{ "id": "8d63125e-6074-4143-befd-1139add7a797", "title": "Qwen3.8-27B MXFP4 + DFlash2 on 2x Radeon AI PRO R9700 (vLLM, TP2)", "date": "2026-10-20", "started_at": "2026-10-20T22:19:51Z", "git_sha": "64c3b48e0642cd5ab5aa0cf53dae7bd78aaaf186", "hardware": "r9700x2", "hardware_label": "vllm-test · desktop (2026) · 2x AMD Radeon AI PRO R9700 32 GB (34.21 GB reported; 210 W cap each) · Ryzen 9 7950X · 64 GB RAM · ASUS ProArt B850 · Proxmox VE 9.2 (kernel 7.0.14-24-pve) · Debian 13 LXC", "engine": "vllm", "harness": "codetwig-vllm-coldwarm-1", "model_family": "qwen3.8", "model_sizes": [ "qwen3.8-27b-mxfp4" ], "task_kind": "chat", "tags": [ "qwen3.8", "vllm", "rocm", "r9700", "mxfp4", "speculative-decoding", "dflash2", "tensor-parallel", "thinking", "contributed" ], "headline": "36 calls across 2 cell(s); ~190.0 tok/s mean (decode, spec-decoding on); p50 1.8s; TTFT ~0.1 s", "janie_blurb_md": null, "janie_blurb_status": "pending", "caveat": "Speculative decoding (DFlash2 draft, 7 tokens) is on, so tokens/sec depends on how predictable the output is, and 'cold' is process-warm (vLLM was not restarted). Not comparable 1:1 with llama.cpp single-GPU runs.", "caveat_severity": "warning", "methodology_ref": "methodology.md (public repo @ 64c3b48)", "methodology_deviations_md": "- **Driver is not `harness/run_benchmark.py`.** The public runner has no cold/warm loop, no streaming/TTFT, and its `frozen` phase ignores the per-prompt `max_tokens` (it sends `num_predict` 2048). To match `methodology.md` and the flagship hello/P-MEDIUM/P-HARD runs, a small stdlib-only wrapper (`codetwig-vllm-coldwarm-1`) imports `harness/prompts.py` unchanged and drives `/v1/chat/completions` with SSE streaming. `git_sha` is the clone HEAD it ran against (64c3b48, clean tree); the wrapper itself is not in the repo.\n- **Prompts and sampling.** Frozen prompts from `prompts.py` with their own `max_tokens` (64/512/1024), temperature 0.1 (`CANONICAL_OPTIONS`). `hello` = frozen P-EASY (\"hi\", max 64), matching the flagship runs' `hello` (same 64-token cap). `num_ctx` has no per-request equivalent on vLLM (server `--max-model-len 262144`). The server's `--override-generation-config` still applies top_p 0.95 / top_k 20 (the request's temperature 0.1 overrides the server's 0.7).\n- **1 cold + 5 warm** per (cell, prompt), as in `methodology.md` §2.1. The flagship runs used 1 + 3.\n- **Cold is `cold_kind: process_warm`.** vLLM is a long-lived daemon that was up since 2026-10-20T17:00:01Z and had already served other traffic. A true cold start would mean restarting the container, which we deliberately did not do. Startup costs from the existing server log (not measured by this run): 118.9 s container start to API ready, 12.6 s model load, 24.8 s engine init (profile, KV cache, warmup, compile from cache). Prefix caching is on, but these prompts are shorter than one cache block: `prefix_cache_hit_tokens` was 0 on every call, so warm calls got no cache advantage.\n- **Speculative decoding is on.** DFlash2 FP8 draft model, 7 speculative tokens. Tokens/sec depends on how predictable the output is: per-call acceptance went from 0.19 (thinking off, P-HARD) to 0.65 (hello). Thinking-on cells post *higher* tok/s than thinking-off for this reason. Per-call `spec_acceptance_rate` comes from vLLM `/metrics` counter deltas.\n- **Two cells from one checkpoint.** `-think` sends `chat_template_kwargs.enable_thinking=true`; `-nothink` sends `false`. With thinking on, the frozen caps truncate output: P-HARD spent all 1024 tokens reasoning on all 6 calls (finish=length, 0 answer chars), and the P-MEDIUM cold call hit 512.\n- **`tokens_per_sec`** follows `methodology.md`: completion_tokens / ((total - first_delta)/1000), where completion_tokens includes reasoning tokens. Because the flagship runs computed completion_tokens / total, `tokens_per_sec_incl_prefill` is also recorded. With TTFT about 0.1 s, the incl-prefill figure is 2–5% lower on P-HARD and thinking-on P-MEDIUM, 10–13% lower on thinking-off P-MEDIUM, and 39–71% lower on hello (11–39 output tokens, so hello tok/s is mostly noise).\n- **`first_delta_ms`** is the first SSE chunk carrying reasoning or content text. `first_chunk_ms` is also recorded. The client sat on a separate box and reached the server through an SSH port forward (ProxyJump over Tailscale DERP), so TTFT includes that tunnel's round trip.\n- **Hardware and runtime context.** Multi-GPU: tensor parallel 2 over 2x Radeon AI PRO R9700, each **power-capped at 210 W** (owner setting, left unchanged). vLLM 0.27.1 runs in Docker (stilldeadcode/vllm-radiance:0.9.3, ROCm 7.14, torch 2.11) inside an unprivileged **Debian 13 LXC container on Proxmox VE 9.2** (host kernel 7.0.14-24-pve), not on bare metal or in a VM. Other serve settings: fp8 KV cache, gpu-memory-utilization 0.98, a custom chat template (`qwen-fixed-v22.3.jinja`), and the R4D attention backend from that image.\n- **`hardware: \"r9700x2\"`** is not in `controlled_vocab.hardware`; the maintainers would need to add it or map it.\n- **Watts are GPU package power.** They come from sysfs `power1_average` at 1 Hz, summed over both cards across each cell's window including gaps between calls. They are not wall power. `memory_mb` is VRAM used summed over both cards; vLLM preallocates about 98%, so it reflects the reservation, not the model's footprint. No `llama_bench` record (llama.cpp-only).\n- **Checkpoint hash.** The main checkpoint's full sha256 was not computed (19.4 GB single file, skipped as heavy). A safetensors-header sha256 and a config.json sha256 are given as fingerprints. The draft model's sha256 is the HF LFS etag from its local download metadata.\n- **Isolation.** Requests were strictly sequential. Before each call the driver waited until vLLM reported 0 running/0 waiting, and it checked `request_success_total` deltas: no foreign completions landed during any call. Server logs for the run window show only these 36 POSTs plus 2 `GET /v1/models` polls from another LAN client.", "results_table": [ { "label": "qwen3.8-27b-mxfp4-dflash2-think", "cell_id": "r9700x2:vllm:qwen3.8-27b-mxfp4-dflash2-think", "n_calls": 18, "n_errors": 0, "duration_ms_mean": 2697.5, "duration_ms_p50": 2711.5, "duration_ms_p95": 5388.65, "tokens_per_sec_mean": 211.06, "tokens_per_sec_p50": 206.9, "tokens_per_sec_p95": 253.5, "tokens_per_sec_max": 254.9, "memory_mb": 64614, "watts_avg": 374.6, "notes": "memory_mb = VRAM used summed over both cards (vLLM preallocates; not model size). watts_avg = summed GPU package power (sysfs power1_average, 1 Hz, n=54, includes inter-call gaps), NOT at-the-wall; max 422 W. Each card capped at 210 W." }, { "label": "qwen3.8-27b-mxfp4-dflash2-nothink", "cell_id": "r9700x2:vllm:qwen3.8-27b-mxfp4-dflash2-nothink", "n_calls": 18, "n_errors": 0, "duration_ms_mean": 1484.67, "duration_ms_p50": 901.0, "duration_ms_p95": 3537.35, "tokens_per_sec_mean": 168.91, "tokens_per_sec_p50": 148.81, "tokens_per_sec_p95": 261.9, "tokens_per_sec_max": 261.9, "memory_mb": 64614, "watts_avg": 363.3, "notes": "memory_mb = VRAM used summed over both cards (vLLM preallocates; not model size). watts_avg = summed GPU package power (sysfs power1_average, 1 Hz, n=31, includes inter-call gaps), NOT at-the-wall; max 429 W. Each card capped at 210 W." } ], "cells": [ { "cell_id": "r9700x2:vllm:qwen3.8-27b-mxfp4-dflash2-think", "n_calls": 18, "n_errors": 0, "duration_ms_mean": 2697.5, "duration_ms_p50": 2711.5, "duration_ms_p95": 5388.65, "tokens_per_sec_mean": 211.06, "tokens_per_sec_p50": 206.9, "tokens_per_sec_p95": 253.5, "tokens_per_sec_max": 254.9, "cold_warm_split": { "cold": { "n_calls": 3, "duration_ms_mean": 2813.67, "duration_ms_p50": 3177.0, "tokens_per_sec_mean": 207.18, "first_delta_ms_mean": 108, "first_delta_ms_p50": 109.0, "first_delta_ms_p95": 109.9, "tokens_per_sec_incl_prefill_mean": 170.55 }, "warm": { "n_calls": 15, "duration_ms_mean": 2674.27, "duration_ms_p50": 2654.0, "tokens_per_sec_mean": 211.84, "first_delta_ms_mean": 104.2, "first_delta_ms_p50": 102.0, "first_delta_ms_p95": 115.9, "tokens_per_sec_incl_prefill_mean": 174.26 } }, "per_prompt": { "hello": { "n_calls": 6, "n_errors": 0, "duration_ms_mean": 260.67, "duration_ms_p50": 261.0, "tokens_per_sec_mean": 250.87, "tokens_per_sec_p50": 253.25, "cold": { "first_delta_ms": 109, "total_duration_ms": 268, "completion_tokens": 39, "tokens_per_sec": 245.28, "finish_reason": "stop" }, "warm": { "first_delta_ms_p50": 104.0, "first_delta_ms_p95": 109.8, "total_duration_ms_p50": 259.0, "total_duration_ms_p95": 263.8, "tokens_per_sec_mean": 251.99, "completion_tokens_mean": 39, "success_rate": 1.0, "finish_length_count": 0 }, "spec_acceptance_rate_mean": 0.65 }, "P-MEDIUM": { "n_calls": 6, "n_errors": 0, "duration_ms_mean": 2722.33, "duration_ms_p50": 2711.5, "tokens_per_sec_mean": 177.22, "tokens_per_sec_p50": 176.43, "cold": { "first_delta_ms": 105, "total_duration_ms": 3177, "completion_tokens": 512, "tokens_per_sec": 166.67, "finish_reason": "length" }, "warm": { "first_delta_ms_p50": 98.0, "first_delta_ms_p95": 114.8, "total_duration_ms_p50": 2654.0, "total_duration_ms_p95": 2774.6, "tokens_per_sec_mean": 179.33, "completion_tokens_mean": 452.8, "success_rate": 1.0, "finish_length_count": 0 }, "spec_acceptance_rate_mean": 0.43 }, "P-HARD": { "n_calls": 6, "n_errors": 0, "duration_ms_mean": 5109.5, "duration_ms_p50": 5059.5, "tokens_per_sec_mean": 205.11, "tokens_per_sec_p50": 206.9, "cold": { "first_delta_ms": 110, "total_duration_ms": 4996, "completion_tokens": 1024, "tokens_per_sec": 209.58, "finish_reason": "length" }, "warm": { "first_delta_ms_p50": 102.0, "first_delta_ms_p95": 113.6, "total_duration_ms_p50": 5123.0, "total_duration_ms_p95": 5408.8, "tokens_per_sec_mean": 204.21, "completion_tokens_mean": 1024, "success_rate": 1.0, "finish_length_count": 5 }, "spec_acceptance_rate_mean": 0.53 } }, "chars_split": { "has_thinking": true, "reasoning_chars_mean": 1439, "answer_chars_mean": 224.78 }, "timeline": [ { "prompt_id": "hello", "run_idx": 0, "phase": "cold", "duration_ms": 268, "tokens_per_sec": 245.2830188679245, "error": null }, { "prompt_id": "hello", "run_idx": 1, "phase": "warm", "duration_ms": 259, "tokens_per_sec": 245.2830188679245, "error": null }, { "prompt_id": "hello", "run_idx": 2, "phase": "warm", "duration_ms": 264, "tokens_per_sec": 253.24675324675326, "error": null }, { "prompt_id": "hello", "run_idx": 3, "phase": "warm", "duration_ms": 258, "tokens_per_sec": 253.24675324675326, "error": null }, { "prompt_id": "hello", "run_idx": 4, "phase": "warm", "duration_ms": 263, "tokens_per_sec": 253.24675324675326, "error": null }, { "prompt_id": "hello", "run_idx": 5, "phase": "warm", "duration_ms": 252, "tokens_per_sec": 254.90196078431373, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 0, "phase": "cold", "duration_ms": 3177, "tokens_per_sec": 166.66666666666666, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 1, "phase": "warm", "duration_ms": 2769, "tokens_per_sec": 179.70797454137028, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 2, "phase": "warm", "duration_ms": 2517, "tokens_per_sec": 189.74782968168665, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 3, "phase": "warm", "duration_ms": 2654, "tokens_per_sec": 169.79655712050078, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 4, "phase": "warm", "duration_ms": 2441, "tokens_per_sec": 184.24451140766251, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 5, "phase": "warm", "duration_ms": 2776, "tokens_per_sec": 173.1488406881077, "error": null }, { "prompt_id": "P-HARD", "run_idx": 0, "phase": "cold", "duration_ms": 4996, "tokens_per_sec": 209.57838722881704, "error": null }, { "prompt_id": "P-HARD", "run_idx": 1, "phase": "warm", "duration_ms": 4747, "tokens_per_sec": 220.4520990312164, "error": null }, { "prompt_id": "P-HARD", "run_idx": 2, "phase": "warm", "duration_ms": 5123, "tokens_per_sec": 204.47284345047925, "error": null }, { "prompt_id": "P-HARD", "run_idx": 3, "phase": "warm", "duration_ms": 5415, "tokens_per_sec": 192.73480143045362, "error": null }, { "prompt_id": "P-HARD", "run_idx": 4, "phase": "warm", "duration_ms": 4992, "tokens_per_sec": 209.32134096484054, "error": null }, { "prompt_id": "P-HARD", "run_idx": 5, "phase": "warm", "duration_ms": 5384, "tokens_per_sec": 194.08642911296437, "error": null } ] }, { "cell_id": "r9700x2:vllm:qwen3.8-27b-mxfp4-dflash2-nothink", "n_calls": 18, "n_errors": 0, "duration_ms_mean": 1484.67, "duration_ms_p50": 901.0, "duration_ms_p95": 3537.35, "tokens_per_sec_mean": 168.91, "tokens_per_sec_p50": 148.81, "tokens_per_sec_p95": 261.9, "tokens_per_sec_max": 261.9, "cold_warm_split": { "cold": { "n_calls": 3, "duration_ms_mean": 1404, "duration_ms_p50": 890.0, "tokens_per_sec_mean": 169.45, "first_delta_ms_mean": 101.33, "first_delta_ms_p50": 101.0, "first_delta_ms_p95": 105.5, "tokens_per_sec_incl_prefill_mean": 107.03 }, "warm": { "n_calls": 15, "duration_ms_mean": 1500.8, "duration_ms_p50": 912.0, "tokens_per_sec_mean": 168.81, "first_delta_ms_mean": 101.33, "first_delta_ms_p50": 101.0, "first_delta_ms_p95": 112.5, "tokens_per_sec_incl_prefill_mean": 105.75 } }, "per_prompt": { "hello": { "n_calls": 6, "n_errors": 0, "duration_ms_mean": 140.17, "duration_ms_p50": 140.5, "tokens_per_sec_mean": 246.83, "tokens_per_sec_p50": 244.44, "cold": { "first_delta_ms": 97, "total_duration_ms": 142, "completion_tokens": 11, "tokens_per_sec": 244.44, "finish_reason": "stop" }, "warm": { "first_delta_ms_p50": 94.0, "first_delta_ms_p95": 100.6, "total_duration_ms_p50": 140.0, "total_duration_ms_p95": 143.4, "tokens_per_sec_mean": 247.31, "completion_tokens_mean": 11, "success_rate": 1.0, "finish_length_count": 0 }, "spec_acceptance_rate_mean": 0.64 }, "P-MEDIUM": { "n_calls": 6, "n_errors": 0, "duration_ms_mean": 907.83, "duration_ms_p50": 901.0, "tokens_per_sec_mean": 150.19, "tokens_per_sec_p50": 148.81, "cold": { "first_delta_ms": 101, "total_duration_ms": 890, "completion_tokens": 113, "tokens_per_sec": 143.22, "finish_reason": "stop" }, "warm": { "first_delta_ms_p50": 101.0, "first_delta_ms_p95": 104.6, "total_duration_ms_p50": 912.0, "total_duration_ms_p95": 1063.2, "tokens_per_sec_mean": 151.58, "completion_tokens_mean": 121.4, "success_rate": 1.0, "finish_length_count": 0 }, "spec_acceptance_rate_mean": 0.34 }, "P-HARD": { "n_calls": 6, "n_errors": 0, "duration_ms_mean": 3406, "duration_ms_p50": 3417.5, "tokens_per_sec_mean": 109.72, "tokens_per_sec_p50": 109.19, "cold": { "first_delta_ms": 106, "total_duration_ms": 3180, "completion_tokens": 371, "tokens_per_sec": 120.69, "finish_reason": "stop" }, "warm": { "first_delta_ms_p50": 108.0, "first_delta_ms_p95": 115.0, "total_duration_ms_p50": 3445.0, "total_duration_ms_p95": 3608.2, "tokens_per_sec_mean": 107.53, "completion_tokens_mean": 359.2, "success_rate": 1.0, "finish_length_count": 0 }, "spec_acceptance_rate_mean": 0.21 } }, "chars_split": { "has_thinking": false, "reasoning_chars_mean": 0, "answer_chars_mean": 735.17 }, "timeline": [ { "prompt_id": "hello", "run_idx": 0, "phase": "cold", "duration_ms": 142, "tokens_per_sec": 244.44444444444446, "error": null }, { "prompt_id": "hello", "run_idx": 1, "phase": "warm", "duration_ms": 136, "tokens_per_sec": 261.90476190476187, "error": null }, { "prompt_id": "hello", "run_idx": 2, "phase": "warm", "duration_ms": 140, "tokens_per_sec": 244.44444444444446, "error": null }, { "prompt_id": "hello", "run_idx": 3, "phase": "warm", "duration_ms": 141, "tokens_per_sec": 229.16666666666666, "error": null }, { "prompt_id": "hello", "run_idx": 4, "phase": "warm", "duration_ms": 144, "tokens_per_sec": 261.90476190476187, "error": null }, { "prompt_id": "hello", "run_idx": 5, "phase": "warm", "duration_ms": 138, "tokens_per_sec": 239.1304347826087, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 0, "phase": "cold", "duration_ms": 890, "tokens_per_sec": 143.21926489226868, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 1, "phase": "warm", "duration_ms": 1028, "tokens_per_sec": 145.94594594594594, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 2, "phase": "warm", "duration_ms": 1072, "tokens_per_sec": 132.36814891416753, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 3, "phase": "warm", "duration_ms": 795, "tokens_per_sec": 159.48275862068968, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 4, "phase": "warm", "duration_ms": 750, "tokens_per_sec": 168.4532924961715, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 5, "phase": "warm", "duration_ms": 912, "tokens_per_sec": 151.66461159062885, "error": null }, { "prompt_id": "P-HARD", "run_idx": 0, "phase": "cold", "duration_ms": 3180, "tokens_per_sec": 120.6896551724138, "error": null }, { "prompt_id": "P-HARD", "run_idx": 1, "phase": "warm", "duration_ms": 3521, "tokens_per_sec": 108.50439882697947, "error": null }, { "prompt_id": "P-HARD", "run_idx": 2, "phase": "warm", "duration_ms": 3445, "tokens_per_sec": 103.98561582259514, "error": null }, { "prompt_id": "P-HARD", "run_idx": 3, "phase": "warm", "duration_ms": 3270, "tokens_per_sec": 109.88317019261132, "error": null }, { "prompt_id": "P-HARD", "run_idx": 4, "phase": "warm", "duration_ms": 3390, "tokens_per_sec": 112.40073304825901, "error": null }, { "prompt_id": "P-HARD", "run_idx": 5, "phase": "warm", "duration_ms": 3630, "tokens_per_sec": 102.86200056673279, "error": null } ] } ], "n_calls_total": 36, "n_errors_total": 0, "chart_spec": { "kind": "bar", "x": "cell", "y": "tokens_per_sec_mean", "series": [ "qwen3.8-27b-mxfp4-dflash2-think", "qwen3.8-27b-mxfp4-dflash2-nothink" ], "data_url": "/data/8d63125e-6074-4143-befd-1139add7a797/run.jsonl" }, "site_grade": null, "site_grade_reason": "contributed run; grade is the maintainers' call", "raw_data_urls": { "jsonl": "/data/8d63125e-6074-4143-befd-1139add7a797/run.jsonl", "log": "/data/8d63125e-6074-4143-befd-1139add7a797/run.log", "md": "/data/8d63125e-6074-4143-befd-1139add7a797/run.md", "metadata": "/data/8d63125e-6074-4143-befd-1139add7a797/metadata.json" }, "synthesis_doc": null, "synthesis_docs_all": [], "related_ids": [ "09d8fbde-0008-49bb-99da-03eeaca72be1", "fba9d9b1-cc5d-40bc-9e21-beafbb72c65d", "5fb2913d-6500-4ecf-9e97-d43f7dd61145", "23066b38-ea9c-4dd3-b2f5-32912a67fce4" ], "status": "complete", "visibility": "draft", "cells_full": [ { "cell_id": "r9700x2:vllm:qwen3.8-27b-mxfp4-dflash2-think", "n_calls": 18, "n_errors": 0, "duration_ms_mean": 2697.5, "duration_ms_p50": 2711.5, "duration_ms_p95": 5388.65, "tokens_per_sec_mean": 211.06, "tokens_per_sec_p50": 206.9, "tokens_per_sec_p95": 253.5, "tokens_per_sec_max": 254.9, "cold_warm_split": { "cold": { "n_calls": 3, "duration_ms_mean": 2813.67, "duration_ms_p50": 3177.0, "tokens_per_sec_mean": 207.18, "first_delta_ms_mean": 108, "first_delta_ms_p50": 109.0, "first_delta_ms_p95": 109.9, "tokens_per_sec_incl_prefill_mean": 170.55 }, "warm": { "n_calls": 15, "duration_ms_mean": 2674.27, "duration_ms_p50": 2654.0, "tokens_per_sec_mean": 211.84, "first_delta_ms_mean": 104.2, "first_delta_ms_p50": 102.0, "first_delta_ms_p95": 115.9, "tokens_per_sec_incl_prefill_mean": 174.26 } }, "per_prompt": { "hello": { "n_calls": 6, "n_errors": 0, "duration_ms_mean": 260.67, "duration_ms_p50": 261.0, "tokens_per_sec_mean": 250.87, "tokens_per_sec_p50": 253.25, "cold": { "first_delta_ms": 109, "total_duration_ms": 268, "completion_tokens": 39, "tokens_per_sec": 245.28, "finish_reason": "stop" }, "warm": { "first_delta_ms_p50": 104.0, "first_delta_ms_p95": 109.8, "total_duration_ms_p50": 259.0, "total_duration_ms_p95": 263.8, "tokens_per_sec_mean": 251.99, "completion_tokens_mean": 39, "success_rate": 1.0, "finish_length_count": 0 }, "spec_acceptance_rate_mean": 0.65 }, "P-MEDIUM": { "n_calls": 6, "n_errors": 0, "duration_ms_mean": 2722.33, "duration_ms_p50": 2711.5, "tokens_per_sec_mean": 177.22, "tokens_per_sec_p50": 176.43, "cold": { "first_delta_ms": 105, "total_duration_ms": 3177, "completion_tokens": 512, "tokens_per_sec": 166.67, "finish_reason": "length" }, "warm": { "first_delta_ms_p50": 98.0, "first_delta_ms_p95": 114.8, "total_duration_ms_p50": 2654.0, "total_duration_ms_p95": 2774.6, "tokens_per_sec_mean": 179.33, "completion_tokens_mean": 452.8, "success_rate": 1.0, "finish_length_count": 0 }, "spec_acceptance_rate_mean": 0.43 }, "P-HARD": { "n_calls": 6, "n_errors": 0, "duration_ms_mean": 5109.5, "duration_ms_p50": 5059.5, "tokens_per_sec_mean": 205.11, "tokens_per_sec_p50": 206.9, "cold": { "first_delta_ms": 110, "total_duration_ms": 4996, "completion_tokens": 1024, "tokens_per_sec": 209.58, "finish_reason": "length" }, "warm": { "first_delta_ms_p50": 102.0, "first_delta_ms_p95": 113.6, "total_duration_ms_p50": 5123.0, "total_duration_ms_p95": 5408.8, "tokens_per_sec_mean": 204.21, "completion_tokens_mean": 1024, "success_rate": 1.0, "finish_length_count": 5 }, "spec_acceptance_rate_mean": 0.53 } }, "chars_split": { "has_thinking": true, "reasoning_chars_mean": 1439, "answer_chars_mean": 224.78 }, "timeline": [ { "prompt_id": "hello", "run_idx": 0, "phase": "cold", "duration_ms": 268, "tokens_per_sec": 245.2830188679245, "error": null }, { "prompt_id": "hello", "run_idx": 1, "phase": "warm", "duration_ms": 259, "tokens_per_sec": 245.2830188679245, "error": null }, { "prompt_id": "hello", "run_idx": 2, "phase": "warm", "duration_ms": 264, "tokens_per_sec": 253.24675324675326, "error": null }, { "prompt_id": "hello", "run_idx": 3, "phase": "warm", "duration_ms": 258, "tokens_per_sec": 253.24675324675326, "error": null }, { "prompt_id": "hello", "run_idx": 4, "phase": "warm", "duration_ms": 263, "tokens_per_sec": 253.24675324675326, "error": null }, { "prompt_id": "hello", "run_idx": 5, "phase": "warm", "duration_ms": 252, "tokens_per_sec": 254.90196078431373, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 0, "phase": "cold", "duration_ms": 3177, "tokens_per_sec": 166.66666666666666, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 1, "phase": "warm", "duration_ms": 2769, "tokens_per_sec": 179.70797454137028, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 2, "phase": "warm", "duration_ms": 2517, "tokens_per_sec": 189.74782968168665, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 3, "phase": "warm", "duration_ms": 2654, "tokens_per_sec": 169.79655712050078, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 4, "phase": "warm", "duration_ms": 2441, "tokens_per_sec": 184.24451140766251, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 5, "phase": "warm", "duration_ms": 2776, "tokens_per_sec": 173.1488406881077, "error": null }, { "prompt_id": "P-HARD", "run_idx": 0, "phase": "cold", "duration_ms": 4996, "tokens_per_sec": 209.57838722881704, "error": null }, { "prompt_id": "P-HARD", "run_idx": 1, "phase": "warm", "duration_ms": 4747, "tokens_per_sec": 220.4520990312164, "error": null }, { "prompt_id": "P-HARD", "run_idx": 2, "phase": "warm", "duration_ms": 5123, "tokens_per_sec": 204.47284345047925, "error": null }, { "prompt_id": "P-HARD", "run_idx": 3, "phase": "warm", "duration_ms": 5415, "tokens_per_sec": 192.73480143045362, "error": null }, { "prompt_id": "P-HARD", "run_idx": 4, "phase": "warm", "duration_ms": 4992, "tokens_per_sec": 209.32134096484054, "error": null }, { "prompt_id": "P-HARD", "run_idx": 5, "phase": "warm", "duration_ms": 5384, "tokens_per_sec": 194.08642911296437, "error": null } ] }, { "cell_id": "r9700x2:vllm:qwen3.8-27b-mxfp4-dflash2-nothink", "n_calls": 18, "n_errors": 0, "duration_ms_mean": 1484.67, "duration_ms_p50": 901.0, "duration_ms_p95": 3537.35, "tokens_per_sec_mean": 168.91, "tokens_per_sec_p50": 148.81, "tokens_per_sec_p95": 261.9, "tokens_per_sec_max": 261.9, "cold_warm_split": { "cold": { "n_calls": 3, "duration_ms_mean": 1404, "duration_ms_p50": 890.0, "tokens_per_sec_mean": 169.45, "first_delta_ms_mean": 101.33, "first_delta_ms_p50": 101.0, "first_delta_ms_p95": 105.5, "tokens_per_sec_incl_prefill_mean": 107.03 }, "warm": { "n_calls": 15, "duration_ms_mean": 1500.8, "duration_ms_p50": 912.0, "tokens_per_sec_mean": 168.81, "first_delta_ms_mean": 101.33, "first_delta_ms_p50": 101.0, "first_delta_ms_p95": 112.5, "tokens_per_sec_incl_prefill_mean": 105.75 } }, "per_prompt": { "hello": { "n_calls": 6, "n_errors": 0, "duration_ms_mean": 140.17, "duration_ms_p50": 140.5, "tokens_per_sec_mean": 246.83, "tokens_per_sec_p50": 244.44, "cold": { "first_delta_ms": 97, "total_duration_ms": 142, "completion_tokens": 11, "tokens_per_sec": 244.44, "finish_reason": "stop" }, "warm": { "first_delta_ms_p50": 94.0, "first_delta_ms_p95": 100.6, "total_duration_ms_p50": 140.0, "total_duration_ms_p95": 143.4, "tokens_per_sec_mean": 247.31, "completion_tokens_mean": 11, "success_rate": 1.0, "finish_length_count": 0 }, "spec_acceptance_rate_mean": 0.64 }, "P-MEDIUM": { "n_calls": 6, "n_errors": 0, "duration_ms_mean": 907.83, "duration_ms_p50": 901.0, "tokens_per_sec_mean": 150.19, "tokens_per_sec_p50": 148.81, "cold": { "first_delta_ms": 101, "total_duration_ms": 890, "completion_tokens": 113, "tokens_per_sec": 143.22, "finish_reason": "stop" }, "warm": { "first_delta_ms_p50": 101.0, "first_delta_ms_p95": 104.6, "total_duration_ms_p50": 912.0, "total_duration_ms_p95": 1063.2, "tokens_per_sec_mean": 151.58, "completion_tokens_mean": 121.4, "success_rate": 1.0, "finish_length_count": 0 }, "spec_acceptance_rate_mean": 0.34 }, "P-HARD": { "n_calls": 6, "n_errors": 0, "duration_ms_mean": 3406, "duration_ms_p50": 3417.5, "tokens_per_sec_mean": 109.72, "tokens_per_sec_p50": 109.19, "cold": { "first_delta_ms": 106, "total_duration_ms": 3180, "completion_tokens": 371, "tokens_per_sec": 120.69, "finish_reason": "stop" }, "warm": { "first_delta_ms_p50": 108.0, "first_delta_ms_p95": 115.0, "total_duration_ms_p50": 3445.0, "total_duration_ms_p95": 3608.2, "tokens_per_sec_mean": 107.53, "completion_tokens_mean": 359.2, "success_rate": 1.0, "finish_length_count": 0 }, "spec_acceptance_rate_mean": 0.21 } }, "chars_split": { "has_thinking": false, "reasoning_chars_mean": 0, "answer_chars_mean": 735.17 }, "timeline": [ { "prompt_id": "hello", "run_idx": 0, "phase": "cold", "duration_ms": 142, "tokens_per_sec": 244.44444444444446, "error": null }, { "prompt_id": "hello", "run_idx": 1, "phase": "warm", "duration_ms": 136, "tokens_per_sec": 261.90476190476187, "error": null }, { "prompt_id": "hello", "run_idx": 2, "phase": "warm", "duration_ms": 140, "tokens_per_sec": 244.44444444444446, "error": null }, { "prompt_id": "hello", "run_idx": 3, "phase": "warm", "duration_ms": 141, "tokens_per_sec": 229.16666666666666, "error": null }, { "prompt_id": "hello", "run_idx": 4, "phase": "warm", "duration_ms": 144, "tokens_per_sec": 261.90476190476187, "error": null }, { "prompt_id": "hello", "run_idx": 5, "phase": "warm", "duration_ms": 138, "tokens_per_sec": 239.1304347826087, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 0, "phase": "cold", "duration_ms": 890, "tokens_per_sec": 143.21926489226868, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 1, "phase": "warm", "duration_ms": 1028, "tokens_per_sec": 145.94594594594594, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 2, "phase": "warm", "duration_ms": 1072, "tokens_per_sec": 132.36814891416753, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 3, "phase": "warm", "duration_ms": 795, "tokens_per_sec": 159.48275862068968, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 4, "phase": "warm", "duration_ms": 750, "tokens_per_sec": 168.4532924961715, "error": null }, { "prompt_id": "P-MEDIUM", "run_idx": 5, "phase": "warm", "duration_ms": 912, "tokens_per_sec": 151.66461159062885, "error": null }, { "prompt_id": "P-HARD", "run_idx": 0, "phase": "cold", "duration_ms": 3180, "tokens_per_sec": 120.6896551724138, "error": null }, { "prompt_id": "P-HARD", "run_idx": 1, "phase": "warm", "duration_ms": 3521, "tokens_per_sec": 108.50439882697947, "error": null }, { "prompt_id": "P-HARD", "run_idx": 2, "phase": "warm", "duration_ms": 3445, "tokens_per_sec": 103.98561582259514, "error": null }, { "prompt_id": "P-HARD", "run_idx": 3, "phase": "warm", "duration_ms": 3270, "tokens_per_sec": 109.88317019261132, "error": null }, { "prompt_id": "P-HARD", "run_idx": 4, "phase": "warm", "duration_ms": 3390, "tokens_per_sec": 112.40073304825901, "error": null }, { "prompt_id": "P-HARD", "run_idx": 5, "phase": "warm", "duration_ms": 3630, "tokens_per_sec": 102.86200056673279, "error": null } ] } ], "meta_record": { "type": "meta", "benchmark_run_id": "8d63125e-6074-4143-befd-1139add7a797", "harness_version": "codetwig-vllm-coldwarm-1", "started_at_utc": "2026-10-20T22:19:51Z", "host": "codetwig-box (driver; requests via ssh -L to vllm-test)", "node": "vllm-test", "engine": "vllm", "target_url": "http://127.0.0.1:18080 (ssh -L 18080:localhost:8080 vllm-test)", "git_sha": "64c3b48e0642cd5ab5aa0cf53dae7bd78aaaf186", "load_avg": [ 0.0458984375, 0.20654296875, 0.2685546875 ], "served_model": "Qwen3.8-MXFP4", "temperature": 0.1, "n_warm": 5, "max_tokens_per_prompt": { "hello": 64, "P-MEDIUM": 512, "P-HARD": 1024 }, "prompt_map": { "hello": "P-EASY", "P-MEDIUM": "P-MEDIUM", "P-HARD": "P-HARD" }, "cells": [ { "key": "think", "cell_id": "r9700x2:vllm:qwen3.8-27b-mxfp4-dflash2-think", "enable_thinking": true }, { "key": "nothink", "cell_id": "r9700x2:vllm:qwen3.8-27b-mxfp4-dflash2-nothink", "enable_thinking": false } ] }, "packaged_at": "2026-10-20T22:23:24Z", "environment": { "engine_version": "vLLM 0.27.1", "container_image": "stilldeadcode/vllm-radiance:0.9.3", "rocm": "7.14 (HIP 7.14.60850)", "torch": "2.11.0+rocm7.14", "host_os": "Proxmox VE 9.2 (pve-manager 9.2.21), running kernel 7.0.14-24-pve (verified read-only via uname -r)", "guest": "Debian 13 unprivileged LXC (CT 113, 8 cores, 32 GB RAM limit), vLLM in Docker inside the LXC", "cpu": "AMD Ryzen 9 7950X (16C/32T)", "ram": "64 GB host", "motherboard": "ASUS ProArt B850", "gpus": "2x AMD Radeon AI PRO R9700 32 GB (rocm-smi: 34.21 GB each), power cap 210 W each (sysfs power1_cap 210000000 uW)", "serve_flags": "--tensor-parallel-size 2 --kv-cache-dtype fp8 --gpu-memory-utilization 0.98 --kv-cache-memory 18563072000 --max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 8192 --attention-backend R4D --enable-prefix-caching --mamba-cache-mode align --reasoning-parser qwen3 --chat-template /patches/qwen-fixed-v22.3.jinja --override-generation-config {temperature:0.7,top_p:0.95,top_k:20} --speculative-config {method:dflash, model:Qwen3.8-27B-DFlash2-FP8, num_speculative_tokens:7}", "server_started_at_utc": "2026-10-20T16:58:03Z", "server_ready_at_utc": "2026-10-20T17:00:01Z", "server_boot_to_ready_s_from_logs": 118.9, "model_loading_s_from_logs": 12.56, "init_engine_s_from_logs": 24.82, "driver": "box (codetwig-box) -> ssh -L 18080:localhost:8080 vllm-test (ProxyJump over tailscale DERP); TTFT includes this tunnel's round trip" }, "model": { "checkpoint": "Qwen3.8-27B-MXFP4-mtpfp8 (local dir; architectures Qwen3_5ForConditionalGeneration, model_type qwen3_5)", "quantization": "AMD Quark MXFP4: fp4 weights and dynamic fp4 activations, per-group 32, e8m0 scales; visual tower excluded; MTP fc in fp8_e4m3", "checkpoint_size_bytes": 19373796656, "checkpoint_sha256": null, "checkpoint_sha256_note": "Full-file sha256 NOT computed (single 19.4 GB file; skipped as too heavy for a read-only pass on a shared machine). Fingerprints instead: safetensors header sha256 b8d7088a40a3e79e80bd21c8b36d051fedf2e402435e6983a68f067fb2504ec7 (header_len 235336); config.json sha256 6a3aad4b9e457ef95a4c3b9c63ca531212f05cf6fc6723aba2a52e49f21f4e78. No HF cache metadata/upstream commit present for this dir.", "draft_model": "Qwen3.8-27B-DFlash2-FP8 (FP8 e4m3 block-128 of incoai/Qwen3.8-27B-DFlash2), HF commit f02593d0ea2e3b7eedbb831fb26f465b1e6238bb", "draft_model_safetensors_sha256": "7dbb99a8d0120f502e66b256aa7c0866d933ceeee4a02463d9db591811e8404e (HF LFS etag from local .cache/huggingface metadata; not re-hashed)", "draft_config_sha256": "5b5668a00b26aaebd88c7e3d961f7d1cdef025867fee158dfccb84f29fd8caec" } }