# Qwen3.8-27B MXFP4 + DFlash2 on 2x Radeon AI PRO R9700 (vLLM, TP2) — 8d63125e-6074-4143-befd-1139add7a797

- Started: 2026-10-20T22:19:51Z
- Driver host: codetwig-box (requests via `ssh -L 18080:localhost:8080 vllm-test`)
- Node: vllm-test (Proxmox LXC 113) → vLLM OpenAI server on :8080
- Engine: vllm 0.27.1 (Docker `stilldeadcode/vllm-radiance:0.9.3`, ROCm 7.14, torch 2.11)
- Harness: codetwig-vllm-coldwarm-1 (imports `harness/prompts.py` @ 64c3b48, unchanged)
- Hardware: vllm-test · desktop (2026) · 2x AMD Radeon AI PRO R9700 32 GB (34.21 GB reported; 210 W cap each) · Ryzen 9 7950X · 64 GB RAM · ASUS ProArt B850 · Proxmox VE 9.2 (kernel 7.0.14-24-pve) · Debian 13 LXC

## Configurations

| Key | Cell | Checkpoint | Thinking | Sampling | Spec decoding |
|---|---|---|---|---|---|
| think | `r9700x2:vllm:qwen3.8-27b-mxfp4-dflash2-think` | Qwen3.8-27B-MXFP4-mtpfp8 (Quark MXFP4) | enable_thinking=true | T=0.1 (request), top_p=0.95 top_k=20 (server override) | DFlash2 FP8 draft, 7 tokens |
| nothink | `r9700x2:vllm:qwen3.8-27b-mxfp4-dflash2-nothink` | Qwen3.8-27B-MXFP4-mtpfp8 (Quark MXFP4) | enable_thinking=false | T=0.1 (request), top_p=0.95 top_k=20 (server override) | DFlash2 FP8 draft, 7 tokens |

Serve: TP=2, fp8 KV cache, gpu-memory-utilization 0.98, max-model-len 262144, prefix caching on, reasoning-parser qwen3.

## VRAM / power snapshot (pre-cell, both cards)

| Cell | VRAM used (GiB, per card) | VRAM total (GiB) | Power now (W) | Power cap (W) |
|---|---|---|---|---|
| think | 31.55 / 31.55 | 31.86 | 17 / 16 | 210 / 210 |
| nothink | 31.55 / 31.55 | 31.86 | 36 / 35 | 210 / 210 |

## Per-prompt latency, throughput, and thinking-vs-answer split

`tokens_per_sec` = completion_tokens / ((total − first_delta)/1000) (methodology.md). Warm = mean/p50 over 5 warm calls.

| Cell | Prompt | Cold TTFT (ms) | Cold dur (ms) | Cold tok/s | Warm TTFT p50 (ms) | Warm tok/s mean | Warm dur p50 (ms) | Warm completion tok (mean) | Finish=length (cold+warm) | Spec accept | Reasoning chars (warm mean) | Answer chars (warm mean) | Errors |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| qwen3.8-27b-mxfp4-dflash2-think | hello | 109 | 268 | 245.3 | 104 | 252.0 | 259 | 39 | 0/6 | 0.65 | 121 | 37 | 0 |
| qwen3.8-27b-mxfp4-dflash2-think | P-MEDIUM | 105 | 3177 | 166.7 | 98 | 179.3 | 2654 | 453 | 1/6 | 0.43 | 1338 | 762 | 0 |
| qwen3.8-27b-mxfp4-dflash2-think | P-HARD | 110 | 4996 | 209.6 | 102 | 204.2 | 5123 | 1024 | 6/6 | 0.53 | 2684 | 0 | 0 |
| qwen3.8-27b-mxfp4-dflash2-nothink | hello | 97 | 142 | 244.4 | 94 | 247.3 | 140 | 11 | 0/6 | 0.64 | 0 | 35 | 0 |
| qwen3.8-27b-mxfp4-dflash2-nothink | P-MEDIUM | 101 | 890 | 143.2 | 101 | 151.6 | 912 | 121 | 0/6 | 0.34 | 0 | 647 | 0 |
| qwen3.8-27b-mxfp4-dflash2-nothink | P-HARD | 106 | 3180 | 120.7 | 108 | 107.5 | 3445 | 359 | 0/6 | 0.21 | 0 | 1533 | 0 |

## Cold vs warm (per cell)

| Cell | cold n | cold tok/s | cold TTFT mean (ms) | warm n | warm tok/s | warm TTFT p50 (ms) | warm/cold tok/s |
|---|---|---|---|---|---|---|---|
| qwen3.8-27b-mxfp4-dflash2-think | 3 | 207.2 | 108 | 15 | 211.8 | 102 | 1.02× |
| qwen3.8-27b-mxfp4-dflash2-nothink | 3 | 169.4 | 101 | 15 | 168.8 | 101 | 1.00× |

Cold here is `cold_kind: process_warm`. vLLM was not restarted. From the existing server log (not measured by this run): container start → API ready 118.9 s, model load 12.6 s, engine init 24.8 s.

## Power (GPU package, not wall)

- `qwen3.8-27b-mxfp4-dflash2-think`: 374.6 W mean, summed over both cards (sysfs `power1_average`, 1 Hz, including gaps between calls). Each card is capped at 210 W.
- `qwen3.8-27b-mxfp4-dflash2-nothink`: 363.3 W mean, summed over both cards (sysfs `power1_average`, 1 Hz, including gaps between calls). Each card is capped at 210 W.

## Methodology deviations

- **Driver is not `harness/run_benchmark.py`.** The public runner has no cold/warm loop, no streaming/TTFT, and its `frozen` phase ignores the per-prompt `max_tokens` (it sends `num_predict` 2048). To match `methodology.md` and the flagship hello/P-MEDIUM/P-HARD runs, a small stdlib-only wrapper (`codetwig-vllm-coldwarm-1`) imports `harness/prompts.py` unchanged and drives `/v1/chat/completions` with SSE streaming. `git_sha` is the clone HEAD it ran against (64c3b48, clean tree); the wrapper itself is not in the repo.
- **Prompts and sampling.** Frozen prompts from `prompts.py` with their own `max_tokens` (64/512/1024), temperature 0.1 (`CANONICAL_OPTIONS`). `hello` = frozen P-EASY ("hi", max 64), matching the flagship runs' `hello` (same 64-token cap). `num_ctx` has no per-request equivalent on vLLM (server `--max-model-len 262144`). The server's `--override-generation-config` still applies top_p 0.95 / top_k 20 (the request's temperature 0.1 overrides the server's 0.7).
- **1 cold + 5 warm** per (cell, prompt), as in `methodology.md` §2.1. The flagship runs used 1 + 3.
- **Cold is `cold_kind: process_warm`.** vLLM is a long-lived daemon that was up since 2026-10-20T17:00:01Z and had already served other traffic. A true cold start would mean restarting the container, which we deliberately did not do. Startup costs from the existing server log (not measured by this run): 118.9 s container start to API ready, 12.6 s model load, 24.8 s engine init (profile, KV cache, warmup, compile from cache). Prefix caching is on, but these prompts are shorter than one cache block: `prefix_cache_hit_tokens` was 0 on every call, so warm calls got no cache advantage.
- **Speculative decoding is on.** DFlash2 FP8 draft model, 7 speculative tokens. Tokens/sec depends on how predictable the output is: per-call acceptance went from 0.19 (thinking off, P-HARD) to 0.65 (hello). Thinking-on cells post *higher* tok/s than thinking-off for this reason. Per-call `spec_acceptance_rate` comes from vLLM `/metrics` counter deltas.
- **Two cells from one checkpoint.** `-think` sends `chat_template_kwargs.enable_thinking=true`; `-nothink` sends `false`. With thinking on, the frozen caps truncate output: P-HARD spent all 1024 tokens reasoning on all 6 calls (finish=length, 0 answer chars), and the P-MEDIUM cold call hit 512.
- **`tokens_per_sec`** follows `methodology.md`: completion_tokens / ((total - first_delta)/1000), where completion_tokens includes reasoning tokens. Because the flagship runs computed completion_tokens / total, `tokens_per_sec_incl_prefill` is also recorded. With TTFT about 0.1 s, the incl-prefill figure is 2–5% lower on P-HARD and thinking-on P-MEDIUM, 10–13% lower on thinking-off P-MEDIUM, and 39–71% lower on hello (11–39 output tokens, so hello tok/s is mostly noise).
- **`first_delta_ms`** is the first SSE chunk carrying reasoning or content text. `first_chunk_ms` is also recorded. The client sat on a separate box and reached the server through an SSH port forward (ProxyJump over Tailscale DERP), so TTFT includes that tunnel's round trip.
- **Hardware and runtime context.** Multi-GPU: tensor parallel 2 over 2x Radeon AI PRO R9700, each **power-capped at 210 W** (owner setting, left unchanged). vLLM 0.27.1 runs in Docker (stilldeadcode/vllm-radiance:0.9.3, ROCm 7.14, torch 2.11) inside an unprivileged **Debian 13 LXC container on Proxmox VE 9.2** (host kernel 7.0.14-24-pve), not on bare metal or in a VM. Other serve settings: fp8 KV cache, gpu-memory-utilization 0.98, a custom chat template (`qwen-fixed-v22.3.jinja`), and the R4D attention backend from that image.
- **`hardware: "r9700x2"`** is not in `controlled_vocab.hardware`; the maintainers would need to add it or map it.
- **Watts are GPU package power.** They come from sysfs `power1_average` at 1 Hz, summed over both cards across each cell's window including gaps between calls. They are not wall power. `memory_mb` is VRAM used summed over both cards; vLLM preallocates about 98%, so it reflects the reservation, not the model's footprint. No `llama_bench` record (llama.cpp-only).
- **Checkpoint hash.** The main checkpoint's full sha256 was not computed (19.4 GB single file, skipped as heavy). A safetensors-header sha256 and a config.json sha256 are given as fingerprints. The draft model's sha256 is the HF LFS etag from its local download metadata.
- **Isolation.** Requests were strictly sequential. Before each call the driver waited until vLLM reported 0 running/0 waiting, and it checked `request_success_total` deltas: no foreign completions landed during any call. Server logs for the run window show only these 36 POSTs plus 2 `GET /v1/models` polls from another LAN client.

## Notes

- Thinking-on runs faster per token than thinking-off (≈204 vs ≈108 tok/s warm on P-HARD) because the draft model's acceptance rate is much higher on reasoning text (0.53 vs 0.21). With speculative decoding, tok/s measures how predictable the content is, not just how fast the hardware is.
- Thinking-on P-HARD never reaches the story: all 6 calls spend the full 1024-token cap on reasoning (finish=length, 0 answer chars). The frozen caps were sized for non-thinking output. The later Predator runs raised them to 512/1024/2048 for this reason.
- The prompts are shorter than one prefix-cache block, so prefix caching played no part (0 hit tokens on every call).
- 36 calls, 0 errors, strictly sequential. No other completions landed on the server during any call (checked via `/metrics` deltas and server logs).
- Checkpoint fingerprint: Full-file sha256 NOT computed (single 19.4 GB file; skipped as too heavy for a read-only pass on a shared machine). Fingerprints instead: safetensors header sha256 b8d7088a40a3e79e80bd21c8b36d051fedf2e402435e6983a68f067fb2504ec7 (header_len 235336); config.json sha256 6a3aad4b9e457ef95a4c3b9c63ca531212f05cf6fc6723aba2a52e49f21f4e78. No HF cache metadata/upstream commit present for this dir.