August 2026

By Davide Ciffa

Qwen3.8-27B on the AMD R9700: up to 227 tok/s

The lucebox engine now serves Qwen3.8-27B on a single AMD Radeon AI PRO R9700 (32 GB, RDNA4), with the DFlash2 block-diffusion drafter from z-lab.

Up to 227 tok/s on code, 208 tok/s HumanEval average, math at 133 tok/s. The quant is off the shelf and matches an 8-bit reference on HumanEval and GSM8K.

And it is lossless. Greedy verification commits only tokens the model itself would produce. The drafter changes how fast you get the answer, never what the answer is.

TL;DR

• The served file is Unsloth's UD-IQ4_XS, straight off HuggingFace. Measured against a Q8_0 reference it is 8-bit-class: mean KL divergence 0.018, 94% top-1 agreement, and it matches the reference on HumanEval and GSM8K within noise. No requantization step, no custom file.
• With speculation: 208 tok/s HumanEval average, 228 peak, 133 tok/s on math, 156 tok/s end-to-end with prefill and HTTP included.
• The lever is the draft block width. DFlash2 ships trained at 8 tokens per block, but its acceptance extrapolates: --draft-block-size 16 takes the HumanEval average from 133.9 to 208.1 tok/s, and on this file widening did not change a byte of output at any width, verified below.
• Faster than llama.cpp running the same DFlash2 drafter: 3.8x on decode (208.1 vs 54.6) and 3.3x end-to-end (156.2 vs 47.2), on the same card with the same weights file.
• Plain decode: 32.3 tok/s against llama.cpp's 29.7 on the same file. This quant streams only 13.3 GiB per token but runs at about 71% of the card's bandwidth, because dynamic quants push some tensors below 4-bit onto slower kernels. A pure IQ4_XS requant reaches 37.4 tok/s plain if autoregressive speed is what you need, hosted ready-made on our HuggingFace; the quantization section below measures that trade both ways.

A block-diffusion drafter, verified exactly

DFlash2 is a 5-layer draft model. It reads the target's hidden states and proposes a whole block of tokens in one masked, parallel pass. A lightweight selector then traces one coherent path through the top candidates per position.

lucebox verifies the proposed chain in a single batched target forward and commits the accepted prefix. That verification reads the full weights once, whether it checks 8 positions or 16. Wide blocks are therefore nearly free, which is how one R9700 turns 32 tok/s of physics into 200+ tok/s of code.

The engine work behind this release starts in the draft graph: the DFlash2 dynamic convolutions and the candidate selector. Then a context-KV ring cache, so drafting stays flat at any context. Then a fused DeltaNet decode path for Qwen3.8's hybrid linear attention.

On top of that, RDNA4-tuned matmul tiles and a widened draft horizon, with byte-exactness validated at every width. Full details and A/B history are in PR #625.

What the block width buys, measured as decode-only speed (the generation phase alone, prefill excluded, as reported by the engine's own per-request timer):

All rows: Qwen3.8-27B UD-IQ4_XS, greedy, single R9700, ROCm 7.2. The 208.1 figure is the mean across the ten prompts, weighted by tokens generated.

227.8 is the fastest individual request, on highly predictable code where the drafter commits its full block. Expect the average, not the peak, on mixed work.

Against llama.cpp, with the same drafter

Comparing a speculative engine against plain decoding is not a fair fight, so we gave llama.cpp the same weapon. Upstream now supports DFlash2 through PR #27342 (--spec-type draft-dflash).

We built that branch for RDNA4 and ran it on the same card. Same UD-IQ4_XS weights file, same z-lab drafter checkpoint, same ten HumanEval prompts at 256 tokens through the OpenAI chat endpoint.

Read the columns for what they are. Decode is the generation phase alone, where the engines actually differ: 208.1 against 54.6, a 3.8x gap on the identical drafter and identical weights.

End-to-end is the whole request, prefill and HTTP included, which is what a user feels: 156.2 against 47.2, or 3.3x. Prompt processing is not the story on these short prompts. The long-context section below measures prefill at length for our engine, and decode at length for both.

How these numbers were taken

Both engines ran from the same client script against the same OpenAI-compatible endpoint. Ten HumanEval-style code prompts, max_tokens 256, temperature 0, one request at a time.

The end-to-end figure is total tokens divided by total wall time across all ten requests, so prefill, sampling and HTTP overhead sit inside it. Both servers were freshly started, so both carry the same first-request warmup.

llama.cpp is PR #27342 built for gfx1201 with -ngl 99 -fa 1 -c 8192 -ctk q8_0 -ctv q8_0, drafting with --spec-draft-n-max 7.

Before anyone asks whether the width flag is the trick: llama.cpp clamps it to the drafter's trained block, 8 for this checkpoint. We measured 7 and 16 (clamped to 8) within noise of each other, 54.6 against 54.3 decode.

Drafting past the trained block is the part that takes engine work. On our side it is worth 133.9 to 208.1 tok/s.

The two engines use different drafter file layouts. llama.cpp ran the official checkpoint from incoai/Qwen3.8-27B-DFlash2-GGUF; lucebox ran the same z-lab weights through our converter.

Decode-only figures come from each engine's own per-request timer (lucebox reports decode_tokens_per_sec, llama.cpp reports predicted_per_second), so neither side is measured by our stopwatch. For reference, llama.cpp's standalone llama-bench gives 30.9 tg128 and 1,283 pp512 on this file, consistent with the server numbers above.

Two lucebox figures appear in this post and they measure different things. 156.2 is end-to-end including prefill, which is what the comparison above needs. 208.1 is decode-only, which is what the block-width table reports.

Neither is a marketing number for the other.

What "lossless" means here, precisely

The drafter never puts a token into the output. Every token is checked against the target model's own logits and committed only if the target agrees. Acceptance rate changes speed and nothing else.

Measured on the served file: blocks of 8, 12 and 16 produce hash-identical bytes on all ten prompts. All ten are also byte-identical to plain decode.

One honest subtlety is worth stating plainly, because anyone testing this on another file may meet it. Verification evaluates sixteen positions in one batched matrix multiply, while plain decode evaluates one position at a time through a vector kernel.

Those two paths sum floating-point numbers in a different order. When two candidate tokens sit within floating-point noise of each other, the winner can differ.

On the served file all ten prompts matched plain decode exactly. On the pure requant measured below, one of ten diverged after 536 identical characters.

Plain decode wrote return result; the speculative path wrote an extra if current: result.append(current) before it.

Both are the model's own argmax under their respective arithmetic. This is a property of batched evaluation in general, not of speculation: changing batch size in any engine can move a tie. What speculation does not do is substitute the drafter's opinion for the model's.

Against llama.cpp running the identical drafter on the identical file, lucebox is 3.3x faster end-to-end. The gap is not the model, and it is not the width flag either.

Their integration is complete: target-feature extraction and the DFlash2 selector are both implemented. But their selector samples candidate paths with an RNG. Ours runs a deterministic best-first search over normalized scores, and keeps the drafter's features exactly current across rollbacks.

Acceptance is where those details compound. lucebox commits long runs of its 16-token block on predictable code, a 6.4x gain over its own plain decode. llama.cpp gains 1.8x over its own at either width setting.

One target forward reads 13.3 GiB of weights whether it checks 4 positions or 16, so the engine that fills that batch wins.

What happens at long context

Short benchmarks flatter speculative engines, so here is the same stack on real long prompts. The input is a growing slice of this repository's C++ source. The question at the end requires reading a label planted at the very top and summarizing the code.

Generation is 128 tokens, greedy, full attention, block 16, --max-ctx 131072.

All of that runs on one 32 GB card, with no offload and no paging. The KV cache at q8_0 costs about 34 KiB per token.

So 128K tokens is 4.2 GiB on top of 13.3 GiB of weights, which is why --max-ctx 131072 fits comfortably. Even 256K would fit, at roughly 22 GiB total.

The label planted at the top was recalled at every length up to 70K. This is real context, not a truncated window.

Two things are worth being blunt about. First, decode here is 24 to 58 tok/s, not 208.

That is mostly a workload difference, not a long-context penalty. This task answers a question in prose, where the drafter's block acceptance sits between 0.21 and 0.36. HumanEval-style code completion runs its decode 6.4x over plain.

Speculation pays in proportion to how predictable the next tokens are. Any engine that quotes one number for every workload is quoting the friendly one.

Second, the slope past the first rows is set by the attention kernel, not by weight traffic. Its cost grows with KV length and with the number of query rows it checks. A wide draft block is therefore a large win on short prompts and a growing tax on long ones.

The engine therefore caps the verify batch at 8 rows once the context passes 8K. That cap follows context length alone, never the observed acceptance rate.

The drafter still proposes its full block. Only the verified width narrows.

Short prompts are untouched, and the narrowing is reproducible for a given prompt rather than chasing a moving target.

The comparison narrows at length. Running llama.cpp with the same DFlash2 drafter over the same growing prompts, decode on its own timer:

From 1.5K through 36K the two engines sit within a few percent of each other, inside run-to-run spread. Past that they pull ahead: at 70K they hold 31.1 against our 23.7.

On unpredictable text at extreme context, their step time degrades less than ours. We would rather print that row than leave it out. It is the next thing to fix, and it is not draft width, because both engines check the same number of positions there.

Our large margin is on code and math, at the prompt lengths where people actually iterate, which is what the model is for.

To make that concrete on the same files, we ran a code-continuation ladder over the identical context lengths. lucebox holds 142 down to 45 tok/s from 1.4K to 61K prompt tokens.

llama.cpp with the same drafter sits between 28 and 54, and passes us only once, at 42K.

The chart puts both workloads side by side. The long-context story depends entirely on whether the drafter can predict what comes next.

Both engines run the same weights file and the same drafter. Solid lines: code continuation, the workload speculation is built for. Dashed: prose summarization, the drafter-hostile case, where both engines converge toward plain-decode speed.

The lucebox code line is jagged because acceptance moves with the content of each slice (0.48 to 0.84 per block).

About the quantization

The served file is Unsloth's UD-IQ4_XS, used exactly as downloaded. That choice came out of measurement, not convenience.

The measurements are worth showing, because two other candidates nearly won. One is bartowski's stock IQ4_XS. The other is a --pure requant of it we made ourselves, the fastest file this card has served.

Quality comes first, against a Q8_0 reference of the same model.

That reference is 27.12 GiB. It does not fit next to the drafter in this card's 32 GB. The second GPU in this lucebox ran it as the offline quality anchor, a Strix Halo with 122 GB of usable unified memory.

How these were measured. Perplexity and KL divergence run over 300 sequential 512-token chunks of the wikitext-2 test set, about 153K tokens. The tool is llama-perplexity --kl-divergence, against saved Q8_0 logits.

That comparison is paired per token, which resolves differences the raw ±0.06 error bars cannot. The PPL ratios above carry uncertainties under a tenth of a percent. Top-1 agreement is how often the quant's argmax token matches the reference's.

HumanEval is all 164 problems, greedy with the DFlash2 drafter attached. They run through the same serving path as every other number here, and execute against the official tests.

Verification is exact, so the drafter cannot change what the model writes. GSM8K is the first 200 test problems, scored on the final answer.

Speed next: same engine, same client, same ten HumanEval prompts, block 16.

Read the two tables together and the decision writes itself. The Unsloth file is 8-bit-class on every quality instrument we have. It matches or beats the Q8_0 reference outright on both task evals, and ties the stock file on speed.

One detail is instructive: it is the smallest file in the table yet not the fastest. Dynamic quants push selected tensors below 4-bit, and the sub-4-bit kernels on gfx1201 run far under IQ4_XS bandwidth. Bytes only buy speed when the kernels that read them keep up.

The pure requant is the opposite trade. On code it is about 12% faster end-to-end than the served file, and 37.4 against 32.3 tok/s on plain decode.

The quality cost is small but real, and it shows up on every instrument. The KL divergence rises by half and top-1 agreement gives up two points. It also hands back seven HumanEval and six GSM8K problems against the served file.

Each task gap on its own sits within a couple of standard errors at these sample sizes. But they all point the same way, and the KL divergence, the sensitive instrument here, is unambiguous.

An earlier version of this post served that file and called its perplexity gap unresolvable inside the error bars. The paired comparison resolves it, and we would rather correct ourselves than round in our own favor.

We now serve the file whose quality we can defend without a footnote. The 12% is yours if you want it: we host the ready-made file at Lucebox/Qwen3.8-27B-IQ4_XS-pure-GGUF, with the trade stated on the card. Or reproduce it yourself with one command:

# from bartowski/Qwen3.8-27B-GGUF
llama-quantize --allow-requantize --pure \
  models/Qwen3.8-27B-IQ4_XS.gguf \
  models/Qwen3.8-27B-IQ4_XS-pure.gguf IQ4_XS

Run it

# build (ROCm 7.2+, RDNA4)
git clone --recurse-submodules https://github.com/Luce-Org/lucebox.git
cd lucebox
cmake -S server -B server/build-hip -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++ \
  -DDFLASH27B_GPU_BACKEND=hip \
  -DDFLASH27B_HIP_ARCHITECTURES=gfx1201 \
  -DGGML_HIP_MMQ_MFMA=ON \
  -DGGML_HIP_NO_VMM=ON
cmake --build server/build-hip --target dflash_server -j"$(nproc)"

# the target quant, used as downloaded, and the z-lab drafter converted to GGUF
mkdir -p models
huggingface-cli download unsloth/Qwen3.8-27B-GGUF \
  Qwen3.8-27B-UD-IQ4_XS.gguf --local-dir models
huggingface-cli download incoai/Qwen3.8-27B-DFlash2 --local-dir models/dflash2
python server/scripts/convert_dflash_to_gguf.py \
  models/dflash2/model.safetensors models/qwen38-dflash2-f16.gguf
python server/scripts/quantize_dflash_draft.py \
  models/qwen38-dflash2-f16.gguf models/qwen38-dflash2-q8_0.gguf --scheme q8_0

# launch: the configuration every number in this post was measured with
# (device auto-detects; on a multi-GPU box add --target-device hip:N)
./server/build-hip/dflash_server models/Qwen3.8-27B-UD-IQ4_XS.gguf \
  --draft models/qwen38-dflash2-q8_0.gguf \
  --draft-block-size 16 --max-ctx 131072 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --port 8216

# it speaks the OpenAI API, so point any client at it
curl -s http://127.0.0.1:8216/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Write a Python LRU cache."}],
       "max_tokens":256,"temperature":0}'

There is nothing else to set. Earlier builds needed four environment variables to reach these numbers. That was a bad default: the tuned path is the one everyone should get, and opt-in mostly means running the slow path by accident.

Exact f32 chain checkpoints, rollback from the first accepted token, memoized q8 activations and the skipped KV rotation are all defaults now. The command above is the fast one.

We also checked the obvious KV knob. --cache-type-k f16 --cache-type-v f16 doubles KV memory and measured slightly slower on this file: 202.5 against 208.1 decode on the same ten prompts. The q8_0 cache in the command stands.

--draft-block-size 16 widens the drafter to double its published horizon. The server accepts up to 2x the checkpoint metadata, and startup logs confirm the exact-verify path.

On prose-heavy workloads the drafter accepts less and the widening stops paying, so leave the flag off there. Code and math are where it shines. The target file is served exactly as downloaded, and the quantization section above measures the alternatives if you want to trade differently.

Bottom line

A single R9700 now serves a 27B frontier-family model at interactive-agent speeds. That is 208 tok/s average on real code, with peaks up to 227. It is 3.3x llama.cpp running the very same drafter on the very same weights file.

Every token is verified against the model's own logits, not trusted from a drafter. The quant is a stock file that matches an 8-bit reference on the task benchmarks we could throw at it. The whole stack is open source.

Source: github.com/Luce-Org/lucebox (PR #625). Drafter checkpoint: incoai/Qwen3.8-27B-DFlash2 (z-lab). Numbers measured on one Radeon AI PRO R9700 32 GB, ROCm 7.2, August 2026.

Related

• DFlash on ggml: up to 207 tok/s Qwen3.5-27B on a RTX 3090
• Laguna XS 2.1 33B on a RTX 3090: 296 tok/s peak, 152 tok/s at 256K context
• Lucebox beats DGX Spark by 3.63x on DeepSeek V4 Flash decode