Measured on this card

NVIDIA GeForce RTX 4060 Ti

16 GB · Ada sm_89 · 288 GB/s

Qwen3.8-27B UD-Q3_K_XL: token generation

Typical app Our engine t/s faster 1.9×

Model: Qwen3.8-27B UD-Q3_K_XL
File: 13.2 GB
Memory used: 14.5 GB / 16 GB
A 500-word answer: 31.5 s → 16.5 s

Memory: 16GB
Bandwidth: 288GB/s
Power limit: 100W
Architecture: Ada sm_89
PCIe: gen3 x8

What fits

16 GB of memory. Each block is a model loaded by our engine, with its context and KV cache.

Qwen3.8-27B UD-Q3_K_XL

14.5 GB

file 13.2 GB used 14.5 GB

LFM2.5-24B QAD-Q4_0

13.6 GB

file 13.6 GB used 13.6 GB

Models for this card

Every measured model that fits in the card’s memory. The best one first.

Qwen3.8-27B

UD-Q3_K_XL 13.2 GB Licence: Apache 2.0: commercial use allowed

Our pick for this card
Writing speed

1.9× faster

Same card, same model, same script. Only the software changes.

Honestly: the details

The same script, other tasks: code edits (T=0) 18.5 → 50.0 → 170.9 t/s; generation after a 16k context (T=0) 17.6 → 33.3 → 40.1 t/s. Honestly: the gain over the best flags is a shorter step (81.6 vs 97.7 ms), not more tokens per step; our prefill is 3.5–3.9% slower, and without speculation (llama-bench tg128) the pace is the same.

Typical app 18.6 t/s

Best flags 30.6 t/s

Our engine 35.4 t/s

Typical app: llama.cpp with default settings, like a typical app (on the RTX 3090 the same as Ollama) · Best flags: upstream llama.cpp with the best flags we know (e.g. MTP speculation) · Our engine: our engine with the same flags plus our patches · spec_sweep through the server, the same script on both sides: chat, 6 prompts × 3 seeds, T=1, fresh server, median; ABBA order, mean of the two passes /; :56

A 500-word answer
derived from t/s × tokens per word /; :56 × 2.69 tok/słowo (raport 06 §2.1.2)/; :56 × 1.17 tok/słowo (raport 06 §2.1.2)

in Polish

in English

Replay at real speed

The same answer, written out at the measured speed.

Typical app: 18.6 t/s, Our engine: 35.4 t/s. spec_sweep through the server, the same script on both sides: chat, 6 prompts × 3 seeds, T=1, fresh server, median; ABBA order, mean of the two passes

Typical app UD-Q3_K_XL

18.6 t/s

First I collect the sensor data and check which readings are reliable. Then I compare them with yesterday’s run, flag the outliers and suggest three fixes. I describe each one briefly, with its risk and cost. Finally I write a summary for the team and a list of questions worth answering before the rollout. If something does not add up, I go back to the raw data, check the calibration and compare the results with last week. I note what I changed and why, so anyone can repeat the steps. Only then do I send the report, with a chart and a short recommendation for what to do tomorrow morning. Each figure links to its source, and every assumption is written down next to it, so nothing has to be taken on trust.

156 / 156 tokens done ·8.4 s
Our engine UD-Q3_K_XL

35.4 t/s

First I collect the sensor data and check which readings are reliable. Then I compare them with yesterday’s run, flag the outliers and suggest three fixes. I describe each one briefly, with its risk and cost. Finally I write a summary for the team and a list of questions worth answering before the rollout. If something does not add up, I go back to the raw data, check the calibration and compare the results with last week. I note what I changed and why, so anyone can repeat the steps. Only then do I send the report, with a chart and a short recommendation for what to do tomorrow morning. Each figure links to its source, and every assumption is written down next to it, so nothing has to be taken on trust.

156 / 156 tokens done ·4.4 s 1.9×
Conditions

spec_sweep through the server, the same script on both sides: chat, 6 prompts × 3 seeds, T=1, fresh server, median; ABBA order, mean of the two passes
- best flags and our engine: 32k context (-c 32768)
- KV q8_0
- -fit off
- -fa on
- ours adds GPU verify
- A0: plain -m (default -fit, KV f16)
- card power-limited to 100 W
- PCIe gen3 x8
- MTP n=4
- ours: 64k draft vocabulary
- IQ4_XS heads
- lookup W16
spec_sweep through the server, the same script on both sides: chat, 6 prompts × 3 seeds, T=1, fresh server, median; ABBA order, mean of the two passes · power limit 100 W (host; default 165 W), PCIe gen3 x8, GDDR6 288 GB/s, host CPU Ryzen 9 3950X, driver 595.84 :25-28
About the model

A dense 27B model with hybrid attention (Gated DeltaNet + full attention) and a built-in MTP head that guesses several tokens ahead.

Qwen at Q3 fits in 16 GB with a 32k context and is 1.9× faster here than the typical app.
- chat
- code
- quality
app/landing/src/lib/data/models.jsonapp/landing/src/lib/data/models.json (raport 06 §2.4.2):52
All measurements
Instruments

Typical app → Our engine
Tokens per step: tokens accepted per model step (MTP speculation) · Step time: The step is longer because it yields ~2.9 tokens. Per token: 53.7 → 28.3 ms (derived: step time ÷ tokens per step). · Prefill 512: reading the prompt, tokens per second · Prefill 4096: reading the prompt, tokens per second · Memory used: best flags and our engine: 32k context (-c 32768), KV q8_0, -fit off, -fa on; ours adds GPU verify; A0: plain -m (default -fit, KV f16); card power-limited to 100 W, PCIe gen3 x8; MTP n=4, ours: 64k draft vocabulary, IQ4_XS heads, lookup W16 (the source gives GiB; shown as GB, the way the vendor labels the card memory) /; :58/; :57/; :70 (14354 MiB)/; :70 (14848 MiB)

Tokens per step

1.00 2.88

↗ better
Step time

53.7 81.6 ms

↘ longer step, ↗ more tokens
Prefill 512

794 763 t/s

↘ worse
Prefill 4096

788 761 t/s

↘ worse
Memory used

14.0 14.5 GB

↘ worse
Long context
Generation as the conversation grows (llama-bench, no speculation). The gain here is small, and we show it as it is. KV q8_0.

our engine upstream context length (linear scale) →
Where the gain comes from

Typical app 18.6 t/s

upstream flags 1.6× MTP

our engine 1.16× 64k draft vocabulary, lookup W16, GPU verify

total 1.9× :9-11,56/; :56
Same model, same quality: measured

Honestly

Saluki 27B

KLD ours vs upstream 0.00065

upstream noise 0.00061
Bonsai 2 27B

KLD ours vs upstream 0.00032

upstream noise 0.00021

In plain mode our engine does not produce bit-identical text to upstream (patches 0001/0002, plus 0013 for Qwen). So we measure quality with KLD: on the RTX 3060 the difference stays within upstream’s own noise.
LFM2.5-24B-A2B

QAD-Q4_0 13.6 GB Licence: LFM Open License v1.0: commercial use only below USD 10M annual revenue
Writing speed

1.3× faster

Same card, same model, same script. Only the software changes.

Typical app 114.4 t/s

Best flags 113.3 t/s

Our engine 149.4 t/s

Typical app: llama.cpp with default settings, like a typical app (on the RTX 3090 the same as Ollama) · Best flags: upstream llama.cpp with the best flags we know (e.g. MTP speculation) · Our engine: our engine with the same flags plus our patches · spec_sweep through the server, the same script on both sides: chat, 6 prompts × 3 seeds, T=1, fresh server, median; ABBA order, mean of the two passes /; :77

A 500-word answer
derived from t/s × tokens per word /; :77 × 2.69 tok/słowo (raport 06 §2.1.2)/; :77 × 1.17 tok/słowo (raport 06 §2.1.2)

in Polish

in English

Replay at real speed

The same answer, written out at the measured speed.

Typical app: 114.4 t/s, Our engine: 149.4 t/s. spec_sweep through the server, the same script on both sides: chat, 6 prompts × 3 seeds, T=1, fresh server, median; ABBA order, mean of the two passes

Typical app QAD-Q4_0

114.4 t/s

First I collect the sensor data and check which readings are reliable. Then I compare them with yesterday’s run, flag the outliers and suggest three fixes. I describe each one briefly, with its risk and cost. Finally I write a summary for the team and a list of questions worth answering before the rollout. If something does not add up, I go back to the raw data, check the calibration and compare the results with last week. I note what I changed and why, so anyone can repeat the steps. Only then do I send the report, with a chart and a short recommendation for what to do tomorrow morning. Each figure links to its source, and every assumption is written down next to it, so nothing has to be taken on trust.

156 / 156 tokens done ·1.4 s
Our engine QAD-Q4_0

149.4 t/s

First I collect the sensor data and check which readings are reliable. Then I compare them with yesterday’s run, flag the outliers and suggest three fixes. I describe each one briefly, with its risk and cost. Finally I write a summary for the team and a list of questions worth answering before the rollout. If something does not add up, I go back to the raw data, check the calibration and compare the results with last week. I note what I changed and why, so anyone can repeat the steps. Only then do I send the report, with a chart and a short recommendation for what to do tomorrow morning. Each figure links to its source, and every assumption is written down next to it, so nothing has to be taken on trust.

156 / 156 tokens done ·1.0 s 1.3×
Conditions

spec_sweep through the server, the same script on both sides: chat, 6 prompts × 3 seeds, T=1, fresh server, median; ABBA order, mean of the two passes
- best flags and our engine: 32k context (-c 32768)
- KV q8_0
- -fit off
- -fa on
- ours adds GPU verify
- A0: plain -m (default -fit, KV f16)
- card power-limited to 100 W
- PCIe gen3 x8
- no speculation
- -b 2048 -ub 2048
spec_sweep through the server, the same script on both sides: chat, 6 prompts × 3 seeds, T=1, fresh server, median; ABBA order, mean of the two passes · power limit 100 W (host; default 165 W), PCIe gen3 x8, GDDR6 288 GB/s, host CPU Ryzen 9 3950X, driver 595.84 :25-28
About the model

A mixture of experts (MoE): 24B parameters, about 2B active per token. That is why it writes several times faster than a dense model.
- speed
- chat
app/landing/src/lib/data/models.jsonapp/landing/src/lib/data/models.json (raport 06 §2.4.2):73
All measurements
Instruments

Typical app → Our engine
Step time: with MTP a step takes longer but accepts several tokens at once · Prefill 512: reading the prompt, tokens per second · Prefill 4096: reading the prompt, tokens per second · Memory used: best flags and our engine: 32k context (-c 32768), KV q8_0, -fit off, -fa on; ours adds GPU verify; A0: plain -m (default -fit, KV f16); card power-limited to 100 W, PCIe gen3 x8; no speculation, -b 2048 -ub 2048 (the source gives GiB; shown as GB, the way the vendor labels the card memory) /; :78/; :82 (14944 MiB)/; :82 (13924 MiB)

Step time

8.7 6.7 ms

↗ better
Prefill 512

3,286 3,278 t/s

≈ no change
Prefill 4096

3,273 3,268 t/s

≈ no change
Memory used

14.6 13.6 GB

↗ better
Where the gain comes from

Typical app 114.4 t/s

upstream flags (no gain) 0.99×

our engine 1.3× sampler + kernels

total 1.3× :14-15,77/; :77
Same model, same quality: measured

Honestly

Saluki 27B

KLD ours vs upstream 0.00065

upstream noise 0.00061
Bonsai 2 27B

KLD ours vs upstream 0.00032

upstream noise 0.00021

In plain mode our engine does not produce bit-identical text to upstream (patches 0001/0002, plus 0013 for Qwen). So we measure quality with KLD: on the RTX 3060 the difference stays within upstream’s own noise.

What next

Compare with another card Check your hardware

NVIDIA GeForce RTX 4060 Ti

Writing speed

A 500-word answer

Replay at real speed

Conditions

About the model

Instruments

Long context

Where the gain comes from

Honestly