Measured on this card

NVIDIA GeForce RTX 3090

24 GB · Ampere sm_86 · 936 GB/s

Qwen3.8-27B UD-Q4_K_XL: token generation

Typical app Our engine t/s faster 2.2×

Model: Qwen3.8-27B UD-Q4_K_XL
File: 17.6 GB
Memory used: 22.9 GB / 24 GB
A 500-word answer: 13.6 s → 6.3 s

Memory: 24GB
Bandwidth: 936GB/s
Power limit: 300W
Architecture: Ampere sm_86
PCIe: gen3 x16

What fits

24 GB of memory. Each block is a model loaded by our engine, with its context and KV cache.

Qwen3.8-27B UD-Q4_K_XL

22.9 GB

file 17.6 GB used 22.9 GB

LFM2.5-24B Q5_K_M

17.5 GB

file 17.0 GB used 17.5 GB

Nemotron 74B IQ2_XXS

23.1 GB

file 22.0 GB used 23.1 GB

Models for this card

Every measured model that fits in the card’s memory. The best one first.

Qwen3.8-27B

UD-Q4_K_XL 17.6 GB Licence: Apache 2.0: commercial use allowed

Our pick for this card
Writing speed

2.2× faster

Same card, same model, same script. Only the software changes.

Honestly: the details

The same script, other tasks: greedy chat (T=0) 42.3 → 76.7 → 103.4 t/s; code edits 42.0 → 109.5 → 348.9 t/s; copying 41.8 → 108.4 → 343.8 t/s; generation after a 32k context 39.1 → 64.0 → 79.9 t/s. Honestly: with a 200k context our engine did not fit a ~55.6k-token prompt in 24 GB; on 24 GB NVIDIA cards we set the context to ≤ 131,072 (22.2 GB peak at 60k).

Typical app 42.9 t/s

Best flags 75.3 t/s

Our engine 93.3 t/s

Typical app: llama.cpp with default settings, like a typical app (on the RTX 3090 the same as Ollama) · Best flags: upstream llama.cpp with the best flags we know (e.g. MTP speculation) · Our engine: our engine with the same flags plus our patches · spec_sweep through the server, the same script on both sides: chat, 6 prompts × 3 seeds, T=1, fresh server, median; ABBA order, mean of the two passes /; :59

A 500-word answer
derived from t/s × tokens per word /; :59 × 2.69 tok/słowo (raport 06 §2.1.2)/; :59 × 1.17 tok/słowo (raport 06 §2.1.2)

in Polish

in English

Replay at real speed

The same answer, written out at the measured speed.

Typical app: 42.9 t/s, Our engine: 93.3 t/s. spec_sweep through the server, the same script on both sides: chat, 6 prompts × 3 seeds, T=1, fresh server, median; ABBA order, mean of the two passes

Typical app UD-Q4_K_XL

42.9 t/s

First I collect the sensor data and check which readings are reliable. Then I compare them with yesterday’s run, flag the outliers and suggest three fixes. I describe each one briefly, with its risk and cost. Finally I write a summary for the team and a list of questions worth answering before the rollout. If something does not add up, I go back to the raw data, check the calibration and compare the results with last week. I note what I changed and why, so anyone can repeat the steps. Only then do I send the report, with a chart and a short recommendation for what to do tomorrow morning. Each figure links to its source, and every assumption is written down next to it, so nothing has to be taken on trust.

156 / 156 tokens done ·3.6 s
Our engine UD-Q4_K_XL

93.3 t/s

First I collect the sensor data and check which readings are reliable. Then I compare them with yesterday’s run, flag the outliers and suggest three fixes. I describe each one briefly, with its risk and cost. Finally I write a summary for the team and a list of questions worth answering before the rollout. If something does not add up, I go back to the raw data, check the calibration and compare the results with last week. I note what I changed and why, so anyone can repeat the steps. Only then do I send the report, with a chart and a short recommendation for what to do tomorrow morning. Each figure links to its source, and every assumption is written down next to it, so nothing has to be taken on trust.

156 / 156 tokens done ·1.7 s 2.2×
Conditions

spec_sweep through the server, the same script on both sides: chat, 6 prompts × 3 seeds, T=1, fresh server, median; ABBA order, mean of the two passes
- best flags and our engine: 200k context (-c 204800)
- KV q4_0
- MTP n=4
- -fa on
- ours adds the 64k draft vocabulary
- lookup W16
- GPU verify
- A0: plain -m (-fit: 90k context, KV f16)
- card at 335 W
spec_sweep through the server, the same script on both sides: chat, 6 prompts × 3 seeds, T=1, fresh server, median; ABBA order, mean of the two passes · power limit 300 W (host), PCIe gen3 x16, memory 936 GB/s, host CPU i7-4790 (2014), driver 615.71.09, CUDA 12.8. · The sheet says 38.9, the raw scorecard 39.01. We show the sheet value.
About the model

A dense 27B model with hybrid attention (Gated DeltaNet + full attention) and a built-in MTP head that guesses several tokens ahead.

The best quality that fits in 24 GB: 2.2× faster than the typical app.
- chat
- code
- quality
app/landing/src/lib/data/models.jsonapp/landing/src/lib/data/models.json (raport 06 §2.4.2)
All measurements
Instruments

Typical app → Our engine
Tokens per step: tokens accepted per model step (MTP speculation) · Step time: The step is longer because it yields ~3.0 tokens. Per token: 23.3 → 10.7 ms (derived: step time ÷ tokens per step). · Prefill 512: reading the prompt, tokens per second · Prefill 4096: reading the prompt, tokens per second · First token: ~2k-token prompt, no cache · Memory used: best flags and our engine: 200k context (-c 204800), KV q4_0, MTP n=4, -fa on; ours adds the 64k draft vocabulary, lookup W16, GPU verify; A0: plain -m (-fit: 90k context, KV f16); card at 335 W (the source gives GiB; shown as GB, the way the vendor labels the card memory) · Kernels per token: step profile (nsys): fewer, larger jobs for the GPU /; :61/; :60/ / ; :98/ / ; :99/; :90/; :104 (22545 MiB)/; :104 (23493 MiB)

Tokens per step

1.00 3.03

↗ better
Step time

23.3 32.4 ms

↘ longer step, ↗ more tokens
Prefill 512

1,585 1,540 t/s

≈ no change
Prefill 4096

1,582 1,543 t/s

≈ no change
First token

1.2 1.6 s

↘ worse
Memory used

22.0 22.9 GB

↘ worse

Kernels per token

1,956 795

↗ better
Long context
Generation as the conversation grows (llama-bench, no speculation). The gain here is small, and we show it as it is. KV q4_0.

our engine upstream context length (linear scale) →
Where the gain comes from

Typical app 42.9 t/s

upstream flags 1.8× MTP

our engine 1.2× lookup W16, share, GPU verify

total 2.2× :20,59/; :59
Same model, same quality: measured

Honestly

Saluki 27B

KLD ours vs upstream 0.00065

upstream noise 0.00061
Bonsai 2 27B

KLD ours vs upstream 0.00032

upstream noise 0.00021

In plain mode our engine does not produce bit-identical text to upstream (patches 0001/0002, plus 0013 for Qwen). So we measure quality with KLD: on the RTX 3060 the difference stays within upstream’s own noise.
LFM2.5-24B-A2B

Q5_K_M 17.0 GB Licence: LFM Open License v1.0: commercial use only below USD 10M annual revenue
Writing speed

1.5× faster

Same card, same model, same script. Only the software changes.

Typical app[179.0–184.0] (Ollama 180.5) 180.2 t/s

Best flags 176.0 t/s

Our engine + card profile[263.0–266.0] 263.9 t/s

Typical app: llama.cpp with default settings, like a typical app (on the RTX 3090 the same as Ollama) · Best flags: upstream llama.cpp with the best flags we know (e.g. MTP speculation) · Our engine + card profile: our engine with the same flags plus our patches · spec_sweep through the server: 6 prompts × 3 seeds, T=1, 400-token answers, short context, median (A0: 180.2) (A1: 175.99) (A2p-t: 263.91)

A 500-word answer
The bars run in real time: this is how long the answer takes at the measured speed. spec_sweep through the server: 6 prompts × 3 seeds, T=1, 400-token answers, short context, median

in Polish

in English

Replay at real speed

The same answer, written out at the measured speed.

Typical app: 180.2 t/s, Our engine + card profile: 263.9 t/s. spec_sweep through the server: 6 prompts × 3 seeds, T=1, 400-token answers, short context, median

Typical app Q5_K_M

180.2 t/s

First I collect the sensor data and check which readings are reliable. Then I compare them with yesterday’s run, flag the outliers and suggest three fixes. I describe each one briefly, with its risk and cost. Finally I write a summary for the team and a list of questions worth answering before the rollout. If something does not add up, I go back to the raw data, check the calibration and compare the results with last week. I note what I changed and why, so anyone can repeat the steps. Only then do I send the report, with a chart and a short recommendation for what to do tomorrow morning. Each figure links to its source, and every assumption is written down next to it, so nothing has to be taken on trust.

156 / 156 tokens done ·0.9 s
Our engine + card profile Q5_K_M

263.9 t/s

First I collect the sensor data and check which readings are reliable. Then I compare them with yesterday’s run, flag the outliers and suggest three fixes. I describe each one briefly, with its risk and cost. Finally I write a summary for the team and a list of questions worth answering before the rollout. If something does not add up, I go back to the raw data, check the calibration and compare the results with last week. I note what I changed and why, so anyone can repeat the steps. Only then do I send the report, with a chart and a short recommendation for what to do tomorrow morning. Each figure links to its source, and every assumption is written down next to it, so nothing has to be taken on trust.

156 / 156 tokens done ·0.6 s 1.5×
Conditions

spec_sweep through the server: 6 prompts × 3 seeds, T=1, 400-token answers, short context, median
- 64k context (-c 65536)
- KV q8_0
- no speculation
spec_sweep through the server: 6 prompts × 3 seeds, T=1, 400-token answers, short context, median · power limit 300 W (host), PCIe gen3 x16, memory 936 GB/s, host CPU i7-4790 (2014), driver 615.71.09, CUDA 12.8. :19-21
About the model

A mixture of experts (MoE): 24B parameters, about 2B active per token. That is why it writes several times faster than a dense model.
- speed
- chat
app/landing/src/lib/data/models.jsonapp/landing/src/lib/data/models.json (raport 06 §2.4.2)
All measurements
Instruments

Typical app → Our engine + card profile
Step time: with MTP a step takes longer but accepts several tokens at once · Prefill 512: reading the prompt, tokens per second · Prefill 4096: reading the prompt, tokens per second · First token: ~2k-token prompt, no cache · Memory used: 64k context (-c 65536), KV q8_0, no speculation · Kernels per token: step profile (nsys): fewer, larger jobs for the GPU

Step time

5.6 3.8 ms

↗ better
Prefill 512

5,168 5,126 t/s

≈ no change
Prefill 4096

5,168 5,164 t/s

≈ no change
First token

0.5 0.5 s

≈ no change
Memory used

19.3 17.5 GB

↗ better

Kernels per token

908 357

↗ better
Long context
Generation as the conversation grows (llama-bench, no speculation). The gain here is small, and we show it as it is. KV q8_0.

our engine upstream context length (linear scale) →
Where the gain comes from

Typical app 180.2 t/s

upstream flags (no gain) 0.98×

our engine 1.5× sampler/host ×1.42, Ampere profilee ×1.06

total 1.5×
Same model, same quality: measured

Honestly

Saluki 27B

KLD ours vs upstream 0.00065

upstream noise 0.00061
Bonsai 2 27B

KLD ours vs upstream 0.00032

upstream noise 0.00021

In plain mode our engine does not produce bit-identical text to upstream (patches 0001/0002, plus 0013 for Qwen). So we measure quality with KLD: on the RTX 3060 the difference stays within upstream’s own noise.
Nemotron 3.5 Bolt 74B-A9B

IQ2_XXS 22.0 GB Licence: OpenMDW-1.1: commercial and non-commercial use per the model card
Writing speed

1.6× faster

Same card, same model, same script. Only the software changes.

Honestly: the details

Honestly: our fork alone with these flags (A2) gave 129.0 t/s, less than upstream (131.4): the step is shorter but the draft hits less often. "Our engine" is the fork with new patches (141.1), which restores draft acceptance. KLD for this model is not measured yet; 500-word times use the Qwen tokenizer ratio.

Typical app[87.0–88.0] 87.2 t/s

Best flags[107.0–141.0] 131.4 t/s

Our engine[112.0–155.0] 141.1 t/s

Typical app: llama.cpp with default settings, like a typical app (on the RTX 3090 the same as Ollama) · Best flags: upstream llama.cpp with the best flags we know (e.g. MTP speculation) · Our engine: our engine with the same flags plus our patches · spec_sweep through the server: 6 prompts × 3 seeds, T=1, 400-token answers, short context, median (A0: 87.18) (A1: 131.4) (tuned: 141.11)

A 500-word answer
The bars run in real time: this is how long the answer takes at the measured speed. spec_sweep through the server: 6 prompts × 3 seeds, T=1, 400-token answers, short context, median

in Polish

in English

Replay at real speed

The same answer, written out at the measured speed.

Typical app: 87.2 t/s, Our engine: 141.1 t/s. spec_sweep through the server: 6 prompts × 3 seeds, T=1, 400-token answers, short context, median

Typical app IQ2_XXS

87.2 t/s

First I collect the sensor data and check which readings are reliable. Then I compare them with yesterday’s run, flag the outliers and suggest three fixes. I describe each one briefly, with its risk and cost. Finally I write a summary for the team and a list of questions worth answering before the rollout. If something does not add up, I go back to the raw data, check the calibration and compare the results with last week. I note what I changed and why, so anyone can repeat the steps. Only then do I send the report, with a chart and a short recommendation for what to do tomorrow morning. Each figure links to its source, and every assumption is written down next to it, so nothing has to be taken on trust.

156 / 156 tokens done ·1.8 s
Our engine IQ2_XXS

141.1 t/s

First I collect the sensor data and check which readings are reliable. Then I compare them with yesterday’s run, flag the outliers and suggest three fixes. I describe each one briefly, with its risk and cost. Finally I write a summary for the team and a list of questions worth answering before the rollout. If something does not add up, I go back to the raw data, check the calibration and compare the results with last week. I note what I changed and why, so anyone can repeat the steps. Only then do I send the report, with a chart and a short recommendation for what to do tomorrow morning. Each figure links to its source, and every assumption is written down next to it, so nothing has to be taken on trust.

156 / 156 tokens done ·1.1 s 1.6×
Conditions

spec_sweep through the server: 6 prompts × 3 seeds, T=1, 400-token answers, short context, median
- 64k context (-c 65536)
- KV q8_0
- DFlash speculation n_max 5 (NVFP4 drafter, 1 GB)
- A0: plain -m (-fit: 4 slots × 104k, KV f16)
spec_sweep through the server: 6 prompts × 3 seeds, T=1, 400-token answers, short context, median · power limit 300 W (host), PCIe gen3 x16, memory 936 GB/s, host CPU i7-4790 (2014), driver 615.71.09, CUDA 12.8. :73; :224
About the model

A Mamba-2 + attention hybrid built as a mixture of experts: 74B parameters, about 9B active per token. At IQ2_XXS it fits in 24 GB together with a small DFlash drafter.
- a big model on 24 GB
- speed
All measurements
Instruments

Typical app → Our engine
Prefill 512: reading the prompt, tokens per second · Prefill 4096: reading the prompt, tokens per second · First token: ~2k-token prompt, no cache · Memory used: 64k context (-c 65536), KV q8_0, DFlash speculation n_max 5 (NVFP4 drafter, 1 GB); A0: plain -m (-fit: 4 slots × 104k, KV f16)

Prefill 512

1,602 1,605 t/s

≈ no change
Prefill 4096

1,566 1,563 t/s

≈ no change
First token

1.9 2.3 s

↘ worse
Memory used

22.0 23.1 GB

↘ worse
Long context
Generation as the conversation grows (llama-bench, no speculation). The gain here is small, and we show it as it is. KV q8_0.

our engine upstream context length (linear scale) →
Where the gain comes from

Typical app 87.2 t/s

upstream flags 1.5× DFlash

our engine 1.07× patches

total 1.6×
Same model, same quality: measured

Honestly

Saluki 27B

KLD ours vs upstream 0.00065

upstream noise 0.00061
Bonsai 2 27B

KLD ours vs upstream 0.00032

upstream noise 0.00021

Nemotron 74B: KLD not measured yet

In plain mode our engine does not produce bit-identical text to upstream (patches 0001/0002, plus 0013 for Qwen). So we measure quality with KLD: on the RTX 3060 the difference stays within upstream’s own noise.

What next

Compare with another card Check your hardware

NVIDIA GeForce RTX 3090

Writing speed

A 500-word answer

Replay at real speed

Conditions

About the model

Instruments

Long context

Where the gain comes from

Honestly

Writing speed

A 500-word answer

Replay at real speed

Conditions

About the model

Instruments

Long context

Where the gain comes from

Honestly

Writing speed

A 500-word answer

Replay at real speed

Conditions

About the model

Instruments

Long context

Where the gain comes from

Honestly

What next

Measured cards