Run AI on your own computer.

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (bytes_per_token / (bandwidth × efficiency) + overhead)

System RAM: 39.775 GB ÷ (307.2 GB/s × 0.7) + 44.61 ms fixed per-token overhead → 4.36 tok/s

System RAM.bandwidth_gbs

307.2 GB/s — component spec

System RAM.efficiency_moe

0.7 — Settings efficiency.server_ddr5 — Server DDR5/MRDIMM single socket: 6 dense + 7 MoE points (small n); fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md. The 44.61 ms per-token overhead is a CPU-engine constant, not a property of DDR5: its fit is set by a 128-core Xeon 6980P running llama.cpp (55.3 ms/token on a 3B model) and SGLang (17.0 ms) with the CPU executing every layer (research 12 §6). A GPU spilling into this memory pays it only for the share of each token read from RAM (lib/perf/decode.ts refinement 3).

System RAM.active_share

1 — placement

per_token_overhead_ms

44.61 ms — Settings efficiency.server_ddr5.overheadMs — calibrated fit (D-013): Server DDR5/MRDIMM single socket: 6 dense + 7 MoE points (small n); fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md. The 44.61 ms per-token overhead is a CPU-engine constant, not a property of DDR5: its fit is set by a 128-core Xeon 6980P running llama.cpp (55.3 ms/token on a 3B model) and SGLang (17.0 ms) with the CPU executing every layer (research 12 §6). A GPU spilling into this memory pays it only for the share of each token read from RAM (lib/perf/decode.ts refinement 3).

bytes_per_token_gb

39.7749 GB

• MoE: only active parameters are read per token (SPEC §3).
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• Batch size 1 (single user, interactive).

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (bytes_per_token / (bandwidth × efficiency) + overhead)

System RAM: 5.274 GB ÷ (307.2 GB/s × 0.81) + 44.61 ms fixed per-token overhead → 15.2 tok/s

System RAM.bandwidth_gbs

307.2 GB/s — component spec

System RAM.efficiency

0.81 — Settings efficiency.server_ddr5 — Server DDR5/MRDIMM single socket: 6 dense + 7 MoE points (small n); fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md. The 44.61 ms per-token overhead is a CPU-engine constant, not a property of DDR5: its fit is set by a 128-core Xeon 6980P running llama.cpp (55.3 ms/token on a 3B model) and SGLang (17.0 ms) with the CPU executing every layer (research 12 §6). A GPU spilling into this memory pays it only for the share of each token read from RAM (lib/perf/decode.ts refinement 3).

System RAM.active_share

1 — placement

per_token_overhead_ms

44.61 ms — Settings efficiency.server_ddr5.overheadMs — calibrated fit (D-013): Server DDR5/MRDIMM single socket: 6 dense + 7 MoE points (small n); fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md. The 44.61 ms per-token overhead is a CPU-engine constant, not a property of DDR5: its fit is set by a 128-core Xeon 6980P running llama.cpp (55.3 ms/token on a 3B model) and SGLang (17.0 ms) with the CPU executing every layer (research 12 §6). A GPU spilling into this memory pays it only for the share of each token read from RAM (lib/perf/decode.ts refinement 3).

bytes_per_token_gb

5.2741 GB

• Dense: every parameter is read per token.
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• Batch size 1 (single user, interactive).

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (Σ_pool bytes_per_token × share_pool / (bandwidth_pool × efficiency_pool) + Σ_pool share_pool × overhead_pool)

GPU VRAM: 1.6 GB ÷ (640 GB/s × 0.48) + System RAM: 7.149 GB ÷ (307.2 GB/s × 0.7) + 36.74 ms fixed per-token overhead (share-weighted) → 13.3 tok/s

GPU VRAM.bandwidth_gbs

640 GB/s — component spec

GPU VRAM.efficiency_moe

0.48 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

0.1829 — placement

System RAM.bandwidth_gbs

307.2 GB/s — component spec

System RAM.efficiency_moe

0.7 — Settings efficiency.server_ddr5 — Server DDR5/MRDIMM single socket: 6 dense + 7 MoE points (small n); fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md. The 44.61 ms per-token overhead is a CPU-engine constant, not a property of DDR5: its fit is set by a 128-core Xeon 6980P running llama.cpp (55.3 ms/token on a 3B model) and SGLang (17.0 ms) with the CPU executing every layer (research 12 §6). A GPU spilling into this memory pays it only for the share of each token read from RAM (lib/perf/decode.ts refinement 3).

System RAM.active_share

0.8171 — placement

per_token_overhead_ms

36.74105353314861 ms — derived: 0.183 × 1.58 ms (GPU VRAM) + 0.817 × 44.61 ms (System RAM) = 36.74 ms

bytes_per_token_gb

8.7487 GB

• MoE: only active parameters are read per token (SPEC §3).
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• Pools are read sequentially per token, so per-pool times add (harmonic combination).
• Per-token overhead on a split placement is DERIVED, not measured: each pool's own single-pool figure is charged in proportion to the share of each token it serves, because the fixed cost is per operation and the split puts that share of the operations on each device.
• Calibrated with that pool serving the whole model (D-013): GPU VRAM (1.58 ms), System RAM (44.61 ms).
• Batch size 1 (single user, interactive).

First reply: estimate — Arithmetic that also depends on a tunable assumption, listed below.

ttft_s = prompt_tokens / prefill_tok_s

= 32,768 / 1,996.44 = 16.4 s

prompt_tokens

32768 tokens

prefill_tok_s

1996.44 tok/s

• No published prefill measurement for this hardware; scaled from peak compute (estimate).
• Model q_dim unknown — attention FLOPs omitted, so long-prompt TTFT is understated.

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (Σ_pool bytes_per_token × share_pool / (bandwidth_pool × efficiency_pool) + Σ_pool share_pool × overhead_pool)

GPU VRAM: 9.535 GB ÷ (640 GB/s × 0.48) + System RAM: 30.24 GB ÷ (307.2 GB/s × 0.7) + 34.3 ms fixed per-token overhead (share-weighted) → 4.86 tok/s

GPU VRAM.bandwidth_gbs

640 GB/s — component spec

GPU VRAM.efficiency_moe

0.48 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

0.2397 — placement

System RAM.bandwidth_gbs

307.2 GB/s — component spec

System RAM.efficiency_moe

0.7 — Settings efficiency.server_ddr5 — Server DDR5/MRDIMM single socket: 6 dense + 7 MoE points (small n); fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md. The 44.61 ms per-token overhead is a CPU-engine constant, not a property of DDR5: its fit is set by a 128-core Xeon 6980P running llama.cpp (55.3 ms/token on a 3B model) and SGLang (17.0 ms) with the CPU executing every layer (research 12 §6). A GPU spilling into this memory pays it only for the share of each token read from RAM (lib/perf/decode.ts refinement 3).

System RAM.active_share

0.7603 — placement

per_token_overhead_ms

34.29514218361304 ms — derived: 0.24 × 1.58 ms (GPU VRAM) + 0.76 × 44.61 ms (System RAM) = 34.3 ms

bytes_per_token_gb

39.7749 GB

• MoE: only active parameters are read per token (SPEC §3).
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• Pools are read sequentially per token, so per-pool times add (harmonic combination).
• Per-token overhead on a split placement is DERIVED, not measured: each pool's own single-pool figure is charged in proportion to the share of each token it serves, because the fixed cost is per operation and the split puts that share of the operations on each device.
• Calibrated with that pool serving the whole model (D-013): GPU VRAM (1.58 ms), System RAM (44.61 ms).
• Batch size 1 (single user, interactive).

First reply: estimate — Arithmetic that also depends on a tunable assumption, listed below.

ttft_s = prompt_tokens / prefill_tok_s

= 32,768 / 577.57 = 56.7 s

prompt_tokens

32768 tokens

prefill_tok_s

577.57 tok/s

• No published prefill measurement for this hardware; scaled from peak compute (estimate).
• Model q_dim unknown — attention FLOPs omitted, so long-prompt TTFT is understated.

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (bytes_per_token / (bandwidth × efficiency) + overhead)

GPU VRAM: 5.274 GB ÷ (640 GB/s × 0.86) + 1.58 ms fixed per-token overhead → 89.59 tok/s

GPU VRAM.bandwidth_gbs

640 GB/s — component spec

GPU VRAM.efficiency

0.86 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

1 — placement

per_token_overhead_ms

1.58 ms — Settings efficiency.gpu_vram.overheadMs — calibrated fit (D-013): Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

bytes_per_token_gb

5.2741 GB

• Dense: every parameter is read per token.
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• Per-token overhead of System RAM (44.61 ms) is not charged: this model does not read from that pool.
• Batch size 1 (single user, interactive).

First reply: estimate — Arithmetic that also depends on a tunable assumption, listed below.

ttft_s = prompt_tokens / prefill_tok_s

= 30,000 / 19,426.65 = 1.5 s

prompt_tokens

30000 tokens

prefill_tok_s

19426.65 tok/s

• No published prefill measurement for this hardware; scaled from peak compute (estimate).
• Model q_dim unknown — attention FLOPs omitted, so long-prompt TTFT is understated.

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (Σ_pool bytes_per_token × share_pool / (bandwidth_pool × efficiency_pool) + Σ_pool share_pool × overhead_pool)

GPU VRAM: 0.8 GB ÷ (960 GB/s × 0.48) + System RAM: 7.949 GB ÷ (307.2 GB/s × 0.7) + 40.68 ms fixed per-token overhead (share-weighted) → 12.6 tok/s

GPU VRAM.bandwidth_gbs

960 GB/s — component spec

GPU VRAM.efficiency_moe

0.48 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

0.0914 — placement

System RAM.bandwidth_gbs

307.2 GB/s — component spec

System RAM.efficiency_moe

0.7 — Settings efficiency.server_ddr5 — Server DDR5/MRDIMM single socket: 6 dense + 7 MoE points (small n); fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md. The 44.61 ms per-token overhead is a CPU-engine constant, not a property of DDR5: its fit is set by a 128-core Xeon 6980P running llama.cpp (55.3 ms/token on a 3B model) and SGLang (17.0 ms) with the CPU executing every layer (research 12 §6). A GPU spilling into this memory pays it only for the share of each token read from RAM (lib/perf/decode.ts refinement 3).

System RAM.active_share

0.9086 — placement

per_token_overhead_ms

40.67552676657429 ms — derived: 0.091 × 1.58 ms (GPU VRAM) + 0.909 × 44.61 ms (System RAM) = 40.68 ms

bytes_per_token_gb

8.7487 GB

• MoE: only active parameters are read per token (SPEC §3).
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• Pools are read sequentially per token, so per-pool times add (harmonic combination).
• Per-token overhead on a split placement is DERIVED, not measured: each pool's own single-pool figure is charged in proportion to the share of each token it serves, because the fixed cost is per operation and the split puts that share of the operations on each device.
• Calibrated with that pool serving the whole model (D-013): GPU VRAM (1.58 ms), System RAM (44.61 ms).
• Batch size 1 (single user, interactive).

First reply: estimate — Arithmetic that also depends on a tunable assumption, listed below.

ttft_s = prompt_tokens / prefill_tok_s

= 32,768 / 1,176.96 = 27.8 s

prompt_tokens

32768 tokens

prefill_tok_s

1176.96 tok/s

• No published prefill measurement for this hardware; scaled from peak compute (estimate).
• Model q_dim unknown — attention FLOPs omitted, so long-prompt TTFT is understated.

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (Σ_pool bytes_per_token × share_pool / (bandwidth_pool × efficiency_pool) + Σ_pool share_pool × overhead_pool)

GPU VRAM: 4.767 GB ÷ (960 GB/s × 0.48) + System RAM: 35.008 GB ÷ (307.2 GB/s × 0.7) + 39.45 ms fixed per-token overhead (share-weighted) → 4.7 tok/s

GPU VRAM.bandwidth_gbs

960 GB/s — component spec

GPU VRAM.efficiency_moe

0.48 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

0.1199 — placement

System RAM.bandwidth_gbs

307.2 GB/s — component spec

System RAM.efficiency_moe

0.7 — Settings efficiency.server_ddr5 — Server DDR5/MRDIMM single socket: 6 dense + 7 MoE points (small n); fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md. The 44.61 ms per-token overhead is a CPU-engine constant, not a property of DDR5: its fit is set by a 128-core Xeon 6980P running llama.cpp (55.3 ms/token on a 3B model) and SGLang (17.0 ms) with the CPU executing every layer (research 12 §6). A GPU spilling into this memory pays it only for the share of each token read from RAM (lib/perf/decode.ts refinement 3).

System RAM.active_share

0.8801 — placement

per_token_overhead_ms

39.45257109180652 ms — derived: 0.12 × 1.58 ms (GPU VRAM) + 0.88 × 44.61 ms (System RAM) = 39.45 ms

bytes_per_token_gb

39.7749 GB

• MoE: only active parameters are read per token (SPEC §3).
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• Pools are read sequentially per token, so per-pool times add (harmonic combination).
• Per-token overhead on a split placement is DERIVED, not measured: each pool's own single-pool figure is charged in proportion to the share of each token it serves, because the fixed cost is per operation and the split puts that share of the operations on each device.
• Calibrated with that pool serving the whole model (D-013): GPU VRAM (1.58 ms), System RAM (44.61 ms).
• Batch size 1 (single user, interactive).

First reply: estimate — Arithmetic that also depends on a tunable assumption, listed below.

ttft_s = prompt_tokens / prefill_tok_s

= 32,768 / 340.5 = 96.2 s

prompt_tokens

32768 tokens

prefill_tok_s

340.5 tok/s

• No published prefill measurement for this hardware; scaled from peak compute (estimate).
• Model q_dim unknown — attention FLOPs omitted, so long-prompt TTFT is understated.

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (bytes_per_token / (bandwidth × efficiency) + overhead)

GPU VRAM: 5.274 GB ÷ (960 GB/s × 0.86) + 1.58 ms fixed per-token overhead → 125.5 tok/s

GPU VRAM.bandwidth_gbs

960 GB/s — component spec

GPU VRAM.efficiency

0.86 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

1 — placement

per_token_overhead_ms

1.58 ms — Settings efficiency.gpu_vram.overheadMs — calibrated fit (D-013): Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

bytes_per_token_gb

5.2741 GB

• Dense: every parameter is read per token.
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• Per-token overhead of System RAM (44.61 ms) is not charged: this model does not read from that pool.
• Batch size 1 (single user, interactive).

First reply: estimate — Arithmetic that also depends on a tunable assumption, listed below.

ttft_s = prompt_tokens / prefill_tok_s

= 30,000 / 11,452.57 = 2.6 s

prompt_tokens

30000 tokens

prefill_tok_s

11452.57 tok/s

• No published prefill measurement for this hardware; scaled from peak compute (estimate).
• Model q_dim unknown — attention FLOPs omitted, so long-prompt TTFT is understated.

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (Σ_pool bytes_per_token × share_pool / (bandwidth_pool × efficiency_pool) + Σ_pool share_pool × overhead_pool)

GPU VRAM: 0.8 GB ÷ (960 GB/s × 0.48) + System RAM: 7.949 GB ÷ (307.2 GB/s × 0.7) + 40.68 ms fixed per-token overhead (share-weighted) → 12.6 tok/s

GPU VRAM.bandwidth_gbs

960 GB/s — component spec

GPU VRAM.efficiency_moe

0.48 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

0.0914 — placement

System RAM.bandwidth_gbs

307.2 GB/s — component spec

System RAM.efficiency_moe

0.7 — Settings efficiency.server_ddr5 — Server DDR5/MRDIMM single socket: 6 dense + 7 MoE points (small n); fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md. The 44.61 ms per-token overhead is a CPU-engine constant, not a property of DDR5: its fit is set by a 128-core Xeon 6980P running llama.cpp (55.3 ms/token on a 3B model) and SGLang (17.0 ms) with the CPU executing every layer (research 12 §6). A GPU spilling into this memory pays it only for the share of each token read from RAM (lib/perf/decode.ts refinement 3).

System RAM.active_share

0.9086 — placement

per_token_overhead_ms

40.67552676657429 ms — derived: 0.091 × 1.58 ms (GPU VRAM) + 0.909 × 44.61 ms (System RAM) = 40.68 ms

bytes_per_token_gb

8.7487 GB

• MoE: only active parameters are read per token (SPEC §3).
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• Pools are read sequentially per token, so per-pool times add (harmonic combination).
• Per-token overhead on a split placement is DERIVED, not measured: each pool's own single-pool figure is charged in proportion to the share of each token it serves, because the fixed cost is per operation and the split puts that share of the operations on each device.
• Calibrated with that pool serving the whole model (D-013): GPU VRAM (1.58 ms), System RAM (44.61 ms).
• Batch size 1 (single user, interactive).

First reply: estimate — Arithmetic that also depends on a tunable assumption, listed below.

ttft_s = prompt_tokens / prefill_tok_s

= 32,768 / 1,176.96 = 27.8 s

prompt_tokens

32768 tokens

prefill_tok_s

1176.96 tok/s

• No published prefill measurement for this hardware; scaled from peak compute (estimate).
• Model q_dim unknown — attention FLOPs omitted, so long-prompt TTFT is understated.

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (Σ_pool bytes_per_token × share_pool / (bandwidth_pool × efficiency_pool) + Σ_pool share_pool × overhead_pool)

GPU VRAM: 4.767 GB ÷ (960 GB/s × 0.48) + System RAM: 35.008 GB ÷ (307.2 GB/s × 0.7) + 39.45 ms fixed per-token overhead (share-weighted) → 4.7 tok/s

GPU VRAM.bandwidth_gbs

960 GB/s — component spec

GPU VRAM.efficiency_moe

0.48 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

0.1199 — placement

System RAM.bandwidth_gbs

307.2 GB/s — component spec

System RAM.efficiency_moe

0.7 — Settings efficiency.server_ddr5 — Server DDR5/MRDIMM single socket: 6 dense + 7 MoE points (small n); fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md. The 44.61 ms per-token overhead is a CPU-engine constant, not a property of DDR5: its fit is set by a 128-core Xeon 6980P running llama.cpp (55.3 ms/token on a 3B model) and SGLang (17.0 ms) with the CPU executing every layer (research 12 §6). A GPU spilling into this memory pays it only for the share of each token read from RAM (lib/perf/decode.ts refinement 3).

System RAM.active_share

0.8801 — placement

per_token_overhead_ms

39.45257109180652 ms — derived: 0.12 × 1.58 ms (GPU VRAM) + 0.88 × 44.61 ms (System RAM) = 39.45 ms

bytes_per_token_gb

39.7749 GB

• MoE: only active parameters are read per token (SPEC §3).
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• Pools are read sequentially per token, so per-pool times add (harmonic combination).
• Per-token overhead on a split placement is DERIVED, not measured: each pool's own single-pool figure is charged in proportion to the share of each token it serves, because the fixed cost is per operation and the split puts that share of the operations on each device.
• Calibrated with that pool serving the whole model (D-013): GPU VRAM (1.58 ms), System RAM (44.61 ms).
• Batch size 1 (single user, interactive).

First reply: estimate — Arithmetic that also depends on a tunable assumption, listed below.

ttft_s = prompt_tokens / prefill_tok_s

= 32,768 / 340.5 = 96.2 s

prompt_tokens

32768 tokens

prefill_tok_s

340.5 tok/s

• No published prefill measurement for this hardware; scaled from peak compute (estimate).
• Model q_dim unknown — attention FLOPs omitted, so long-prompt TTFT is understated.

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (bytes_per_token / (bandwidth × efficiency) + overhead)

GPU VRAM: 5.274 GB ÷ (960 GB/s × 0.86) + 1.58 ms fixed per-token overhead → 125.5 tok/s

GPU VRAM.bandwidth_gbs

960 GB/s — component spec

GPU VRAM.efficiency

0.86 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

1 — placement

per_token_overhead_ms

1.58 ms — Settings efficiency.gpu_vram.overheadMs — calibrated fit (D-013): Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

bytes_per_token_gb

5.2741 GB

• Dense: every parameter is read per token.
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• Per-token overhead of System RAM (44.61 ms) is not charged: this model does not read from that pool.
• Batch size 1 (single user, interactive).

First reply: estimate — Arithmetic that also depends on a tunable assumption, listed below.

ttft_s = prompt_tokens / prefill_tok_s

= 30,000 / 11,452.57 = 2.6 s

prompt_tokens

30000 tokens

prefill_tok_s

11452.57 tok/s

• No published prefill measurement for this hardware; scaled from peak compute (estimate).
• Model q_dim unknown — attention FLOPs omitted, so long-prompt TTFT is understated.

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (Σ_pool bytes_per_token × share_pool / (bandwidth_pool × efficiency_pool) + Σ_pool share_pool × overhead_pool)

GPU VRAM: 1.6 GB ÷ (640 GB/s × 0.48) + System RAM: 7.149 GB ÷ (307.2 GB/s × 0.7) + 36.74 ms fixed per-token overhead (share-weighted) → 13.3 tok/s

GPU VRAM.bandwidth_gbs

640 GB/s — component spec

GPU VRAM.efficiency_moe

0.48 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

0.1829 — placement

System RAM.bandwidth_gbs

307.2 GB/s — component spec

System RAM.efficiency_moe

0.7 — Settings efficiency.server_ddr5 — Server DDR5/MRDIMM single socket: 6 dense + 7 MoE points (small n); fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md. The 44.61 ms per-token overhead is a CPU-engine constant, not a property of DDR5: its fit is set by a 128-core Xeon 6980P running llama.cpp (55.3 ms/token on a 3B model) and SGLang (17.0 ms) with the CPU executing every layer (research 12 §6). A GPU spilling into this memory pays it only for the share of each token read from RAM (lib/perf/decode.ts refinement 3).

System RAM.active_share

0.8171 — placement

per_token_overhead_ms

36.74105353314861 ms — derived: 0.183 × 1.58 ms (GPU VRAM) + 0.817 × 44.61 ms (System RAM) = 36.74 ms

bytes_per_token_gb

8.7487 GB

• MoE: only active parameters are read per token (SPEC §3).
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• Pools are read sequentially per token, so per-pool times add (harmonic combination).
• Per-token overhead on a split placement is DERIVED, not measured: each pool's own single-pool figure is charged in proportion to the share of each token it serves, because the fixed cost is per operation and the split puts that share of the operations on each device.
• Calibrated with that pool serving the whole model (D-013): GPU VRAM (1.58 ms), System RAM (44.61 ms).
• Batch size 1 (single user, interactive).

First reply: estimate — Arithmetic that also depends on a tunable assumption, listed below.

ttft_s = prompt_tokens / prefill_tok_s

= 32,768 / 1,996.44 = 16.4 s

prompt_tokens

32768 tokens

prefill_tok_s

1996.44 tok/s

• No published prefill measurement for this hardware; scaled from peak compute (estimate).
• Model q_dim unknown — attention FLOPs omitted, so long-prompt TTFT is understated.

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (Σ_pool bytes_per_token × share_pool / (bandwidth_pool × efficiency_pool) + Σ_pool share_pool × overhead_pool)

GPU VRAM: 9.535 GB ÷ (640 GB/s × 0.48) + System RAM: 30.24 GB ÷ (307.2 GB/s × 0.7) + 34.3 ms fixed per-token overhead (share-weighted) → 4.86 tok/s

GPU VRAM.bandwidth_gbs

640 GB/s — component spec

GPU VRAM.efficiency_moe

0.48 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

0.2397 — placement

System RAM.bandwidth_gbs

307.2 GB/s — component spec

System RAM.efficiency_moe

0.7 — Settings efficiency.server_ddr5 — Server DDR5/MRDIMM single socket: 6 dense + 7 MoE points (small n); fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md. The 44.61 ms per-token overhead is a CPU-engine constant, not a property of DDR5: its fit is set by a 128-core Xeon 6980P running llama.cpp (55.3 ms/token on a 3B model) and SGLang (17.0 ms) with the CPU executing every layer (research 12 §6). A GPU spilling into this memory pays it only for the share of each token read from RAM (lib/perf/decode.ts refinement 3).

System RAM.active_share

0.7603 — placement

per_token_overhead_ms

34.29514218361304 ms — derived: 0.24 × 1.58 ms (GPU VRAM) + 0.76 × 44.61 ms (System RAM) = 34.3 ms

bytes_per_token_gb

39.7749 GB

• MoE: only active parameters are read per token (SPEC §3).
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• Pools are read sequentially per token, so per-pool times add (harmonic combination).
• Per-token overhead on a split placement is DERIVED, not measured: each pool's own single-pool figure is charged in proportion to the share of each token it serves, because the fixed cost is per operation and the split puts that share of the operations on each device.
• Calibrated with that pool serving the whole model (D-013): GPU VRAM (1.58 ms), System RAM (44.61 ms).
• Batch size 1 (single user, interactive).

First reply: estimate — Arithmetic that also depends on a tunable assumption, listed below.

ttft_s = prompt_tokens / prefill_tok_s

= 32,768 / 577.57 = 56.7 s

prompt_tokens

32768 tokens

prefill_tok_s

577.57 tok/s

• No published prefill measurement for this hardware; scaled from peak compute (estimate).
• Model q_dim unknown — attention FLOPs omitted, so long-prompt TTFT is understated.

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (bytes_per_token / (bandwidth × efficiency) + overhead)

GPU VRAM: 5.274 GB ÷ (640 GB/s × 0.86) + 1.58 ms fixed per-token overhead → 89.59 tok/s

GPU VRAM.bandwidth_gbs

640 GB/s — component spec

GPU VRAM.efficiency

0.86 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

1 — placement

per_token_overhead_ms

1.58 ms — Settings efficiency.gpu_vram.overheadMs — calibrated fit (D-013): Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

bytes_per_token_gb

5.2741 GB

• Dense: every parameter is read per token.
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• Per-token overhead of System RAM (44.61 ms) is not charged: this model does not read from that pool.
• Batch size 1 (single user, interactive).

First reply: estimate — Arithmetic that also depends on a tunable assumption, listed below.

ttft_s = prompt_tokens / prefill_tok_s

= 30,000 / 19,426.65 = 1.5 s

prompt_tokens

30000 tokens

prefill_tok_s

19426.65 tok/s

• No published prefill measurement for this hardware; scaled from peak compute (estimate).
• Model q_dim unknown — attention FLOPs omitted, so long-prompt TTFT is understated.

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (Σ_pool bytes_per_token × share_pool / (bandwidth_pool × efficiency_pool) + Σ_pool share_pool × overhead_pool)

GPU VRAM: 3.2 GB ÷ (640 GB/s × 0.48) + System RAM: 5.549 GB ÷ (307.2 GB/s × 0.7) + 28.87 ms fixed per-token overhead (share-weighted) → 15.36 tok/s

GPU VRAM.bandwidth_gbs

640 GB/s — component spec

GPU VRAM.efficiency_moe

0.48 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

0.3657 — placement

System RAM.bandwidth_gbs

307.2 GB/s — component spec

System RAM.efficiency_moe

0.7 — Settings efficiency.server_ddr5 — Server DDR5/MRDIMM single socket: 6 dense + 7 MoE points (small n); fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md. The 44.61 ms per-token overhead is a CPU-engine constant, not a property of DDR5: its fit is set by a 128-core Xeon 6980P running llama.cpp (55.3 ms/token on a 3B model) and SGLang (17.0 ms) with the CPU executing every layer (research 12 §6). A GPU spilling into this memory pays it only for the share of each token read from RAM (lib/perf/decode.ts refinement 3).

System RAM.active_share

0.6343 — placement

per_token_overhead_ms

28.872107066297204 ms — derived: 0.366 × 1.58 ms (GPU VRAM) + 0.634 × 44.61 ms (System RAM) = 28.87 ms

bytes_per_token_gb

8.7487 GB

• MoE: only active parameters are read per token (SPEC §3).
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• Pools are read sequentially per token, so per-pool times add (harmonic combination).
• layer split across 2 GPUs: capacity adds, per-token bandwidth does not (layers run sequentially)
• Per-token overhead on a split placement is DERIVED, not measured: each pool's own single-pool figure is charged in proportion to the share of each token it serves, because the fixed cost is per operation and the split puts that share of the operations on each device.
• Calibrated with that pool serving the whole model (D-013): GPU VRAM (1.58 ms), System RAM (44.61 ms).
• Batch size 1 (single user, interactive).

First reply: estimate — Arithmetic that also depends on a tunable assumption, listed below.

ttft_s = prompt_tokens / prefill_tok_s

= 32,768 / 1,996.44 = 16.4 s

prompt_tokens

32768 tokens

prefill_tok_s

1996.44 tok/s

• No published prefill measurement for this hardware; scaled from peak compute (estimate).
• Model q_dim unknown — attention FLOPs omitted, so long-prompt TTFT is understated.

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (Σ_pool bytes_per_token × share_pool / (bandwidth_pool × efficiency_pool) + Σ_pool share_pool × overhead_pool)

GPU VRAM: 19.069 GB ÷ (640 GB/s × 0.48) + System RAM: 20.706 GB ÷ (307.2 GB/s × 0.7) + 23.98 ms fixed per-token overhead (share-weighted) → 5.48 tok/s

GPU VRAM.bandwidth_gbs

640 GB/s — component spec

GPU VRAM.efficiency_moe

0.48 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

0.4794 — placement

System RAM.bandwidth_gbs

307.2 GB/s — component spec

System RAM.efficiency_moe

0.7 — Settings efficiency.server_ddr5 — Server DDR5/MRDIMM single socket: 6 dense + 7 MoE points (small n); fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md. The 44.61 ms per-token overhead is a CPU-engine constant, not a property of DDR5: its fit is set by a 128-core Xeon 6980P running llama.cpp (55.3 ms/token on a 3B model) and SGLang (17.0 ms) with the CPU executing every layer (research 12 §6). A GPU spilling into this memory pays it only for the share of each token read from RAM (lib/perf/decode.ts refinement 3).

System RAM.active_share

0.5206 — placement

per_token_overhead_ms

23.980284367226076 ms — derived: 0.479 × 1.58 ms (GPU VRAM) + 0.521 × 44.61 ms (System RAM) = 23.98 ms

bytes_per_token_gb

39.7749 GB

• MoE: only active parameters are read per token (SPEC §3).
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• Pools are read sequentially per token, so per-pool times add (harmonic combination).
• layer split across 2 GPUs: capacity adds, per-token bandwidth does not (layers run sequentially)
• Per-token overhead on a split placement is DERIVED, not measured: each pool's own single-pool figure is charged in proportion to the share of each token it serves, because the fixed cost is per operation and the split puts that share of the operations on each device.
• Calibrated with that pool serving the whole model (D-013): GPU VRAM (1.58 ms), System RAM (44.61 ms).
• Batch size 1 (single user, interactive).

First reply: estimate — Arithmetic that also depends on a tunable assumption, listed below.

ttft_s = prompt_tokens / prefill_tok_s

= 32,768 / 577.57 = 56.7 s

prompt_tokens

32768 tokens

prefill_tok_s

577.57 tok/s

• No published prefill measurement for this hardware; scaled from peak compute (estimate).
• Model q_dim unknown — attention FLOPs omitted, so long-prompt TTFT is understated.

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (bytes_per_token / (bandwidth × efficiency) + overhead)

GPU VRAM: 5.274 GB ÷ (640 GB/s × 0.86) + 1.58 ms fixed per-token overhead → 89.59 tok/s

GPU VRAM.bandwidth_gbs

640 GB/s — component spec

GPU VRAM.efficiency

0.86 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

1 — placement

per_token_overhead_ms

1.58 ms — Settings efficiency.gpu_vram.overheadMs — calibrated fit (D-013): Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

bytes_per_token_gb

5.2741 GB

• Dense: every parameter is read per token.
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• layer split across 2 GPUs: capacity adds, per-token bandwidth does not (layers run sequentially)
• Per-token overhead of System RAM (44.61 ms) is not charged: this model does not read from that pool.
• Batch size 1 (single user, interactive).

First reply: estimate — Arithmetic that also depends on a tunable assumption, listed below.

ttft_s = prompt_tokens / prefill_tok_s

= 30,000 / 19,426.65 = 1.5 s

prompt_tokens

30000 tokens

prefill_tok_s

19426.65 tok/s

• No published prefill measurement for this hardware; scaled from peak compute (estimate).
• Model q_dim unknown — attention FLOPs omitted, so long-prompt TTFT is understated.