Run AI on your own computer.

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (bytes_per_token / (bandwidth × efficiency) + overhead)

GPU VRAM: 39.775 GB ÷ (1,792 GB/s × 0.48) + 1.58 ms fixed per-token overhead → 20.91 tok/s

GPU VRAM.bandwidth_gbs

1792 GB/s — component spec

GPU VRAM.efficiency_moe

0.48 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

1 — placement

per_token_overhead_ms

1.58 ms — Settings efficiency.gpu_vram.overheadMs — calibrated fit (D-013): Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

bytes_per_token_gb

39.7749 GB

• MoE: only active parameters are read per token (SPEC §3).
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• layer split across 2 GPUs: capacity adds, per-token bandwidth does not (layers run sequentially)
• Per-token overhead of System RAM (44.61 ms) is not charged: this model does not read from that pool.
• Batch size 1 (single user, interactive).

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (bytes_per_token / (bandwidth × efficiency) + overhead)

GPU VRAM: 5.274 GB ÷ (1,792 GB/s × 0.86) + 1.58 ms fixed per-token overhead → 199.91 tok/s

GPU VRAM.bandwidth_gbs

1792 GB/s — component spec

GPU VRAM.efficiency

0.86 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

1 — placement

per_token_overhead_ms

1.58 ms — Settings efficiency.gpu_vram.overheadMs — calibrated fit (D-013): Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

bytes_per_token_gb

5.2741 GB

• Dense: every parameter is read per token.
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• layer split across 2 GPUs: capacity adds, per-token bandwidth does not (layers run sequentially)
• Per-token overhead of System RAM (44.61 ms) is not charged: this model does not read from that pool.
• Batch size 1 (single user, interactive).

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (bytes_per_token / (bandwidth × efficiency) + overhead)

GPU VRAM: 8.749 GB ÷ (1,792 GB/s × 0.48) + 1.58 ms fixed per-token overhead → 85.1 tok/s

GPU VRAM.bandwidth_gbs

1792 GB/s — component spec

GPU VRAM.efficiency_moe

0.48 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

1 — placement

per_token_overhead_ms

1.58 ms — Settings efficiency.gpu_vram.overheadMs — calibrated fit (D-013): Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

bytes_per_token_gb

8.7487 GB

• MoE: only active parameters are read per token (SPEC §3).
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• layer split across 3 GPUs: capacity adds, per-token bandwidth does not (layers run sequentially)
• Per-token overhead of System RAM (44.61 ms) is not charged: this model does not read from that pool.
• Batch size 1 (single user, interactive).

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (bytes_per_token / (bandwidth × efficiency) + overhead)

GPU VRAM: 39.775 GB ÷ (1,792 GB/s × 0.48) + 1.58 ms fixed per-token overhead → 20.91 tok/s

GPU VRAM.bandwidth_gbs

1792 GB/s — component spec

GPU VRAM.efficiency_moe

0.48 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

1 — placement

per_token_overhead_ms

1.58 ms — Settings efficiency.gpu_vram.overheadMs — calibrated fit (D-013): Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

bytes_per_token_gb

39.7749 GB

• MoE: only active parameters are read per token (SPEC §3).
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• layer split across 3 GPUs: capacity adds, per-token bandwidth does not (layers run sequentially)
• Per-token overhead of System RAM (44.61 ms) is not charged: this model does not read from that pool.
• Batch size 1 (single user, interactive).

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (bytes_per_token / (bandwidth × efficiency) + overhead)

GPU VRAM: 5.274 GB ÷ (1,792 GB/s × 0.86) + 1.58 ms fixed per-token overhead → 199.91 tok/s

GPU VRAM.bandwidth_gbs

1792 GB/s — component spec

GPU VRAM.efficiency

0.86 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU VRAM.active_share

1 — placement

per_token_overhead_ms

1.58 ms — Settings efficiency.gpu_vram.overheadMs — calibrated fit (D-013): Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

bytes_per_token_gb

5.2741 GB

• Dense: every parameter is read per token.
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• layer split across 3 GPUs: capacity adds, per-token bandwidth does not (layers run sequentially)
• Per-token overhead of System RAM (44.61 ms) is not charged: this model does not read from that pool.
• Batch size 1 (single user, interactive).

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (bytes_per_token / (bandwidth × efficiency) + overhead)

GPU HBM3e: 8.749 GB ÷ (7,100 GB/s × 0.48) + 1.58 ms fixed per-token overhead → 241.13 tok/s

GPU HBM3e.bandwidth_gbs

7100 GB/s — component spec

GPU HBM3e.efficiency_moe

0.48 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU HBM3e.active_share

1 — placement

per_token_overhead_ms

1.58 ms — Settings efficiency.gpu_vram.overheadMs — calibrated fit (D-013): Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

bytes_per_token_gb

8.7487 GB

• MoE: only active parameters are read per token (SPEC §3).
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• Batch size 1 (single user, interactive).

First reply: estimate — Arithmetic that also depends on a tunable assumption, listed below.

ttft_s = prompt_tokens / prefill_tok_s

= 32,768 / 52,262.73 = 0.6 s

prompt_tokens

32768 tokens

prefill_tok_s

52262.73 tok/s

• No published prefill measurement for this hardware; scaled from peak compute (estimate).
• Model q_dim unknown — attention FLOPs omitted, so long-prompt TTFT is understated.

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (bytes_per_token / (bandwidth × efficiency) + overhead)

GPU HBM3e: 39.775 GB ÷ (7,100 GB/s × 0.48) + 1.58 ms fixed per-token overhead → 75.47 tok/s

GPU HBM3e.bandwidth_gbs

7100 GB/s — component spec

GPU HBM3e.efficiency_moe

0.48 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU HBM3e.active_share

1 — placement

per_token_overhead_ms

1.58 ms — Settings efficiency.gpu_vram.overheadMs — calibrated fit (D-013): Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

bytes_per_token_gb

39.7749 GB

• MoE: only active parameters are read per token (SPEC §3).
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• Batch size 1 (single user, interactive).

First reply: estimate — Arithmetic that also depends on a tunable assumption, listed below.

ttft_s = prompt_tokens / prefill_tok_s

= 32,768 / 15,119.69 = 2.2 s

prompt_tokens

32768 tokens

prefill_tok_s

15119.69 tok/s

• No published prefill measurement for this hardware; scaled from peak compute (estimate).
• Model q_dim unknown — attention FLOPs omitted, so long-prompt TTFT is understated.

Speed: estimate — Arithmetic that also depends on a tunable assumption, listed below.

tok_s = 1 / (bytes_per_token / (bandwidth × efficiency) + overhead)

GPU HBM3e: 5.274 GB ÷ (7,100 GB/s × 0.86) + 1.58 ms fixed per-token overhead → 409.21 tok/s

GPU HBM3e.bandwidth_gbs

7100 GB/s — component spec

GPU HBM3e.efficiency

0.86 — Settings efficiency.gpu_vram — Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

GPU HBM3e.active_share

1 — placement

per_token_overhead_ms

1.58 ms — Settings efficiency.gpu_vram.overheadMs — calibrated fit (D-013): Discrete GPU: 10 dense + 3 MoE points; fit on published benchmarks, leave-one-out validated — docs/research/12b-estimator-validation.md

bytes_per_token_gb

5.2741 GB

• Dense: every parameter is read per token.
• Effective bits include embeddings and higher-precision tensors in mixed quants.
• Assumes active weights are quantized like the file average.
• KV cache is re-read on every token (DECISIONS D-008).
• Batch size 1 (single user, interactive).

First reply: estimate — Arithmetic that also depends on a tunable assumption, listed below.

ttft_s = prompt_tokens / prefill_tok_s

= 30,000 / 508,550.93 = 0.1 s

prompt_tokens

30000 tokens

prefill_tok_s

508550.93 tok/s

• No published prefill measurement for this hardware; scaled from peak compute (estimate).
• Model q_dim unknown — attention FLOPs omitted, so long-prompt TTFT is understated.