How Cerebras Fits a Trillion-Parameter Model Across 44 GB Chips
A Cerebras chip holds 44 GB, while a three-trillion-parameter model needs about 1.5 TB even at 4 bits. Why does Cerebras map it to 59 wafers, not just enough for the weights?
- Layers and request state must share memory, so weight size alone does not determine the wafer count.
- Cerebras maps its roughly three-trillion-parameter example to 59 wafers using 4-bit weights and layer-specific cache allowances.
- Active parameters describe computation; full-model storage and complete server configurations require different counts.
Distribute the layers, then reserve space for request state
Cerebras’ worked example uses 44 GB WSE-3T wafers, 4-bit weights, an 8-bit KV cache and an assumed average request length of 256k. Its 69 linear-attention layers occupy 35 wafers at two per wafer; 24 full-attention layers each occupy one, producing 59. Intermediate activations move between wafers.
The weight arithmetic is straightforward: three trillion parameters at four bits each occupy 1.5 trillion bytes, or 1.5 TB in decimal units. Spread evenly over 93 layers, that is about 16.1 GB per layer. Dividing the weight payload by chip capacity alone leaves out the space required to serve requests.
Kimi K3 lists 2.8T total parameters, not 3T exactly
Moonshot AI’s model card separates total parameters from the parameters activated for each token.
| Model-card field | Value | Meaning |
|---|---|---|
| Total parameters | 2.8T | The complete model. |
| Activated parameters | 104B | The active parameter count per token. |
| Attention composition | 69 KDA + 24 Gated MLA | The attention types across 93 layers. |
Sparse activation describes computation. It does not make the inactive weights disappear from a full-model storage calculation. Likewise, the rounded quantity in an architectural example should not replace the model card’s precise total.
A wafer is not a complete CS-4 system
The CS-4 product page specifies three WSE-3 Turbo processors per system. Its I/O design connects wafers within and across racks. A chip mapping therefore cannot be read directly as a count of servers to order.
CS-4 also separates compute, power and I/O through the Nexus platform. Each wafer assembly incorporates power conversion, direct liquid cooling, high-speed I/O and control electronics. Those are deployment requirements beyond the weight-capacity calculation; a purchasing configuration needs to account for them.
Capacity does not establish the user’s wait time
Cerebras’ Dedicated Inference metrics distinguish request, token and latency measurements. Time to first token, or TTFT, measures the initial response: a p95 TTFT of 0.5 seconds means 95% of requests receive their first token within that interval. It says nothing by itself about when a whole answer finishes.
The metrics API serves dedicated endpoints and aggregates the last complete minute. For a deployment decision, pair the capacity plan with response measurements at the request lengths the application will actually use.
Frequently asked questions
Can storage be sized using only active parameters?
The active count describes per-token computation, not the complete set of model weights.
Does a wafer count determine a hardware quote?
No. Systems package multiple chips and require power, cooling and interconnect capacity.
Keep four quantities distinct: total weights, per-layer headroom, system configuration and measured request latency.