Disaggregated Quantization: Specializing LLM Prefill and Decode

1. Task

What problem is being solved: Large Language Model (LLM) inference consists of two distinct phases with different computational characteristics. Prefill (processing the input prompt) is compute-bound, while decode (generating tokens autoregressively) is memory bandwidth-bound.

Why it matters and is hard: Maximizing efficiency requires quantization, but standard quantization schemes apply the exact same format to both phases, forcing a trade-off. Weight-only compression accelerates memory-bound decode but slows down prefill compared to hardware-native compute formats. Conversely, formats that quantize both weights and activations (like NVFP4) speed up prefill but hurt accuracy and fail to maximize memory bandwidth during decode.

The gap targeted: This paper targets this rigid trade-off by proposing "Disaggregated Quantization" (DQ). DQ specializes the computation formats, weight formats, and even weight storage locations to the specific bottleneck of each phase.

2. Core idea

Instead of forcing a single quantization scheme on the entire model, optimize each inference phase independently. The most extreme version (Full Disaggregation) trains the model to use hardware-native, compute-efficient weights (NVFP4) for the prefill phase to rapidly build the KV cache, and extremely compressed, weight-only representations (like 1-3 bit LUT) for the decode phase to maximize memory bandwidth. Both are jointly optimized so the decoder can flawlessly interpret the prefiller's representations.

3. Mechanism + mental model

1. Quantization-Aware Distillation with Disaggregation (QADD): The model is distilled from a teacher. The forward pass uses a label mask to select the pathway: prompt tokens are processed via the prefill quantization format, while generated tokens use the decode format.
2. Format Disaggregation: A simple drop-in intervention where master weights are shared, but activation quantization is completely disabled during decode (e.g., NVFP4 prefill + NVFP4A16 decode).
3. Full Disaggregation: Two entirely separate sets of weights are maintained. Prefill uses its own NVFP4 weights, while decode uses its own 2-3 bit weight-only weights.
4. Offloaded Disaggregated Prefill (ODP): To avoid holding two sets of weights in limited VRAM, the prefill weights reside on an SSD. During prefill, these weights are streamed into device memory block-by-block, overlapping loading with matrix compute, temporarily borrowing space from idle decode weights.

Mental model: Prefill is compute-bound, so give it hardware-native quantized compute formats (NVFP4); decode is memory bandwidth-bound, so give it maximally compact weights (1-4 bit) without the accuracy penalty of activation quantization.

4. Metrics / data

• Benchmarks: Decode-heavy: GSM8K (5-shot), MATH-500 (4-shot), MMLU-Pro (5-shot, text). Prefill-heavy: RULER (13 tasks, 4K to 32K context). Visual: MMMU-Pro (zero-shot).
• Models: Qwen 3 (0.6B to 8B, and 27B), Gemma 3 (270M to 12B). Large-scale PTQ on Qwen 3.8, Gemma 4, Muse Glimmer, Nemotron 3, Kimi-K3 (up to 2.8T parameters).
• Hardware: NVIDIA DGX Spark (GB10, sm_121) for local stack timings; GB300 nodes for large-scale vLLM.
• Serving stack: vLLM and a custom llama.cpp extension for ODP.
• Workload: Batch size 1 for decode (where bandwidth strictly dominates), text and multi-modal reasoning.

5. vs. baseline

• Baselines: Dense unquantized models (BF16), uniform non-disaggregated NVFP4, and uniform weight-only (2-3 bit) models. The baselines were implemented and run by the authors.
• Matched comparisons: Yes, baselines are perfectly matched—same models, hardware, and serving harness.
• Headline gains:
  • On Qwen3.8-27B (1-bit decoder, IQ1_S), training an NVFP4 prefiller improves MMLU-Pro accuracy by +32.5 points and MMMU-Pro by +35.3 points over the weight-only baseline.
  • At 8K prompt length in llama.cpp, ODP achieves a 1.78x time-to-first-token (TTFT) speedup over the weight-only baseline.
  • On massive models (up to 2.8T) with 4-bit Post-Training Quantization (PTQ), format disaggregation (disabling activation quantization on decode) yields small but statistically significant gains (e.g., +1.13 on Gemma-4-31B MMMU-Pro) at zero extra inference cost.

6. Open source

Code and llama.cpp fork released at: https://github.com/IST-DASLab/disaggregated-quantization and https://github.com/IST-DASLab/disaggregated-llama.cpp. Weights (Qwen3.8-27B NVFP4 prefiller) are released on the Hugging Face Hub.

7. Key numbers, verified

8. What the abstract doesn't say

• Regime Boundaries (Bitwidth Sweet Spot): The massive accuracy gains of specialized prefillers vanish and can even reverse at higher bitwidths. For instance, on Qwen3.8-27B at 3-bit decode (IQ3_S), the prefiller causes a 0.63 point drop on MMLU-Pro and a 2.77 point drop on MMMU-Pro. The benefits strictly target extremely compressed (1-2 bit) decoders.
• ODP Overhead at Short Contexts: SSD loading makes ODP slower than resident prefill for short prompts. On DGX Spark, compute only overtakes SSD loading at around 8K context. At >16K, offloading adds a <5% latency overhead.
• Generation Length Alteration: Using a prefiller drastically alters generation behavior. On MMMU-Pro (IQ1_S), the mean response length halved (from 14k to 6.5k tokens) while accuracy skyrocketed. Conversely, on MMLU-Pro, the prefiller often increased the mean length due to a longer generated tail.
• Failure on Mixture of Experts (MoE): The ODP streaming trick does not gracefully transfer to MoE models. For MoEs, the ratio of compute cost to loading cost scales down with the active parameter fraction, making SSD loading substantially more expensive than compute even at extreme context lengths.
• Metric Hygiene: Measurements rigorously disaggregate inference speed into time-to-first-token (TTFT) and per-output-token latency, acknowledging that decoding is tested at batch size 1 where bandwidth limitations are most acute.

9. Reachability + takeaways

• Reachability: High for single-node deployment. The codebase and llama.cpp fork are open-source. For users, loading the provided 27B prefill weights makes ODP instantly usable. Training new prefillers requires QADD distillation, but format disaggregation (turning off activation quantization for decode) is a free PTQ intervention.
• Takeaway (Inference): The KV cache is a universal bottleneck interface. We do not need to compress the model uniformly. We can use thick, compute-friendly formats to build the cache quickly, and heavily compressed, memory-friendly formats to read it during autoregressive generation.
• Takeaway (Inference): ODP is a brilliant systems trick for local LLMs. Since prefill weights are swept exactly once per prompt, they can live on SSD and be streamed just-in-time, keeping VRAM dedicated strictly to the KV cache and decode weights.

10. Confidence + next steps

• Mechanism: High. The core intuition distinguishing compute-bound prefill from memory-bound decode is physically robust.
• Reported gains: Medium-High. The +35 point gains at 1-bit are real, but they are relative to an exceptionally weak, nearly broken baseline. 2-3 bit gains are more measured and realistic.
• Generality: Medium. Datacenter Disaggregated Serving already segregates prefill/decode onto separate machines, but adapting this to single-device local serving via ODP is a strong contribution for dense models (though explicitly noted to fail for MoEs).
• Worth reproducing? Yes. Integrating ODP into consumer stacks like llama.cpp or Ollama could drastically improve UX for running massive dense models on Macs/consumer GPUs.
• Next steps / Open questions:
  • Investigate how this interacts with prompt caching. If KV caches are largely reused, prefill weights are needed even less often.
  • How robust is this in multi-turn chat? If the KV cache is built iteratively over multiple prefill cycles, does the representation drift from what the decode weights naturally produce?

11. Audit

• Downgraded ODP speedup to ⚠️: The 1.78x speedup is at 8K, which is the single most favorable configuration. At contexts < 4K, ODP is actually slower than weight-only prefill due to the constant SSD streaming overhead.
• Downgraded NVFP4 prefiller 32.5 pts gain to ⚠️: This headline number is from the single most favorable configuration (1-bit), where the baseline is severely degraded. The benefit vanishes or reverses at 3-bit.
• Downgraded Full disaggregation vs LUT2A16 to ⚠️: Full disaggregation uses an additional 4-bit (NVFP4) prefill checkpoint, effectively doubling the weight storage required unless ODP is used, making it an unfair capacity comparison against a purely 2-bit model without streaming.
• Verified: All other numbers, claims, and formatting disaggregation points match the paper and are leniently fair. No unverifiable claims found.