Inference Notes · Paper Deep Reads

中文

Full-text deep reads of Agent and LLM-inference papers: mechanism, verified numbers, what the abstract leaves out.

All · 5AgentInference
2026-09
AgentInferenceRead 2026-09-23 · arXiv 2609.26779

CliffCompaction: Cost-Efficient Compaction for Long-Horizon Coding Agents

Trang Nguyen, Eulrang Cho, Bingqing Chen, Tim Dettmers

Problem: Long-horizon coding agents accumulate millions of tokens of context; how to compact it without losing what matters and without re-prefilling (and paying) on every step.

The mechanism is sound and worth borrowing, but headline numbers are oversold: cost savings are solid, 'quality improves' sits within seed noise, and 'matches Opus 4.7' / '25% over specialized methods' are cross-harness / cross-model comparisons.

Trust: Mechanism high / cost medium-high / quality gains low-medium
AgentInferenceRead 2026-09-23 · arXiv 2609.26693

Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation

Lijuan Tang, Yuemeng Zheng

Problem: Local serving stacks silently confound tool-use evaluation by rejecting or altering requests before they reach the model, leading evaluators to misattribute infrastructure failures to model incompetence.

The paper demonstrates that 0% tool-use rates are often artifacts of the serving stack rather than model inability. However, the measurements suffer from extremely small denominators and high variance, meaning the numbers highlight the serving confound but cannot reliably rank model capabilities.

Trust: High confidence in mechanism and measurement caveats, but authors explicitly state their results do not support cross-model capability rankings.
InferenceRead 2026-09-23 · arXiv 2609.26333

Disaggregated Quantization: Specializing LLM Prefill and Decode

Andrei Panferov, Maximilian Kleinegger, Sweta Priyadarshi, Tijmen Blankevoort, Dan Alistarh

Problem: Accelerating LLM inference without sacrificing accuracy by resolving the conflicting quantization requirements of the compute-bound prefill phase and the memory-bound decode phase.

Shows that decoupling quantization formats for prefill (compute-native) and decode (weight-only) significantly improves accuracy and speed, but the SSD streaming optimization (ODP) is ineffective for MoE architectures and short context prompts.

Trust: High confidence in mechanism and metrics. Massive 1-bit gains are robust but rely on a heavily degraded weight-only baseline. Generalizes well to dense models but not MoEs.
AgentInferenceRead 2026-09-23 · arXiv 2609.25611

Qwen3.8-Omni: Towards Native Omni-Modal Agents

Qwen Team

Problem: How to enable omni models to perform long-horizon multimodal productivity tasks, such as video editing and analysis, without incurring prohibitive token costs from processing full-length videos.

Demonstrates that wrapping an omni-model in a native agent harness drastically reduces token consumption for long videos while improving reasoning, though the model still lags behind Gemini 3.8 Flash on several agentic and video reasoning tasks, and some core multilingual translation capabilities regress slightly.

Trust: High confidence in mechanism and benchmark wins; medium confidence in exact cost reduction claims due to missing hardware/serving ablation.
AgentInferenceRead 2026-09-23 · arXiv 2609.25510

Hill Sampling for Test-Time Scaling: A Simple and Better Alternative to Repeated Sampling, Evolution, and Training

Jacob Beck, Philip V. Ogren, Ari Kobren

Problem: The task is to allocate test-time compute for LLMs to effectively discover verifiable algorithmic and mathematical solutions, questioning how much complex evolutionary machinery is actually necessary.

It shows that token sampling with a simple greedy update is a stronger and cheaper exploration mechanism than perturbing weights or maintaining complex populations, though its headline state-of-the-art results rely on the single best seed in float-sensitive environments and struggle without domain-specific prompts.

Trust: Mechanism: High (simple algorithm). Gains: Medium (3 seeds, some float sensitivity). Generality: Medium (only scalar math tasks).