CliffCompaction: Cost-Efficient Compaction for Long-Horizon Coding Agents

1. Verdict

The mechanism is sound and worth borrowing, but headline numbers are oversold. The core contribution is a very simple rule: let the context grow until a threshold, compact only by truncating/dropping (never rewriting), and discard the previous compaction block every time. The cost reduction (cache reads −80%) is solid; "quality maintained or improved" on Terminal-Bench is within seed noise; "matches Opus 4.7" and "25% over specialized methods" are cross-harness / cross-model comparisons. Trust: mechanism high / cost medium-high / quality gains low-medium.

2. Mechanism + mental model

Trigger: only when LLM() raises ContextWindowExceeded (Alg. 1). On compaction, older turns are processed message by message:

1. System prompt + first user message (task description) kept verbatim
2. Most recent K turns kept verbatim
3. Older turns:
   • assistant thinking truncated to 300 chars
   • tool calls reduced to a signature (tool name + path and key arguments, ≤150 chars; inline file contents in write calls removed)
   • tool results ≤500 chars kept, >500 chars dropped entirely
   • the previous compaction block is discarded, never nested into the new one
4. New context = [system, task, this compaction block] + last K turns

Mental model (paper evidence + my inference):

• Cost side: most of an agent's bill is cache reads (78% uncompacted, Terminal-Bench / GLM 5.1), because every step re-reads the whole prefix. Changing the prefix only at the "cliff" keeps the prefix cache valid for the whole segment in between; sliding windows and microcompaction change the prefix almost every step and keep invalidating the cache. So when you compact matters more for cost than what you compact into.
• Quality side: dropping content leaves an obvious hole (signature present, result gone) — the agent knows what it doesn't know and re-reads the file to get ground truth. An LLM summary leaves a plausible-looking paraphrase — the agent believes it already knows, stops going back to the source, and drifts. The authors call this "high compaction precision, low recall".

3. Key numbers, verified

4. Checklist findings (agent checklist)

• Evaluation validity: SWE-bench Verified / Terminal-Bench 2.0 (89 tasks) / KernelBench L3 (50 tasks) are all execution-judged. Fine.
• Reliability: ± std reported but number of seeds not reported; the TB "improvement" is within std. In the Claude Code experiment Cliff's std is 1.41 vs. 3.72 for native compaction — a notable positive signal (lower variance).
• Controlled comparison: main tables compare same model, same scaffold — clean. But model versions drift across experiments: TB/SWE main tables use Kimi K2.6 / GLM 5.1, the compaction comparison uses Kimi K2.7 / GLM 5.2, KernelBench uses Kimi K2.7. The Claude Code comparison is matched on mean peak context (~45K), which is reasonable.
• Serving confounds: cost depends heavily on the provider's prefix-cache behavior (Kimi hit rate 96–98%, GLM 58–91%). The main-table Cost column is an idealized perfect-cache estimate; metered costs are in a separate appendix table and don't match (e.g. GLM 5.1 TB full context: $0.54 main table vs. $0.89 metered).
• Cost reporting: thorough (metered cost, cache hit rate, wall-clock −23% / −34%). The most solid part of the paper.
• Generalization: the SGV selector is trained on another model's rollouts on the same 89 tasks; main results use the 100%-of-tasks version. The appendix "25% of tasks works just as well" only partly mitigates this (evaluation still includes training tasks). SWE-bench features include repo_code (repository identity), a prior-type feature.
• Failure analysis: a good behavioral analysis — re-read counts (appendix Fig. extra-steps) support the precision argument. The 8K threshold clearly degrades; authors acknowledge it.

5. What the abstract doesn't say

• On SWE-bench, test-time scaling doesn't benefit from Cliff: uncompacted Kimi + SGV k=3 = 78.0% @ $284 vs. Cliff 16K k=3 = 76.4% @ $264 (appendix Tab. swebench-scaling). The abstract only cites Terminal-Bench.
• Kimi barely saves money on SWE-bench (cache already 96% hit, short trajectories). Gains concentrate in long trajectories with imperfect cache hit rates.
• Only meaningful for medium-to-long horizon tasks (stated in Limitations).
• Self-hosted vLLM measurement (appendix Tab. glm-flash-time; GLM-4.7-Flash, vLLM, RTX PRO 6000 Blackwell, SWE-bench Verified): total LLM-call time 234.2h → 146.0 / 111.6 / 88.9h, i.e. 1.60× / 2.10× / 2.62× at 32K/16K/8K. But the same model's accuracy (Tab. swe-bench) is 43.20 → 43.60 / 41.20 / 30.70 — the 2.62× costs 12.5 points; 16K is the sensible setting. The appendix reports only the speedup.
• Main-table costs use a "perfect-caching model" (§A.1): any prompt longer than the previous one counts as a full prefix hit; any shorter prompt counts as a full invalidation. This is optimistic for the full-context baseline (real provider hit rates are 58–98%) and conservative for Cliff's re-prefills (the system+task prefix would still hit). Metered costs are in a separate table.
• Cost breakdown (Tab. cost-attribution, GLM 5.1 TB): savings come almost entirely from tool results (−61%) and tool calls (−58%); thoughts save only 11%, system+task actually rise +22% (re-prefill). OpenHands signature rules are in Tab. openhands-signatures (command ≤120 chars, old_str ≤60 chars, task_tracker dropped entirely).
• Commented-out text in the LaTeX source (not part of the paper; my observation):
  • A removed limitation: proprietary models have much shorter trajectories — Opus 4.8 uses ~0.5M tokens / 22 turns on TB 2.1 vs. ~1.7M / 35 for Kimi K2.7 — i.e. the benefit falls mainly on open-weight models. This sentence is gone from the final version.
  • An older compaction comparison had weaker baselines (sliding 2.76×, Anthropic-style 2.71×) and claimed both "underperform no compaction per dollar"; in the final version baseline numbers are higher (2.86× / 3.33× / 3.47×) and that claim was removed.

6. Takeaways (my inference)

1. Compaction timing is an inference-cost problem, not just an agent-design problem: cache hit rate drives the bill. Any context management that edits the prefix (including Claude Code's microcompaction) fights the prefix cache. This directly links the agent and serving tracks — read alongside Hot-Cold KV tiering (2609.25782) and PatchKV (2609.26219, KV repair after editing mid-context).
2. Leave a hole rather than fill it with a paraphrase: letting the agent clearly see what it lost — with a signature to fetch it back — is more reliable than a plausible summary. A transferable harness design principle; same idea as DTOC (placeholder + on-demand restore).
3. For your own Claude Code use: the proxy can sit directly in front of Claude Code. But Claude trajectories are short and cache hit rates high, so per the paper's own deleted paragraph the benefit may be small — better suited to long-running open-weight-model agents.

7. Next steps

• Worth reading the code (the proxy is small) to see how tool-call signatures are built for Claude Code's schema.
• Open question: how many seeds? Is Cliff vs. full context on TB significant across seeds?
• Read next: Anthropic context editing (clear_tool_uses), DTOC (2609.26121), PatchKV (2609.26219).

8. Cross-check: independent full read by Gemini 3.1 Pro

All 22 source files (commented-out drafts removed) were given to Gemini 3.1 Pro (via agy, no tools) for an independent read, without this note. Every point below was re-checked against the source:

• It caught, I had missed: k=3 costs more than Opus 4.6 (now in the verification table); appendix parts I had not read end to end (wall-clock, signature rules, price table).
• I had, it missed: the "matched settings +25%" is a cross-model comparison (it marked it as holding); Opus 4.7's score is from the system card, not the same harness; test-time scaling doesn't help on SWE-bench; summarization ties Cliff on KernelBench; the 8K accuracy drop behind the self-hosted speedup.
• It overstated: that the selector is "evaluated on its own training data". The selector is trained on another model's trajectories; the overlap is in tasks, not trajectories — task-level overlap, not direct label leakage.
• Both agree: "cache reads −90%" has no support in the body (it's 80%); seeds not reported and the TB gain is within std; model versions drift across experiments; at 8K cost can go up rather than down.