Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation
ProblemLocal serving stacks silently confound tool-use evaluation by rejecting or altering requests before they reach the model, leading evaluators to misattribute infrastructure failures to model incompetence.
VerdictThe paper demonstrates that 0% tool-use rates are often artifacts of the serving stack rather than model inability. However, the measurements suffer from extremely small denominators and high variance, meaning the numbers highlight the serving confound but cannot reliably rank model capabilities.
1. Task
What problem is being solved, in concrete terms (inputs, outputs, setting).
The paper investigates how local serving stacks (like Ollama, llama.cpp, vLLM, SGLang) evaluate and handle tool-use requests from coding agents before they are passed to the model. In a standard local agent setup, a harness sends a request containing a prompt and a tool schema (tools=). The server must route this request, format it, and parse the response into a structured tool_call.
Why it matters and why it is hard.
This protocol step is critical but often conflated with model capability. If the server fails to handle the tools= request or fails to parse a valid model output, it usually returns an error string to the harness. Typical per-turn analyses read these transport errors as ordinary non-call responses from the model, leading to silent contamination of the measurement. It is difficult to measure true model capability when the intervening software layer alters or rejects the request independent of the model's weights.
What prior approaches do and where they fall short. Prior benchmarks like BFCL evaluate function calling but do not treat the serving interface itself as an experimental variable. Agent harnesses generally record the assistant message stream but fail to preserve structured error metadata (like HTTP 400 rejection). Thus, when a model like Phi-3 or Gemma-3 is evaluated, it may score 0% fidelity not because it lacks capability, but because the serving layer rejected its native tool-call request entirely, an issue previous benchmarks failed to isolate.
2. Core idea
The core insight is that in local tool-use evaluation, the serving stack—not just the model—determines whether a tool-call request succeeds. By separating the tool channel (native tools= vs. prompt text) and probing different serving backends, the authors show that a "0% tool-use capability" often means the serving layer simply refused the request or failed to parse the response, misattributing an infrastructure failure to model incompetence.
3. Mechanism + mental model
The paper isolates serving layer confounds using three configurations on a fixed aggregation task (8 seeds per model):
- Native: Harness sends default
tools=request. Ollama gates this per model (e.g., Llama-3.2 gets native calls; Qwen gets text; Phi-3/Gemma-3 are rejected with HTTP 400). - Native+hint:
tools=is sent, plus a text tool list and explicit JSON format injected into the prompt. - Text-tools:
tools=is dropped; uniform text guidance is provided, and calls are parsed from text.
Mental model: The serving layer is an opaque gatekeeper: if the model is not on the server's VIP list for native tools, the request bounces before reaching the weights, but the harness records it as the model choosing to do nothing.
4. Metrics / data
- Benchmarks/Datasets: A fixed aggregation task requiring several tool calls. Replicated on a dependency-chain task (4 seeds) and single-turn HumanEval (6 seeds).
- Metrics: Protocol fidelity (valid in-schema call rate over turns the model actually produced). Also reports turn-pooled rate, per-seed mean (with seed-level bootstrap 95% CIs), and per-turn outcome composition.
- Models: Qwen2.5-Coder (0.5B, 1.5B, 3B, 7B, 14B), Llama-3.2-3B, Phi-3-mini, Gemma-3-4B, Gemma-3-270m. Cloud anchor: deepseek-v4-flash.
- Hardware: Local CPU-only for Ollama/llama.cpp. vLLM (T4) and SGLang (A800 80GB) used for cross-stack checks.
- Serving stack/harness: ReAct coding-agent harness (LOCA-bench extension). Serving backends: Ollama (0.30.8), llama.cpp, vLLM, SGLang.
5. vs. baseline
The paper does not propose a new model or method, but compares standard serving setups against control configurations.
- Native vs. Text-tools: On Ollama, Phi-3 and Gemma-3 score 0% (rejected) under native, but achieve 38% and 38%-100% respectively under text-tools. Qwen models (0.5B-14B) see massive gains (up to 59 percentage points, reaching 89%) when a text hint is added.
- Llama-3.2 exception: Native+hint reaches 82%, but uniform text-tools protocol drops it to 44% because it discards Llama's real native support.
- Cross-stack baseline: The same GGUF weights that Ollama rejects run successfully on llama.cpp. vLLM defaults to rejecting the request entirely (unless specific flags and parsers are provided).
- Constrained Decoding baseline: Using structured outputs guarantees 100% valid in-schema calls on a single step for all 9 local models, but causes weaker models to enter infinite loops (600-900 turns) in agentic settings. All baselines were run by the authors under matched task and hardware conditions.
6. Open source
Code is released at https://github.com/LijuanTang94/serving-confound-repo
7. Key numbers, verified
| Claim | Where (§/Tab/Fig) | Actual condition | Holds? (✅/⚠️/❌) | Note |
|---|---|---|---|---|
| Phi-3 and Gemma-3 reported at 0% due to native rejection | §1, §4, Table 1 | Ollama native FC | ✅ | 100% of turns are rejected HTTP 400 requests. Holds specifically for the Ollama version tested (0.30.8). |
| Adding text hint to native channel recovers most fidelity for accepted models | §4, Fig 3, Table 1 | Qwen 0.5B-14B, Llama-3.2 | ✅ | Jumps from 0-60% to 35-89% per-seed. The gain is substantial, though absolute magnitudes remain noisy. |
| Uniform text protocol lowers fidelity for Llama-3.2 | §4, Fig 3, Table 1 | Llama-3.2-3B | ✅ | Drops from 82% (native+hint) to 44% (text-tools) as the text parser discards its native capability. |
| Turn-pooled vs per-seed estimates differ by up to 55 points | Abstract, §4, Table 1 | Qwen-0.5B text-tools | ⚠️ | True, but this 51-56 point difference is an extreme outlier driven by a single 41-turn looping episode, not typical variance. |
| Constrained decoding removes protocol failure but induces non-termination | §4 | All 9 local models | ⚠️ | 100% valid 1-step holds for all 9 models, but the 600-900 turn infinite loops only occur on the weaker models, not all 9. |
| Same weights rejected by Ollama run on llama.cpp | §4, Table 2 | Phi-3, Gemma-3 | ✅ | llama.cpp handles them with 200 text/native. |
8. What the abstract doesn't say
- Inference/Serving Confounds: The rejection mechanisms are stack-and-version specific. Ollama 0.30.8 gates by model template. vLLM requires explicit
--enable-auto-tool-choiceand--tool-call-parser. SGLang accepts by default but returns text unless parsed. - Metric Hygiene: Denominators for weaker models are extremely thin (8-16 turns over 8 seeds), leading to massive bootstrap 95% CIs (e.g., Qwen-0.5B text-tools is 34% [9,59]).
- Non-response vs Fidelity: The "protocol fidelity" metric excludes non-responses. For instance, under text-tools, Gemma-3-4B appears to have 100% fidelity, but this comes from emitting only one valid call per seed and then failing to respond (Fig 4).
- Generalization: The paper explicitly refuses to make claims about model scale laws, model families, or reasoning capabilities, as the noise and variance are too high.
- Your Inference: Practitioners must log transport errors (HTTP 4xx, timeouts) separately from model responses. Treating the serving stack as a transparent layer is statistically dangerous.
9. Reachability + takeaways
- Reachability: High. Code and prompts are available. Can be run locally on standard hardware (CPU/consumer GPU). Requires pinning the specific Ollama version (0.30.8) to replicate the exact native gating behavior.
- Takeaways (Inference): Always pin and report the serving stack and its version. A "0% success rate" might be a 400 Bad Request error. Check if your serving backend requires explicit parser flags before declaring a model lacks tool-use.
- Takeaways (Agents): Agent harnesses must persist structured error metadata, not just the assistant message stream. Use per-seed rates with confidence intervals rather than turn-pooled rates, which can be artificially inflated by a single infinite loop.
10. Confidence + next steps
- Mechanism: High. The cross-stack probe clearly isolates the serving layer as the cause.
- Reported gains/variance: High. The authors are extremely transparent about thin denominators and wide intervals.
- Generality: Medium. Specific gating behaviors are tied to Ollama 0.30.8, but the broader principle of serving confounds applies universally.
- Worth reproducing? Yes, the checklist for logging transport errors should be implemented in any local evaluation harness immediately.
- What to read next? BFCL (for benchmark baselines), LOCA-bench, and literature on inference backends as measurement variables (e.g., SilentHyper).
- Open questions: How do standardized cloud APIs (e.g., OpenAI, Anthropic) gate native tool-use for unsupported or fine-tuned endpoints? Does the Model Context Protocol (MCP) introduce similar translation confounds?
11. Audit
- task: Updated meta.json task statement to be a direct problem statement rather than a description of what the paper investigates.
- verdict: Added the paper's main limitation (extremely small denominators and high variance making cross-model ranking unreliable) to the meta.json verdict.
- Qwen models (0.5B-14B) see massive gains: Clarified the phrase "gains (up to 89%)" to mean an absolute gain of up to 59 percentage points (reaching 89%).
- Phi-3 and Gemma-3 reported at 0% due to native rejection: Added a note that the gating behavior is specifically tied to Ollama version 0.30.8.
- Adding text hint to native channel recovers most fidelity: Included Llama-3.2 as an accepted model that also recovered fidelity, not just Qwen models.
- Turn-pooled vs per-seed estimates differ by up to 55 points: Downgraded to ⚠️ because the 55-point difference is an extreme outlier driven by a single 41-turn looping episode on a weak model, not a typical variance.
- Constrained decoding removes protocol failure but induces non-termination: Downgraded to ⚠️ because the note incorrectly claimed all 9 local models looped 600-900 turns; the paper states this non-termination only happens for the weaker models.
- Unverifiable: Whether the specific error text 'Failed to get response after multiple retries' applies to all ReAct harness implementations, though it is true for the LOCA-bench extension tested by the authors.