Language models trained on static corpora are frozen at their training cutoff: they cannot fetch tomorrow’s weather, query a live database, execute arithmetic with guaranteed precision, or invoke an external service. These limitations are not incidental deficiencies but structural properties of the autoregressive generation paradigm. Tool use is the mechanism by which agents bridge the gap between language model reasoning and world action, converting natural-language intention into structured external calls whose results ground subsequent reasoning.
229.1 Why Tools Are Necessary
A language model generates tokens by approximating the conditional distribution \(P(x_t \mid x_1, \ldots, x_{t-1})\) learned from training data. This distribution encodes knowledge about the world only up to the training cutoff and only insofar as that knowledge appeared in the training corpus. Three categories of tasks are therefore structurally outside the model’s native capability.
Real-time knowledge. Events after the training cutoff are unknown to the model. A user asking “what is the current Euro/USD exchange rate?” receives either an admission of ignorance or a confidently hallucinated stale figure. A tool call to a currency API grounds the response in live data.
Precise computation. Transformer models are poor calculators. While they can approximate arithmetic for small numbers seen frequently in training, they have no reliable mechanism for multi-digit multiplication or floating-point arithmetic. A Python interpreter tool executes such computations exactly.
External state manipulation. Creating a calendar event, sending an email, writing to a database, or triggering a deployment pipeline requires side effects in external systems. Language generation alone cannot produce these effects; only structured API calls can.
Tool use does not require fine-tuning. A sufficiently capable model can learn to produce structured outputs from in-context demonstrations or system prompts. Fine-tuning improves reliability and reduces hallucination, as the Gorilla work demonstrates (Section 4), but the architectural pattern is prompt-level.
229.2 Function Calling Mechanics
The canonical mechanism for tool use is function calling, introduced in the OpenAI Chat Completions API and now a near-universal interface. The protocol has four components.
Schema declaration. At inference time, the model receives a list of JSON Schema objects describing available tools. Each schema specifies the function name, a natural-language description, and a parameters object describing argument names, types, and descriptions. The model uses these schemas to determine when to call a tool and how to construct a valid argument bundle.
{"name":"get_current_weather","description":"Retrieve current weather conditions for a location.","parameters":{"type":"object","properties":{"location":{"type":"string","description":"City and country, e.g. 'Paris, France'"},"unit":{"type":"string","enum":["celsius","fahrenheit"],"description":"Temperature unit to return"}},"required":["location"]}}
Structured call generation. When the model determines that a tool call is appropriate, it generates a structured object rather than free text. In the OpenAI format this appears in a dedicated tool_calls field of the assistant message, containing id, type: "function", and a function object with name and arguments (a JSON string). The model does not continue generating conversational tokens until the tool result is returned.
Execution and result injection. The runtime deserializes the argument bundle, dispatches to the appropriate function, and captures the return value. The result is injected into the conversation as a tool message keyed to the original call ID. In multi-tool scenarios, multiple calls may be batched in a single assistant turn and results returned in parallel.
Continued reasoning. After receiving tool results the model resumes generation with a richer context: its original reasoning, the structured call it made, and the observed output. This grounds subsequent generation in verified external data.
The conversation structure is therefore a sequence of alternating roles:
For multi-step tasks this sequence may cycle through the tool-call/tool-result pair multiple times before a final answer is produced.
229.2.1 Tools as Code: The Programmatic Alternative
The protocol above carries a constraint that is easy to miss, because it is baked into the transport rather than stated anywhere. Every argument bundle and every return value crosses the context window as JSON text, and every invocation costs an inference turn: the model emits a call, generation halts, the runtime replies, generation resumes. For a single weather lookup this is invisible. For a task requiring forty calls, or a chain of twelve in which each argument depends on the previous result, the transport becomes the binding constraint rather than the reasoning.
A different arrangement is available to any model that can write code. Instead of publishing JSON Schemas and waiting for structured calls, the runtime publishes the tools as typed function stubs, a source module the model reads and imports, and gives the model a sandbox in which to run scripts. The model writes a program that calls the stubs, loops over them, branches on their results and prints a reduced answer; the runtime executes that program in a subprocess and returns stdout. Patel et al. (2026) name this programmatic tool calling (PTC) and give the first systematic comparison against native JSON calling on an established benchmark across model generations.
The weather tool of the previous subsection becomes a signature rather than a schema:
# tools.py, whose source is injected into the system promptfrom typing import Literaldef get_current_weather( location: str, unit: Literal["celsius", "fahrenheit"] ="celsius",) ->dict:"""Retrieve current weather conditions for a location. Args: location: City and country, e.g. 'Paris, France' unit: Temperature unit to return """
The annotations carry what properties carried, the default carries what required carried, and the docstring carries the description fields. Nothing in the contract is lost. It is restated in the notation the model has seen most often during training, which is the paper’s title in miniature.
The four components of the JSON protocol map across directly, and the third is where the difference lives.
Table 229.1: Correspondence between the two calling conventions.
JSON calling
Programmatic calling
Schema declaration: JSON Schema objects passed through the API tools parameter
Stub declaration: annotated function source placed in the system prompt
Structured call generation: one tool_calls object per invocation
Script generation: one program containing all invocations, with loops and conditionals
Execution and result injection: runtime dispatches each call and appends a tool message
Sandbox execution: runtime runs the script in a subprocess and appends captured stdout
Continued reasoning: resumes after every call
Continued reasoning: resumes once, after the script terminates
The conversational shape collapses accordingly. Where JSON calling for an \(N\)-invocation task produces
for every \(N\). The iteration has moved out of the conversation and into the script, where it costs no inference turns and no context.
Three structural consequences.Serial depth becomes cheap: a twelve-step chain in which each call consumes the previous return value is twelve statements in one program rather than twelve round trips, so the compounding dependency that Section 6 identifies as the primary source of fragility is spanned by a single generation instead of twelve. Fan-out width stops being rationed by the model’s willingness to emit many parallel call objects in one message, because a loop over fifty items is the same three lines as a loop over five. Context pressure falls, because the script filters and aggregates before printing, so intermediate payloads never enter the window at all. The first two are efficiency claims. The third is an accuracy claim, and it is the one that matters most under adversarial conditions.
A token model for fan-out. Let a task require \(N\) invocations. Under JSON calling the context accumulates a fixed schema block of \(S\) tokens plus, for each invocation, an argument bundle \(c_a\), a returned payload \(c_r\), and per-message protocol overhead \(c_o\) covering call identifiers, role wrappers and tool-message envelopes:
Under programmatic calling the context accumulates the stub module source \(P\), the script the model writes, and the text the script prints:
\[
C_{\mathrm{ptc}}(N) = P + \ell(N) + \rho(N).
\tag{229.2}\]
Two facts about the right-hand side do the work. When the \(N\) invocations are homogeneous, the script is a loop whose source length does not grow with \(N\), so \(\ell(N) \approx \ell_0\). And because the script aggregates before printing, \(\rho\) scales with the size of the answer rather than with the number of calls. Writing \(\gamma\) for whatever residual marginal cost survives, \(C_{\mathrm{ptc}}(N) \approx (P + \ell_0) + \gamma N\) with \(\gamma \ll c_a + c_r + c_o\). Two lines with a positive intercept gap and a larger slope gap cross exactly once, at
The numerator is positive in practice, because stub source carrying full signatures, docstrings and the prose that teaches the calling convention is bulkier than the equivalent schemas. Programmatic calling is therefore strictly more expensive on narrow tasks and strictly cheaper on wide ones, and \(N^{*}\) is the only quantity a practitioner needs in order to choose between them.
Patel et al. (2026) measure both sides of Equation 229.3. The fixed penalty is real: programmatic calling consumes roughly \(1.5\times\) the input tokens of the baseline on their chaining ablation, which is \(P > S\) appearing exactly where Equation 229.2 says it should, on tasks whose \(N\) is small. The two slopes are recoverable from a pair of reported fan-out measurements, and the crossover they imply is then a check on the model rather than an assumption fed into it.
JSON marginal 85.4 tok/call, fixed block 996 tok
PTC marginal 8.6 tok/call, fixed block 3122 tok
crossover N* = 27.7 (authors report approximately 26)
Figure 229.1: Left: input token cost against fan-out width, with each paradigm fitted as a straight line through the two measurements reported by Patel et al. (2026) at N = 30 and N = 48. The fixed stub-module preamble makes programmatic calling more expensive on narrow tasks while the per-call JSON overhead makes it cheaper on wide ones, producing a single crossover near N = 28 against the value of about 26 the authors report. Right: the enumeration accuracy cliff measured on Claude Sonnet 5, where JSON calling holds full accuracy out to a fan-out of 70 and then collapses within thirty further items, while the programmatic form is still complete at 100.
What the benchmark shows. The evaluation covers fourteen models released between November 2024 and July 2026, on a 309-entry subset of BFCL v4 sampled proportionally across eight task categories. On the main subset, programmatic calling matches or exceeds JSON calling for eleven of the fourteen, with the strongest gains around ten percentage points. Chaining and fan-out, the two regimes the structural argument above predicts should favour it, are where the separation is widest: on chains of length two to twenty the leading models rise from roughly 81% to the mid-nineties, and on enumeration tasks programmatic calling matches or beats the baseline in thirteen of fourteen models. Latency tracks the turn count, with programmatic calling completing chaining entries in roughly half the wall-clock time for thirteen of the fourteen.
The context-rot ablation is the most interesting result, because it probes the third consequence listed above rather than the first two. Flooding the context with irrelevant material degrades JSON calling by 2.3% on average, and a filesystem-discovery variant by 32.0%, while programmatic calling improves by 5.5%. The asymmetry follows from where intermediate results live. Under JSON calling every payload joins the same window the distractors are polluting; under programmatic calling those payloads stay in the sandbox, and only the script’s summary is ever exposed to the pollution.
The bitter lesson. The three models that lose badly, by roughly twenty to twenty-seven percentage points, are all older OpenAI releases, and the cause is not a reasoning failure. They emit literal \n escape sequences where a multiline script needs real newlines, so the subprocess raises a syntax error before any tool is reached. Newer releases in the same family do not do this. The gap therefore tracks model generation rather than model family, and it closes without anyone redesigning the scaffold. That is the argument the title makes: a calling convention engineered around the assumption that models cannot reliably write code is a convention with an expiry date, and the structural advantages of expressing tool use in a general-purpose language accrue automatically as the underlying models improve.
What the evaluation does not establish. Three limits are worth carrying forward, two of them stated by the authors. BFCL v4 stubs are echo-return, so the benchmark scores argument serialization accuracy rather than end-to-end tool-use correctness; a script that constructs perfect arguments and then mishandles a real API’s error response scores identically to one that does not. The ablations run on 31 to 52 entries, so the aggregate direction is informative while individual per-model deltas are not. And the work is a recent preprint without peer review. The structural claims about turn count and context isolation follow from the mechanics and would survive a change of benchmark; the specific accuracy figures should be read as a first measurement rather than a settled quantity.
229.3 ReAct: Interleaved Reasoning and Acting
Yao et al. (2023) formalized the observation that language models produce better tool-augmented outputs when reasoning and action are explicitly interleaved rather than either separated or implicit. The ReAct (Reason + Act) framework structures agent traces as alternating thought and action steps.
A ReAct trace for a question-answering task might read:
Thought: I need to find the year Beyonce's sixth album was released.
Action: Search("Beyonce sixth studio album")
Observation: Beyonce's sixth studio album "Lemonade" was released in April 2016.
Thought: I now know the release year is 2016. I should verify the director of the title track.
Action: Search("Lemonade Beyonce title track director")
Observation: The title track was directed by Kahlil Joseph and Beyonce herself.
Thought: I have enough information to answer.
Action: Finish("Lemonade was released in 2016; the title track was co-directed by Kahlil Joseph.")
This structure differs from chain-of-thought (CoT) prompting in a critical way: CoT generates reasoning entirely within the model’s parametric knowledge, while ReAct grounds each reasoning step in observed external evidence. The observation at each step constrains subsequent thoughts, preventing the compounding hallucinations that afflict multi-step CoT.
Yao et al. evaluated ReAct on HotpotQA (multi-hop question answering), Fever (fact verification), ALFWorld (text-based household tasks), and WebShop (product-search navigation). ReAct outperformed CoT-only baselines on all four tasks and produced traces that human raters found more interpretable and easier to correct. On HotpotQA, ReAct achieved 35.1% success versus 29.4% for CoT with the same model, a substantial gain attributable entirely to grounding.
The information-theoretic motivation is straightforward. Let \(K_\theta\) be the model’s parametric knowledge and \(K_e\) be external evidence retrieved at step \(t\). CoT conditions only on \(K_\theta\); ReAct conditions on \(K_\theta \cup \bigcup_t K_e^{(t)}\). Because \(K_e^{(t)}\) is retrieved in response to intermediate reasoning, it is specifically relevant to the current subproblem. The effective knowledge available to ReAct is therefore a superset of CoT’s knowledge, targeted rather than broad.
229.4 Gorilla: Reducing API Hallucination via Retrieval-Aware Training
A persistent failure mode in tool use is parameter hallucination: the model invokes a real function name but fabricates argument names, types, or values that do not conform to the actual API. This is particularly dangerous for SDK calls where incorrect arguments cause silent wrong behavior rather than runtime errors.
Patil et al. (2023) addressed this with Gorilla, a LLaMA-based model fine-tuned on over 1,600 API documentation pages spanning TorchHub, TensorFlow Hub, and HuggingFace model APIs. The contribution has two parts: a dataset and a training strategy.
APIBench. The authors constructed APIBench by scraping API documentation, writing questions that require API calls to answer, and recording the correct API invocation as the label. The benchmark covers three domains with distinct calling conventions: TorchHub uses Python torch.hub.load, TensorFlow Hub uses tensorflow_hub.load, and HuggingFace uses pipeline with model identifiers. Hallucination is measured as the fraction of generated calls that either name a nonexistent API or supply incorrect arguments.
Retrieval-aware training. The key insight is that API documentation is too voluminous to fit in context but too specific to memorize reliably. Gorilla’s training procedure pairs each training example with retrieved documentation: a retriever fetches the most relevant API documentation for the question, that documentation is prepended to the context, and the model is trained to generate the correct API call given this augmented context. At inference time the same retriever runs before generation, simulating the training distribution.
This contrasts with naive RAG, where retrieval is added at inference time without training-time conditioning. By training with retrieved context, the model learns to use documentation rather than to override it with parametric memorization. The result is a substantial reduction in hallucination: Gorilla achieves 20.43% AST accuracy on HuggingFace versus 8.22% for GPT-3.5-Turbo in the zero-shot setting, nearly a 2.5x improvement.
The architectural takeaway is general: when tools have complex or frequently changing interfaces, fine-tuning with retrieval-in-context produces more reliable callers than prompting alone. This principle extends beyond API calling to any domain where the “schema” of valid actions evolves faster than model retraining cycles.
229.5 MM-ReAct: Multimodal Tool Composition
Yang et al. (2023) extended the ReAct framework to multimodal inputs by treating specialized vision models as tools. MM-ReAct allows a language model (GPT-4 in the original paper) to reason about images through a composition of external vision APIs rather than through its own vision encoder.
The available tools include an image captioner, an OCR engine, a celebrity/landmark recognizer, a Bing image search, an image editing tool, and a math solver for image-embedded equations. When presented with an image, the language model does not process pixel values directly. Instead, it decomposes the visual question into a sequence of tool calls:
Thought: The image may contain text I need to read.
Action: OCR(image)
Observation: "Net Income: $4.2B (up 12% YoY)"
Thought: I need context about this company to interpret the number.
Action: BingSearch("company name from financial report header")
Observation: The report header shows "Acme Corp Q3 2024"
Thought: I can now answer the question about profitability trends.
This decomposition has an important epistemic property: each tool call produces a verifiable, human-interpretable intermediate result. The trace is auditable in a way that end-to-end vision-language generation is not. A reviewer can inspect whether the OCR output was correct, whether the search query was appropriate, and whether the final reasoning step is valid given the observations.
MM-ReAct demonstrates that tool use is not merely a patch for knowledge gaps but a compositional architecture: a language model can serve as an orchestrator that coordinates specialist models, with each specialist optimized for a narrow perceptual or computational task. The language model contributes planning, disambiguation, and synthesis; the tools contribute precision.
229.6 Chain-of-Abstraction Reasoning
Gao et al. (2024) identified a subtle failure mode in direct tool use: when a model must invoke multiple tools in a chain, it tends to fix its reasoning structure around the first tool call’s result, making it brittle to errors or unexpected outputs in that call. Chain-of-Abstraction (CoA) reasoning addresses this by separating the reasoning graph from tool execution.
In CoA, the model first produces a complete reasoning trace using abstract placeholders for tool outputs:
Step 1: Compute the compound interest on $1000 at 5% for 3 years. Call this CALC_1.
Step 2: The population in 2024 is BASE_POP. Retrieve this from the census API. Call this LOOKUP_1.
Step 3: The answer is CALC_1 divided by LOOKUP_1.
Only after the reasoning structure is fixed does the runtime make tool calls to fill in CALC_1 and LOOKUP_1, substituting results and evaluating the final expression. The advantage is that the reasoning graph is produced by the model in a single forward pass, without the sequential dependency on intermediate results that makes direct tool use brittle. Errors in one tool call can be diagnosed at a specific node in the graph rather than propagating through downstream reasoning.
Gao et al. report that CoA outperforms direct ReAct-style tool use on multi-step mathematical reasoning and knowledge-intensive QA tasks. The gain is largest on tasks requiring three or more sequential tool calls, consistent with the hypothesis that compounding dependency is the primary source of fragility.
229.7 Tool Design Principles
Effective tool use depends as much on tool design as on model capability. Four principles govern reliable tool interfaces.
Reliability. Tools should produce deterministic or near-deterministic outputs for a given input. Stochastic tools (e.g., tools that call another language model internally) introduce variance that the orchestrating model cannot account for. When stochasticity is unavoidable, tools should communicate uncertainty explicitly in their return values.
Documentation. The JSON Schema description fields are the only signal the model has about what a tool does and when to use it. Descriptions must be precise, unambiguous, and include examples of correct argument values. Vague descriptions (“performs some data operation”) lead to misuse; specific descriptions (“returns the closing price of a NASDAQ ticker on a given trading day in YYYY-MM-DD format”) enable reliable invocation.
Atomicity. Tools should have a single, well-defined responsibility. A tool that “searches the web and summarizes results” conflates retrieval with summarization; if either step fails, the failure is opaque. Separate tools for “web search returning raw snippets” and “summarize text” give the model control over each step and produce interpretable error signals.
Parseable outputs. Tool return values should be structured (JSON, typed dataclasses) rather than free text. Free-text returns require the model to parse natural language before reasoning over results, introducing an additional failure mode. Structured returns enable direct field access in subsequent reasoning.
Two of these four principles bind harder under the programmatic convention of Section 229.2.1, and for the same underlying reason: the consumer of a tool’s output changes from a language model reading a transcript to a program indexing a value. Atomicity stops being merely good hygiene and becomes compositional necessity, because the model is now writing control flow over the tools rather than selecting one per turn, and a tool that fuses retrieval with summarization cannot be placed inside a loop that needs the raw snippets. Parseable outputs stop being a convenience and become a hard requirement, because a free-text return that a model could have read charitably will instead be subscripted by a script and raise an exception. Documentation shifts rather than tightens, moving out of description strings and into type annotations, defaults and docstrings, where the same information is expressed in the notation the model has the most training exposure to.
229.7.1 Context Management as a Tool Rather Than a Harness Policy
The four principles above concern tools that reach outward, into the world. A less obvious application turns the same machinery inward. The agent’s context window is itself a scarce resource, and the operations that manage it can be exposed as callable tools rather than hard-wired into the runtime loop. Li et al. (2026) makes precisely this move with Agentic Context Management (ACM), which equips agents with purpose-built context editing tools and lets the agent decide when to invoke them. Their diagnosis of the status quo is sharp: existing compression schemes lose information and fire on rigid heuristic rules, so the moment of compression is chosen by a policy that knows nothing about what the agent currently cares about.
A budgeted buffer. Model the context as a buffer of capacity \(B\) tokens into which each turn appends roughly \(a\) tokens, so the window holds about \(k = B/a\) turns of history. Define the reuse distance\(D_i\) of an appended item as the number of turns between the moment it is written and the next turn at which it is actually needed. Under threshold truncation that drops the oldest entries, item \(i\) is destroyed before its next use exactly when \(D_i > k\), so the probability that a still-needed token has been evicted is simply the tail of the reuse distance distribution:
\[\Pr[\text{loss}] = \Pr[D > k], \qquad k \approx B/a.\]
This is a discouraging quantity, because reuse distances in long-horizon trajectories are heavy tailed. A file read in the opening minutes of a coding episode is consulted again after two hundred intervening turns; a search result is revisited after a long detour into an unrelated subgoal. If \(D\) has a Pareto tail with index \(\alpha\), then \(\Pr[D > k] \propto k^{-\alpha}\), and for \(\alpha\) near one, doubling the context budget barely halves the loss. Buying more window is a weak lever against a fat tail.
Eviction as a decision rather than a consequence. Let \(E_i \in \{0,1\}\) indicate that item \(i\) leaves the working context before its next use, and \(Y_i \in \{0,1\}\) indicate that item \(i\) is in fact needed again. Expected unrecoverable loss over an episode is
where \(r_i\) is the probability the content can be fetched back when the need arises. Threshold truncation is the degenerate corner of this expression: \(E_i\) is a deterministic function of position alone, carrying no information about \(Y_i\) beyond what recency itself implies, and \(r_i = 0\) because destroyed content is destroyed. Each term collapses back to \(\Pr[D > k]\).
ACM attacks both factors independently. Offloading rather than deleting makes \(r_i > 0\): discarded content is written to an external memory system and queried back on demand, so lossless here names a move rather than a drop. And letting the agent choose the moment makes \(E_i\) a function of the agent’s own belief about future use, introducing a correlation
between the eviction decision and the true future-use indicator that no positional rule can have. Read the second identity carefully, because the sign is the whole point. Holding the number of evicted items fixed pins \(\mathbb{E}[E]\), so \(L\) is increasing in \(\rho\): a policy that evicts precisely the items it will need again is the worst possible one. The objective is to drive \(\rho\) as far negative as the marginals allow, evicting exactly the material the agent does not expect to consult. A policy with \(\rho = 0\) is evicting blind, and pays the base rate.
This is the classical frame of cache replacement theory, and it is worth naming explicitly. Belady’s MIN algorithm evicts the line with the largest forward reuse distance and is optimal precisely because it has oracle access to \(Y\), which is to say it attains the most negative \(\rho\) the marginals permit. FIFO and fixed-threshold truncation sit some way up from that floor, using recency as a weak proxy that buys only a mildly negative \(\rho\). Real caches close the gap with profiled or learned hints, and ACM’s wager is that a model reasoning over its own trajectory is a better hint generator than a positional rule, because it knows which subgoal is active and therefore which observations are finished being useful. The post-training pipeline in Li et al. (2026) exists to sharpen that signal, constructing demonstrations of good context management so the policy learns when to edit, not merely how.
Tool interface. The framework reduces to three atomic operations over a single external store. The schemas below are our own illustrative rendering of those operations in the style of Section 2, written to show what the interface has to express; they are not a transcription of the published ACM API, and field names, defaults, and argument shapes in Li et al. (2026) differ. Note that the three satisfy the atomicity principle: compression, eviction, and retrieval are separable, so a failed query is distinguishable from a bad compression.
[{"name":"context_compress","description":"Replace a contiguous span of the working context with a summary, keeping the span retrievable in full from external memory. Use when a subtask is complete and only its conclusion is still relevant.","parameters":{"type":"object","properties":{"span_ids":{"type":"array","items":{"type":"string"},"description":"Message or observation IDs to compress"},"summary":{"type":"string","description":"Replacement text preserving conclusions and open questions"}},"required":["span_ids","summary"]}},{"name":"context_offload","description":"Move items out of the working context into external memory without leaving a summary. Use for observations that are finished being useful, e.g. raw tool output already acted upon.","parameters":{"type":"object","properties":{"span_ids":{"type":"array","items":{"type":"string"}},"tags":{"type":"array","items":{"type":"string"},"description":"Retrieval keys, e.g. file paths or entity names"}},"required":["span_ids"]}},{"name":"context_query","description":"Retrieve previously compressed or offloaded content back into the working context. Returns matching items with their original IDs.","parameters":{"type":"object","properties":{"query":{"type":"string","description":"Natural-language description of the needed content"},"top_k":{"type":"integer","description":"Maximum items to return (default 3)"}},"required":["query"]}}]
Li et al. (2026) report three effects from this arrangement, and the third is easy to overlook. Peak token pressure falls, because the agent compresses before the buffer is desperate rather than after. Explorations run longer, because the effective horizon is no longer capped by the raw window. And solutions become more consistent across independent trials, which is a statement about variance rather than about the mean: two runs of the same task that discard different material diverge, whereas two runs that retain the same decisive evidence converge. The simulation below speaks to the second and third only. It cannot speak to the first, because a fair comparison has to impose the same hard budget and the same compaction trigger on every policy, which fixes peak occupancy by construction and leaves nothing to measure there. What remains is the pair of quantities the setup genuinely isolates: how much needed material each policy destroys, and how much that quantity varies from run to run.
import numpy as npimport matplotlib.pyplot as pltSEED =20260726B =12_000# context budget in tokensN_TURNS =400# turns in one long horizon episodeALPHA =0.9# Pareto tail index of the reuse distanceP_REUSE =0.45# fraction of items ever needed againSIGNAL_ACC =0.85# agent accuracy at predicting future reuseRECALL =0.90# probability an offloaded item is queried backN_TRIALS =40HI, LO =0.92* B, 0.60* B # identical trigger and target for every policy,# so peak occupancy is held fixed by constructiondef trajectory(rng):"""One episode: token size and next-need turn for each appended item.""" size = rng.lognormal(np.log(400.0), 0.6, N_TURNS) reused = rng.random(N_TURNS) < P_REUSE dist = np.ceil(rng.pareto(ALPHA, N_TURNS) +1.0).astype(int) nxt = np.arange(N_TURNS) + distreturn size, np.where(reused & (nxt < N_TURNS), nxt, -1)def run(policy, rng): size, need_at = trajectory(rng) truth = need_at >=0 flipped = rng.random(N_TURNS) > SIGNAL_ACC pred = np.where(flipped, ~truth, truth) # agent's noisy "still useful" belief schedule = {}for i, t inenumerate(need_at):if t >=0: schedule.setdefault(int(t), []).append(i) offloads = policy =="agent_offload" live, order, memory = {}, [], set() occ = peak =0.0 needed = lost = queries =0def make_room(incoming):nonlocal occif occ + incoming <= HI:returnif offloads: # agent ranks by predicted future use cand = [i for i in order ifnot pred[i]] + [i for i in order if pred[i]]elif policy =="random_evict": cand = [int(i) for i in rng.permutation(order)]else: # rigid threshold truncation: drop oldest cand =list(order)for i in cand:if occ + incoming <= LO:break occ -= live.pop(i) order.remove(i)if offloads: memory.add(i)for t inrange(N_TURNS): make_room(size[t]) live[t] = size[t]; order.append(t); occ += size[t] peak =max(peak, occ)for i in schedule.get(t, []): needed +=1if i in live:continueif i in memory: # recoverable: pay a query, not a loss queries +=1if rng.random() < RECALL: make_room(size[i]) live[i] = size[i]; order.append(i); occ += size[i] peak =max(peak, occ)continue lost +=1return peak / B, lost /max(needed, 1), queriesPOLICIES = ["threshold_truncate", "random_evict", "agent_offload"]res = {p: {"peak": [], "loss": [], "q": []} for p in POLICIES}master = np.random.default_rng(SEED)for _ inrange(N_TRIALS): s =int(master.integers(1<<31)) # same trajectory for all three policiesfor p in POLICIES: pk, ls, q = run(p, np.random.default_rng(s)) res[p]["peak"].append(pk); res[p]["loss"].append(ls); res[p]["q"].append(q)print(f"{'policy':<20}{'loss':>9}{'loss sd':>10}{'spread':>9}{'queries':>9}{'peak/B':>9}")print("="*66)for p in POLICIES: ls = np.array(res[p]["loss"]); pk = np.array(res[p]["peak"])print(f"{p:<20}{ls.mean():>9.4f}{ls.std(ddof=1):>10.4f}{ls.max() - ls.min():>9.4f}"f"{np.mean(res[p]['q']):>9.1f}{pk.mean():>9.3f}")print("\npeak/B is equal by construction (shared cap), not a measured difference")LABELS = ["threshold\ntruncation", "random\neviction", "agent-invoked\noffload"]COLORS = ["#b5651d", "#7a7a7a", "#1f6f8b"]fig, ax = plt.subplots(1, 2, figsize=(8.6, 3.6))x = np.arange(3)mu = np.array([np.mean(res[p]["loss"]) for p in POLICIES])sd = np.array([np.std(res[p]["loss"], ddof=1) for p in POLICIES])ax[0].bar(x, mu, yerr=sd, capsize=5, color=COLORS)ax[0].set_ylabel("needed but unrecoverable")ax[0].set_title("Information loss (mean $\\pm$ sd)")jit = np.random.default_rng(1).normal(0, 0.06, (3, N_TRIALS))for j, p inenumerate(POLICIES): v = np.array(res[p]["loss"]) ax[1].scatter(j + jit[j], v, s=14, alpha=0.65, color=COLORS[j]) ax[1].hlines(v.mean(), j -0.28, j +0.28, color="k", lw=1.6)ax[1].set_ylabel("loss rate per trial")ax[1].set_title(f"Run to run spread ({N_TRIALS} trials)")for a in ax: a.set_xticks(x); a.set_xticklabels(LABELS, fontsize=8.5) a.spines[["top", "right"]].set_visible(False)fig.tight_layout()plt.show()
policy loss loss sd spread queries peak/B
==================================================================
threshold_truncate 0.0589 0.0204 0.0754 0.0 0.919
random_evict 0.1925 0.0297 0.1387 0.0 0.919
agent_offload 0.0106 0.0071 0.0279 14.1 0.919
peak/B is equal by construction (shared cap), not a measured difference
Figure 229.2: Long-horizon buffer simulation over 40 independent trials. All three policies run under an identical hard budget and an identical compaction trigger, so peak occupancy is held fixed by construction and only the eviction rule varies. Agent-invoked offload with retrieval on miss cuts unrecoverable information loss (left) and tightens the run-to-run spread (right) relative to rigid threshold truncation and random eviction.
Random eviction is the control that isolates the mechanism: it runs under the same budget and the same trigger as the other two but with \(\rho \approx 0\), and at a loss rate of 0.193 it destroys roughly three times as much needed material as threshold truncation at 0.059. Threshold truncation does better only because recency is a weak proxy for future use, worth a mildly negative \(\rho\). Agent-invoked offload combines a more negative \(\rho\) with \(r_i > 0\) and lands at 0.011, a further factor of about five, at a cost of roughly 14 retrieval queries per episode. The run-to-run standard deviation falls with it, from 0.020 under truncation to 0.007, which is the variance result in miniature: policies that destroy material at rule-chosen moments make outcomes depend on which trajectory happened to cross the threshold when. Peak occupancy is 0.919 for all three, which is the shared cap read back out rather than a finding. Compacting early is a design choice a harness can impose on any eviction rule, and it is imposed identically here precisely so that it cannot flatter the agent policy.
The design lesson closes the loop on this section’s argument about where behavior belongs. Threshold truncation lives in the harness because the harness is the only component that can see the token counter, but it is the wrong place to make the decision, since the harness cannot see intent. The correct division is that the harness owns the mechanism and the invariant (it enforces the hard budget and will truncate if the agent does not act) while the agent owns the policy, invoking context_compress and context_offload at moments it selects. Exposing context management as a tool is not a convenience wrapper; it moves a decision to the only component holding the information needed to make it well, which is the same argument that justifies giving the model a calculator instead of a rounding rule.
229.8 Implementing a Tool-Use Loop
The following implementation demonstrates the complete tool-use loop without external API dependencies, using a rule-based dispatcher in place of a language model to make the mechanics visible.
import jsonimport mathfrom typing import Any, CallableTOOL_REGISTRY: dict[str, dict[str, Any]] = {}def tool(schema: dict[str, Any]) -> Callable:def decorator(fn: Callable) -> Callable: TOOL_REGISTRY[schema["name"]] = {"fn": fn, "schema": schema}return fnreturn decorator@tool({"name": "calculator","description": "Evaluate a mathematical expression. Returns a float.","parameters": {"type": "object","properties": {"expression": {"type": "string","description": "A Python arithmetic expression, e.g. '2 ** 10 + sqrt(144)'" } },"required": ["expression"] }})def calculator(expression: str) ->float: allowed = {k: getattr(math, k) for k indir(math) ifnot k.startswith("_")} allowed["abs"] =absreturnfloat(eval(expression, {"__builtins__": {}}, allowed))@tool({"name": "lookup_capital","description": "Return the capital city of a country.","parameters": {"type": "object","properties": {"country": {"type": "string", "description": "Country name in English"} },"required": ["country"] }})def lookup_capital(country: str) ->str: db = {"France": "Paris", "Germany": "Berlin", "Japan": "Tokyo","Brazil": "Brasilia", "Australia": "Canberra", "Canada": "Ottawa", }return db.get(country, f"Unknown capital for '{country}'")def dispatch(call: dict[str, Any]) -> Any: name = call["name"] args = call.get("arguments", {})ifisinstance(args, str): args = json.loads(args)if name notin TOOL_REGISTRY:return {"error": f"No tool named '{name}'"}return TOOL_REGISTRY[name]["fn"](**args)def mock_llm(messages: list[dict]) ->dict:""" Minimal rule-based mock that mimics the assistant turn of an LLM with function-calling capability. Parses the last user message to decide whether to call a tool or produce a final answer. """ last_user =next( (m["content"] for m inreversed(messages) if m["role"] =="user"), "" ) tool_results = [m for m in messages if m["role"] =="tool"]if"capital"in last_user.lower() andnot tool_results: country ="France"if"france"in last_user.lower() else"Japan"return {"role": "assistant","tool_calls": [ {"id": "tc_1", "type": "function","function": {"name": "lookup_capital","arguments": json.dumps({"country": country})}} ] }if"sqrt"in last_user.lower() or"**"in last_user.lower():ifnot tool_results: expr ="sqrt(144) + 2 ** 10"return {"role": "assistant","tool_calls": [ {"id": "tc_2", "type": "function","function": {"name": "calculator","arguments": json.dumps({"expression": expr})}} ] }if tool_results: last_result = tool_results[-1]["content"]return {"role": "assistant","content": f"Based on the tool result: {last_result}" }return {"role": "assistant", "content": "I can answer that directly: 42."}def run_agent(user_query: str, max_steps: int=6) ->str: messages = [{"role": "user", "content": user_query}]for step inrange(max_steps): response = mock_llm(messages) messages.append(response)if"tool_calls"notin response:return response.get("content", "")for call in response["tool_calls"]: fn_info = call["function"] result = dispatch(fn_info) messages.append({"role": "tool","tool_call_id": call["id"],"content": str(result) })return"Max steps reached without final answer."queries = ["What is the capital of France?","Compute sqrt(144) + 2**10","What is the meaning of life?",]for q in queries: answer = run_agent(q)print(f"Q: {q}\nA: {answer}\n")
Running this loop produces:
Q: What is the capital of France?
A: Based on the tool result: Paris
Q: Compute sqrt(144) + 2**10
A: Based on the tool result: 1036.0
Q: What is the meaning of life?
A: I can answer that directly: 42.
The mock demonstrates the structural mechanics: messages accumulate, tool calls are dispatched to a typed registry, results are injected as tool-role messages, and the agent terminates on a content-only assistant turn. Replacing mock_llm with an actual API call (passing tools=[s["schema"] for s in TOOL_REGISTRY.values()] and parsing the response’s tool_calls field) converts this skeleton into a production loop.
The second demonstration shows tool composition: computing a result that requires the calculator tool while leaving the reasoning structure in the calling code.
This pattern implements Chain-of-Abstraction manually: the computation graph is expressed in Python (the “abstract” layer), and tool calls fill in the numerical leaves.
Call precision and recall. Does the model invoke the correct tool for each sub-problem (precision), and does it invoke all necessary tools (recall)? APIBench measures this via Abstract Syntax Tree (AST) comparison of generated versus reference API calls.
Argument correctness. Even when the correct tool is named, arguments may be wrong. Argument correctness requires field-level comparison and is more sensitive than name-level matching. Gorilla’s primary gain over baseline models is in argument correctness rather than tool selection.
Efficiency. An agent that correctly answers a question in 10 tool calls when 2 suffice is expensive to run. Trajectory length is a proxy for reasoning efficiency and correlates with latency and API cost.
Saturation under fan-out. All three dimensions above are measured at a task’s natural width, which on most benchmark entries is a handful of calls. Holding the task fixed and sweeping the number of required invocations exposes a fourth dimension that fixed-width evaluation cannot see. Patel et al. (2026) run this sweep and find a cliff rather than a slope: on an enumeration task, JSON calling on Claude Sonnet 5 returns every item correctly out to a fan-out of 70, drops to 75% at 72, and reaches zero by 100, while the programmatic form of Section 229.2.1 is still complete at 100. Any evaluation whose entries all sit to the left of an agent’s cliff will report perfect enumeration and predict nothing about production traffic that occasionally sits to the right of it.
The cliff also matters because of how it fails. A model that drops thirty of a hundred required calls does not announce an error; it emits a well-formed, plausible answer over the subset it did enumerate, and every downstream check on syntactic validity and argument correctness passes. Detecting silent truncation requires an expected-cardinality check against the request, which is a property of the harness rather than of the model, and which most tool-use scaffolds do not implement.
Common failure modes include: invoking a tool when parametric knowledge suffices (over-reliance), failing to invoke a tool when external grounding is needed (under-reliance), generating syntactically valid but semantically incorrect argument bundles (hallucination), and entering cycles where repeated tool calls return the same observation without triggering a state change in reasoning.
Mitigations include explicit stopping criteria in system prompts, tool-use training data that includes negative examples (situations where no tool call is correct), and structured output validation before dispatch.
229.10 References
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=WE_vluYUL-X
Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023). Gorilla: Large Language Model Connected with Massive APIs. arXiv preprint. https://arxiv.org/abs/2305.15334
Yang, Z., Li, L., Wang, J., Lin, K., Azarnasab, E., Ahmed, F., Liu, Z., Liu, C., Zeng, M., & Wang, L. (2023). MM-ReAct: Prompting ChatGPT for Multimodal Reasoning and Action. arXiv preprint. https://arxiv.org/abs/2303.11381
Gao, J., Gu, L., Su, J., Guo, Y., Li, M., Li, J., Chen, W., & Bi, W. (2024). Efficient Tool Use with Chain-of-Abstraction Reasoning. arXiv preprint. https://arxiv.org/abs/2401.17464
Li, X., Ming, R., Chu, M., Shao, S., Jin, R., & Xiong, C. (2026). ACM: Agentic Context Management for Long Horizon Tasks. arXiv preprint. https://arxiv.org/abs/2607.23809
Patel, I., Sen, S., Lumer, E., & Subbiah, V. K. (2026). The Bitter Lesson of Tool Calling. arXiv preprint. https://arxiv.org/abs/2608.06370
Li, Xiaochuan, Ryan Ming, Meng Chu, Shuai Shao, Rong Jin, and Chenyan Xiong. 2026. “ACM: Agentic Context Management for Long Horizon Tasks.”arXiv Preprint arXiv:2607.23809.
Patel, Ishan, Sahil Sen, Elias Lumer, and Vamse Kumar Subbiah. 2026. “The Bitter Lesson of Tool Calling.”arXiv Preprint arXiv:2608.06370. https://doi.org/10.48550/arXiv.2608.06370.
Source Code
# Tool Use in Language Model Agents {#sec-tool-use}Language models trained on static corpora are frozen at their training cutoff: they cannot fetch tomorrow's weather, query a live database, execute arithmetic with guaranteed precision, or invoke an external service. These limitations are not incidental deficiencies but structural properties of the autoregressive generation paradigm. Tool use is the mechanism by which agents bridge the gap between language model reasoning and world action, converting natural-language intention into structured external calls whose results ground subsequent reasoning.## Why Tools Are NecessaryA language model generates tokens by approximating the conditional distribution $P(x_t \mid x_1, \ldots, x_{t-1})$ learned from training data. This distribution encodes knowledge about the world only up to the training cutoff and only insofar as that knowledge appeared in the training corpus. Three categories of tasks are therefore structurally outside the model's native capability.**Real-time knowledge.** Events after the training cutoff are unknown to the model. A user asking "what is the current Euro/USD exchange rate?" receives either an admission of ignorance or a confidently hallucinated stale figure. A tool call to a currency API grounds the response in live data.**Precise computation.** Transformer models are poor calculators. While they can approximate arithmetic for small numbers seen frequently in training, they have no reliable mechanism for multi-digit multiplication or floating-point arithmetic. A Python interpreter tool executes such computations exactly.**External state manipulation.** Creating a calendar event, sending an email, writing to a database, or triggering a deployment pipeline requires side effects in external systems. Language generation alone cannot produce these effects; only structured API calls can.Tool use does not require fine-tuning. A sufficiently capable model can learn to produce structured outputs from in-context demonstrations or system prompts. Fine-tuning improves reliability and reduces hallucination, as the Gorilla work demonstrates (Section 4), but the architectural pattern is prompt-level.## Function Calling MechanicsThe canonical mechanism for tool use is function calling, introduced in the OpenAI Chat Completions API and now a near-universal interface. The protocol has four components.**Schema declaration.** At inference time, the model receives a list of JSON Schema objects describing available tools. Each schema specifies the function name, a natural-language description, and a `parameters` object describing argument names, types, and descriptions. The model uses these schemas to determine when to call a tool and how to construct a valid argument bundle.```json{"name":"get_current_weather","description":"Retrieve current weather conditions for a location.","parameters": {"type":"object","properties": {"location": {"type":"string","description":"City and country, e.g. 'Paris, France'" },"unit": {"type":"string","enum": ["celsius","fahrenheit"],"description":"Temperature unit to return" } },"required": ["location"] }}```**Structured call generation.** When the model determines that a tool call is appropriate, it generates a structured object rather than free text. In the OpenAI format this appears in a dedicated `tool_calls` field of the assistant message, containing `id`, `type: "function"`, and a `function` object with `name` and `arguments` (a JSON string). The model does not continue generating conversational tokens until the tool result is returned.**Execution and result injection.** The runtime deserializes the argument bundle, dispatches to the appropriate function, and captures the return value. The result is injected into the conversation as a tool message keyed to the original call ID. In multi-tool scenarios, multiple calls may be batched in a single assistant turn and results returned in parallel.**Continued reasoning.** After receiving tool results the model resumes generation with a richer context: its original reasoning, the structured call it made, and the observed output. This grounds subsequent generation in verified external data.The conversation structure is therefore a sequence of alternating roles:$$\text{system} \to \text{user} \to \text{assistant (tool\_call)} \to \text{tool (result)} \to \text{assistant (final)}$$For multi-step tasks this sequence may cycle through the tool-call/tool-result pair multiple times before a final answer is produced.### Tools as Code: The Programmatic Alternative {#sec-485-programmatic-tool-calling}The protocol above carries a constraint that is easy to miss, because it is baked into the transport rather than stated anywhere. Every argument bundle and every return value crosses the context window as JSON text, and every invocation costs an inference turn: the model emits a call, generation halts, the runtime replies, generation resumes. For a single weather lookup this is invisible. For a task requiring forty calls, or a chain of twelve in which each argument depends on the previous result, the transport becomes the binding constraint rather than the reasoning.A different arrangement is available to any model that can write code. Instead of publishing JSON Schemas and waiting for structured calls, the runtime publishes the tools as **typed function stubs**, a source module the model reads and imports, and gives the model a sandbox in which to run scripts. The model writes a program that calls the stubs, loops over them, branches on their results and prints a reduced answer; the runtime executes that program in a subprocess and returns stdout. @patel2026bitter name this **programmatic tool calling** (PTC) and give the first systematic comparison against native JSON calling on an established benchmark across model generations.The weather tool of the previous subsection becomes a signature rather than a schema:```python# tools.py, whose source is injected into the system promptfrom typing import Literaldef get_current_weather( location: str, unit: Literal["celsius", "fahrenheit"] ="celsius",) ->dict:"""Retrieve current weather conditions for a location. Args: location: City and country, e.g. 'Paris, France' unit: Temperature unit to return """```The annotations carry what `properties` carried, the default carries what `required` carried, and the docstring carries the `description` fields. Nothing in the contract is lost. It is restated in the notation the model has seen most often during training, which is the paper's title in miniature.The four components of the JSON protocol map across directly, and the third is where the difference lives.| JSON calling | Programmatic calling ||---|---|| Schema declaration: JSON Schema objects passed through the API `tools` parameter | Stub declaration: annotated function source placed in the system prompt || Structured call generation: one `tool_calls` object per invocation | Script generation: one program containing all invocations, with loops and conditionals || Execution and result injection: runtime dispatches each call and appends a tool message | Sandbox execution: runtime runs the script in a subprocess and appends captured stdout || Continued reasoning: resumes after every call | Continued reasoning: resumes once, after the script terminates |: Correspondence between the two calling conventions. {#tbl-485-ptc-correspondence}The conversational shape collapses accordingly. Where JSON calling for an $N$-invocation task produces$$\text{system} \to \text{user} \to \big(\text{assistant (tool\_call)} \to \text{tool (result)}\big)^{N} \to \text{assistant (final)},$$the programmatic form produces$$\text{system (stubs)} \to \text{user} \to \text{assistant (script)} \to \text{tool (stdout)} \to \text{assistant (final)}$$for every $N$. The iteration has moved out of the conversation and into the script, where it costs no inference turns and no context.**Three structural consequences.** *Serial depth* becomes cheap: a twelve-step chain in which each call consumes the previous return value is twelve statements in one program rather than twelve round trips, so the compounding dependency that Section 6 identifies as the primary source of fragility is spanned by a single generation instead of twelve. *Fan-out width* stops being rationed by the model's willingness to emit many parallel call objects in one message, because a loop over fifty items is the same three lines as a loop over five. *Context pressure* falls, because the script filters and aggregates before printing, so intermediate payloads never enter the window at all. The first two are efficiency claims. The third is an accuracy claim, and it is the one that matters most under adversarial conditions.**A token model for fan-out.** Let a task require $N$ invocations. Under JSON calling the context accumulates a fixed schema block of $S$ tokens plus, for each invocation, an argument bundle $c_a$, a returned payload $c_r$, and per-message protocol overhead $c_o$ covering call identifiers, role wrappers and tool-message envelopes:$$C_{\mathrm{json}}(N) = S + N\,(c_a + c_r + c_o).$$ {#eq-485-json-tokens}Under programmatic calling the context accumulates the stub module source $P$, the script the model writes, and the text the script prints:$$C_{\mathrm{ptc}}(N) = P + \ell(N) + \rho(N).$$ {#eq-485-ptc-tokens}Two facts about the right-hand side do the work. When the $N$ invocations are homogeneous, the script is a loop whose source length does not grow with $N$, so $\ell(N) \approx \ell_0$. And because the script aggregates before printing, $\rho$ scales with the size of the answer rather than with the number of calls. Writing $\gamma$ for whatever residual marginal cost survives, $C_{\mathrm{ptc}}(N) \approx (P + \ell_0) + \gamma N$ with $\gamma \ll c_a + c_r + c_o$. Two lines with a positive intercept gap and a larger slope gap cross exactly once, at$$N^{*} = \frac{(P + \ell_0) - S}{(c_a + c_r + c_o) - \gamma}.$$ {#eq-485-crossover}The numerator is positive in practice, because stub source carrying full signatures, docstrings and the prose that teaches the calling convention is bulkier than the equivalent schemas. Programmatic calling is therefore strictly more expensive on narrow tasks and strictly cheaper on wide ones, and $N^{*}$ is the only quantity a practitioner needs in order to choose between them.@patel2026bitter measure both sides of @eq-485-crossover. The fixed penalty is real: programmatic calling consumes roughly $1.5\times$ the input tokens of the baseline on their chaining ablation, which is $P > S$ appearing exactly where @eq-485-ptc-tokens says it should, on tasks whose $N$ is small. The two slopes are recoverable from a pair of reported fan-out measurements, and the crossover they imply is then a check on the model rather than an assumption fed into it.```{python}#| label: fig-485-ptc-token-crossover#| fig-cap: "Left: input token cost against fan-out width, with each paradigm fitted as a straight line through the two measurements reported by Patel et al. (2026) at N = 30 and N = 48. The fixed stub-module preamble makes programmatic calling more expensive on narrow tasks while the per-call JSON overhead makes it cheaper on wide ones, producing a single crossover near N = 28 against the value of about 26 the authors report. Right: the enumeration accuracy cliff measured on Claude Sonnet 5, where JSON calling holds full accuracy out to a fan-out of 70 and then collapses within thirty further items, while the programmatic form is still complete at 100."#| code-fold: falseimport numpy as npimport matplotlib.pyplot as plt# Reported input-token consumption at two fan-out widths (Patel et al., 2026).N_OBS = np.array([30.0, 48.0])JSON_OBS = np.array([3559.0, 5097.0])PTC_OBS = np.array([3380.0, 3535.0])def fit_line(x, y):"""Two points determine the affine token model of each paradigm.""" slope = (y[1] - y[0]) / (x[1] - x[0])return slope, y[0] - slope * x[0]m_json, b_json = fit_line(N_OBS, JSON_OBS) # (c_a + c_r + c_o), and Sm_ptc, b_ptc = fit_line(N_OBS, PTC_OBS) # gamma, and (P + l_0)n_star = (b_ptc - b_json) / (m_json - m_ptc) # the crossover equationprint(f"JSON marginal {m_json:5.1f} tok/call, fixed block {b_json:6.0f} tok")print(f"PTC marginal {m_ptc:5.1f} tok/call, fixed block {b_ptc:6.0f} tok")print(f"crossover N* = {n_star:.1f} (authors report approximately 26)")# Reported enumeration accuracy for Claude Sonnet 5 under increasing fan-out.fanout = np.array([10.0, 40.0, 70.0, 72.0, 100.0])json_acc = np.array([100.0, 100.0, 100.0, 75.0, 0.0])ptc_acc = np.full_like(json_acc, 100.0)fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(11, 4.2))grid = np.linspace(0, 60, 200)ax1.plot(grid, b_json + m_json * grid, color="#c44e52", lw=2, label="JSON tool calling")ax1.plot(grid, b_ptc + m_ptc * grid, color="#4c72b0", lw=2, label="Programmatic tool calling")ax1.scatter(N_OBS, JSON_OBS, color="#c44e52", zorder=5, s=45)ax1.scatter(N_OBS, PTC_OBS, color="#4c72b0", zorder=5, s=45)ax1.axvline(n_star, color="0.4", ls=":", lw=1.4)ax1.annotate(f"$N^*\\approx{n_star:.0f}$", xy=(n_star, 0), xytext=(n_star +2.5, 1500), color="0.25")ax1.set_xlabel("fan-out width $N$ (tool invocations)")ax1.set_ylabel("input tokens consumed")ax1.set_title("Token cost crosses over once")ax1.set_ylim(0, 6400)ax1.legend(frameon=False, loc="upper left", fontsize=9)ax2.plot(fanout, json_acc, "o-", color="#c44e52", lw=2, label="JSON tool calling")ax2.plot(fanout, ptc_acc, "s--", color="#4c72b0", lw=2, label="Programmatic tool calling")ax2.set_xlabel("fan-out width $N$ (items to enumerate)")ax2.set_ylabel("enumeration accuracy (%)")ax2.set_title("The fan-out cliff (Claude Sonnet 5)")ax2.set_ylim(-5, 108)ax2.legend(frameon=False, loc="lower left", fontsize=9)for ax in (ax1, ax2): ax.spines[["top", "right"]].set_visible(False)fig.tight_layout()plt.show()```**What the benchmark shows.** The evaluation covers fourteen models released between November 2024 and July 2026, on a 309-entry subset of BFCL v4 sampled proportionally across eight task categories. On the main subset, programmatic calling matches or exceeds JSON calling for eleven of the fourteen, with the strongest gains around ten percentage points. Chaining and fan-out, the two regimes the structural argument above predicts should favour it, are where the separation is widest: on chains of length two to twenty the leading models rise from roughly 81% to the mid-nineties, and on enumeration tasks programmatic calling matches or beats the baseline in thirteen of fourteen models. Latency tracks the turn count, with programmatic calling completing chaining entries in roughly half the wall-clock time for thirteen of the fourteen.The context-rot ablation is the most interesting result, because it probes the third consequence listed above rather than the first two. Flooding the context with irrelevant material degrades JSON calling by 2.3% on average, and a filesystem-discovery variant by 32.0%, while programmatic calling *improves* by 5.5%. The asymmetry follows from where intermediate results live. Under JSON calling every payload joins the same window the distractors are polluting; under programmatic calling those payloads stay in the sandbox, and only the script's summary is ever exposed to the pollution.**The bitter lesson.** The three models that lose badly, by roughly twenty to twenty-seven percentage points, are all older OpenAI releases, and the cause is not a reasoning failure. They emit literal `\n` escape sequences where a multiline script needs real newlines, so the subprocess raises a syntax error before any tool is reached. Newer releases in the same family do not do this. The gap therefore tracks model *generation* rather than model *family*, and it closes without anyone redesigning the scaffold. That is the argument the title makes: a calling convention engineered around the assumption that models cannot reliably write code is a convention with an expiry date, and the structural advantages of expressing tool use in a general-purpose language accrue automatically as the underlying models improve.**What the evaluation does not establish.** Three limits are worth carrying forward, two of them stated by the authors. BFCL v4 stubs are echo-return, so the benchmark scores argument serialization accuracy rather than end-to-end tool-use correctness; a script that constructs perfect arguments and then mishandles a real API's error response scores identically to one that does not. The ablations run on 31 to 52 entries, so the aggregate direction is informative while individual per-model deltas are not. And the work is a recent preprint without peer review. The structural claims about turn count and context isolation follow from the mechanics and would survive a change of benchmark; the specific accuracy figures should be read as a first measurement rather than a settled quantity.## ReAct: Interleaved Reasoning and ActingYao et al. (2023) formalized the observation that language models produce better tool-augmented outputs when reasoning and action are explicitly interleaved rather than either separated or implicit. The **ReAct** (Reason + Act) framework structures agent traces as alternating thought and action steps.A ReAct trace for a question-answering task might read:```Thought: I need to find the year Beyonce's sixth album was released.Action: Search("Beyonce sixth studio album")Observation: Beyonce's sixth studio album "Lemonade" was released in April 2016.Thought: I now know the release year is 2016. I should verify the director of the title track.Action: Search("Lemonade Beyonce title track director")Observation: The title track was directed by Kahlil Joseph and Beyonce herself.Thought: I have enough information to answer.Action: Finish("Lemonade was released in 2016; the title track was co-directed by Kahlil Joseph.")```This structure differs from chain-of-thought (CoT) prompting in a critical way: CoT generates reasoning entirely within the model's parametric knowledge, while ReAct grounds each reasoning step in observed external evidence. The observation at each step constrains subsequent thoughts, preventing the compounding hallucinations that afflict multi-step CoT.Yao et al. evaluated ReAct on HotpotQA (multi-hop question answering), Fever (fact verification), ALFWorld (text-based household tasks), and WebShop (product-search navigation). ReAct outperformed CoT-only baselines on all four tasks and produced traces that human raters found more interpretable and easier to correct. On HotpotQA, ReAct achieved 35.1% success versus 29.4% for CoT with the same model, a substantial gain attributable entirely to grounding.The information-theoretic motivation is straightforward. Let $K_\theta$ be the model's parametric knowledge and $K_e$ be external evidence retrieved at step $t$. CoT conditions only on $K_\theta$; ReAct conditions on $K_\theta \cup \bigcup_t K_e^{(t)}$. Because $K_e^{(t)}$ is retrieved in response to intermediate reasoning, it is specifically relevant to the current subproblem. The effective knowledge available to ReAct is therefore a superset of CoT's knowledge, targeted rather than broad.## Gorilla: Reducing API Hallucination via Retrieval-Aware TrainingA persistent failure mode in tool use is parameter hallucination: the model invokes a real function name but fabricates argument names, types, or values that do not conform to the actual API. This is particularly dangerous for SDK calls where incorrect arguments cause silent wrong behavior rather than runtime errors.Patil et al. (2023) addressed this with **Gorilla**, a LLaMA-based model fine-tuned on over 1,600 API documentation pages spanning TorchHub, TensorFlow Hub, and HuggingFace model APIs. The contribution has two parts: a dataset and a training strategy.**APIBench.** The authors constructed APIBench by scraping API documentation, writing questions that require API calls to answer, and recording the correct API invocation as the label. The benchmark covers three domains with distinct calling conventions: TorchHub uses Python `torch.hub.load`, TensorFlow Hub uses `tensorflow_hub.load`, and HuggingFace uses `pipeline` with model identifiers. Hallucination is measured as the fraction of generated calls that either name a nonexistent API or supply incorrect arguments.**Retrieval-aware training.** The key insight is that API documentation is too voluminous to fit in context but too specific to memorize reliably. Gorilla's training procedure pairs each training example with retrieved documentation: a retriever fetches the most relevant API documentation for the question, that documentation is prepended to the context, and the model is trained to generate the correct API call given this augmented context. At inference time the same retriever runs before generation, simulating the training distribution.This contrasts with naive RAG, where retrieval is added at inference time without training-time conditioning. By training with retrieved context, the model learns to use documentation rather than to override it with parametric memorization. The result is a substantial reduction in hallucination: Gorilla achieves 20.43% AST accuracy on HuggingFace versus 8.22% for GPT-3.5-Turbo in the zero-shot setting, nearly a 2.5x improvement.The architectural takeaway is general: when tools have complex or frequently changing interfaces, fine-tuning with retrieval-in-context produces more reliable callers than prompting alone. This principle extends beyond API calling to any domain where the "schema" of valid actions evolves faster than model retraining cycles.## MM-ReAct: Multimodal Tool CompositionYang et al. (2023) extended the ReAct framework to multimodal inputs by treating specialized vision models as tools. **MM-ReAct** allows a language model (GPT-4 in the original paper) to reason about images through a composition of external vision APIs rather than through its own vision encoder.The available tools include an image captioner, an OCR engine, a celebrity/landmark recognizer, a Bing image search, an image editing tool, and a math solver for image-embedded equations. When presented with an image, the language model does not process pixel values directly. Instead, it decomposes the visual question into a sequence of tool calls:```Thought: The image may contain text I need to read.Action: OCR(image)Observation: "Net Income: $4.2B (up 12% YoY)"Thought: I need context about this company to interpret the number.Action: BingSearch("company name from financial report header")Observation: The report header shows "Acme Corp Q3 2024"Thought: I can now answer the question about profitability trends.```This decomposition has an important epistemic property: each tool call produces a verifiable, human-interpretable intermediate result. The trace is auditable in a way that end-to-end vision-language generation is not. A reviewer can inspect whether the OCR output was correct, whether the search query was appropriate, and whether the final reasoning step is valid given the observations.MM-ReAct demonstrates that tool use is not merely a patch for knowledge gaps but a compositional architecture: a language model can serve as an orchestrator that coordinates specialist models, with each specialist optimized for a narrow perceptual or computational task. The language model contributes planning, disambiguation, and synthesis; the tools contribute precision.## Chain-of-Abstraction ReasoningGao et al. (2024) identified a subtle failure mode in direct tool use: when a model must invoke multiple tools in a chain, it tends to fix its reasoning structure around the first tool call's result, making it brittle to errors or unexpected outputs in that call. **Chain-of-Abstraction (CoA)** reasoning addresses this by separating the reasoning graph from tool execution.In CoA, the model first produces a complete reasoning trace using abstract placeholders for tool outputs:```Step 1: Compute the compound interest on $1000 at 5% for 3 years. Call this CALC_1.Step 2: The population in 2024 is BASE_POP. Retrieve this from the census API. Call this LOOKUP_1.Step 3: The answer is CALC_1 divided by LOOKUP_1.```Only after the reasoning structure is fixed does the runtime make tool calls to fill in `CALC_1` and `LOOKUP_1`, substituting results and evaluating the final expression. The advantage is that the reasoning graph is produced by the model in a single forward pass, without the sequential dependency on intermediate results that makes direct tool use brittle. Errors in one tool call can be diagnosed at a specific node in the graph rather than propagating through downstream reasoning.Gao et al. report that CoA outperforms direct ReAct-style tool use on multi-step mathematical reasoning and knowledge-intensive QA tasks. The gain is largest on tasks requiring three or more sequential tool calls, consistent with the hypothesis that compounding dependency is the primary source of fragility.## Tool Design PrinciplesEffective tool use depends as much on tool design as on model capability. Four principles govern reliable tool interfaces.**Reliability.** Tools should produce deterministic or near-deterministic outputs for a given input. Stochastic tools (e.g., tools that call another language model internally) introduce variance that the orchestrating model cannot account for. When stochasticity is unavoidable, tools should communicate uncertainty explicitly in their return values.**Documentation.** The JSON Schema description fields are the only signal the model has about what a tool does and when to use it. Descriptions must be precise, unambiguous, and include examples of correct argument values. Vague descriptions ("performs some data operation") lead to misuse; specific descriptions ("returns the closing price of a NASDAQ ticker on a given trading day in YYYY-MM-DD format") enable reliable invocation.**Atomicity.** Tools should have a single, well-defined responsibility. A tool that "searches the web and summarizes results" conflates retrieval with summarization; if either step fails, the failure is opaque. Separate tools for "web search returning raw snippets" and "summarize text" give the model control over each step and produce interpretable error signals.**Parseable outputs.** Tool return values should be structured (JSON, typed dataclasses) rather than free text. Free-text returns require the model to parse natural language before reasoning over results, introducing an additional failure mode. Structured returns enable direct field access in subsequent reasoning.Two of these four principles bind harder under the programmatic convention of @sec-485-programmatic-tool-calling, and for the same underlying reason: the consumer of a tool's output changes from a language model reading a transcript to a program indexing a value. Atomicity stops being merely good hygiene and becomes compositional necessity, because the model is now writing control flow over the tools rather than selecting one per turn, and a tool that fuses retrieval with summarization cannot be placed inside a loop that needs the raw snippets. Parseable outputs stop being a convenience and become a hard requirement, because a free-text return that a model could have read charitably will instead be subscripted by a script and raise an exception. Documentation shifts rather than tightens, moving out of `description` strings and into type annotations, defaults and docstrings, where the same information is expressed in the notation the model has the most training exposure to.### Context Management as a Tool Rather Than a Harness Policy {#sec-485-context-editing-tools}The four principles above concern tools that reach outward, into the world. A less obvious application turns the same machinery inward. The agent's context window is itself a scarce resource, and the operations that manage it can be exposed as callable tools rather than hard-wired into the runtime loop. @li2026acm makes precisely this move with **Agentic Context Management (ACM)**, which equips agents with purpose-built context editing tools and lets the agent decide when to invoke them. Their diagnosis of the status quo is sharp: existing compression schemes lose information and fire on rigid heuristic rules, so the moment of compression is chosen by a policy that knows nothing about what the agent currently cares about.**A budgeted buffer.** Model the context as a buffer of capacity $B$ tokens into which each turn appends roughly $a$ tokens, so the window holds about $k = B/a$ turns of history. Define the **reuse distance** $D_i$ of an appended item as the number of turns between the moment it is written and the next turn at which it is actually needed. Under threshold truncation that drops the oldest entries, item $i$ is destroyed before its next use exactly when $D_i > k$, so the probability that a still-needed token has been evicted is simply the tail of the reuse distance distribution:$$\Pr[\text{loss}] = \Pr[D > k], \qquad k \approx B/a.$$This is a discouraging quantity, because reuse distances in long-horizon trajectories are heavy tailed. A file read in the opening minutes of a coding episode is consulted again after two hundred intervening turns; a search result is revisited after a long detour into an unrelated subgoal. If $D$ has a Pareto tail with index $\alpha$, then $\Pr[D > k] \propto k^{-\alpha}$, and for $\alpha$ near one, doubling the context budget barely halves the loss. Buying more window is a weak lever against a fat tail.**Eviction as a decision rather than a consequence.** Let $E_i \in \{0,1\}$ indicate that item $i$ leaves the working context before its next use, and $Y_i \in \{0,1\}$ indicate that item $i$ is in fact needed again. Expected unrecoverable loss over an episode is$$L = \sum_i \Pr[E_i = 1,\, Y_i = 1]\,(1 - r_i),$$where $r_i$ is the probability the content can be fetched back when the need arises. Threshold truncation is the degenerate corner of this expression: $E_i$ is a deterministic function of position alone, carrying no information about $Y_i$ beyond what recency itself implies, and $r_i = 0$ because destroyed content is destroyed. Each term collapses back to $\Pr[D > k]$.ACM attacks both factors independently. Offloading rather than deleting makes $r_i > 0$: discarded content is written to an external memory system and queried back on demand, so **lossless** here names a move rather than a drop. And letting the agent choose the moment makes $E_i$ a function of the agent's own belief about future use, introducing a correlation$$\rho = \mathrm{corr}(E, Y), \qquad \Pr[E = 1,\, Y = 1] = \mathbb{E}[E]\,\mathbb{E}[Y] + \rho\sqrt{\mathrm{Var}(E)\,\mathrm{Var}(Y)},$$between the eviction decision and the true future-use indicator that no positional rule can have. Read the second identity carefully, because the sign is the whole point. Holding the number of evicted items fixed pins $\mathbb{E}[E]$, so $L$ is *increasing* in $\rho$: a policy that evicts precisely the items it will need again is the worst possible one. The objective is to drive $\rho$ as far negative as the marginals allow, evicting exactly the material the agent does not expect to consult. A policy with $\rho = 0$ is evicting blind, and pays the base rate.This is the classical frame of **cache replacement theory**, and it is worth naming explicitly. Belady's MIN algorithm evicts the line with the largest forward reuse distance and is optimal precisely because it has oracle access to $Y$, which is to say it attains the most negative $\rho$ the marginals permit. FIFO and fixed-threshold truncation sit some way up from that floor, using recency as a weak proxy that buys only a mildly negative $\rho$. Real caches close the gap with profiled or learned hints, and ACM's wager is that a model reasoning over its own trajectory is a better hint generator than a positional rule, because it knows which subgoal is active and therefore which observations are finished being useful. The post-training pipeline in @li2026acm exists to sharpen that signal, constructing demonstrations of good context management so the policy learns *when* to edit, not merely *how*.**Tool interface.** The framework reduces to three atomic operations over a single external store. The schemas below are our own illustrative rendering of those operations in the style of Section 2, written to show what the interface has to express; they are not a transcription of the published ACM API, and field names, defaults, and argument shapes in @li2026acm differ. Note that the three satisfy the atomicity principle: compression, eviction, and retrieval are separable, so a failed query is distinguishable from a bad compression.```json[ {"name":"context_compress","description":"Replace a contiguous span of the working context with a summary, keeping the span retrievable in full from external memory. Use when a subtask is complete and only its conclusion is still relevant.","parameters": {"type":"object","properties": {"span_ids": {"type":"array","items": {"type":"string"},"description":"Message or observation IDs to compress"},"summary": {"type":"string","description":"Replacement text preserving conclusions and open questions"} },"required": ["span_ids","summary"] } }, {"name":"context_offload","description":"Move items out of the working context into external memory without leaving a summary. Use for observations that are finished being useful, e.g. raw tool output already acted upon.","parameters": {"type":"object","properties": {"span_ids": {"type":"array","items": {"type":"string"}},"tags": {"type":"array","items": {"type":"string"},"description":"Retrieval keys, e.g. file paths or entity names"} },"required": ["span_ids"] } }, {"name":"context_query","description":"Retrieve previously compressed or offloaded content back into the working context. Returns matching items with their original IDs.","parameters": {"type":"object","properties": {"query": {"type":"string","description":"Natural-language description of the needed content"},"top_k": {"type":"integer","description":"Maximum items to return (default 3)"} },"required": ["query"] } }]```@li2026acm report three effects from this arrangement, and the third is easy to overlook. Peak token pressure falls, because the agent compresses before the buffer is desperate rather than after. Explorations run longer, because the effective horizon is no longer capped by the raw window. And solutions become **more consistent across independent trials**, which is a statement about variance rather than about the mean: two runs of the same task that discard different material diverge, whereas two runs that retain the same decisive evidence converge. The simulation below speaks to the second and third only. It cannot speak to the first, because a fair comparison has to impose the same hard budget and the same compaction trigger on every policy, which fixes peak occupancy by construction and leaves nothing to measure there. What remains is the pair of quantities the setup genuinely isolates: how much needed material each policy destroys, and how much that quantity varies from run to run.```{python}#| label: fig-485-context-editing-policies#| fig-cap: "Long-horizon buffer simulation over 40 independent trials. All three policies run under an identical hard budget and an identical compaction trigger, so peak occupancy is held fixed by construction and only the eviction rule varies. Agent-invoked offload with retrieval on miss cuts unrecoverable information loss (left) and tightens the run-to-run spread (right) relative to rigid threshold truncation and random eviction."#| code-fold: falseimport numpy as npimport matplotlib.pyplot as pltSEED =20260726B =12_000# context budget in tokensN_TURNS =400# turns in one long horizon episodeALPHA =0.9# Pareto tail index of the reuse distanceP_REUSE =0.45# fraction of items ever needed againSIGNAL_ACC =0.85# agent accuracy at predicting future reuseRECALL =0.90# probability an offloaded item is queried backN_TRIALS =40HI, LO =0.92* B, 0.60* B # identical trigger and target for every policy,# so peak occupancy is held fixed by constructiondef trajectory(rng):"""One episode: token size and next-need turn for each appended item.""" size = rng.lognormal(np.log(400.0), 0.6, N_TURNS) reused = rng.random(N_TURNS) < P_REUSE dist = np.ceil(rng.pareto(ALPHA, N_TURNS) +1.0).astype(int) nxt = np.arange(N_TURNS) + distreturn size, np.where(reused & (nxt < N_TURNS), nxt, -1)def run(policy, rng): size, need_at = trajectory(rng) truth = need_at >=0 flipped = rng.random(N_TURNS) > SIGNAL_ACC pred = np.where(flipped, ~truth, truth) # agent's noisy "still useful" belief schedule = {}for i, t inenumerate(need_at):if t >=0: schedule.setdefault(int(t), []).append(i) offloads = policy =="agent_offload" live, order, memory = {}, [], set() occ = peak =0.0 needed = lost = queries =0def make_room(incoming):nonlocal occif occ + incoming <= HI:returnif offloads: # agent ranks by predicted future use cand = [i for i in order ifnot pred[i]] + [i for i in order if pred[i]]elif policy =="random_evict": cand = [int(i) for i in rng.permutation(order)]else: # rigid threshold truncation: drop oldest cand =list(order)for i in cand:if occ + incoming <= LO:break occ -= live.pop(i) order.remove(i)if offloads: memory.add(i)for t inrange(N_TURNS): make_room(size[t]) live[t] = size[t]; order.append(t); occ += size[t] peak =max(peak, occ)for i in schedule.get(t, []): needed +=1if i in live:continueif i in memory: # recoverable: pay a query, not a loss queries +=1if rng.random() < RECALL: make_room(size[i]) live[i] = size[i]; order.append(i); occ += size[i] peak =max(peak, occ)continue lost +=1return peak / B, lost /max(needed, 1), queriesPOLICIES = ["threshold_truncate", "random_evict", "agent_offload"]res = {p: {"peak": [], "loss": [], "q": []} for p in POLICIES}master = np.random.default_rng(SEED)for _ inrange(N_TRIALS): s =int(master.integers(1<<31)) # same trajectory for all three policiesfor p in POLICIES: pk, ls, q = run(p, np.random.default_rng(s)) res[p]["peak"].append(pk); res[p]["loss"].append(ls); res[p]["q"].append(q)print(f"{'policy':<20}{'loss':>9}{'loss sd':>10}{'spread':>9}{'queries':>9}{'peak/B':>9}")print("="*66)for p in POLICIES: ls = np.array(res[p]["loss"]); pk = np.array(res[p]["peak"])print(f"{p:<20}{ls.mean():>9.4f}{ls.std(ddof=1):>10.4f}{ls.max() - ls.min():>9.4f}"f"{np.mean(res[p]['q']):>9.1f}{pk.mean():>9.3f}")print("\npeak/B is equal by construction (shared cap), not a measured difference")LABELS = ["threshold\ntruncation", "random\neviction", "agent-invoked\noffload"]COLORS = ["#b5651d", "#7a7a7a", "#1f6f8b"]fig, ax = plt.subplots(1, 2, figsize=(8.6, 3.6))x = np.arange(3)mu = np.array([np.mean(res[p]["loss"]) for p in POLICIES])sd = np.array([np.std(res[p]["loss"], ddof=1) for p in POLICIES])ax[0].bar(x, mu, yerr=sd, capsize=5, color=COLORS)ax[0].set_ylabel("needed but unrecoverable")ax[0].set_title("Information loss (mean $\\pm$ sd)")jit = np.random.default_rng(1).normal(0, 0.06, (3, N_TRIALS))for j, p inenumerate(POLICIES): v = np.array(res[p]["loss"]) ax[1].scatter(j + jit[j], v, s=14, alpha=0.65, color=COLORS[j]) ax[1].hlines(v.mean(), j -0.28, j +0.28, color="k", lw=1.6)ax[1].set_ylabel("loss rate per trial")ax[1].set_title(f"Run to run spread ({N_TRIALS} trials)")for a in ax: a.set_xticks(x); a.set_xticklabels(LABELS, fontsize=8.5) a.spines[["top", "right"]].set_visible(False)fig.tight_layout()plt.show()```Random eviction is the control that isolates the mechanism: it runs under the same budget and the same trigger as the other two but with $\rho \approx 0$, and at a loss rate of 0.193 it destroys roughly three times as much needed material as threshold truncation at 0.059. Threshold truncation does better only because recency is a weak proxy for future use, worth a mildly negative $\rho$. Agent-invoked offload combines a more negative $\rho$ with $r_i > 0$ and lands at 0.011, a further factor of about five, at a cost of roughly 14 retrieval queries per episode. The run-to-run standard deviation falls with it, from 0.020 under truncation to 0.007, which is the variance result in miniature: policies that destroy material at rule-chosen moments make outcomes depend on which trajectory happened to cross the threshold when. Peak occupancy is 0.919 for all three, which is the shared cap read back out rather than a finding. Compacting early is a design choice a harness can impose on any eviction rule, and it is imposed identically here precisely so that it cannot flatter the agent policy.The design lesson closes the loop on this section's argument about where behavior belongs. Threshold truncation lives in the harness because the harness is the only component that can see the token counter, but it is the wrong place to make the decision, since the harness cannot see intent. The correct division is that the harness owns the *mechanism* and the *invariant* (it enforces the hard budget and will truncate if the agent does not act) while the agent owns the *policy*, invoking `context_compress` and `context_offload` at moments it selects. Exposing context management as a tool is not a convenience wrapper; it moves a decision to the only component holding the information needed to make it well, which is the same argument that justifies giving the model a calculator instead of a rounding rule.## Implementing a Tool-Use LoopThe following implementation demonstrates the complete tool-use loop without external API dependencies, using a rule-based dispatcher in place of a language model to make the mechanics visible.```pythonimport jsonimport mathfrom typing import Any, CallableTOOL_REGISTRY: dict[str, dict[str, Any]] = {}def tool(schema: dict[str, Any]) -> Callable:def decorator(fn: Callable) -> Callable: TOOL_REGISTRY[schema["name"]] = {"fn": fn, "schema": schema}return fnreturn decorator@tool({"name": "calculator","description": "Evaluate a mathematical expression. Returns a float.","parameters": {"type": "object","properties": {"expression": {"type": "string","description": "A Python arithmetic expression, e.g. '2 ** 10 + sqrt(144)'" } },"required": ["expression"] }})def calculator(expression: str) ->float: allowed = {k: getattr(math, k) for k indir(math) ifnot k.startswith("_")} allowed["abs"] =absreturnfloat(eval(expression, {"__builtins__": {}}, allowed))@tool({"name": "lookup_capital","description": "Return the capital city of a country.","parameters": {"type": "object","properties": {"country": {"type": "string", "description": "Country name in English"} },"required": ["country"] }})def lookup_capital(country: str) ->str: db = {"France": "Paris", "Germany": "Berlin", "Japan": "Tokyo","Brazil": "Brasilia", "Australia": "Canberra", "Canada": "Ottawa", }return db.get(country, f"Unknown capital for '{country}'")def dispatch(call: dict[str, Any]) -> Any: name = call["name"] args = call.get("arguments", {})ifisinstance(args, str): args = json.loads(args)if name notin TOOL_REGISTRY:return {"error": f"No tool named '{name}'"}return TOOL_REGISTRY[name]["fn"](**args)def mock_llm(messages: list[dict]) ->dict:""" Minimal rule-based mock that mimics the assistant turn of an LLM with function-calling capability. Parses the last user message to decide whether to call a tool or produce a final answer. """ last_user =next( (m["content"] for m inreversed(messages) if m["role"] =="user"), "" ) tool_results = [m for m in messages if m["role"] =="tool"]if"capital"in last_user.lower() andnot tool_results: country ="France"if"france"in last_user.lower() else"Japan"return {"role": "assistant","tool_calls": [ {"id": "tc_1", "type": "function","function": {"name": "lookup_capital","arguments": json.dumps({"country": country})}} ] }if"sqrt"in last_user.lower() or"**"in last_user.lower():ifnot tool_results: expr ="sqrt(144) + 2 ** 10"return {"role": "assistant","tool_calls": [ {"id": "tc_2", "type": "function","function": {"name": "calculator","arguments": json.dumps({"expression": expr})}} ] }if tool_results: last_result = tool_results[-1]["content"]return {"role": "assistant","content": f"Based on the tool result: {last_result}" }return {"role": "assistant", "content": "I can answer that directly: 42."}def run_agent(user_query: str, max_steps: int=6) ->str: messages = [{"role": "user", "content": user_query}]for step inrange(max_steps): response = mock_llm(messages) messages.append(response)if"tool_calls"notin response:return response.get("content", "")for call in response["tool_calls"]: fn_info = call["function"] result = dispatch(fn_info) messages.append({"role": "tool","tool_call_id": call["id"],"content": str(result) })return"Max steps reached without final answer."queries = ["What is the capital of France?","Compute sqrt(144) + 2**10","What is the meaning of life?",]for q in queries: answer = run_agent(q)print(f"Q: {q}\nA: {answer}\n")```Running this loop produces:```Q: What is the capital of France?A: Based on the tool result: ParisQ: Compute sqrt(144) + 2**10A: Based on the tool result: 1036.0Q: What is the meaning of life?A: I can answer that directly: 42.```The mock demonstrates the structural mechanics: messages accumulate, tool calls are dispatched to a typed registry, results are injected as tool-role messages, and the agent terminates on a content-only assistant turn. Replacing `mock_llm` with an actual API call (passing `tools=[s["schema"] for s in TOOL_REGISTRY.values()]` and parsing the response's `tool_calls` field) converts this skeleton into a production loop.The second demonstration shows tool composition: computing a result that requires the `calculator` tool while leaving the reasoning structure in the calling code.```pythondef multi_step_demo() ->dict[str, float]: results = {} interest_rate =0.05 principal =1000.0 years =3 expr =f"{principal} * (1 + {interest_rate}) ** {years}" results["compound_interest"] = calculator(expression=expr) results["log2_population_estimate"] = calculator(expression="log2(8e9)") results["answer"] = results["compound_interest"] / results["log2_population_estimate"]return resultsprint(multi_step_demo())```This pattern implements Chain-of-Abstraction manually: the computation graph is expressed in Python (the "abstract" layer), and tool calls fill in the numerical leaves.## Evaluation and Failure ModesBenchmarking tool-use agents requires metrics beyond end-task accuracy. Four dimensions matter.**Call precision and recall.** Does the model invoke the correct tool for each sub-problem (precision), and does it invoke all necessary tools (recall)? APIBench measures this via Abstract Syntax Tree (AST) comparison of generated versus reference API calls.**Argument correctness.** Even when the correct tool is named, arguments may be wrong. Argument correctness requires field-level comparison and is more sensitive than name-level matching. Gorilla's primary gain over baseline models is in argument correctness rather than tool selection.**Efficiency.** An agent that correctly answers a question in 10 tool calls when 2 suffice is expensive to run. Trajectory length is a proxy for reasoning efficiency and correlates with latency and API cost.**Saturation under fan-out.** All three dimensions above are measured at a task's natural width, which on most benchmark entries is a handful of calls. Holding the task fixed and sweeping the number of required invocations exposes a fourth dimension that fixed-width evaluation cannot see. @patel2026bitter run this sweep and find a cliff rather than a slope: on an enumeration task, JSON calling on Claude Sonnet 5 returns every item correctly out to a fan-out of 70, drops to 75% at 72, and reaches zero by 100, while the programmatic form of @sec-485-programmatic-tool-calling is still complete at 100. Any evaluation whose entries all sit to the left of an agent's cliff will report perfect enumeration and predict nothing about production traffic that occasionally sits to the right of it.The cliff also matters because of *how* it fails. A model that drops thirty of a hundred required calls does not announce an error; it emits a well-formed, plausible answer over the subset it did enumerate, and every downstream check on syntactic validity and argument correctness passes. Detecting silent truncation requires an expected-cardinality check against the request, which is a property of the harness rather than of the model, and which most tool-use scaffolds do not implement.Common failure modes include: invoking a tool when parametric knowledge suffices (over-reliance), failing to invoke a tool when external grounding is needed (under-reliance), generating syntactically valid but semantically incorrect argument bundles (hallucination), and entering cycles where repeated tool calls return the same observation without triggering a state change in reasoning.Mitigations include explicit stopping criteria in system prompts, tool-use training data that includes negative examples (situations where no tool call is correct), and structured output validation before dispatch.## References1. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. *International Conference on Learning Representations (ICLR)*. https://openreview.net/forum?id=WE_vluYUL-X2. Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023). Gorilla: Large Language Model Connected with Massive APIs. *arXiv preprint*. https://arxiv.org/abs/2305.153343. Yang, Z., Li, L., Wang, J., Lin, K., Azarnasab, E., Ahmed, F., Liu, Z., Liu, C., Zeng, M., & Wang, L. (2023). MM-ReAct: Prompting ChatGPT for Multimodal Reasoning and Action. *arXiv preprint*. https://arxiv.org/abs/2303.113814. Gao, J., Gu, L., Su, J., Guo, Y., Li, M., Li, J., Chen, W., & Bi, W. (2024). Efficient Tool Use with Chain-of-Abstraction Reasoning. *arXiv preprint*. https://arxiv.org/abs/2401.174645. Li, X., Ming, R., Chu, M., Shao, S., Jin, R., & Xiong, C. (2026). ACM: Agentic Context Management for Long Horizon Tasks. *arXiv preprint*. https://arxiv.org/abs/2607.238096. Patel, I., Sen, S., Lumer, E., & Subbiah, V. K. (2026). The Bitter Lesson of Tool Calling. *arXiv preprint*. https://arxiv.org/abs/2608.06370
