226  Tool Use in Language Model Agents

Language models trained on static corpora are frozen at their training cutoff: they cannot fetch tomorrow’s weather, query a live database, execute arithmetic with guaranteed precision, or invoke an external service. These limitations are not incidental deficiencies but structural properties of the autoregressive generation paradigm. Tool use is the mechanism by which agents bridge the gap between language model reasoning and world action, converting natural-language intention into structured external calls whose results ground subsequent reasoning.

226.1 1. Why Tools Are Necessary

A language model generates tokens by approximating the conditional distribution \(P(x_t \mid x_1, \ldots, x_{t-1})\) learned from training data. This distribution encodes knowledge about the world only up to the training cutoff and only insofar as that knowledge appeared in the training corpus. Three categories of tasks are therefore structurally outside the model’s native capability.

Real-time knowledge. Events after the training cutoff are unknown to the model. A user asking “what is the current Euro/USD exchange rate?” receives either an admission of ignorance or a confidently hallucinated stale figure. A tool call to a currency API grounds the response in live data.

Precise computation. Transformer models are poor calculators. While they can approximate arithmetic for small numbers seen frequently in training, they have no reliable mechanism for multi-digit multiplication or floating-point arithmetic. A Python interpreter tool executes such computations exactly.

External state manipulation. Creating a calendar event, sending an email, writing to a database, or triggering a deployment pipeline requires side effects in external systems. Language generation alone cannot produce these effects; only structured API calls can.

Tool use does not require fine-tuning. A sufficiently capable model can learn to produce structured outputs from in-context demonstrations or system prompts. Fine-tuning improves reliability and reduces hallucination, as the Gorilla work demonstrates (Section 4), but the architectural pattern is prompt-level.

226.2 2. Function Calling Mechanics

The canonical mechanism for tool use is function calling, introduced in the OpenAI Chat Completions API and now a near-universal interface. The protocol has four components.

Schema declaration. At inference time, the model receives a list of JSON Schema objects describing available tools. Each schema specifies the function name, a natural-language description, and a parameters object describing argument names, types, and descriptions. The model uses these schemas to determine when to call a tool and how to construct a valid argument bundle.

{
  "name": "get_current_weather",
  "description": "Retrieve current weather conditions for a location.",
  "parameters": {
    "type": "object",
    "properties": {
      "location": {
        "type": "string",
        "description": "City and country, e.g. 'Paris, France'"
      },
      "unit": {
        "type": "string",
        "enum": ["celsius", "fahrenheit"],
        "description": "Temperature unit to return"
      }
    },
    "required": ["location"]
  }
}

Structured call generation. When the model determines that a tool call is appropriate, it generates a structured object rather than free text. In the OpenAI format this appears in a dedicated tool_calls field of the assistant message, containing id, type: "function", and a function object with name and arguments (a JSON string). The model does not continue generating conversational tokens until the tool result is returned.

Execution and result injection. The runtime deserializes the argument bundle, dispatches to the appropriate function, and captures the return value. The result is injected into the conversation as a tool message keyed to the original call ID. In multi-tool scenarios, multiple calls may be batched in a single assistant turn and results returned in parallel.

Continued reasoning. After receiving tool results the model resumes generation with a richer context: its original reasoning, the structured call it made, and the observed output. This grounds subsequent generation in verified external data.

The conversation structure is therefore a sequence of alternating roles:

\[\text{system} \to \text{user} \to \text{assistant (tool\_call)} \to \text{tool (result)} \to \text{assistant (final)}\]

For multi-step tasks this sequence may cycle through the tool-call/tool-result pair multiple times before a final answer is produced.

226.3 3. ReAct: Interleaved Reasoning and Acting

Yao et al. (2023) formalized the observation that language models produce better tool-augmented outputs when reasoning and action are explicitly interleaved rather than either separated or implicit. The ReAct (Reason + Act) framework structures agent traces as alternating thought and action steps.

A ReAct trace for a question-answering task might read:

Thought: I need to find the year Beyonce's sixth album was released.
Action: Search("Beyonce sixth studio album")
Observation: Beyonce's sixth studio album "Lemonade" was released in April 2016.
Thought: I now know the release year is 2016. I should verify the director of the title track.
Action: Search("Lemonade Beyonce title track director")
Observation: The title track was directed by Kahlil Joseph and Beyonce herself.
Thought: I have enough information to answer.
Action: Finish("Lemonade was released in 2016; the title track was co-directed by Kahlil Joseph.")

This structure differs from chain-of-thought (CoT) prompting in a critical way: CoT generates reasoning entirely within the model’s parametric knowledge, while ReAct grounds each reasoning step in observed external evidence. The observation at each step constrains subsequent thoughts, preventing the compounding hallucinations that afflict multi-step CoT.

Yao et al. evaluated ReAct on HotpotQA (multi-hop question answering), Fever (fact verification), ALFWorld (text-based household tasks), and WebShop (product-search navigation). ReAct outperformed CoT-only baselines on all four tasks and produced traces that human raters found more interpretable and easier to correct. On HotpotQA, ReAct achieved 35.1% success versus 29.4% for CoT with the same model, a substantial gain attributable entirely to grounding.

The information-theoretic motivation is straightforward. Let \(K_\theta\) be the model’s parametric knowledge and \(K_e\) be external evidence retrieved at step \(t\). CoT conditions only on \(K_\theta\); ReAct conditions on \(K_\theta \cup \bigcup_t K_e^{(t)}\). Because \(K_e^{(t)}\) is retrieved in response to intermediate reasoning, it is specifically relevant to the current subproblem. The effective knowledge available to ReAct is therefore a superset of CoT’s knowledge, targeted rather than broad.

226.4 4. Gorilla: Reducing API Hallucination via Retrieval-Aware Training

A persistent failure mode in tool use is parameter hallucination: the model invokes a real function name but fabricates argument names, types, or values that do not conform to the actual API. This is particularly dangerous for SDK calls where incorrect arguments cause silent wrong behavior rather than runtime errors.

Patil et al. (2023) addressed this with Gorilla, a LLaMA-based model fine-tuned on over 1,600 API documentation pages spanning TorchHub, TensorFlow Hub, and HuggingFace model APIs. The contribution has two parts: a dataset and a training strategy.

APIBench. The authors constructed APIBench by scraping API documentation, writing questions that require API calls to answer, and recording the correct API invocation as the label. The benchmark covers three domains with distinct calling conventions: TorchHub uses Python torch.hub.load, TensorFlow Hub uses tensorflow_hub.load, and HuggingFace uses pipeline with model identifiers. Hallucination is measured as the fraction of generated calls that either name a nonexistent API or supply incorrect arguments.

Retrieval-aware training. The key insight is that API documentation is too voluminous to fit in context but too specific to memorize reliably. Gorilla’s training procedure pairs each training example with retrieved documentation: a retriever fetches the most relevant API documentation for the question, that documentation is prepended to the context, and the model is trained to generate the correct API call given this augmented context. At inference time the same retriever runs before generation, simulating the training distribution.

This contrasts with naive RAG, where retrieval is added at inference time without training-time conditioning. By training with retrieved context, the model learns to use documentation rather than to override it with parametric memorization. The result is a substantial reduction in hallucination: Gorilla achieves 20.43% AST accuracy on HuggingFace versus 8.22% for GPT-3.5-Turbo in the zero-shot setting, nearly a 2.5x improvement.

The architectural takeaway is general: when tools have complex or frequently changing interfaces, fine-tuning with retrieval-in-context produces more reliable callers than prompting alone. This principle extends beyond API calling to any domain where the “schema” of valid actions evolves faster than model retraining cycles.

226.5 5. MM-ReAct: Multimodal Tool Composition

Yang et al. (2023) extended the ReAct framework to multimodal inputs by treating specialized vision models as tools. MM-ReAct allows a language model (GPT-4 in the original paper) to reason about images through a composition of external vision APIs rather than through its own vision encoder.

The available tools include an image captioner, an OCR engine, a celebrity/landmark recognizer, a Bing image search, an image editing tool, and a math solver for image-embedded equations. When presented with an image, the language model does not process pixel values directly. Instead, it decomposes the visual question into a sequence of tool calls:

Thought: The image may contain text I need to read.
Action: OCR(image)
Observation: "Net Income: $4.2B (up 12% YoY)"
Thought: I need context about this company to interpret the number.
Action: BingSearch("company name from financial report header")
Observation: The report header shows "Acme Corp Q3 2024"
Thought: I can now answer the question about profitability trends.

This decomposition has an important epistemic property: each tool call produces a verifiable, human-interpretable intermediate result. The trace is auditable in a way that end-to-end vision-language generation is not. A reviewer can inspect whether the OCR output was correct, whether the search query was appropriate, and whether the final reasoning step is valid given the observations.

MM-ReAct demonstrates that tool use is not merely a patch for knowledge gaps but a compositional architecture: a language model can serve as an orchestrator that coordinates specialist models, with each specialist optimized for a narrow perceptual or computational task. The language model contributes planning, disambiguation, and synthesis; the tools contribute precision.

226.6 6. Chain-of-Abstraction Reasoning

Gao et al. (2024) identified a subtle failure mode in direct tool use: when a model must invoke multiple tools in a chain, it tends to fix its reasoning structure around the first tool call’s result, making it brittle to errors or unexpected outputs in that call. Chain-of-Abstraction (CoA) reasoning addresses this by separating the reasoning graph from tool execution.

In CoA, the model first produces a complete reasoning trace using abstract placeholders for tool outputs:

Step 1: Compute the compound interest on $1000 at 5% for 3 years. Call this CALC_1.
Step 2: The population in 2024 is BASE_POP. Retrieve this from the census API. Call this LOOKUP_1.
Step 3: The answer is CALC_1 divided by LOOKUP_1.

Only after the reasoning structure is fixed does the runtime make tool calls to fill in CALC_1 and LOOKUP_1, substituting results and evaluating the final expression. The advantage is that the reasoning graph is produced by the model in a single forward pass, without the sequential dependency on intermediate results that makes direct tool use brittle. Errors in one tool call can be diagnosed at a specific node in the graph rather than propagating through downstream reasoning.

Gao et al. report that CoA outperforms direct ReAct-style tool use on multi-step mathematical reasoning and knowledge-intensive QA tasks. The gain is largest on tasks requiring three or more sequential tool calls, consistent with the hypothesis that compounding dependency is the primary source of fragility.

226.7 7. Tool Design Principles

Effective tool use depends as much on tool design as on model capability. Four principles govern reliable tool interfaces.

Reliability. Tools should produce deterministic or near-deterministic outputs for a given input. Stochastic tools (e.g., tools that call another language model internally) introduce variance that the orchestrating model cannot account for. When stochasticity is unavoidable, tools should communicate uncertainty explicitly in their return values.

Documentation. The JSON Schema description fields are the only signal the model has about what a tool does and when to use it. Descriptions must be precise, unambiguous, and include examples of correct argument values. Vague descriptions (“performs some data operation”) lead to misuse; specific descriptions (“returns the closing price of a NASDAQ ticker on a given trading day in YYYY-MM-DD format”) enable reliable invocation.

Atomicity. Tools should have a single, well-defined responsibility. A tool that “searches the web and summarizes results” conflates retrieval with summarization; if either step fails, the failure is opaque. Separate tools for “web search returning raw snippets” and “summarize text” give the model control over each step and produce interpretable error signals.

Parseable outputs. Tool return values should be structured (JSON, typed dataclasses) rather than free text. Free-text returns require the model to parse natural language before reasoning over results, introducing an additional failure mode. Structured returns enable direct field access in subsequent reasoning.

226.7.1 Context Management as a Tool Rather Than a Harness Policy

The four principles above concern tools that reach outward, into the world. A less obvious application turns the same machinery inward. The agent’s context window is itself a scarce resource, and the operations that manage it can be exposed as callable tools rather than hard-wired into the runtime loop. Li et al. (2026) makes precisely this move with Agentic Context Management (ACM), which equips agents with purpose-built context editing tools and lets the agent decide when to invoke them. Their diagnosis of the status quo is sharp: existing compression schemes lose information and fire on rigid heuristic rules, so the moment of compression is chosen by a policy that knows nothing about what the agent currently cares about.

A budgeted buffer. Model the context as a buffer of capacity \(B\) tokens into which each turn appends roughly \(a\) tokens, so the window holds about \(k = B/a\) turns of history. Define the reuse distance \(D_i\) of an appended item as the number of turns between the moment it is written and the next turn at which it is actually needed. Under threshold truncation that drops the oldest entries, item \(i\) is destroyed before its next use exactly when \(D_i > k\), so the probability that a still-needed token has been evicted is simply the tail of the reuse distance distribution:

\[\Pr[\text{loss}] = \Pr[D > k], \qquad k \approx B/a.\]

This is a discouraging quantity, because reuse distances in long-horizon trajectories are heavy tailed. A file read in the opening minutes of a coding episode is consulted again after two hundred intervening turns; a search result is revisited after a long detour into an unrelated subgoal. If \(D\) has a Pareto tail with index \(\alpha\), then \(\Pr[D > k] \propto k^{-\alpha}\), and for \(\alpha\) near one, doubling the context budget barely halves the loss. Buying more window is a weak lever against a fat tail.

Eviction as a decision rather than a consequence. Let \(E_i \in \{0,1\}\) indicate that item \(i\) leaves the working context before its next use, and \(Y_i \in \{0,1\}\) indicate that item \(i\) is in fact needed again. Expected unrecoverable loss over an episode is

\[L = \sum_i \Pr[E_i = 1,\, Y_i = 1]\,(1 - r_i),\]

where \(r_i\) is the probability the content can be fetched back when the need arises. Threshold truncation is the degenerate corner of this expression: \(E_i\) is a deterministic function of position alone, carrying no information about \(Y_i\) beyond what recency itself implies, and \(r_i = 0\) because destroyed content is destroyed. Each term collapses back to \(\Pr[D > k]\).

ACM attacks both factors independently. Offloading rather than deleting makes \(r_i > 0\): discarded content is written to an external memory system and queried back on demand, so lossless here names a move rather than a drop. And letting the agent choose the moment makes \(E_i\) a function of the agent’s own belief about future use, introducing a correlation

\[\rho = \mathrm{corr}(E, Y), \qquad \Pr[E = 1,\, Y = 1] = \mathbb{E}[E]\,\mathbb{E}[Y] + \rho\sqrt{\mathrm{Var}(E)\,\mathrm{Var}(Y)},\]

between the eviction decision and the true future-use indicator that no positional rule can have. Read the second identity carefully, because the sign is the whole point. Holding the number of evicted items fixed pins \(\mathbb{E}[E]\), so \(L\) is increasing in \(\rho\): a policy that evicts precisely the items it will need again is the worst possible one. The objective is to drive \(\rho\) as far negative as the marginals allow, evicting exactly the material the agent does not expect to consult. A policy with \(\rho = 0\) is evicting blind, and pays the base rate.

This is the classical frame of cache replacement theory, and it is worth naming explicitly. Belady’s MIN algorithm evicts the line with the largest forward reuse distance and is optimal precisely because it has oracle access to \(Y\), which is to say it attains the most negative \(\rho\) the marginals permit. FIFO and fixed-threshold truncation sit some way up from that floor, using recency as a weak proxy that buys only a mildly negative \(\rho\). Real caches close the gap with profiled or learned hints, and ACM’s wager is that a model reasoning over its own trajectory is a better hint generator than a positional rule, because it knows which subgoal is active and therefore which observations are finished being useful. The post-training pipeline in Li et al. (2026) exists to sharpen that signal, constructing demonstrations of good context management so the policy learns when to edit, not merely how.

Tool interface. The framework reduces to three atomic operations over a single external store. The schemas below are our own illustrative rendering of those operations in the style of Section 2, written to show what the interface has to express; they are not a transcription of the published ACM API, and field names, defaults, and argument shapes in Li et al. (2026) differ. Note that the three satisfy the atomicity principle: compression, eviction, and retrieval are separable, so a failed query is distinguishable from a bad compression.

[
  {
    "name": "context_compress",
    "description": "Replace a contiguous span of the working context with a summary, keeping the span retrievable in full from external memory. Use when a subtask is complete and only its conclusion is still relevant.",
    "parameters": {
      "type": "object",
      "properties": {
        "span_ids": {"type": "array", "items": {"type": "string"},
                     "description": "Message or observation IDs to compress"},
        "summary": {"type": "string",
                    "description": "Replacement text preserving conclusions and open questions"}
      },
      "required": ["span_ids", "summary"]
    }
  },
  {
    "name": "context_offload",
    "description": "Move items out of the working context into external memory without leaving a summary. Use for observations that are finished being useful, e.g. raw tool output already acted upon.",
    "parameters": {
      "type": "object",
      "properties": {
        "span_ids": {"type": "array", "items": {"type": "string"}},
        "tags": {"type": "array", "items": {"type": "string"},
                 "description": "Retrieval keys, e.g. file paths or entity names"}
      },
      "required": ["span_ids"]
    }
  },
  {
    "name": "context_query",
    "description": "Retrieve previously compressed or offloaded content back into the working context. Returns matching items with their original IDs.",
    "parameters": {
      "type": "object",
      "properties": {
        "query": {"type": "string", "description": "Natural-language description of the needed content"},
        "top_k": {"type": "integer", "description": "Maximum items to return (default 3)"}
      },
      "required": ["query"]
    }
  }
]

Li et al. (2026) report three effects from this arrangement, and the third is easy to overlook. Peak token pressure falls, because the agent compresses before the buffer is desperate rather than after. Explorations run longer, because the effective horizon is no longer capped by the raw window. And solutions become more consistent across independent trials, which is a statement about variance rather than about the mean: two runs of the same task that discard different material diverge, whereas two runs that retain the same decisive evidence converge. The simulation below speaks to the second and third only. It cannot speak to the first, because a fair comparison has to impose the same hard budget and the same compaction trigger on every policy, which fixes peak occupancy by construction and leaves nothing to measure there. What remains is the pair of quantities the setup genuinely isolates: how much needed material each policy destroys, and how much that quantity varies from run to run.

import numpy as np
import matplotlib.pyplot as plt

SEED = 20260726
B = 12_000                 # context budget in tokens
N_TURNS = 400              # turns in one long horizon episode
ALPHA = 0.9                # Pareto tail index of the reuse distance
P_REUSE = 0.45             # fraction of items ever needed again
SIGNAL_ACC = 0.85          # agent accuracy at predicting future reuse
RECALL = 0.90              # probability an offloaded item is queried back
N_TRIALS = 40
HI, LO = 0.92 * B, 0.60 * B   # identical trigger and target for every policy,
                              # so peak occupancy is held fixed by construction

def trajectory(rng):
    """One episode: token size and next-need turn for each appended item."""
    size = rng.lognormal(np.log(400.0), 0.6, N_TURNS)
    reused = rng.random(N_TURNS) < P_REUSE
    dist = np.ceil(rng.pareto(ALPHA, N_TURNS) + 1.0).astype(int)
    nxt = np.arange(N_TURNS) + dist
    return size, np.where(reused & (nxt < N_TURNS), nxt, -1)

def run(policy, rng):
    size, need_at = trajectory(rng)
    truth = need_at >= 0
    flipped = rng.random(N_TURNS) > SIGNAL_ACC
    pred = np.where(flipped, ~truth, truth)   # agent's noisy "still useful" belief

    schedule = {}
    for i, t in enumerate(need_at):
        if t >= 0:
            schedule.setdefault(int(t), []).append(i)

    offloads = policy == "agent_offload"

    live, order, memory = {}, [], set()
    occ = peak = 0.0
    needed = lost = queries = 0

    def make_room(incoming):
        nonlocal occ
        if occ + incoming <= HI:
            return
        if offloads:                       # agent ranks by predicted future use
            cand = [i for i in order if not pred[i]] + [i for i in order if pred[i]]
        elif policy == "random_evict":
            cand = [int(i) for i in rng.permutation(order)]
        else:                              # rigid threshold truncation: drop oldest
            cand = list(order)
        for i in cand:
            if occ + incoming <= LO:
                break
            occ -= live.pop(i)
            order.remove(i)
            if offloads:
                memory.add(i)

    for t in range(N_TURNS):
        make_room(size[t])
        live[t] = size[t]; order.append(t); occ += size[t]
        peak = max(peak, occ)

        for i in schedule.get(t, []):
            needed += 1
            if i in live:
                continue
            if i in memory:                # recoverable: pay a query, not a loss
                queries += 1
                if rng.random() < RECALL:
                    make_room(size[i])
                    live[i] = size[i]; order.append(i); occ += size[i]
                    peak = max(peak, occ)
                    continue
            lost += 1

    return peak / B, lost / max(needed, 1), queries

POLICIES = ["threshold_truncate", "random_evict", "agent_offload"]
res = {p: {"peak": [], "loss": [], "q": []} for p in POLICIES}
master = np.random.default_rng(SEED)
for _ in range(N_TRIALS):
    s = int(master.integers(1 << 31))      # same trajectory for all three policies
    for p in POLICIES:
        pk, ls, q = run(p, np.random.default_rng(s))
        res[p]["peak"].append(pk); res[p]["loss"].append(ls); res[p]["q"].append(q)

print(f"{'policy':<20}{'loss':>9}{'loss sd':>10}{'spread':>9}{'queries':>9}{'peak/B':>9}")
print("=" * 66)
for p in POLICIES:
    ls = np.array(res[p]["loss"]); pk = np.array(res[p]["peak"])
    print(f"{p:<20}{ls.mean():>9.4f}{ls.std(ddof=1):>10.4f}{ls.max() - ls.min():>9.4f}"
          f"{np.mean(res[p]['q']):>9.1f}{pk.mean():>9.3f}")
print("\npeak/B is equal by construction (shared cap), not a measured difference")

LABELS = ["threshold\ntruncation", "random\neviction", "agent-invoked\noffload"]
COLORS = ["#b5651d", "#7a7a7a", "#1f6f8b"]
fig, ax = plt.subplots(1, 2, figsize=(8.6, 3.6))
x = np.arange(3)

mu = np.array([np.mean(res[p]["loss"]) for p in POLICIES])
sd = np.array([np.std(res[p]["loss"], ddof=1) for p in POLICIES])
ax[0].bar(x, mu, yerr=sd, capsize=5, color=COLORS)
ax[0].set_ylabel("needed but unrecoverable")
ax[0].set_title("Information loss (mean $\\pm$ sd)")

jit = np.random.default_rng(1).normal(0, 0.06, (3, N_TRIALS))
for j, p in enumerate(POLICIES):
    v = np.array(res[p]["loss"])
    ax[1].scatter(j + jit[j], v, s=14, alpha=0.65, color=COLORS[j])
    ax[1].hlines(v.mean(), j - 0.28, j + 0.28, color="k", lw=1.6)
ax[1].set_ylabel("loss rate per trial")
ax[1].set_title(f"Run to run spread ({N_TRIALS} trials)")

for a in ax:
    a.set_xticks(x); a.set_xticklabels(LABELS, fontsize=8.5)
    a.spines[["top", "right"]].set_visible(False)
fig.tight_layout()
plt.show()

policy                   loss   loss sd   spread  queries   peak/B
==================================================================
threshold_truncate     0.0589    0.0204   0.0754      0.0    0.919
random_evict           0.1925    0.0297   0.1387      0.0    0.919
agent_offload          0.0106    0.0071   0.0279     14.1    0.919

peak/B is equal by construction (shared cap), not a measured difference

Figure 226.1: Long-horizon buffer simulation over 40 independent trials. All three policies run under an identical hard budget and an identical compaction trigger, so peak occupancy is held fixed by construction and only the eviction rule varies. Agent-invoked offload with retrieval on miss cuts unrecoverable information loss (left) and tightens the run-to-run spread (right) relative to rigid threshold truncation and random eviction.

Random eviction is the control that isolates the mechanism: it runs under the same budget and the same trigger as the other two but with \(\rho \approx 0\), and at a loss rate of 0.193 it destroys roughly three times as much needed material as threshold truncation at 0.059. Threshold truncation does better only because recency is a weak proxy for future use, worth a mildly negative \(\rho\). Agent-invoked offload combines a more negative \(\rho\) with \(r_i > 0\) and lands at 0.011, a further factor of about five, at a cost of roughly 14 retrieval queries per episode. The run-to-run standard deviation falls with it, from 0.020 under truncation to 0.007, which is the variance result in miniature: policies that destroy material at rule-chosen moments make outcomes depend on which trajectory happened to cross the threshold when. Peak occupancy is 0.919 for all three, which is the shared cap read back out rather than a finding. Compacting early is a design choice a harness can impose on any eviction rule, and it is imposed identically here precisely so that it cannot flatter the agent policy.

The design lesson closes the loop on this section’s argument about where behavior belongs. Threshold truncation lives in the harness because the harness is the only component that can see the token counter, but it is the wrong place to make the decision, since the harness cannot see intent. The correct division is that the harness owns the mechanism and the invariant (it enforces the hard budget and will truncate if the agent does not act) while the agent owns the policy, invoking context_compress and context_offload at moments it selects. Exposing context management as a tool is not a convenience wrapper; it moves a decision to the only component holding the information needed to make it well, which is the same argument that justifies giving the model a calculator instead of a rounding rule.

226.8 8. Implementing a Tool-Use Loop

The following implementation demonstrates the complete tool-use loop without external API dependencies, using a rule-based dispatcher in place of a language model to make the mechanics visible.

import json
import math
from typing import Any, Callable

TOOL_REGISTRY: dict[str, dict[str, Any]] = {}

def tool(schema: dict[str, Any]) -> Callable:
    def decorator(fn: Callable) -> Callable:
        TOOL_REGISTRY[schema["name"]] = {"fn": fn, "schema": schema}
        return fn
    return decorator

@tool({
    "name": "calculator",
    "description": "Evaluate a mathematical expression. Returns a float.",
    "parameters": {
        "type": "object",
        "properties": {
            "expression": {
                "type": "string",
                "description": "A Python arithmetic expression, e.g. '2 ** 10 + sqrt(144)'"
            }
        },
        "required": ["expression"]
    }
})
def calculator(expression: str) -> float:
    allowed = {k: getattr(math, k) for k in dir(math) if not k.startswith("_")}
    allowed["abs"] = abs
    return float(eval(expression, {"__builtins__": {}}, allowed))

@tool({
    "name": "lookup_capital",
    "description": "Return the capital city of a country.",
    "parameters": {
        "type": "object",
        "properties": {
            "country": {"type": "string", "description": "Country name in English"}
        },
        "required": ["country"]
    }
})
def lookup_capital(country: str) -> str:
    db = {
        "France": "Paris", "Germany": "Berlin", "Japan": "Tokyo",
        "Brazil": "Brasilia", "Australia": "Canberra", "Canada": "Ottawa",
    }
    return db.get(country, f"Unknown capital for '{country}'")

def dispatch(call: dict[str, Any]) -> Any:
    name = call["name"]
    args = call.get("arguments", {})
    if isinstance(args, str):
        args = json.loads(args)
    if name not in TOOL_REGISTRY:
        return {"error": f"No tool named '{name}'"}
    return TOOL_REGISTRY[name]["fn"](**args)

def mock_llm(messages: list[dict]) -> dict:
    """
    Minimal rule-based mock that mimics the assistant turn of an LLM
    with function-calling capability. Parses the last user message to
    decide whether to call a tool or produce a final answer.
    """
    last_user = next(
        (m["content"] for m in reversed(messages) if m["role"] == "user"), ""
    )
    tool_results = [m for m in messages if m["role"] == "tool"]

    if "capital" in last_user.lower() and not tool_results:
        country = "France" if "france" in last_user.lower() else "Japan"
        return {
            "role": "assistant",
            "tool_calls": [
                {"id": "tc_1", "type": "function",
                 "function": {"name": "lookup_capital",
                              "arguments": json.dumps({"country": country})}}
            ]
        }
    if "sqrt" in last_user.lower() or "**" in last_user.lower():
        if not tool_results:
            expr = "sqrt(144) + 2 ** 10"
            return {
                "role": "assistant",
                "tool_calls": [
                    {"id": "tc_2", "type": "function",
                     "function": {"name": "calculator",
                                  "arguments": json.dumps({"expression": expr})}}
                ]
            }
    if tool_results:
        last_result = tool_results[-1]["content"]
        return {
            "role": "assistant",
            "content": f"Based on the tool result: {last_result}"
        }
    return {"role": "assistant", "content": "I can answer that directly: 42."}

def run_agent(user_query: str, max_steps: int = 6) -> str:
    messages = [{"role": "user", "content": user_query}]
    for step in range(max_steps):
        response = mock_llm(messages)
        messages.append(response)
        if "tool_calls" not in response:
            return response.get("content", "")
        for call in response["tool_calls"]:
            fn_info = call["function"]
            result = dispatch(fn_info)
            messages.append({
                "role": "tool",
                "tool_call_id": call["id"],
                "content": str(result)
            })
    return "Max steps reached without final answer."

queries = [
    "What is the capital of France?",
    "Compute sqrt(144) + 2**10",
    "What is the meaning of life?",
]
for q in queries:
    answer = run_agent(q)
    print(f"Q: {q}\nA: {answer}\n")

Running this loop produces:

Q: What is the capital of France?
A: Based on the tool result: Paris

Q: Compute sqrt(144) + 2**10
A: Based on the tool result: 1036.0

Q: What is the meaning of life?
A: I can answer that directly: 42.

The mock demonstrates the structural mechanics: messages accumulate, tool calls are dispatched to a typed registry, results are injected as tool-role messages, and the agent terminates on a content-only assistant turn. Replacing mock_llm with an actual API call (passing tools=[s["schema"] for s in TOOL_REGISTRY.values()] and parsing the response’s tool_calls field) converts this skeleton into a production loop.

The second demonstration shows tool composition: computing a result that requires the calculator tool while leaving the reasoning structure in the calling code.

def multi_step_demo() -> dict[str, float]:
    results = {}
    interest_rate = 0.05
    principal = 1000.0
    years = 3
    expr = f"{principal} * (1 + {interest_rate}) ** {years}"
    results["compound_interest"] = calculator(expression=expr)
    results["log2_population_estimate"] = calculator(expression="log2(8e9)")
    results["answer"] = results["compound_interest"] / results["log2_population_estimate"]
    return results

print(multi_step_demo())

This pattern implements Chain-of-Abstraction manually: the computation graph is expressed in Python (the “abstract” layer), and tool calls fill in the numerical leaves.

226.9 9. Evaluation and Failure Modes

Benchmarking tool-use agents requires metrics beyond end-task accuracy. Three dimensions matter.

Call precision and recall. Does the model invoke the correct tool for each sub-problem (precision), and does it invoke all necessary tools (recall)? APIBench measures this via Abstract Syntax Tree (AST) comparison of generated versus reference API calls.

Argument correctness. Even when the correct tool is named, arguments may be wrong. Argument correctness requires field-level comparison and is more sensitive than name-level matching. Gorilla’s primary gain over baseline models is in argument correctness rather than tool selection.

Efficiency. An agent that correctly answers a question in 10 tool calls when 2 suffice is expensive to run. Trajectory length is a proxy for reasoning efficiency and correlates with latency and API cost.

Common failure modes include: invoking a tool when parametric knowledge suffices (over-reliance), failing to invoke a tool when external grounding is needed (under-reliance), generating syntactically valid but semantically incorrect argument bundles (hallucination), and entering cycles where repeated tool calls return the same observation without triggering a state change in reasoning.

Mitigations include explicit stopping criteria in system prompts, tool-use training data that includes negative examples (situations where no tool call is correct), and structured output validation before dispatch.

226.10 References

1. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023. https://arxiv.org/abs/2210.03629

2. Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023). Gorilla: Large Language Model Connected with Massive APIs. arXiv preprint. https://arxiv.org/abs/2305.15334

3. Yang, Z., Li, L., Wang, J., Lin, K., Azarnasab, E., Ahmed, F., Liu, Z., Liu, C., Zeng, M., & Wang, L. (2023). MM-ReAct: Prompting ChatGPT for Multimodal Reasoning and Action. arXiv preprint. https://arxiv.org/abs/2303.11381

4. Gao, J., Gu, L., Su, J., Guo, Y., Li, M., Li, J., Chen, W., & Bi, W. (2024). Efficient Tool Use with Chain-of-Abstraction Reasoning. arXiv preprint. https://arxiv.org/abs/2401.17464

5. Li, X., Ming, R., Chu, M., Shao, S., Jin, R., & Xiong, C. (2026). ACM: Agentic Context Management for Long Horizon Tasks. arXiv preprint. https://arxiv.org/abs/2607.23809

Li, Xiaochuan, Ryan Ming, Meng Chu, Shuai Shao, Rong Jin, and Chenyan Xiong. 2026. “ACM: Agentic Context Management for Long Horizon Tasks.” arXiv Preprint arXiv:2607.23809.