When an Agent Calls a Tool, What Happens to Its KV Cache?

A normal completion moves from prompt prefill into autoregressive decode and eventually terminates. A tool-using agent interrupts that lifecycle. The model emits a tool call, waits for an API, database, browser, code sandbox, human approval, or another external system, and resumes after the result is appended to its context.

The pause may last a millisecond or a minute. No tokens are generated during that interval, but the request may still own scarce accelerator memory through its accumulated key-value state.

The serving layer now has to decide whether to keep that state, move it somewhere else, or discard and rebuild it. This is where agent-runtime semantics meet inference scheduling.

Complete source code: ihiteshsharma/agent-tool-pause-kv-cache
Clone the verified implementation or use it to compare each step in the experiment.

The intercepted generation lifecycle

One tool round looks like this:

prompt prefill
    -> model decode
    -> tool call emitted
    -> tool wait
    -> tool result appended
    -> model resumes
    -> final answer or another tool call

Before the pause, the runtime has processed the system prompt, conversation, retrieved documents, tool definitions, and generated tool-call tokens. Their keys and values represent reusable computation.

When the tool returns, the model needs that previous context plus the new result. Discarding the cache can force another prefill of the retained history. Keeping it avoids that work but prevents the memory from serving something else while the model waits on an unrelated system.

The central trade-off is:

A paused request exchanges resume latency and recomputation for memory capacity that could serve other work.

A paused request still owns state

For the Qwen2.5-3B-style dimensions used in the preceding capacity walkthrough, one 4096-token f16 cache is 144 MiB:

36 layers × 2 × 2 KV heads × 128 dimensions × 2 bytes
= 36 KiB per retained token
= 144 MiB at 4096 tokens

Cache size alone is not enough to price a pause. Duration matters too. A useful first-order quantity is memory-time:

accelerator MiB-seconds = cache MiB × pause seconds

A 144 MiB cache held for 50 ms spends 7.2 MiB-seconds. The same cache held for five seconds spends 720 MiB-seconds. The request occupies the same memory in both cases, but the opportunity cost is radically different.

Policy 1: keep the state

The simplest policy leaves KV state in accelerator memory until the tool returns.

Keeping provides immediate resume, no transfer, no repeated prefill, and simple lifecycle accounting. It works well for short, predictable tools or when memory pressure is low.

The cost is continued occupancy. Slow or abandoned tools can pin memory, reduce batch size, delay admission, or force another request to be preempted. Enough paused workflows can turn an inference server into a cache for inactive work.

Keep is therefore a latency-first choice, not a free default.

Policy 2: swap the state

A swapping policy transfers paused KV state to host or another memory tier and restores it when the tool returns.

This frees accelerator capacity and preserves previous computation. In exchange, swap-out and swap-in consume bandwidth, restore adds resume delay, simultaneous transfers contend with model traffic, and the host tier becomes another capacity and failure boundary.

If swap-out overlaps the tool wait, user-visible restore cost is dominated by swap-in:

resume delay = cache size / effective transfer bandwidth

The word effective is important. Peak interconnect bandwidth is not application-observed bandwidth under concurrent transfers, NUMA effects, and model traffic.

Policy 3: discard and recompute

The runtime can release the state completely and reconstruct it by prefilling the retained context after the tool returns.

This requires no host cache or transfer protocol and frees accelerator memory during the wait. The cost is repeated computation:

resume delay = retained tokens / effective prefill tokens per second

Long contexts and repeated tool rounds amplify the cost. Later rounds can reprocess an increasingly long conversation, and the recomputation competes with new prefill and decode work.

Recompute is attractive when context is cheap, transfers are congested, or preserved state is unavailable. It is not automatically cheaper merely because it frees memory.

Tool latency is a distribution, not one number

A policy based on average tool latency will fail at the tail. A tool with a 50 ms median and a 20 s P99 can pin memory during exactly the overload periods where capacity matters most.

Measure bounded latency classes with percentiles and deadlines. Distinguish local reads, remote reads, mutating APIs, browser tasks, sandboxes, human approval, and long-running jobs. The class is a scheduling hint, not a guarantee.

The policy also needs retained tokens at the pause, cache pressure, transfer queues, prefill capacity, priority, client lifetime, and the likelihood of another tool round.

The scheduler needs bounded agent semantics

A completion scheduler sees tokens, queues, and device state. An agent runtime knows which tool was called, whether it is read-only or mutating, its deadline, whether cancellation is safe, and whether the workflow must resume.

A pause contract can expose only the bounded information the serving layer needs:

{
  "run_id": "run-42",
  "tool_class": "remote_read",
  "deadline_ms": 2000,
  "retained_tokens": 4096,
  "resume_required": true,
  "priority": "interactive"
}

The serving layer combines this with actual cache occupancy, transfer contention, prefill throughput, and SLOs. A model should never choose its own resource priority, and an untrusted tool name should not control isolation or admission.

Thresholds are better than a universal policy

Keep, swap, and recompute optimize different resources. A practical policy therefore uses thresholds:

1. Keep while predicted remaining wait is short and memory pressure is low.
2. Swap when the expected wait exceeds restore cost and a protected host tier has capacity.
3. Recompute when transfer is congested, preserved state is unavailable, or the retained context is cheap to prefill.
4. Cancel and release all state when the deadline or client lifetime ends.

The thresholds should come from observed distributions, not intuition. Compare marginal memory-time, transfer time under contention, prefill time under contention, resume-latency objectives, and the goodput impact on other admitted requests.

Multiple tool rounds amplify every choice

Agents often alternate generation and tools several times. Discarding after every call can repeatedly prefill a growing history. Keeping every paused request accumulates a large inactive working set. Swapping every time can make transfers dominate.

Evaluate complete trajectories rather than one pause in isolation. Record the number of tool rounds, retained tokens at each pause, pause duration, policy decision, bytes moved, tokens recomputed, resume delay, task success, cost, and impact on unrelated requests.

That last measurement prevents a selfish policy from appearing optimal. Keeping one interactive workflow fast may reduce service goodput or violate another tenant's SLO.

What InferCept changes in the serving conversation

InferCept studies augmented LLM workloads where tools or humans pause generation. It manages intercepted contexts through scheduling, preservation, swapping, and recomputation decisions.

The broader lesson is that agentic workloads need different serving treatment from isolated completions. The paper does not establish the local numbers below; those values come only from the declared architecture and analytical assumptions.

A decision framework for paused state

Before selecting a policy, ask:

1. How much KV state is retained when the workflow pauses?
2. What is the tool's latency distribution and hard deadline?
3. What are measured swap and prefill costs under contention?
4. How much accelerator and host headroom exists?
5. Which resume-latency and system-goodput objectives apply?
6. What happens on cancellation, client disconnect, or a tool that never returns?

The agent's tool call pauses generation. It does not pause the cost of owning the generated state.

Experiment: compare keep, swap, and recompute

The standalone lab uses two 4096-token paused requests: one waits 50 ms and one waits 5000 ms. It declares two hypothetical system inputs:

• 8 GiB/s effective transfer bandwidth;
• 20,000 effective prefill tokens/s.

It also assumes swap-out overlaps the tool wait. These values expose the trade-off; they are not observed results from llama.cpp, vLLM, or the Apple M2 Pro host.

Run it:

git clone https://github.com/ihiteshsharma/agent-tool-pause-kv-cache.git
cd agent-tool-pause-kv-cache
python3 -m unittest -v test_lab.py
python3 tool_pause_lab.py

The script calculates each policy independently and refuses malformed requests, non-positive inputs, unknown policies, and empty workloads.

Verification and observed behavior

The fresh run executed seven tests and returned "verdict": "passed".

The totals cover both requests. Swap moves each 144 MiB cache out and back, or 288 MiB per request. Recompute processes 4096 retained tokens per request. Keep spends 7.2 MiB-seconds on the short pause and 720 MiB-seconds on the long pause.

What the analytical model establishes

Keeping minimizes resume delay while consuming accelerator memory throughout the pause. Swap releases that memory by paying transfer cost. Recompute releases it without a host cache but pays prefill cost.

No policy wins until the workload supplies an objective and resource constraints. The table is a comparison of first-order costs, not a scheduler benchmark or a recommendation to swap every paused agent.

The model omits allocator fragmentation, concurrent scheduling, transfer contention, device topology, prefix sharing, partial preservation, distributed routing, failures during swap, and runtime-specific cache formats.

Failure injection: the tool that never returns

An unbounded tool can leak useful serving capacity even though no process crashes. The runtime needs a tool deadline, cancellation propagation, maximum retained-cache time, a swap-or-discard threshold, client-disconnect cleanup, and a terminal trace outcome.

Timeout does not prove that a mutating tool failed safely. It may have committed before its response was lost. The agent's idempotency and reconciliation boundary controls whether retry is safe; the inference layer's cache policy controls how expensive waiting and resuming are. These are related failure windows, not one mechanism.

Observability, security, and production implications

Instrument explicit lifecycle states:

prefill -> decode -> tool_wait -> restoring -> decode -> completed
                                  \-> cancelled

Record transition timestamps, retained tokens and cache bytes, tool class and deadline, keep/swap/recompute/cancel decisions, bytes moved, tokens recomputed, restore queue and duration, resume latency, task and trajectory verdicts, cache occupancy, and service goodput.

Do not label tool wait as model latency or hide restore time inside tool latency. Those boundaries determine which subsystem can diagnose the delay.

Paused KV state encodes sensitive context. Moving it to host or remote storage expands the attack surface. Protect transfer channels, isolate tenants, restrict dumps, define retention, and verify cleanup after cancellation or crash. Tool results remain untrusted inputs; preserving computation does not preserve trust.

A production experiment should replay real agent traces against the target runtime, replace assumed rates with measured distributions, inject slow and abandoned tools, and evaluate P95/P99 resume latency, memory occupancy, preemption, task success, and system goodput.

Cleanup

The analytical script creates no external resource and writes no persistent state. A runtime experiment must also release paused contexts, stop the server, confirm listeners are gone, and remove any cache dump containing derived user state.

Sources

• InferCept: Efficient Intercept Support for Augmented Large Language Model Inference
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
• llama.cpp server documentation