AI agent tool routing can turn a simple delivery question into a surprisingly long wait. The customer asks where an order is, and the agent searches products, reads policy documents and checks the customer profile before finally looking up the shipment.
Each tool may work correctly. The route through them is wasteful. Let’s give straightforward requests a short path before asking the model to decide everything.
Recognise the work with a clear answer path
Start with a few requests you can identify reliably: an explicit order-status button, a known menu action or a structured command from your interface. Map those to ordinary application functions.
Don’t begin with a giant list of loose keyword checks. “Where is my refund?” and “where is my order?” may need different services and permissions. Ambiguous text should go through a clarification or interpretation path.
Anthropic distinguishes predefined workflows from agents that choose their own steps. That distinction is useful here: you can use both within one product. Agent architecture reference.
Bound the path that still needs an agent
For open-ended requests, give the agent a small relevant tool set and a task budget. Track tool calls, elapsed time and business actions separately. A six-call limit doesn’t prevent one call from running for a minute.
Reserve the ability to explain a failure. If the budget runs out, return the verified progress and the next useful question. Don’t ask the model to invent a completed answer to satisfy a response deadline.
Keep authorisation in the tools. A deterministic route still needs to verify that the customer owns the order; a shorter path isn’t permission to skip checks.
Build the route before the generation layer
This example handles one explicit status command directly. Other input reaches a bounded fallback. It also reuses the same in-flight lookup within one request.
import assert from 'node:assert/strict';
let reads = 0;
async function status(id) { reads++; return { id, state: 'in_transit' }; }
function requestCache() {
const pending = new Map();
return (key, load) => {
if (!pending.has(key)) pending.set(key, Promise.resolve().then(load));
return pending.get(key);
};
}
async function route(input) {
if (input.kind === 'order_status' && input.orderId) {
const once = requestCache();
const key = JSON.stringify([input.tenant, input.orderId]);
const [a,b] = await Promise.all([once(key,()=>status(input.orderId)), once(key,()=>status(input.orderId))]);
return { route: 'direct', answer: a.state, same: a === b };
}
let calls = 0;
while (calls < 2) calls++;
return { route: 'bounded_fallback', calls, outcome: 'clarify' };
}
assert.deepEqual(await route({ kind:'order_status',tenant:'a',orderId:'1' }), { route:'direct',answer:'in_transit',same:true });
assert.equal(reads, 1);
assert.equal((await route({ kind:'free_text' })).calls, 2);
console.log('PASS: direct routing, in-flight read reuse and bounded fallback');
The status path makes one underlying lookup even when two parts of the handler request it. The fallback stops at its budget. No model is called here; the example tests the control flow you put around a model.
Keep reuse inside the right scope
A request-scoped cache can prevent repeated reads during one answer. Key it by tenant, operation and normalised arguments. Cache the promise while the read is in flight so simultaneous callers share the work.
Don’t reuse mutation calls just because their argument strings match. Two deliberate purchases of the same item are different operations. Mutation safety needs stable business operation IDs and an idempotency contract.
Likewise, a request cache shouldn’t quietly become an hour-long shared cache. An order status or current stock result can change. Give any cross-request cache a clear freshness and invalidation policy.
Record the route taken
Store an execution trace with the selected route, tool names, durations, result classes and final task outcome. Exclude sensitive arguments unless they are necessary and protected.
The route label helps explain regressions. If an explicit menu action suddenly starts using the open-ended agent path, tool count and latency may rise without a change to the tools themselves.
Also record why a direct route was rejected. A missing order ID is useful evidence for improving the interface. It shouldn’t be hidden inside a generic “agent failed” metric.
Test task completion, not fewer calls alone
A route that avoids every tool is cheap and useless. Evaluate whether the customer gets the correct order status, whether ownership was checked and whether uncertain input prompted clarification.
Use the same task set for your baseline and your new route. Include paraphrases, multiple order references and requests that look simple but need two dependent reads. Measure cost and latency only after you know the task outcomes are acceptable.
The local fixture proves a routing rule and a budget. To compare real model behaviour, run repeated evaluations with representative tool outputs and record variability. Don’t publish a percentage reduction from this small deterministic example.
Keep the simple paths maintainable
Put direct routes in ordinary tested functions. Give them explicit inputs and typed outputs. Avoid copying business rules into both a direct handler and a separate agent tool; let both call the same authorised service.
Review the route catalogue as the product changes. If most traffic uses a complex fallback because the common actions are missing, add the right structured entry point rather than increasing every agent’s budget.
Start with the most common clear request in your logs. Build its shortest correct route, keep the fallback bounded and compare completed outcomes. You can remove wasted work without turning every conversation into a rigid menu.
Run the example and adapt it
Save the code as 08-bounded-agent-routing.mjs and run it with Node.js 24 or later:
node 08-bounded-agent-routing.mjs
The code above includes assertions you can run locally. It tests the local decision logic with controlled inputs; it doesn’t connect to a live provider or establish production performance. Keep that boundary when adapting it to your application.
