Cutting AI Agent Cost and Latency

Where an agent's tokens and time go, and how streaming, prompt caching, batching and model sizing cut both without hurting quality.

An e-commerce team launched two AI Agents at once. The first answers customer chats; the second summarises the sales report every Monday morning. A month later they had two problems: customers complained the chat was slow, and the API bill was higher than planned. The two problems need different fixes, and every agent builder runs into them sooner or later.

Where agent cost and latency come from

Agent cost is charged by tokens, both the ones you send in (input) and the ones the model generates (output). Latency comes from the time to generate output plus the number of rounds that call the model and tools. The catch is that an agent runs many rounds through the Agent Loop, and each round carries a longer context. Anthropic reports that agents use about 4 times the tokens of a normal chat, and multi-agent systems about 15 times.

Cutting latency: first understand what takes time

OpenAI's latency optimization guide sets out 7 principles: process tokens faster, generate fewer tokens, use fewer input tokens, make fewer requests, parallelize, make users wait less, and don't default to an LLM. The points people most often get wrong:

Output matters more than input

The guide says that cutting output tokens by 50% may cut latency by about 50%, but cutting the prompt by 50% may only improve latency by 1 to 5%. If chat feels slow, first check whether the model is answering at too much length.

Combine steps and run in parallel

Steps that always run in sequence can be merged into one request, and independent work should run at the same time. Anthropic reports that running subagents and tools in parallel in its Research system cut time on complex queries by up to about 90%.

Streaming

OpenAI's guide calls streaming the most effective way to make users feel they wait less, because they see the answer appear right away.

Cutting cost: four main levers

1. Prompt caching

Agents often send the same system prompt and tool descriptions every round. Claude's prompt caching documentation explains that the cache works on the beginning of the prompt in the order tools, system, then messages. Cache reads are priced at about 0.1 times the normal input price for many models, and by default the cache lives for 5 minutes and is refreshed each time it is used. The trick is to put what never changes at the front and what changes every time at the end.

2. Batch work that can wait

The Monday report does not need to arrive within seconds. Anthropic announced that its Message Batches API costs 50% less than standard calls, in exchange for processing within 24 hours.

3. Match model size to the task

OpenAI notes that smaller models are faster and, with good prompts or examples, can come close to larger models on specific tasks. Classifying messages or extracting fields from forms rarely needs the largest model.

4. Control tool output size

Long search results flowing back into context every round are a hidden cost. Anthropic says Claude Code limits tool responses to 25,000 tokens by default. Your system should use pagination or filters from the start.

Back to the e-commerce team

• Chat agent: instruct it to answer more briefly, turn on streaming, move shop policies into a cacheable system prompt, and use a small model to classify questions first
• Report agent: move it to an overnight batch and fetch only the columns it needs from the database

Watch out for

• Cheaper can mean worse: Anthropic found that token usage explained 80% of the performance variance on BrowseComp. Cutting too many tokens can hurt quality, so measure with evals after every change
• Prices change often: always check the provider's current pricing page before budgeting
• Caching breaks if the start of the prompt changes: even a small change, such as putting the date and time at the top, prevents cache hits

The best agent is not the cheapest one. It is the one that only pays for tokens that create value.

What to do next

1. Record input tokens, output tokens and number of rounds for each agent task
2. Separate work that needs an instant answer from work that can wait
3. Order prompts so the fixed parts come first, then turn on caching
4. Try small models on simple subtasks
5. Measure quality before and after every change

Managing the cost and performance of AI systems is a practical skill for technology teams of any size. Browse related roles on Plants.work Jobs, or use Resume Lab to present this kind of work clearly on your resume.