Reduce your agent's token spend

The first LLM cost spike usually gets blamed on traffic. More users, more calls, more spend, which sounds right until you look at the invoices properly. The growth tracks with context per request, not requests per day. Your token count per call has been creeping up, and you pay for it on every single inference.

This page is for teams whose agent is in production, or close to it, and whose costs are climbing faster than usage. You have probably already tried the obvious levers. This is about the one that actually moves the number, which is the shape of what you send.

What you have probably already tried

The standard fixes each help a little and none of them address the cause.

Cheaper models reduce cost per token, but a smaller model is less able to pull the right answer out of noisy context. You turn a cost problem into a quality problem and often end up paying for retries.

Prompt caching is worth doing and genuinely helps with static prefixes, but conversation history and retrieved chunks are exactly the parts that change every request, so the growing portion of your context is the part caching cannot cover.

Shorter conversations work by penalising your users, which is not a fix so much as a tax on the experience.

Summarisation is lossy in a way that bites later. The summariser decides what matters before knowing what will be asked, and that decision cannot be reversed. It also strips the specificity that made the context useful: the exact date, the exact constraint, the exact preference.

Bigger context windows make it worse rather than better. Effective context sits at roughly 50 to 65 percent of marketed capacity, performance degrades by more than 30 percent when relevant information lands in the middle of the input, and every model tested in one study of 18 production LLMs showed monotonically declining accuracy as input length grew. You pay more and get less accurate answers at the same time. Our BEAM benchmark write-up collects the research on this, and context window overflow covers the practical version.

Where the tokens actually go

A typical agent request is a system prompt of around 500 tokens, the full conversation history at anywhere from 3,000 to 30,000 depending on session length, retrieved chunks at another 1,500 to 3,000, and the user's actual message at perhaps 100. By the time it reaches the model you are sending 15,000 tokens, and most of them are not helping answer this particular question.

Conversation history is the largest source of waste. A 20-message thread might be 12,000 tokens, and when the user asks what plan they are on, the answer depends on one exchange from message four. The other 11,500 tokens are greetings, rephrasings, and the assistant's own preamble from earlier turns. You are paying for all of it, every time.

Retrieval compounds it. The standard approach pulls the top five chunks by vector similarity and injects all of them, which is around 2,500 tokens. Maybe one or two contain the answer. The rest scored well because they are about the same general topic, and similarity is not the same as relevance. You are paying for the difference.

How Exabase changes the shape

Two mechanisms replace bulk context with precise context.

Memory replaces conversation history. Send a conversation through and Exabase extracts the facts that matter, storing them as structured memories rather than transcript. An 8,000-token conversation might reduce to around 50 tokens of durable facts: preferences, decisions, stated constraints. On the next request you retrieve only what is relevant to the current query, which is typically 200 to 400 tokens instead of 12,000. The important property is that this stays nearly flat as conversations lengthen, because you are retrieving relevant facts rather than proportional history.

It also resolves contradictions when memories are written rather than at read time. If a user was on the free plan at message five and upgraded at message 22, only the current state is stored, so the model receives one clean fact instead of two conflicting chunks it has to reconcile. That is the memory drift problem handled at the source.

Deep Search replaces chunk dumps. Rather than top-K retrieval of fixed-size blocks, it finds specific passages at the sub-document level, with a precision setting you control per query. Turn it up for factual questions where one passage holds the answer, turn it down for research queries where breadth matters. Either way you are deciding token spend per request instead of accepting whatever the vector store returns.

There is independent evidence this architecture is the cheaper one. On the BEAM benchmark, M-1 holds state of the art at 100K, 1M, and 10M tokens while consuming roughly 20 percent fewer total tokens per query than the next best system, and doing it on a model that is 4 to 6 times cheaper than the one competing systems used. Precise retrieval is not a trade against quality. It is what produces both.

Get started

The token cost calculator is the fastest way to size this against your own traffic. For the technical walkthrough with code, how to cut your token costs covers the patterns in full, how you're overspending on tokens is the diagnostic version, and how Exabase reduces your token spend is the implementation view. The Memory and Deep Search pages have the API detail, and there is a free tier to test against.

FAQs

Why is my LLM bill growing faster than my user count?

Because cost tracks context per request rather than requests per day. As conversations lengthen and knowledge bases grow, the tokens injected into each call creep up, and you pay that on every inference. The number to watch is average input tokens per request, not request volume.

Doesn't prompt caching already solve this?

Prompt caching helps with the static parts of your prompt, and it is worth using. It does not help with conversation history or retrieved chunks, which change on every request and are the parts that grow without bound. Use caching alongside better context construction rather than instead of it.

Won't a bigger context window fix it?

It makes both cost and quality worse. Effective context is roughly 50 to 65 percent of marketed capacity, accuracy drops sharply when relevant information sits in the middle of a long input, and tested models show declining performance as input grows. The research is collected in our BEAM write-up.

How is extracted memory different from summarising the conversation?

A summary is written before anyone knows what will be asked, so it discards detail that turns out to matter and cannot be recovered. Memory extraction stores discrete facts that stay individually retrievable, and resolves contradictions as facts change, so what you retrieve is current and specific rather than a lossy paraphrase.

How much can I actually expect to save?

The illustrative case here is about 81 percent fewer input tokens, from roughly 15,100 to 2,800 on a single request. Real savings depend on how long your conversations run and how much you retrieve. The token cost calculator estimates it for your workload.

Does cutting context hurt answer quality?

The opposite, generally. Models degrade with irrelevant context, so a smaller, precisely targeted context usually produces better answers than a large, roughly related one. On BEAM, M-1 scored highest at every scale while using around 20 percent fewer tokens than the next best system and a substantially cheaper model.

Do I have to rewrite my agent to adopt this?

The change is at the context layer: retrieve memories and precise passages before the call instead of assembling raw history and top-K chunks. Your agent logic, prompts, and model choice stay as they are. The walkthrough shows the shape of the integration.