Blog

How you're overspending on tokens

Most teams think LLM costs scale with users. They actually scale with how much context you inject per request. Here is where the waste is and how to find it.

If your LLM costs are growing faster than your user base, the instinct is to look at request volume. More users, more calls, more spend. But in most agent architectures the real driver is not how many requests you are making, it is how many tokens you are putting into each one.

A single request to GPT-4o with 15,000 tokens of context costs roughly ten times more than the same request with 1,500 tokens. The model does not care whether those extra tokens helped it answer the question. You pay for all of them regardless.

Most teams are overspending not because they are using the wrong model or handling too many requests, but because they are sending context that is not contributing to the answer. Here is where that waste typically comes from.

Full conversation history injection

This is the largest source of unnecessary token spend in most agent architectures. The standard approach is to append the full conversation history to every request so the model has access to everything the user has said.

The problem is that most of that history is not relevant to the current question. A 20-message conversation might be 12,000 tokens, and the answer to "what plan am I on" depends on a single exchange from message 4. The other 11,500 tokens are filler: greetings, tangents, the user rephrasing their question, the assistant's preamble on earlier responses.

The cost scales linearly with conversation length. By message 40 you might be injecting 25,000 tokens per request, most of which the model will never use. And this is not just a cost problem: research shows that models degrade with irrelevant context, so you are paying more and getting worse answers at the same time.

Naive top-K RAG retrieval

The standard RAG pattern retrieves the top-K chunks from a vector store by semantic similarity and injects all of them into the prompt. Top-5 chunks at 500 tokens each is 2,500 tokens of context per request.

The problem is that similarity is not the same as relevance. Of those five chunks, maybe one or two actually contain the answer. The rest scored highly because they are about the same general topic but do not contain the specific information the model needs. You are paying for 2,500 tokens to deliver what 400 tokens could have covered.

As the knowledge base grows, this gets worse. Semantic collapse means the retrieval layer starts returning things that sound right rather than things that are right, and you are paying for all of them.

LLM calls that should be lookups

Some questions your agent handles do not need a language model at all. What plan is this user on, what is their preferred language, have they contacted support before. These are fact lookups, not generation tasks.

But if those facts only exist inside past conversation history, the only way to access them is to stuff that history into a prompt and ask the model to find them. You are paying for thousands of tokens of context plus the inference cost of a full LLM call to retrieve a single fact that could have been a database query.

Why the obvious fixes do not work

When teams notice their token costs climbing, the typical responses make the problem worse or just move it around.

Shorter conversations penalise users. Asking people to start new sessions or limiting conversation length is a product decision that trades user experience for cost management. It does not address the underlying architecture.

Cheaper models sacrifice quality. Moving to a smaller model reduces cost per token but the model is also less capable of extracting the right answer from noisy context, which means the context problem becomes a quality problem instead of a cost problem.

Summarisation is lossy. Compressing conversation history into summaries reduces token count but discards detail that may be needed later. The summariser decides what matters before knowing what future questions will be asked, and that decision cannot be reversed. See context window overflow for a fuller treatment of why this approach has a ceiling.

Larger context windows make it worse. The instinct that more context capacity solves the problem is wrong. Research consistently shows that model performance degrades with context length. A larger window means more room for irrelevant tokens that the model cannot reliably attend to.

How to diagnose it

The single most useful metric is average tokens per request over time. If this number is growing faster than your request count, you have a context problem, not a volume problem.

Break it down by component: how many tokens are system prompt, how many are conversation history, how many are RAG chunks, how many are the user's message. In most architectures, conversation history and RAG chunks together account for 80% or more of the total, and that is where the opportunity is.

Try the token cost calculator to estimate what your current architecture is costing you and what precise retrieval would change.

What actually works

The fix is architectural rather than operational. Instead of stuffing raw context into the prompt and hoping the model sorts it out, you extract the facts that matter, store them in structured memory, and retrieve only what is relevant to each specific query. Instead of dumping top-K chunks from a vector store, you search at the sub-document level and return precise passages.

The result is fewer tokens per request, better answers because the model is working from clean context rather than noise, and costs that scale flat with usage rather than linearly with conversation length.

For the technical patterns behind this, see how to cut token costs. For how Exabase implements these patterns as managed infrastructure, see how Exabase reduces your token spend.