Exabase M-1: State of the Art on BEAM at 100K, 1M, and 10M Tokens

Exabase Research July 2026

Abstract

We present results from evaluating M-1 against the BEAM benchmark (Tavakoli et al., ICLR 2026) at 100K, 1M, and 10M token scales. M-1 achieves state-of-the-art results at all three scales, scoring 76.9% on BEAM-100K, 75.0% on BEAM-1M, and 68.0% on BEAM-10M, surpassing all previously reported systems. M-1 achieves this using Gemini 3 Flash, beating previous frontier results which depended on a more expensive model (Gemini 3 Pro) and higher token usage, while also consuming approximately 20% less total tokens than the next best-ranked system. Combined with our recent LongMemEval results (96.4%), M-1 is now the only memory system to hold state-of-the-art results across both major conversational memory benchmarks at every evaluated scale.

1. Introduction

Our earlier evaluation on the LongMemEval benchmark showed that it is retrieval architecture, and not model scale, that drives memory system quality. But a natural question follows: does this hold as scale increases?

LongMemEval evaluates using 115,000 tokens of conversational history. At that scale, a sufficiently large context window can technically hold the entire corpus, meaning a naive "dump everything into context" approach can still participate in the benchmark and produce a result.

BEAM was designed for a higher level of rigor:

At 1M and 10M tokens, two problems emerge. The first and most obvious, the text corpus physically exceeds most context windows. GPT-4o's 128K window holds approximately 1.3% of a 10M token corpus. Even Gemini 3 Pro's 1M window covers only ~10%. At this scale, selective retrieval becomes a physical necessity to even run the benchmark. It's not possible here to "dump everything into context".

Second, even where "context stuffing" is technically possible, it is ultimately counterproductive. Context windows are just containers, not brains. NVIDIA's RULER benchmark tested 17 long-context models, finding that the effective context length actually sits at roughly 50 to 65 percent of marketed capacity (Hsieh et al., 2024). Liu et al. (2024) demonstrated that performance follows a U-shaped function of information position, degrading by over 30% when relevant information is buried in the middle of the input. Tokens near the beginning of the input get attention (because early positions get favored structurally), tokens near the end of the input get attention (because they're closer to wherever the model is currently generating), but the tokens in the middle of a longer context get squeezed out – a bias which increases progressively with larger context. The middle tokens are far from both ends, so they lose out on both factors.

However, the degradation problem runs deeper than positional bias. Adobe's NoLiMa benchmark (Modarressi et al., ICML 2025) removed literal keyword overlap between questions and relevant information, requiring associative reasoning instead of surface-level matching. At 32K tokens, 11 of 13 tested models dropped below 50% of their short-context baselines. Chroma Research (Hong et al., 2025) tested 18 production LLMs on multi-hop reasoning tasks across 10,000 to 500,000 token contexts and found that all 18 showed monotonically decreasing F1 scores as input length grew. They coined the term "context rot" for this phenomenon. A separate study on intelligence degradation found that models maintain strong performance up to roughly 40% of maximum context length, after which performance can collapse catastrophically (arXiv:2601.15300, 2026).

To sum, adding more tokens to the context window not only doesn't help beyond a threshold, but the additional tokens actively drive performance degradation. A system that retrieves 5-10K precisely targeted and pertinent tokens will outperform one that stuffs 200K of somewhat relevant context, because the model attends to less noise and can waste fewer tokens on information that it cannot reliably process. This means that selective retrieval isn't just necessary at scales where stuffing is impossible, but it also produces better results even at smaller scales, where stuffing is an option.

This is the problem-set M-1 was built to solve.

2. About BEAM

BEAM ("Beyond a Million Tokens") was introduced by Tavakoli et al. at ICLR 2026 to address limitations in earlier memory benchmarks. Where benchmarks like LongMemEval concatenate separate user sessions to approximate long context, BEAM generates coherent, chronologically-ordered conversations that include realistic conversational follow-ups and clarifications across a range of 100k, 1M, and 10M token scales.

To make that token scale more tangible: 10 million is roughly a year of daily conversations with an AI agent, or a company's entire internal documentation corpus, or the complete output of a software project across hundreds of sessions. Rather than a more contrived stress test, BEAM is a real volume challenge that production systems accumulate over time.

The benchmark tests ten distinct memory abilities:

• Information Extraction

• Multi-hop Reasoning

• Knowledge Update

• Temporal Reasoning

• Summarization

• Preference Following

• Abstention

• Contradiction Resolution

• Event Ordering

• Instruction Following

The last three are currently only evaluated by the BEAM benchmark.

The benchmark's scoring uses a "nugget"-based methodology: where reference answers are decomposed into more atomic criteria, and an LLM judge scores the system's responses against these "nuggets" (0, 0.5, or 1). This means getting partial credit is achievable when the answer was within reasonable bounds, but getting a perfect score on a question is difficult.

BEAM is primarily designed to have resistance to context/memory shortcuts. At the scale of 1M+ tokens, the corpus is too large for any context window to hold reliably, so the only path to a good score is building a memory system that can reliably retrieve the right facts. And then at the 10M tier, the real differentiation happens: retrieval performance and information gathering become the only thing that matters.

3. Methodology

3.1 Evaluation Setup

We built our evaluation harness by forking the open-source benchmarking script released by Hindsight. We made the following modifications:

Storage and retrieval were fully replaced with M-1's retrieval engine.

Prompt simplification. We used the runner's prompt structure with an adjustment to better match production usage. Our interest was in measuring M-1's retrieval quality, not in prompt optimisation. You can view the prompt generator here.

Answering and judging model. We used a smaller model: Gemini 3 Flash, for both answering questions and judging correctness. This is discussed further in the next point.

3.2 Model Choice

As in our LongMemEval benchmark run, we used Gemini 3 Flash. All other systems on the BEAM leaderboard reported results based on Gemini 3 Pro. Our reasoning remains consistent: a memory system designed for production use should be able to deliver results with a practical model. Gemini 3 Flash is faster and up to 6x cheaper, which stacks up with every query. This matters even more at BEAM's scale, where the corpus is beyond an order of magnitude larger.

3.3 Token Efficiency

For this evaluation we also tracked the total token consumption across the pipeline: embedding generation, query decomposition, candidate retrieval, re-ranking, and answer generation.

If a system needs more tokens, it may be relying on the model to filter the additional noise. Token efficiency provides an additional signal of retrieval precision, as well as improving the practical cost-effectiveness of using a system in production.

4. Results

4.1 Overall Results

Our results show that M-1 leads at every scale, while once again using a model that is 4-6x cheaper and faster than the one used by the next best two competing systems.

Efficiency: M-1 consumed approximately 20% fewer total tokens per query than the next best system, whilst achieving higher scores across all scales.

4.2 Performance Across Scale

As token volume increases, M-1 shows graceful degradation: 76.9% at 100K, 75.0% at 1M, 68.0% at 10M. The drop from 100K to 1M is just 1.9 points, and the steeper decline to 10M is expected given the order-of-magnitude increase in corpus size.

The competitive gap tells a more interesting story. At 100K, M-1 leads Hindsight by 3.5 points. At 1M, the gap narrows to 1.1 points. At 10M, it widens again to 3.9 points. The widening gap at the largest scale would suggest that as the retrieval conditions become more challenging, M-1's retrieval architecture advantage is most pronounced.

Honcho's results show a different pattern: roughly flat performance change from 100K to 1M (63.0% to 63.1%), then a sharp collapse to 40.6% at 10M. The 27.4 point gap between M-1 and Honcho at 10M, compared to 13.9 points at 100K demonstrates how retrieval architecture differences get amplified by scale.

4.3 Category-level results for M-1

BEAM-100K (40 questions per category)

BEAM-1M (70 questions per category)

BEAM-10M (20 questions per category)

Observations:

The top tier (Preference Following, Instruction Following, Summarization, and Abstention) is consistently strong across all scales, never falling below 89%. This set of categories evaluate whether the system can find and follow explicit signals in the corpus, something that M-1's retrieval handles well regardless of corpus size.

The middle tier (Event Ordering, Information Extraction, Contradiction Resolution, and Temporal Reasoning) shows a clear degradation pattern. Event Ordering drops from 83.6% at 100K, down to 78.7% at 1M, and to 67.5% at 10M. Information Extraction follows a similar trajectory: 81.0% to 73.2% to 66.3%. These categories test the system's ability to locate specific facts and reason about their relationships, which progressively becomes more difficult with increased information volume.

Knowledge Update shows generally expected score falloff, with 52.5% at 100K, 55.7% at 1M, and 45.0% at 10M. This category evaluates the system's ability to recognise information change over time, and surface the most recent version. At greater scales, the volume of potentially contradictory information increases, making it harder to identify the current state of an entity, fact or synthesized fact.

Multi-session Reasoning is the clear (downwards) outlier, declining from 44.7% at 100K to 48.8% at 1M, before collapsing to 9.6% at 10M. This category requires piecing together fragments from multiple separate conversations to gather together an answer for a single question. At 10M scale, the relevant fragments are scattered across a huge and noisy volume of information. This is the hardest retrieval problem in the benchmark, and it clearly remains an open challenge at this scale. Notably, these challenges are observed across all existing memory systems, suggesting it represents a challenge for the field of AI memory still to solve, rather than anything specific to M-1.

5. Analysis

5.1 The 10M Tier: Where Architecture Is All That Matters

The 10M results are the most revealing. This scale is well beyond the effective region of context windows, beyond brute-force, and the only path to a good BEAM score is real memory and retrieval that finds the right information in a massive, noisy corpus.

M-1 scores 68.0% at 10M. Hindsight scores 64.1%. Honcho scores 40.6%. The spread between the latest systems is widest at 10M, confirming that scale amplifies the difference in memory system capability. A system that achieves competitive results at 100K through hacks or coasting on model performance will be exposed at 10M.

5.2 Why Context Windows Are Not Enough

BEAM at 1M and 10M tokens is far beyond the effective reasoning capacity of any current model. The research supports this: effective context is roughly 50-65% of marketed capacity (Hsieh et al., 2024), performance degrades by more than 30% in the middle of the input (Liu et al., 2024), associative reasoning collapses at just 32K tokens when keyword shortcuts are removed (Modarressi et al., 2025), and every tested model shows monotonically decreasing performance as input size grows (Hong et al., 2025). Some models exhibit phase-transition-like collapse at as low as 40% of maximum context length (arXiv:2601.15300).

This means that even as context windows continue to grow to meaningful scales (1M+), the case for memory systems only gets stronger. Larger windows simply give you a larger container of text that the model cannot reliably reason through. Context rot means that a model processing 200K tokens of somewhat relevant context will perform significantly worse than one processing 10K tokens of precisely focused context, because it has to attend to more noise, losing coherence in the middle of the input.

M-1's results at BEAM-10M are a direct demonstration of the above points. By surfacing a radically cleaner, well-targeted context from a 10M token corpus, M-1 allows a smaller model to outperform previously-frontier systems that depend on larger models to eat their way through a larger, noisier context.

5.3 Token Efficiency as a Quality Signal

M-1's 20% lower token consumption at higher accuracy goes further in service of this. It reflects what M-1's retrieval pipeline is doing: surfacing well-targeted, low-noise context that the answering model can act on. The combination of lower tokens and higher accuracy indicate that the M-1 system is doing more work prior to the model seeing any context, which is the ideal division of labour for a production-practical memory system.

This also directly lowers the system's cost in real use. At 1M+ token scale, the token savings are obvious and compounding: 20% fewer tokens on every query across thousands or more of daily queries represents a material cost reduction, which is further on top of the 4-6x savings from the ability to use a lighter, cheaper LLM. The total cost advantage of M-1 over other memory systems can be as great as an 86%+ saving.

5.4 SOTA across the board: LongMemEval + BEAM

M-1 is now the only system to hold state-of-the-art results across both major memory benchmarks at every evaluated scale:

The consistent performance across scales and separate benchmarks indicate that M-1's advantage is architectural rather than benchmark-specific. It also demonstrates that a memory system's performance is primarily determined by retrieval architecture rather than the model.

6. Limitations

• Results reflect one model configuration (Gemini 3 Flash) with no ablation studies

• M-1's architecture details remain intentionally limited

• Multi-session Reasoning at 10M scale remains an open challenge, with M-1 scoring 9.6% — this category appears to be somewhat architecturally unsolved for all current memory systems at BEAM scale

• BEAM's LLM judge introduces scoring variance; the paper does not fully standardize the judge model or evaluation prompts, which may affect cross-implementation comparability

• Known noise in BEAM corpus (repeated strings, leaked generation plans) which may affect a subset of individual question scores

• Production conditions (adversarial inputs, concurrent multi-user load, privacy constraints) not tested

• Token consumption comparison is approximate and based on published results; competing systems may have different pipeline architectures that make direct comparison imprecise

7. Conclusion

M-1 achieves state-of-the-art results on BEAM at 100K, 1M, and 10M token scales, while using a smaller model and consuming fewer total tokens than competing systems. Combined with our LongMemEval result, M-1 is the only memory system to hold the top score on both major benchmarks at every evaluated scale.

M-1 represents a pareto-frontier improvement: combining the highest recorded accuracy scores with lowest cost and latency.

The competitive gap shows further widening at the largest and toughest scales: where memory and retrieval architecture are the only factors, M-1 leads by the largest margin. As context windows grow but effective reasoning capacity lags behind, the case for precise retrieval architecture becomes stronger. These results demonstrate that M-1's approach scales.

The category-level results reveal both strengths and open challenges yet to solve. M-1 maintains near-perfect performance on Preference Following, Instruction Following, Summarization, and Abstention at all scales. However, Multi-session Reasoning at 10M remains the hardest problem in the benchmark, and improving retrieval for scattered, cross-session information at extreme scale is a priority for our next iteration.

References

Tavakoli et al. (2025). "Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs." ICLR 2026.

Wang et al. (2024). "LongMemEval: Benchmarking Long-Term Memory in AI Assistants."

Hsieh, C-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., and Ginsburg, B. (2024). "RULER: What's the Real Context Size of Your Long-Context Language Models?" Conference on Language Modeling (COLM).

Liu, N.F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2024). "Lost in the Middle: How Language Models Use Long Contexts." Transactions of the Association for Computational Linguistics.

Modarressi, A., Deilamsalehy, H., Dernoncourt, F., Bui, T., Rossi, R.A., Yoon, S., and Schütze, H. (2025). "NoLiMa: Long-Context Evaluation Beyond Literal Matching." International Conference on Machine Learning (ICML).

Hong, K., Troynikov, A., and Huber, J. (2025). "Context Rot: How Increasing Input Tokens Impacts LLM Performance." Technical Report, Chroma. https://research.trychroma.com/context-rot

"Intelligence Degradation in Long-Context LLMs: Critical Threshold Determination via Natural Length Distribution Analysis." arXiv:2601.15300, January 2026.

"How Long Context Models Actually Work Under the Hood." Let's Data Science, March 2026. https://letsdatascience.com/blog/long-context-models-working-with-1m-token-windows

"Unlocking the Effective Context Length: Benchmarking the Granite-3.1-8B Model." Red Hat Blog, April 2026. https://www.redhat.com/en/blog/unlocking-effective-context-length-benchmarking-granite-31-8b-model

Links

• Generic answerer prompt generator

• 100k Results JSON

• 1M Results JSON

• 10M Results JSON

• Hindsight benchmark runner