Benchmarks

Latest LongMemEval run (500 questions). Retrieval is deterministic; answers generated with gpt-5.4:

We also report the stricter version, did we surface every memory a question needed, not just one:

• 97.4% found everything needed, top 10
• 92.7% found everything needed, top 5
• 0.863 MRR: on average, how close to the very top the first correct memory lands.

Retrieval uses no LLM at search time, so these numbers are deterministic: the same memory and the same question return the same results every run. They do not drift when answer models change.

Current LoCoMo result (older snapshot):

• 89.1% found everything needed, top 10
• 93.4% right memory in top 10
• 0.681 MRR

Read these together. The retrieval numbers show whether the right past sessions reached the model. The answer number shows whether the full system produced the right answer once that memory was in hand.

A memory system can be measured at three levels, easiest to hardest:

A system can clear level 1 and still fail level 3: the memory showed up, but the answer was wrong. Most tools report only the first. Total Recall reports all three.

The S-setting used here contains 500 questions across 6 question types. The answer is often not in the most recent session, and the hard questions need more than keyword matching.

Per-type retrieval:

Total Recall reaches these with deterministic local retrieval and no LLM calls at search time.

This run freezes retrieval first, then answers the 500 questions from that fixed retrieved context, then judges the answers after all 500 outputs are present. The answer model was gpt-5.4, and judging used question-type-specific LongMemEval judge prompts.

Per-type answer accuracy:

This was a full 500-question run, not a tuning split and not a partial sample.

Public comparisons are useful, but they are not controlled head-to-head experiments. Different systems use different answer models, different architectures, and sometimes different harnesses. Some publish end-to-end answer accuracy; some publish only retrieval. Read the table as landscape context, with the model shown for each result.

Sources:

• Mastra OM: Observational Memory: 95% on LongMemEval
• Hindsight: vectorize-io/hindsight-benchmarks
• Honcho: evals.honcho.dev
• Mem0: self-reported, April 2026 algorithm (arXiv 2504.19413)
• ByteRover: self-reported, LongMemEval-S (byterover.dev)
• Zep: 71.2% from the Vectorize comparison paper (arXiv 2512.12818), a competitor-authored run but the most apples-to-apples number available; Zep's own self-report is 63.8% on temporal tasks (gpt-4o)
• Public comparison snapshot accessed 2026-04-18, Honcho/Mem0/ByteRover added 2026-06-02

Retrieval-only tools that report Recall@K but no end-to-end answer accuracy are not directly comparable on this column.

LoCoMo evaluates memory over long conversations rather than many separate sessions. Evaluated on 1,986 questions across 10 conversations, categories 1-4. Pure retrieval, zero LLM calls at search time.

Per-category retrieval:

These are the widely used memory systems that publish a LoCoMo result. Published LoCoMo numbers measure different things: Total Recall reports retrieval Recall@10, while Honcho, Mem0, and Zep publish LLM-judged answer accuracy from their own runs. The two metric types are not directly comparable, so every number below is labeled by what it measures. Read this as landscape context, not one ranking.

Sources:

• Honcho: honcho.dev/evals (LLM-judged score; their own no-memory baseline, Haiku alone, scores 83.9%)
• Mem0: self-reported LLM-judged accuracy (arXiv 2504.19413)
• Zep: 58.4% from Mem0's corrected rerun in getzep/zep-papers#5; Zep's rebuttal computes ~75.1%. A vendor dispute, not a neutral authority.

The Total Recall LoCoMo number comes from an older snapshot and is the current improvement target. The theoretical ceiling on this benchmark is about 93.6% given a 6.4% label error rate in the dataset.

These benchmarks force memory systems to do real work under long-context conditions. They are far better than toy retrieval demos. They are still benchmarks, not a complete theory of product quality.

That is why this page separates retrieval quality from end-to-end answer quality, and treats LongMemEval and LoCoMo as complementary rather than interchangeable.

The retrieval numbers are deterministic. Retrieval runs in code with no LLM calls at search time, so the same memory and the same query produce the same retrieved set every run. For the locked LongMemEval run, retrieval was frozen first, answers were generated for all 500 questions, and judging was run only after the full set was complete, which keeps the retrieval result and the answer result independently interpretable. The benchmark definitions are external:

• LongMemEval
• LoCoMo