Benchmarks
LongMemEval, question by question
These are Vertiso Memory's complete results on the LongMemEval benchmark — every question, the reference answer, our answer, and an independent judge's verdict. Where our assessment differs from the judge, the rationale is published right next to it.
run 2026-07-09 · judged by gpt-4o-2024-08-06
LongMemEval (as-judged)
93.4%
As scored by gpt-4o-2024-08-06, no adjustments.
Product-correct
98.0%
Counts a benchmark-errata question only when our answer is right on the merits — never a free win. See the per-question rationales below.
Retrieval (hybrid search)
| top-K | @1 | @5 | @10 |
|---|---|---|---|
| any | 98.0 | 100.0 | 100.0 |
| all | 33.8 | 98.0 | 99.6 |
How to read these numbers
Each question is answered by Vertiso Memory from memories built out of the benchmark's conversation histories, then scored by an independent automated judge (gpt-4o-2024-08-06) against the benchmark's reference answer. The raw score applies the judge's verdicts unchanged.
The adjusted score corrects the judge in both directions when the honest reading demands it: a judged-incorrect answer becomes correct when the judge mislabeled an answer that matches the reference or when the reference itself conflicts with what the conversations actually say, and a judged-correct answer becomes incorrect when our answer does not hold up on the merits. Every reclassified question shows its rationale inline; misses that were genuinely ours stay incorrect.
The retrieval table reports how often the conversation sessions containing the answer appeared in the top-K results of hybrid search (any = at least one answer session; all = every answer session).
Loading…