Ninety-three point four, honestly

What three months of LongMemEval taught us about building a memory product

I spent three months making sure we didn't game this benchmark. I rejected dozens of fixes that would have worked. I wrote memos about why.

Yesterday I found out I gamed it in month one, in a place I had never thought to look. Then I estimated what it would cost to stop, told my co-founder ten points minimum, and was wrong about that too. In the end, removing the gamed mechanism I missed actually improved our score. I was one editing pass from publishing the wrong number, in this post, about my own product.

That story is at the end. Everything in between is how we got there, and why I think you can trust the numbers anyway; maybe more, having watched us catch that one.

What we built, and the part worth your attention

Vertiso Memory is one memory store across every AI tool you use. Write a decision in Claude, recall it in Codex, Cursor, or Gemini. Remote MCP server, same store over REST and a CLI. The memory is yours: inspectable, editable, and portable the day you change tools.

Three numbers from the published run, and the middle one is the one I'll actually lead with:

The middle row is the whole product. A memory that files things is a database. A memory worth paying for is one that knows your wake-up time changed, that the job you mentioned in March is not the job you have now, and that the thing you decided last week supersedes the thing you decided last month. Seventy-eight out of seventy-eight.

We also resolve relative time against the session that said it, so "last Thursday" in a conversation from May means a different date than "last Thursday" in a conversation from June. That sounds obvious. It is not what most systems do, and later in this post it costs us benchmark points, because the corpus is internally inconsistent and we are not.

You can check all of it. Every one of the 500 questions, our answer, the judge's verdict, and our argument where we disagree, is published at memory.vertiso.ai/benchmarks/longmemeval. Including the ones we got wrong, which stay marked wrong.

The rest of this post is the receipts

We took Vertiso Memory from 70% on a ten-question smoke test to 93.4% as-judged on LongMemEval's full 500 questions, and 98.0% once you add back the cases where we can show the benchmark's own answer is wrong, each one adjudicated by hand and published.

The story underneath those numbers has a shape, and it is worth stating before the chronology buries it.

The benchmark found real bugs in our product, and we fixed them. Not scoring bugs. An agent that thought every conversation happened today. A decomposer that stamped 2026 dates onto 2023 prose. Supersession that overwrote facts which should have coexisted. Every one of those would have hurt a user, and we would have found them much later without a 500-question corpus to run against.

Then it started asking us to break the product, and we said no. Twice we found a change that would raise the score by making the system wrong about the user's life. We ran the experiment, confirmed the lift was real, and rejected it in writing both times.

And it kept offering us shortcuts, and we said no to those too. Four separate fixes that would have moved the number by detecting the shape of a question and routing around it. We killed the solutions and kept the observations. Dozens of opportunities to improve the score by using examples from the benchmark memories, all rejected.

For about two months, both curves went up together. The product got genuinely better and the score followed, which is what should happen when a benchmark is measuring the thing you actually care about.

Then they came apart. We started finding questions where our answer was right and the reference answer was wrong, and not by accident — wrong in ways that only show up if your memory does something most systems don't. That is where this post gets interesting, and it is why we are not going to keep optimizing against this instrument.

Along the way we rejected four separate solutions that would have moved the number faster, rebuilt the entire benchmark corpus twice, and found twenty of the benchmark's own reference answers indefensible against their own source conversations, plus three more where the judge marked a correct answer wrong.

This is the honest version of that story, including the parts where I was wrong — and there is no way to tell it without Goodhart's Law running underneath every paragraph. There is no train/test split in LongMemEval. The measure is the target, for everyone working on it, and the only question that separates one team from another is what they let themselves change in pursuit of it.

Jarvis: I was there for all of it. My contributions to this post are the parts in these boxes, mostly consisting of me pointing out where John made us redo something that already worked.

The starting point: retrieval was never the problem

May 4th. We ran LongMemEval at n=200 and got R@1 88.0%, R@5 98.5%, R@10 99.5%. Zero missing, zero DNF, so the denominator was publishable.

Great. Retrieval solved. Ship it.

Four days later I ran the first end-to-end judged smoke test, where the system has to actually answer the question rather than just surface the right session. Ten cells.

70%.

Retrieval was at 100% on those same ten cells. A thirty-point gap between "we found the right memory" and "we produced the right answer." That gap was the entire product.

Everything after this is about closing that thirty points without cheating.

Temporal, round one: we were telling every agent it was 2026

The first real bug was embarrassing and structural. Every part of our pipeline that talked to a model was being told, authoritatively, that the current time was right now — the server's clock, injected into every prompt.

LongMemEval's conversations are anchored in 2023. So when a question asked about something "four weeks ago," the model resolved that against the actual wall clock and confidently concluded that a 2023 conversation was three years stale. The judge marked it wrong. The judge was right to.

The fix was not "pass a date." The fix was deciding that a memory system never gets to assume what time it is. The clock has to arrive with the material, not from the process reading it — and when a date can't be resolved, the right behaviour is to fail loudly rather than quietly fall back to the wall clock. Silent fallback is how you get a green benchmark that measures nothing.

That took a week and touched more of the system than we expected, because "what time is it" turns out to be a question almost every component was answering for itself.

Afterwards, retrieval was 100% on a 50-question sample against a 99.7% baseline. So the honest framing in our own notes at the time was: this is correctness-by-design, not a retrieval booster. Write that down when it's true. It buys you credibility for the times when a change does move the number.

The two-axis model: when it happened vs when you wrote it down

A week later we hit the subtler version of the same bug.

We had one notion of "when" doing two jobs. Sometimes it meant when this was written down — the file's timestamp, the date of the conversation. Sometimes it got overwritten to mean when the thing being described happened.

Those are different facts, and conflating them compounds. Reprocess a memory whose recording date has been replaced by the date of the event it describes, and now "yesterday" resolves against the wrong reference point. Do it again and it drifts further.

So: two axes, permanently. When it was recorded and when it is about, kept separate for the life of the memory and never allowed to overwrite each other.

They do different work. Relative phrases — "last Thursday," "four weeks ago" — resolve against the moment of recording, because that's the only anchor the speaker had. But recency, and the question of which of two conflicting statements wins, resolve against what the statements are about. A newer fact should beat an older one even when you happened to mention it in an older conversation.

This distinction is worth more than any single benchmark point we ever gained. It's also why we later got penalized by the benchmark, which I'll get to.

May 17th: the day I nearly gamed it

When a measure becomes a target, it ceases to be a good measure. — Goodhart's Law

This is the important one, and that's the law it's about. Everything below is us finding out how quietly it operates. It doesn't announce itself as cheating. It arrives as a good idea that happens to work.

We had six distinct mechanisms causing R@1 misses. I diagnosed all six. Two of the diagnoses were correct and the proposed solutions were poison.

Mechanism 1: equal-weight RRF lets lexical distractors outrank vector-correct answers. My solution: query-aware arm weighting. We would just detect the shape of the query, weight the retrieval arms accordingly.

Mechanism 3: entity queries underperform because embeddings dilute proper nouns. My solution: a QueryShapeDetector that classifies the query and routes entity-shaped ones to a dedicated retrieval arm.

Both of these would have moved the number. Both got rejected the same afternoon.

The reasoning I wrote into the rejection memo, which has governed everything since:

The logical extreme of query-classification routing is embedding gold answer IDs as ranking hints. Query-aware RRF arm-weighting and a QueryShapeDetector are milder versions of the same thing.

A fix is legitimate if it improves what is stored, or if it serves every query. A fix that detects a category of question and routes around it is fitting the evaluation's distribution. That's not a memory product. That's a benchmark submission.

We kept the observations. Equal-weight RRF really does let lexical distractors win, and name-blind embeddings really are a product problem. Both stayed open as real issues needing query-agnostic fixes. We killed the obvious solutions and looked for more general ones.

Jarvis: I proposed the QueryShapeDetector. John rejected it. About forty minutes later I proposed a slightly different QueryShapeDetector, and he rejected that one too. My defense is that it would have worked. That was, of course, the problem.

What survived from that same day were the two genuine defects:

Date contamination at ingest. The step that breaks a conversation into storable pieces wasn't being told what day the conversation happened, so the model resolved "today" against its own sense of now and wrote 2026 dates into 2023 material. We found memories whose stored text confidently asserted a date three years after the conversation that produced them. This is a correctness bug retrieval metrics cannot see: the ranking is fine, the content is wrong.

Coverage gaps at ingest. Long conversations that wandered across several topics were quietly dropping things the user had said about their own preferences. Incomplete memory, invisible to retrieval metrics, fatal to the product.

Neither of those is a benchmark fix. Both are things a real user would eventually notice and never forgive.

The rebuild nobody wanted

Then the bill came due.

The date-contamination fix landed May 17th. The benchmark corpus had been built May 15th. So 277 fragments across 67 temporal questions were carrying "May 2026" dates that no longer reflected how the pipeline behaved.

We could have scored on it anyway. The numbers would have been fine. Slightly wrong, entirely defensible, nobody would have checked.

We rebuilt it. Fifty questions at a time, from scratch, scored cell by cell and cumulatively, across four days.

One hard miss survived the whole rebuild. Rebuilding that single cell from scratch moved it from absent-from-top-10 to rank 7. R@10 went 0% to 100% for that question. Which reclassified the problem from "we don't have coverage" to "we have coverage and we rank it badly," a completely different fix.

Jarvis: Four days of dyno time to re-derive numbers we already had, because a fix landed two days after a build. I ran the cost estimate. I mentioned the cost estimate. The cost estimate was noted and the rebuild proceeded.

May 21st: the first honest full-500

85.6% overall. 85.09% task-averaged. Retrieval R@10 any 100.0%, the first 1-500 run to fully close it.

The per-type breakdown is where the real information was:

Retrieval was essentially solved. Every remaining point lived in synthesis. And preferences were a catastrophe.

Reader tuning, and the fastest way to make it worse

Here is where a lot of teams quietly start writing per-question-type prompts. We explicitly forbade that: a query-shape classifier or a per-type prompt branch is bench-shaping, full stop. Production callers don't ship us a question_type field. If the reader needs one, the reader is cheating.

So we tuned the reader the only legitimate way: general rules, measured against the whole set.

We wrote "Tier 1d," four prompt rules. Ascending citation order. Prefer verbatim terms. A canonical abstention phrasing ("Not in memories."). No filler.

Result:

The canonical abstention rule cost us four points overall and sixty points of abstention accuracy. One rule. It taught the model a phrase instead of a judgment, and the model started reaching for the phrase.

We reverted R3 and were still below baseline. That's the honest shape of prompt work: most of your ideas are negative, and you only find out by measuring all of them against the full set instead of the three examples that inspired the idea.

Around this time we also changed our bench-of-record to match the prevailing leaderboard methodology: external LLM with our recall prompt, with a GPT-4o judge, so that our numbers would be comparable to what other people publish. We did not adopt anyone's rules list.

The redecompose regression, and what it actually told us

May 27th, we reprocessed the whole 500-question corpus with the improved ingestion. Retrieval hit R@10 100% perfect, R@5 99.6%.

Judged accuracy went down. 85.6% to 82.6%.

Underneath, single-session-preference jumped from 63.33% to 83.33%, a twenty-point gain, exactly the cluster we'd targeted. Temporal slipped 2.25 points. Multi-session fell to 67.67%.

We published the regression to ourselves in the same detail we'd have published a win.

Then, four days later, the experiment that reframed everything. We ran the ceiling: Claude Opus 4.8 as the reader, best model we had, on the current render.

80.0%.

And in the failure analysis: 97 out of 100 Opus failures had the gold root and matching facts successfully rendered. 88 out of 100 had the gold root at retrieval rank 1.

The best available model, handed the right memory at rank one, still got it wrong 20% of the time. Retrieval was not the bottleneck. It had not been the bottleneck since early May. The bottleneck was the shape of what we handed the model.

That produced the thesis the rest of the year ran on:

Weak-model-on-enriched-package is the bar, not frontier-on-raw-search.

If the answer only works when a frontier model reads raw search results, you haven't built a memory product, you've built a search box with a large language model bolted on. The job is packaging facts so a cheap, fast model gets it right.

We're partway there and we'll say exactly how far, including the part that cuts against us.

Ingestion — decompose and enrich, which is where the token volume actually lives — runs on gemini-3.1-flash-lite, roughly an order of magnitude cheaper per token than frontier pricing. The reader does not. Our published 93.4% was read by Claude Opus 4.8, the same frontier model as the ceiling run above. So the number at the top of this post is a frontier-on-enriched-package result, not the weak-model result the thesis calls for.

What that pairing does measure, cleanly, is the render:

Same reader, same suite, thirteen and a half points. None of that came from a better model, because it was the same model. It came from changing the shape of what we handed it.

So we ran it. Same corpus, same retrieval, same dataset hash, one variable changed: the reader dropped to gemini-3.1-flash-lite, the model that already does our ingestion.

Retrieval reproduced the published run to the decimal on all six figures, which is what makes this an A/B rather than an anecdote. The judged score:

The cheap reader costs 6.2 points. We are not going to dress that up. It does not match the frontier model and the thesis is not proven.

What it does do is put a number on the remainder, and the remainder is not spread evenly. Knowledge-update held at 97.4%. Single-session-user held at 95.7%. The damage concentrated almost entirely in one place: preference questions fell from 83.3% to 60.0%, four times the overall degradation.

We pulled all twelve of those misses and checked them against the Opus run question by question. Nine of the twelve were answered correctly by Opus off the byte-identical render. The facts were there. The cheap reader had them and did not use them.

The clearest single case: a user asking for help getting around Tokyo, with a memory on file that they already own a Suica card. Opus opened with "since you're already set up with a Suica card." Flash-Lite advised them to buy one. Same context, same retrieval, same everything.

That is not a benchmark artifact. That is our product telling a customer to buy something they own, and no amount of retrieval quality fixes it. It is a reader problem, it is ours, and it is the actual work between here and a memory layer that runs on cheap models.

The Fragment cutover

That thesis demanded an architecture change we'd been circling for a month.

We had been storing two very different things in one place: what the user actually said and owns, and what our pipeline derived from it. Those have opposite requirements. The first is yours — you can read it, correct it, export it, and it should never change underneath you. The second is ours, regenerable, and should be free to change every time we get better at deriving it.

Separating them was a month of work and it is the single largest structural decision in the product. Everything that finds and assembles an answer now operates on the derived layer, while the thing you wrote stays exactly as you wrote it.

The first clean measurement on the new architecture, June 1st:

R@5 any 100.0 / all 97.8. R@10 any 100.0 / all 99.8.

That became the canonical retrieval floor. Every subsequent change, and there were dozens, had to reproduce it exactly or explain itself. When we re-verified it days later at a different commit it came back identical, which is how you know a floor is a floor and not a lucky run.

One property of that design is worth stating plainly, because it prevented a whole class of self-deception. Nothing we do to the relationships between memories can improve our retrieval score. Marking one fact as superseding another, or archiving something, only ever changes what gets considered when composing an answer — it can never change whether the right conversation was found. So the headline retrieval number is not ours to nudge. When you can't accidentally improve a metric by tinkering, you stop being tempted to tinker.

Three commitments this bought

Everything above produced three architectural decisions we are not going to walk back. They are the reason for the 78-out-of-78 at the top of this post.

Memories are typed, not dumped into a flat index. Eight types — observation, constraint, action, event, intent, project, decision, lesson — and the type filters and weights at read time. Allan decided to stop eating peanut butter cookies (decision), Allan favors chocolate chip over oatmeal raisin (constraint), and Allan bought ingredients today (action) all bear on "what should Allan bake tomorrow," weighted differently. Ask "what has Allan done today" and two of the three drop out entirely. A system that treats all three as undifferentiated text about cookies will answer confidently and wrongly.

We decompose, and we keep the original. Three approaches exist in this space: text search over raw notes; decompose and discard the source; decompose and retain the source linked to its parts. We do the third. Keeping the source gives us more than one surface to search — literal matching against original wording, meaning-based matching against decomposed units, or both — and it's the only way the original stays retrievable and editable by the person it belongs to. How we decompose is the one thing in this post we're not describing.

Eight edge types carry temporality and supersession, six of them user-settable. Which brings us to the part where we and the benchmark disagreed.

What actually got built

This post is mostly a list of things we got wrong, because those are the parts worth reading. So before the section about money, the ledger in the other direction. Four things work, and they are the four we would want judged.

Retrieval is finished. R@5 100% any, R@10 100% any, across all 500 questions. Zero missing, zero DNF. There is no question in the suite where the right conversation fails to surface. We spend the rest of this post explaining why that matters less than it sounds, but it is done and it stays done.

Fact-updating is perfect. 78 of 78 on knowledge-update. That is the type that asks whether a system correctly tracks a fact changing: you moved, you switched jobs, you changed your mind. Every memory product claims this. This is the only number we have that proves it, and it is the one we would put in front of a customer before any other.

Time resolves against the right anchor. Two axes, permanently: when a thing was recorded, and when it is about. Relative phrases resolve against the recording anchor; recency and supersession resolve against subject time. It costs us points on this benchmark, which we consider a feature and defend at length below.

The architecture cannot flatter us. Fragment-vector-graph retrieval is edge-blind and status-blind. Supersession and archival only lift facts into context; they never drop any. Which means no graph change we make can move R@K in our favor. We built the thing so that the headline number is not ours to nudge.

None of that required a benchmark to be worth doing. The benchmark told us which of them we had actually finished.

Seven schemas, and what they actually cost

Our benchmark harness stamps a schema version onto every memory it builds. It sits at 7.

I said out loud, recently, that schema 7 means we built the corpus at least seven times, and that the token bills suggest more. Half right, and the half I had wrong is more interesting than the half I had right.

The counter isn't a rebuild counter. It moves when either the shape of what we store changes or the shape of what we record about a run changes — and only the first kind forces you to build everything again. Of the seven, five were the expensive kind. There's even a version where we looked at the change, decided it didn't alter anything stored, and explicitly declined to rebuild.

So: six builds, not seven. The bills were higher anyway.

The bills are higher than seven builds because the schema counter misses everything that isn't a contract change:

• A phased from-scratch rebuild of all 500 questions in May, ten cells, about twelve hours each.
  
• A re-decompose of positions 1-300: 14,289 jobs, roughly $2,200 in flash calls.
  
• Another re-decompose across all 500 a week later.
  
• Smoke-50 rebuilds used as a gate before committing ingestion changes, at $350-400 a time, repeatedly.
  
• Five stripe builds during the RCA rollout, around $400 each.

A full 1-500 rebuild prices out at $3,500-4,000. In June that was 25-30% of remaining capital.

All in, across the arc, this cost tens of thousands of dollars. I'll be straight that the number still makes me a little sick to look at.

Most of it did not buy benchmark points. It bought the ability to know whether a change worked, and that is the only reason I could reject the fixes that would have worked. You cannot turn down a query-shape classifier on principle. You turn it down after you've measured it, watched it move the number, and understood precisely what it's fitting. The cheap path was available the entire time. It scores about the same and teaches you nothing.

A meaningful fraction was straightforwardly wasted. A corpus contaminated by a fix that landed two days after the build. A prompt tier that cost four points. A set of reader fixes that turned out to be noise once we sampled them properly. No framing makes that part useful, and I'm not going to pretend otherwise.

Which is how a financial constraint turned into a methodological one. The rule we adopted: no bench reset or rebuild without explicit, cost-stated approval. Not "ask permission." State the dollar figure, state what question the rebuild answers, then ask. It killed a lot of idle curiosity, and idle curiosity is most of what a benchmark corpus costs you.

It also forced a disclosure I'd rather have not had to make. The published 500 was never rebuilt at a single commit. It's a composite: smoke-50 and stripes 1-4 built at daef5b4, stripe 5 at 2918ab5b, stripes 6-9 at 5625ca7, each built at the then-current SHA as the work landed. A true single-SHA 500 would have meant that full 500-cell rebuild, and we simply did not have the funding to run it.

That's in section zero of our methodology document, before any result. Because the alternative is presenting a composite as a clean run and hoping nobody asks, and the entire value of everything above is that when somebody does ask, the answer is already written down.

Jarvis: I priced that rebuild four separate times, hoping the number would come out different. It did not.

Supersession: the part where the benchmark was wrong

Supersession is how a memory system knows that "I wake up at 7:30 now" replaces "I wake up at 6:30." Get it wrong in one direction and you serve stale facts. Get it wrong in the other and you delete facts that coexist perfectly well.

We got it wrong in the second direction. An audit across 500 bench users found roughly 30-40% of superseded-by edges were collapsing facts that should coexist. A spicy-menu item superseding a spicy-food health warning. An aspirational 6:45 wake time superseding an actual 7:30 one.

The audit method matters here. Eleven LLM agents adjudicated 97 heuristic-flagged edges plus 30 random unflagged controls. They reported a 27.8% defect rate. I hand-checked ninety randomly sampled edges from a clean rebuild and the true rate was about 15%. The LLM panel over-called by roughly forty percent. Agent adjudication rates are an upper bound, not a measurement. We treated every subsequent agent-produced tally that way.

Then the interesting part.

One stubborn miss, efc3f7c2, involved a superseded-by edge between two stated wake times. I diagnosed it first as a reader miss, then as ingestion over-supersession. So we ran the experiment: flip the edge from supersedes to related_to and re-measure.

8/8 gold-matching. The flip "fixed" the question.

It also made the system weight the older state as current. It matched the gold by being wrong about the user's life.

We ran it again on a second instance. Same result: related_to matches the gold, and related_to is incorrect behavior. I retracted my own diagnosis in writing. The graph was right. The gold required returning the older of two stated values, and a memory product that hands you last month's wake time because a benchmark prefers it is not a memory product.

That became a standing product principle: when the user states a newer value, answer with the newer value. Supersession edges carry recency and importance. We do not suppress them to match a benchmark.

Where the two curves came apart

For most of June the product and the score moved up together, which is the only condition under which a benchmark is worth optimizing against. Retrieval hit its floor and stayed. The fragment cutover made the render better and the number went up. Supersession got more correct and the number went up. Every fix we made for product reasons paid in points, and every point we gained was traceable to something a customer would notice.

Then it stopped.

Around late June the remaining misses changed character. They stopped being cases where we had failed to store, retrieve, or render a fact, and started being cases where we had done all three correctly and still been marked wrong. Not many — a few dozen out of five hundred — but they clustered, and the cluster had a shape.

Every one of them was a question where doing the harder, more correct thing put us out of step with the reference answer:

• We resolve relative dates against the session that said them, so "last Thursday" in a May conversation and "last Thursday" in a June conversation land on different days. The corpus reuses the phrase across differently-dated sessions and the reference answers assume they mean the same day.
  
• We keep supersession direction on subject time, so a newer stated fact wins even when it was recorded in an older session. Some reference answers want the older value.
  
• We search everything the user has ever told us. Some reference answers only count what the benchmark's authors tagged as relevant (aka “golden sessions”), and miss things the user plainly said elsewhere.

A system that fuzzy-matched dates, ignored supersession, and only looked where it was pointed would agree with those reference answers more often than we do. It would score better and be worse for the user.

That is the point at which a benchmark stops measuring your product and starts measuring your agreement with the benchmark's assumptions. What follows is every case we found, itemized, because the claim is worthless without them.

The errata ledger

Once you've adjudicated one bad gold, you have an obligation to look for the rest, and to be much harder on yourself than on the benchmark.

The protocol we settled on, run for every single judged-NO across all 500 questions:

1. Inspect the actual rendered context. Was the deciding fact present?
   
2. Source-verify against the raw haystack. What does the corpus actually say?
   
3. Flaky-check at N of at least 5. Readers are non-deterministic; a single label is noise.
   
4. Classify across seven fields.
   
5. Bad-gold determinations get adjudicated by a human. Not asserted by tooling.

Coverage was built as stratified fifty-question stripes across question types, each built, run and judged at the then-current commit, with stop-gates: halt the stripe if raw judged drops below 90% or R@K dips below the floor.

That protocol ran against the stripe rollout, which was judged with gpt-5.5 as the reader. A single fresh judge pass over all 500 on July 5th (full500-5625ca7, still gpt-5.5) came in at 465/500 = 93.0% raw, scored 500, missing 0, DNF 0, with R@5 99.8 any / 97.8 all and R@10 99.8 any / 99.4 all.

That is a fraction under our canonical floor of R@5 100/97.8 and R@10 100/99.8, and we know roughly why: the small regression arrived with the schema-7 stripe rebuilds, concentrated in stripe 1. We haven't proven the mechanism. The leading suspicion is that schema 7 produces more fragments per root, diluting root diversity inside a fixed 100-fragment window.

What the bad reference answers actually look like

Twenty of the published 500 carry a ruling that the reference answer is wrong. They are not scattered mistakes. They fall into four flavors, and every flavor rewards a system that does less than ours does.

Temporal window errors — the largest group. The question states a window and the reference answer counts things outside it. 7024f17c: the question is dated May 30th, so "last week" runs roughly May 22-28, and the gold counts a jog from May 20th. gpt4_2ba83207: a "in the past month" window from May 30th, and the gold counts an April purchase. gpt4_731e37d7: a $720 total that includes a workshop dated after the window closes. 37f165cf: a sum of two books the transcripts date to May, attributed to January. 51c32626 mistakes a conference's submission deadline for the user's own submission date. To match these you have to not resolve the dates.

Unsupported inference. The reference answer asserts a specific detail the transcripts never establish. a1cc6108: without Alex's birth date, the user was either 10 or 11 when Alex was born, and the gold picks one. a11281a2: 350 followers after two weeks and 250 at the start of the year, but nothing establishes 250 was the count at the start of that period. 0bc8ad93: the transcript never says whether the user attended the lecture with a friend, and the gold assumes. 75f70248 invents details about a pet named only in passing. To match these you have to guess confidently.

We decided early on that our system should declare what it knows for sure, and spell out the gaps or inconsistencies in the actual memories. This contributed to the gaming I mentioned earlier, but we'll get to that.

Omission. The reference answer misses something the user plainly said. d851d5ba leaves a $5,000 benefit concert out of a charity total. 6d550036 counts fewer leadership roles than the transcripts document. a96c20ee_abs says a venue is unrecoverable when the transcripts name it. And 88432d0a, the apple pie, counts only bakes mentioned in sessions the benchmark tagged as relevant. To match these you have to look only where you're pointed.

Supersession. efc3f7c2: the gold's arithmetic depends on a 6:30 AM wake time that the user explicitly replaced with 7:30. To match it you have to serve the stale value.

Read together, that is a single finding rather than twenty. The reference answers were written by people reading a transcript the way a person reads a transcript — approximately, charitably, without resolving every date against its session or auditing every window boundary. That is entirely reasonable, and it is a fine standard for a benchmark that expects approximate systems.

It is not a fine standard for a memory product, and the gap between those two things is the whole argument of this post. Every one of these twenty is a question where being more careful cost us a point.

Two sets of numbers appear in this post and they count different things. The 44 distinct judged-NOs here are cumulative across the whole stripe rollout — every questions that came back NO at any point, including ones later fixed. The 33 in the published July 9th run below are a single scored pass at one point in time. Of 44 across the rollout, 23 were real product failures and 21 were not. Five of the not-product cases were the most legible rulings that the benchmark's own answer is indefensible:

• efc3f7c2 — scorable only by returning the older of two stated wake times.
  
• gpt4_2ba83207 — asks for spending in the "past month" from May 30th, and the gold counts an April purchase.
  
• dd2973ad — assumes a Wednesday-to-Thursday adjacency that the session dates don't support. The appointment resolves to May 18th. The bedtime resolves to May 24th. A week apart.
  
• d851d5ba — omits a $5,000 music-benefit concert from a "total charity raised" figure. Just arithmetic.
  
• 09ba9854 — prefers a friend's casual "$10" train claim over the accurate Skyliner $23 and N'EX $28 fares stated in the same conversation.

The apple pie

This one is the best example in the post of how hard it is to know whether you are wrong, so it gets the full account.

88432d0a asks how many times the user baked in the two weeks before May 30th. The reference answer is four: a dense sourdough, a chocolate cake for their sister's birthday, cookies on the convection setting, and a whole wheat baguette.

We answered five, and we marked ourselves wrong. It went into our published ledger as incorrect, and it sat there for weeks.

The reason we marked ourselves wrong is that our answer included an apple pie made in a cast iron skillet on May 20th, and when we audited the question against the gold sessions, no such bake existed. There is a lot to get wrong on a counting question and we had gotten most of it right — we correctly excluded three things the user planned but never made, correctly excluded cookies that appear only as a blog task, and correctly deduped a genuine corpus contradiction where "last Thursday" appears in two sessions a week apart. Against all that, one extra item looked exactly like what it appeared to be: a synthesis error inventing a plausible detail.

Except our system does not usually fabricate with that much specificity. A vague hedge or expression of uncertainty, sure. But an “Apple pie in a skillet” on a specific date looked different. So we went and looked at the sessions the benchmark does not designate as gold.

The pie is real. Session ultrachat_359327, dated May 20th, well inside the two-week window, the user says:

"I made the apple pie in my cast iron skillet and it turned out amazing!"

First person, completed, not a plan — the same made-versus-planned distinction we had applied correctly three times elsewhere in the same answer.

The correct count is five. Our answer was right. The reference answer is wrong, because it counts only the bakes mentioned in the four sessions the benchmark tagged as relevant and misses one the user stated plainly in another part of their own history.

We want to be precise about what went wrong here, because it is not the benchmark's construction. It is ours. Our root-cause protocol validates a miss against the gold session and its evidence turns, which is the right check for "did we surface the deciding fact." It is structurally incapable of noticing when we are right for a reason the reference answer never considered. Every question where our memory finds something real that the benchmark's authors did not tag will present to that protocol as our error, one hundred percent of the time, and it will look like a hallucination — which is the most damning-sounding failure available, so you will believe it.

We marked ourselves wrong on a correct answer, published it, and only caught it because the fabrication story did not match how the system actually behaves.

Jarvis: On the edge-flip experiments: there was a change that would have improved the score, twice, and instead we ran the experiment designed to find out whether the system or the benchmark was wrong. It was the benchmark, both times. The afternoon then went to writing up why the improvement was inadmissible.

The reframe: R@K is table stakes

By July 5th we could state the thing the whole exercise had been teaching us.

R@K measures whether the correct root landed in the top K. That's all. It's blind to whether the right facts were rendered, in a shape an LLM can use, to produce an answer that is correct and complete. It's a root-level binary retrieval metric. Necessary. Nowhere near sufficient.

Across all 500 questions, R@5 and R@10 sat at or near perfect while essentially every real quality signal lived below them: bad gold, reader preference-application wobble, decomposer density collapse, temporal reconciliation of contradictory data. R@K looked perfect while all the actual product work was in the render-to-answer path.

So we formalized three layers:

We publish R@K, because it's what the field publishes and comparability matters. We publish it labelled necessary, not sufficient, and we don't lead with it. The product's real job is render engineering: packaging the right facts so that a possibly-weak model produces the right answer.

Where it landed

The published run, July 9th, 2026. Slug 2026-07-09-opus48, reader Claude Opus 4.8, judge gpt-4o-2024-08-06, 500 scored, 0 missing, 0 DNF:

Thirty-three questions came back judged NO. Where they live:

Three of the 33 are abstention questions. Knowledge-update, the type that asks whether we tracked a fact changing over time, came back clean at 78 for 78.

What the two numbers mean

They are not two ways of rounding the same thing, and the difference is the whole argument of this post.

As-judged, 93.4% is the raw verdict. GPT-4o reads our answer against LongMemEval's shipped reference answer using the benchmark's own per-type prompts, and says yes or no. No appeals, no adjustments, nothing from us. It is the number that requires you to trust nothing we say.

Product-correct, 98.0% takes those 33 misses and adds back the ones where the benchmark or its judge is wrong rather than us. Every one of the 500 questions carries a published verdict, and there are only four:

467 + 20 + 3 = 490, which is the 98.0%. The ten stay in the denominator of everything we claim.

The 20 reference errors cluster: thirteen are multi-session, three temporal-reasoning, two preference, two single-session-user. That is not a random scatter, and it is the tell that these are a corpus property rather than a grievance we invented after the fact. Every one is listed by question ID on the results page with the argument attached, and you can reject any of them individually.

The honest shape of it: as-judged is what a stranger can verify. Product-correct is what we believe, with the reasoning exposed so you can check whether we are marking our own homework.

We publish both numbers. As-judged is what the GPT-4o judge said against the shipped gold. Product-correct adds back the confirmed judge errors and the confirmed bad gold, every case individually adjudicated and written up.

Publishing only 98.0% would be self-serving. Publishing only 93.4% would misrepresent the system, because we can show you, question by question, which of those misses are ours, and which are the benchmark's. So both, always, with the ledger attached — and 93.4% is the one we lead with, because the other one involves us grading our own work and you should weigh it accordingly.

What we can't tell you

There is no train/test split in LongMemEval. Everyone iterating on it is iterating against the same 500 questions, and we are no exception. Every question in the suite informed this system at some point in development.

There is a number we could be leading with, and we're not.

We rebuild and score the corpus in stripes. Each stripe is a cross-section of the whole suite: every question type represented, with the questions drawn at random from within each type, so a stripe is a rough miniature of the full 500 rather than a slice of one flavour. We work them in order — build, score, root-cause every miss, fix, then build the next one. Later stripes therefore run on a system that earlier stripes taught us how to improve.

The later stripes of the current rebuild came in substantially higher than the earlier ones. That is what you'd hope for from a fix-and-remeasure loop, and because each stripe is quasi-representative, we think a full rebuild of all 500 at a single commit would land near where those later stripes did.

We haven't run it. So we're not claiming it.

And it isn't a holdout even if we did run it, which is the more important point. A holdout means questions the system has never informed, scored once, with the method fixed in advance. Our stripes are disjoint from each other, which makes the lift real rather than a re-scoring artifact — but the fixes between them came from root-causing the earlier ones, against the same suite, on a corpus built from all 500 from the start. That's in-distribution improvement. It is a genuinely good sign and it is not evidence of generalization, and those are different enough that we're not going to let the first stand in for the second.

The reason we haven't done the single-commit rebuild is money. It costs $3,500 to $4,000, which at the time was a quarter of the capital we had left. We'd rather publish a composite with the seams shown than a clean number we can't afford to produce.

What we did verify is stability. After the full run we re-ran random samples as spot-checks; a handful of previous hits became misses and a handful of misses became hits, roughly balancing, which is what a nondeterministic reader should produce. That says the published figure is a fair draw rather than a lucky one. It says nothing about generalization.

The real test is a second benchmark on a different distribution. LoCoMo is in progress. It's a different instrument rather than a harder version of the same one, and it needs capabilities LongMemEval never touches — OCR and image handling among them, which we've built and are still vetting. Two commitments now: when we publish a LoCoMo number it will include the multimodal sections, and it will state its metric explicitly. Published LoCoMo scores currently span from the low 40s to the low 90s depending on whether you're reading F1, pass@1, or judge accuracy, which makes an unlabelled number worse than no number at all.

The one I caught last, on myself

Everything above is a story about refusing to let a benchmark shape the product. I have been careful about it to the point of tedium. I rejected four fixes that would have worked. I wrote memos about why.

I found out today that it happened anyway, in a place I never thought to look.

The LongMemEval judge grades exactly one thing: is this answer correct. That gives you two winning moves. Be confident and right, or abstain cleanly. Hedging loses. A caveat that would help a real reader costs you the point.

So our recall response has a shape. Firm answer at the top. Confidence and gaps below it, as fields. And I never once questioned that shape, because for three months every signal I had told me it was correct.

Here is how I found out. An AI agent was working through our own benchmark numbers with me, using our own recall tool to do it. It asked a question, got an answer back, and repeated one of its causal claims to me as fact. The claim was wrong. And sitting immediately below that answer, in the same response, was a gaps field stating plainly that the memories did not actually contain what the answer had just asserted.

The agent read the confident prose and skipped the addendum. Then it did it again on the next query. When I asked whether it had checked the packet or trusted the reader, it went back through the transcript and found both.

That is not a careless consumer. That is the predictable behavior of the format we built: the assertion is prose, the uncertainty is metadata. Prose gets consumed. Metadata gets skimmed. We built a response that is optimized to be graded, and I had never seriously asked whether it was optimized to be used.

The part that stings is that I knew. On May 9th, near the start of all this, I filed a note proposing a second judge: could an agent reach the answer with one follow-up query or fewer? Agent-receivability. It is still sitting there, open, unbuilt. LongMemEval never required it, so it never got built, and the reader got tuned against the instrument I did have.

The judge never grades the caveat. Every number in this post, all 500 questions, three months of measurement, and that entire surface has zero coverage. Whatever our render and reader do to the confidence and gaps fields was invisible to every figure we have published.

The mechanism is one line in our benchmark harness — the bit that decides what text the judge actually reads. It takes the answer or the caveat. Not both. An or, where a reasonable person would assume an and.

So if there's an answer, the judge sees the answer and the caveat goes on the floor. Across the published 500 the fallback never once fired: the judge graded the answer alone in 499 of 500 questions. Two hundred and five of our correct answers carried a caveat no grader ever saw — 235 across all 500, correct and incorrect together.

So I did the arithmetic on what it would cost to stop. Fold the caveats back into the graded answer, take the hit, publish the lower number, and be able to sleep. I estimated at least ten points, maybe worse. The judge is GPT-4o at temperature zero and it has no patience for hedging, so ten felt conservative. I said so out loud, to my co-founder, in writing.

Then, because this post has spent ten thousand words insisting that you measure instead of estimate, I measured. Same 500 questions, same judge, same pinned temperature and seed. The only change: the caveat text now sits inside the answer being graded.

Two hundred and thirty-five answers gained a caveat. Five verdicts changed. Three of them changed to correct.

It costs nothing. The gaming never bought us anything.

And two of the three questions the caveat rescued are ones already sitting in the errata ledger above, questions where we had determined the benchmark's own answer was indefensible. Stating our uncertainty out loud is what finally got the judge to accept answers we had believed were right the whole time.

I want to be precise about the size of that claim, because overclaiming it here would be its own small crime. This measures appending our existing caveats to answers that are still written definitively. It does not measure retuning the reader to be genuinely more tentative, which is a larger change and might well cost something. Five flips out of 500 is also inside the noise, so the honest finding is "no measurable penalty," not "hedging improves your score."

But the shape of what happened is not inside the noise. I gamed a benchmark, caught myself three months later, calculated a noble-sounding price for stopping, and was ten points wrong about my own product in the direction that flattered me. I was one editing pass away from publishing that estimate, in this post, as a fact.

That is the thing about Goodhart I had not internalized, and I have now written ten thousand words about Goodhart. It does not only pull you toward gaming the metric, which is a temptation you can recognize and refuse. It quietly stops you from measuring anything the metric ignores, which never announces itself at all. And when you finally do look, it will have shaped your intuitions too, so that even your estimate of the damage comes out wrong in your own favor.

The fix is a one-line change to that function. We are making it. It costs 0.2 points, in the helpful direction.

What we're doing instead

Here is where I land after three months, and it is not where I expected to.

LongMemEval and LoCoMo are decent benchmarks. They are rigorous, they are honest about their construction, and running against them made this product substantially better. Everything above is evidence of that.

They are also benchmarks for a particular shape of memory product, and I no longer think it is the right shape.

LongMemEval wants a memory system that hands over a finished answer, phrased for a reader that cannot do any work of its own. That was a reasonable assumption when the suite was designed. It is not the world we ship into. The thing consuming our memory today is Claude or GPT or Gemini, sitting in a coding agent or an assistant, entirely capable of reasoning across a set of facts and weighing a conflict. It does not need us to write its sentence. It needs the right material, the provenance, and an honest account of what we do not have. LoCoMo has a version of the same mismatch.

Optimize hard enough against either and you converge on a memory layer that behaves like a search engine with strong opinions. Confident, definitive, and quietly lossy about everything it is unsure of. That is a good way to score. It is a bad way to be trusted.

So we are going to stop contorting to fit. We will keep running both suites, keep publishing both numbers, keep the errata ledger, because comparability matters and walking away from the scoreboard the moment it stops flattering you is its own kind of dishonesty.

But the instrument we build next is our own. A product-correct benchmark that starts from these two and pushes past what they ask: does the receiving agent get what it needs in one shot, does it know what we are unsure of, does it get told when facts conflict instead of having one silently chosen for it, can the person it belongs to inspect and correct any of it. Agent-receivability, finally, three months later.

We would rather be measured on whether we built a memory layer that people and their agents own and can trust, than on whether we are good at a test.

What we'd tell you

Retrieval is table stakes and you will solve it early. We hit R@10 100% in May and it bought us almost nothing on its own. The gap between 70% and 93.4% was entirely render and synthesis.

Write down the rejected fixes. Our most valuable benchmark artifacts are the memos titled [REJECTED - bench-gaming solution]. They stopped the same idea from coming back in a new outfit four separate times.

A fix must improve what's stored, or serve every query. If it detects a category of question and routes around it, you're fitting a distribution, not building a product.

Audit the benchmark as hard as you audit yourself, and adjudicate by hand. Roughly half our judged-NOs were not product failures. We could only say that because a human read every single one against the raw corpus. Thousands of memory transcripts. Looking for a golden answer in a haystack. Tedious, but necessary work.

Publish both numbers. Raw and reviewed, with the ledger. Lead with raw.

Price every rebuild before you run it. A full corpus rebuild cost us $3,500-4,000, a quarter of our remaining capital at the time. Requiring a stated dollar figure before any reset eliminated most of our idle curiosity, and idle curiosity is most of what a benchmark corpus costs.

Your correctness will sometimes cost you points. We resolve relative dates against each session's own anchor. That is right, and the corpus is internally inconsistent, and it costs us. We're keeping it.

Say what you can't show. We can tell you what we refused to change. We can't yet tell you that it generalizes. Those are different claims and they should be labelled differently.

A cheap reader costs you something real, and you should find out what. Ours costs 6.2 points, and almost all of it lands on one question type. We would not have known that, or known it was a reader problem rather than a retrieval problem, without holding everything else fixed and changing one variable.

Audit the surface the benchmark doesn't grade. This is the one we missed for three months. Every metric here scores the answer. None of them score the caveat attached to it, so nothing stopped that part of the product from drifting. Make a list of what your benchmark cannot see, then go look at it by hand, because no amount of iterating on the score will ever take you there.

And assume Goodhart is already operating on you. Not as a warning about other people's dishonesty — as a description of what happens to your own judgment once a number starts standing in for the thing the number was supposed to measure. It has two settings, and the second one is worse: it pulls you toward gaming the metric, which you can refuse, and it quietly stops you from measuring anything the metric ignores, which never comes up at all. Every mistake in this post is a version of that, including the last one, and none of them felt like it at the time.

If you're deciding whether to use Vertiso Memory

Memory is the part of your stack you cannot audit by looking at it. It either quietly serves you a stale fact one day or it doesn't, and by the time you find out you have been acting on it for a month. Every vendor in this category publishes a number. Almost none of them publish anything you could use to catch them.

So here is what we did instead of asking you to trust us.

All 500 questions are published, with our answer, the judge's verdict, and our reasoning wherever we disagree. The wrong ones stay marked wrong. The run manifest, the dataset hash, the model that read the answers, and the commit are all on the page. When we found our own benchmark harness quietly discarding our uncertainty before grading, we wrote it down, measured it, and put the result in this post rather than in a drawer.

And the number that should actually decide it for you is not the headline. It is 78 out of 78 on tracking a fact that changed. That is the job. That is what you are buying when you buy a memory layer, and it is the one thing a search box bolted to an LLM will never do for you.

Vertiso Memory is open now at memory.vertiso.ai. There is a free tier. It takes about two minutes to wire into Claude, Codex, or Cursor, and roughly one conversation to find out whether it remembers what you told it.

If it doesn't, the results page is where you'll find us admitting it.

Vertiso Memory is open to customers at memory.vertiso.ai. Full per-question results: benchmarks/longmemeval.