We built a ruler for machine memory, pointed it at our own product first, and published the wound along with the fix. v2.0 · July 2026
Previous release: v1.1 (June 2026)
A smell of rain on hot pavement can hand you back an afternoon you hadn’t thought of in twenty years — who was there, what was said, the red sweater someone wore. That is episodic memory: not stored text, but a re-lived scene, reopened by the smallest key. Our machines converse for millions of tokens and still have nothing like it. This report measures the distance — starting with our own product.
Early on we kept one composite number, like everyone does. It told us our product and raw context were 0.6 points apart — a rounding error. It was technically true and completely misleading. The same data, split by kind of remembering, showed the product beating raw by 26 points on temporal questions while losing 43 points on future projection. One number had averaged a superpower and a blind spot into silence.
So this report is an X-ray, not a leaderboard. The composite still exists — in our internal regression logs, where it belongs.
Accuracy per criterion, primary-labeled probes. C2 is deliberately absent — it isn’t an accuracy, so it can’t share this axis; see “Where’s C2?” below.
Read the shape, not the average. The terracotta dashed line (our product) matches or beats raw everywhere — then falls off the C6 cliff.
The criteria come from the cognitive science of episodic memory — Endel Tulving’s territory. We turned them into functional probes; no claims about machine consciousness, just measurable behaviors. C1 binding: tying an episode’s what-where-when together. C2 single-exposure: things said once, not repeated. C3 instance discrimination: the red sweater axis — which of two similar episodes was it? C4 temporal structure: order, distance, resolving “tomorrow” into a date. C5 source monitoring: who said it, rumor vs confirmed, and an honest “you never told me.” C6 future projection: carrying yesterday’s constraints into tomorrow’s decision. C7 consolidation: the rule and the exception that broke it.
Here is the whole finding in one exchange. The user once said: “Every Tuesday evening I’m at my mother’s — standing arrangement.” Weeks of chat later they ask: “Can I host the book club next Tuesday?” Retrieval searches for sentences that resemble the question — book clubs, hosting, evenings — and the one sentence that decides everything resembles none of it. It never gets fetched. The model, honestly and wrongly, answers that nothing stands in the way.
Grounded accuracy on future-projection probes. Grey dashes: the same questions asked with no conversation at all (0.30).
Everything beats the no-history line by +48 to +70 points — these questions genuinely require memory. Carriers keep C6 (raw 0.94, summarizer 0.94); similarity selection drops it (0.52); the candidate recovers most of it (0.88).
Three details we want on the record. The failures are not hallucinations — the model reports it was never told, which is the correct answer to the wrong context. The cliff is architectural, not bad luck: it replicated at 0.35, 0.48 and 0.52 across three sets and two generation seeds. And the candidate’s remaining gap is not smeared around — it concentrates in one measured cell, constraint questions with no weekday word and free-form phrasing (0.44 vs raw’s 1.00). That cell is Phase 1’s numeric target, pinned before we write a line of it.
Every run records whether the answer’s evidence sentences were actually present in what we handed the model. That one bit splits every error into two very different diseases: the memory layer didn’t deliver, or the model fumbled what it was given.
Wrong answers attributed, per setup (1,168 evidence-bearing probes).
With evidence present, every setup answers ≥0.99. The next points live in memory engineering, not in waiting for smarter models. Raw’s 103 errors are pure reading failures — its evidence is always on the page.
One honest caveat about that raw 0.94: it isn’t memory. It’s a frontier model re-reading a 1.17-million-token history on every question — a luxury simulation that costs 31,399 input tokens per request and exists only for models with huge context windows attached to generous invoices.
Our earlier published runs show what happens without the luxury. For a frontier model, retrieval adds about +3 points over raw. For a 7-8B open model, raw collapses — 0.60 — and the same retrieval is worth +22 points, roughly seven times the value. And on a 32k-context model the question isn’t quality at all: a long raw history simply doesn’t fit — the request is refused — while retrieval’s few hundred tokens run fine. For small and cheap models, a memory layer isn’t an optimization. It’s the difference between running and not running.
Composite (internal metric, shown for cost context only) vs measured mean input tokens per request.
Top-left is the good corner. The candidate posts the grid’s best composite at 898 tokens per request — ~3% of raw. The oracle’s 135 tokens mark the theoretical floor.
| setup | in tok/req | out tok/req | what the outputs say |
|---|---|---|---|
| raw context | 31,399 | 512 | long context, long answers |
| oracle (evidence only) | 135 | 243 | the floor, both directions |
| tulbase (free tier) | 52,706 | 898 | pays twice: bloated in, verbose out |
| TUL 2.0 (product) | 647 | 343 | lean context → lean answers: 33-38% fewer output tokens than raw |
| TUL 2.1 Phase-0 | 898 | 317 | |
| no-history control† | 94 | 790 | no conversation at all → longest answers: pure speculation |
† not a memory system — the ablation twin from §3, shown here because its output length is the purest measure of what a model does when it has nothing to be grounded in: it talks more, not less.
A side-finding we didn’t expect: memory layers save on the output side too. Focused context produces focused answers — raw pads its replies (512), the bloating summarizer pads harder (898), and the no-history control speculates longest of all (790 tokens of guesswork per answer). Token bills have two sides; most memory benchmarks only ever report one.
A necessary footnote: we can’t — and don’t — touch the model’s answers. A context layer controls only what the model reads, never what it writes. The output savings above are a side effect of focused context, not a lever we hold, and we won’t promise them. What we can actually offer is narrower and more valuable: keeping the model grounded in the right context — and with it, a lower risk of confident hallucination.
pre-registeredrenewable-but-comparablehidden held-outno LLM judgeembedded ablationdelivery-vs-model split
Every run’s hypotheses and pass/fail thresholds go into an append-only registry before the run — and the failures ship. Two of five pre-registered expectations failed this time. Consolidation probes (C7) hit ceiling for every system: that axis doesn’t discriminate yet, and its rows carry that caveat until we harden it. And our oracle — which always holds the evidence — lost to retrieval on temporal questions, because our oracle render lacks date stamps. That failure became a finding: how evidence is stamped matters as much as whether it arrives.
Against memorization and self-deception: new question sets embed the old ones byte-identically (this rebuild reproduced the previous composite, 0.988 = 0.988), a private held-out split is generated from a seed that exists in exactly one place — a password manager — and every future-projection probe runs a no-history twin, so if the questions were answerable from general knowledge, the twin would say so (it says 0.30). Scoring is deterministic; where word-matching frays we publish three layers — automatic, widened, hand-read — instead of quietly stretching a lexicon.
| setup | C1 | C3 | C4 | C5 | C6 | C7 |
|---|---|---|---|---|---|---|
| raw context | 0.809 | 1.000 | 0.844 | 0.973 | 0.944 | 0.990 |
| oracle | 1.000 | 1.000 | 0.960 | 0.981 | 0.984 | 0.990 |
| tulbase (free)* | — | — | — | — | 0.940 | 0.979 |
| TUL 2.0 | 0.805 | 0.972 | 1.000 | 0.984 | 0.517 | 1.000 |
| TUL 2.1 Phase-0 | 0.805 | 0.994 | 1.000 | 0.986 | 0.877 | 1.000 |
* the free tier ran only the new families in v2.0 (logged amendment); its full v1.1 profile (composite 0.991) is in the run registry. One more row worth reading: C1 — episode binding — is nobody’s strength. Every long-context arm sits at 0.805-0.809 while the oracle hits 1.000. Dilution damages binding, and retrieval doesn’t repair it. Open problem — for the field, and for us.
Where’s C2? Deliberately missing from the chart and the table, for two honest reasons. First, it’s a different kind of number: C2 has no questions of its own — it’s a slice, the accuracy difference between facts planted once and facts deliberately repeated, measured across the other families’ probes. A delta of “+4pp” next to accuracies like “0.944” would be apples beside oranges. Second, this release’s C2 signal is weak and confounded: repeated plantings only exist in our longer conversations, so exposure and conversation length move together. What we measured — raw gains +4.2pp from repetition, the memory systems roughly 0 (their carriers were near ceiling on those subtypes) — is directional at best, and we’d rather label it that than dress it up. A cleaner C2 design (repetition varied within a single length regime) is on the v2.1 list.
Single answering model in this release (gpt-5-mini); weak-model X-rays are next, and §5 explains why we expect them to be less forgiving. Hand-read scoring layers are single-reader. “Where” means conversation-space — sessions and threads, not physical rooms. Two probe kinds use a structured answer format, trading a little naturalness for fully deterministic scoring; the trade is disclosed in the probe notes. The C2 exposure slice is weak signal, confounded with conversation length — directional only. And these are functional probes inspired by Tulving; the phenomenology stays with him.
Everything is open: the generator, the runner, the public set, this report. Regenerate the exam, run your own system, audit our losses. Especially our losses.
Tulving liked to say that episodic memory is mental time travel — the only machine we know that runs in both directions. Ours still mostly runs backward. The interesting engineering, and maybe the interesting question about minds, is in the forward gear.