T-bench: an X-ray for episodic memory

T-bench is meant to be pointed at your own context/memory system to show where it forgets, confuses a source, or loses the thread — not to rank who wins. We built it because the benchmarks we relied on could not see these failures, and we are still early in understanding them ourselves. Everything here is reproducible. Fork it and run it on your system.

1 · What it measures

Most long-context benchmarks score whether a fact is retrieved. T-bench scores how an answer is reconstructed — the episodic properties: where a fact came from, when it was said, whether it was later corrected, and whether the system honestly knows what it does and does not know.

We use Endel Tulving's episodic-memory framework as inspiration, mapped to functional, measurable probes — not as a claim that any system "satisfies Tulving's criteria." T-bench does not measure autonoesis. That boundary is part of the design, not a footnote.

2 · How it is built

Pre-registered. Every run's hypothesis and pass/fail threshold are written before the run, in an append-only registry. Thresholds are never moved after seeing results, and negative results are published. Deterministic scoring from the generating graph — reproducibility does not depend on a judge model (an LLM judge is used for one free-text sub-probe only). Synthetic, graph-based generation makes contamination controllable: a public split plus a private held-out split with a hidden seed, regenerable on demand. v1.1 is 976 probes over a ~1.15M-token haystack, three size regimes (S ≈ 45 turns / M ≈ 200 / L ≈ 1,450), answered by three models.

Arms. raw (full context, control) · oracle (only the evidence turns — an upper bound separating model error from architecture error) · naive RAG (policy-free embedding top-k, an independent baseline) · a blind-summarization baseline (LexRank, no query awareness) · retrieval (query-aware, TUL 2.0) · retrieval + role/anchor (TUL 2.1).

Context structure (production). In production, Compresh structures the prompt as a stable, cacheable prefix (system prompt + append-stable compressed history) plus a small, query-aware suffix — the turn's retrieved evidence and the recent raw turns. The bulk stays cache-warm while the dense, relevant evidence sits at the end. Note: T-bench does not exercise prefix caching — each probe is an independent request with no repeated prefix — so this is an architectural property of the product, not a number measured on the bench.

3 · Main grid

Strong answerer (gpt-5-mini), v1.1, 976 probes. "ctx" = mean tokens fed per probe; "saving" relative to raw.

The two baselines, read together. naive RAG (policy-free embedding top-k) and blind summarization are the obvious cheap options, and both fall short. At the same tiny budget as retrieval (270 vs 275 tokens), naive RAG scores 0.900 against 0.988 — and it trails even raw (0.955) at ~1% of the tokens, because top-k cosine misses the relational and temporal evidence turns (F3 0.667, F4 0.792). Blind summarization keeps quality but grows the context ~1.2× (the −19% "saving"): LexRank earns its savings by stripping modality cruft — code, terminal dumps, tool output, boilerplate — and summarizing long turns, none of which T-bench's clean short-turn prose contains, while the episodic role/turn tags it adds are pure overhead. On this workload the token win comes only from query-aware retrieval, not from naive retrieval or blind compression. (In production a blind rewrite also breaks the cacheable prefix described above — another reason its real cost exceeds the token count.)

raw quality falls with context length while retrieval stays flat:

4 · The finding, in four parts

The headline is not "we win." It is that the value of context architecture is inversely proportional to the strength of the answering model.

The pattern holds across three answerer models — the weaker the model, the larger retrieval's gain:

* Qwen raw: the long-regime history exceeds the model's 32,768-token context — the provider rejects the request; retrieval (~280 tokens) fits, so on the cheapest models retrieval is what makes the task runnable at all. tulngin = tulbase + TUL 2.0 (live; 2.1 on the way).

5 · Where we were wrong — on the record

A pre-registered hypothesis failed. H1 predicted role/anchor (TUL 2.1) would beat plain retrieval by ≥ +5pp on source attribution (F1). Measured: +0.9pp. Rejected. The honest verdict: role/anchor rendering is a net-positive, zero-cost change — decisive on weak models, marginal on strong ones — but not the across-the-board win we hypothesized. A trap we added, "relayed then claimed as one's own," actually regressed under role labels, which is why the net is small. That points at the next real problem (epistemic propagation), not a marketing line.

The benchmark caught our own bugs. A \btomorrow\b pattern in the 2.1 anchor matched inside "day after tomorrow" and mis-anchored 32/32 hard probes — an incorrect anchor, worse than none. We also caught two silent fallbacks (a summarizer and an embedder degrading to a no-op) that had quietly invalidated three earlier runs. Each is logged with its correction. An instrument that never catches its builder is not an instrument.

Production = what we measured. The live production code, run as its own arm, scores 0.921 and reproduces the retrieval thesis; its lower calibration score (F5 = 0.800) exposed a real confabulation issue we are now tracking.

6 · Run it on your own system

git clone https://github.com/compresh/tbench
cd tbench/runner
pip install -r requirements.txt
python run_tbench.py --data ../data/v1.1 --arm your_system --out results/you.jsonl

Scoring is deterministic, so your numbers are reproducible. Share what you find in Discussions, or open a pull request adding your .jsonl to results/ with full provenance. This is collaborative, not competitive — we want to see where your system's X-ray differs from ours.

T-bench is a living benchmark. We keep adding tests, models, and harder probes over time — and we are glad to publish results, ours and the community's. If you run it and find something we missed, send it: that is the instrument working.

7 · Honest limits

T-bench measures functional analogs, not autonoesis. v1.1 still ceilings on two families (F2, F3) for the strong model — they are on the v1.2 hardening list. The noise band is ≈ ±0.4pp (±1 probe), so sub-1pp differences are not interpreted. At launch the grid includes one independent baseline we run ourselves (naive embedding RAG); reflection approaches and the long tail of memory systems are added over time and by contribution — the harness is public and deterministic, so others run their own systems rather than trusting us to configure them.

Built by Compresh. T-bench v1.1 · figures from run tbench-008fix (gpt-5-mini), tbench-011/011b (weak models), tbench-010 (production). Dataset released as measured (CC BY-SA 4.0, with attribution — see ATTRIBUTION.md in the repo). We are early in this area too — if T-bench tells you something we got wrong, that is the instrument working.