主题
Chapter 17 — Benchmark Controversy and Evaluation Traps
In March 2026 a team building a CRM assistant switched its memory system from A to B. The decision rested on a public comparison: B scored 12 points higher than A on a long-horizon dialogue benchmark. In the first month after the switch, the sales team reported that "the assistant cannot remember the budget definition the customer changed last week", and complaints doubled compared with before the switch. The team re-tested with 80 real customer sessions they sampled themselves: A scored 63, B scored 58 — the ranking had reversed.
Neither result was fabricated. The problem was that B's high score in the public comparison came from single-session retrieval, whereas the real business needed cross-session alignment of multiple revisions to the same customer. The evaluation answered a different question from the business one. The score was valid inside the benchmark and invalid outside its boundary. And the invalidity comes without an error message, which is what makes it hardest to notice.
This chapter deals with that boundary. The previous chapter addressed "whether a score within a single benchmark can be trusted"; this chapter addresses "why two scores that are both correct still cannot be compared". These two things are often conflated, with the consequence that reproducibility is mistaken for comparability: however rigorous a benchmark is, it holds only under the conditions it defines.
The boundary is drawn by five variables: dataset version, judge implementation, prompt template, number of retrieval rounds, and whether multi-hop is permitted. Any one difference is enough to produce a gap of a dozen points, and that gap is usually larger than the real capability difference between systems. This means that in most cases, cross-vendor score comparisons measure condition differences, not capability differences.
This chapter first unpacks the five variables (17.1), then explains the difference in nature between the two main benchmarks (17.2), then converts that difference into a selection question a reader can use directly (17.3). 17.4 handles mutual accusations within the industry — the method there is to give only decision rules and not to restate unverified numbers. 17.5 lists five concrete traps, and 17.6 gives a minimum viable scheme for enterprise-built evaluation.
17.1 Why no two vendors' scores are directly comparable
17.1.1 The concrete impact of the five variables
Dataset version. The same benchmark may add or remove questions, revise labels, and adjust difficulty distribution between versions. When the version number is not stated, two scores may have been produced on two different question sets. More insidious is the silent revision: upstream fixes a few labelling errors, and the resulting score change gets attributed to system improvement.
Judge implementation. For the question "the correct answer is Berlin", strict matching requires the output to contain "Berlin", while lenient semantic judging accepts "the user now lives in the capital of Germany". The same predictions can differ by a dozen percentage points under the two judges. The judge's own version drifts too — swap the underlying model and the judging standard changes.
Prompt template. How retrieved memories are assembled into the prompt directly determines whether the system can use them. The same batch of retrieval results, assembled as "flattened in reverse chronological order" versus "grouped by relevance with sources annotated", yields different answer accuracy. The template is usually not published, and it belongs to the system implementation, not to the benchmark.
Number of retrieval rounds. Allowing one round versus allowing iterative multi-round retrieval are two different difficulty classes. Multi-round retrieval lets the system first locate with a broad query and then narrow, which helps substantially in long-context scenarios. If one side runs one round and the other runs three, the scores are incomparable.
Whether multi-hop is permitted. For questions that require reasoning across three memory chains, a system with graph traversal has a structural advantage. That is not cheating, but it measures a different capability.
| Variable | Typical difference in setting | Typical impact on score | Usually published |
|---|---|---|---|
| Dataset version | Before or after revision; subset versus full | A few to a dozen points | Partially |
| Judge implementation | Strict matching / lenient semantic | A dozen points | Rarely |
| Prompt template | Flattened / grouped with sources | A few points | Very rarely |
| Retrieval rounds | Single round / iterative multi-round | A dozen points | Rarely |
| Multi-hop permission | Forbidden / graph traversal allowed | Depends on question type | Rarely |
Table 17-1 — The five uncontrollable variables in cross-system comparison
17.1.2 The compound effect of variables stacking
The five variables do not stack independently. Relaxing the judge amplifies the benefit of additional retrieval rounds — multi-round retrieval readily recalls a batch of marginally related content; under strict matching that content does not count as a hit, while under lenient judging part of it does. When two variables change together, the gap is not additive but multiplicative.
This compound effect explains a recurring phenomenon: the same pair of systems can be ranked in completely opposite orders in two comparison documents, and each document honestly describes its conditions. The difference is not in honesty but in how the variables were combined.
A team working on industrial spare-parts retrieval ran a cross-validation: they tested the candidate systems under "strict matching + single round" and under "lenient judging + three rounds", and the rankings swapped. Their conclusion was not "one side cheated" but "without locking the configuration, no ranking holds".
17.1.3 Why a unified harness still cannot solve cross-benchmark comparison
One might reason: since there are so many variables, just run all systems through the same harness. That reasoning is only half right. A unified harness solves comparability within one benchmark; it cannot solve comparison across benchmarks, because the benchmarks themselves are defined differently — different question sets, different capability dimensions, different passing standards.
A team doing industrial equipment maintenance tried this route: they connected all three candidate systems to the same self-built question set, and comparability did improve. But they then found that their question set covered only the "look up historical work orders" class of task, whereas 40% of queries in production were "what is the current status of this device" — questions requiring temporal reasoning, of which the set contained none. The unified harness gave them a precise ranking on the wrong dimension.
This is where the cost shows: building a unified harness takes real effort (writing adapters, labelling ground truth, maintaining the question set), and what it guarantees is internal consistency, not dimensional coverage. A precise ranking with incomplete coverage is more dangerous than a rough ranking with complete coverage, because it looks more trustworthy.
17.2 The difference in nature between LoCoMo and LongMemEval
17.2.1 LoCoMo: multi-session dialogue memory
LoCoMo provides a dialogue history spanning several sessions and asks questions about facts within it. Its core concern is "can you find the right information in a long conversation", and its evaluation focus is recall and cross-session association.
This class of benchmark is close to the real shape of conversational products: the user reveals information gradually over multiple exchanges, and the system must associate fragments scattered along the timeline. Its strength is testing the localization ability of retrieval within a long history; its weakness is that it does not distinguish "found" from "correctly used" — recalling the correct fragment is usually enough to score.
17.2.2 LongMemEval: a composite benchmark over five abilities
LongMemEval (ICLR 2025, arXiv:2410.10813) is a composite benchmark of 500 questions evaluating five abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge update, and abstention.
Five abilities in parallel means it measures not only "can it find it" but also "does it know when to say it does not know" and "does it know which piece of information is newer". This structure contrasts with LoCoMo's single focus: a high score on LongMemEval requires all five directions simultaneously, and any one weakness drags the total down. Conversely, a high LongMemEval score does not guarantee good performance in LoCoMo-style scenarios, because the dialogue length distributions differ.
More importantly, the LongMemEval paper reports a phenomenon: commercial assistants and long-context models show roughly a 30% accuracy drop in sustained interaction. That number is not a defect of any particular system; it is a shared characteristic of this class of architecture over long-period use.
| Dimension | LoCoMo | LongMemEval |
|---|---|---|
| Question count | This book did not verify the exact count | 500 questions |
| Capability coverage | Cross-session dialogue memory | Information extraction, multi-session reasoning, temporal reasoning, knowledge update, abstention |
| Main failure mode | Recall drift in long dialogue | Strong on single points but uneven overall |
| Applicable judgement | Baseline recall for conversational products | Overall capability for long-horizon assistants |
Table 17-2 — The difference in nature between the two benchmarks
17.2.3 What the 30% drop means
The 30% drop deserves a separate reading, because it is the most frequently cited and least frequently understood number in this field.
It does not mean "the system becomes 30% worse after 30 turns". It means that when the same question is asked across sustained interaction, the measured accuracy drops by roughly 30 percentage points compared with short-conversation conditions. The mechanism has several possible sources: accumulated noise diluting the relevant memory; contradictory facts entering the store; the retrieval budget being spread over more candidates; and genuine user preference drift that the system has not yet absorbed.
The practical implication is blunt: a system that scores 95% on 5-turn conversations may score 65% on 50-turn conversations. The first question in selection should therefore be "how many turns did your benchmark run", not "what is your score".
17.2.4 The correct way to read industry controversy
When evaluation is conducted by parties with commercial interests and there is no neutral arbiter, any party's reproduction conclusion should be treated as a claim rather than a fact. This book's approach is not to restate any unverified accusation or its numbers — the specific figures differ across accounts, and restating them would give unverified data a second life.
Three decision rules are more useful than any particular dispute:
Rule one: a claim is not a fact. A party that both publishes a score and benefits from it is a stakeholder. That does not make the score wrong; it makes independent verification necessary.
Rule two: methodology outranks conclusion. A published harness can be checked and refuted. A published conclusion can only be believed or disbelieved.
Rule three: a list of condition differences is more useful than a verdict. Knowing that one side used strict matching and single-round retrieval, and the other lenient judging and three rounds, tells you more than knowing who "won".
17.3 Converting the difference into a selection question
17.3.1 The question to ask
Instead of "what is your score", ask: under what conditions was this score produced, and do those conditions match my production path?
Concretely, four sub-questions:
- How many turns did the dialogue history span, and how does that compare with my typical session length?
- How many retrieval rounds were permitted, and how many does my production path allow?
- Was multi-hop traversal permitted, and does my architecture support it?
- Which judge, and how strict is it relative to the accuracy my business requires?
17.3.2 A worked comparison
Suppose a candidate reports 94 on a long-horizon benchmark, and the disclosure states: 50-turn histories, three retrieval rounds, multi-hop permitted, lenient semantic judge.
Your production path: 10-turn histories, one retrieval round, no multi-hop, strict accuracy requirements.
The right conclusion is not "this system will score 94 for me" but "this system scored 94 under conditions substantially more generous than mine, so the number is an upper bound whose distance from my reality is unknown". The honest next step is a small self-built evaluation on your own traces, not a leap of faith.
17.3.3 Why the self-built evaluation is not optional
A self-built evaluation of 100 questions drawn from your own traces, labelled by hand, with a fixed judge version, will tell you more about production behaviour than any public benchmark. It costs perhaps two person-weeks and it removes the largest source of selection error.
The reason teams skip it is that it produces an unglamorous number that cannot be compared with anyone else's. That is precisely its value: it is comparable with your own future self, which is the only comparison that matters for an operating system.
17.4 Five concrete traps
Trap one: the question set leaks into training or into memory. If the evaluation questions were seen during training or stored as memories, the score measures memorization, not retrieval. The symptom is a suspiciously uniform high score across question categories.
Trap two: the judge shares provenance with the system under test. When the judging model is the same model the system uses, systematic biases align. The symptom is that scores rise together whenever the base model is upgraded.
Trap three: measuring recall without measuring abstention. A system that confidently answers every question scores well on recall-only metrics and badly in production, because a wrong memory is more damaging than no memory. LongMemEval includes abstention for exactly this reason.
Trap four: ignoring recency, letting stale facts interfere. If the evaluation does not include knowledge-update questions, a system that returns the oldest matching fact looks as good as one that returns the newest. In production the two are not remotely equivalent.
Trap five: hiding long-tail collapse behind an average. An average of 85 can conceal a category at 40. Always read the per-category breakdown, and always ask for the worst category rather than the best.
| Trap | What it inflates | How to detect it |
|---|---|---|
| Question-set leakage | Memorization, not retrieval | Uniformly high scores across categories |
| Judge shares provenance | Systematic bias | Scores move together with base-model upgrades |
| No abstention testing | Over-answering | High recall, high wrong-answer rate in production |
| No recency testing | Stale-fact returns | Old fact returned when a newer one exists |
| Average hides long tail | Overall score | Per-category breakdown shows a collapsed category |
Table 17-3 — Five evaluation traps and how to detect them
17.5 A minimum viable scheme for enterprise-built evaluation
Four steps.
Step one: sample from your own traces. Take 100 real tasks spanning at least three task types. Do not synthesize them; synthetic questions encode your assumptions about what matters, which is exactly what you are trying to test.
Step two: label ground truth by hand. For each task, write down what a correct answer looks like and which memories are relevant. This is the expensive step and the one that cannot be skipped.
Step three: fix the judge version. Pin the judging model, the prompt, and the matching rule. Record them in the repository alongside the results. Any change to the judge invalidates comparison with previous runs.
Step four: run A/B and record the diff. Compare candidates under identical conditions, and store the machine-readable output so future runs can be diffed against it.
The scheme is deliberately minimal. Its purpose is not to produce a publishable benchmark but to give the team a number that is comparable with itself over time — which is the only property an operating system actually needs.
17.6 Chapter Summary
This chapter establishes three judgments.
First, most cross-vendor score comparisons measure condition differences rather than capability differences. Five variables — dataset version, judge, prompt template, retrieval rounds, multi-hop permission — each capable of producing a double-digit gap, and the gaps compound multiplicatively rather than additively.
Second, turn count matters more than score. The roughly 30% drop that LongMemEval reports in sustained interaction means that a high score on short conversations can systematically collapse over long-period use. The first selection question should be "how many turns did your benchmark run", not "what is your score".
Third, when both sides are stakeholders and no neutral arbiter exists, any reproduction conclusion is a claim. Reproducible methodology is more valuable than a reproduced conclusion, and a list of condition differences is more useful than a verdict.
This chapter deliberately does not restate any specific accusation or its numbers. Not because the dispute is unimportant, but because repeating unverified figures would give them a second life — and the discipline of not repeating them is itself the point.