The Scorer Is Not the System: Three Separable Ways Offline Evaluation and a Deployed Scoring Path Can Disagree
Built on SenpaiScope. The paper separates three things that offline recommender evaluation usually conflates — the answer key, the metric, and the scorer — and shows each one independently changing which system wins.
The answer key
80% of the catalogue's mass has zero inclusion propensity, so inverse-propensity scoring is undefined over most of the thing being measured — not noisy, undefined.
The metric
Swapping random candidates for popularity-matched ones, and then sweeping how tight that match is, showed a popularity prior's entire benefit was residual confound (p = 6 × 10⁻³⁴). The prior was deleted from the live system.
The scorer
The same 1,190 probes pick e5-large as the winner on the research scorer (+0.0062) and as the loser on the production one (−0.0019). The offline winner did not transfer.
Retracting its own result
Section 8.4 re-runs the whole comparison against a co-consumption answer key and withdraws one of the paper's earlier conclusions.
Preprint — independent research. Code, data and every table are public.
Written to be checked. All 23 tables regenerate byte-identical from persisted JSON, so no number in the paper was ever typed by hand; the Zenodo deposit was verified file-by-file against the stamped commit rather than assumed to match it; and the cohort it draws on is pseudonymised to row indices, with a flag that restricts the released superset back to the exact one the paper used. Two rounds of adversarial review against my own draft found that the transfer arms had been run on different probe sets — which meant the headline rested on nothing — and that got fixed before anyone else saw it.