I don't know what to do with most benchmark deltas anymore.
I used to read higher as better and plan around it. For a while that worked reasonably well. The models were weaker, the tests separated them cleanly, and the scores tracked behavior I cared about. I've now watched too many high-scoring models fall apart in an ordinary repository to trust a small leaderboard gap.
A four-point jump on SWE-bench gets treated like four points of engineering ability even though the units don't work that way.
Contamination is one reason: test cases turn up in training data, copied or paraphrased, and the model can half-remember an answer instead of solving it. A held-out variant can catch that without making every high score evidence of memorization.
The harness matters enough that scores from two labs often aren't the same test. The tasks also have to be gradeable, with a clean spec and a checkable answer, which isn't how my week is shaped.
The benchmark task
A clean spec, one correct answer, and a grader that can tell pass from fail with no human in the loop.
Selected for gradeability
The task that eats my week
The spec is wrong, the repository disagrees with itself, and nobody agrees what "done" even means.
No oracle
The scores aren't useless: near zero is still bad, and large gaps that keep showing up across independent tests probably mean something. Good benchmark designers already fight contamination and harness drift.
I use the score to pick what to try, then run the model on my own ugly cases and see where it fails.
Where this stands
A score can get a model onto my test list without telling me how it will hold up in my repository.
- Benchmarks with held-out, rotating test sets that I can't contaminate.
- A standard harness reported next to every score, so two labs are running the same test.
- Tasks that include the ambiguous, no-oracle problems that make real work hard.