Leaderboard scores don’t hold up when we look more closely at benchmarks that are considered standards. The models dramatically improved on the exact tasks we trained them on, yet those gains often did not show up on other benchmarks or on slightly different tasks in the same codebase. This blog post is about our research on that meaning gap between what coding benchmarks measure and what we wish they measured – and how to improve these benchmarks.
