Beyond the resolve rate
Imagine plugging two LLMs from different frontier labs into the same coding agent and finding that they solve exactly the same number of benchmark tasks. If the evaluation stopped there, you might conclude that the models are interchangeable and simply choose the cheaper o…
