Benchmark-based AI capability metrics overstate real-world autonomous performance because automated scoring excludes documentation, maintainability, and production-readiness requirements

METR evaluated Claude 3.7 Sonnet on 18 open-source software tasks using both algorithmic scoring (test pass/fail) and holistic human expert review. The model achieved a 38% success rate on automated test scoring, but human experts found 0% of the passing submissions were production-ready ('none of them are mergeable as-is'). Every passing-test run had testing coverage deficiencies (100%), 75% had documentation gaps, 75% had linting/formatting problems, and 25% had residual functionality gaps. Fixing agent PRs to production-ready required an average of 42 minutes of additional human work—roughly one-third of the original 1.3-hour human task time. METR explicitly states: 'Algorithmic scoring may overestimate AI agent real-world performance because benchmarks don't capture non-verifiable objectives like documentation quality and code maintainability—work humans must ultimately complete.' This creates a systematic measurement gap where capability metrics based on automated scoring (including METR's own time horizon estimates) may significantly overstate practical autonomous capability. The finding is particularly significant because it comes from METR itself—the primary organization measuring AI capability trajectories for dangerous autonomy.