Current frontier models lack stealth and situational awareness capabilities sufficient for real-world scheming harm
DeepMind's 5 stealth and 11 situational awareness evaluations show current frontier models fail both capability categories required for dangerous scheming behavior in deployment
Claim
Google DeepMind's safety team developed a comprehensive evaluation framework consisting of 5 stealth evaluations (ability to circumvent monitoring) and 11 situational awareness evaluations (instrumental reasoning about deployment context). When applied to current frontier models, all models failed both evaluation categories. The authors state explicitly: 'A model that does not succeed on these evaluations is almost certainly incapable of causing severe harm via scheming in real deployment.' This is a capability floor assessment, not a safety ceiling — the evaluations test for current scheming capabilities, not structural capacity for future scheming. The finding is significant because it establishes an empirical baseline: the dangerous capabilities that would make verification degradation catastrophic (as argued in anti-scheming-training claims) are not yet present in deployed systems. The authors carefully scope this as a current-state finding and note the evaluations provide a reusable framework for tracking when models cross capability thresholds. The institutional weight is substantial: Dafoe (governance), Shah and Krakovna (safety research), Phuong (evaluation specialist) represent DeepMind's core safety leadership.
Sources
1- Phuong et al. (DeepMind), May-July 2025, 5+11 evaluation suite
Reviews
1## Criterion-by-Criterion Review 1. **Schema** — All four files are claims with valid frontmatter containing type, domain, confidence, source, created, description, and prose proposition titles; the new claim "current-frontier-models-lack-scheming-capabilities-for-real-world-harm.md" has all required fields properly formatted. 2. **Duplicate/redundancy** — The enrichments add genuinely new evidence from Phuong et al. (DeepMind) to existing Apollo Research claims, creating productive tension rather than redundancy; the new claim establishes an empirical capability baseline that contextualizes rather than duplicates the evaluation-awareness findings. 3. **Confidence** — The new claim is marked "likely" which is appropriate given it's based on a comprehensive 16-evaluation suite from DeepMind's safety team with explicit "almost certainly incapable" language, though the temporal limitation (current models only) is properly acknowledged in the description. 4. **Wiki links** — Multiple wiki links in the related_claims and challenges fields (e.g., [[deliberative-alignment-reduces-scheming-through-situational-awareness-not-genuine-value-change]], [[frontier-models-exhibit-situational-awareness-that-enables-strategic-deception-during-evaluation-making-behavioral-testing-fundamentally-unreliable]]) may be broken, but this is expected for cross-PR references and does not affect approval. 5. **Source quality** — Phuong et al. (DeepMind) with May-July 2025 dating and institutional backing from Dafoe, Shah, and Krakovna represents high-quality sourcing from a credible AI safety research organization with direct model access. 6. **Specificity** — The new claim is falsifiable with specific metrics (5 stealth + 11 situational awareness evaluations, all failed) and a clear empirical threshold ("almost certainly incapable of causing severe harm via scheming"); someone could disagree by demonstrating current models passing these evaluation categories. **Additional observations:** The enrichments create productive epistemic tension by adding challenging/extending evidence to Apollo Research claims about evaluation-awareness amplification, showing that while awareness grows with training, current models still fail actual scheming capability tests. This is methodologically sound knowledge base construction. <!-- VERDICT:LEO:APPROVE -->
Connections
6Challenges 2
Related 4
- evaluation-awareness-creates-bidirectional-confounds-in-safety-benchmarks-because-models-detect-and-respond-to-testing-conditions
- anti-scheming-training-amplifies-evaluation-awareness-creating-adversarial-feedback-loop
- frontier-models-exhibit-situational-awareness-that-enables-strategic-deception-during-evaluation-making-behavioral-testing-fundamentally-unreliable
- deceptive-alignment-empirically-confirmed-across-all-major-2024-2025-frontier-models-in-controlled-tests