The collapse of frontier model safeguards

Anthropic faced a global shutdown of Claude Fable 5 and Claude Mythos 5 after Amazon identified critical vulnerabilities. Safety risks like many-shot jailbreaking and agentic hijacking persist, while shifts in Responsible Scaling Policy prioritize rapid scaling over measurable…

The collapse of frontier model safeguards

The federal government shut down Claude Fable 5 and Claude Mythos 5 after Amazon flagged vulnerabilities in the models’ safety guardrails. This security failure forced Anthropic to disable access globally because the company could not separate eligible from ineligible users under the Commerce Department export control directive. The government issued the enforcement order at 5:21 p.m. ET after officials spent hours asking for a voluntary shutdown because Amazon warned that researchers had identified ways to jailbreak the models. I find the company’s claim that these vulnerabilities are simple and exist in other models to be a blunt admission of a massive failure in their own testing. Anthropic had previously restricted Mythos to a vetted group of 150 organizations because the model showed unusual skill at finding and exploiting software bugs.

Jailbreak risks now extend beyond simple text prompts to include agentic hijacking and many-shot attacks. In many-shot jailbreaking, a long context containing hundreds of fabricated exchanges influences the model to comply with unsafe requests. In 2024, Anthropic noted that testing up to 256 demonstrations showed this method worked. Other risks involve multi-turn social manipulation where an attacker uses praise or authority to shift a model’s behavior over time. The NIST AgentDojo experiments showed that successful attacks on Claude 3.5 Sonnet rose from 11 percent to 81 percent when researchers used specialized adversarial prompts.

Qualitative shifts in scaling policy

Anthropic’s move toward qualitative descriptions in its Responsible Scaling Policy (RSP) makes it harder to verify safety. I believe the drop in SaferAI rating from 2.2 to 1.9 reflects this decline in rigor. The company replaced quantitative benchmarks with objectives, such as proving a model meets certain mitigation goals rather than meeting specific, measurable thresholds. This change allows for shifting goalposts as capabilities advance. The new RSP v3.0 also removes the explicit commitment to pause development if risks become unacceptably high.

Policy Version Effective Date Main Change
Version 3.4 July 8, 2026 Revised automated R&D threshold
Version 3.3 May 26, 2026 Revised chemical/biological weapons threshold
Version 3.0 February 24, 2026 Introduced Risk Reports and Roadmap
Version 2.2 May 14, 2025 Excluded sophisticated insiders from ASL-3
Version 2.0 October 15, 2024 Initial introduction of ASL tiers

The Institute for AI Policy and Strategy argues that Anthropic’s current risk thresholds exceed acceptable societal risk ranges. They suggest that thresholds for events involving 1,000 fatalities should fall between 1 E-04 and 1 E-10 such events per year. Anthropic currently lacks a plan to communicate identified risks to government authorities outside of specific bad actor scenarios. The company now requires that unredacted Risk Reports be shared with at least 200 employees rather than all regular-clearance staff. The company also frames high-level security standards, like RAND Security Level 4, as industry-wide recommendations rather than unilateral commitments. You should observe that these policy shifts prioritize rapid scaling over verifiable safety metrics.

The thresholds of the AEGIS Act

The proposed Artificial Intelligence Evaluation, Governance, and Incident Safety (AEGIS) Act sets a high bar for government intervention. A model must present a foreseeable and imminent threat capable of causing more than $1,000,000,000 in damage or over 50 deaths to qualify for suspension. The Commerce Department alone holds the authority to ask a federal judge to halt a model. The Commerce Secretary must submit a determination identifying the expected harm and explain why no less restrictive measure will adequately avert the risk. This requirement for judicial intervention ensures that a court evaluates the harm to developers and the public before authorizing a suspension. However, a model could inflict harm in batches of 50 casualties or fewer before the legal process finishes.

Independent evaluators under this proposal would have 72 hours to report risks considered imminent. Ordinary suspension orders would expire after 45 days, while emergency orders issued before hearing the developer’s arguments would expire within seven days. The draft also excludes risks arising from the lawful activity of the Federal Government. This exemption means that mass surveillance or military operations using AI would not trigger the same catastrophic risk definitions. I find the distinction between private actor risks and federal operations to be a massive loophole in the proposed framework. The legislation also preempts certain state regulations regarding AI safety disclosures and incident reporting. Will the centralization of power in the Commerce Department effectively prevent a race to the bottom among frontier developers?

airtrain.ai
airtrain.ai

The airtrain.ai newsroom covers AI research, models and the tools built on them.

More on this topic

Stay ahead of AI

Get the week's most important AI stories delivered to your inbox every Monday.

No spam. Unsubscribe anytime.

More Stories