Policy on the AI Exponential

Anthropic recently released a policy document that turns the usual tech industry stance on its head. Instead of resisting regulation, the company proactively proposed a two-part framework that calls for governments to have the legal authority to block or deter high-risk AI deployments. This is driven by the exponential growth of AI capabilities—their own Claude Mythos Preview model recently discovered thousands of high-severity vulnerabilities across every major operating system and browser, a stark reminder of the dual-use potential of advanced AI.

The proposal consists of two frameworks. The first, the Advanced AI Framework, targets only the most powerful models by setting clear thresholds: models trained with more than 10²⁵ floating-point operations (FLOPs) and developed by companies with over $500 million in AI revenue or $1 billion in AI R&D. This precise targeting aims to avoid stifling innovation from smaller players. Under this framework, frontier AI developers would be required to conduct rigorous testing, publicly disclose their findings, submit to independent evaluation, and maintain robust security programs. Crucially, Anthropic argues that governments should have the power to block or deter deployments that pose catastrophic risks, with civil penalties scaled to a company’s global annual revenue and escalating for repeat violations. The framework identifies four types of catastrophic risk: biological risks (such as facilitating bioweapons development), cyber risks (exploiting vulnerabilities at scale), loss of control risks (models acting beyond human direction), and other yet unspecified categories.

The second piece, the Economic Policy Framework, addresses the impacts on workers and the economy. While less detailed in the release, its core question is how to ensure that the financial benefits of AI are broadly shared, rather than exacerbating inequality.

Anthropic is not alone in calling for regulation—OpenAI and DeepMind have done so as well—but its proposal is unusually concrete, with quantifiable thresholds and specific enforcement mechanisms. This reflects a broader trend: top AI labs are shifting from passive compliance to actively designing the regulatory rules they want to see. They understand that waiting for a catastrophe to spur reactive regulation could lead to far harsher, one-size-fits-all rules that could hamstring the entire field. By proposing thoughtful, targeted regulation now, they hope to shape a governance environment that is both protective and supportive of continued progress.

For most developers and startups, the immediate impact is negligible. The 10²⁵ FLOPs threshold is extremely high, currently reached by only a tiny handful of models. However, the long-term signal is clear: if your ambition is to train massive models, you will need to embed safety testing, transparency, and government engagement into your development process from day one. For the broader tech workforce, the economic framework is a reminder that AI-driven productivity gains will inevitably lead to job displacement and career shifts. Upskilling and public policy discussions are no longer optional.

Perhaps the most counterintuitive takeaway is that regulation can actually be an ally of innovation. By setting clear rules and thresholds, it removes uncertainty and prevents an unchecked race to the bottom. In that sense, well-designed regulation is not a barrier but a guardrail that protects long-term, responsible innovation.

Overall, Anthropic’s policy paper is both a reference answer for governments and a wake-up call for the industry. AI has entered a phase where institutional innovation is just as critical as technical breakthroughs. Striking the right balance between development and safety will define the coming decade.