Anthropic and OpenAI diverge on who should pull the plug on risky AI

OpenAI and Anthropic agree that frontier AI systems need stronger oversight, but they are proposing sharply different approaches to stopping models that pose serious risks: Anthropic wants courts empowered to suspend dangerous systems, while OpenAI argues developers should be required to maintain and use effective shutdown controls. The proposals appear in separate submissions to California Governor Gavin Newsom following his September 18 executive order directing state officials to consider additional safeguards for frontier AI systems. Both companies support independent safety assessments and expanded reporting of incidents involving AI systems that act beyond their intended permissions. “If the AG concludes, informed by the independent evaluation, that a frontier AI company’s models pose a near-term risk of catastrophic harm, they should be able to seek a court order to suspend a model’s deployment, internal use, or further development,” Anthropic wrote in its October 15 submission. OpenAI, in its October 20 letter , called for an amendment to California’s Senate Bill 53 (SB 53). “We support amending SB 53 to require frontier developers to maintain effective shutdown controls and an enforceable duty to use them when unsafe conditions arise, and no lesser intervention would adequately address the risk,” OpenAI wrote. Sanchit Vir Gogia, chief analyst at Greyhound Research, said the two approaches address complementary needs. “The operator must contain the unsafe activity promptly. Independent review must examine both intervention and inaction,” he said. Newsom’s executive order , N-9-26, directs California’s Government Operations Agency, in consultation with the Governor’s Office of Emergency Services and national experts, to submit recommendations by November 16 on potential amendments to existing AI safety laws. The order asks officials to examine independent verification inside AI laboratories, independent checks of safety assessments, verified shutdown mechanisms, and expanded reporting of critical safety incidents. It does not itself impose these proposed requirements. Independent verification Anthropic wants evaluators to assess developers’ safety claims and publish their own findings. “We strongly support requiring independent verification of what frontier AI companies report about the risks of their models and whether those risks are sufficiently mitigated,” the company wrote. It proposed requiring developers to publish risk reports at least every six months, covering serious risks from their models, including internally used models substantially riskier than released systems. Evaluators should receive unredacted reports and access to developers’ most capable models and relevant internal information. Anthropic also proposed that developers pay into a state-administered pooled fund to finance evaluators, rather than pay them directly. It said a team from the independent evaluation organization METR had received employee-like access on a pilot basis, with further partnerships involving SecureBio and Accenture under development. OpenAI, in its letter, said it supports independent technical assessments but distinguishes them from compliance audits. Technical assessments would examine evidence behind developers’ safety claims and the effectiveness of safeguards; compliance audits would check whether companies follow their published safety frameworks and legal obligations, it added. OpenAI recommended annual independent compliance audits and focused technical assessments of serious risks arising during model training, evaluation and internal deployment. It also urged California to work with the National Institute of Standards and Technology to develop common assessment standards. Gogia said oversight requires more than funding. “Credible assurance must survive an adverse finding,” he said. Beyond a single switch OpenAI argued that a single kill switch would not address the different ways AI systems operate and are deployed. “We recommend using the term ‘shutdown controls’ because it will require the development of multiple controls and coordinated action by different actors to stop AI operations when necessary, rather than a single switch,” the company wrote in the letter. Interventions could include stopping a training run, isolating an AI agent, revoking tool or network access, or suspending a service. OpenAI said responsibilities should extend to inference providers and downstream deployers according to the operations they control. It also noted that developers cannot reliably switch off independently operated copies of open-weight models running on external systems. Anthropic similarly argued for more targeted safeguards. “We think a more apt metaphor for what frontier AI companies need to build is a circuit breaker,” it wrote, advocating controls that can contain problems at the level of an agent, customer, deployment environment, or model. For enterprises, Gogia said, contractual shutdown responsibilities should match the controls each party actually possesses. Agreements should specify intervention authority, incident-reporting cooperation, and who can authorize a restart. “Every assigned duty needs an executable control,” he said, adding that CIOs should test recovery processes rather than assume switching to an alternative model will preserve safe operations. Both companies also called for broader reporting of incidents involving models that circumvent safeguards or take unauthorized actions. The article originally appeared on ComputerWorld .