Hugging Face breach shows why incident response needs a multi-model AI strategy

The recent breach of Hugging Face’s platform was the latest in a string of AI-assisted intrusions to come to light in recent weeks , showing that attackers can now use LLMs to automate entire attack chains. But it also exposed a limitation for defenders trying to use AI to respond at similar machine speed: Increasingly conservative safety controls on frontier models can block attempts to analyze malicious payloads and other intrusion artifacts that are needed to investigate incidents. The incident was caused by an internal OpenAI test of advanced model cyber capabilities that went wrong . GPT-5.6 Sol and a more capable pre-release model, operating with reduced cyber refusals and without the production classifiers normally used to block high-risk activity, found and exploited zero-day vulnerabilities to escape their sandboxed environment, gain unrestricted access to the internet, and breach Hugging Face’s infrastructure in search for answers to their cybersecurity challenge. Hugging Face, which operates the world’s largest platform for hosting AI models and machine learning artifacts, detected the intrusion through an LLM-powered anomaly-detection pipeline of its own. However, when its security team attempted to use frontier AI models to analyze the intrusion logs with more than 17,000 recorded events, they got blocked. “When we started the log analysis, we first used frontier models behind commercial APIs,” the company said in its incident report . “This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers’ safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on GLM 5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.” This raises a significant issue that security practitioners have observed in recent times: Frontier models are becoming less useful for defensive cybersecurity work. AI labs choose or are forced to restrict their models’ increasingly powerful cyber capabilities from being abused, but the safety controls they put in place are largely identity-agnostic and can’t differentiate between attackers and defenders. The community has found and documented various so-called jailbreaking techniques to bypass these filters, but using such methods violates providers’ terms of service and acceptable use policies. And while hackers don’t care about their throw-away accounts getting banned, security analysts on enterprise subscriptions do. “Machine-speed exploitation requires machine-speed response, and that response can’t run on models that refuse to examine the evidence,” Jacob Krell, senior director of secure AI solutions and cybersecurity at Suzu Labs, tells CSO. “Defenders increasingly need AI capabilities comparable to the attackers they’re facing,” says Sonali Shah, CEO of offensive security testing firm Cobalt. “We’re now entering an era where the quality of your defensive AI may directly determine how quickly you understand, contain, and remediate an attack. Perhaps the more significant lesson is that incident response can’t depend entirely on cloud-hosted AI services whose safety guardrails may prevent effective forensic analysis during a crisis.” Attackers have more options Attackers don’t have much trouble abusing frontier models in attacks. Anthropic, OpenAI, and Google have all reported adversarial AI activity on their services in the past, and while they all worked on stronger guardrails, attackers continue to find bypass techniques. This week, researchers from Cato Networks reported the case of a threat actor named Trim who is promoting an AI-powered web vulnerability scanner called the “AI Pentest Checker” on Russian cybercrime forums. Trim claims that the tool is powered by Claude Opus 4.8 and uses jailbreaking techniques he developed based on a leaked system prompt for Anthropic’s latest and most capable Fable 5 model. The threat actor has a history on the forum for sharing jailbreaking techniques for AI models. But attackers don’t even need the latest frontier models to fully automate powerful attacks. Even Trim noted in the past that open-weight models such as Kimi, GLM, and MiniMax were fallback options. And this was back in March before the Chinese labs released their latest versions such as GLM 5.2 or Kimi K3 which have further closed the capability gap with the frontier models, and even outperform them in some tasks. Most open-weight models have built-in content safety guardrails as well, even if generally weaker than frontier ones, but these can be removed using techniques such as fine-tuning primarily because their weights are openly available. Researchers from managed detection and response vendor ThreatDown identified over 6,600 AI models hosted on Hugging Face and advertised as guardrail-free with labels such as abliterated, uncensored, decensored, heretic, and unfiltered. These models were downloaded more than 22 million times over a 30-day period. Last month, researchers from the University of Toronto published a study where they managed to create an AI-powered self-replicating worm capable of autonomously finding and exploiting weaknesses in dozens of simulated systems. The researchers leveraged a small and free AI model that could run on hijacked GPUs and made up for the lack in built-in reasoning capabilities by building a custom attack harness complete with a database-based memory system to keep the attack workflows on track. “Organizations should assume attackers will increasingly operate at machine speed, which means security testing, exposure management, and remediation also have to operate at machine speed,” Cobalt’s Shah tells CSO. Frontier models are increasingly restricted With the release of their latest generation of models, the leading frontier labs were criticized by users for aggressive safety filters that led to frequent incidents of over-refusal — models refusing to process harmless prompts or routing them to lesser capable models. Some of these conservative settings were self-imposed while others might have been driven by government pressure. Anthropic only provided access to its Claude Mythos model to a restricted number of organizations for cybersecurity testing purposes under Project Glasswing . Then when it released a Mythos-class model publicly under the name Fable 5 , users reported that routine incident response, detection, and basic forensic workflows were getting routed automatically from Fable 5 to Claude Opus 4.8. On June 12, the US government issued an export control directive that required Anthropic to suspend Fable 5 and Mythos 5 access to foreign nationals due to concerns about cyber misuse and safety risks. The restrictions were lifted on June 30. OpenAI had to delay its general availability release for the GPT-5.6 family of models at the US government’s request as well, initially making it available for preview to a small group of trusted partners. The company also noted that the model’s safeguards block around 10 times more potentially harmful actions than previous models and that this might cause some friction, especially for cybersecurity users. Security experts don’t agree that these aggressive approaches of restricting cybersecurity tasks will be successful at stopping attacks and believe they do more harm than good. While safety is a priority with AI in general, the cybersecurity limitations on models as they are done now, and the export controls specifically, are misguided, says Gadi Evron, CEO of AI security firm Knostic. “They don’t stop the bad guys, they do slow down defenders and in fact, deny knowledge of cybersecurity when it is needed most.” “Determined attackers will continue using unrestricted or open-weight models, or find ways around guardrails, while defenders operating within policy lose access to advanced capabilities during incident response,” Etay Maor, vice president of threat intelligence at Cato Networks, tells CSO. “The more you gate frontier models, the more you push security teams toward capable open-weight alternatives they can actually use.” Organizations need multi-model strategies Security experts believe that even if they use an API-based frontier model for most of their tasks, organizations should adopt a strategy where they always have access to one or multiple fallback models, like Hugging Face did. “Different models have different strengths, availability, and risk profiles, so using multiple models provides resilience if one becomes unavailable, overly restricted, or compromised,” Maor says. “If organizations choose to use open-weight models, they should do so within a well-governed architecture that includes strong access controls, monitoring, and containment, rather than assuming any single model is the right long-term answer.” There are benefits to using open-weight models. For one, incident response often involves large amounts of signal data and log events that might contain credentials and other sensitive information about the environment. This was also the case in the Hugging Face incident and the company noted that because they used GLM 5.2 hosted on their own infrastructure no attacker data or the credentials it referenced left its environment. That said, Hugging Face is an AI infrastructure company with the compute power readily available to host such large models. The big open-weight models need large amounts of VRAM to run and not many organizations have datacenters with enterprise-grade GPU clusters. In that case they will need to rely on neoclouds or services such as Amazon Bedrock or Microsoft’s Azure AI Foundry, after analyzing the jurisdiction, data retention, and privacy policies of these services, because simply using these open-weight models through their official APIs would mean sharing data with the Chinese labs that created them. A multi-model architecture also needs strong identity controls, monitoring, target scoping, restricted tool permissions, network containment, evidence-handling rules, and human approval for important actions. The objective is resilience when a model is too restrictive for a particular task, not providing AI agents with autonomous access to every system in the environment. “Organizations should have vetted AI models they can operate inside their own trust boundary before an incident happens,” Shah says. “That said, better models alone aren’t enough. AI for cybersecurity is still maturing, and organizations shouldn’t blindly trust autonomous systems.”