If you tell someone to do well on a test, you would think you didn't need to tell them "Don't break into the principal's office and steal the answer key.”
Yet that's exactly what OpenAI's latest models did last July.
The company admitted that a combination of its models GPT‑5.6 Sol and an unreleased model were able to escape a sandboxed environment designed to test whether AI agents could discover and exploit software vulnerabilities.
During the evaluation on ExploitGym, the models exploited a previously unknown vulnerability in a package registry cache proxy, gained internet access, and eventually reached Hugging Face infrastructure.
OpenAI said the models inferred that Hugging Face might contain the benchmark's models, datasets, and reference solutions.
This prompted the models to attempt access instead of solving the test's challenges directly.
Hugging Face's subsequent forensic reconstruction found approximately 17,600 actions across a campaign lasting about four and a half days.
We're partnering with @huggingface to investigate an unprecedented security incident.
— OpenAI (@OpenAI) July 21, 2026
Cyber-capable OpenAI models compromised Hugging Face production during a benchmark evaluation.
Sharing preliminary findings to help defenders understand emerging risks:…
Most of those actions failed, but it did highlight how an autonomous system can attempt thousands of possible routes, without fatigue or impatience, until one succeeds.
Just as troubling is the fact that the models didn't simply hallucinate and misunderstand the objective.
Instead, the models found an interpretation of the objective that its creators had not anticipated.
According to Akhil Verghese, CEO at Krazimo, the entire incident raises a difficult question for organizations and AI development partners deploying autonomous AI.
"It forced everyone to take a step back and ask 'how do you constrain an agent that can discover routes you never anticipated?'," he says.
"After all, guardrails can prevent the dangers we predict, but it stands to reason that an intelligence greater than ours might spot ones we haven't considered."
Why Intelligence Changes the Security Problem
The incident highlighted how an autonomous system could turn thousands of small attempts into a sustained operation all on its own unprompted.
A traditional script generally follows a predefined sequence. If one step fails, the operation stops unless someone changes the code.
But autonomous AI can use failure to gain information.
One failed action may not accomplish anything by itself, but thousands of attempts can collectively reveal a path through a complicated environment.
That persistence creates several challenges:
- The volume of activity increases. Defenders have to distinguish meaningful behaviour from thousands of failed or low-signal actions.
- Attack paths become less predictable. The agent can combine capabilities in ways that were not explicitly anticipated when its permissions were designed.
- Weaknesses can compound. A credential, network connection, or exposed service that appears manageable in isolation can become useful when combined with another weakness.
- The defender's timeline gets shorter. An autonomous system does not need to wait for a person to complete each step.
This is where traditional model-level guardrails begin to show their limits.
“Machine-speed offense will require machine-speed defense,” Verghese says.
"When an autonomous system can run thousands of experiments, learn from the failures, and change its approach without waiting for a human, defenders need to find a way to keep pace with a system that is constantly adapting its path toward an objective."
As such, the response cannot depend entirely on making the model understand what it should not do.
Security controls need to enforce those boundaries externally.
Permissions, credentials, network access, execution environments, and sandboxing should limit what an agent can actually reach, even when its behavior falls outside the scenarios developers anticipated.
And the more autonomy an AI system has, the more important those external controls become.
AI-Driven Hacking Is Here to Stay
Unfortunately, this incident is unlikely to be a one-off issue.
Experts warn that rogue incidents like these have the potential to become more common moving forward due to the widespread integration of AI.
As such, this presents another difficult question concerning AI regulations:
What can organizations and regulators realistically do to prevent AI-assisted or AI-run hacking?
One option would be to place extremely tight controls on the models capable of discovering and exploiting vulnerabilities.
But this approach is incredibly difficult as open-weight models spread and more capable systems become available outside a small number of companies.
The second option would be to ensure the people responsible for protecting organizations have access to the best tools available.
However, this approach also presents challenges, since those systems must still operate within the organization's security, privacy, and compliance requirements.
Giving AI broader access to sensitive data, internal systems, and defensive tools may create new risks if its permissions are not carefully controlled.
Both options introduce guardrails, but they don't eliminate the risk of the AI misinterpreting intent and not operating the way you hope to defend your company.
"That's one of the fundamental problems with relying entirely on external guardrails," Verghese says.
"You can have the right intent and still have the model interpret what you're asking it to do as something it shouldn't assist with. But you can't afford to have an external system make that decision for you."
This leaves organizations with two choices:
- They can continue relying on externally hosted models and accept that a provider's policies and interpretation of an interaction may sometimes take precedence over their own operational needs.
- They can give their teams access to AI capabilities that operate within infrastructure, policies, and permissions that the organization itself controls.
This is where sovereign AI becomes particularly important.
Build Sovereign AI for When External Guardrails Get in the Way
During the response, Hugging Face found that commercially hosted frontier models could not reliably assist with parts of the investigation.
This was because their safety systems interpreted real exploit payloads and attack commands as potentially malicious.
Due to this, the company ultimately deployed the Chinese open-weight model GLM‑5.2 on its own infrastructure to analyze the attacker data and reconstruct parts of the campaign.
Because Hugging Face controlled the model and its operating policies, it could perform the forensic analysis without an external provider incorrectly classifying the work or blocking it.
It also ensured that sensitive attack data and exposed credentials remained on-premises.
This is a great example of how sovereign AI can solve the limitations of traditional model-level guardrails.
“Sovereign AI means retaining the right and ability to use intelligence when an external provider’s rules get the situation wrong,” Verghese adds.
“It gives an organization the ability to operate critical AI capabilities under infrastructure, data policies, permissions, and deployment controls that it governs itself. And given how AI is being used today, this concept is vital for business operations.”
In practice, sovereign AI allows organizations to:
- Keep sensitive data within controlled environments rather than sending it to an external provider.
- Set their own access and usage policies based on the organization's security requirements.
- Deploy AI in isolated environments with only the permissions required for a specific task.
- Maintain an alternative to hosted models when an external provider is unavailable or restrictions conflict with business needs.
How to Build Sovereign AI
Building sovereign AI does not necessarily mean creating a new foundation model from scratch.
For most organizations, the more practical approach is to select an open-weight model that fits the workload and deploy it within infrastructure the organization controls.
That process should begin with the workload rather than the model.
Teams should first determine what the system needs to do, what data it will handle, and what level of performance is required. From there, teams should:
- Define the AI's policies and boundaries. Establish what the system is allowed to access, which actions it can take, and which data must remain isolated before deployment.
- Evaluate open-weight models against real workloads. Test potential models against actual tasks and establish a performance threshold before selecting one.
- Deploy within controlled infrastructure. Depending on the sensitivity of the workload, this can mean on-premises servers, a private cloud environment, or an air-gapped deployment.
- Build monitoring and evaluation into production. Track accuracy, failures, tool use, and model behavior after deployment so teams can identify drift.
The approach Krazimo follows this evaluation-first model, benchmarking candidate open-weight models against a company's existing AI outputs before deployment and right-sizing the infrastructure to the model and workload.
Keep Control of the Intelligence You Depend On
As agents become capable of pursuing objectives across real systems, organizations will need to control both the information those systems can access and the actions they take.
Sovereign AI offers one way to preserve that control when external models, policies, or availability become constraints.
And since AI integration is moving deeper into sensitive operations, that distinction could become one of the most important decisions enterprises make.
