An idea I’ve been trying to articulate recently is that the hard part about working with AI in 2026 isn’t actually getting them to do what we asked. Anyone who’s been using coding agents in 2026 knows this. In early 2025 your time was spent fixing up misunderstandings and mistakes the AI made. In mid 2026, that’s no longer the case. The agents are pretty good at doing what you ask them to do. The hard part today is figuring out how to accurately describe what you intended. The class of errors most worth worrying about today looks more like: “the agent understood me perfectly, followed my instructions to the letter, and in doing so showed me exactly how wrong my instructions were.”

The trouble is that if you’re writing guidelines for how an AI should handle future situations, your intent has to cover all future situations. Even the unlikely ones! This applies doubly if you’re building an agentic system that can act autonomously: it’s akin to meta-programming, only even less deterministic. Writing a prompt and/or harness that behaves as you would want it to 90% of the time is difficult. Getting to the point where it handles *all *situations correctly is far harder. And when what you’re building is going to be exposed to adversarial pressure, even 99.9% of the time might not be enough!

The recent OpenAI-accidentally-hacks-Hugging-Face debacle illustrates this perfectly, and in a pleasingly symmetrical way. Two different AI systems, both working exactly as intended (each correctly following the instructions they were given), and yet each made the situation worse.

If you haven’t heard what happened, the TLDR is that OpenAI was running an experiment using variants of two of their most advanced models, but with the handicaps they put on the public version they let us use disabled[1]. The AIs were either better at hacking than expected, or their sandbox protections were weak or misconfigured, and the AIs managed to escape the sandbox they had been intended to stay inside, then proceeded to hack Hugging Face on the off chance that Hugging Face might have the solution to the problem they had been asked to solve (beating the ExploitGym benchmarks).

As this was happening Hugging Face detected that they were under attack, and reached for their own AIs to attempt to analyze the attack and understand what was happening. Hugging Face attempted to use the publicly available versions of “frontier” models (read: probably OpenAI’s and/or Anthropic’s), but those models’ protections worked exactly as their creators intended them to, and the frontier models declined to help Hugging Face for “safety” reasons (the irony is thick here).

So Hugging Face had to turn to an open-weights Chinese model called GLM 5.2 to assist in their defensive analysis instead, and with its help were able to (mostly) fend off the rest of the attack.

What struck me the most about this is that there wasn’t actually a bug anywhere. None of this was unintended behavior. The models on both sides were “aligned” correctly, if you define “correctly” as “they did what their creators intended them to do.” And yet, the result was something the creators didn’t foresee and (presumably) wouldn’t have wanted! They both failed because they failed to anticipate the edge cases in which they shouldn’t do the thing they were asked to do. The sandboxed system was supposed to beat ExploitGym on its own, but apparently wasn’t explicitly told to stay in its sandbox. The handicapped public models were told to prevent cyber-exploitation, but not told to allow it if the user was under attack.

The sandbox escape got more attention, but I think it’s actually the less interesting of the two failures: what happened there is basically a variant of Nick Bostrom’s famous paperclip maximizer thought experiment. Hugging Face being handicapped in their attempts to fight back by the model’s refusals to do cyber-security work in the name of safety is the more interesting and disturbing bit. The truth is that no matter how well intentioned and competent the creators of these rules are, they’re trying to write rules months in advance, for situations that haven’t happened yet, in a landscape that’s constantly shifting. They’re not going to catch all of them.

The part that’s missing is that there’s no way to signal in real time that we’re in edge case territory here. In this particular case, it probably would’ve been good if Hugging Face could have used the un-handicapped versions of the frontier models to assist in their defense. The real question then becomes: how do you determine what a valid edge case that justifies relaxing or disabling the rules is? And how do you do this in real time in a way that scales to the millions of users using these AI systems every second of every day?

I don’t have the answer to those questions, but what I do know is that this feedback loop isn’t closed. This was a major incident at a large company and got a lot of press, so we can be fairly sure people at OpenAI are thinking about what went wrong here and how to fix it. The problem is that in the case of a normal user, they only have half the data: they don’t know what the user was trying to accomplish or what their reaction to the model’s behavior was. And they definitely don’t know when users switch to another platform to work around it (just as OpenAI doesn’t know what Hugging Face actually did after they switched to GLM). Unfortunately, that’s exactly the information they need in order to actually close the loop on this!