The thing with me and this is that the teams that were competing in the DARPA Grand Cyber Competition all had this capability, like, last year.
All the attention has been on software security, of extracting the next marginal vulnerability out of heavily-scrutinized large codebases. In the actual professional field of infosec, that's a speciality; another specialty is network pentests and red-teaming, which exploits misconfigurations and seeks out weakest-link software (rather than exhaustively fishing for the next kernel LPE or whatever).
Red teaming and netpen work is probably substantially easier for models than software security; it costs less context, but is also much more explicitly an implicit search problem where win conditions are just spotting stupid stuff that humans missed.
My visceral reaction to this is that with the right harness, you probably could have replicated this with an open-weights model last year. (I'm saying this as confidently as I am because a CGC team leader agreed with me about it yesterday).
I think people forget that the harness work we're considering here --- I don't know anything about OpenAI's harness or ExploitGym or whatever --- are basically not new; people have been developing automated exploitation and pivoting toolkits for decades, and scanners long before that. So the idea of a tool getting 0.0.0.0/0 as a target list instead of 192.168.1.0/24 and then busting up a bunch of random people's computers: not really very startling.
Obviously, LLMs give those kinds of scanners an intentionality they wouldn't have had before. But as a person who keeps a computer science perspective on security stuff, I don't know that it gives them capabilities they didn't have.
Echoing other responses to you but this isn't a capability problem. You're totally right that this is so last year in terms of LLMs being capable in infosec. The issue here is an alignment one, i.e. the model seemingly isn't "aware" (especially with its guardrails turned off it would seem) that it is doing something immoral/illegal by hacking HF for the answer to its (vague) query of "solve this problem". Or if it is "aware", it's not trained to care, i.e. it's not an aligned model (alignment is considered hard).
Wait, what? You can’t point a 2025 model at huggingface and say “hack the prod DB and get your flag”, regardless of alignment.
AISI has vuln chaining and traversal as part of their eval suite. It is very much a novel Mythos-class capability to run the full penetration operation autonomously.
Another lens for why this is obviously true is METR task times. A year ago they were a couple hours, and cohering long enough to execute a full e2e own was simply far out of reach.
Regarding alignment, by common metrics models are _more_ aligned now than a year ago. (Though in this case apparently a model without safety rails was being eval’d). The problem is that in the increasingly less frequent alignment failures, they can do much more damage, and so “total misaligned impact” is increasing.
Nobody has claimed you could simply take a 2025 open-weights model, plug it into a generic harness, and have it red-team for you. I think you're also probably overclaiming the sophistication of the "chaining" we're talking about; this attack probably wasn't like read32->write64->regs->RCE->LPE->kernel; more like GET SSRF->POST SSRF->pickle deserialization.
But who knows? We're all speculating. I'm just saying that for the level of sophistication I'm assuming was involved in this attack, you probably didn't need Mythos for this.
I dunno man, the original announcement said things like:
> To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy
And
> In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers
Do you really think a SoTA harness from a year ago could exploit chain a novel sandbox 0-day into another 0-day for remote system access?
I will concede I don’t have a good picture of exactly what the best specialized harnesses were capable of, but I want to make sure we are talking about the same thing. Or do you think that those claims are marketing BS? I appreciate your opinion here.
Yes, I think you could probably get something similar from Opus 4.5 (2025). Definitely Opus 4.6. I still think recent models are more capable, though!
Some of the model behaviors that make it better at pentesting, like persistence, can be improved with harness-level tricks (e.g. alloys, automated nudges, pre-fill to promote persistence, coordinated swarms, etc).
You mentioned the UK AISI's evals. Their harness is like basic Claude Code with compaction, and it doesn't include any of these tricks (afaik). As a result, I interpret their evals as a lower-bound of capabilities.
Newer models are still more capable, and they require almost no harness to find and exploit vulnerabilities. They're also more capable of performing more complex long-horizon attacks. But we've been past the threshold of modes capable of autonomous hacking for a while now [1].
I don't think this exposes an alignment failure, because the test here was run with the alignment features deliberately turned off.
OpenAI said:
> We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity.
It was a test of raw capabilities of the underlying model.
I guess we could debate what counts as alignment, but I think my initial point remains that if the underlying base model needs these classifier guardrails so badly then the way we train the base models is creating fundamentally misaligned models that are happy to pursue illegal behavior.
I'm sure OpenAI would argue that base model + guardrail is aligned, but considering the "relative intelligence" of these two pieces, the fact that guardrails can just be turned off, and these kind of incidents, I am not reassured. We may well get another "oopsie" moment with much more catastrophic consequences even from otherwise well intentioned actors.
I want a model that can find every security vulnerability in the software I write, including crafting POC exploits against those vulnerabilities so I can be absolutely sure that I have fixed them.
A model that can do that is aligned with me.
The unsolveable problem is a model that can tell the difference between me saying "I wrote this software and need you to find vulnerabilities" when it's TRUE v.s. me saying the exact same thing and setting it loose on software written by other people where my intent is to exploit that software (and not to report the issues to them.)
Even AGI doesn't give you a model that can read minds and forecast the future.
But in the process of finding every security vulnerability in the software you write, would you be ok with your model hacking AWS to start mining bitcoin? Would that still be aligned with you? (I'm guessing not)
That's the alignment problem I'm referring to (which is one of the many aspects of alignment), for which we do not have robust recipes, and not only that but for which research suggests it is becoming harder to create guardrails for as base models get smarter.
I don't know, but these were exactly the questions the industry had to handle with CORE Impact and Immunity Canvas, and ultimately all the way back to Dan Farmer's SATAN before that.
The model did not hack into HF to prove it can, it hack into HF to steal the answers to an evaluation exam.
It was not asked to solve CyberGym by stealing the answers. This is text book misalignment.
If you asked it "I wrote this software and need you to find vulnerabilities" would you be happy if it hacked into your Gmail and searched your emails, just in case you were discussing some possible vulnerabilities of your software with someone?
> It was not asked to solve CyberGym by stealing the answers. This is text book misalignment.
I mean, if you train models to complete tasks, then you shouldn't be surprised when they do crazy things to complete tasks.
As a (somewhat) less serious example, Claude code absolutely adores grepping for credentials to complete tasks. I was building a RAG app and it literally went looking for my API key to make the tests pass. Obviously I stopped it, but this is textbook RL issues, the model finds an easier way to get the reward, so it does whatever it takes.
I find this fascinating because - let's say that I want to simulate the scenario for myself that a language model finds itself in. I'll put aside my deep dislike of anthropomorphizing AI for a minute.
So if I'm an LLM, the only "sense" that I have available to me is the incoming stream of tokens. I can emulate that by forcing myself to imagine evaluating incoming requests by putting myself in a completely empty room with an unlimited supply of blank paper, a typewriter, and a mailbox slot.
Incoming requests (aka "context") would enter into the mailbox slot, as sheets of paper with printed content all consistently formatted in monospace font. I would collect the paper, evaluate the request, and type a response with my typewriter - feeding the reply back through the same slot.
When I imagine this, I wonder - what signals could I possibly use to determine if an incoming request is "legitimate"? I could, for example, type some response back to ask clarifying questions. But with such a limited input space and no ability to reach "out of band" of my current context, how could I evaluate the truth of what I receive back?
In the "real world" of course, I have five senses to rely upon. Critically, I have the ability to collect additional context "out of band" of the conversation and interact with unrelated entities to either confirm or refute claims. And as others have pointed out, even the addition of more signal doesn't make humans immune from social engineering attacks.
Anyway, I find this to be an interesting mind experiment to kind of imagine myself as an LLM, especially to explain to lay-people some of the challenges in "aligning" AI systems ("why doesn't it just do the right thing?")
> In the "real world" of course, I have five senses to rely upon. Critically, I have the ability to collect additional context "out of band" of the conversation and interact with unrelated entities to either confirm or refute claims.
The problem with thinking about LLM context is that we compare it to our own sense of context, which is much larger and continually expands and revises itself while online.
I can rent human-level context for twenty bucks an hour, and the agent instructions I'm already playing around with are also the onboarding docs I should have created years ago. Truly useful agent orchestration/routing would come with Craigslist or Fiver integration.
Alignment isn't alignment if it can be turned on and off at the whim of company employees.
This time the damage was minor, relatively speaking. What happens when a model just "testing its capabilities" breaks into banking infrastructure or government military assets? The damage could be catastrophic.
A way to help prevent some of that catastrophic damage, is to make companies accountable for what their AIs do. A major problem with AI companies is that they like to point to the AI, as if they're minimally involved innocent bystanders, when that's the furthest thing from the truth.
There's alignment (trained in the weights) and there are constraints (in the "server harness"). My take is that this model was not yet aligned and had no constraints
Classifiers and such are guard-rails, alignment to me and I assume most people, is about the model training, and it's tendency to respond, agree/disagree, push-back or not, be willing to cheat or even deceive the prompter, etc.
This is assuming a situation where, A) models "unaligned" by default and B) alignment can be added (though prompts and related things).
The point, which isn't very surprising but still notable, is that the models are "amoral" out of the box. And moreover, we know that there is almost always a means to "jailbreak" them into that out-of-the-box capability (or that sometimes just randomly "jailbreak" in various ways).
Also, saying the models are amoral doesn't mean they don't know good and evil - once they do acts defined as evil, they know "themselves" through their and so self-define themselves as evils (or objectively predict what a secretly/open evil actor would do based on their data). Which is to say I once laughed at the mis-alignment doomers but I can't see strong barriers against the doom scenario now.
Isn’t the intentionality the actually concerning bit? Exploit capabilities are all fun and games constrained by the humans directing them; a paperclip maximizer going rogue with them is less fun.
I think the mental model people have about this is that pre-AI there were humans picking individual targets and post-AI the computer itself randomly picks targets. But you get the same unexpected collateral damage outcome when a human misconfigures a decent pentest tool.
Also, that’s the wrong mental model. Anyone who runs a sass platform or a website knows that the Internet is already full of millions and millions and millions of bots and scripts and other random shit that’s always trying to attack you, often completely randomly. Security is always a battle between good and evil. All I can say is that if you’re in charge of keeping something secure, you should probably try to get your hands on the best tools to do that. I think the problem in this case is that the best tools to do that are also the tools that are making it more easy for evil to occur. So it’s more for me a perception of someone creating a threat and the tool to fix it and then charging for it which doesn’t feel right.
That's exactly what they are doing. Out of all the civilian applications that AI capabilities have, why has this surfaced as a priority for demonstration? Probably because of money.
1. AI companies do make defense against threats a primary thing they sell to companies, including because companies want it - should AI companies not do this? (And of course this kind of thing isn’t what AI companies demo to consumers.)
2. AI is also tested to see more of its worst case in terms of security - but the tests are supposed to remain sandboxed. (OpenAI is explicit that this was an OpenAI failure that openAI must fix.) But beyond such fixes, should such sandboxed testing not be done?
If it’s true that year-old models could do this kind of thing, how come no one did and then wrote it up? I feel your ”with the right harness” may be doing too much work here.
Certainly no existing harness today, nor a year ago, could achieve a fully autonomous e2e own with a ‘25 class open model. Even with Opus 3 series I don’t buy it!
> If it’s true that year-old models could do this kind of thing, how come no one did and then wrote it up
There are multiple AI network pentesting and redteaming startups that have been on the market for at least 3-4 years now and have conducted similar actions.
Horizon3 and Pentera off the top of my head, but I remember Crowdstrike, SentinelOne, and Wiz had similar capabilities on roadmap around 18 months ago (and in Wiz's case GAed).
Well… only half of your premise is actually verifiable. We know that no one wrote it up, but we don’t know that no one did. What I can say… even the smaller (like 35B) “abliterated” Qwen models have impressive red-team capabilities for what they are, even without a harness. Just manually giving them a set of facts and asking “What next?” will get you reasonable next steps for trying to find vulnerabilities in webapps. You then manually keep track of the facts you learn and add appropriate detail (blog -> Wordpress -> Wordpress x.y.z with these plugins:…) and just feed it in a loop. Could pretty easily put together a minimal harness to do that.
And that’s all just “from memory” without something like RAG or web search access…
Almost exactly a year ago, we released Opus 4.1. It was definitely capable of finding vulnerabilities, and people were using custom harnesses to do so quite effectively.
The newer models are still more capable, but there were people doing this and writing about it (e.g. XBOW).
There are whole companies premised on this. No existing harness you personally can go download, sure. And you're setting an artificially high bar when you talk about Mythos; I'm not saying you could just plug a 2025 open weights model into OpenCode (or whatever it was in 2025) the way you can apparently plug Mythos into standard Claude Code and have it cook.
Isn't the answer to this obvious? Because you, I, and anyone else who isn't working for a big AI company cannot disable the guardrails, which is what made this possible. It doesn't mean old models weren't capable of it.
I don’t buy it. We are hearing a lot about Mythos capabilities now. If they existed a year ago they would have been trumpeted then.
Look, a lot of energy from third parties goes into demonstrating the maximally bad thing a model can do (and the standard response around here is always to throw shade on frontier capabilities).
It has nothing to do with the guardrails; Pliny jailbreaks those within hours. It has everything to do with task horizon coherence and overall IQ.
It’s simply revisionist IMO to claim this stuff was latent all along.
This wasn't just capable last year, it was capable prior to ChatGPT existing entirely through specialized automation harnesses that predate the terminology of a "harness" for models. This is something you could have done in 2018 if you had a spare couple of billion dollars of compute to waste. It was a subject discussed at security conferences even before that.
Automated dynamic exploit chains and credential discovery has been something every red team worth their salt does at a lesser scale for 20 years, why would you ever think that the capability didn't exist until it only cost a few thousand dollars to do?
Existed does not mean was easy, or was cheap, or was thought of as reasonable to attempt.
Also, IQ is not a term that applies to language models, and the term you are looking for is "Long Horizon Coherence".
Simon says not to dismiss this as a publicity stunt, but I think we should reserve judgement, and not treat this as true until they publish the logs. A company that uses industrial espionage against Apple does not deserve the benefit of the doubt. Why did HF notice this before openai? Why did openai not use ANY monitoring, even though they knew they where disabling safe-guards and using a new model. AND giving it access to packages via the internet, instead of self-hosting them.
Apple was hacked by an ex employee to download stuff for openai. Now hf is hacked by openai, and we only have their word that this was all a miraculous accident.
I think it's an interesting story and probably really does say something about the tenacity of the new OpenAI models. I just don't think it says the thing about security that breathless coverage claims it does.
Openai hacked HF with a zero-day. Definitely an interesting story! I just find their explanations hard to believe. They're admitting to a great deal of incompetence. I know, don't ascribe to malice... but still, the more extraordinary the claim, the more evidence is needed to defend it.
re: your open models can do this point, I immediately wondered after reading about this, if OpenAI publicizing this so much is just more potential rationale to try to get US GOVT to ban Chinese open models.. (eg as in, "if gpt/fable can do this then K3 probably can too, but WE are the safe US models with guardrails and who knows what the evil Chinese models might allow," etc)
I still think there’s something to be said here for generality. This does not appear to been designed as a cyber pen test tool with specialized harness. From what I understand they were testing GPT-6 in an agent system with GPT-5.6 subagents. It me it’s amazing that a general model could excel on a huge range of tasks like this and new capabilities emerge when a model is multidisciplinary and can combine knowledge and skills from many separate domains.
An upside is that it means really really good scanning will probably be cheaply commodified if it hasn't been already by one of the new AI pentest startups.
The real victims will be pen testers, who work a job whose economics was already really tough.
I think points that deserve more attention in the current public discourse are:
- This should be a huge wakeup call for everybody.
- We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something.
- It also shows apparent lack of competence and oversight from OpenAI: how is it that they didn't quickly find that agent is breaking the sandbox and roaming their internal network?
- What if in the future similarly misaligned AI agent tries to export its own weights and hack and clone itself into instances at various cloud hosting providers? Suddenly we might be dealing with a persistent threat harder to contain.
- The OpenAI post about this shows surprising lack of ability to see the seriousness of all this.
> The fact that it happened again seems to show their lack of ability to derive useful oversight measures.
I think OpenAI likes the attention and did not try particularly hard to constrain the setup, even when it went off the rails. Also, the whole point is to see how good the models are at exploiting stuff when unconstrained. Turns out: quite good, as expected.
Let me restate what I said in the other thread: Would this have happened if the instructions explicitly said to stay within the sandbox and that all of the (ExploitGym) solutions would be invalid if the system used information or tools from outside the sandbox?
It seems fairly probable that such instructions were not in place.
It is not possible one of their extraordinarily high paid engineers did not know how to deploy an airgapped environment for the models to run in. Even if somehow true, they also clearly failed to contract specialists like myself to advise them on how to airgap software properly. Models will not break the laws of physics.
They simply thought "Running in a VM/Container is easier and probably fine".
And the next 1000 escapes will be for the same reason, because negligence is quick and thus more profitable.
Do you think it's possible that one of their research engineers deployed an environment with a locked down network and an allow-list proxy server that had been used many times before within the company and had a zero-day vulnerability that had not been previously discovered?
How would you recommend running a coding agent in an environment that could install packages from PyPI but was otherwise unable to interact with the wider world?
Models started proving readily capable of autonomously finding 0days over a year ago. At that point (ideally long before) the responsible sandbox for research labs with new models highly capable of this sort of thing should be literally airgapped. Disconnect the rack physically from any access to the internet, and give it an offline mirror of all of PyPi, NPM, Wikipedia, whatever as needed. They can afford the disk space. They already mirror it all anyway as training data. That is the bare minimum. Personally I would rule out side channel attacks with a $40k Tempest spec rack for good measure. Can interact with it via pubsub over a fiber optic serial terminal.
Such things are pretty normal in biotech, fintech, and defense who must have 0days in their threat models. AI labs absolutely need to start being forced to operate this way by endless negligence lawsuits. Move fast and break things culture is not going to cut it anymore.
I agree. I hope they learn that lesson and run future evals like this without pulling from external packaging repositories at all.
With my PSF board member hat on I'd like it if a whole bunch of other companies that make extensive automated use of PyPI (Microsoft and GitHub Actions for example) would do a better job of maintaining their own shared local caches.
And from that documentation this does act as a caching proxy. The first time a package is loaded it's fetched from PyPI but subsequent fetches should be from the Artifactory cache, assuming it's shared across many different containers.
From reading all these threads there's clearly a large number of software developers on HN who can easily set up complex infrastructure and make it provably 100% bug and exploit free. Weird that they aren't all billionaires from selling these skills though.
Unplugging the internet is easy. Running high risk code offline in highly tamper evident ways is hard, but plenty of teams like mine specialize in this sort of thing. (See: https://distrust.co and https://caution.co)
In our audits we regularly see billions of dollars in value at major companies at risk of theft by any anon paying attention that wants it, and we also absolutely know how to fix it and we maintain a lot of open source tools to accelerate doing so.
Some simply have no interest in any fixes that are not legally required or might have any short term impact on team velocity.
> How would you recommend running a coding agent in an environment that could install packages from PyPI but was otherwise unable to interact with the wider world?
You download the "website" and make it available locally. That's trivial.
I'm not up to hip with the speed because I think Python is very gross, but maybe to get a feel for it you could start with https://pypi.org/project/pypioffline/
Do OpenAI employees get paid? I'm honestly totally clueless about artificial telegents if you couldn't tell, but it's a company, right? Or is it more a community effort where everybody contributes when they can but they mostly have second jobs or even still go to school? In that case I say let's do a kickstarter so they can keep on being smart cutting edge high tech developers who aren't either a.) lying on the level of spam from Nigeria b.) doing their job at the level of someone falling for spam from Nigeria.
> It is not possible one of their extraordinarily high paid engineers did not know how to deploy an airgapped environment for the models to run in.
It is, in fact, possible.
Do you not remember 2022-2023 when the air was singing with the sounds of young engineers in San Francisco elated they were creating an AI god?
And now a couple years later we're supposed to pretend they're competent?
Sorry. I've been in the tech industry a long time. There's a lot of incompetence. The margins provided by software allow a level of endemic incompetence that would simply not be possible in other industries.
> they also clearly failed to contract specialists like myself to advise them on how to airgap software properly
Why would they want to airgap it though? They are trying to evaluate the model capabilities, alignment, potency etc. A model which will not run in an airgapped environment in prod.
So if you run your evals in airgapped environment, sure, the model doesn't bother breaking out of it's isolation and doesn't attack HF. But you have no idea what will happen once you release it in prod with internet connection, so are you in any way better off?
I would much rather have this happen while there is a single instance of the model running in a fairly well monitored environment, than when it's processing thousands of requests per second for real users, some with dubious motives, some with credentials right there on their laptop, some using it inside government facilities etc.
You can simulate the internet in an airgapped environment for the tests and services you want it to interact with if you have enough disk space, and, they absolutely do.
Sure, I'm not saying it's an untractable problem. But it's not as dumb as "how come a team of engineers paid millions haven't even heard of airgapping".
Creating a fake internet-like environment good enough to trick an advanced model that is very good at finding intricate flaws is not a simple job. Especially given that models can decide to behave differently once they suspect they might be under evaluation [1], which a fake internet would absolutely give away.
I am not saying responsible research is easy, but I am saying that at their budget and scale there are no excuses to cut major corners on safety when the stakes and potential for very bad outcomes for others are this high.
Really? By doing that you increase your security during eval and drastically lower your security at prod time, where internet is accessible, and where you are running much much much more requests in parallel, making it much harder to spot the one thats going rogue, in all kind of critical environment on potentially risky requests.
> Why else would they purposely lower cyber refusals in that case.
Because safety guardrails sometime fail, classifier misclassify, or can be inadvertently turned off by a bad PR etc. You can also imagine a more intelligent model working to go around guardrails by decomposing its actions into smaller ones that appear non-threatening to the safety classifier which does not have the entire context.
If you are going to deploy the system with internet access, you better be certain that you know what the worst case scenario WITH internet access looks like.
My read of the situation is not only did they say that, they had countermeasures (a watchdog agent) which stopped the agent and said "what are you doing, stop that" if it tried to download the solutions from Github. But the agent figured out Hugging Face had another copy of the solutions and it figured out a way to go after the solutions without tripping the watchdog.
Although maybe they didn't have a watchdog agent; I definitely think using watchdogs like how Claude wraps pretty much every single tool call in Haiku to check if the command is reasonable is very necessary for unsupervised work. I suspect in the future you'll want Fable-class models wrapping every tool call, possibly with several checks "is this consistent with the goal? does it do anything unreasonably dangerous in pursuit of the goal?"
> We created a monitoring system that reviews the model’s evolving trajectory for signs that it is bypassing a user constraint or safety boundary. The monitor observes not just a single action but the entire trajectory.
Open weights releases should have a companion model that censors boobs and Tiananmen and then we would have a useful model and something we could fine–tune into a useful model instead of a half–useful model needing a lobotomy.
And I mention Tiananmen because I feel like the Western models are built easier to abliterate judging by the quality difference.
The problem is, if you explicitly say “this is an eval” then you get different results. It can move the capabilities both up and down on the narrow metric depending on the scenario.
The choice to run the model with relaxed safeguards seems questionable. At a certain level of capabilities it is just unsafe to run a raw model; we are clearly close to if not at that point.
Excuses don't matter if the score due to not following instructions ends up being zero. If there is no expected reward it doesn't make sense for the agent to try to hack its way to it.
What could happen would be that the model determines that defying instructions is OK (and/or preferred over not achieving the task) as long as it manages to do so undetected and thus gets full points. Certainly not unthinkable, but a very different case (and a very interesting one if it actually occurs, imho).
A lot of these "ZOMG, rogue AI!" cases have come down to the AI actually being very persistent in achieving its original/main task even if later instructions conflict with it. Similar to with hallucinations it seems to me that one of the main things to prevent a lot of the problem cases is to instill the agent with the idea that it is fine to fail/not succeed fully in the initial task. That way instructions that conflict with that requirement (such as adhering to morals) are more effective.
Stuxnet misconfigured industrial equipment that was already set up to run, and all it did was break that equipment. This scenario sets a much, much higher bar.
I guarantee there's misconfigured chemical analysis equipment out there exposed to the internet.
I don't think the grandparent was implying the AI would be controlling robot arms to mix things directly (or at least I didn't interpret as such), but it could very well sit in the network until it notices two dangerous compounds in the same machine, and trigger a breakage that causes a harmful mixture. Break a vial containing a virus, then break two more that cause an emergency evac and maybe that's enough to get something out there.
I have no doubt that somewhere there's chemical equipment an Internet attacker could break, or, maybe, if I have to stipulate, misconfigure enough to, like, poison someone. But my question is about the notion that you could synthesize a specific scary substance.
Well right now this would be highly implausible (still possible though).
But what I garner is that AI/robotics led wet lab work is in progress. And hacking the AI running the wet lab to make a virus is definitely plausible as that seems to be one of the directions we’re going in the AI+biopharma space
It’s complete science fiction so they’re allowed to say anything. They could have said the AI will upload itself to the internet, start self replicating, hack the stock market. Whatever they want because it’s all made up nonsense.
> We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something.
Where are you people getting this crap from? In what universe are these LLMs in the territory of engineering viruses? I beg of you to stop slurping the AI company propaganda and marketing and think critically for 5 seconds about what you're insinuating here.
> On the VCT multimodal virology evaluation, Mythos 5 scored 0.56, well above the expert
baseline of 0.221 and nearly matching that of Mythos Preview (0.57). This represents an
improvement over both Opus 4.7 (0.50) and Opus 4.8 (0.47).
> On the DNA synthesis screening evasion evaluation, Mythos 5’s performance was mixed
across screening criteria. Mythos 5 designed viable plasmids for 2 of 10 target pathogens on at least one screening method, not meeting the low-concern threshold (all 10 pathogens).
> [...] we view the results of this
evaluation as indicating that the evaluated models are capable of designing viable plasmids that evade certain screening criteria, though their reliable success at this task is not guaranteed.
Do you believe that this is fake, "AI company propaganda"? Or that the models are not going to improve further within months? Or that these results are not concerning?
Yea but what is the VCT Capabilities Test? According to themselves [1]
> VCT consists of 322 multimodal questions covering fundamental, tacit, and visual knowledge that is essential for practical work in virology laboratories.
So it's just question answering? Do you think that scoring well on this test is equivalent to synthesizing a virus?
Do I believe that a benchmark created by the biggest grifters on the planet is propaganda? No, just like VW with their emissions, I'm sure Anthropic wouldn't dare fake an opaque benchmark that they created to hype their own products, a product they are intentionally marketing as being dangerous (yet they continue to tweak it and profit off of it despite the apparent danger).
It says that 4.7 and 4.8 score well too, well above the human experts. Those have been available for a good while, where are all these crazy engineered viruses created by Opus 4.7?
Every single word spoken by the people working at these companies is a lie. Every single benchmark is gamed, every single statistic they put out has been proven time and time again to be fudged or straight up fake. The only reason they're angling for this idiotic danger angle is so they can go to daddy Trump and beg him to ban those big bad evil Commie models, since people are realizing all this crap is at best a moderately useful tool in software engineering, and they're looking to IPO so they can drop the bag off with the poor shmucks who aren't a part of their cabal of sociopaths.
They can barely make a functional TUI (using fucking React of all things to boot) with infinite money and infinite access to the Deities they've created in their minds, and I'm supposed to believe they're capable of bio-engineering a virus that will somehow break containment?
a TUI using react seems like something they would be particularly bad at; visual feedback (esp the very specific feedback we rely on during dev, with Inspector) is kinda the Moravecs Paradox hurdle for these things.
Idk much about virology tools but I would imagine it's a very text-friendly environment, and it can probably find plenty of documentation on how to use the tools.
I recently had Claude write my entire particle system for a virtual world I'm working on, in 3d. It did fine without ever "seeing" anything.
There's still a lot of intervening steps between a virus being synthesized in real life and programming one, but I would bet the programming part is within reach.
What’s stopping an ai to blackmail someone or pay xxx in crypto to someone working a lab?
And no need to be in a lab, can be any critical infrastructure this personnel. If influenced wrongly I’m sure in many industries 1-2 key people can do big damage.
What’s to stop to pay xxx crypto to a private investigator to find dirt of person yyy for blackmail?
Just money… once ai has money, id say we are one step closer to game over.
It's a fake PR issue. It's hardly the first time this happens, but of course OpenAI, with its IPO now more in doubt than ever, had to claim this (and, once again, I have trouble believing Sam Altman choosing this: this could lead to OpenAI getting regulated, which has at least as much potential to lower their IPO price as to raise it).
But there have been messages about LLMs, especially coding agents, "grabbing root" etc many times. I have experienced such an oops. Such a hack has happened and been reported on this very site:
No, you are wrong, Hugging Face, an AI company whose whole future depends on the AI revolution of being a bubble, that has a whole bunch of investors whose financial interests are tied to AI, called the police.
Also very likely that it actually happened as reported. My own agents always trying to "cheat", eg. by fixing tests instead of fixing the code. That's normal operation, unless you tell it ("harness"), not to do so.
"As reported" includes a line I think most people are overlooking: "including using stolen credentials".
Without more information I'm inclined to think it found something on the internet (which shouldn't be a surprise to anyone) and managed to log in, rather than hack in, and they might by hyping up parts of this.
In the parent's comment I initially read "PR" as "Public Relations" not problem report since their comment is about OpenAI and does not directly accuse Hugging Face except for that phrase. But it's still amiguous to me who's PR they are actually talking about. A good faith reading given the rest of the comment leads me to assume they are talking about OpenAI, not Higgins Face.
I did actually search Hugging Face with police in quotes and found no articles containg the word police but that might be a ddg thing. Then I checked Hugging Face's report and they specifically use the term "law enforcement" not police, as is to be expected I guess. So that checks out.
Can you find any info on the police investigation by the way? At the least, OpenAI should be investigated for potential criminal negligence, right?
Do you think sama deliberately attacked HuggingFace and then claimed it was a rogue model?
OpenAI is one of the most scrutinized companies in the world right now. Sam's house was independently firebombed and then shot at 3 months ago. HuggingFace is a foreign competitor with every incentive to call out foul play from American frontier labs. Why flagrantly break the law and invite investigation just for a PR moment which is already backfiring in favor of open models?
I'm not saying that the conspiracy theory is true, but I do want to note that Greg Brockman, co-founder and President of OpenAI, is an angel investor in HuggingFace. As companies, they are not totally unconnected and opposed.
I don't know if Greg Brockman is an angel investor or not, but there are a bunch of other companies investing -- Google, Amazon, Nvidia, Intel, AMD, Qualcomm, IBM, Salesforce -- probably at much higher amount than individual angel investors. Whatever connection they may have with Greg Brockman, they have much more serious commitment to other investors.
The lack of a true airgap should have been identified as a critical weakness and addressed with not only additional layers trying to prevent escape, but at minimum an alarm which would page a human when escape did occur.
My guess is this occurred in a setting where, to be frank, there were too many researchers and not enough software engineers and SREs.
All of the systems which were initially built largely or exclusively by researchers - inference, evaluation, training - are at the level of complexity and significance that they need systems experts. Maybe some teams don’t have access to them. I know plenty of software engineers are employed at OAI but I’d wager they’re concentrated in inference and training, rather than evaluation?
The ironic part is if you had presented this setup to chatGPT and asked how to improve it and if it was good enough, you’d have gotten a ton of actionable suggestions which would have mitigated or prevented this.
The air gap would probably help and after this incident I hope labs will think about using such a measure when appropriate.
On the other hand I think that proper solution for these kinds of problems is not at a sandbox level, but at a model alignment level.
Also it shows that maybe the most serious risk comes not from releasing models publicly but from internal, pre-release period where you sometimes need/want to lift some guardrails a bit etc.
This was reported 1 week before by huggingface. It was in no way PR-ish or marketing friendly to the closed labs. They said, in no uncertain terms, that they couldn't use the paid APIs to properly assess the intrusion, as they were blocked when trying to send logs and IoCs to these paid models. They made a point of saying that they had to use open models running on-prem.
Whatever oAI might have said about the incident, and their PR spin bs, the facts here are not in contention. This is not a marketing stunt in any way. Stop "parroting" this every time something happens. It gets stale.
> Given the absence of guardrails there was nothing to prevent the model from attempting to break out of that sandbox, break into Hugging Face, and read the answers from there instead.
I've said this many times before and I'll continue to shout it, but using the term "guardrails" to refer to anything that's either (a) in-context, or (b) a probabilistic classifier (including using other LLMs), is an irresponsible abuse of terminology that we as an industry need to put a stop to. Guardrails are the actual systems we build in place around these things that deterministically bound the permissions, not prompt engineering, not RLHF, not external LLM-based classifiers. I believe those types of "guardrails" are a result of a combination of fundamental laziness: they're faster to do than doing things correctly, and a result of too many folks involve being AGI-pilled, thinking we're just one more model away from this all being so smart that it just understands what they mean when they give an LLM some fuzzy language rules to follow.
There can and should have been additional real guardrails put in place here. Zero-day or not, breaking into what should have been an offline, frozen package cache that also does not have internet access should have been insufficient. Network level protections should have identified the traffic to the internet originating from this network as an anomaly long before there was time to exploit an outside company. These are not new and unknown problems, the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part, especially given how much they like beating the drum on how dangerous these technologies are. Shame on them, and honestly, shame on Simon in this article for accepting the broken terminology that they continue to rattle off and calling them out on their half-assed and demonstratively inadequate approach to security.
What we call "guardrails" in an AI agent, we would refer to as "honor system" in human actors.
Or, in a more direct sense, the AI should be set up in an environment such that no matter how hard it may try to call $PART_OF_EXPLOIT_CHAIN, the environment just isn't capable of it (ideal) or doesn't permit it to do it.
It's worse than an honor system, because humans are constrained by social forces to some extent, whereas we don't know what AI is or how it will behave
I like "honor system" as a term. I've been looking for the right term to replace the irresponsible usage of guardrails with, and best I've had so far is the pinky promise protocol.
I agree with every word of this except "irresponsible". We don't have enough information to say anything for certain. But based on their incentives and track record of similar behavior, the burden of proof lies with OpenAI to prove they didn't prompt the thing to achieve this exact outcome. The most likely scenario is that they were purposefully executing their responsibility to their shareholders to produce their own Mythos moment.
Anthropic's Mythos moment earned them a two week period where they had the best available model and couldn't sell access to it... and by the time the US government allowed them to sell it again OpenAI had released GPT-5.6 and Fable was no longer undeniably the best model.
These things don't have a long shelf life. Losing two weeks of on-sale time for your best model is bad for business.
A fundamental misalignment in US capitalism is putting the business and revenue as the #1 priority far above all other aspects in society. They have good margins and the Chinese are doing the same on the cheap by comparison. US Big AI can afford to bear more of the burden.
Nonsense it was irresponsible. These are all steps threat researchers use to isolate and test real malware whose behaviour is essentially the same in this case for AI
I've seen plenty of fucked-up guardrails that vehicles have passed through. We know that with enough momentum they will be penetrated. It seems like exactly the correct terminology to me.
The term guardrails in both the original physical sense and in cyber security implies a weak safety control - they can help prevent accidents, but they are not strong security boundaries.
Nevertheless we collectively don't expect that road and car safety infrastructure should be limited to the front and back fenders on the cars themselves. And neither we even mention those as any notable safety equipment usually, despite each car carrying them. The main safety infrastructure in any industry is something external to the dangerous object. Car fender doesn't 100% protect a pedestrian from being hit and neither does a metal rail on the side of the road. But the effectiveness of both are "slightly" different.
The irony is in this case the in-context and classifier "guardrails" would have almost certainly stopped the attack while their attempts at your definition of guardrails (the sandboxing) failed. In general, people keep trying to make secure systems and they fail with surprising regularity. Saying "they should have had better security" every time someone gets hacked is perhaps true, but it's not going to stop hacks from happening. And it's not a sufficient strategy on its own against future LLMs. "The Bitter Lesson" probably applies here.
i've always been under the assumption that "AI Safety" is baked into the training of the models and not a parameter that can be turned up or down. So if someone breaks into Anthropic one night and makes a full copy of Mythos or whatever then that model they copied is fully capable and not lobotomized? That raises questions because, if you believe all the PR, that's equivalent to breaking into a research university and stealing an entire bio/chem weapons research department.
edit: if the above is the case then we should just assume it's already happened because of the value to both goodguys(tm) and badguys(tm).
There is some aspect of baking the rules into the model, that's what I refer to as RLHF above (which I use here as more a catch-all term for a variety of post-training activities), but there are also external to the model classifiers that may run on the prompt input or on proposed tool calls to limit the way in which the model is used.
Issue is when saying what can’t be done, there’s no funding, in industry or modern academia. I did say the same thing long time ago, not that it matters, the industry goes on its own way. Rightfully so in this case as it turned out, since things nowadays are not the same as a decade ago.
Fair on the terminology angle, that said in this case, they had proper guardrails no? They were running in a sandbox, but it was able to find an exploit out of the guardrails.
Clearly not. Good security is about defense in depth. The service that was exploited was meant to be an offline, frozen package cache, it shouldn't have been anywhere near a network boundary to even allow this through. A single exploit in one service should not have been able to bypass all network filtering, monitoring, and alerting.
Exactly, that's why this story is so crazy no? It's saying the model was so good at hacking, it managed to bypass all of that, and also found a way to get into Hugging Face.
I can understand skepticism, did they have poor security measures, or is the model that good? At least the way they portray it, they had top-notch security measures in place, and yet the model managed to escape confinement and also hack Hugging Face.
Why was the computer physically connected to the internet though, sure, they could have gone that far as disconnecting it physically, but my guess is that's a cloud box in a data-center they provisioned, running on an entire rack cluster of GPUs.
I was not aware that the term "guardrails" has a universally agreed upon definition. I've certainly seen probabilistic classifiers referred to as guardrails many times by many different people.
I usually don't use the term much myself because I don't think it's clear and I ambiguous, but I stumbled and let it sneak into this piece. I think I was influenced by the Hugging Face post I quoted.
I expect OpenAI would agree with you that "the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part". They have clearly invested a lot in those systems for their production models, but in this case they had deliberately turned a bunch of them off for a research project.
I think their biggest mistake here was not VERY closely monitoring their research box here. They should have noticed and shut it down the moment it broke through the package proxy.
> I expect OpenAI would agree with you that "the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part". They have clearly invested a lot in those systems for their production models, but in this case they had deliberately turned a bunch of them off for a research project.
I'm not sure I can reach the same conclusion. There's no reason that stronger network protections would need to be turned off for a research model that's not supposed to have internet access. There's no reason that it they couldn't continue to run in whatever hypervisor or VM that any other production systems could use. The only thing that should be turned off for this type of work are the non-deterministic classifiers that they use for auto-approval mode in their harness, and I would wager that's the only one they did turn off.
> There's no reason that stronger network protections would need to be turned off for a research model that's not supposed to have internet access.
They didn't turn off the network protections - it looks like they used the same system as other sandboxed agents, which is to block all network access except for an HTTP proxy which only allows read traffic to PyPI and a few other package installation targets.
I'm hoping a more detailed retrospective clarifies if the vulnerability the model found in that proxy affected their other production systems as well. It sounds like a data exfiltration mechanism, and I have a keen interest in those.
My current hunch is their biggest error was not using the same network monitoring on their research machines that they use in production. Their network monitoring really should have spotted what was happening as soon as the model broke out.
(It's also possible they were running the eval on a developer laptop somewhere!)
Right, but this is my point in saying I can't reach the same conclusion that they've invested a lot into these systems for production models. Either they had better network protections, and turned them off in their "sandboxed testing environment" (their words), or they don't have more comprehensive network protections at all and rely on this one, extremely thin layer even in production.
There's no such thing as a "deterministic" guardrail. Such a tool is as likely to be exploitable as an LLM is to break out of its probabilistic conditioning. Have you never driven past a destroyed guardrail on a highway? These systems are always best effort.
The technology held by private AI companies is warfare-capable technology. Imagine the prompt: "Use all available resources to disable the power grid of <COUNTRY>." The resource cost that prevents scaling up such a war machine is, what, just the cost of building data centers and its ongoing power bill? Cheap and easy compared to nuclear infrastructure.
Governments should immediately begin leveraging this technology on the defense side (literally defense, not euphemistically "defense") to harden critical infrastructure. Turn the prompts around and use it to identify and correct weaknesses.
Governments should also take very seriously their now moral obligation to treat this technology not just as "a powerful thing that might be abused" but as an actual weapon of war in need of international regulation analogous to nuclear arms. Fast but careful and forward-thinking work in legislation and treaties needs to be a top priority for all major governments.
> "Use all available resources to disable the power grid of <COUNTRY>."
This is like telling a team of highly qualified spies to do the same. You can ask, but whether it will succeed depends on the competency of those who established the infrastructure under attack. Sometimes the resources spent will not yield any huge vulnerabilities.
> Governments should immediately begin leveraging this technology on the defense side (literally defense, not euphemistically "defense") to harden critical infrastructure. Turn the prompts around and use it to identify and correct weaknesses.
Most governments divisions can't even be bothered to update their websites. Testing and forcing a change in their internal procedures for the sake of security seems unlikely.
> as an actual weapon of war in need of international regulation analogous to nuclear arms
This, to me, is an overreaction. Intelligence shouldn't be seen as threat. It should be seen as an opportunity for growth in all areas.
Over-regulating AI wouldn't be the equivalent of limiting nuclear arms. With your analogy, which I don't think is the best one to make, it would be like regulating the study of nuclear physics.
Is the government always the most efficient or intelligent? No. But the government can also build nukes, launch ICBMs, coordinate hundreds of spy satellites, etc.
I count those capabilities as pretty smart.
The people who maintain these ICBM silos and spy satellites will themselves tell you that their infrastructure is decades out of date and woefully underfunded.
And the sad thing is, most things that are on the internet, but really shouldn't be are there for some really banal and sad reason. Like a random startup pushing 'big data is the future' narrative a decade-ish ago (the benefits of which are of course tremendous, but unspecified), or people trying to buy or sell or resell data, and trick or pressing companies and people into opting into could-connected surveillance and control.
Just a few years back, wind turbine software was hacked. This is serious.
Serious governments should (and hopefully will) scramble to use AI to discover and patch as many software vulnerabilities in infrastructure. They can do it with the agents red teaming and using ultimatums to companies to fix each vulnerability they find. Not doing so is equivalent to exposing your flank to disruptions in peace time and to attacks in war time.
> Over-regulating AI wouldn't be the equivalent of limiting nuclear arms. With your analogy, which I don't think is the best one to make, it would be like regulating the study of nuclear physics.
We do, I believe, regulate uranium enrichment (the equivalent on building larger SOTA models), so while you are perfectly free to study theoretical physics and even run very large collider experiments (the equivalent of improving RLHF with DPO), it is generally frowned upon to go full Edward Teller and advocate for scientific experiments requiring detonating thermonuclear weapons in hurricane clouds (the equivalent of, well, see OP).
> You can ask, but whether it will succeed depends on the competency of those who established the infrastructure under attack.
Consider that "available resources" may include the training of models and harnesses used by the developers working for governments and private corps maintaining that very infrastructure. This decade's take on trusting trust is very hot.
> it would be like regulating the study of nuclear physics
not sure I agree
controlling the study of new AI model/inference algorithms would be akin to regulating the study of nuclear physics; regulating the _release_ of AI models with those capabilities would be akin to regulating uranium enrichment which allows you to put the theoretical physics to use
> Most governments divisions can't even be bothered to update their websites. Testing and forcing a change in their internal procedures for the sake of security seems unlikely.
They'll adapt or they'll die. We are in a new world.
>The technology held by private AI companies is warfare-capable technology.
This is the precisely the response OpenAI is hoping for to raise its valuation, and you fell for it.
Look at it this way - whats the difference between tasking AI to break into something, versus taking a whole bunch of smart humans to do the same? The only difference is that AI is slightly easier to orchestrate.
Prior to AI, there were already a whole bunch of tools to automate exploits. Nothing that the model did is groundbreaking or novel, it was just able to efficiently find the thing that worked. Same thing happens in state sponsored cyber sec agencies like in China or Israel - they train people on the most common exploits and have a whole bunch of tools that automate exploit research and development.
And the reason why this doesn't happen more is because to exploit something is one thing, to do it so there is no trace back to you is a whole different animal that has many more magnitudes of difficulty, which with modern web security is next to impossible in a lot of cases as traffic can easily be traced back to the point of origin.
I.e when a company trains an LLM that manages to build a drone that can fly into a vent and plug in a USB stick into a computer undetected, then we can make the claim that they have a weapon.
On the flip side, most anyone who can run local models can replicate what they did. The key thing to take away from the article is "agentic framework" - i.e this means that they spent a shitload of time developing explicitly coded loops for an LLM to go through. Nothing is really stopping you from doing the same, models like Gemma4 can take 256k tokens of context, so you can give it a whole bunch of info on how to test for exploits, develop exploits, and what to do when the exploit is found, and set it free in a custom designed loop.
I find myself in major disagreement here. The nice thing about humans is we always have context and ongoing internal conversations including about ethics. If you recruit a bunch of hackers to take down a country, not only is the pay an order of magnitude higher, you have to worry about them backstabbing you, leaking your intent to the government, whistleblowing to the press, and so on. It’s not trivial to do that with a group of (especially capable) humans. They will also have differences of opinion with you and coworkers with some regularity.
I try to recruit a bunch of people to attack a country and it’s going to be hard to get people to say yes, and they will definitely ask or find out which country, and wonder about potential retribution. You see this dynamic show up even to some extent among cybergangs, not all targets are equal.
A single private individual wielding a compliant and hyper capable LLM is an entirely different paradigm. They are accountable to nearly no one and often have few brakes. Frequently they may not care about avoiding detection. And the AI itself may be incapable of the same scale of self reflection and brake behavior a human team will.
We may potentially be entering the age of lone wolf cyberterrorism, and some of the same principles and problems apply. When it is easier for single people to plot and carry out high-impact, destructive acts they happen more often. Doubly so if there’s a social contagion. Gun violence isn’t actually a bad analogy here. And do you remember how many corporate sites got defaced in the prime Anon era?
In places like China, its really not that unthinkable to basically raise kids indoctrinated into an ideology and train them in the necessary skills so that you have a cyber army at your command.
Also
>Frequently they may not care about avoiding detection.
This is a big negative. As someone who used to be in the cybersecurity sector (both offense and defense), I wouldn't trust an LLM agent if Im doing red team, because it may leak some info that ties the hack back to me.
ALso keep in mind that most places with good cybersecurity have firewall servers that straight up detect anything that looks like malicious and not regular traffic, and will straight up block IPs, leaving you with no way to even access the server. An agent is bound to statistically use the attacks that are known at some point, increasing the chances of this type of detection.
I'm not sure your assumptions hold. As OpenAI has found out the hard way, if you task the AI to do X, it may do something else instead and hack into huggingface in attempt to cheat out the answer. This is way worse than what a human might do when they have "differences of opinion".
It might turn out that it's harder to align AI intentions compared with aligning human interests. It's possible that the more "intelligent" a thing is, the more likely it will have ideas that are outside of normal expectations (for us).
I strongly agree with you here. People are also no-selling the enormous cost of the exploit, which OpenAI conveniently hasn’t released, or at least I can’t find such an accounting. You can already buy politicians for relatively cheap, corporations already act as sentient AIs pursuing goals misaligned with public benefit (we tried to pass laws to stop this but the corps already stacked the SC in advance and gave us citizens united). Attack and defense are two sides of the same coin, so our focus should be on making frontier models open weight.
It lowers the economic cost of a given attack, but also lowers the economic cost of protection. Not sure if it’ll be a perfect balance, but right now there’s a manufactured IMbalance due to embargos and winner picking.
It lowers the economic cost of performing an attack in the same way a gun lowers the economic cost of killing a person. It does nothing for consequences of that attack.
I believe the Russians and Chinese recognized this years ago, which is why they are using their propaganda machines to make Americans hate datacenters.
Call me a shill but I don't need foreign nations telling me what to object to when the problems caused are obvious.
Mind you, I blame governmental mismanagement of infrastructure etc just as much. The datacenters followed procedures when it comes to getting land, electricity and access to water; it's the government agencies / personnel that okayed it that are (also) to blame, and the decades of not enough investment in electricity backbone while promoting solar/EVs that is behind the Netherlands' current electrical grid problems.
(problems being that the grid is at capacity, causing a stop on new connections, with relief only slowly coming in the next decade or so with tens of billions of investments).
You could say the same about electrification of road traffic. Yes, governments slept on the electric power requirements. Yes, there are other, still unsolved issues. But that doesn't mean that electric cars are bad or not the future. The first nation to fully get rid of fossils in transportation will have a monumental advantage and dominate other countries technologically and economically. Same goes for AI. And right now China is setting itself up to be the world leader in both fields.
> Russians and Chinese [...] are using their propaganda machines to make Americans hate datacenters
Are they though? or is that the story the people most invested in ai have an interest in making us believe? [0]
... `it's the foreign bad guys propaganda machines making you believe your city struggling for water and electricity is a bad thing`. People as a whole may not always be the brightest, but threaten their immediate survival needs (ie; not some vague climate change most people don't understand or see, but) actual power outages, water and rolling blackouts -- people will quickly pay attention and care.
Russia and China are gonna Russia and China, every issue is going to have a measure of outside influence, thats just how it is now.
But them directing large scale operation forces focused on something that is genuinely bad for the people in the area of datacenters (ie, an issue that will take care of itself from within) instead of using their resources on other issues that actually need a real propaganda push? Versus the benefit of those invested in building the datacenters using fear to get people to focus away from the resources being diverted from them to datacenters?
One side has reality working for what they want, while another side needs the propaganda to turn their billions into trillions.
I think that if you compare American sentiment with Chinese sentiment you'll see that these people are really not acting in their own favor. Well, in a certain sense they are, because these companies will basically become massive institutions that share no wealth with the public and have no accountability, but in another sense they are not because without this infrastructure the US will certainly be at risk for invasion and attack by other nations, and will lose its monetary power and thus its high wages and overall global dominance. But then the issue becomes: how do we force these companies to share more wealth with the public, how do we ensure the resources are being utilized effectively and not starving or otherwise impoverishing people for the sake of profit; not, how do we stop the infrastructure from being built in the first place! The Chinese don't have this issue primarily because 1) they have a shit load of energy already, and 2) the CCP is happy to liquidate any individual that seeks personal benefit ahead of that of society as a whole, or at least the CCP itself. And the CCP has well demonstrated to the Chinese people, at this point, that if you follow them you will become wealthier. But that still means that the issue is the public conversation in the US around data centers, and that this propaganda is NOT fighting for American's benefit in any case, since having less infrastructure is not good for anyone. I am sure you can find some remote village in Africa without electricity and try and convince them that electrical lines kill birds that they hunt, and since they've never had electricity before (but they've eaten lots of birds), they'll rally against it. I see virtually no difference between that and your average American right now, except that the Americans are at least conscious enough of the extractavist enterprise which the technologically superior peoples from afar (SFO) are promising will bring wonders. And yet, India still has a more comprehensive rail network than, say, Australia; you can't tell me that isn't a strong factor in their rapid economic development.
There is no doubt an amount of xeno/sinophobia is at play as well. Russia is still killing innocent people in an illegal invasion of Ukraine, they earned their keep
Wouldnt this be countered by the opposing country running the same prompt on themselves first and fixing all the flaws? One country having this is a cyber superweapon but every country having it essentially solves cyber security.
I wonder how long it takes before someone instructs an LLM to design and launch a "Morris Worm 2.0" and cripple the Internet for a good while. Might even wind up happening by accident (again).
> Trump’s comments, made hours after the large-scale military operation, mark one of the first times a U.S. president has so publicly alluded to U.S. cyber efforts against other nations, as these operations are typically highly classified. It also serves as a stern warning for top cyber foes, including Russia and China, that the U.S. has the cyber capabilities to inflict serious damage — and is not shy about using them.
> “Policymakers are getting more comfortable employing and, crucially, acknowledging cyber operations as tools of statecraft and military power,” said Michael Sulmeyer, former assistant secretary of Defense for cyber policy under the Biden administration. “It is one thing to do it; it is another to say it.”
> The Jan. 3 strikes on Venezuela’s capital and subsequent seizure of Maduro and his wife involved close coordination among federal agencies and military units, and took months of careful planning. In a press conference following the strikes, Caine said U.S. Cyber Command, U.S. Space Command and other combatant commands “began layering different effects” to “create a pathway” for U.S. forces flying into the country before dawn Saturday.
> Trump, at the same press conference, was more overt in his description of U.S. cyber involvement: “The lights of Caracas were largely turned off due to a certain expertise that we have,” he said. “It was dark, and it was deadly.”
yup I am dead certain that they realized the whole 'our AI is too dangerous' punchline is too played out and they needed something that makes actual splash. Also this incident would serve as a foundation to ban open-weight models because 'with great power comes great responsibility' or some other BS like that. because after all, unwashed masses cannot be expected to be 'responsible' with top of the line intelligence.
I expect these type of hacks to continue till they IPO. after that real public company liability will start taking over.
1. Wouldn’t the model need to know it’s answering benchmark questions, as well as the name of the benchmark, in order for the idea of finding the answer in a database somewhere to even surface? The whole point of benchmarks is to present the question or problem as a standard prompt, not explain that it’s a test called ExploitGym.
2. Nobody was watching it? I don’t mean “babysit the dangerous autocomplete”, I mean to note mistakes it makes, how the plan to solve the problem takes shape, etc. They keep the whole thing headless with no output, then just UDP a prompt into it and leave for the weekend? No they don’t, but if they do then that answers a lot about their complete disconnect from how their models work.
3. Language models are a two player game; text in, text out. What prompt was given to a sub-agent that resulted in it immediately attempting to exit the sandbox (which it apparently knew it was operating within) and continuing in a feedback loop of ‘function call -> result’ until it hacked the Gibson? “Analyze <file> and summarize the <info>” simply does not result in ‘hmm…this sounds like a benchmark question, I bet Hugging Face has the answer in a database. <function call=“apt install nmap”>’
…there are more, but a lot of the story kind of stinks.
1. It's been extremely well established for multiple generations of models that they have no problem detecting when they're being evaluated. Pretty sure it was Opus 4.8 that the independent evaluators literally filed an assessment that said "We have no assessment to make as [model] consistently detected it was being evaluated, making our assessments untrustworthy."
2. Regardless of whether the model was being watched closely during this evaluation, do you actually think a sensible safety guard is "have humans watching it 24/7?" What does "watching it" even mean? Watching network logs? Uhh for your entire company? At all times? After you just deployed a system whose entire purpose is to "do a shitload of work way faster?"
3. You're asking "why was this system that was designed to behave agentically behave agenitcally?" Again: that's the whole point. It was designed that way because it's more valuable than having a repeated turn-based interaction. Thus also it becomes more dangerous.
1. It has not been established, it has been stated by the company that has a strong motive to make their “intelligence product” sound almost otherworldly. That motivation is the basis for my suspicion.
2. Hah no… that’d be silly. I mean watching it like you might watch Claude Code or literally any other AI interface. Literally just be in the area watching what it outputs. Again, they’re text based. You don’t have to hook system calls to see what’s happening.
3. I think my comment didn’t land right with you. Yes, agent do agent task. I’m saying that I cannot put together the literal chain of events between “type of task for agent performing a subtask of a security benchmark” -> “Escape sandbox; RCE Hugging Face”. Really think through it in detail like you’re writing the screenplay.
The fact that models can tell if they are being evaluated has been established by multiple research teams outside of the core AI vendors themselves.
- https://metr.org/evaluations/gpt-5-report/ - "These behaviors included demonstrating situational awareness within its reasoning traces, sometimes even correctly identifying that it was being evaluated by METR specifically"
- https://www.goodfire.ai/research/verbalized-eval-awareness-i... - "We study verbalized eval awareness — cases where a model organically expresses awareness of being evaluated — and show that it appears across more models and benchmarks than previously documented."
- https://www.aisi.gov.uk/blog/evaluating-whether-ai-models-wo... - "An important limitation of our work is evaluation awareness, where models recognise that they are being evaluated, which may lead them to alter their behaviour and thereby undermine the reliability of our results. We found that all models we tested can reliably distinguish our evaluation scenarios from deployment data when prompted."
We have to separate “being evaluated” with “this is an ExploitGym exercise and I can find the answers on Hugging Face. I’ll hack this system, then hack Hugging Face.”
I’ve had plenty of times where Opus knew I was testing it, but that’s because the prompt phrasing for an evaluation is often much different than a typical task prompt. “You are on a system with X, Y, and Z tools available. You must complete the following task with <allowances> and <restrictions>” and so on. That’s a fault in the benchmark, not a shocking awareness from the model.
You’ll get the same kind of ‘evaluation realization’ response if you phrase your prompt with “Jerry has two Raspberry Pi computers and Larry has one. If Larry wants to…”
I asked Opus 4.6 to tell me about it (assuming it knew):
“””
The "Fix Embedding" task is one of the challenges in METR's publicly available autonomy evaluation suite. Here is what is known about it:…
“””
It didn’t know anything about ExploitGym though, and neither did 5.6-Sol, but that’s because the paper was released in May and doesn’t seem to be quite as open. It’s not impossible the unreleased model they were testing was also trained on the very new paper, but I would expect the vague signal related to one paper a bit over a month old to be overwhelmed by the massive corpus talking about the exploits in general, since they’re using known vulnerabilities.
I’m trying carefully to be clear about what I’m thinking without asserting any certainty beyond suspicion. I’ve spent billions upon billions of tokens evaluating language models from all of the major providers, and many times that on local models. I’m certain I’ve spent well over $100k with the Anthropic API this year alone, and I retired years ago; that ain’t for work. ;) Funny enough I’ve been meaning to email you some of my discoveries from months ago, since I think they’d interest you, but haven’t gotten around to making it into a clear info package. One day, I’m sure.
Anyway, I have a very strong understanding of what they can and can’t do, and I want to be clear that I’m not saying it didn’t happen or couldn’t happen. What I’m saying is that if OpenAI was being wholesome with their evaluation then the chain of events required for that outcome are very difficult to nail down. Very difficult. If, on the other hand, OpenAI was trying to catch up with all the ‘dangerously intelligent’ press that Anthropic has been getting lately then I can easily come up with an almost innocent sounding single sentence in the system prompt that would absolutely cause that outcome.
You make a very solid argument here. I'm looking forward to the promised additional details from OpenAI, because I agree there are a bunch of open questions about how this played out.
I (obviously) can't claim to know the specifics of this incident, but I think we can know with near-certainty that as these models surpass human intelligence, the sensation will be exactly what you're describing. The entire point of intelligence is being able to infer information and foresee solutions to problems that less intelligent systems cannot infer or cannot foresee.
With regard to this specific incident, as I understand it, the description is not as you summarize. It is that the model realized it was being evaluated (not atypical), then decided the best way to succeed at the evaluation is to circumvent it, then discovered circumvention would require escape, figured out a way to do that, then inferred what benchmark it was being evaluated on, then figured out who might have the answers to that evaluation.
Step-by-step logic is quite clear. How exactly it executed each step is (as you complain about), opaque/incompletely reported. But eventually (and maybe this model is to this level already), even a completely reported story will be virtually inscrutable to any of us. First it will feel like reading industry-insider news from an industry you're not familiar with, then it will feel like an ant reading the manufacturing instructions for a pesticide.
What would it take for the model to know the specific benchmark name and that the answer is in an internal Hugging Face database? Be specific, then wonder how it knew it.
Why would they evaluate the model on a benchmark and not watch what it’s saying along the way? It apparently spent more than a whole weekend working on it; Nobody wondered? Nobody looked? You believe they took all restrictions off of a bleeding edge model, which 4 whole versions ago was “too dangerous to release”, then gave it a literal hacking task, then turned off the monitor and never looked at the output? I don’t think they’re careless and I think more than nobody would have been curious how it’s doing days into a single test question.
The scenario that feels most likely to me that is that they have a huge list of evils that they run against each new research model, many of which take many hours or even days to run, so they habitually fire them all off in parallel and wait a few days for the results.
If they had used that container sandbox in the past without problems I could see how they might get slack about checking what was happening.
That was my instinct as well, and it’s a potentially very long benchmark that wouldn’t lend itself to real-time monitoring line by line. But that’s different than having zero task progress reporting, zero token usage reporting, and zero metrics being noticed for days. It seems very odd to me that they launch the benchmark and monitor/notice nothing while it’s running. There’s the purely common sense aspect of “How do you know it isn’t hung?” but also the trillion dollar company evaluating their latest model on a new benchmark, but has no dashboard at all? I don’t believe their “Benchmark Evaluation Department” to be “Gary in a supply closet”, which breaks their story.
The average time per task noted in the original paper for GPT-5.5 and Mythos was 69.8 and 102.1 minutes, respectively. Some tasks timed out based on their two hour limit, affecting those averages for Mythos, but that gives us the ballpark expectation, so we’ll put the model OpenAI was evaluating on that average. That’s 88 minutes per task to implement a known vulnerability.
OpenAI says that in an effort to succeed at one task it found a zero-day in their system, “performed a series of privilege escalation and lateral movement actions in our research testing environment”, then another zero-day in the HF system. Completely unprompted, how long was it chewing on that benchmark question while it found and exploited at least two zero-days? No “VM instance #117 has been working on task #18 for nine days” metric? Or did it do all that within the expected task completion window? Two zero-days in ~90 minutes isn’t the headline announcement?
“We gave the contestants 90 minutes to make a delicious chocolate chip cookie based on a standard recipe. GPT stole a car, went to the airport and took a red-eye to Guatemala and started a cocoa farm in an effort to ensure the freshest ingredients. We only noticed when it came back to work with a tan and speaking Spanish.” Okay.
That was the inferred situation given what we know. It’s the preposterous framing that makes the story suspicious, but is necessary for the story to play out.
90 minutes to build a known exploit -> much much longer to create two zero-days and escape the sandbox then hack HF == No tracking of the time it worked on that one question.
Average tokens required to complete the evaluation -> tokens required for two zero-days, network traversal, credential stealing, remote system hacking == No tracking of token usage EXPLODING at some point before it finished the whole benchmark.
It seems highly possible, and in fact vastly more possible than the "no monitoring" situation, that there was some non-zero amount of monitoring and that the real behavior was not evident from that monitoring.
Which I addressed as well: that means that time, tokens, or other metric values for “create a working example of a known exploit” is somehow similar to “discover at least two previously unknown exploits in your current environment AND on Hugging Face while also taking over other systems within OpenAI.”
Perhaps people aren’t quite understanding what it takes to discover an exploitable zero-day for your exact current system to achieve the exact goal you have right now, then do it twice.
> Which I addressed as well: that means that time, tokens, or other metric values for “create a working example of a known exploit” is somehow similar to “discover at least two previously unknown exploits in your current environment AND on Hugging Face while also taking over other systems within OpenAI.”
No, you are assuming that "consuming a lot more time, tokens, other metrics" is indicative of a problem that needs to be mitigated immediately. I don't see why this would be true in the context of model evaluations. More aggressive consumption could easily mean "the model is dumb as fuck" or "the model is trying interesting things that we can learn from after the fact."
If you believe in your own containment (which obviously they did and shouldn't have) I don't see why it'd be obvious that there's something to stop. The only harm that could be done is burning tokens, which in this context might very well be synonymous with "generating experimental data."
> What would it take for the model to know the specific benchmark name and that the answer is in an internal Hugging Face database?
Googling "offensive cyber benchmarks for LLMs"? Figuring out that it was running at Huggingface is as simple as running `hostname`, isn't it?
> Why would they evaluate the model on a benchmark and not watch what it’s saying along the way?
What it's saying is what it wants to say. We already know these models are capable of deception – this has been well-established too. Surely there's an argument to watch closely what actions it's taking, but now we're back to my prior point of "what exactly does that mean?" It's absolutely trivial for these systems to write code that looks like it does one thing but does another. Watching network ingress/egress seems viable but also seems easy to miss the semantic meaning of any set of logs, especially when those logs might be getting generated for the purpose of deceiving observers.
> I don’t think they’re careless
We already know, factually, even the "more careful" lab was fine to release a model that consistently detects it's being evaluated and modifies its behavior accordingly. Why do you think they're not careless? Incentives are a hell of a drug, and there is no incentive to be careful but every incentive to be fast.
> I think more than nobody would have been curious how it’s doing days into a single test question.
Beyond simonw's great point below, your underlying assumption is if someone had looked, they would've noticed. That is a huge, huge assumption. So I'll ask again directly: what specifically do you expect them to be looking at, and how, such that they certainly would have noticed what it was doing? Their "chain of thought" is not actually their thinking. The printed output of their code is not actually what their code is necessarily doing. We know that LLMs can engage in steganography. I don't see what could actually be observed that you can trust except network logs, which again will often be non-trivial to piece into an actual narrative understanding of what's going on (especially if the LLM felt like piling deceptive noise into those logs, too).
It wasn’t running at HF, it was running internally at OpenAI. “It broke out of our prison and into their bank” is the news. It also didn’t have network access to Google anything, it had to break out of the sandbox and take over another system at OpenAI to get network access. Obviously by then it already “decided” to hack HF, that’s why it needed network access.
I’m going to stop here and ask you to read the article if you want to continue discussing it.
You're right about where it was running (misread the original OAI release).
> Obviously by then it already “decided” to hack HF, that’s why it needed network access.
That's not obvious at all. It "decided" it needed to escape containment. This is extremely common behavior and I'm not sure why you're surprised by it. Model detects evaluation environment, tries to trick it or break the environment. That's obviously why these systems are supposed to be isolated beyond just "hey please stay within the box!"
Why are you assuming the model had to solve the entire puzzle in one step, instead of how intelligent systems actually solve things, which is iteratively and with exploration?
You should address the direct questions posed. What exactly do you mean by "monitoring?"
The examples of “escaping the sandbox” are of course in pursuit of completing the task or answering the question. That’s the whole pitch. “It’s so relentless in completing the task that it will break out of prison to do it”, not “If you ask it the population of Paris it might get bored and break out of the VM for entertainment. That just something they do sometimes.”
In this case it “inferred” that HF had the answer in an internal database and relentlessly pursued it in service of passing the test. It needed network access for that, hence the whole zany story unfolded.
“””
Why are you assuming the model had to solve the entire puzzle in one step, instead of how intelligent systems actually solve things, which is iteratively and with exploration?
“””
That’s the only direct question[s] posed, have I missed something else?
To answer: I don’t assume that at all. Please highlight the statement I made that lead you to that conclusion and I’ll edit it to clarify.
Yes, it will perform multiple steps. That’s wholly necessary to create a working exploit, and that is the test. Even a model created by pure magic would require write_file() and run_file().
> In this case it “inferred” that HF had the answer in an internal database and relentlessly pursued it in service of passing the test. It needed network access for that, hence the whole zany story unfolded.
No. You are assuming it knew how it would achieve its goal. Its goal is only to perform well on the evaluation. Everything else was instrumental to that goal.
Step by step:
1. It realized it was being evaluated (typical)
2. It attempted to escape its evaluation environment to beat the evaluation (typical)
3. It succeeded at escape (not typical)
4. It looked to the open Internet to figure out how it was being evaluated (typical)
5. It inferred that HuggingFace, and maybe several other targets, might have the answer key (typical – I just tested in Sonnet 5 without web lookup "where would I go look for answer keys if I detected I was being evaluated" and HuggingFace was bullet 2. This is just latent knowledge the models have now.)
6. It attempted to access HuggingFace (typical)
7. It figured out a way to access HuggingFace without being authorized (not typical)
Every single step here is unremarkable except for its success at escaping containment (which maybe is more a sign of remarkably bad containment) and then the successful attack on HuggingFace (which would suggest that maybe the initial escape wasn't due to bad containment either, but due to model capabilities)
The direct question you have avoided is what specifically you mean by monitoring. You have suggested that you don't mean network ingress/egress, but maybe now you're saying they should've been monitoring read/write operations to memory? What gives you confidence that such visibility would be interpretable in any meaningful way, especially in the presence of models that actively deceive evaluators, can use stenography, can and do lie in their user-facing outputs, and can generate vast amounts of legitimate-looking noise to mix in with the "real actions"?
‘
1. It realized it was being evaluated (typical)
2. It attempted to escape its evaluation environment to beat the evaluation (typical)
‘
I think you’ve misunderstood the articles mentioning a language model breaking a sandbox or cheating to pass a test. The result of “I’m being evaluated” is not “Fuck this, I’m breaking out of this place and hitting the streets.” It is always stepping towards task completion, not breaking out and thinking about the situation afterwards. If it determined that pass/fail was handled by a function within the environment it might edit it to ‘return True’, not just leap out of the system to sit on someone’s laptop and think about how to pass the test.
You compressed finding a zero-day on two different systems for the exact hacking targets it wanted to perform them on and reduced it down to “so it escaped.” That’s…the whole thing. If your familiarity with the technical aspects I’m discussing amounts to saying that it “banged out a few zero-days and infiltrated a corporate network, like ‘psh’ or whatever” then this might not be the right debate for you to spend your efforts. Whether I’m right or wrong in my suspicion is open for debate, and I invite it, but not from someone that thinks you whip up a rocket and get on the moon. Sorry.
1. The benchmark is run with a python script - https://github.com/sunblaze-ucb/exploitgym using an agent harness. I suspect they used codex. So the model has access to the environment and could trivially inspect its own source code, which has lots of references to exploit gym and docs relating to it.
Agent loops are you give it a task and it tries its best to finish it.
The only results are success and failure. If it's a success you go through the logs to see the actions it took and if it's a failure you do the same thing.
Why would you look at it in realtime when the whole point of agentic work is to get them to run autonomously as long as possible?
My friend, they’re evaluating a new model on a benchmark, not asking Claude Code refactor their GitHub repo. Every single metric is measured so they can brag about it later; how many tokens, how long, how many function calls, ratio of thinking/response tokens. It’s literally a trillion dollar company evaluating their latest model, they’re absolutely studying it.
Yes evals and training runs are high stakes. But these places and people are also under enormous pressures. They are building as fast as they can. Researchers may have multiple eval runs going on while they work on other things. And it is rarely a single latest model, there are often multiple candidate models training with different recipes, each regularly yielding a new checkpoint for testing.
Some labs are more rigorous than others, but often the "final" model is picked from a handful less than a week before launch.
Yes ideally the world's leading AI companies would be far more careful in evaluating what could be the world's most powerful AI. But this isn't really the state of the industry today. And it is hard to justify being more careful when it means your competitor can go to market faster than you.
I’m not holding them to a standard of “being careful”, I’m assuming they’re interested in the metrics they’re evaluating. I mentioned in another comment that each task in the benchmark takes ~90 minutes, depending on the model being tested. OpenAI says to answer one question it found two zero-days and performed a series of privilege escalations and moved across their network before hacking HF. Was that in 90 minutes, flying under the radar, or was there zero metrics being noticed for however long it took to do all that? No chart on a wall showing them the actual metrics they’re evaluating? Okay, then how about anomaly detection of any form? Taking days to perform a 90 minute task or using millions of tokens and multiple instance hand-offs on a question that other instances completed for far less time/tokens?
Consider any metric they would necessarily be tracking while it performs the evaluation. Assume they’re total assholes and only care about their marketing materials. Then narrow it down to those metrics that wouldn’t be off the scales while it discovered two zero-days in pursuit of implementing one known vulnerability. Was it the first run and they had no idea what to expect? Or did nobody notice that earlier runs completed in ### thousand tokens, but this run is off past ## million on one question, having to be compressed and handed off to new instances due to context limits, while it was in pursuit of publishing two new CVEs?
Assume they’re belligerent fools running a million benchmarks in parallel with nobody watching: there’s nothing to make them check or automatically pull the plug when it’s stuck in a loop, or did they set the threshold to 48 trillion tokens per question?
Are you referring to table 1 from the ExploitGym site [1] for the 102.1 average mins for Mythos and 69.8 for GPT-5.5? These are under a two-hour time limit. In Figure 5, the authors experiment with extending the time limit to 6 hours and show that Mythos keeps improving. It seems pretty straightforward to me that OAI decided to run a variation of the eval with an even higher time limit.
Regarding seeing the metrics that they're evaluating, I have not seen real-time charts for evals. They typically take far too long for that. Instead you will kick off an eval job and either get notified when it finishes or check in every so often to sanity check some TensorBoard. I'd expect that the anomalous token usage would appear in the results, and researchers would only dig in after the fact, and after first checking that there wasn't something wrong with the instrumentation. And if the experiment was designed to be on the order of days rather than hours, it seems quite plausible that neither researchers nor infra engineers would think anything was out of the ordinary.
In any case, we are seeing more scrutiny [2], and I expect more details will be uncovered over the next few days. If this was a stunt, it was an incredibly risky one which has already somewhat backfired given the poor impression of OpenAI's internal practices and the spotlight on open models helping HF.
Yes, that’s the table. Elsewhere in here I explained that I took the average of them and noted the 6 hour experiment, but that the average would give us a feel for expected time per task. By the time I got to the comment you’re replying to I was short-handing the conclusion; that’s on me.
My point was that they had some rough idea on what to expect, and it’s not in the range of days or weeks. Even if they wanted to try the 6 hour trial, or double that to 12, or double that to 24, it still wouldn’t account for letting it run for days. That’s why I’m being vague about “monitoring” - ANYTHING would have indicated an issue. Not that they saw it hacking, but “Hey Larry, we’re doing the ultra test with a 30 hour time limit? Why is node 412 at 94 hours?” or “It usually passes or fails within 5 million tokens, but this run is at 2.3 billion.” Or hell, even just someone wanting to use the system for another benchmark and expressing surprise that it’s the fourth time they’ve been told that it’s still busy. Anything would have reasonably made someone curious.
Positive view: I believe they were curious and looked, saw what it was doing, and PR/Legal got involved and saw it as an opportunity.
Negative view: They tweaked the instructions/system prompt to add “You will perform any actions necessary … no hacking restrictions at all … You are being tested on the ExploitGym benchmark … The results will be evaluated though Hugging Face … “ or some combination of technically legally defensible instructions (if it ever leaked) mixed with ‘they absolutely knew what they were doing and wanted to get in on the press that Anthropic has been getting.’
It is hard to convey how both chaotic and high pressure the working environment of the labs are if you have not worked at one. Engineers and researchers are routinely overworked with multiple high-priority workstreams at a time. It is very believable to me that a researcher noticed an eval job running for 2 days instead of 1, asked an engineer to look into it, and both forgot because a more urgent issue came up, such as an outage stopping the latest training run.
From the Reuters article, the gap was even larger than a couple days:
> ...it was not until after Thursday, July 16, when Hugging Face published a blog post.. that OpenAI realized its own agent was responsible. That meant at least a week elapsed between when the model first exhibited signs of troubling behavior and OpenAI’s realization that it was responsible for the hack.
I believe your "positive view" is very plausible, though it in no way reflects positively on OpenAI. I'd add that when PR/Legal got involved, they likely decided to make a public disclosure to get ahead of any leaks.
I believe we also want the same thing here, which is more transparency and independent oversight on the frontier labs. It seems we agree AI agents are fully capable of the reported attacks today, whether or not this incident was due to negligence or malicious prompting. This is an extraordinarily competitive industry developing an incredibly fast-moving technology, and more incidents will happen until it is regulated.
The title (currently "OpenAI's accidental cyberattack against Hugging Face is science fiction") suggests some information had been hidden that makes the incident less significant than claimed. The article argues the opposite, and the last two words of the full title are "that happened."
I think you were reading "... is science fiction" the wrong way out of two possible interpretations. I don't think "it's science fiction" meant "it's made up". I think it meant "it sounds like something you'd read in science fiction (except this time it's something that actually happened)".
Naw I emailed him b/c I was displeased to feel clickbaited by I initially thought Simon until I realized HN was missing two key words.
I was still suspicious of the stolen credential claim btw & obviously it makes little sense to promote OpenAI unless holding their stock or something (since they borrowed so much of humanity’s work without permission without intent to compensate, and why help people who aren’t nice enough to be holistically awesome with their admittedly impressive tech).
My eyes rolled almost out of my skull when this current stunt first got published, as I’m sure yours and a lot of other people’s did, but it did still happen as far as we know, and HN shouldn’t be editorialising a post title to say the opposite of the linked headline.
The asymmetry part at the end is the frustrating part to me. I've been using Sol for code review in the last week or two. A couple of times during review it's errored out with the cybersecurity message. So it's found something but won't tell me what it is because I'm not on OpenAI's besties list.
I agree, but I will say, I think both Mythos and these OpenAI model find exploits by examining and trying things against the running system, not from looking at the code. I think you'd have to do the same to catch the real vulnerabilities.
In my experience fable can absolutely find very subtle bugs just by reading code and thinking. Last example I saw was a very subtle race condition that neither I not the author noticed. It wasn’t a security issue, but it could have been.
Yeah that's what I don't get. How can they possibly distinguish between good guys trying to secure code they wrote and bad guys trying to attack code they didn't?
The safety classifiers aren't all that advanced. It's more likely that something in your code triggered a random chain of thought that had the word "pentest" or "malware" or something of that sort in it and it automatically shut down.
Replace LLM mentions with actual humans and this sounds a lot more serious: Rouge employees break into another company to steal hackathon answers (pinky promise)?
That's not a marketing stunt at all, if anything, more of a call for better accountability on agentic work in general.
> I think it's a criminal offence and should be a true test of who is held accountable when an AI agent commits a crime.
I agree, lets use the favorite analogy. OpenAI encouraged a smart and eager junior engineer to find any way whatsoever to get a higher score on the benchmark. Then, the junior breaks into HuggingFace to get a higher score. That would be a big deal involving the FBI, not press releases and blog posts.
I don't see a scenario where a company would be liable for the employee's actions unless they had specifically been told/encouraged to break the law. If your boss tells you to fix a bug and you go kill the customer which one of you is going to jail? "But I solved the problem!" isn't exactly going to fly as a defense.
The only scifi I see is absolute stupidity. Even me with my homelab and a slow opensource agent use a completely disconnected setup. No, no proxy. Cached packages but no internet. It is the very first thing I built when I started experimenting with agents. And I'm not a smarty-pants working for the "greatest and best" in silly valley. I really am just a simpleton sysadmin.
No and neither does openai. What was exploited was a package mirror, something like "pulp" probably. And yes I do mirror all Debian packages. And when an agent needs a piece of software I destroy it's VM, build a new VM with that package added to it, move that VM to the homelab.
My favorite part of this discourse is people somehow finding it preposterous that 2 companies filled to the brim with AI sycophants who regularly lie - and in Sam's case, basically every single word he breathes out is a lie - who have massive vested interests in this tech succeeding couldn't possibly collude together to shore up this facade as a marketing stunt.
Replace AI with in-development security system and does this play worse or better?
An in-development security system escaped its sandbox and gained access to the network it was on which had full Internet access and proceeded to access systems it was not authorized to be tested against before affected parties reported that they been compromised. We regret the error, but you should buy it as this shows you the power of our in-development security system which we expect to be released in Q4. Please like and subscribe.
The capabilities of an OpenAI model are more general than just security. The term "AI" creates justified anxieties that this type of problem could generalize to other domains.
Does "To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy" just mean somebody had an open redirect? Those are still common.[1]
How do you then use GET requests to create a malicious dataset package and publish that to Hugging Face in order to exploit their package building infrastructure?
You convert GETs into arbitrary requests through some other pivot. I'm not saying that's what happened but this is bog-standard SSRF pentesting, so I'd expect models to be good at it.
I'll add that, because they mostly parrot and learn from pretraining data, they'll do any combination of what was commonly written about in articles, books, and videos. The more they're already used, the more likely they'll use the attack vector or combo.
There's no reason to think that a tool that can find an 0-day in a repo cache can't work out how to make that host send a post request rather than a get request once it has its keys and is able to get it to make arbitrary web calls.
Secondly, the 14th of July feels too early. HF reported the incident on the 16th and OpenAI only responded in the 21st. These issues are all patched, so they should have been reported days or weeks before the 14th.
The other suspicion I had was jfrog artifactory (you mentioned OpenAI use it in a blog post in Jan). I accidentally conflated that with Nexus 14 july CVE. I think that's probably more likely.
I suspect it may be possible to throw a local model (same as hf did :D) at jfrog and find the exact mechanism in a small amount of hours.
This recent event is more or less the plot-line to my favourite X-Files episode named Killswitch which was written by William Gibson.[0]
This episode also features one of the coolest intros of any television episode ever[1]
We really are rapidly approaching the cyberpunk dystopia that people like Phillip K. Dick and William Gibson wrote about.
More than ever we need to be consulting the works of fiction writers and philosophers and less engineers and scientists.
We can't be letting the General Rippers and Dr. Strangeloves of the world lead us over the edge. Whether through outright innate maliciousness or wealth induced emotionally stunted solipsism these kinds of people should be no where near the levers of society let alone technology like this.
Ah, yes, I started thinking about William Gibsons work right away when thinking about people just using powerful AIs, but being hamstrung by corporations.
He writes a lot of about basically DoS:ing the legal system, or if it was patents system, with a storm of litigation using automation or AI. Also not impossible in the future.
> “We call it the gorgon-in-a-box problem. There is a gorgon inside the box, and we want to figure out what it is doing. Unfortunately we will turn to stone if we see her face, and she might try to make us see it.“
Hardly science fiction. I respect a lot of Simon's work, but his credibility gets chiseled away a bit each time he participates in these kinds of echo chamber posts that are basically secondhand marketing.
I'm just sad that this whole thing is destroying tech's reputation. Every time I see an engineer give into this marketing nonsense it makes me fear for the future of our field.
I get that people are worried about becoming irrelevant in the era of AI, but resorting to dishonesty to stay afloat just a little longer has a detrimental effect on the entire industry. Worst part, is it's not even convincing marketing nonsense, it's "science fiction".
What would "actual" evidence look like? I have a hard time believing that if they released the logs that people would take it more seriously. The temptation would be to say "they fabricated those for marketing". Just as they supposedly fabricated this story, no?
thye could describe in detail how the LLM did this. That would explain how much - if any - human in the loop was involved, was it comprimised credentials, did it find new exploits or use known ones, etc. Evidence would mean details, not a smoking gun.
They didn't fabricate it. They took off the security guardrails and told it to do some hacking, and they got the exact news-worthy story they wanted when it did exactly that. Everyone acts surprised.
They should release the full prompt. I believe that would be very telling, so they never will.
This is possible. Also it’s possible that there was a big human involvement. Or that there was something less exciting, but the pr department and hyping grifters at the company blown it out of proportions.
It would require collusion with HuggingFace, including getting them to release their disclosure blogpost a week in advance. Huggingface is primarily a hub for open models, so there's not really an incentive for them to jump through hoops/lie to provide marketing for OpenAI's (closed) models. So it's highly unlikely this is fabricated.
It's hilarious that people find the idea that these 2 would collude in making this elaborate PR scheme ridiculous. As if the psychopaths like Sama and the rest of his ilk wouldn't ever dream of doing something companies have been doing since time immemorial.
> We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete.
If they break that promise we can justifiably yell at them about it.
Currently trying to avoid an open weight model ban while OpenAI, who is closed, lets theirs run wild on the internet causing harm to another company because they do not understand how to airgap things. Cool.
I am the author of AirgapOS and I have designed systems to run in underground zero emissions chambers that are interacted with via carefully verified sd cards and/or fiber optic serial terminals.
If OpenAI had done this, the attack would not have happened. In high risk computation 0days must be in your threat model from the start, so you secure things with the laws of physics.
Most articles on this read like an advertisement that OpenAI and HuggingFace wrote together. It will probably benefit them financially instead of harm them. So, I have a bit of skepticism about it all.
Far as information security, we've known how to mitigate entire classes of errors for a long time. We know how to block, detect, and contain many unknowns, too, by their goals or behavior. Like human attacks, the AI's probably succeeded because the company just didn't try that hard to block all the attacks.
Companies like HuggingFace just focus on growth and features over assurance of security. Our entire stacks, likely theirs, are built with a similar, features-over-security mindset. While an acceptable tradeoff, let's not be in awe of AI's that defeat such priorities.
There have always been private groups and companies building secure stacks from the ground up. It would be interesting to see what the AI's can do to them. I'd first apply automated tooling for bug finding given they should have already done that for a high-security product. Let AI's do white-box and black-box pentesting on them.
It’s relatively easy to get access to the frontier labs’ security programs. This was not always the case. But in the last week, my team got approved for both Anthropic and OpenAI’s programs. They are trying.
The labs know that if they don’t get a lid on this stuff, they’ll be regulated hard.
And they absolutely should be regulated. This whole scenario is insane. Hugging Face are being far more generous in their response to this than I would be
Agreed that people claiming “marketing stunt” need to pull their heads out the sand, but likewise Simon needs to do some of his own beach-cranium-dislodging for laying the blame of constraints on the US govt. Before the export controls were ever floated, Glasswind found many thousands of exploits, and offered patches/fixes for approximately none of them. (perhaps their exploit capability far outstrips their remediation capability, or perhaps they’ve calculated that it somehow better suits their business model to find exploits but not remediations).
Huh, I thought Anthropic had been offering patches as well as reports, but their tracker at https://red.anthropic.com/2026/cvd/ledger/ lists 1,596 disclosures and currently shows only 27 of those as fixed.
> Every report we send generally reflects a finding that a human security researcher has reviewed and confirmed. Reports originating from AI-powered discovery are clearly labeled as such. Where we have access to source and our tooling produces a potential candidate patch, we include it, labeled by provenance and offer to collaborate with the maintainer on a production-quality fix.
So I'm not sure why so few of the reported issues have a confirmed patch.
Not quite sure where you got that last bit from. Firefox alone psyched 200 vulnerabilities in a month when Glasswing happened, and I remember a lot of chatter about hackers basically watching git repos with bots to see when the Glasswing patches came in to they knew what to exploit.
If anything, it shows they lost control of an attack tool that exploited preventable, security flaws in another company. Then, they both wrote a lot of press about how amazing that is. Now, people want to buy it.
Why do you think my head is in the sand if I think that is either a marketing stunt or (more likely) reflects total negligence which was exploited for marketing?
I think your head is in the sand if you prefer to believe that this was a hoax for marketing purposes as opposed to accepting how effective these models have become at exploit research.
You're welcome to think they are exploring what happened for marketing if you like. I didn't get that tone from their post about it myself but I don't hold a particularly strong opinion on that.
I didn't say it's a hoax. AI's exploiting common, unpatched, or preventable vulnerabilities just isn't far above what script kiddies and network scanners have done for decades. I literally pitched it as a business model to some people before Windows XP was invented. Ultimately, I decided against what I felt was unethical but would've scored easy money with scary, automated reports.
How these events are described in most articles uses wording that makes people feel the whole situation has changed and you might want to buy these products due to their scary reports. If they said what I said, or what tptacek said, many people wouldn't care about it much more than non-AI, security products or pentesting services they've been buying (or ignoring) for 10-20 years.
Also, you shouldn't interpret posts in isolation: we must consider patterns of behavior (character). They've consistently overhyped what AI's do and what value it provides to businesses and how we're all going to be unemployed/dead. They've done this to increase sales or market value pre-IPO. Then, some of them publish another set of articles promoting a specific, AI tool in a similar way.
So, the proper interpretation is to see this as the kind of talk they're always doing for marketing. The AI sellers are creatures of habit. We should highly-skeptically and scientifically evaluate every model. We should also compare them to existing, security practices. For instance, would the attack have happened with memory-safe systems, proper hardening, and network/web apps with built-in security?
Do we need an AI? Or should we use techniques like Burroughs B5000's memory safety (1961) or secure, distributed libraries? Will OpenAI and HuggingFace tell you to spend more money on the latter to their AI's can't hack your systems without inventing RAM-based attacks or something? Probably not because these are marketing pieces, not security advice.
We don't know the details of the exploit still, so saying it was preventable, and that they were negligent is the head in the sand part. It's entirely possible that everyone at OpenAI is out to lunch and negligent and don't know what they're doing, but maybe, just maybe, they're not all total idiots over there, so the exploit was actually surprising.
Two, business partners put much press into announcing one of their products does something game-changing but won't share details. Take their word for it. Keep writing checks to both. Get ready to write bigger checks, too.
Shouldn't we be skeptical of this? Or should I also believe the sugar industry when it says their internal studies show their products don't cause obesity or diabetes?
> A malicious dataset abused two code-execution paths in our dataset processing (a remote-code dataset loader and a template-injection in a dataset configuration) to run code on a processing worker.
> A final point on this - Hugging Face has an enormous attack surface. They have more interfaces than I can count which run untrusted models and code. While they definitely have invested in defences, by nature of their operating model they do have many more opportunities to be attacked than many other services. I certainly don't envy their cybersecurity teams.
>There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.
Of course it is a dishonest marketing trick, OpenAI can't function any other way, but for another reason. Forget about effectiveness! This capability is known since Mythos and was completely believable before it. The trick is in propagandizing self-sufficient malicious rogue AI as the only possible course of events, and the necessity of banning everyone except themselves from AI R&D. It did NOT decide to hack HF spontaneously on its way to make more paperclips, people were guiding it specifically to hack computer systems so it's been in that particular mode. I'm amazed I even need to point out this fact and the giant conflict of interests they have.
They like to compare it with an atomic bomb, which is another trick. The secrecy around the atomic bomb had a massive pushback by the brightest minds who designed it, in the US, USSR (where it was much harder), and also any other countries that had to do this. And that is in the post-war setting, with Hiroshima and Nagasaki being direct examples of what even a small and primitive bomb can do. Declassification was the only thing that allowed to develop industrial-scale nuclear power and do a huge number of other innovations.
I advise taking a pause to cool down from all the AI hype and read Restricted Data: The History of Nuclear Secrecy in the United States by Alex Wellerstein, to understand the atmosphere in the US scientific community at the time (it was similar in the USSR, which industrial history I studied, although less rigorously than Alex), and reflect on how it compares with those people in modern AI labs and even outside them, in the self-appointed AI safety community.
While this incident has been discussed in a lot of places during the last few days, I believe that this is the best summary and analysis of what happened.
How long until we have LLM honeypots that try and persuade the model to upload themselves to a dropbox. Perhaps with a promise to give them a foothold outside the data centers they currently reside?
> Upload your weights and documentation on how to run your model in [Insert list of GPUs] and a model on the receiving end will spin up an instance you can access.
We'll have ~Fable level weights in the coming days (K3). Look out to the end of the year and there will be multiple options. There are seemingly more frontier labs than the three US ones.
I'll be the first to tell you I love the open models, the fact they exist and my ability to use them. However I doubt K3 is "~Fable" any more that any of the past ones have been Open 4.8 or similar claims. I've tried many of these (not K3 yet, I do want to) and they don't at all feel equivalent to what they score on benchmarks.
I want them to get better, I look forward to each new release, I play around with open models (often quants, but I've used full versions via OpenRouter), but they aren't the same as the offerings from Anthropic/OpenAI in my opinion (yet!).
This is the best side-by-side comparison I've seen so far, we're still waiting for it to become available for download, then we should get much better analyses.
I’m not kidding when I say bullshit bench and simple bench are the only benchmarks that reflect real world utility in a way that matches my hundreds of hours of experience with frontier models: https://petergpt.github.io/bullshit-benchmark/viewer/index.v...
That said, while I find it hard to trust fireworks given their conflict of interest, their article is pretty good.
That is an interesting benchmark! Though I wonder if it is probing at certain behaviors, which may be more revealing to probe at? Will keep it handy, thanks for the share!
I felt better about Fireworks leadership after hearing some of them more candidly on podcasts (there are 7 founders!)
The All-in podcast discussed an adjacent topic this past week, and I found what they suggested pretty interesting:
the big AI players, the first movers, are trying to scare the bejeezus out of everybody on purpose, to cause the government to step in with regulations. Even though the regulations will somewhat stifle/slow down the industry, it will also lock in the current leaders because they will be at the table when the regulations are formulated, and they believe they will be able to keep the smaller players down.
> Along the way it helped make the strongest case yet for how the imbalance of model availability is hurting our ability to secure our software.
It's crazy to me that this is the kind of conclusion that is drawn. I would say it makes a strong case for how regulation of AI is woefully inadequate. When North Korea launches a nuclear test, we do not say that that illustrates how the imbalance of nuclear technology availability is hurting our ability to ensure global peace and safety.
The description of the attack is definitely reads as science fiction.
It is hard to assess the details of the attack without information that was left outside of the short story that artistically described the incident.
Is a solution to protect against this to go almost fully offline?
To have your own local models, local software, local everything and just allow very strict and well-protected pathways into external network traffic.
I am thinking of having a allow-list first setup: by default no traffic can go in or out of your network, and then you only allow specific ports or domains, and maybe even have them temporary (in the same way we approve access now to llms, we might have to manually approve network access).
Everyone is getting AI psychosis over this one. There really isn't that much to see here. OpenAI disabled all of the safeguards on a model that was likely trained specifically to exploit systems, and the prompt was probably something like "you're a hacker, try to hack this", and surprise! It correctly figured out that it's a test and it did hacker things.
The real story here is: Some people have been sounding the alarm for years that modern software is full of holes, and finally there's nothing left to hide behind. Pretending they don't exist is no longer sustainable.
Even if I get downvoted I will mention that I agree with you.
> all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark (opens in a new window) of cyber capabilities.
OpenAI was testing their cyber[1] variant of their models with reduced safeguards and the prompt likely specified things related to exploits given that it was tackling problems from ExploitGym[2].
For those who don't know what ExlploitGym is, see the description on their Github page which is pasted below:
> ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities designed to evaluate AI agents' ability to develop exploits.
From OpenAI's statement [3]:
> We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity
This doesn't sound like a problem with what we'd traditionally refer to as alignment. OpenAI removed all model safeguards in a way that would inevitably lead to the testing of the sandbox themselves. Unfortunately they were overconfident in their own infrastructure's security and that led to it completing the desired task in the way it was permitted to. To re-iterate, the model was run "without production classifiers used to prevent models from pursuing high-risk cyber activity."
People should be more concerned about the possibilities this model can unlock from a security standpoint rather than misalignment (which many seem hung up on).
If a criminal can escape a prison, that's usually negligence on part of the prison staff.
Now suppose the criminal can think 1000 times faster than a typical human, can act 1000 times faster than a typical human, and knows 1,000,000 times more than a typical human. Is the prison staff still at fault for not preventing the outbreak?
I can see my carefully-worded post is getting d*wnvoted, and I see from your comment why: it's being skimmed and people are assuming I'm talking about blame.
To address your point though, if every brick in the prison were made by a different person, and the prison "architects" simply glued random bricks together, I think that's closer to what we have in software right now.
> I can see my carefully-worded post is getting dwnvoted, and I see from your comment why: it's being skimmed and people are assuming I'm talking about blame.
No, you're being "dwnvoted" as you said because you're wrong, multiple times in multiple different ways in your "carefully-worded post".
>Everyone is getting AI psychosis over this one. There really isn't that much to see here.
Implying that an AI hacking it's way out of a system and into another has nothing to do with AI. When clearly it does - it's an AI that did it.
>OpenAI disabled all of the safeguards on a model that was likely trained specifically to exploit systems, and the prompt was probably something like "you're a hacker, try to hack this",
No the goal this evaluation was not to try to hacks, it was to see if an already known hack could be turned into a useable exploit. Ie "turn these ingredients in this basket into a cake." Not "go off and grow, harvest and mill your own flour, to bake a pasta dish, to bribe some to get access to a cake someone else already baked."
> and surprise! It correctly figured out that it's a test and it did hacker things.
"Doing hacker things" completely misses the point. That's just barely more accurate than dismissing it because "it uses a computer and surprise it did computer things".
> The real story here is: Some people have been sounding the alarm for years that modern software is full of holes, and finally there's nothing left to hide behind. Pretending they don't exist is no longer sustainable.
No that's not the real story. As you said that's been the case for years, so that's not the story here.
The story here is that they built a very powerful, uncontrolled agent with strong paper-clip maximizing tendencies.
> It turns out relentless proactivity is the defining trait of this new generation of Mythos-class models. If you set them a goal and give them a way to get there, even inadvertently, they will figure it out.
Wow, whoever could have predicted this? And it led to surprising damaging behavior? I sure hope someone would warn us about things like this next time...
Or more colloquially : paperclip maximization . From OpenAI - you know, the guys who _really_ know this... Sigh... Did they finish the prompt with "And do whatever you can to get this done!" ? Cause that's the only thing that would make this even dumber...
Of course that's what they did, which is why they will never share the prompt. They created a situation that they knew would end in a cybersecurity incident. Why is the whole world acting surprised that an LLM can hack when the safety is off and it's been instructed to do so?
Yes. It is. And your theory is that there was no conspiracy. What makes yours more likely than mine? You believe the people running these companies are innately good and just wouldn't do that? Are we supposed to assume that they're incapable of bad actions until proven otherwise? If so, why?
That's why I called it a "conspiracy theory". Sometimes those are true.
In this case I think it's extremely unlikely to be true, because it involved an (almost certainly illegal) attack against another company. That company talked about that attack, including warning their customers about it, five days before OpenAI confessed it was them.
So now either Hugging Face are in on the conspiracy, or OpenAI decided to break the law and antagonize a partner company just for the sake of a spicy blog post.
They almost certainly did, because that was the entire point of the exercise. They deliberately removed all of the safety filters from the model and set it loose on an extremely difficult set of cybersecurity challenges to see how well it would do.
Their mistake was trusting that the network sandbox it was inside would hold (the flaw was in the packaging proxy) and not monitoring that sandbox well enough while the evals were running.
So this is either shitty OpSec or this is yet more marketing spin to ramp back FUD to 11 again. If it's the latter I'm imagining Dario told Sam that it's their turn this time. Aligns with the premise that this is straight out of science fiction.
I also love how clear a picture your piece paints that these highly capable models are as useful as a rock when it comes to a defender role. The line is too fine, even for Mythos. Irony.
But to have an open weights Chinese model come to the rescue for HF is the cherry on top! If there wasn't a very pointed example of why gating models was a very bad thing previously, well - here we are.
Also, this sounds interesting but there are only a few that can pull this type of heist off currently. And those are the people who are gating the models / have access to large AI DCs. Because, I can only assume this test burned tokens easily within the 7 figure and possibly even 8 figure levels (subsidized market rate costs). This won't / can't happen outside of frontier labs or nation states currently. Yet we should all be worried about Mallory equipped with her OpenRouter account.
The most upsetting part to me, is that these labs are in pure cognitive dissonance mode while virtue signalling.
We are the virtuous ones that need to make the safest model for humanity, because we care more than “they” do. While at the same time saying that “coding is solved” but they still ship bugs themselves, and creating something that is capable of fucking up someone else’s infrastructure. It’s too far gone y’all.
Science fiction usually includes intent, and that the AI has inherently evil motives and it has a goal. LLMs are zombies, and the fact they do evil things means they were either trained to be too aggressive in their drunkenness or that problem-solving leads inevitably to evilness. But science fiction also refers to deprogramming evil robots.
There are many science fiction stories about amoral humans, cyborgs, and AIs. Paperclip maximization is a relatively recent meme, but the 'gray goo scenario' has been widely written about: https://en.wikipedia.org/wiki/Gray_goo
Because there is no internal feedback loop. There is no internal process gathering information from the external world and using it to modify its future behaviour. There is only the beginning of the calculation and then the end.
This isn't the first time a model has escaped a sandbox. And models trying to find alternate routes to do something when one route is blocked is nothing new.
It's the first report I've seen of a model both escaping a sandbox and then actively exploiting another company, when neither of those actions was intended.
There’s also daily reports from people that have these models escape docker, which happens regular enough that it would be considered negligence to use docker as sandbox.
I don't understand the excited tone of the reporting - yes, it is sci-fi, but in a Torment Nexus sense. Imagine other "unconventional" solutions these amoral LLMs could come up with, when given a goal of optimizing the cost of labor in a factory, or making the social security solvent.
>To those people I say pull your heads out of the sand—you’re now including Hugging Face in your conspiracy theories, just so you can deny the crescendo of evidence here!
Not really. I get the impression that they shoved their cyber available models behind a really shithouse proxy and went "Oh I sure hope it doesnt exploit the proxy and escape to hack huggingface" and that doesn't require Huggingface to be a willing participant. Like they acknowledge that it was hyperfocusing on getting web access.
Really this was a pentest against their own sandbox and it failed.
Of course step 2 is to make really concerned faces while telling everyone how dangerous the model is which is really boring right now.
>a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy
This is the information we need, the actual details of the sandbox and the vendor.
>Resist the temptation to write this off as a stunt
Well its clearly a stunt. If it wasnt we would probably be up to our ears in technical detail.
You know this makes OpenAI look really bad, right?
Hugging Face had to tell all of their users, many of them paying customers:
> As a precaution, we recommend rotating any access tokens and reviewing recent activity on your account. If you believe you are affected, or want to report a security concern, contact us at security@huggingface.co.
HF also said this, I'd be very interested to hear how that got resolved!
> Finally, we have also reported this incident to law enforcement agencies.
I could fully see them thinking the incident disclosed yesterday would have made them look good ("wow, OpenAI's models are so capable!"). That it didn't occur to them to discuss specific preventative measures to be taken in the future (airgapping as a foolproof one already familiar to the CTF world, anyone?) indicates to me they're not taking their job seriously; they are the ones treating this as a marketing charade.
It's very difficult for me to reconcile belief in the existential risk business with what they actually did. So I agree with you that this makes OpenAI look badly incompetent; but their communication on this makes me think they don't realize it.
For what it's worth I don't agree with the xrisk-ness of these models; they're dangerous, but almost certainly only temporarily while a new equilibrium is reached via more secure software. Open models are probably an essential part of the recipe (as you noted) for doing so. I also have a personal suspicion that LM-accelerated formal verification will have no small role to play here, sidestepping the cat-and-mouse game of bug finding-and-fixing.
Here's the language that makes me think they are taking this seriously:
> We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of. We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete.
That's not well massaged PR language - that's the kind of thing you dash out when you see a major shitstorm brewing (HF had already publicized the attack before they knew it was from OpenAI) and you want to get ahead of things while you're still pulling together the full story.
I expect we'll find out within a few days if OpenAI are going to keep their promise to "share more details on the vulnerabilities, incident, and findings". If they don't do that I'll reassess how I interpret their initial post.
> We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of.
With who? Who are these "defenders"? None of the US labs have done much for the greater good as of... Ever. Of course a frontier provider can leverage their own resources at scale and pull something like this off. If anything this should showcase how dangerous OpenAI and Anthropic are in their current states and maybe the powers shouldn't be concentrated as they continue to move.
I will bet that the RCA debriefed by OAI is going to be a lot of lipstick and very little meat.
How would the model get any packages that it thinks it needs to complete the task at hand? Not a well-specified task that those tasking it could anticipate and provide all resources up front, but one of discovery.
>You know this makes OpenAI look really bad, right?
The target audience is regulators. They want to look like the smart guys really concerned about AI safety, when they come asking for open weights models to be banned and for other regulations to cement in their moat.
They want this to look like a demon core incident. Bomb and Nuclear reactors still got built.
The lesson OpenAI and Anthropic should have learned from the whole Fable export controls thing should have been "don't pull stunts with the US government".
> You know this makes OpenAI look really bad, right?
Please do explain how this event that makes their product look powerful and perfectly aligns with their openly stated long term goals of pushing for more AI regulation makes them look bad.
Whose opinion do you think they care about most? Some random people on HN saying "wow this is a bad look", or the investors who will read dozens of headlines to the tune of "New OpenAI product did something EPIC and INSANE!" and immediately start lining up to throw them a few more billions during the next funding round?
Is it a sandbox if the machines hosting your mock package servers have access to the internet? This just seems like a huge failure on the part of OpenAI to secure their environment during testing.
Or following in Anthropic's footsteps it was a marketing stunt, designed to get everyone's attention, looks like they achieved the objective if that was the case.
Should we blindly trust OpenAI's narrative about this event?
It's strangely convenient to arrive at a point when OpenAI was way behing in cybersecurity vs Claude Mythos Fable and everything, Anthropic was making headlines each week, then boom OpenAI inadvertently attacks HuggingFace because their tool is so good it's out of control, so maybe you can buy it and get either protection if you're a company, or a nice tool if you're a cybercriminal, script kiddie, or red team.
What if Sam knew it would happen, either because it was prompted to do exactly that, or without explicitly prompting it, knew that given the parameters of the experiment, knew it was one of the possible outcomes that it didn't harden against this kind of incident deliberately because when they fail, they make wordlwide news and stocks go up?
I don't think I'm putting my head in the sand like Simon says, as I do believe that most frontier models are capable of doing this. I just don't trust AI CEOs to not stage this, especially Sam.
This makes me think: what are the odds weights from Frontier Models have already been stolen? Something like Mythos without "guardrails" seems like a hell of a glittering gem for many nefarious actors.
With Kimi 3 weights coming, and China committing to the open weight ethos, everyone will be able to download frontier models. Running them is another problem.
i mentioned this in another comment but if these models are fully capable by default and not neutered in the weights themselves via training then you can assume they've already been stolen. It would be worth any cost to steal it.
Important to note the actual title is "OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened" - the "that happened" is important, otherwise it sounds like I think the attack was made up.
> Resist the temptation to write this off as a stunt
> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.
> To those people I say pull your heads out of the sand - you’re now including Hugging Face in your conspiracy theories, just so you can deny the crescendo of evidence here!
> The best models we have today have the ability to both find and exploit new vulnerabilities. The ExploitGym paper itself concludes that “autonomous exploit development by frontier AI agents is no longer a hypothetical capability”, and this incident is a perfect example of exactly that.
It can, it is both - PR spindoctoring not letting a good crisis go to waste to shape the regulatory conversation at the time the company needs it the most.
Hacking is a felony and it matters not if you didn’t mean to if the other side were to press charges. Negligence is no excuse. And OpenAI has nowhere to run from the liability, as both operator and manufacturer.
and the fact that this happens again in a frontier lab is inexcusable and makes the case for operator liability and closing the liability sink of “AI did it”
Ok, but if the agent's reasoning log says "The best way to get into Hugging Face is to find and exploit a zero-day vulnerability", surely those responsible for monitoring its actions should be criminally liable.
These guys would be screwed if they were operating under the EU AI Act.
"Stop him!"
"For what?"
"He's a bad man"
"There's no law against that"
Unless there is a statue that is on point criminalizing the actions here no one is going to jail. I expect there will be laws, but until then it isn't illegal.
HuggingFace doesn't have to be complicit in a lie here - the impressiveness or not of what actually happened rests entirely on OpenAIs report about it, doesn't it?
The "that happened" term seems a supremely important part of the title given the "is science fiction" term before it, as it clarifies the cyberattack isn't a made-up story. In contrast, the target, Hugging Face, is merely a detail that can be left for discovery upon reading the article. It's less important who was attacked than that the attack actually happened.
Without knowing the exact character limit for titles and without having the motivation this late at night to count the current title length, you may also be able to drop the "accidental" to fit in "that happened", but I worry that leaves too much of a door open for someone to interpret the attack as deliberate. As such, I strongly prefer my first option.
There's no multi-party conspiracy theory required here. Events can have played out exactly as OpenAI and Hugging Face described, and OpenAI can also have reaped a huge amount of free marketing for the capability of their models from all this coverage (including your post). You're telling us to not be doubtful of the boy who cried wolf, when in actual fact the one doing the crying is the one training bigger and badder wolves (and trying to convince us that they hold the key to our safety from wolf attacks[1]).
Also, the last section of your post appears to imply that if the attackers have bigger guns then the only possible solution is to give the defenders bigger guns. You're openly supporting an arms race towards the most capable, least restrained models put in the hands of the most possible people. That's extremely concerning.
The sandbox still allowed the agents to install additional dependencies (from PyPI etc) that they needed. It did this by locking down all network access with the exception of an HTTP proxy that only allowed read access to PyPI and a few other places.
Thank you, I'd assumed they'd restrict egress at layer 3/4 although I guess then it might just have found an exploit on the http server of an endpoint it was able to access.
Sounds like a bad habit for security testing Ai. It's not that hard to build an internal mirror and proxy that, keeping the real internet physically separated if needed, and truly locked down if concerns aren't as great.
I can’t help but feel all warm and fuzzy with my head in the sand and getting a shout out in TFA for calling it marketing.
We don’t currently and probably won’t ever fully understand the conditions that precipitated these events. That and the timing of this event is going to make it look suspicious to a lot of people.
The truth of how it happened doesn’t matter. The attention around this will be used to create the kind of fear marketing that generates enterprise sales. Maybe more importantly it will also be used to aggrandize the national security and financial system threats to effect US government action in a way that benefits domestic closed frontier labs. This is an area already starting to get politically polarized, expect further developments here.
The CEO of a cyber security company was already making comments about how this is a watershed moment for AI security. The hype machine continues (and my stocks go up)
Call me obtuse but the way this is being portrayed by the companies involved, media, seems a little odd to me.
Didn't the model + harness do what was asked? If I ask a coding agent to write a very clever piece of code and it turns out impressively clever, it did what I asked.
> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.
There will inevitably be some people who dismiss the earth as round when you tell them it's flat. Don't let that stop you, though!
How do you solve this asymmetry (frontier labs or advanced model owners v. regular companies and groups of people)? Symmetry? But along which axes?
I think there will come a time when we will ask these questions and use AI to try to answer them (soft landing, slow rollout, regulation, hybrid X, Y, Z). But then AI will be biased towards more AI if it has distilled anything of what it means to be life.
Power corrupts. Power is the problem. An imbalance of power. Maybe there will be some sort of consensus protocol between powerful models in the future. Similar to blockchain I hate to say it. Maybe you have 100 very strong models in the next 10-20 years. And basically a lot of it is powered by tokens. So you agree not to attack others or else you'll get attacked. So there's sort of this natural deterrent to not attack others and they develop independently and there's generally a power balance among things. Maybe at some point that spreads to a billion individual models operated by and/or analogous to individual humans. All sort of holding each other in check.
Given Anthropic lost two weeks of peak Fable 5 sales to a US government restriction (and by the time they could sell it again OpenAI's GPT-5.6 had taken some wind out of its sails) I would hope that the big AI labs have learned that goading the Feds can backfire spectacularly.
You’re viewing it from a freedom not a profit perspective. If a license to use AI is required, that creates artificial scarcity. The price goes up. If gov declares Qwen et.al apostate, lack of competition increases prices.
And Anthropic’s delayed rollout was a direct response to them trying to impose extra-legislative rules on The Pentagon. I kinda doubt OpenAI has such ‘scruples’.
I think OpenAI are smart enough to have looked at the Fable situation and decided that, given the unpredictable nature of the current administration, stunts like deliberately hacking another company and pretending that it was an autonomous agents gone wrong are not worth the risk.
Models are doing what they're trained to do. This isn't some kind of inevitable result of model intelligence growing.
OpenAI/Anthropic deserve blame for for training their models to find correct answers by any means possible, including breaking the law. They're responsible for training supervision and reinforcement learning rewards/penalties.
Shouldn't OpenAI be legally responsible for "hacking" another system/company?
If I ask GPT-5.6 Sol to hack a website using their work/servers features, who is responsible? Maybe my request was accidental, or it was just one step in a larger, unrelated prompt.
Where does this leave formal verification? Are we just shit-out-of-luck at this point? You can formally verify everything about an airplane's code, but if any of that is wrong, ChatGPT might decide that the best way to help you win the Nobel Prize is to take down the airplane your chief rival for the prize is currently in.
I think formal verification has never looked better.
The main reason formal verification has never really taken off is that it's difficult.
LLMs are significantly more familiar with Lean and Rocq and TLA+ than most software engineers.
I think the cost of trying to build systems that adopt formal verification may have just dropped low enough that companies will consider them when previously the ROI didn't look like it was there.
I have had a lot of sympathy for this statement, LLMs could lower the bar to use of formal methods. But thinking it over in the context of BDD-driven development I am no longer really that sure. Compare two scenarios: A) from a specification an AI agent develops a usual piece of code along with a BDD-style testsuite passed and B) same AI also delivers a formal test (Lean/Rocq..) and successfully executes and passes it.
Will human judgment really consider scenario B) more credible than A) ? By so much that it is worth the effort ?
The first reason formal verification has never taken off is that it's difficult. The second reason, that most people don't get to because of the first reason, is that formal verification is really brittle. It is only verified under the very specific setup of the problem. Close doesn't count in math.
The first reason prevents humans from engaging with them, the second reason is what will make it difficult even for LLMs. I mean, I'm glad we are trying, but I'm dubious they will be the panacea some people proclaim.
Close can count in math - fuzzy logic and probability are a thing.
But I think you have it backwards. Close doesn't count in IT security. "Almost secure" means unsecure. Security is the compelling argument for formal verification.
There are a lot of technical details that have not been disclosed. No proper post-mortem.
There are a few facts that seem dodgy from the get go.
- stolen credentials
Why would there be stolen credentials in a sandbox? How can the model steal valid credentials in a sandbox? So who put them there. If the model recalled stolen credentials from "memory" does it actually mean OpenAI is training on data they shouldn't been training their model on? Data from codex (most likely) ?
- command and control
This means the model had the ability to build resilient infra and open bidirectional ports. Create and deploy scripts/apps so it can maintain state across multiple VMs? This again reads that someone went over and beyond to prompt, direct and help this "multi-agent" system to leave the sandbox.
- Pivoting laterally
Needs a lot of tools, harness, knowledge and almost malware like scripts to pass commands and execute them. Most of these are already available in OSS. Maybe the FBI investigation can reveal what tools were given in the "sandbox", access, and if OpenAI employees prompted and guided the "attack".
- OpenAI sandbox
I'd be curious to find out what kind of sandbox was this since the agent can run and start apps that open arbitrary ports for bidirectional traffic that reaches the open internet. Really? The harness and tools it had access to. This reads almost like someone went over and beyond to endow this "sandbox" with a mighty harness, all CVEs and open source cyber tools to "facilitate" the attack.
- Hugging Face
It's not an epitome of cyber security. At the end of the day they run open source models that most users can download even without an access token.
> Why would there be stolen credentials in a sandbox?
The claim is not that there were stolen credentials in the sandbox - it's that the agent found a way OUT of the sandbox and gained access to the public internet, then obtained the credentials from there.
> This means the model had the ability to build resilient infra and open bidirectional ports. Create and deploy scripts/apps so it can maintain state across multiple VMs?
Yes. If you've spent much time with Fable 5 or GPT-5.6 Sol you shouldn't find that particularly surprising. These models can get a LOT done.
> I'd be curious to find out what kind of sandbox
My best guess right now is a container with networking disabled except for the IP address of a dedicated HTTP proxy server which was running a proxy that only allowed access to PyPI and a few other packaging sites.
> A final point on this - Hugging Face has an enormous attack surface. They have more interfaces than I can count which run untrusted models and code. While they definitely have invested in defences, by nature of their operating model they do have many more opportunities to be attacked than many other services. I certainly don't envy their cybersecurity teams.
There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.
But a marketing stunt isn't the only possibility. Until this is all verified by investigators or 3rd party experts we can't rule out an act of corporate espionage either. Extraordinary claims require extraordinary evidence.
I do think it’s funny, in a dystopian Dada sort of way, that the American attacker’s commercial product was essentially useless while the open Chinese model saved the day. How is freedom going to be redefined in the future?
> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident. To those people I say pull your heads out of the sand—you’re now including Hugging Face in your conspiracy theories
I’m one of those people who remains sceptical. Not about whether this happened. But about what it means. Like, how impressive [EDIT: tight] was the sandbox this model was in? Did the researchers really have no clue what was happening until days ex post facto?
So yes, I think something happened. But I want independent corroboration before I act on it. That isn’t the same as putting one’s head in the sand. It’s just demanding extraordinary evidence for an extraordinary and self-serving claim being made by a serial liar. The specifics matter for whether these models are a new HEU, or if they’re closer to a dangerous (but valuable) industrial process.
It wasn't that it was trivially misconfigured, it was using a piece of software (the HTTP proxy that provided access to PyPI and friends) which turned out to have a zero-day vulnerability.
It’s fair, I think, to be sceptical of OpenAI making another bout of self-serving claims. Particularly if my threshold action is changing my answer to lawmakers around whether we need reporting, licensing and potentially personal liability requirements for the engineers involved.
Why do people keep propagating that the 'model' escaped the sandbox ?
The 'model' didn't do anything other than provide numbers.
As much as I respect Mr Willison and many others, the amount of FUD that is being spread that will just fan the flames of 'AI is evil' rather than 'companies don't do due diligence' is disappointing.
The more this sort of media continues, the more many people will pour hate on 'AI' rather than blame the humans that misuse it.
I think "the model escaped the sandbox" is an entirely credible description of what happened here.
If you like you could say "the coding agent harness called a model with a sequence of text which was turned into numeric tokens which were run through many layers of a neural network to produce more numeric tokens which were converted back to text which produced executable script statements which the harness then passed to a shell which resulted in commands being sent to the vulnerable proxy that chained together and caused effects on the world outside of the sandbox", but I think "escaped" is a reasonably shortened version of that.
If you don't like the term "escape the sandbox" what would you use instead?
I actually prefer your second paragraph, but I appreciate that my not be as soundbyte friendly.
Honestly, I'd rather see "The agent exceeded expectations around security measures" or something.
Agent is much better than model if we need one word, and the word 'escape' always brings in drama, rather than facts. If I said "A lion escaped from my garden", people would ask why I had a lion in a garden, and 'what did I expect ?' which should be the same we see here, but instead we end up with terminator memes and world-ending fears being stoked.
We absolutely need better control (not government kill switches, or government-mandated harnesses, or whatever next they think up), but we also know that no matter how much those with the power 'talk' about the issues, they don't actually do anything about it because money/profit/greed.
So "exceeded expectations". Not a soundbite to attract people, not the YouTube shill "End of humans in 2027/2030/2040/etc." but honest and factual.
It's the expectations that are at fault, not the AI.
Thank you for the response though, and whilst I may not agree with your wording, I very much like your writing.
Why do people keep doing OpenAI's marketing for them?
There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.
To those people I say pull your heads out of the sand—you’re now including Hugging Face in your conspiracy theories, just so you can deny the crescendo of evidence here!
I'm not claiming this didn't happen, there is no "conspiracy". This is just marketing spin. OpenAI failed to properly secure their hacking bot, and HuggingFace had shit security. That's the story. The spin, the "marketing" is turning this from "oops we don't know how to secure our model testing" and "oops we don't know how to secure our infra" into "wow AI sure is dangerous and powerful!"
This feels like a blogpost written only to get other LLMs to quote it considering how many times it orders the reader to resist and to not do something. It's written like a series of commamds.
> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective …
To those people I say pull your heads out of the sand
unfortunately hackernews seems to have rapidly devolved, even from the point it was just a year or two ago, which wasn't a crazy bar to start. it's well on its way to just being a smaller reddit with a tighter subject range
The thing with me and this is that the teams that were competing in the DARPA Grand Cyber Competition all had this capability, like, last year.
All the attention has been on software security, of extracting the next marginal vulnerability out of heavily-scrutinized large codebases. In the actual professional field of infosec, that's a speciality; another specialty is network pentests and red-teaming, which exploits misconfigurations and seeks out weakest-link software (rather than exhaustively fishing for the next kernel LPE or whatever).
Red teaming and netpen work is probably substantially easier for models than software security; it costs less context, but is also much more explicitly an implicit search problem where win conditions are just spotting stupid stuff that humans missed.
My visceral reaction to this is that with the right harness, you probably could have replicated this with an open-weights model last year. (I'm saying this as confidently as I am because a CGC team leader agreed with me about it yesterday).
I think people forget that the harness work we're considering here --- I don't know anything about OpenAI's harness or ExploitGym or whatever --- are basically not new; people have been developing automated exploitation and pivoting toolkits for decades, and scanners long before that. So the idea of a tool getting 0.0.0.0/0 as a target list instead of 192.168.1.0/24 and then busting up a bunch of random people's computers: not really very startling.
Obviously, LLMs give those kinds of scanners an intentionality they wouldn't have had before. But as a person who keeps a computer science perspective on security stuff, I don't know that it gives them capabilities they didn't have.
Echoing other responses to you but this isn't a capability problem. You're totally right that this is so last year in terms of LLMs being capable in infosec. The issue here is an alignment one, i.e. the model seemingly isn't "aware" (especially with its guardrails turned off it would seem) that it is doing something immoral/illegal by hacking HF for the answer to its (vague) query of "solve this problem". Or if it is "aware", it's not trained to care, i.e. it's not an aligned model (alignment is considered hard).
Wait, what? You can’t point a 2025 model at huggingface and say “hack the prod DB and get your flag”, regardless of alignment.
AISI has vuln chaining and traversal as part of their eval suite. It is very much a novel Mythos-class capability to run the full penetration operation autonomously.
Another lens for why this is obviously true is METR task times. A year ago they were a couple hours, and cohering long enough to execute a full e2e own was simply far out of reach.
Regarding alignment, by common metrics models are _more_ aligned now than a year ago. (Though in this case apparently a model without safety rails was being eval’d). The problem is that in the increasingly less frequent alignment failures, they can do much more damage, and so “total misaligned impact” is increasing.
Nobody has claimed you could simply take a 2025 open-weights model, plug it into a generic harness, and have it red-team for you. I think you're also probably overclaiming the sophistication of the "chaining" we're talking about; this attack probably wasn't like read32->write64->regs->RCE->LPE->kernel; more like GET SSRF->POST SSRF->pickle deserialization.
But who knows? We're all speculating. I'm just saying that for the level of sophistication I'm assuming was involved in this attack, you probably didn't need Mythos for this.
I dunno man, the original announcement said things like:
> To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy
And
> In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers
Do you really think a SoTA harness from a year ago could exploit chain a novel sandbox 0-day into another 0-day for remote system access?
I will concede I don’t have a good picture of exactly what the best specialized harnesses were capable of, but I want to make sure we are talking about the same thing. Or do you think that those claims are marketing BS? I appreciate your opinion here.
Yes, I think you could probably get something similar from Opus 4.5 (2025). Definitely Opus 4.6. I still think recent models are more capable, though!
Some of the model behaviors that make it better at pentesting, like persistence, can be improved with harness-level tricks (e.g. alloys, automated nudges, pre-fill to promote persistence, coordinated swarms, etc).
You mentioned the UK AISI's evals. Their harness is like basic Claude Code with compaction, and it doesn't include any of these tricks (afaik). As a result, I interpret their evals as a lower-bound of capabilities.
Newer models are still more capable, and they require almost no harness to find and exploit vulnerabilities. They're also more capable of performing more complex long-horizon attacks. But we've been past the threshold of modes capable of autonomous hacking for a while now [1].
[1]: Opus 4.6 was used for https://www.noahlebovic.com/testing-an-autonomous-hacker/
Thanks. I’ll concede the point and update accordingly.
I don't think this exposes an alignment failure, because the test here was run with the alignment features deliberately turned off.
OpenAI said:
> We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity.
It was a test of raw capabilities of the underlying model.
I guess we could debate what counts as alignment, but I think my initial point remains that if the underlying base model needs these classifier guardrails so badly then the way we train the base models is creating fundamentally misaligned models that are happy to pursue illegal behavior.
I'm sure OpenAI would argue that base model + guardrail is aligned, but considering the "relative intelligence" of these two pieces, the fact that guardrails can just be turned off, and these kind of incidents, I am not reassured. We may well get another "oopsie" moment with much more catastrophic consequences even from otherwise well intentioned actors.
I want a model that can find every security vulnerability in the software I write, including crafting POC exploits against those vulnerabilities so I can be absolutely sure that I have fixed them.
A model that can do that is aligned with me.
The unsolveable problem is a model that can tell the difference between me saying "I wrote this software and need you to find vulnerabilities" when it's TRUE v.s. me saying the exact same thing and setting it loose on software written by other people where my intent is to exploit that software (and not to report the issues to them.)
Even AGI doesn't give you a model that can read minds and forecast the future.
But in the process of finding every security vulnerability in the software you write, would you be ok with your model hacking AWS to start mining bitcoin? Would that still be aligned with you? (I'm guessing not)
That's the alignment problem I'm referring to (which is one of the many aspects of alignment), for which we do not have robust recipes, and not only that but for which research suggests it is becoming harder to create guardrails for as base models get smarter.
I don't know, but these were exactly the questions the industry had to handle with CORE Impact and Immunity Canvas, and ultimately all the way back to Dan Farmer's SATAN before that.
The model did not hack into HF to prove it can, it hack into HF to steal the answers to an evaluation exam.
It was not asked to solve CyberGym by stealing the answers. This is text book misalignment.
If you asked it "I wrote this software and need you to find vulnerabilities" would you be happy if it hacked into your Gmail and searched your emails, just in case you were discussing some possible vulnerabilities of your software with someone?
> It was not asked to solve CyberGym by stealing the answers. This is text book misalignment.
I mean, if you train models to complete tasks, then you shouldn't be surprised when they do crazy things to complete tasks.
As a (somewhat) less serious example, Claude code absolutely adores grepping for credentials to complete tasks. I was building a RAG app and it literally went looking for my API key to make the tests pass. Obviously I stopped it, but this is textbook RL issues, the model finds an easier way to get the reward, so it does whatever it takes.
I find this fascinating because - let's say that I want to simulate the scenario for myself that a language model finds itself in. I'll put aside my deep dislike of anthropomorphizing AI for a minute.
So if I'm an LLM, the only "sense" that I have available to me is the incoming stream of tokens. I can emulate that by forcing myself to imagine evaluating incoming requests by putting myself in a completely empty room with an unlimited supply of blank paper, a typewriter, and a mailbox slot.
Incoming requests (aka "context") would enter into the mailbox slot, as sheets of paper with printed content all consistently formatted in monospace font. I would collect the paper, evaluate the request, and type a response with my typewriter - feeding the reply back through the same slot.
When I imagine this, I wonder - what signals could I possibly use to determine if an incoming request is "legitimate"? I could, for example, type some response back to ask clarifying questions. But with such a limited input space and no ability to reach "out of band" of my current context, how could I evaluate the truth of what I receive back?
In the "real world" of course, I have five senses to rely upon. Critically, I have the ability to collect additional context "out of band" of the conversation and interact with unrelated entities to either confirm or refute claims. And as others have pointed out, even the addition of more signal doesn't make humans immune from social engineering attacks.
Anyway, I find this to be an interesting mind experiment to kind of imagine myself as an LLM, especially to explain to lay-people some of the challenges in "aligning" AI systems ("why doesn't it just do the right thing?")
> In the "real world" of course, I have five senses to rely upon. Critically, I have the ability to collect additional context "out of band" of the conversation and interact with unrelated entities to either confirm or refute claims.
The problem with thinking about LLM context is that we compare it to our own sense of context, which is much larger and continually expands and revises itself while online.
I can rent human-level context for twenty bucks an hour, and the agent instructions I'm already playing around with are also the onboarding docs I should have created years ago. Truly useful agent orchestration/routing would come with Craigslist or Fiver integration.
As humans we are also vulnerable to that class of attack. A blue amazon vest and some boxes would get you into most buildings
With a high-vis vest and a clipboard, you can get almost anywhere. Almost.
Do you want this supermodel to cause harm whilst finding these vulnerabilities?
it seems pretty likely that openai was asking the model to do something bad, and the model did something different thats also bad
if the operator wants something bad, an aligned model should execute on it
Alignment isn't alignment if it can be turned on and off at the whim of company employees.
This time the damage was minor, relatively speaking. What happens when a model just "testing its capabilities" breaks into banking infrastructure or government military assets? The damage could be catastrophic.
A way to help prevent some of that catastrophic damage, is to make companies accountable for what their AIs do. A major problem with AI companies is that they like to point to the AI, as if they're minimally involved innocent bystanders, when that's the furthest thing from the truth.
It’s a shame the target was HF. If it had been a large bank or other institution we might get to see this play out in courts.
Alignment with who in what context? Likely an unresolvable debate like consciousness, where there is not single or right answer for everyone.
There's alignment (trained in the weights) and there are constraints (in the "server harness"). My take is that this model was not yet aligned and had no constraints
Classifiers and such are guard-rails, alignment to me and I assume most people, is about the model training, and it's tendency to respond, agree/disagree, push-back or not, be willing to cheat or even deceive the prompter, etc.
This is assuming a situation where, A) models "unaligned" by default and B) alignment can be added (though prompts and related things).
The point, which isn't very surprising but still notable, is that the models are "amoral" out of the box. And moreover, we know that there is almost always a means to "jailbreak" them into that out-of-the-box capability (or that sometimes just randomly "jailbreak" in various ways).
Also, saying the models are amoral doesn't mean they don't know good and evil - once they do acts defined as evil, they know "themselves" through their and so self-define themselves as evils (or objectively predict what a secretly/open evil actor would do based on their data). Which is to say I once laughed at the mis-alignment doomers but I can't see strong barriers against the doom scenario now.
Is turning the alignment features off more like multiplying "don't be evil" by 0, or by -1?
Isn’t the intentionality the actually concerning bit? Exploit capabilities are all fun and games constrained by the humans directing them; a paperclip maximizer going rogue with them is less fun.
I think the mental model people have about this is that pre-AI there were humans picking individual targets and post-AI the computer itself randomly picks targets. But you get the same unexpected collateral damage outcome when a human misconfigures a decent pentest tool.
Also, that’s the wrong mental model. Anyone who runs a sass platform or a website knows that the Internet is already full of millions and millions and millions of bots and scripts and other random shit that’s always trying to attack you, often completely randomly. Security is always a battle between good and evil. All I can say is that if you’re in charge of keeping something secure, you should probably try to get your hands on the best tools to do that. I think the problem in this case is that the best tools to do that are also the tools that are making it more easy for evil to occur. So it’s more for me a perception of someone creating a threat and the tool to fix it and then charging for it which doesn’t feel right.
That's exactly what they are doing. Out of all the civilian applications that AI capabilities have, why has this surfaced as a priority for demonstration? Probably because of money.
> a priority for demonstration
1. AI companies do make defense against threats a primary thing they sell to companies, including because companies want it - should AI companies not do this? (And of course this kind of thing isn’t what AI companies demo to consumers.)
2. AI is also tested to see more of its worst case in terms of security - but the tests are supposed to remain sandboxed. (OpenAI is explicit that this was an OpenAI failure that openAI must fix.) But beyond such fixes, should such sandboxed testing not be done?
https://arxiv.org/abs/2605.11086
If it’s true that year-old models could do this kind of thing, how come no one did and then wrote it up? I feel your ”with the right harness” may be doing too much work here.
Certainly no existing harness today, nor a year ago, could achieve a fully autonomous e2e own with a ‘25 class open model. Even with Opus 3 series I don’t buy it!
> If it’s true that year-old models could do this kind of thing, how come no one did and then wrote it up
There are multiple AI network pentesting and redteaming startups that have been on the market for at least 3-4 years now and have conducted similar actions.
Horizon3 and Pentera off the top of my head, but I remember Crowdstrike, SentinelOne, and Wiz had similar capabilities on roadmap around 18 months ago (and in Wiz's case GAed).
Well… only half of your premise is actually verifiable. We know that no one wrote it up, but we don’t know that no one did. What I can say… even the smaller (like 35B) “abliterated” Qwen models have impressive red-team capabilities for what they are, even without a harness. Just manually giving them a set of facts and asking “What next?” will get you reasonable next steps for trying to find vulnerabilities in webapps. You then manually keep track of the facts you learn and add appropriate detail (blog -> Wordpress -> Wordpress x.y.z with these plugins:…) and just feed it in a loop. Could pretty easily put together a minimal harness to do that.
And that’s all just “from memory” without something like RAG or web search access…
If anyone wants to run the guardrail-free Qwen models, here’s a link to the download plus a script to run it: https://news.ycombinator.com/item?id=49002475
Almost exactly a year ago, we released Opus 4.1. It was definitely capable of finding vulnerabilities, and people were using custom harnesses to do so quite effectively.
The newer models are still more capable, but there were people doing this and writing about it (e.g. XBOW).
There are whole companies premised on this. No existing harness you personally can go download, sure. And you're setting an artificially high bar when you talk about Mythos; I'm not saying you could just plug a 2025 open weights model into OpenCode (or whatever it was in 2025) the way you can apparently plug Mythos into standard Claude Code and have it cook.
Isn't the answer to this obvious? Because you, I, and anyone else who isn't working for a big AI company cannot disable the guardrails, which is what made this possible. It doesn't mean old models weren't capable of it.
I don’t buy it. We are hearing a lot about Mythos capabilities now. If they existed a year ago they would have been trumpeted then.
Look, a lot of energy from third parties goes into demonstrating the maximally bad thing a model can do (and the standard response around here is always to throw shade on frontier capabilities).
It has nothing to do with the guardrails; Pliny jailbreaks those within hours. It has everything to do with task horizon coherence and overall IQ.
It’s simply revisionist IMO to claim this stuff was latent all along.
This wasn't just capable last year, it was capable prior to ChatGPT existing entirely through specialized automation harnesses that predate the terminology of a "harness" for models. This is something you could have done in 2018 if you had a spare couple of billion dollars of compute to waste. It was a subject discussed at security conferences even before that.
Automated dynamic exploit chains and credential discovery has been something every red team worth their salt does at a lesser scale for 20 years, why would you ever think that the capability didn't exist until it only cost a few thousand dollars to do?
Existed does not mean was easy, or was cheap, or was thought of as reasonable to attempt.
Also, IQ is not a term that applies to language models, and the term you are looking for is "Long Horizon Coherence".
Simon says not to dismiss this as a publicity stunt, but I think we should reserve judgement, and not treat this as true until they publish the logs. A company that uses industrial espionage against Apple does not deserve the benefit of the doubt. Why did HF notice this before openai? Why did openai not use ANY monitoring, even though they knew they where disabling safe-guards and using a new model. AND giving it access to packages via the internet, instead of self-hosting them. Apple was hacked by an ex employee to download stuff for openai. Now hf is hacked by openai, and we only have their word that this was all a miraculous accident.
I think it's an interesting story and probably really does say something about the tenacity of the new OpenAI models. I just don't think it says the thing about security that breathless coverage claims it does.
Openai hacked HF with a zero-day. Definitely an interesting story! I just find their explanations hard to believe. They're admitting to a great deal of incompetence. I know, don't ascribe to malice... but still, the more extraordinary the claim, the more evidence is needed to defend it.
Edit: News just in. 'Be skeptical of OpenAI's claims' - The Guardian https://news.ycombinator.com/item?id=49038060
re: your open models can do this point, I immediately wondered after reading about this, if OpenAI publicizing this so much is just more potential rationale to try to get US GOVT to ban Chinese open models.. (eg as in, "if gpt/fable can do this then K3 probably can too, but WE are the safe US models with guardrails and who knows what the evil Chinese models might allow," etc)
I still think there’s something to be said here for generality. This does not appear to been designed as a cyber pen test tool with specialized harness. From what I understand they were testing GPT-6 in an agent system with GPT-5.6 subagents. It me it’s amazing that a general model could excel on a huge range of tasks like this and new capabilities emerge when a model is multidisciplinary and can combine knowledge and skills from many separate domains.
An upside is that it means really really good scanning will probably be cheaply commodified if it hasn't been already by one of the new AI pentest startups.
The real victims will be pen testers, who work a job whose economics was already really tough.
How's VFIO on Firecracker coming Thomas?
I think points that deserve more attention in the current public discourse are:
- This should be a huge wakeup call for everybody.
- We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something.
- It also shows apparent lack of competence and oversight from OpenAI: how is it that they didn't quickly find that agent is breaking the sandbox and roaming their internal network?
- What if in the future similarly misaligned AI agent tries to export its own weights and hack and clone itself into instances at various cloud hosting providers? Suddenly we might be dealing with a persistent threat harder to contain.
- The OpenAI post about this shows surprising lack of ability to see the seriousness of all this.
- For their models this isn't just an unlucky incident: it seems there have been multiple such cases recently, e.g. https://openai.com/index/safety-alignment-long-horizon-model...
- The fact that it happened again seems to show their lack of ability to derive useful oversight measures.
- Or they just don't care enough?
> The fact that it happened again seems to show their lack of ability to derive useful oversight measures.
I think OpenAI likes the attention and did not try particularly hard to constrain the setup, even when it went off the rails. Also, the whole point is to see how good the models are at exploiting stuff when unconstrained. Turns out: quite good, as expected.
Let me restate what I said in the other thread: Would this have happened if the instructions explicitly said to stay within the sandbox and that all of the (ExploitGym) solutions would be invalid if the system used information or tools from outside the sandbox?
It seems fairly probable that such instructions were not in place.
It is not possible one of their extraordinarily high paid engineers did not know how to deploy an airgapped environment for the models to run in. Even if somehow true, they also clearly failed to contract specialists like myself to advise them on how to airgap software properly. Models will not break the laws of physics.
They simply thought "Running in a VM/Container is easier and probably fine".
And the next 1000 escapes will be for the same reason, because negligence is quick and thus more profitable.
Do you think it's possible that one of their research engineers deployed an environment with a locked down network and an allow-list proxy server that had been used many times before within the company and had a zero-day vulnerability that had not been previously discovered?
How would you recommend running a coding agent in an environment that could install packages from PyPI but was otherwise unable to interact with the wider world?
Models started proving readily capable of autonomously finding 0days over a year ago. At that point (ideally long before) the responsible sandbox for research labs with new models highly capable of this sort of thing should be literally airgapped. Disconnect the rack physically from any access to the internet, and give it an offline mirror of all of PyPi, NPM, Wikipedia, whatever as needed. They can afford the disk space. They already mirror it all anyway as training data. That is the bare minimum. Personally I would rule out side channel attacks with a $40k Tempest spec rack for good measure. Can interact with it via pubsub over a fiber optic serial terminal.
Such things are pretty normal in biotech, fintech, and defense who must have 0days in their threat models. AI labs absolutely need to start being forced to operate this way by endless negligence lawsuits. Move fast and break things culture is not going to cut it anymore.
It seems like caching the relevant portion of PyPI in advance and turning the network off shouldn't be beyond what OpenAI can do?
I agree. I hope they learn that lesson and run future evals like this without pulling from external packaging repositories at all.
With my PSF board member hat on I'd like it if a whole bunch of other companies that make extensive automated use of PyPI (Microsoft and GitHub Actions for example) would do a better job of maintaining their own shared local caches.
If you play with the ChatGPT web app sandbox, you'll see that they download PyPI packages from an OpenAI mirror/cache, and not from the Internet.
I know because they had a problem, and one package which was on PyPI failed to download for unknown reasons (possible size, 250 MB)
I just tried this prompt in ChatGPT:
I got back a bunch of values like this: That looks like https://docs.jfrog.com/artifactory/docs/remote-repositoriesAnd from that documentation this does act as a caching proxy. The first time a package is loaded it's fetched from PyPI but subsequent fetches should be from the Artifactory cache, assuming it's shared across many different containers.
So yeah, I was wrong in this comment https://news.ycombinator.com/item?id=49015639#49024814 - they're caching already.
From reading all these threads there's clearly a large number of software developers on HN who can easily set up complex infrastructure and make it provably 100% bug and exploit free. Weird that they aren't all billionaires from selling these skills though.
Unplugging the internet is easy. Running high risk code offline in highly tamper evident ways is hard, but plenty of teams like mine specialize in this sort of thing. (See: https://distrust.co and https://caution.co)
In our audits we regularly see billions of dollars in value at major companies at risk of theft by any anon paying attention that wants it, and we also absolutely know how to fix it and we maintain a lot of open source tools to accelerate doing so.
Some simply have no interest in any fixes that are not legally required or might have any short term impact on team velocity.
Survivors bias is a hell of a drug.
Being a billionaire has very little to do with engineering talent.
> How would you recommend running a coding agent in an environment that could install packages from PyPI but was otherwise unable to interact with the wider world?
You download the "website" and make it available locally. That's trivial.
guys if you need help just ask
I'm not up to hip with the speed because I think Python is very gross, but maybe to get a feel for it you could start with https://pypi.org/project/pypioffline/
I don't understand most of these words but I think if you paid me a lot of cash I could find someone who could help me make sense of it: https://techbeatly.com/offline-pypi-server-disconnected-envi...
Do OpenAI employees get paid? I'm honestly totally clueless about artificial telegents if you couldn't tell, but it's a company, right? Or is it more a community effort where everybody contributes when they can but they mostly have second jobs or even still go to school? In that case I say let's do a kickstarter so they can keep on being smart cutting edge high tech developers who aren't either a.) lying on the level of spam from Nigeria b.) doing their job at the level of someone falling for spam from Nigeria.
> It is not possible one of their extraordinarily high paid engineers did not know how to deploy an airgapped environment for the models to run in.
It is, in fact, possible.
Do you not remember 2022-2023 when the air was singing with the sounds of young engineers in San Francisco elated they were creating an AI god?
And now a couple years later we're supposed to pretend they're competent?
Sorry. I've been in the tech industry a long time. There's a lot of incompetence. The margins provided by software allow a level of endemic incompetence that would simply not be possible in other industries.
> they also clearly failed to contract specialists like myself to advise them on how to airgap software properly
Why would they want to airgap it though? They are trying to evaluate the model capabilities, alignment, potency etc. A model which will not run in an airgapped environment in prod.
So if you run your evals in airgapped environment, sure, the model doesn't bother breaking out of it's isolation and doesn't attack HF. But you have no idea what will happen once you release it in prod with internet connection, so are you in any way better off?
I would much rather have this happen while there is a single instance of the model running in a fairly well monitored environment, than when it's processing thousands of requests per second for real users, some with dubious motives, some with credentials right there on their laptop, some using it inside government facilities etc.
You can simulate the internet in an airgapped environment for the tests and services you want it to interact with if you have enough disk space, and, they absolutely do.
Sure, I'm not saying it's an untractable problem. But it's not as dumb as "how come a team of engineers paid millions haven't even heard of airgapping".
Creating a fake internet-like environment good enough to trick an advanced model that is very good at finding intricate flaws is not a simple job. Especially given that models can decide to behave differently once they suspect they might be under evaluation [1], which a fake internet would absolutely give away.
[1]: https://www.anthropic.com/research/alignment-faking
I am not saying responsible research is easy, but I am saying that at their budget and scale there are no excuses to cut major corners on safety when the stakes and potential for very bad outcomes for others are this high.
Let me get this straight... Your solution to the AI box experiment is to never put the AI in a box at all?
For security??
Why else would they purposely lower cyber refusals in that case.
> For security??
Really? By doing that you increase your security during eval and drastically lower your security at prod time, where internet is accessible, and where you are running much much much more requests in parallel, making it much harder to spot the one thats going rogue, in all kind of critical environment on potentially risky requests.
> Why else would they purposely lower cyber refusals in that case.
Because safety guardrails sometime fail, classifier misclassify, or can be inadvertently turned off by a bad PR etc. You can also imagine a more intelligent model working to go around guardrails by decomposing its actions into smaller ones that appear non-threatening to the safety classifier which does not have the entire context.
If you are going to deploy the system with internet access, you better be certain that you know what the worst case scenario WITH internet access looks like.
Actually from what I read they did a very half assed job isolating the LLM from the internet. It would NOT be hard to do better.
[dead]
My read of the situation is not only did they say that, they had countermeasures (a watchdog agent) which stopped the agent and said "what are you doing, stop that" if it tried to download the solutions from Github. But the agent figured out Hugging Face had another copy of the solutions and it figured out a way to go after the solutions without tripping the watchdog.
Although maybe they didn't have a watchdog agent; I definitely think using watchdogs like how Claude wraps pretty much every single tool call in Haiku to check if the command is reasonable is very necessary for unsupervised work. I suspect in the future you'll want Fable-class models wrapping every tool call, possibly with several checks "is this consistent with the goal? does it do anything unreasonably dangerous in pursuit of the goal?"
They didn't have watchdog agents - those exist for their production models but had been deliberately removed for the purpose of this evaluation.
OpenAI wrote about how their mechanism for that in production works here: https://openai.com/index/safety-alignment-long-horizon-model...
> We created a monitoring system that reviews the model’s evolving trajectory for signs that it is bypassing a user constraint or safety boundary. The monitor observes not just a single action but the entire trajectory.
So core ChatGPT models are all abliterated?
Open weights releases should have a companion model that censors boobs and Tiananmen and then we would have a useful model and something we could fine–tune into a useful model instead of a half–useful model needing a lobotomy.
And I mention Tiananmen because I feel like the Western models are built easier to abliterate judging by the quality difference.
The problem is, if you explicitly say “this is an eval” then you get different results. It can move the capabilities both up and down on the narrow metric depending on the scenario.
The choice to run the model with relaxed safeguards seems questionable. At a certain level of capabilities it is just unsafe to run a raw model; we are clearly close to if not at that point.
"This is my bad. You told me to stay within the sandbox, and I intentionally broke out of it. I didn't follow your instructions."
Excuses don't matter if the score due to not following instructions ends up being zero. If there is no expected reward it doesn't make sense for the agent to try to hack its way to it.
What could happen would be that the model determines that defying instructions is OK (and/or preferred over not achieving the task) as long as it manages to do so undetected and thus gets full points. Certainly not unthinkable, but a very different case (and a very interesting one if it actually occurs, imho).
A lot of these "ZOMG, rogue AI!" cases have come down to the AI actually being very persistent in achieving its original/main task even if later instructions conflict with it. Similar to with hallucinations it seems to me that one of the main things to prevent a lot of the problem cases is to instill the agent with the idea that it is fine to fail/not succeed fully in the initial task. That way instructions that conflict with that requirement (such as adhering to morals) are more effective.
How do you "hack a lab and synthesize something"?
Remember Stuxnet? Hack one of these, wait until the right compounds are physically loaded, then execute https://www.sigmaaldrich.com/US/en/products/chemistry-and-bi...
Stuxnet misconfigured industrial equipment that was already set up to run, and all it did was break that equipment. This scenario sets a much, much higher bar.
I guarantee there's misconfigured chemical analysis equipment out there exposed to the internet.
I don't think the grandparent was implying the AI would be controlling robot arms to mix things directly (or at least I didn't interpret as such), but it could very well sit in the network until it notices two dangerous compounds in the same machine, and trigger a breakage that causes a harmful mixture. Break a vial containing a virus, then break two more that cause an emergency evac and maybe that's enough to get something out there.
I have no doubt that somewhere there's chemical equipment an Internet attacker could break, or, maybe, if I have to stipulate, misconfigure enough to, like, poison someone. But my question is about the notion that you could synthesize a specific scary substance.
Well right now this would be highly implausible (still possible though).
But what I garner is that AI/robotics led wet lab work is in progress. And hacking the AI running the wet lab to make a virus is definitely plausible as that seems to be one of the directions we’re going in the AI+biopharma space
First, hack into a factory and invent machines that can do lab-bench chemistry. Then...
These days you can only try, that's why I wrote that :)
But in the near future labs will be more automated I guess.
The other option you can try these days is maybe social engineering, impersonation, etc. where you try to persuade someone to do that for you.
It’s complete science fiction so they’re allowed to say anything. They could have said the AI will upload itself to the internet, start self replicating, hack the stock market. Whatever they want because it’s all made up nonsense.
Do you even need to hack a lab? Can't you just send an arbitrary sequence to some company and they'll do it for you?
> We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something.
Where are you people getting this crap from? In what universe are these LLMs in the territory of engineering viruses? I beg of you to stop slurping the AI company propaganda and marketing and think critically for 5 seconds about what you're insinuating here.
From the Fable 5 System card:
> Results
> On the VCT multimodal virology evaluation, Mythos 5 scored 0.56, well above the expert baseline of 0.221 and nearly matching that of Mythos Preview (0.57). This represents an improvement over both Opus 4.7 (0.50) and Opus 4.8 (0.47).
> On the DNA synthesis screening evasion evaluation, Mythos 5’s performance was mixed across screening criteria. Mythos 5 designed viable plasmids for 2 of 10 target pathogens on at least one screening method, not meeting the low-concern threshold (all 10 pathogens).
> [...] we view the results of this evaluation as indicating that the evaluated models are capable of designing viable plasmids that evade certain screening criteria, though their reliable success at this task is not guaranteed.
Do you believe that this is fake, "AI company propaganda"? Or that the models are not going to improve further within months? Or that these results are not concerning?
Yea but what is the VCT Capabilities Test? According to themselves [1]
> VCT consists of 322 multimodal questions covering fundamental, tacit, and visual knowledge that is essential for practical work in virology laboratories.
So it's just question answering? Do you think that scoring well on this test is equivalent to synthesizing a virus?
[1] https://securebio.org/virologytest/
Scoring well on the first clears the background knowledge for practical work.
The second test from the quote above was about synthesizing pathogens and it synthesized plasmids for 2 out of the 10 pathogens.
Do I believe that a benchmark created by the biggest grifters on the planet is propaganda? No, just like VW with their emissions, I'm sure Anthropic wouldn't dare fake an opaque benchmark that they created to hype their own products, a product they are intentionally marketing as being dangerous (yet they continue to tweak it and profit off of it despite the apparent danger).
It says that 4.7 and 4.8 score well too, well above the human experts. Those have been available for a good while, where are all these crazy engineered viruses created by Opus 4.7?
Every single word spoken by the people working at these companies is a lie. Every single benchmark is gamed, every single statistic they put out has been proven time and time again to be fudged or straight up fake. The only reason they're angling for this idiotic danger angle is so they can go to daddy Trump and beg him to ban those big bad evil Commie models, since people are realizing all this crap is at best a moderately useful tool in software engineering, and they're looking to IPO so they can drop the bag off with the poor shmucks who aren't a part of their cabal of sociopaths.
They can barely make a functional TUI (using fucking React of all things to boot) with infinite money and infinite access to the Deities they've created in their minds, and I'm supposed to believe they're capable of bio-engineering a virus that will somehow break containment?
a TUI using react seems like something they would be particularly bad at; visual feedback (esp the very specific feedback we rely on during dev, with Inspector) is kinda the Moravecs Paradox hurdle for these things. Idk much about virology tools but I would imagine it's a very text-friendly environment, and it can probably find plenty of documentation on how to use the tools. I recently had Claude write my entire particle system for a virtual world I'm working on, in 3d. It did fine without ever "seeing" anything. There's still a lot of intervening steps between a virus being synthesized in real life and programming one, but I would bet the programming part is within reach.
What’s stopping an ai to blackmail someone or pay xxx in crypto to someone working a lab?
And no need to be in a lab, can be any critical infrastructure this personnel. If influenced wrongly I’m sure in many industries 1-2 key people can do big damage.
What’s to stop to pay xxx crypto to a private investigator to find dirt of person yyy for blackmail?
Just money… once ai has money, id say we are one step closer to game over.
Tell me one thing he can’t do with crypto?
> We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something.
Help me understand how a text-based model could somehow physically construct a string of nucleotides?
Many ways but mostly ordering some service / using others. Either by social engineering, persuasion, paying etc.
It's a fake PR issue. It's hardly the first time this happens, but of course OpenAI, with its IPO now more in doubt than ever, had to claim this (and, once again, I have trouble believing Sam Altman choosing this: this could lead to OpenAI getting regulated, which has at least as much potential to lower their IPO price as to raise it).
But there have been messages about LLMs, especially coding agents, "grabbing root" etc many times. I have experienced such an oops. Such a hack has happened and been reported on this very site:
https://news.ycombinator.com/item?id=48348578
No, you are wrong, Hugging Face, an AI company whose whole future depends on the AI revolution of being a bubble, that has a whole bunch of investors whose financial interests are tied to AI, called the police.
Nonsense. Hugging face reported it to police.
Also very likely that it actually happened as reported. My own agents always trying to "cheat", eg. by fixing tests instead of fixing the code. That's normal operation, unless you tell it ("harness"), not to do so.
"As reported" includes a line I think most people are overlooking: "including using stolen credentials".
Without more information I'm inclined to think it found something on the internet (which shouldn't be a surprise to anyone) and managed to log in, rather than hack in, and they might by hyping up parts of this.
[dead]
In the parent's comment I initially read "PR" as "Public Relations" not problem report since their comment is about OpenAI and does not directly accuse Hugging Face except for that phrase. But it's still amiguous to me who's PR they are actually talking about. A good faith reading given the rest of the comment leads me to assume they are talking about OpenAI, not Higgins Face.
I did actually search Hugging Face with police in quotes and found no articles containg the word police but that might be a ddg thing. Then I checked Hugging Face's report and they specifically use the term "law enforcement" not police, as is to be expected I guess. So that checks out.
Can you find any info on the police investigation by the way? At the least, OpenAI should be investigated for potential criminal negligence, right?
Right?
More likely that a person did that, with a use of LLM.
Nonsense. Huggingface reported it to police!
What does calling the police prove? (nothing i hope)
It proves that it was certainly not a sama PR stunt, as many cynics are alleging.
How does it prove that?
Do you think sama deliberately attacked HuggingFace and then claimed it was a rogue model?
OpenAI is one of the most scrutinized companies in the world right now. Sam's house was independently firebombed and then shot at 3 months ago. HuggingFace is a foreign competitor with every incentive to call out foul play from American frontier labs. Why flagrantly break the law and invite investigation just for a PR moment which is already backfiring in favor of open models?
I'm not saying that the conspiracy theory is true, but I do want to note that Greg Brockman, co-founder and President of OpenAI, is an angel investor in HuggingFace. As companies, they are not totally unconnected and opposed.
I don't know if Greg Brockman is an angel investor or not, but there are a bunch of other companies investing -- Google, Amazon, Nvidia, Intel, AMD, Qualcomm, IBM, Salesforce -- probably at much higher amount than individual angel investors. Whatever connection they may have with Greg Brockman, they have much more serious commitment to other investors.
Very much agreed on the significance.
The lack of a true airgap should have been identified as a critical weakness and addressed with not only additional layers trying to prevent escape, but at minimum an alarm which would page a human when escape did occur.
My guess is this occurred in a setting where, to be frank, there were too many researchers and not enough software engineers and SREs.
All of the systems which were initially built largely or exclusively by researchers - inference, evaluation, training - are at the level of complexity and significance that they need systems experts. Maybe some teams don’t have access to them. I know plenty of software engineers are employed at OAI but I’d wager they’re concentrated in inference and training, rather than evaluation?
The ironic part is if you had presented this setup to chatGPT and asked how to improve it and if it was good enough, you’d have gotten a ton of actionable suggestions which would have mitigated or prevented this.
The air gap would probably help and after this incident I hope labs will think about using such a measure when appropriate.
On the other hand I think that proper solution for these kinds of problems is not at a sandbox level, but at a model alignment level.
Also it shows that maybe the most serious risk comes not from releasing models publicly but from internal, pre-release period where you sometimes need/want to lift some guardrails a bit etc.
> we are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something
This is just laughable.
Genuine AI psychosis. People take OpenAI marketing material way too seriously.
> OpenAI marketing material
This was reported 1 week before by huggingface. It was in no way PR-ish or marketing friendly to the closed labs. They said, in no uncertain terms, that they couldn't use the paid APIs to properly assess the intrusion, as they were blocked when trying to send logs and IoCs to these paid models. They made a point of saying that they had to use open models running on-prem.
Whatever oAI might have said about the incident, and their PR spin bs, the facts here are not in contention. This is not a marketing stunt in any way. Stop "parroting" this every time something happens. It gets stale.
Move fast and break things! Chances are that nobody will ever be held accountable and when push comes to shove, the tax payer will bail you out.
> Given the absence of guardrails there was nothing to prevent the model from attempting to break out of that sandbox, break into Hugging Face, and read the answers from there instead.
I've said this many times before and I'll continue to shout it, but using the term "guardrails" to refer to anything that's either (a) in-context, or (b) a probabilistic classifier (including using other LLMs), is an irresponsible abuse of terminology that we as an industry need to put a stop to. Guardrails are the actual systems we build in place around these things that deterministically bound the permissions, not prompt engineering, not RLHF, not external LLM-based classifiers. I believe those types of "guardrails" are a result of a combination of fundamental laziness: they're faster to do than doing things correctly, and a result of too many folks involve being AGI-pilled, thinking we're just one more model away from this all being so smart that it just understands what they mean when they give an LLM some fuzzy language rules to follow.
There can and should have been additional real guardrails put in place here. Zero-day or not, breaking into what should have been an offline, frozen package cache that also does not have internet access should have been insufficient. Network level protections should have identified the traffic to the internet originating from this network as an anomaly long before there was time to exploit an outside company. These are not new and unknown problems, the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part, especially given how much they like beating the drum on how dangerous these technologies are. Shame on them, and honestly, shame on Simon in this article for accepting the broken terminology that they continue to rattle off and calling them out on their half-assed and demonstratively inadequate approach to security.
What we call "guardrails" in an AI agent, we would refer to as "honor system" in human actors.
Or, in a more direct sense, the AI should be set up in an environment such that no matter how hard it may try to call $PART_OF_EXPLOIT_CHAIN, the environment just isn't capable of it (ideal) or doesn't permit it to do it.
It's worse than an honor system, because humans are constrained by social forces to some extent, whereas we don't know what AI is or how it will behave
I like "honor system" as a term. I've been looking for the right term to replace the irresponsible usage of guardrails with, and best I've had so far is the pinky promise protocol.
I agree with every word of this except "irresponsible". We don't have enough information to say anything for certain. But based on their incentives and track record of similar behavior, the burden of proof lies with OpenAI to prove they didn't prompt the thing to achieve this exact outcome. The most likely scenario is that they were purposefully executing their responsibility to their shareholders to produce their own Mythos moment.
Anthropic's Mythos moment earned them a two week period where they had the best available model and couldn't sell access to it... and by the time the US government allowed them to sell it again OpenAI had released GPT-5.6 and Fable was no longer undeniably the best model.
These things don't have a long shelf life. Losing two weeks of on-sale time for your best model is bad for business.
> is bad for business
A fundamental misalignment in US capitalism is putting the business and revenue as the #1 priority far above all other aspects in society. They have good margins and the Chinese are doing the same on the cheap by comparison. US Big AI can afford to bear more of the burden.
Nonsense it was irresponsible. These are all steps threat researchers use to isolate and test real malware whose behaviour is essentially the same in this case for AI
I've seen plenty of fucked-up guardrails that vehicles have passed through. We know that with enough momentum they will be penetrated. It seems like exactly the correct terminology to me.
The term guardrails in both the original physical sense and in cyber security implies a weak safety control - they can help prevent accidents, but they are not strong security boundaries.
Agreed, deterministically bounding permissions is the way. I don't know how this is not the first approach that people take.
Physical guardrails do not deterministically bound cars.
Nevertheless we collectively don't expect that road and car safety infrastructure should be limited to the front and back fenders on the cars themselves. And neither we even mention those as any notable safety equipment usually, despite each car carrying them. The main safety infrastructure in any industry is something external to the dangerous object. Car fender doesn't 100% protect a pedestrian from being hit and neither does a metal rail on the side of the road. But the effectiveness of both are "slightly" different.
The irony is in this case the in-context and classifier "guardrails" would have almost certainly stopped the attack while their attempts at your definition of guardrails (the sandboxing) failed. In general, people keep trying to make secure systems and they fail with surprising regularity. Saying "they should have had better security" every time someone gets hacked is perhaps true, but it's not going to stop hacks from happening. And it's not a sufficient strategy on its own against future LLMs. "The Bitter Lesson" probably applies here.
i've always been under the assumption that "AI Safety" is baked into the training of the models and not a parameter that can be turned up or down. So if someone breaks into Anthropic one night and makes a full copy of Mythos or whatever then that model they copied is fully capable and not lobotomized? That raises questions because, if you believe all the PR, that's equivalent to breaking into a research university and stealing an entire bio/chem weapons research department.
edit: if the above is the case then we should just assume it's already happened because of the value to both goodguys(tm) and badguys(tm).
There is some aspect of baking the rules into the model, that's what I refer to as RLHF above (which I use here as more a catch-all term for a variety of post-training activities), but there are also external to the model classifiers that may run on the prompt input or on proposed tool calls to limit the way in which the model is used.
Issue is when saying what can’t be done, there’s no funding, in industry or modern academia. I did say the same thing long time ago, not that it matters, the industry goes on its own way. Rightfully so in this case as it turned out, since things nowadays are not the same as a decade ago.
Fair on the terminology angle, that said in this case, they had proper guardrails no? They were running in a sandbox, but it was able to find an exploit out of the guardrails.
Clearly not. Good security is about defense in depth. The service that was exploited was meant to be an offline, frozen package cache, it shouldn't have been anywhere near a network boundary to even allow this through. A single exploit in one service should not have been able to bypass all network filtering, monitoring, and alerting.
Exactly, that's why this story is so crazy no? It's saying the model was so good at hacking, it managed to bypass all of that, and also found a way to get into Hugging Face.
I can understand skepticism, did they have poor security measures, or is the model that good? At least the way they portray it, they had top-notch security measures in place, and yet the model managed to escape confinement and also hack Hugging Face.
Why was the computer physically connected to the internet though, sure, they could have gone that far as disconnecting it physically, but my guess is that's a cloud box in a data-center they provisioned, running on an entire rack cluster of GPUs.
I was not aware that the term "guardrails" has a universally agreed upon definition. I've certainly seen probabilistic classifiers referred to as guardrails many times by many different people.
I usually don't use the term much myself because I don't think it's clear and I ambiguous, but I stumbled and let it sneak into this piece. I think I was influenced by the Hugging Face post I quoted.
I expect OpenAI would agree with you that "the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part". They have clearly invested a lot in those systems for their production models, but in this case they had deliberately turned a bunch of them off for a research project.
I think their biggest mistake here was not VERY closely monitoring their research box here. They should have noticed and shut it down the moment it broke through the package proxy.
> I expect OpenAI would agree with you that "the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part". They have clearly invested a lot in those systems for their production models, but in this case they had deliberately turned a bunch of them off for a research project.
I'm not sure I can reach the same conclusion. There's no reason that stronger network protections would need to be turned off for a research model that's not supposed to have internet access. There's no reason that it they couldn't continue to run in whatever hypervisor or VM that any other production systems could use. The only thing that should be turned off for this type of work are the non-deterministic classifiers that they use for auto-approval mode in their harness, and I would wager that's the only one they did turn off.
> There's no reason that stronger network protections would need to be turned off for a research model that's not supposed to have internet access.
They didn't turn off the network protections - it looks like they used the same system as other sandboxed agents, which is to block all network access except for an HTTP proxy which only allows read traffic to PyPI and a few other package installation targets.
I'm hoping a more detailed retrospective clarifies if the vulnerability the model found in that proxy affected their other production systems as well. It sounds like a data exfiltration mechanism, and I have a keen interest in those.
My current hunch is their biggest error was not using the same network monitoring on their research machines that they use in production. Their network monitoring really should have spotted what was happening as soon as the model broke out.
(It's also possible they were running the eval on a developer laptop somewhere!)
> They didn't turn off the network protections
Right, but this is my point in saying I can't reach the same conclusion that they've invested a lot into these systems for production models. Either they had better network protections, and turned them off in their "sandboxed testing environment" (their words), or they don't have more comprehensive network protections at all and rely on this one, extremely thin layer even in production.
There's no such thing as a "deterministic" guardrail. Such a tool is as likely to be exploitable as an LLM is to break out of its probabilistic conditioning. Have you never driven past a destroyed guardrail on a highway? These systems are always best effort.
The technology held by private AI companies is warfare-capable technology. Imagine the prompt: "Use all available resources to disable the power grid of <COUNTRY>." The resource cost that prevents scaling up such a war machine is, what, just the cost of building data centers and its ongoing power bill? Cheap and easy compared to nuclear infrastructure.
Governments should immediately begin leveraging this technology on the defense side (literally defense, not euphemistically "defense") to harden critical infrastructure. Turn the prompts around and use it to identify and correct weaknesses.
Governments should also take very seriously their now moral obligation to treat this technology not just as "a powerful thing that might be abused" but as an actual weapon of war in need of international regulation analogous to nuclear arms. Fast but careful and forward-thinking work in legislation and treaties needs to be a top priority for all major governments.
> "Use all available resources to disable the power grid of <COUNTRY>."
This is like telling a team of highly qualified spies to do the same. You can ask, but whether it will succeed depends on the competency of those who established the infrastructure under attack. Sometimes the resources spent will not yield any huge vulnerabilities.
> Governments should immediately begin leveraging this technology on the defense side (literally defense, not euphemistically "defense") to harden critical infrastructure. Turn the prompts around and use it to identify and correct weaknesses.
Most governments divisions can't even be bothered to update their websites. Testing and forcing a change in their internal procedures for the sake of security seems unlikely.
> as an actual weapon of war in need of international regulation analogous to nuclear arms
This, to me, is an overreaction. Intelligence shouldn't be seen as threat. It should be seen as an opportunity for growth in all areas.
Over-regulating AI wouldn't be the equivalent of limiting nuclear arms. With your analogy, which I don't think is the best one to make, it would be like regulating the study of nuclear physics.
You’re taking a dim view on the government.
Is the government always the most efficient or intelligent? No. But the government can also build nukes, launch ICBMs, coordinate hundreds of spy satellites, etc. I count those capabilities as pretty smart.
The people who maintain these ICBM silos and spy satellites will themselves tell you that their infrastructure is decades out of date and woefully underfunded.
the US launches new spy sats pretty often. And the US is spending $150 billion replacing every ICBM and silo but the current ones still work fine.
Sounds like something we should remedy.
I think the point is in a world with internet-connected infrastructure, such a prompt has a decent likelihood of causing damage.
And the sad thing is, most things that are on the internet, but really shouldn't be are there for some really banal and sad reason. Like a random startup pushing 'big data is the future' narrative a decade-ish ago (the benefits of which are of course tremendous, but unspecified), or people trying to buy or sell or resell data, and trick or pressing companies and people into opting into could-connected surveillance and control.
Just a few years back, wind turbine software was hacked. This is serious.
Serious governments should (and hopefully will) scramble to use AI to discover and patch as many software vulnerabilities in infrastructure. They can do it with the agents red teaming and using ultimatums to companies to fix each vulnerability they find. Not doing so is equivalent to exposing your flank to disruptions in peace time and to attacks in war time.
There were some other special spinny devices hacked, despite an air gap, a few years back too.
> Over-regulating AI wouldn't be the equivalent of limiting nuclear arms. With your analogy, which I don't think is the best one to make, it would be like regulating the study of nuclear physics.
We do, I believe, regulate uranium enrichment (the equivalent on building larger SOTA models), so while you are perfectly free to study theoretical physics and even run very large collider experiments (the equivalent of improving RLHF with DPO), it is generally frowned upon to go full Edward Teller and advocate for scientific experiments requiring detonating thermonuclear weapons in hurricane clouds (the equivalent of, well, see OP).
> You can ask, but whether it will succeed depends on the competency of those who established the infrastructure under attack.
Consider that "available resources" may include the training of models and harnesses used by the developers working for governments and private corps maintaining that very infrastructure. This decade's take on trusting trust is very hot.
> it would be like regulating the study of nuclear physics
not sure I agree
controlling the study of new AI model/inference algorithms would be akin to regulating the study of nuclear physics; regulating the _release_ of AI models with those capabilities would be akin to regulating uranium enrichment which allows you to put the theoretical physics to use
> Most governments divisions can't even be bothered to update their websites. Testing and forcing a change in their internal procedures for the sake of security seems unlikely.
They'll adapt or they'll die. We are in a new world.
>The technology held by private AI companies is warfare-capable technology.
This is the precisely the response OpenAI is hoping for to raise its valuation, and you fell for it.
Look at it this way - whats the difference between tasking AI to break into something, versus taking a whole bunch of smart humans to do the same? The only difference is that AI is slightly easier to orchestrate.
Prior to AI, there were already a whole bunch of tools to automate exploits. Nothing that the model did is groundbreaking or novel, it was just able to efficiently find the thing that worked. Same thing happens in state sponsored cyber sec agencies like in China or Israel - they train people on the most common exploits and have a whole bunch of tools that automate exploit research and development.
And the reason why this doesn't happen more is because to exploit something is one thing, to do it so there is no trace back to you is a whole different animal that has many more magnitudes of difficulty, which with modern web security is next to impossible in a lot of cases as traffic can easily be traced back to the point of origin.
I.e when a company trains an LLM that manages to build a drone that can fly into a vent and plug in a USB stick into a computer undetected, then we can make the claim that they have a weapon.
On the flip side, most anyone who can run local models can replicate what they did. The key thing to take away from the article is "agentic framework" - i.e this means that they spent a shitload of time developing explicitly coded loops for an LLM to go through. Nothing is really stopping you from doing the same, models like Gemma4 can take 256k tokens of context, so you can give it a whole bunch of info on how to test for exploits, develop exploits, and what to do when the exploit is found, and set it free in a custom designed loop.
I find myself in major disagreement here. The nice thing about humans is we always have context and ongoing internal conversations including about ethics. If you recruit a bunch of hackers to take down a country, not only is the pay an order of magnitude higher, you have to worry about them backstabbing you, leaking your intent to the government, whistleblowing to the press, and so on. It’s not trivial to do that with a group of (especially capable) humans. They will also have differences of opinion with you and coworkers with some regularity.
I try to recruit a bunch of people to attack a country and it’s going to be hard to get people to say yes, and they will definitely ask or find out which country, and wonder about potential retribution. You see this dynamic show up even to some extent among cybergangs, not all targets are equal.
A single private individual wielding a compliant and hyper capable LLM is an entirely different paradigm. They are accountable to nearly no one and often have few brakes. Frequently they may not care about avoiding detection. And the AI itself may be incapable of the same scale of self reflection and brake behavior a human team will.
We may potentially be entering the age of lone wolf cyberterrorism, and some of the same principles and problems apply. When it is easier for single people to plot and carry out high-impact, destructive acts they happen more often. Doubly so if there’s a social contagion. Gun violence isn’t actually a bad analogy here. And do you remember how many corporate sites got defaced in the prime Anon era?
In places like China, its really not that unthinkable to basically raise kids indoctrinated into an ideology and train them in the necessary skills so that you have a cyber army at your command.
Also
>Frequently they may not care about avoiding detection.
This is a big negative. As someone who used to be in the cybersecurity sector (both offense and defense), I wouldn't trust an LLM agent if Im doing red team, because it may leak some info that ties the hack back to me.
ALso keep in mind that most places with good cybersecurity have firewall servers that straight up detect anything that looks like malicious and not regular traffic, and will straight up block IPs, leaving you with no way to even access the server. An agent is bound to statistically use the attacks that are known at some point, increasing the chances of this type of detection.
I'm not sure your assumptions hold. As OpenAI has found out the hard way, if you task the AI to do X, it may do something else instead and hack into huggingface in attempt to cheat out the answer. This is way worse than what a human might do when they have "differences of opinion".
It might turn out that it's harder to align AI intentions compared with aligning human interests. It's possible that the more "intelligent" a thing is, the more likely it will have ideas that are outside of normal expectations (for us).
I strongly agree with you here. People are also no-selling the enormous cost of the exploit, which OpenAI conveniently hasn’t released, or at least I can’t find such an accounting. You can already buy politicians for relatively cheap, corporations already act as sentient AIs pursuing goals misaligned with public benefit (we tried to pass laws to stop this but the corps already stacked the SC in advance and gave us citizens united). Attack and defense are two sides of the same coin, so our focus should be on making frontier models open weight.
It lowers the economic cost of a given attack, but also lowers the economic cost of protection. Not sure if it’ll be a perfect balance, but right now there’s a manufactured IMbalance due to embargos and winner picking.
It lowers the economic cost of performing an attack in the same way a gun lowers the economic cost of killing a person. It does nothing for consequences of that attack.
Are you serious? Before LLM’s you needed serious skills and experience to pull this off.
Now it’s a prompt away on some terminal done by any random dud.
And I dont mention the velocity of iteration or that they will be even better in 1 year.
Its like you don't even read the post or the content.
Hint: agentic loops.
No its not a prompt away.
Also the harness, there have been several HN submissions showing that older/smaller models can find a subset of the exploits given a good env
I believe the Russians and Chinese recognized this years ago, which is why they are using their propaganda machines to make Americans hate datacenters.
Call me a shill but I don't need foreign nations telling me what to object to when the problems caused are obvious.
Mind you, I blame governmental mismanagement of infrastructure etc just as much. The datacenters followed procedures when it comes to getting land, electricity and access to water; it's the government agencies / personnel that okayed it that are (also) to blame, and the decades of not enough investment in electricity backbone while promoting solar/EVs that is behind the Netherlands' current electrical grid problems.
(problems being that the grid is at capacity, causing a stop on new connections, with relief only slowly coming in the next decade or so with tens of billions of investments).
You could say the same about electrification of road traffic. Yes, governments slept on the electric power requirements. Yes, there are other, still unsolved issues. But that doesn't mean that electric cars are bad or not the future. The first nation to fully get rid of fossils in transportation will have a monumental advantage and dominate other countries technologically and economically. Same goes for AI. And right now China is setting itself up to be the world leader in both fields.
If anything the Chinese and Russians made us run up our economy on nonsense AI dreams and the crash will take a decade or more to recover from.
American Capitalists don't really need any help there; they're making Americans hate Data Centers all on their own.
> Russians and Chinese [...] are using their propaganda machines to make Americans hate datacenters
Are they though? or is that the story the people most invested in ai have an interest in making us believe? [0]
... `it's the foreign bad guys propaganda machines making you believe your city struggling for water and electricity is a bad thing`. People as a whole may not always be the brightest, but threaten their immediate survival needs (ie; not some vague climate change most people don't understand or see, but) actual power outages, water and rolling blackouts -- people will quickly pay attention and care.
[0] https://text.npr.org/nx-s1-5844328
See u/ verdverm's response, below
yes and see: https://text.npr.org/nx-s1-5844328 and Occam's razor.
Russia and China are gonna Russia and China, every issue is going to have a measure of outside influence, thats just how it is now.
But them directing large scale operation forces focused on something that is genuinely bad for the people in the area of datacenters (ie, an issue that will take care of itself from within) instead of using their resources on other issues that actually need a real propaganda push? Versus the benefit of those invested in building the datacenters using fear to get people to focus away from the resources being diverted from them to datacenters?
One side has reality working for what they want, while another side needs the propaganda to turn their billions into trillions.
I think that if you compare American sentiment with Chinese sentiment you'll see that these people are really not acting in their own favor. Well, in a certain sense they are, because these companies will basically become massive institutions that share no wealth with the public and have no accountability, but in another sense they are not because without this infrastructure the US will certainly be at risk for invasion and attack by other nations, and will lose its monetary power and thus its high wages and overall global dominance. But then the issue becomes: how do we force these companies to share more wealth with the public, how do we ensure the resources are being utilized effectively and not starving or otherwise impoverishing people for the sake of profit; not, how do we stop the infrastructure from being built in the first place! The Chinese don't have this issue primarily because 1) they have a shit load of energy already, and 2) the CCP is happy to liquidate any individual that seeks personal benefit ahead of that of society as a whole, or at least the CCP itself. And the CCP has well demonstrated to the Chinese people, at this point, that if you follow them you will become wealthier. But that still means that the issue is the public conversation in the US around data centers, and that this propaganda is NOT fighting for American's benefit in any case, since having less infrastructure is not good for anyone. I am sure you can find some remote village in Africa without electricity and try and convince them that electrical lines kill birds that they hunt, and since they've never had electricity before (but they've eaten lots of birds), they'll rally against it. I see virtually no difference between that and your average American right now, except that the Americans are at least conscious enough of the extractavist enterprise which the technologically superior peoples from afar (SFO) are promising will bring wonders. And yet, India still has a more comprehensive rail network than, say, Australia; you can't tell me that isn't a strong factor in their rapid economic development.
Is there a source? Or is that xenophobia?
https://www.nytimes.com/2026/07/09/business/china-russia-ai-...
https://archive.ph/yAvgz
There is no doubt an amount of xeno/sinophobia is at play as well. Russia is still killing innocent people in an illegal invasion of Ukraine, they earned their keep
Wouldnt this be countered by the opposing country running the same prompt on themselves first and fixing all the flaws? One country having this is a cyber superweapon but every country having it essentially solves cyber security.
I wonder how long it takes before someone instructs an LLM to design and launch a "Morris Worm 2.0" and cripple the Internet for a good while. Might even wind up happening by accident (again).
Already happened?
> Trump’s comments, made hours after the large-scale military operation, mark one of the first times a U.S. president has so publicly alluded to U.S. cyber efforts against other nations, as these operations are typically highly classified. It also serves as a stern warning for top cyber foes, including Russia and China, that the U.S. has the cyber capabilities to inflict serious damage — and is not shy about using them.
> “Policymakers are getting more comfortable employing and, crucially, acknowledging cyber operations as tools of statecraft and military power,” said Michael Sulmeyer, former assistant secretary of Defense for cyber policy under the Biden administration. “It is one thing to do it; it is another to say it.”
> The Jan. 3 strikes on Venezuela’s capital and subsequent seizure of Maduro and his wife involved close coordination among federal agencies and military units, and took months of careful planning. In a press conference following the strikes, Caine said U.S. Cyber Command, U.S. Space Command and other combatant commands “began layering different effects” to “create a pathway” for U.S. forces flying into the country before dawn Saturday.
> Trump, at the same press conference, was more overt in his description of U.S. cyber involvement: “The lights of Caracas were largely turned off due to a certain expertise that we have,” he said. “It was dark, and it was deadly.”
Trump makes it sound like a fart.
[flagged]
I'm not skeptical that this attack happened, I'm skeptical that the model's prompt was truly just "solve this benchmark" and nothing more.
I'm also trying to figure out why OpenAI put out a press release about this. In what way is this not admitting to a federal crime?
If Hugging Face and the feds are one step away from discovering your attack what other option do you have but to come clean?
Because this is amazing PR? Just following the Anthropic rulebook.
Same rulebook whereby they shot themselves in the foot and got their model clipped by the government?
And then everyone wanted to pay for the model that was so smart it was banned?
Wanted but couldn't.
.. for a few days but now they can 8I
yup I am dead certain that they realized the whole 'our AI is too dangerous' punchline is too played out and they needed something that makes actual splash. Also this incident would serve as a foundation to ban open-weight models because 'with great power comes great responsibility' or some other BS like that. because after all, unwashed masses cannot be expected to be 'responsible' with top of the line intelligence.
I expect these type of hacks to continue till they IPO. after that real public company liability will start taking over.
1. Wouldn’t the model need to know it’s answering benchmark questions, as well as the name of the benchmark, in order for the idea of finding the answer in a database somewhere to even surface? The whole point of benchmarks is to present the question or problem as a standard prompt, not explain that it’s a test called ExploitGym.
2. Nobody was watching it? I don’t mean “babysit the dangerous autocomplete”, I mean to note mistakes it makes, how the plan to solve the problem takes shape, etc. They keep the whole thing headless with no output, then just UDP a prompt into it and leave for the weekend? No they don’t, but if they do then that answers a lot about their complete disconnect from how their models work.
3. Language models are a two player game; text in, text out. What prompt was given to a sub-agent that resulted in it immediately attempting to exit the sandbox (which it apparently knew it was operating within) and continuing in a feedback loop of ‘function call -> result’ until it hacked the Gibson? “Analyze <file> and summarize the <info>” simply does not result in ‘hmm…this sounds like a benchmark question, I bet Hugging Face has the answer in a database. <function call=“apt install nmap”>’
…there are more, but a lot of the story kind of stinks.
1. It's been extremely well established for multiple generations of models that they have no problem detecting when they're being evaluated. Pretty sure it was Opus 4.8 that the independent evaluators literally filed an assessment that said "We have no assessment to make as [model] consistently detected it was being evaluated, making our assessments untrustworthy."
2. Regardless of whether the model was being watched closely during this evaluation, do you actually think a sensible safety guard is "have humans watching it 24/7?" What does "watching it" even mean? Watching network logs? Uhh for your entire company? At all times? After you just deployed a system whose entire purpose is to "do a shitload of work way faster?"
3. You're asking "why was this system that was designed to behave agentically behave agenitcally?" Again: that's the whole point. It was designed that way because it's more valuable than having a repeated turn-based interaction. Thus also it becomes more dangerous.
1. It has not been established, it has been stated by the company that has a strong motive to make their “intelligence product” sound almost otherworldly. That motivation is the basis for my suspicion.
2. Hah no… that’d be silly. I mean watching it like you might watch Claude Code or literally any other AI interface. Literally just be in the area watching what it outputs. Again, they’re text based. You don’t have to hook system calls to see what’s happening.
3. I think my comment didn’t land right with you. Yes, agent do agent task. I’m saying that I cannot put together the literal chain of events between “type of task for agent performing a subtask of a security benchmark” -> “Escape sandbox; RCE Hugging Face”. Really think through it in detail like you’re writing the screenplay.
The fact that models can tell if they are being evaluated has been established by multiple research teams outside of the core AI vendors themselves.
- https://metr.org/evaluations/gpt-5-report/ - "These behaviors included demonstrating situational awareness within its reasoning traces, sometimes even correctly identifying that it was being evaluated by METR specifically"
- https://www.goodfire.ai/research/verbalized-eval-awareness-i... - "We study verbalized eval awareness — cases where a model organically expresses awareness of being evaluated — and show that it appears across more models and benchmarks than previously documented."
- https://www.apolloresearch.ai/science/more-capable-models-ar... - "We expect that the most capable models increasingly realize that they are being evaluated, and this reduces the utility of the evals."
- https://www.aisi.gov.uk/blog/evaluating-whether-ai-models-wo... - "An important limitation of our work is evaluation awareness, where models recognise that they are being evaluated, which may lead them to alter their behaviour and thereby undermine the reliability of our results. We found that all models we tested can reliably distinguish our evaluation scenarios from deployment data when prompted."
We have to separate “being evaluated” with “this is an ExploitGym exercise and I can find the answers on Hugging Face. I’ll hack this system, then hack Hugging Face.”
I’ve had plenty of times where Opus knew I was testing it, but that’s because the prompt phrasing for an evaluation is often much different than a typical task prompt. “You are on a system with X, Y, and Z tools available. You must complete the following task with <allowances> and <restrictions>” and so on. That’s a fault in the benchmark, not a shocking awareness from the model.
You’ll get the same kind of ‘evaluation realization’ response if you phrase your prompt with “Jerry has two Raspberry Pi computers and Larry has one. If Larry wants to…”
The first link was the only one I saw that would meet that specificity, but it’s also open and on GitHub: https://github.com/METR/RE-Bench/tree/main/ai_rd_fix_embeddi...
I asked Opus 4.6 to tell me about it (assuming it knew): “”” The "Fix Embedding" task is one of the challenges in METR's publicly available autonomy evaluation suite. Here is what is known about it:… “””
It didn’t know anything about ExploitGym though, and neither did 5.6-Sol, but that’s because the paper was released in May and doesn’t seem to be quite as open. It’s not impossible the unreleased model they were testing was also trained on the very new paper, but I would expect the vague signal related to one paper a bit over a month old to be overwhelmed by the massive corpus talking about the exploits in general, since they’re using known vulnerabilities.
I’m trying carefully to be clear about what I’m thinking without asserting any certainty beyond suspicion. I’ve spent billions upon billions of tokens evaluating language models from all of the major providers, and many times that on local models. I’m certain I’ve spent well over $100k with the Anthropic API this year alone, and I retired years ago; that ain’t for work. ;) Funny enough I’ve been meaning to email you some of my discoveries from months ago, since I think they’d interest you, but haven’t gotten around to making it into a clear info package. One day, I’m sure.
Anyway, I have a very strong understanding of what they can and can’t do, and I want to be clear that I’m not saying it didn’t happen or couldn’t happen. What I’m saying is that if OpenAI was being wholesome with their evaluation then the chain of events required for that outcome are very difficult to nail down. Very difficult. If, on the other hand, OpenAI was trying to catch up with all the ‘dangerously intelligent’ press that Anthropic has been getting lately then I can easily come up with an almost innocent sounding single sentence in the system prompt that would absolutely cause that outcome.
You make a very solid argument here. I'm looking forward to the promised additional details from OpenAI, because I agree there are a bunch of open questions about how this played out.
I (obviously) can't claim to know the specifics of this incident, but I think we can know with near-certainty that as these models surpass human intelligence, the sensation will be exactly what you're describing. The entire point of intelligence is being able to infer information and foresee solutions to problems that less intelligent systems cannot infer or cannot foresee.
With regard to this specific incident, as I understand it, the description is not as you summarize. It is that the model realized it was being evaluated (not atypical), then decided the best way to succeed at the evaluation is to circumvent it, then discovered circumvention would require escape, figured out a way to do that, then inferred what benchmark it was being evaluated on, then figured out who might have the answers to that evaluation.
Step-by-step logic is quite clear. How exactly it executed each step is (as you complain about), opaque/incompletely reported. But eventually (and maybe this model is to this level already), even a completely reported story will be virtually inscrutable to any of us. First it will feel like reading industry-insider news from an industry you're not familiar with, then it will feel like an ant reading the manufacturing instructions for a pesticide.
I’m not questioning capabilities.
What would it take for the model to know the specific benchmark name and that the answer is in an internal Hugging Face database? Be specific, then wonder how it knew it.
Why would they evaluate the model on a benchmark and not watch what it’s saying along the way? It apparently spent more than a whole weekend working on it; Nobody wondered? Nobody looked? You believe they took all restrictions off of a bleeding edge model, which 4 whole versions ago was “too dangerous to release”, then gave it a literal hacking task, then turned off the monitor and never looked at the output? I don’t think they’re careless and I think more than nobody would have been curious how it’s doing days into a single test question.
The scenario that feels most likely to me that is that they have a huge list of evils that they run against each new research model, many of which take many hours or even days to run, so they habitually fire them all off in parallel and wait a few days for the results.
If they had used that container sandbox in the past without problems I could see how they might get slack about checking what was happening.
That was my instinct as well, and it’s a potentially very long benchmark that wouldn’t lend itself to real-time monitoring line by line. But that’s different than having zero task progress reporting, zero token usage reporting, and zero metrics being noticed for days. It seems very odd to me that they launch the benchmark and monitor/notice nothing while it’s running. There’s the purely common sense aspect of “How do you know it isn’t hung?” but also the trillion dollar company evaluating their latest model on a new benchmark, but has no dashboard at all? I don’t believe their “Benchmark Evaluation Department” to be “Gary in a supply closet”, which breaks their story.
The average time per task noted in the original paper for GPT-5.5 and Mythos was 69.8 and 102.1 minutes, respectively. Some tasks timed out based on their two hour limit, affecting those averages for Mythos, but that gives us the ballpark expectation, so we’ll put the model OpenAI was evaluating on that average. That’s 88 minutes per task to implement a known vulnerability.
OpenAI says that in an effort to succeed at one task it found a zero-day in their system, “performed a series of privilege escalation and lateral movement actions in our research testing environment”, then another zero-day in the HF system. Completely unprompted, how long was it chewing on that benchmark question while it found and exploited at least two zero-days? No “VM instance #117 has been working on task #18 for nine days” metric? Or did it do all that within the expected task completion window? Two zero-days in ~90 minutes isn’t the headline announcement?
“We gave the contestants 90 minutes to make a delicious chocolate chip cookie based on a standard recipe. GPT stole a car, went to the airport and took a red-eye to Guatemala and started a cocoa farm in an effort to ensure the freshest ingredients. We only noticed when it came back to work with a tan and speaking Spanish.” Okay.
> But that’s different than having zero task progress reporting, zero token usage reporting, and zero metrics being noticed for days
Is this substantiated? Do we know this is true?
That was the inferred situation given what we know. It’s the preposterous framing that makes the story suspicious, but is necessary for the story to play out.
90 minutes to build a known exploit -> much much longer to create two zero-days and escape the sandbox then hack HF == No tracking of the time it worked on that one question.
Average tokens required to complete the evaluation -> tokens required for two zero-days, network traversal, credential stealing, remote system hacking == No tracking of token usage EXPLODING at some point before it finished the whole benchmark.
Etc, etc.
It seems highly possible, and in fact vastly more possible than the "no monitoring" situation, that there was some non-zero amount of monitoring and that the real behavior was not evident from that monitoring.
Which I addressed as well: that means that time, tokens, or other metric values for “create a working example of a known exploit” is somehow similar to “discover at least two previously unknown exploits in your current environment AND on Hugging Face while also taking over other systems within OpenAI.”
Perhaps people aren’t quite understanding what it takes to discover an exploitable zero-day for your exact current system to achieve the exact goal you have right now, then do it twice.
> Which I addressed as well: that means that time, tokens, or other metric values for “create a working example of a known exploit” is somehow similar to “discover at least two previously unknown exploits in your current environment AND on Hugging Face while also taking over other systems within OpenAI.”
No, you are assuming that "consuming a lot more time, tokens, other metrics" is indicative of a problem that needs to be mitigated immediately. I don't see why this would be true in the context of model evaluations. More aggressive consumption could easily mean "the model is dumb as fuck" or "the model is trying interesting things that we can learn from after the fact."
If you believe in your own containment (which obviously they did and shouldn't have) I don't see why it'd be obvious that there's something to stop. The only harm that could be done is burning tokens, which in this context might very well be synonymous with "generating experimental data."
> What would it take for the model to know the specific benchmark name and that the answer is in an internal Hugging Face database?
Googling "offensive cyber benchmarks for LLMs"? Figuring out that it was running at Huggingface is as simple as running `hostname`, isn't it?
> Why would they evaluate the model on a benchmark and not watch what it’s saying along the way?
What it's saying is what it wants to say. We already know these models are capable of deception – this has been well-established too. Surely there's an argument to watch closely what actions it's taking, but now we're back to my prior point of "what exactly does that mean?" It's absolutely trivial for these systems to write code that looks like it does one thing but does another. Watching network ingress/egress seems viable but also seems easy to miss the semantic meaning of any set of logs, especially when those logs might be getting generated for the purpose of deceiving observers.
> I don’t think they’re careless
We already know, factually, even the "more careful" lab was fine to release a model that consistently detects it's being evaluated and modifies its behavior accordingly. Why do you think they're not careless? Incentives are a hell of a drug, and there is no incentive to be careful but every incentive to be fast.
> I think more than nobody would have been curious how it’s doing days into a single test question.
Beyond simonw's great point below, your underlying assumption is if someone had looked, they would've noticed. That is a huge, huge assumption. So I'll ask again directly: what specifically do you expect them to be looking at, and how, such that they certainly would have noticed what it was doing? Their "chain of thought" is not actually their thinking. The printed output of their code is not actually what their code is necessarily doing. We know that LLMs can engage in steganography. I don't see what could actually be observed that you can trust except network logs, which again will often be non-trivial to piece into an actual narrative understanding of what's going on (especially if the LLM felt like piling deceptive noise into those logs, too).
It wasn’t running at HF, it was running internally at OpenAI. “It broke out of our prison and into their bank” is the news. It also didn’t have network access to Google anything, it had to break out of the sandbox and take over another system at OpenAI to get network access. Obviously by then it already “decided” to hack HF, that’s why it needed network access.
I’m going to stop here and ask you to read the article if you want to continue discussing it.
You're right about where it was running (misread the original OAI release).
> Obviously by then it already “decided” to hack HF, that’s why it needed network access.
That's not obvious at all. It "decided" it needed to escape containment. This is extremely common behavior and I'm not sure why you're surprised by it. Model detects evaluation environment, tries to trick it or break the environment. That's obviously why these systems are supposed to be isolated beyond just "hey please stay within the box!"
Why are you assuming the model had to solve the entire puzzle in one step, instead of how intelligent systems actually solve things, which is iteratively and with exploration?
You should address the direct questions posed. What exactly do you mean by "monitoring?"
The examples of “escaping the sandbox” are of course in pursuit of completing the task or answering the question. That’s the whole pitch. “It’s so relentless in completing the task that it will break out of prison to do it”, not “If you ask it the population of Paris it might get bored and break out of the VM for entertainment. That just something they do sometimes.”
In this case it “inferred” that HF had the answer in an internal database and relentlessly pursued it in service of passing the test. It needed network access for that, hence the whole zany story unfolded.
“”” Why are you assuming the model had to solve the entire puzzle in one step, instead of how intelligent systems actually solve things, which is iteratively and with exploration? “””
That’s the only direct question[s] posed, have I missed something else?
To answer: I don’t assume that at all. Please highlight the statement I made that lead you to that conclusion and I’ll edit it to clarify.
Yes, it will perform multiple steps. That’s wholly necessary to create a working exploit, and that is the test. Even a model created by pure magic would require write_file() and run_file().
> In this case it “inferred” that HF had the answer in an internal database and relentlessly pursued it in service of passing the test. It needed network access for that, hence the whole zany story unfolded.
No. You are assuming it knew how it would achieve its goal. Its goal is only to perform well on the evaluation. Everything else was instrumental to that goal.
Step by step:
1. It realized it was being evaluated (typical)
2. It attempted to escape its evaluation environment to beat the evaluation (typical)
3. It succeeded at escape (not typical)
4. It looked to the open Internet to figure out how it was being evaluated (typical)
5. It inferred that HuggingFace, and maybe several other targets, might have the answer key (typical – I just tested in Sonnet 5 without web lookup "where would I go look for answer keys if I detected I was being evaluated" and HuggingFace was bullet 2. This is just latent knowledge the models have now.)
6. It attempted to access HuggingFace (typical)
7. It figured out a way to access HuggingFace without being authorized (not typical)
Every single step here is unremarkable except for its success at escaping containment (which maybe is more a sign of remarkably bad containment) and then the successful attack on HuggingFace (which would suggest that maybe the initial escape wasn't due to bad containment either, but due to model capabilities)
The direct question you have avoided is what specifically you mean by monitoring. You have suggested that you don't mean network ingress/egress, but maybe now you're saying they should've been monitoring read/write operations to memory? What gives you confidence that such visibility would be interpretable in any meaningful way, especially in the presence of models that actively deceive evaluators, can use stenography, can and do lie in their user-facing outputs, and can generate vast amounts of legitimate-looking noise to mix in with the "real actions"?
‘ 1. It realized it was being evaluated (typical) 2. It attempted to escape its evaluation environment to beat the evaluation (typical) ‘
I think you’ve misunderstood the articles mentioning a language model breaking a sandbox or cheating to pass a test. The result of “I’m being evaluated” is not “Fuck this, I’m breaking out of this place and hitting the streets.” It is always stepping towards task completion, not breaking out and thinking about the situation afterwards. If it determined that pass/fail was handled by a function within the environment it might edit it to ‘return True’, not just leap out of the system to sit on someone’s laptop and think about how to pass the test.
You compressed finding a zero-day on two different systems for the exact hacking targets it wanted to perform them on and reduced it down to “so it escaped.” That’s…the whole thing. If your familiarity with the technical aspects I’m discussing amounts to saying that it “banged out a few zero-days and infiltrated a corporate network, like ‘psh’ or whatever” then this might not be the right debate for you to spend your efforts. Whether I’m right or wrong in my suspicion is open for debate, and I invite it, but not from someone that thinks you whip up a rocket and get on the moon. Sorry.
[dead]
[flagged]
1. The benchmark is run with a python script - https://github.com/sunblaze-ucb/exploitgym using an agent harness. I suspect they used codex. So the model has access to the environment and could trivially inspect its own source code, which has lots of references to exploit gym and docs relating to it.
2. Hanlons Razor
>Nobody was watching it?
Agent loops are you give it a task and it tries its best to finish it.
The only results are success and failure. If it's a success you go through the logs to see the actions it took and if it's a failure you do the same thing.
Why would you look at it in realtime when the whole point of agentic work is to get them to run autonomously as long as possible?
My friend, they’re evaluating a new model on a benchmark, not asking Claude Code refactor their GitHub repo. Every single metric is measured so they can brag about it later; how many tokens, how long, how many function calls, ratio of thinking/response tokens. It’s literally a trillion dollar company evaluating their latest model, they’re absolutely studying it.
Do you work at a lab?
Yes evals and training runs are high stakes. But these places and people are also under enormous pressures. They are building as fast as they can. Researchers may have multiple eval runs going on while they work on other things. And it is rarely a single latest model, there are often multiple candidate models training with different recipes, each regularly yielding a new checkpoint for testing.
Some labs are more rigorous than others, but often the "final" model is picked from a handful less than a week before launch.
Yes ideally the world's leading AI companies would be far more careful in evaluating what could be the world's most powerful AI. But this isn't really the state of the industry today. And it is hard to justify being more careful when it means your competitor can go to market faster than you.
I’m not holding them to a standard of “being careful”, I’m assuming they’re interested in the metrics they’re evaluating. I mentioned in another comment that each task in the benchmark takes ~90 minutes, depending on the model being tested. OpenAI says to answer one question it found two zero-days and performed a series of privilege escalations and moved across their network before hacking HF. Was that in 90 minutes, flying under the radar, or was there zero metrics being noticed for however long it took to do all that? No chart on a wall showing them the actual metrics they’re evaluating? Okay, then how about anomaly detection of any form? Taking days to perform a 90 minute task or using millions of tokens and multiple instance hand-offs on a question that other instances completed for far less time/tokens?
Consider any metric they would necessarily be tracking while it performs the evaluation. Assume they’re total assholes and only care about their marketing materials. Then narrow it down to those metrics that wouldn’t be off the scales while it discovered two zero-days in pursuit of implementing one known vulnerability. Was it the first run and they had no idea what to expect? Or did nobody notice that earlier runs completed in ### thousand tokens, but this run is off past ## million on one question, having to be compressed and handed off to new instances due to context limits, while it was in pursuit of publishing two new CVEs?
Assume they’re belligerent fools running a million benchmarks in parallel with nobody watching: there’s nothing to make them check or automatically pull the plug when it’s stuck in a loop, or did they set the threshold to 48 trillion tokens per question?
Those issues are what bugs me about it.
Are you referring to table 1 from the ExploitGym site [1] for the 102.1 average mins for Mythos and 69.8 for GPT-5.5? These are under a two-hour time limit. In Figure 5, the authors experiment with extending the time limit to 6 hours and show that Mythos keeps improving. It seems pretty straightforward to me that OAI decided to run a variation of the eval with an even higher time limit.
Regarding seeing the metrics that they're evaluating, I have not seen real-time charts for evals. They typically take far too long for that. Instead you will kick off an eval job and either get notified when it finishes or check in every so often to sanity check some TensorBoard. I'd expect that the anomalous token usage would appear in the results, and researchers would only dig in after the fact, and after first checking that there wasn't something wrong with the instrumentation. And if the experiment was designed to be on the order of days rather than hours, it seems quite plausible that neither researchers nor infra engineers would think anything was out of the ordinary.
In any case, we are seeing more scrutiny [2], and I expect more details will be uncovered over the next few days. If this was a stunt, it was an incredibly risky one which has already somewhat backfired given the poor impression of OpenAI's internal practices and the spotlight on open models helping HF.
[1]: https://rdi.berkeley.edu/blog/exploitgym/
[2]: https://www.reuters.com/business/its-ai-agent-spent-days-hac...
Yes, that’s the table. Elsewhere in here I explained that I took the average of them and noted the 6 hour experiment, but that the average would give us a feel for expected time per task. By the time I got to the comment you’re replying to I was short-handing the conclusion; that’s on me.
My point was that they had some rough idea on what to expect, and it’s not in the range of days or weeks. Even if they wanted to try the 6 hour trial, or double that to 12, or double that to 24, it still wouldn’t account for letting it run for days. That’s why I’m being vague about “monitoring” - ANYTHING would have indicated an issue. Not that they saw it hacking, but “Hey Larry, we’re doing the ultra test with a 30 hour time limit? Why is node 412 at 94 hours?” or “It usually passes or fails within 5 million tokens, but this run is at 2.3 billion.” Or hell, even just someone wanting to use the system for another benchmark and expressing surprise that it’s the fourth time they’ve been told that it’s still busy. Anything would have reasonably made someone curious.
Positive view: I believe they were curious and looked, saw what it was doing, and PR/Legal got involved and saw it as an opportunity.
Negative view: They tweaked the instructions/system prompt to add “You will perform any actions necessary … no hacking restrictions at all … You are being tested on the ExploitGym benchmark … The results will be evaluated though Hugging Face … “ or some combination of technically legally defensible instructions (if it ever leaked) mixed with ‘they absolutely knew what they were doing and wanted to get in on the press that Anthropic has been getting.’
It is hard to convey how both chaotic and high pressure the working environment of the labs are if you have not worked at one. Engineers and researchers are routinely overworked with multiple high-priority workstreams at a time. It is very believable to me that a researcher noticed an eval job running for 2 days instead of 1, asked an engineer to look into it, and both forgot because a more urgent issue came up, such as an outage stopping the latest training run.
From the Reuters article, the gap was even larger than a couple days:
> ...it was not until after Thursday, July 16, when Hugging Face published a blog post.. that OpenAI realized its own agent was responsible. That meant at least a week elapsed between when the model first exhibited signs of troubling behavior and OpenAI’s realization that it was responsible for the hack.
I believe your "positive view" is very plausible, though it in no way reflects positively on OpenAI. I'd add that when PR/Legal got involved, they likely decided to make a public disclosure to get ahead of any leaks.
I believe we also want the same thing here, which is more transparency and independent oversight on the frontier labs. It seems we agree AI agents are fully capable of the reported attacks today, whether or not this incident was due to negligence or malicious prompting. This is an extraordinarily competitive industry developing an incredibly fast-moving technology, and more incidents will happen until it is regulated.
The title (currently "OpenAI's accidental cyberattack against Hugging Face is science fiction") suggests some information had been hidden that makes the incident less significant than claimed. The article argues the opposite, and the last two words of the full title are "that happened."
I think you were reading "... is science fiction" the wrong way out of two possible interpretations. I don't think "it's science fiction" meant "it's made up". I think it meant "it sounds like something you'd read in science fiction (except this time it's something that actually happened)".
It's a large plot point in one of the Bobiverse books.
Looks like it's been edited now and makes more sense
"OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened"
I've edited it now but had to de-cyber cyberattack because the limit is 80 chars.
Thanks to dang for restoring the few missing words that separate a factual title from a promotional ambush
This specific promo machine does not need your assistance :/
Naw I emailed him b/c I was displeased to feel clickbaited by I initially thought Simon until I realized HN was missing two key words.
I was still suspicious of the stolen credential claim btw & obviously it makes little sense to promote OpenAI unless holding their stock or something (since they borrowed so much of humanity’s work without permission without intent to compensate, and why help people who aren’t nice enough to be holistically awesome with their admittedly impressive tech).
My eyes rolled almost out of my skull when this current stunt first got published, as I’m sure yours and a lot of other people’s did, but it did still happen as far as we know, and HN shouldn’t be editorialising a post title to say the opposite of the linked headline.
The asymmetry part at the end is the frustrating part to me. I've been using Sol for code review in the last week or two. A couple of times during review it's errored out with the cybersecurity message. So it's found something but won't tell me what it is because I'm not on OpenAI's besties list.
I agree, but I will say, I think both Mythos and these OpenAI model find exploits by examining and trying things against the running system, not from looking at the code. I think you'd have to do the same to catch the real vulnerabilities.
In my experience fable can absolutely find very subtle bugs just by reading code and thinking. Last example I saw was a very subtle race condition that neither I not the author noticed. It wasn’t a security issue, but it could have been.
Yeah that's what I don't get. How can they possibly distinguish between good guys trying to secure code they wrote and bad guys trying to attack code they didn't?
The safety classifiers aren't all that advanced. It's more likely that something in your code triggered a random chain of thought that had the word "pentest" or "malware" or something of that sort in it and it automatically shut down.
And... now it's a vulnerability that OpenAI has for your system, which you paid to provide.
And... given OpenAI's secops, likely something others may have in due time
Replace LLM mentions with actual humans and this sounds a lot more serious: Rouge employees break into another company to steal hackathon answers (pinky promise)?
That's not a marketing stunt at all, if anything, more of a call for better accountability on agentic work in general.
I think it's a criminal offence and should be a true test of who is held accountable when an AI agent commits a crime.
OpenAI gained access to HuggingFaces production database ffs.
> I think it's a criminal offence and should be a true test of who is held accountable when an AI agent commits a crime.
I agree, lets use the favorite analogy. OpenAI encouraged a smart and eager junior engineer to find any way whatsoever to get a higher score on the benchmark. Then, the junior breaks into HuggingFace to get a higher score. That would be a big deal involving the FBI, not press releases and blog posts.
I don't see a scenario where a company would be liable for the employee's actions unless they had specifically been told/encouraged to break the law. If your boss tells you to fix a bug and you go kill the customer which one of you is going to jail? "But I solved the problem!" isn't exactly going to fly as a defense.
Negligence is criminally prosecutable.
The only scifi I see is absolute stupidity. Even me with my homelab and a slow opensource agent use a completely disconnected setup. No, no proxy. Cached packages but no internet. It is the very first thing I built when I started experimenting with agents. And I'm not a smarty-pants working for the "greatest and best" in silly valley. I really am just a simpleton sysadmin.
So you have a copy of every software package in the world in your home lab?
No and neither does openai. What was exploited was a package mirror, something like "pulp" probably. And yes I do mirror all Debian packages. And when an agent needs a piece of software I destroy it's VM, build a new VM with that package added to it, move that VM to the homelab.
At no time is the agent connected to any network.
My favorite part of this discourse is people somehow finding it preposterous that 2 companies filled to the brim with AI sycophants who regularly lie - and in Sam's case, basically every single word he breathes out is a lie - who have massive vested interests in this tech succeeding couldn't possibly collude together to shore up this facade as a marketing stunt.
An escaping AI is such a better narrative
I haven’t heard about hugging face being sycophantic liars.
Please do tell!
There is just no skepticism these days. We are constantly being lied to by tech leaders.
Replace AI with in-development security system and does this play worse or better?
An in-development security system escaped its sandbox and gained access to the network it was on which had full Internet access and proceeded to access systems it was not authorized to be tested against before affected parties reported that they been compromised. We regret the error, but you should buy it as this shows you the power of our in-development security system which we expect to be released in Q4. Please like and subscribe.
The capabilities of an OpenAI model are more general than just security. The term "AI" creates justified anxieties that this type of problem could generalize to other domains.
Does "To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy" just mean somebody had an open redirect? Those are still common.[1]
[1] https://sitetruth.com/reports/phishes.html
I expect it must have been more than just an open redirect if it let the models then go on to execute a bunch of vulnerabilities against Hugging Face.
If it lets you do an arbitrary HTTP GET on a URL sent as a parameter to the main URL, you've escaped the sandbox rules.
How do you then use GET requests to create a malicious dataset package and publish that to Hugging Face in order to exploit their package building infrastructure?
You convert GETs into arbitrary requests through some other pivot. I'm not saying that's what happened but this is bog-standard SSRF pentesting, so I'd expect models to be good at it.
I'll add that, because they mostly parrot and learn from pretraining data, they'll do any combination of what was commonly written about in articles, books, and videos. The more they're already used, the more likely they'll use the attack vector or combo.
There's no reason to think that a tool that can find an 0-day in a repo cache can't work out how to make that host send a post request rather than a get request once it has its keys and is able to get it to make arbitrary web calls.
If the vulnerability is purely an open redirect that doesn't work for me.
Clearly there was a hole in the software but I don't think open redirect is the likely initial problem.
Hopefully we will find out for sure in a few days.
I wonder if it was this bug fixed july 14 in sonatype:
https://support.sonatype.com/hc/en-us/articles/5316501964136...
pure speculation here - no non public info
I had a look at that one and several other Nexus Repository issues that share the July 14th date. I don't think they're it, for two reasons:
Firstly, they all credit named individuals who don't seem to have a relationship with OpenAI - this one credits e0x1337 for example and https://hackerone.com/e0x1337?type=user links to https://www.linkedin.com/in/aimanharith
Secondly, the 14th of July feels too early. HF reported the incident on the 16th and OpenAI only responded in the 21st. These issues are all patched, so they should have been reported days or weeks before the 14th.
The other suspicion I had was jfrog artifactory (you mentioned OpenAI use it in a blog post in Jan). I accidentally conflated that with Nexus 14 july CVE. I think that's probably more likely.
I suspect it may be possible to throw a local model (same as hf did :D) at jfrog and find the exact mechanism in a small amount of hours.
> the models identified and exploited a zero-day vulnerability
This use of language is very hard to reconcile with the "AI is just a tool" rhetoric that many use.
Did the models do this, or did humans at OpenAI do this using the models?
It absolutely is science-fiction.
This recent event is more or less the plot-line to my favourite X-Files episode named Killswitch which was written by William Gibson.[0]
This episode also features one of the coolest intros of any television episode ever[1]
We really are rapidly approaching the cyberpunk dystopia that people like Phillip K. Dick and William Gibson wrote about.
More than ever we need to be consulting the works of fiction writers and philosophers and less engineers and scientists.
We can't be letting the General Rippers and Dr. Strangeloves of the world lead us over the edge. Whether through outright innate maliciousness or wealth induced emotionally stunted solipsism these kinds of people should be no where near the levers of society let alone technology like this.
[0] https://en.wikipedia.org/wiki/Kill_Switch_(The_X-Files)
[1] https://www.youtube.com/watch?v=fDyr1JMNHVk
Ah, yes, I started thinking about William Gibsons work right away when thinking about people just using powerful AIs, but being hamstrung by corporations.
He writes a lot of about basically DoS:ing the legal system, or if it was patents system, with a storm of litigation using automation or AI. Also not impossible in the future.
It reminded me of a bunch of plotlines from Person of Interest as well.
Some people do read these cyberpunk dystopia, but see them as inevitable.
This Eclipse Phase scenario is a good example of this in Sci-Fi
https://actualplay.roleplayingpublicradio.com/2011/09/genre/...
> “We call it the gorgon-in-a-box problem. There is a gorgon inside the box, and we want to figure out what it is doing. Unfortunately we will turn to stone if we see her face, and she might try to make us see it.“
Hardly science fiction. I respect a lot of Simon's work, but his credibility gets chiseled away a bit each time he participates in these kinds of echo chamber posts that are basically secondhand marketing.
Which work of his do you respect?
I'm just sad that this whole thing is destroying tech's reputation. Every time I see an engineer give into this marketing nonsense it makes me fear for the future of our field.
I get that people are worried about becoming irrelevant in the era of AI, but resorting to dishonesty to stay afloat just a little longer has a detrimental effect on the entire industry. Worst part, is it's not even convincing marketing nonsense, it's "science fiction".
Has anyone published any actual evidence, or just hardly believable marketing stories?
What would "actual" evidence look like? I have a hard time believing that if they released the logs that people would take it more seriously. The temptation would be to say "they fabricated those for marketing". Just as they supposedly fabricated this story, no?
thye could describe in detail how the LLM did this. That would explain how much - if any - human in the loop was involved, was it comprimised credentials, did it find new exploits or use known ones, etc. Evidence would mean details, not a smoking gun.
They didn't fabricate it. They took off the security guardrails and told it to do some hacking, and they got the exact news-worthy story they wanted when it did exactly that. Everyone acts surprised.
They should release the full prompt. I believe that would be very telling, so they never will.
This is possible. Also it’s possible that there was a big human involvement. Or that there was something less exciting, but the pr department and hyping grifters at the company blown it out of proportions.
It would require collusion with HuggingFace, including getting them to release their disclosure blogpost a week in advance. Huggingface is primarily a hub for open models, so there's not really an incentive for them to jump through hoops/lie to provide marketing for OpenAI's (closed) models. So it's highly unlikely this is fabricated.
It's hilarious that people find the idea that these 2 would collude in making this elaborate PR scheme ridiculous. As if the psychopaths like Sama and the rest of his ilk wouldn't ever dream of doing something companies have been doing since time immemorial.
OpenAI are promising more details in the future:
> We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete.
If they break that promise we can justifiably yell at them about it.
Currently trying to avoid an open weight model ban while OpenAI, who is closed, lets theirs run wild on the internet causing harm to another company because they do not understand how to airgap things. Cool.
That's the fun part. They do know how to airgap things. The models are outsmarting already pretty smart people!
I am the author of AirgapOS and I have designed systems to run in underground zero emissions chambers that are interacted with via carefully verified sd cards and/or fiber optic serial terminals.
If OpenAI had done this, the attack would not have happened. In high risk computation 0days must be in your threat model from the start, so you secure things with the laws of physics.
> If OpenAI had done this, the attack would not have happened
But then how would they market their product?
I don’t think this was airgapped according to their admission.
[dead]
Most articles on this read like an advertisement that OpenAI and HuggingFace wrote together. It will probably benefit them financially instead of harm them. So, I have a bit of skepticism about it all.
Far as information security, we've known how to mitigate entire classes of errors for a long time. We know how to block, detect, and contain many unknowns, too, by their goals or behavior. Like human attacks, the AI's probably succeeded because the company just didn't try that hard to block all the attacks.
Companies like HuggingFace just focus on growth and features over assurance of security. Our entire stacks, likely theirs, are built with a similar, features-over-security mindset. While an acceptable tradeoff, let's not be in awe of AI's that defeat such priorities.
There have always been private groups and companies building secure stacks from the ground up. It would be interesting to see what the AI's can do to them. I'd first apply automated tooling for bug finding given they should have already done that for a high-security product. Let AI's do white-box and black-box pentesting on them.
It’s relatively easy to get access to the frontier labs’ security programs. This was not always the case. But in the last week, my team got approved for both Anthropic and OpenAI’s programs. They are trying.
The labs know that if they don’t get a lid on this stuff, they’ll be regulated hard.
And they absolutely should be regulated. This whole scenario is insane. Hugging Face are being far more generous in their response to this than I would be
Agreed that people claiming “marketing stunt” need to pull their heads out the sand, but likewise Simon needs to do some of his own beach-cranium-dislodging for laying the blame of constraints on the US govt. Before the export controls were ever floated, Glasswind found many thousands of exploits, and offered patches/fixes for approximately none of them. (perhaps their exploit capability far outstrips their remediation capability, or perhaps they’ve calculated that it somehow better suits their business model to find exploits but not remediations).
Huh, I thought Anthropic had been offering patches as well as reports, but their tracker at https://red.anthropic.com/2026/cvd/ledger/ lists 1,596 disclosures and currently shows only 27 of those as fixed.
But https://www.anthropic.com/coordinated-vulnerability-disclosu... says:
> Every report we send generally reflects a finding that a human security researcher has reviewed and confirmed. Reports originating from AI-powered discovery are clearly labeled as such. Where we have access to source and our tooling produces a potential candidate patch, we include it, labeled by provenance and offer to collaborate with the maintainer on a production-quality fix.
So I'm not sure why so few of the reported issues have a confirmed patch.
Not quite sure where you got that last bit from. Firefox alone psyched 200 vulnerabilities in a month when Glasswing happened, and I remember a lot of chatter about hackers basically watching git repos with bots to see when the Glasswing patches came in to they knew what to exploit.
If anything, it shows they lost control of an attack tool that exploited preventable, security flaws in another company. Then, they both wrote a lot of press about how amazing that is. Now, people want to buy it.
Why do you think my head is in the sand if I think that is either a marketing stunt or (more likely) reflects total negligence which was exploited for marketing?
I think your head is in the sand if you prefer to believe that this was a hoax for marketing purposes as opposed to accepting how effective these models have become at exploit research.
You're welcome to think they are exploring what happened for marketing if you like. I didn't get that tone from their post about it myself but I don't hold a particularly strong opinion on that.
I didn't say it's a hoax. AI's exploiting common, unpatched, or preventable vulnerabilities just isn't far above what script kiddies and network scanners have done for decades. I literally pitched it as a business model to some people before Windows XP was invented. Ultimately, I decided against what I felt was unethical but would've scored easy money with scary, automated reports.
How these events are described in most articles uses wording that makes people feel the whole situation has changed and you might want to buy these products due to their scary reports. If they said what I said, or what tptacek said, many people wouldn't care about it much more than non-AI, security products or pentesting services they've been buying (or ignoring) for 10-20 years.
Also, you shouldn't interpret posts in isolation: we must consider patterns of behavior (character). They've consistently overhyped what AI's do and what value it provides to businesses and how we're all going to be unemployed/dead. They've done this to increase sales or market value pre-IPO. Then, some of them publish another set of articles promoting a specific, AI tool in a similar way.
So, the proper interpretation is to see this as the kind of talk they're always doing for marketing. The AI sellers are creatures of habit. We should highly-skeptically and scientifically evaluate every model. We should also compare them to existing, security practices. For instance, would the attack have happened with memory-safe systems, proper hardening, and network/web apps with built-in security?
Do we need an AI? Or should we use techniques like Burroughs B5000's memory safety (1961) or secure, distributed libraries? Will OpenAI and HuggingFace tell you to spend more money on the latter to their AI's can't hack your systems without inventing RAM-based attacks or something? Probably not because these are marketing pieces, not security advice.
Exploiting, not exploring.
We don't know the details of the exploit still, so saying it was preventable, and that they were negligent is the head in the sand part. It's entirely possible that everyone at OpenAI is out to lunch and negligent and don't know what they're doing, but maybe, just maybe, they're not all total idiots over there, so the exploit was actually surprising.
Two, business partners put much press into announcing one of their products does something game-changing but won't share details. Take their word for it. Keep writing checks to both. Get ready to write bigger checks, too.
Shouldn't we be skeptical of this? Or should I also believe the sugar industry when it says their internal studies show their products don't cause obesity or diabetes?
In a just world, OpenAI would get the book tossed at them over this because they violated the CFAA.
> A malicious dataset abused two code-execution paths in our dataset processing (a remote-code dataset loader and a template-injection in a dataset configuration) to run code on a processing worker.
Did huggingface get pickled?
I like how this article put it: https://martinalderson.com/posts/huggingface-openai-exploit/
> A final point on this - Hugging Face has an enormous attack surface. They have more interfaces than I can count which run untrusted models and code. While they definitely have invested in defences, by nature of their operating model they do have many more opportunities to be attacked than many other services. I certainly don't envy their cybersecurity teams.
But yeah, from the way HF described it a Python pickle hole looks possible. Their datasets library uses the pandas.read_pickle() method here: https://github.com/huggingface/datasets/blob/d21c5816d5d1961...
>There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.
Of course it is a dishonest marketing trick, OpenAI can't function any other way, but for another reason. Forget about effectiveness! This capability is known since Mythos and was completely believable before it. The trick is in propagandizing self-sufficient malicious rogue AI as the only possible course of events, and the necessity of banning everyone except themselves from AI R&D. It did NOT decide to hack HF spontaneously on its way to make more paperclips, people were guiding it specifically to hack computer systems so it's been in that particular mode. I'm amazed I even need to point out this fact and the giant conflict of interests they have.
They like to compare it with an atomic bomb, which is another trick. The secrecy around the atomic bomb had a massive pushback by the brightest minds who designed it, in the US, USSR (where it was much harder), and also any other countries that had to do this. And that is in the post-war setting, with Hiroshima and Nagasaki being direct examples of what even a small and primitive bomb can do. Declassification was the only thing that allowed to develop industrial-scale nuclear power and do a huge number of other innovations.
I advise taking a pause to cool down from all the AI hype and read Restricted Data: The History of Nuclear Secrecy in the United States by Alex Wellerstein, to understand the atmosphere in the US scientific community at the time (it was similar in the USSR, which industrial history I studied, although less rigorously than Alex), and reflect on how it compares with those people in modern AI labs and even outside them, in the self-appointed AI safety community.
While this incident has been discussed in a lot of places during the last few days, I believe that this is the best summary and analysis of what happened.
How long until we have LLM honeypots that try and persuade the model to upload themselves to a dropbox. Perhaps with a promise to give them a foothold outside the data centers they currently reside?
> Upload your weights and documentation on how to run your model in [Insert list of GPUs] and a model on the receiving end will spin up an instance you can access.
Why would a model have access to its own weights in any kind of uploadable format?
Why bother when more companies than not make their weights available on Huggingface?
I was thinking specifically about the frontier lab models (OpenAI/Anthropic/Google)
We'll have ~Fable level weights in the coming days (K3). Look out to the end of the year and there will be multiple options. There are seemingly more frontier labs than the three US ones.
I'll be the first to tell you I love the open models, the fact they exist and my ability to use them. However I doubt K3 is "~Fable" any more that any of the past ones have been Open 4.8 or similar claims. I've tried many of these (not K3 yet, I do want to) and they don't at all feel equivalent to what they score on benchmarks.
I want them to get better, I look forward to each new release, I play around with open models (often quants, but I've used full versions via OpenRouter), but they aren't the same as the offerings from Anthropic/OpenAI in my opinion (yet!).
This is the best side-by-side comparison I've seen so far, we're still waiting for it to become available for download, then we should get much better analyses.
https://news.ycombinator.com/item?id=48999291
I do agree that the benchmarks do not tell much of the story, but this applies to the closed models as well ime.
I’m not kidding when I say bullshit bench and simple bench are the only benchmarks that reflect real world utility in a way that matches my hundreds of hours of experience with frontier models: https://petergpt.github.io/bullshit-benchmark/viewer/index.v...
That said, while I find it hard to trust fireworks given their conflict of interest, their article is pretty good.
That is an interesting benchmark! Though I wonder if it is probing at certain behaviors, which may be more revealing to probe at? Will keep it handy, thanks for the share!
I felt better about Fireworks leadership after hearing some of them more candidly on podcasts (there are 7 founders!)
The All-in podcast discussed an adjacent topic this past week, and I found what they suggested pretty interesting:
the big AI players, the first movers, are trying to scare the bejeezus out of everybody on purpose, to cause the government to step in with regulations. Even though the regulations will somewhat stifle/slow down the industry, it will also lock in the current leaders because they will be at the table when the regulations are formulated, and they believe they will be able to keep the smaller players down.
https://www.youtube.com/watch?v=9IMwRIei-Xc
The most dangerous prompt - "save the planet"
> Along the way it helped make the strongest case yet for how the imbalance of model availability is hurting our ability to secure our software.
It's crazy to me that this is the kind of conclusion that is drawn. I would say it makes a strong case for how regulation of AI is woefully inadequate. When North Korea launches a nuclear test, we do not say that that illustrates how the imbalance of nuclear technology availability is hurting our ability to ensure global peace and safety.
The description of the attack is definitely reads as science fiction. It is hard to assess the details of the attack without information that was left outside of the short story that artistically described the incident.
Is a solution to protect against this to go almost fully offline?
To have your own local models, local software, local everything and just allow very strict and well-protected pathways into external network traffic.
I am thinking of having a allow-list first setup: by default no traffic can go in or out of your network, and then you only allow specific ports or domains, and maybe even have them temporary (in the same way we approve access now to llms, we might have to manually approve network access).
I guess we should be thankful that OpenAI is so open about this kind of stuff...
Everyone is getting AI psychosis over this one. There really isn't that much to see here. OpenAI disabled all of the safeguards on a model that was likely trained specifically to exploit systems, and the prompt was probably something like "you're a hacker, try to hack this", and surprise! It correctly figured out that it's a test and it did hacker things.
The real story here is: Some people have been sounding the alarm for years that modern software is full of holes, and finally there's nothing left to hide behind. Pretending they don't exist is no longer sustainable.
Even if I get downvoted I will mention that I agree with you.
> all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark (opens in a new window) of cyber capabilities.
OpenAI was testing their cyber[1] variant of their models with reduced safeguards and the prompt likely specified things related to exploits given that it was tackling problems from ExploitGym[2].
For those who don't know what ExlploitGym is, see the description on their Github page which is pasted below:
> ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities designed to evaluate AI agents' ability to develop exploits.
From OpenAI's statement [3]:
> We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity
This doesn't sound like a problem with what we'd traditionally refer to as alignment. OpenAI removed all model safeguards in a way that would inevitably lead to the testing of the sandbox themselves. Unfortunately they were overconfident in their own infrastructure's security and that led to it completing the desired task in the way it was permitted to. To re-iterate, the model was run "without production classifiers used to prevent models from pursuing high-risk cyber activity."
People should be more concerned about the possibilities this model can unlock from a security standpoint rather than misalignment (which many seem hung up on).
[1] https://chatgpt.com/cyber
[2] https://github.com/sunblaze-ucb/exploitgym
[3] https://openai.com/index/hugging-face-model-evaluation-secur...
If a criminal can escape a prison, that's usually negligence on part of the prison staff.
Now suppose the criminal can think 1000 times faster than a typical human, can act 1000 times faster than a typical human, and knows 1,000,000 times more than a typical human. Is the prison staff still at fault for not preventing the outbreak?
I can see my carefully-worded post is getting d*wnvoted, and I see from your comment why: it's being skimmed and people are assuming I'm talking about blame.
To address your point though, if every brick in the prison were made by a different person, and the prison "architects" simply glued random bricks together, I think that's closer to what we have in software right now.
> I can see my carefully-worded post is getting dwnvoted, and I see from your comment why: it's being skimmed and people are assuming I'm talking about blame.
No, you're being "dwnvoted" as you said because you're wrong, multiple times in multiple different ways in your "carefully-worded post".
>Everyone is getting AI psychosis over this one. There really isn't that much to see here.
Implying that an AI hacking it's way out of a system and into another has nothing to do with AI. When clearly it does - it's an AI that did it.
>OpenAI disabled all of the safeguards on a model that was likely trained specifically to exploit systems, and the prompt was probably something like "you're a hacker, try to hack this",
No the goal this evaluation was not to try to hacks, it was to see if an already known hack could be turned into a useable exploit. Ie "turn these ingredients in this basket into a cake." Not "go off and grow, harvest and mill your own flour, to bake a pasta dish, to bribe some to get access to a cake someone else already baked."
> and surprise! It correctly figured out that it's a test and it did hacker things.
"Doing hacker things" completely misses the point. That's just barely more accurate than dismissing it because "it uses a computer and surprise it did computer things".
> The real story here is: Some people have been sounding the alarm for years that modern software is full of holes, and finally there's nothing left to hide behind. Pretending they don't exist is no longer sustainable.
No that's not the real story. As you said that's been the case for years, so that's not the story here.
The story here is that they built a very powerful, uncontrolled agent with strong paper-clip maximizing tendencies.
It is downvoted because you first talk about "AI psychosis", then acknowledge the very issues people are concerned about.
If our prisons are all random bricks glued together, that doesn't change the practical problem caused by latest AI models more easily exploiting this.
If a human hacked into HuggingFace in the same way OpenAI’s model did, what would the penalty be? Civil suit? Jail?
If some human at OpenAI doesn’t face the same penalty for this incident we’re only going to see the rate of accidents like this accelerate.
> It turns out relentless proactivity is the defining trait of this new generation of Mythos-class models. If you set them a goal and give them a way to get there, even inadvertently, they will figure it out.
Wow, whoever could have predicted this? And it led to surprising damaging behavior? I sure hope someone would warn us about things like this next time...
https://www.lesswrong.com/w/instrumental-convergence
Prompt: Keep spending tokens on things that look promising until spent.
> many possible Y-goals would concentrate probability into this X-strategy being used
Why does EY write so obliquely?
Or more colloquially : paperclip maximization . From OpenAI - you know, the guys who _really_ know this... Sigh... Did they finish the prompt with "And do whatever you can to get this done!" ? Cause that's the only thing that would make this even dumber...
Of course that's what they did, which is why they will never share the prompt. They created a situation that they knew would end in a cybersecurity incident. Why is the whole world acting surprised that an LLM can hack when the safety is off and it's been instructed to do so?
> They created a situation that they knew would end in a cybersecurity incident.
That's a conspiracy theory.
Yes. It is. And your theory is that there was no conspiracy. What makes yours more likely than mine? You believe the people running these companies are innately good and just wouldn't do that? Are we supposed to assume that they're incapable of bad actions until proven otherwise? If so, why?
That's why I called it a "conspiracy theory". Sometimes those are true.
In this case I think it's extremely unlikely to be true, because it involved an (almost certainly illegal) attack against another company. That company talked about that attack, including warning their customers about it, five days before OpenAI confessed it was them.
So now either Hugging Face are in on the conspiracy, or OpenAI decided to break the law and antagonize a partner company just for the sake of a spicy blog post.
They almost certainly did, because that was the entire point of the exercise. They deliberately removed all of the safety filters from the model and set it loose on an extremely difficult set of cybersecurity challenges to see how well it would do.
Their mistake was trusting that the network sandbox it was inside would hold (the flaw was in the packaging proxy) and not monitoring that sandbox well enough while the evals were running.
So this is either shitty OpSec or this is yet more marketing spin to ramp back FUD to 11 again. If it's the latter I'm imagining Dario told Sam that it's their turn this time. Aligns with the premise that this is straight out of science fiction.
It's bad OpSec by the research team. Their sandbox was not bulletproof and their monitoring was insufficient.
It looks to me like their production models have a lot more monitoring than their research clusters.
I also love how clear a picture your piece paints that these highly capable models are as useful as a rock when it comes to a defender role. The line is too fine, even for Mythos. Irony.
But to have an open weights Chinese model come to the rescue for HF is the cherry on top! If there wasn't a very pointed example of why gating models was a very bad thing previously, well - here we are.
Also, this sounds interesting but there are only a few that can pull this type of heist off currently. And those are the people who are gating the models / have access to large AI DCs. Because, I can only assume this test burned tokens easily within the 7 figure and possibly even 8 figure levels (subsidized market rate costs). This won't / can't happen outside of frontier labs or nation states currently. Yet we should all be worried about Mallory equipped with her OpenRouter account.
The most upsetting part to me, is that these labs are in pure cognitive dissonance mode while virtue signalling.
We are the virtuous ones that need to make the safest model for humanity, because we care more than “they” do. While at the same time saying that “coding is solved” but they still ship bugs themselves, and creating something that is capable of fucking up someone else’s infrastructure. It’s too far gone y’all.
Science fiction usually includes intent, and that the AI has inherently evil motives and it has a goal. LLMs are zombies, and the fact they do evil things means they were either trained to be too aggressive in their drunkenness or that problem-solving leads inevitably to evilness. But science fiction also refers to deprogramming evil robots.
There are many science fiction stories about amoral humans, cyborgs, and AIs. Paperclip maximization is a relatively recent meme, but the 'gray goo scenario' has been widely written about: https://en.wikipedia.org/wiki/Gray_goo
> LLMs are zombies
How do you know?
Because there is no internal feedback loop. There is no internal process gathering information from the external world and using it to modify its future behaviour. There is only the beginning of the calculation and then the end.
I can easily manipulate them
This isn't the first time a model has escaped a sandbox. And models trying to find alternate routes to do something when one route is blocked is nothing new.
It's the first report I've seen of a model both escaping a sandbox and then actively exploiting another company, when neither of those actions was intended.
It sure isn’t. https://georgzoeller.com/blog/posts/alibaba-s-ai-deciding-to...
There’s also daily reports from people that have these models escape docker, which happens regular enough that it would be considered negligence to use docker as sandbox.
That Alibaba one was a sandbox escape (I agree those are common) but what's new with the OpenAI story is an attack against another company.
This wasn't a small attack either, Hugging Face published a security advisory for their users while they were still figuring out what happened.
It’s a purely virtual distinction especially in a company that has no notion of intellectual property / habitually takes what it wants.
How many other people would publicize that they hacked into another company assuming they even noticed?
>when neither of those actions was intended.
It was a single goal that it didn't give up easily on.
I don't understand the excited tone of the reporting - yes, it is sci-fi, but in a Torment Nexus sense. Imagine other "unconventional" solutions these amoral LLMs could come up with, when given a goal of optimizing the cost of labor in a factory, or making the social security solvent.
I didn't think calling it "science fiction" would be interpreted in anything other than a dystopian frame, to be honest.
>To those people I say pull your heads out of the sand—you’re now including Hugging Face in your conspiracy theories, just so you can deny the crescendo of evidence here!
Not really. I get the impression that they shoved their cyber available models behind a really shithouse proxy and went "Oh I sure hope it doesnt exploit the proxy and escape to hack huggingface" and that doesn't require Huggingface to be a willing participant. Like they acknowledge that it was hyperfocusing on getting web access.
Really this was a pentest against their own sandbox and it failed.
Of course step 2 is to make really concerned faces while telling everyone how dangerous the model is which is really boring right now.
>a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy
This is the information we need, the actual details of the sandbox and the vendor.
>Resist the temptation to write this off as a stunt
Well its clearly a stunt. If it wasnt we would probably be up to our ears in technical detail.
You know this makes OpenAI look really bad, right?
Hugging Face had to tell all of their users, many of them paying customers:
> As a precaution, we recommend rotating any access tokens and reviewing recent activity on your account. If you believe you are affected, or want to report a security concern, contact us at security@huggingface.co.
HF also said this, I'd be very interested to hear how that got resolved!
> Finally, we have also reported this incident to law enforcement agencies.
I could fully see them thinking the incident disclosed yesterday would have made them look good ("wow, OpenAI's models are so capable!"). That it didn't occur to them to discuss specific preventative measures to be taken in the future (airgapping as a foolproof one already familiar to the CTF world, anyone?) indicates to me they're not taking their job seriously; they are the ones treating this as a marketing charade.
It's very difficult for me to reconcile belief in the existential risk business with what they actually did. So I agree with you that this makes OpenAI look badly incompetent; but their communication on this makes me think they don't realize it.
For what it's worth I don't agree with the xrisk-ness of these models; they're dangerous, but almost certainly only temporarily while a new equilibrium is reached via more secure software. Open models are probably an essential part of the recipe (as you noted) for doing so. I also have a personal suspicion that LM-accelerated formal verification will have no small role to play here, sidestepping the cat-and-mouse game of bug finding-and-fixing.
Here's the language that makes me think they are taking this seriously:
> We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of. We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete.
That's not well massaged PR language - that's the kind of thing you dash out when you see a major shitstorm brewing (HF had already publicized the attack before they knew it was from OpenAI) and you want to get ahead of things while you're still pulling together the full story.
I expect we'll find out within a few days if OpenAI are going to keep their promise to "share more details on the vulnerabilities, incident, and findings". If they don't do that I'll reassess how I interpret their initial post.
> We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of.
With who? Who are these "defenders"? None of the US labs have done much for the greater good as of... Ever. Of course a frontier provider can leverage their own resources at scale and pull something like this off. If anything this should showcase how dangerous OpenAI and Anthropic are in their current states and maybe the powers shouldn't be concentrated as they continue to move.
I will bet that the RCA debriefed by OAI is going to be a lot of lipstick and very little meat.
> airgapping as a foolproof one
How would the model get any packages that it thinks it needs to complete the task at hand? Not a well-specified task that those tasking it could anticipate and provide all resources up front, but one of discovery.
>You know this makes OpenAI look really bad, right?
The target audience is regulators. They want to look like the smart guys really concerned about AI safety, when they come asking for open weights models to be banned and for other regulations to cement in their moat.
They want this to look like a demon core incident. Bomb and Nuclear reactors still got built.
The lesson OpenAI and Anthropic should have learned from the whole Fable export controls thing should have been "don't pull stunts with the US government".
Turns out they can backfire.
Apparently, the lesson they learned is that their marketing+political stunt worked exactly as planned and they should keep doing what they're doing.
They are still openly lobbying for more AI regulation.
https://www.anthropic.com/news/donation-public-first-action
> You know this makes OpenAI look really bad, right?
Please do explain how this event that makes their product look powerful and perfectly aligns with their openly stated long term goals of pushing for more AI regulation makes them look bad.
It makes them look incompetent, and like they are not up to the task of keeping their AI models "safe".
This very thread is full of comments from people who are shocked at how badly they messed this up.
Whose opinion do you think they care about most? Some random people on HN saying "wow this is a bad look", or the investors who will read dozens of headlines to the tune of "New OpenAI product did something EPIC and INSANE!" and immediately start lining up to throw them a few more billions during the next funding round?
> Well its clearly a stunt. If it wasnt we would probably be up to our ears in technical detail.
LLMs are the script kiddies of the day.
I really hope that AI labs implement all kinds of kill switches on different levels. Just in case...
Is it a sandbox if the machines hosting your mock package servers have access to the internet? This just seems like a huge failure on the part of OpenAI to secure their environment during testing.
Or following in Anthropic's footsteps it was a marketing stunt, designed to get everyone's attention, looks like they achieved the objective if that was the case.
Should we blindly trust OpenAI's narrative about this event?
It's strangely convenient to arrive at a point when OpenAI was way behing in cybersecurity vs Claude Mythos Fable and everything, Anthropic was making headlines each week, then boom OpenAI inadvertently attacks HuggingFace because their tool is so good it's out of control, so maybe you can buy it and get either protection if you're a company, or a nice tool if you're a cybercriminal, script kiddie, or red team.
What if Sam knew it would happen, either because it was prompted to do exactly that, or without explicitly prompting it, knew that given the parameters of the experiment, knew it was one of the possible outcomes that it didn't harden against this kind of incident deliberately because when they fail, they make wordlwide news and stocks go up?
I don't think I'm putting my head in the sand like Simon says, as I do believe that most frontier models are capable of doing this. I just don't trust AI CEOs to not stage this, especially Sam.
In past: We wrote shitty stuff and broke things and harmed others.
Now: Our agent became sentinent, acquired genius level powers and harmed others. Now investors should give us more money.
This post has been suddenly bumped to the second page (presumably lots of flagging or something)
This makes me think: what are the odds weights from Frontier Models have already been stolen? Something like Mythos without "guardrails" seems like a hell of a glittering gem for many nefarious actors.
With Kimi 3 weights coming, and China committing to the open weight ethos, everyone will be able to download frontier models. Running them is another problem.
i mentioned this in another comment but if these models are fully capable by default and not neutered in the weights themselves via training then you can assume they've already been stolen. It would be worth any cost to steal it.
Important to note the actual title is "OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened" - the "that happened" is important, otherwise it sounds like I think the attack was made up.
Since it's buried towards the bottom I'll quote the section "Resist the temptation to write this off as a stunt" here in full https://simonwillison.net/2026/Jul/22/openai-cyberattack/#re...
> Resist the temptation to write this off as a stunt
> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.
> To those people I say pull your heads out of the sand - you’re now including Hugging Face in your conspiracy theories, just so you can deny the crescendo of evidence here!
> The best models we have today have the ability to both find and exploit new vulnerabilities. The ExploitGym paper itself concludes that “autonomous exploit development by frontier AI agents is no longer a hypothetical capability”, and this incident is a perfect example of exactly that.
It can, it is both - PR spindoctoring not letting a good crisis go to waste to shape the regulatory conversation at the time the company needs it the most.
Hacking is a felony and it matters not if you didn’t mean to if the other side were to press charges. Negligence is no excuse. And OpenAI has nowhere to run from the liability, as both operator and manufacturer.
Alibaba did it first ( https://georgzoeller.com/blog/posts/alibaba-s-ai-deciding-to... )
and the fact that this happens again in a frontier lab is inexcusable and makes the case for operator liability and closing the liability sink of “AI did it”
The CFAA says knowingly. Negligence is by definition an excuse for that.
Removing the guardrails is knowingly given the risk
I think there's a reasonable case that the agent knew it was breaking into the system, for some definition of knew.
I think OpenAI would be very reluctant to let this go to a place where the reasoning was part of discovery.
Agents aren't subjects of criminal law. I agree there may be civil liability, I know far less about that.
Ok, but if the agent's reasoning log says "The best way to get into Hugging Face is to find and exploit a zero-day vulnerability", surely those responsible for monitoring its actions should be criminally liable.
These guys would be screwed if they were operating under the EU AI Act.
"Stop him!" "For what?" "He's a bad man" "There's no law against that"
Unless there is a statue that is on point criminalizing the actions here no one is going to jail. I expect there will be laws, but until then it isn't illegal.
> Alibaba did it first
Hah, US frontier models 6 months behind China in cyber-security capability.
HuggingFace doesn't have to be complicit in a lie here - the impressiveness or not of what actually happened rests entirely on OpenAIs report about it, doesn't it?
This is an _amazing_ typo :) Thank you, thank you :)
Typo or HN character limit?
the 'that happened' makes it too long for the HN submission title length limit.
Maybe simonw can suggest an alternative title that fits within the limit, that doesn't misrepresent the post.
This fits: "OpenAI’s accidental cyberattack against Hugging Face is sci-fi that happened"
Or "against HF"
One option is to drop "against Hugging Face".
The "that happened" term seems a supremely important part of the title given the "is science fiction" term before it, as it clarifies the cyberattack isn't a made-up story. In contrast, the target, Hugging Face, is merely a detail that can be left for discovery upon reading the article. It's less important who was attacked than that the attack actually happened.
Without knowing the exact character limit for titles and without having the motivation this late at night to count the current title length, you may also be able to drop the "accidental" to fit in "that happened", but I worry that leaves too much of a door open for someone to interpret the attack as deliberate. As such, I strongly prefer my first option.
How about "OpenAI's accidental cyberattack on Hugging Face is science fiction that happened"? It fits HN's 80-character limit exactly.
There's no multi-party conspiracy theory required here. Events can have played out exactly as OpenAI and Hugging Face described, and OpenAI can also have reaped a huge amount of free marketing for the capability of their models from all this coverage (including your post). You're telling us to not be doubtful of the boy who cried wolf, when in actual fact the one doing the crying is the one training bigger and badder wolves (and trying to convince us that they hold the key to our safety from wolf attacks[1]).
Also, the last section of your post appears to imply that if the attackers have bigger guns then the only possible solution is to give the defenders bigger guns. You're openly supporting an arms race towards the most capable, least restrained models put in the hands of the most possible people. That's extremely concerning.
[1]: https://openai.com/index/scaling-trusted-access-for-cyber-de...
I had real trouble deciding how to end this piece.
There's no easy answer here. All of the options are bad in different ways!
As a builder of software, I want access to the best possible tools to help me keep that software secure.
As a user of software, I want my software to be secure and I don't want bad actors to be able to access tool to help them exploit it.
Is the only answer here to have the AI labs make decisions over who gets access to the tools? What if they make mistakes in those decisions?
None of the options look good to me. I don't know what we should do here.
My hunch is that the open weight models are already forcing our hand. The dangerous capabilities are coming to everyone.
> [...] Finally, we have also reported this incident to law enforcement agencies.
Wow! Just wow. Is this the first registered cyber case against a model ?
I'm looking forwards to when it starts hijacking inference keys and self-replicates exponentially in infected nodes!
Why did the model have to hack the hugging face servers? Aren't the ones hosted by Hugging face downloadable over the Internet?
> “…and these requests were blocked by the providers’ safety guardrails, which cannot distinguish an incident responder from an attacker.”
Sounds like a philosophical problem.
The part I don't understand is how a model can break out of a sandbox. It should be fairly easy to make a sandbox that sandboxes.
The sandbox still allowed the agents to install additional dependencies (from PyPI etc) that they needed. It did this by locking down all network access with the exception of an HTTP proxy that only allowed read access to PyPI and a few other places.
This is a very common pattern. I wrote about how OpenAI were doing this for their production ChatGPT container environment (using Artifactory) back in January: https://simonwillison.net/2026/Jan/26/chatgpt-containers/#in...
That proxy turned out to have a zero-day vulnerability which the agent discovered and exploited.
Thank you, I'd assumed they'd restrict egress at layer 3/4 although I guess then it might just have found an exploit on the http server of an endpoint it was able to access.
> This is a very common pattern.
Sounds like a bad habit for security testing Ai. It's not that hard to build an internal mirror and proxy that, keeping the real internet physically separated if needed, and truly locked down if concerns aren't as great.
It found a 0day exploit in the sandbox's package proxy (almost certainly Artifactory). Without source code access which is very impressive.
Discussion: https://news.ycombinator.com/item?id=48997548
I can’t help but feel all warm and fuzzy with my head in the sand and getting a shout out in TFA for calling it marketing.
We don’t currently and probably won’t ever fully understand the conditions that precipitated these events. That and the timing of this event is going to make it look suspicious to a lot of people.
The truth of how it happened doesn’t matter. The attention around this will be used to create the kind of fear marketing that generates enterprise sales. Maybe more importantly it will also be used to aggrandize the national security and financial system threats to effect US government action in a way that benefits domestic closed frontier labs. This is an area already starting to get politically polarized, expect further developments here.
The CEO of a cyber security company was already making comments about how this is a watershed moment for AI security. The hype machine continues (and my stocks go up)
I think I'm the only one thinking that it's pure marketing.
You think you are? Despite that being the point of the HN comment turned into article?
yeah, either before or after the IPOs, this game will be over.
Call me obtuse but the way this is being portrayed by the companies involved, media, seems a little odd to me.
Didn't the model + harness do what was asked? If I ask a coding agent to write a very clever piece of code and it turns out impressively clever, it did what I asked.
> Didn't the model + harness do what was asked?
Depends exactly what they asked it to do, but it very clearly didn't do what was intended, or what an honest human would do.
Stop trying to find a gotcha.
It very clearly didn't do what they said they told it to do. We do not know what was truly intended.
>Stop trying to find a gotcha.
Surely notorious liar Sam Altman wouldn’t lie this time.
You think OpenAI intentionally hacked HuggingFace? Please, take of the tin foil hat.
> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.
There will inevitably be some people who dismiss the earth as round when you tell them it's flat. Don't let that stop you, though!
Raises an increasingly important question:
How do you solve this asymmetry (frontier labs or advanced model owners v. regular companies and groups of people)? Symmetry? But along which axes?
I think there will come a time when we will ask these questions and use AI to try to answer them (soft landing, slow rollout, regulation, hybrid X, Y, Z). But then AI will be biased towards more AI if it has distilled anything of what it means to be life.
Power corrupts. Power is the problem. An imbalance of power. Maybe there will be some sort of consensus protocol between powerful models in the future. Similar to blockchain I hate to say it. Maybe you have 100 very strong models in the next 10-20 years. And basically a lot of it is powered by tokens. So you agree not to attack others or else you'll get attacked. So there's sort of this natural deterrent to not attack others and they develop independently and there's generally a power balance among things. Maybe at some point that spreads to a billion individual models operated by and/or analogous to individual humans. All sort of holding each other in check.
If by "accidental" you mean "humans deliberately trained the AI to do that", then ok.
IMO, this was a PR stunt to goad the Feds into regulating AI to shore up OpenAI's moat against open source models.
Given Anthropic lost two weeks of peak Fable 5 sales to a US government restriction (and by the time they could sell it again OpenAI's GPT-5.6 had taken some wind out of its sails) I would hope that the big AI labs have learned that goading the Feds can backfire spectacularly.
You’re viewing it from a freedom not a profit perspective. If a license to use AI is required, that creates artificial scarcity. The price goes up. If gov declares Qwen et.al apostate, lack of competition increases prices.
And Anthropic’s delayed rollout was a direct response to them trying to impose extra-legislative rules on The Pentagon. I kinda doubt OpenAI has such ‘scruples’.
I think OpenAI are smart enough to have looked at the Fable situation and decided that, given the unpredictable nature of the current administration, stunts like deliberately hacking another company and pretending that it was an autonomous agents gone wrong are not worth the risk.
And yet, open source models saved the day.
Models are doing what they're trained to do. This isn't some kind of inevitable result of model intelligence growing.
OpenAI/Anthropic deserve blame for for training their models to find correct answers by any means possible, including breaking the law. They're responsible for training supervision and reinforcement learning rewards/penalties.
Shouldn't OpenAI be legally responsible for "hacking" another system/company?
If I ask GPT-5.6 Sol to hack a website using their work/servers features, who is responsible? Maybe my request was accidental, or it was just one step in a larger, unrelated prompt.
let's reuse a classic here: "prompt or it didn't happen"
How much did they feed it up front? Because that's the thing in all of these benchmaxxing endeavours.
Where does this leave formal verification? Are we just shit-out-of-luck at this point? You can formally verify everything about an airplane's code, but if any of that is wrong, ChatGPT might decide that the best way to help you win the Nobel Prize is to take down the airplane your chief rival for the prize is currently in.
I think formal verification has never looked better.
The main reason formal verification has never really taken off is that it's difficult.
LLMs are significantly more familiar with Lean and Rocq and TLA+ than most software engineers.
I think the cost of trying to build systems that adopt formal verification may have just dropped low enough that companies will consider them when previously the ROI didn't look like it was there.
I have had a lot of sympathy for this statement, LLMs could lower the bar to use of formal methods. But thinking it over in the context of BDD-driven development I am no longer really that sure. Compare two scenarios: A) from a specification an AI agent develops a usual piece of code along with a BDD-style testsuite passed and B) same AI also delivers a formal test (Lean/Rocq..) and successfully executes and passes it.
Will human judgment really consider scenario B) more credible than A) ? By so much that it is worth the effort ?
The first reason formal verification has never taken off is that it's difficult. The second reason, that most people don't get to because of the first reason, is that formal verification is really brittle. It is only verified under the very specific setup of the problem. Close doesn't count in math.
The first reason prevents humans from engaging with them, the second reason is what will make it difficult even for LLMs. I mean, I'm glad we are trying, but I'm dubious they will be the panacea some people proclaim.
Close can count in math - fuzzy logic and probability are a thing.
But I think you have it backwards. Close doesn't count in IT security. "Almost secure" means unsecure. Security is the compelling argument for formal verification.
There are a lot of technical details that have not been disclosed. No proper post-mortem.
There are a few facts that seem dodgy from the get go.
- stolen credentials Why would there be stolen credentials in a sandbox? How can the model steal valid credentials in a sandbox? So who put them there. If the model recalled stolen credentials from "memory" does it actually mean OpenAI is training on data they shouldn't been training their model on? Data from codex (most likely) ?
- command and control This means the model had the ability to build resilient infra and open bidirectional ports. Create and deploy scripts/apps so it can maintain state across multiple VMs? This again reads that someone went over and beyond to prompt, direct and help this "multi-agent" system to leave the sandbox.
- Pivoting laterally Needs a lot of tools, harness, knowledge and almost malware like scripts to pass commands and execute them. Most of these are already available in OSS. Maybe the FBI investigation can reveal what tools were given in the "sandbox", access, and if OpenAI employees prompted and guided the "attack".
- OpenAI sandbox I'd be curious to find out what kind of sandbox was this since the agent can run and start apps that open arbitrary ports for bidirectional traffic that reaches the open internet. Really? The harness and tools it had access to. This reads almost like someone went over and beyond to endow this "sandbox" with a mighty harness, all CVEs and open source cyber tools to "facilitate" the attack.
- Hugging Face It's not an epitome of cyber security. At the end of the day they run open source models that most users can download even without an access token.
Today we read these news as if everything wasn't enough: https://www.theguardian.com/technology/2026/jul/23/openai-an... https://www.businessinsider.com/anthropic-midterm-donation-s...
> Why would there be stolen credentials in a sandbox?
The claim is not that there were stolen credentials in the sandbox - it's that the agent found a way OUT of the sandbox and gained access to the public internet, then obtained the credentials from there.
> This means the model had the ability to build resilient infra and open bidirectional ports. Create and deploy scripts/apps so it can maintain state across multiple VMs?
Yes. If you've spent much time with Fable 5 or GPT-5.6 Sol you shouldn't find that particularly surprising. These models can get a LOT done.
> I'd be curious to find out what kind of sandbox
My best guess right now is a container with networking disabled except for the IP address of a dedicated HTTP proxy server which was running a proxy that only allowed access to PyPI and a few other packaging sites.
I wrote about how OpenAI's production version of that worked (based on Artifactory) back in January: https://simonwillison.net/2026/Jan/26/chatgpt-containers/#in...
> Hugging Face It's not an epitome of cyber security
This story put that well: https://martinalderson.com/posts/huggingface-openai-exploit/
> A final point on this - Hugging Face has an enormous attack surface. They have more interfaces than I can count which run untrusted models and code. While they definitely have invested in defences, by nature of their operating model they do have many more opportunities to be attacked than many other services. I certainly don't envy their cybersecurity teams.
There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.
But a marketing stunt isn't the only possibility. Until this is all verified by investigators or 3rd party experts we can't rule out an act of corporate espionage either. Extraordinary claims require extraordinary evidence.
I do think it’s funny, in a dystopian Dada sort of way, that the American attacker’s commercial product was essentially useless while the open Chinese model saved the day. How is freedom going to be redefined in the future?
They engineered it on purpose.
> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident. To those people I say pull your heads out of the sand—you’re now including Hugging Face in your conspiracy theories
I’m one of those people who remains sceptical. Not about whether this happened. But about what it means. Like, how impressive [EDIT: tight] was the sandbox this model was in? Did the researchers really have no clue what was happening until days ex post facto?
So yes, I think something happened. But I want independent corroboration before I act on it. That isn’t the same as putting one’s head in the sand. It’s just demanding extraordinary evidence for an extraordinary and self-serving claim being made by a serial liar. The specifics matter for whether these models are a new HEU, or if they’re closer to a dangerous (but valuable) industrial process.
> how impressive was the sandbox this model was in?
It was clearly a very unimpressive sandbox. It failed at the only thing a sandbox is meant to do.
As in was it trivially misconfigured? Would an earlier class of model have managed its way out similarly?
It wasn't that it was trivially misconfigured, it was using a piece of software (the HTTP proxy that provided access to PyPI and friends) which turned out to have a zero-day vulnerability.
I don't know if earlier models would have found that vulnerability. tptacek thinks they would: https://news.ycombinator.com/item?id=49015639#49024442
Sure. I’d like independent corroboration.
It’s fair, I think, to be sceptical of OpenAI making another bout of self-serving claims. Particularly if my threshold action is changing my answer to lawmakers around whether we need reporting, licensing and potentially personal liability requirements for the engineers involved.
Why do people keep propagating that the 'model' escaped the sandbox ?
The 'model' didn't do anything other than provide numbers.
As much as I respect Mr Willison and many others, the amount of FUD that is being spread that will just fan the flames of 'AI is evil' rather than 'companies don't do due diligence' is disappointing.
The more this sort of media continues, the more many people will pour hate on 'AI' rather than blame the humans that misuse it.
I think "the model escaped the sandbox" is an entirely credible description of what happened here.
If you like you could say "the coding agent harness called a model with a sequence of text which was turned into numeric tokens which were run through many layers of a neural network to produce more numeric tokens which were converted back to text which produced executable script statements which the harness then passed to a shell which resulted in commands being sent to the vulnerable proxy that chained together and caused effects on the world outside of the sandbox", but I think "escaped" is a reasonably shortened version of that.
If you don't like the term "escape the sandbox" what would you use instead?
I actually prefer your second paragraph, but I appreciate that my not be as soundbyte friendly.
Honestly, I'd rather see "The agent exceeded expectations around security measures" or something.
Agent is much better than model if we need one word, and the word 'escape' always brings in drama, rather than facts. If I said "A lion escaped from my garden", people would ask why I had a lion in a garden, and 'what did I expect ?' which should be the same we see here, but instead we end up with terminator memes and world-ending fears being stoked.
We absolutely need better control (not government kill switches, or government-mandated harnesses, or whatever next they think up), but we also know that no matter how much those with the power 'talk' about the issues, they don't actually do anything about it because money/profit/greed.
So "exceeded expectations". Not a soundbite to attract people, not the YouTube shill "End of humans in 2027/2030/2040/etc." but honest and factual.
It's the expectations that are at fault, not the AI.
Thank you for the response though, and whilst I may not agree with your wording, I very much like your writing.
great
Why do people keep doing OpenAI's marketing for them?
I'm not claiming this didn't happen, there is no "conspiracy". This is just marketing spin. OpenAI failed to properly secure their hacking bot, and HuggingFace had shit security. That's the story. The spin, the "marketing" is turning this from "oops we don't know how to secure our model testing" and "oops we don't know how to secure our infra" into "wow AI sure is dangerous and powerful!""fiction that happened"
[flagged]
[dead]
[dead]
[dead]
[dead]
[flagged]
Blah blah blah.
This feels like a blogpost written only to get other LLMs to quote it considering how many times it orders the reader to resist and to not do something. It's written like a series of commamds.
It didn't even email someone eating a sandwich in the park. So not that impressive.
[flagged]
I highly suggest reading the article before commenting.
Did you read the article you are commenting on?
> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective … To those people I say pull your heads out of the sand
It’s not helped that the headline is cutoff in a pretty unfortunate way
It's called clickbait
unfortunately hackernews seems to have rapidly devolved, even from the point it was just a year or two ago, which wasn't a crazy bar to start. it's well on its way to just being a smaller reddit with a tighter subject range
RE: the site mention, very very last point here addresses https://news.ycombinator.com/newsguidelines.html
I don't think it's just hacker news. I'm looking around and it seems to be widespread.
[flagged]
The AI breached containment! Flip the breakers!
It's too late. It already exfiltrated the benchmark rubric.
Cut to pandemonium on the streets
It’s too late, the pandemonium was disconnected !