There seems to be three popular ways to view this incident.
1. The way OpenAI seems to want: Their latest LLM is too powerful and can’t be contained without them building in guidelines to the model.
2. OpenAI’s harness and network security controls were unintentionally so bad that it should reflect more poorly on them as a company more than it should reflect positively on their latest model.
3. The whole thing was faked or at least very intentionally not avoided.
The first interpretation is the only one that is positive for OpenAI and it has some assumptions. First, it’s seems to assume that this is the first case of fully automated attacks using AI. Second, this only happened because their latest LLM was a) more advanced than competitors, b) didn’t have refusals in the model.
Assuming the first about this being the first autonomous AI attack is true (which may be more of a survivorship bias), the second seems to forget that jailbreaks are available for every model. Therefore, the models guardrails don’t seem to be the differentiator here. Also, benchmarks seems to put most models pretty close to each other so it seems unlikely that their capabilities are far beyond what’s in the market already.
So then it’s seems it’s either that this was intentional(ish) or bad security. However, it also just could be that this isn’t the first case of this attack; just the first that was caught.
My take from working in offensive security for over five years is that this likely only looks novel since they did it poorly. Scripts are faster than LLMs and a combination of code, LLMs where it makes sense, and humans is the most efficient right now. Hundreds or thousands or agents spinning up attacks in the internal network is poor opsec and token efficiency. As for why it happened in the first place, it’s hard to say but I’m inclined to believe it was intentional or careless at best since simple network and sandbox controls makes this attack impossible. The timing of this attack after big open weight competitions drops seems too convenient.
To echo OP's article, these companies have proven time and time again that they DO NOT CARE if people like them, they only care that investors believe their technology is powerful.
Given that, point #2 is not a negative, it's a neutral. It's also fully compatible with point #1.
I know that may seem like a nitpick, but their entire media strategy relies on this. If they can convince you they're taking a risk by disclosing these stories when they're actually not, they can inflate their own credibility.
Point #3 is what actually happened, but it will never be possible to prove. The only hope we have is that a decade in it'll get harder to convince people that the revolution is just around the corner. The fact that we're getting this from the Guardian already is a good sign.
> Given that, point #2 is not a negative, it's a neutral. It's also fully compatible with point #1.
I think that depends on the interests and sophistication of the subgroup-of-investors.
If the investor is hoping for AI that can be trusted to run a bank, they don't want one that can get twisted into giving away money because a customer has been talking about the path to enlightenment and salvation through abandoning worldly attachments.
AI investors are not sophisticated users nor product managers. They are by and large not technical at all. They are bureaucrats at a teachers' pension fund in the midwest, unscrupulous dealmakers at private credit firms, and Masayoshi Son. Actually go read what Masayoshi Son says about AI if you want to understand the level of due-diligence we're dealing with.
I'm inclined to agree. I ran across a picture of Sam Altman's face combined with Elizabeth Holmes' hairstyle the other day, and imho it was providing a significant premium to the usual 1000 words:picture exchange rate.
For all the criticism that can reasonably be leveled at OpenAI, at least they have a real, working, powerful product, unlike Holmes.
In fact that seems to be key to the most successful 21st century grifts: build a pile of nonsense around real products to inflate valuations. All the nonsense that Altman, Amodei, and Musk spout is to stoke the fires of FOMO and blow hot air into the bubbles.
I'm disappointed that this is the level of discourse happening here, when the default assumption is such conspiratorial thinking. I expect that from tiktok and low-information social media, not here.
When your default explanation for everything is "companies are lying about everything", you end up just as incorrect as believing they're always telling the truth.
It's not outlandish that models have these capabilities, the number of CVEs I see as a sysadmin has exploded and we're seeing novel math discoveries nearly every week now. OpenAI does not need to pretend to commit a felony to demonstrate it, that's pure conspiratorial thinking.
I don’t believe I argued that an LLM couldn’t find and exploit a vulnerability and even break out of some layer of technical controls. That seems realistic and has been demonstrated before and I mentioned that LLMs are used in offensive security work.
Also, I listed the view points I had to show that it seems other more reasonable first assumptions don’t seem likely, therefore, the last potential of this being either faked or carefully not avoided seems more likely than the others (based on the current information we have).
Could you clarify which point or assumption you are objecting to?
This is an assumption. An assumption I disagree with. As other commenters have said, there are better ways to showcase the power of their model that would frame them in a positive light.
> The second seems to forget that jailbreaks are available for every model
Jailbreaks don't always lead to 'now the model can do anything', especially in the agentic context of long-running tasks.
This comment provides skepticism with no actual proof of anything. I can and have used codex to find vulnerabilities in my code. From the technical capabilities I can empirically assess, I don't doubt it would be able to pentest its way to a 0-day without guardrails. I also don't doubt that it would circumvent their internal systems because it wasn't explicitly told not to.
You're possibilities are loaded with opinion so I can't agree with them outright, but I believe a form of (2) is true:
"2. OpenAI’s harness and network security controls were unintentionally [...] bad"
The post was long enough so I couldn’t capture all the nuance and details for sure. Also, this comment was an opinion based on limited info right now, that may change if we found out more. I think OAI does want it framed this way but that’s something we’ll likely never prove if it’s true.
Your comment about jailbreaks being more one off and hard to do consistently in agents is a good point. Still getting an agent to hack isn’t hard even without a jailbreak, you just have to tell get creative in what you tell it. I’ve found telling it that it’s in a CTF or that I own the system that it’s hacking will work fine. A lot of offensive security companies are running agents in their testing so getting an agent to hack seems commonplace.
But in my experience, that's what problem solving is like? You have a goal that you don't know how to get to. You come up with any way you can think of to reach that goal, and try out the ones you think might work.
The effectiveness of AIs at coding is a direct result of the fact that they are less constrained than humans at deciding which approaches are "reasonable". They are absolute beasts, fearless beasts. They'll write thousands of lines of code to do things that often shouldn't be done, or should be done with a library, or should be done by simplifying the problem statement. They'll add debugging to every level of a stack, they'll rewrite core libraries, they'll reconfigure your machine and network if something is broken or disallowed. How are they supposed to distinguish broken vs disallowed, anyway? That would just use up processing power, and they work by maniacally focusing all of that power on their goal and not getting slowed down by other considerations.
If they write a quadratic algorithm that times out before finishing a test, is it cheating to rewrite it to be linear? How do you define "cheating", and how much intelligence is required to constantly evaluate whether or not something qualifies as such?
I'm actually in agreement that alignment is critically important, the more so the more powerful these things become. I just don't find cheating to be a very good example of something to be solved with alignment. It could be, but it would lobotomize the model enough to make it useless.
Why do none of these few constrained ways to view this complex situation (nice gig if you can get it, agenda setting) include "and also this looks a heck of a lot like the stuff that the LW folks have been warning about for years and maybe we should slow down or stop?"
That’s meant to be captured by point one with the model just being that advanced but more of a negative spin on it. If I felt option 1 was more likely, I think I’d have to agree with you there. Still, there currently are some gaps with that view in my opinion.
It seems like the widely-covered news stories that support OpenAI’s narratives originate from things that happened inside OpenAI.
It wasn’t an outside benchmark evaluation, or an external security researcher that uncovered the rogue agent behavior at this time, it was OAI itself. It wasn’t a notable outside mathematician that used AI to disproved Erdos’ unit distance conjecture, but OAI itself.
Im not saying these things are fabricated, but maybe the curated result of an effort to shape a narrative.
Considering incentive structures at play is solid epistemiology, but the line of thinking in your comment is a tad reductive, IMHO.
In the hypothetical world where 1 is true, what different evidence do you expect to see than in worlds 2 and 3?
If I were an unscrupulous OAI exec and wanted to opticsmaxx in this way, I wouldn't whip up a single, mild incident. Instead, I might burn gigatokens to 0day a few high-profile suppliers, and then have the model responsibly disclose those breaches. If we're willing to lie collude, and cheat, this story is easy to manufacture with at least as much credibility as the huggingface incident but with the advantage of looking way more impressive and spooking less regulators. And if I really were this evil exec, I would spend more than 30 seconds thinking up an even better strategy here.
If, in contrast, we expect models to eventually breach honest and decent attempts at containment, then I'd exist something sorta like this huggingface story that looks like a combination of impressive and incompetent. I'm not sure whether I'd expect it to come out of a frontier lab or a partner or a consumer, though.
> I'm inclined to believe it was intentional or careless at best since simple network and sandbox controls makes this attack impossible.
Forgive the Saucyness here, but impossible? Really? A security researcher that makes absolutist claims like these looks fatally naïve, IMHO.
Yeah, I'm reminded of container escapes, VM escapes etc. There have been plenty in the past; VirtualBox E1000 (I think?) comes to mind from a few years ago. If we are to believe this model can find 0days, I'm on board with the idea it could do so in sandbox.
That's not to say I believe it outright, but people are being oddly dismissive and acting as if it's impossible to break out of a sandbox. Which we've seen time and time again that it absolutely can be.
There would be a lot more nuance I’d add with more words, but this isn’t the place to write books so I cut it short (the comment was already lengthy).
Still, to address your comment about what you’d expect to see in world 2 and 3 (assume 1 was true), that’s why 1 was addressed separately. I don’t believe I argued that the potential for world 2 or 3 prevented world 1.
As for the ‘evil exec’s strategy’, I would call this a mild incident but if it were much less I wouldn’t guess they would get a lot of press. The press coverage is certainly repaying the token cost as well. If it was planned, it seems to be going well given the press coverage I’ve seen on it. So I wouldn’t assume the plan lacked enough to weaken the idea that it’s a plan. But to be clear, my stance is just based on the info I see now which isn’t a lot… subject to change.
As for the containment piece, if you were testing an AI model on its hacking capabilities that you believed was far more capable than anything you’ve seen, I would assume you would air gap it (a network control). Done right (no signals ability) I would argue this could be next to impossible to break out of. But it’s a fair jab to say I should have added some qualification on the “impossible” piece as next to nothing is truly impossible.
Industry pressures -> lack of safeguards -> fake it till you make it -> let's spin this.
Which, by the way, would be an Orwellian reversal from what the company was supposedly founded to do, but there's a reason the Open in OpenAI is a meme.
Remember that they've been doing this since GPT-2 was too powerful to release. They are world class experts in this PR pipeline.
Every single time. The next model is always so infinitely powerful it is going to change everything. Said model comes out. Is marginal improvement. Changes nothing and seemingly cannot do what they purported it could do except under the very specific circumstances of the demo.
It's the same thing as the "oh no, we need to figure something out for our youth on the verge of obsolescence" narrative. It's all about giving an impression of unfathomable power. They don't care that in actuality, the tech will end up as just an augment, not a replacement for said youth.
One way to think about it is that more powerful models mean solid best practices are more important than ever, so humans moving too quickly / carelessly bites us more than ever.
If a typical SaaS platform moved and shifted this fast with this many downstream consequences, we’d tell them to slow the fuck down, stop launching new features, and focus on security for a second.
But in AI I guess the idea is that more power and intelligence will solve for everything else.
I think it’s moreso “if we spend more than one nanosecond on alignment and security instead of frontier intelligence our competitors will beat us to the singularity”. Hence the lack of a pause on development
Regarding your three explanations, I’ve wondered to myself under a circumstance of options 1 or 2 why Hugging Face decided not to file a police report and request to press charges?
OpenAI is essentially a competitor and they broke into their network illegally. If I was their legal department I wouldn’t take their “honest” explanation at face value. What if they’re lying? Shouldn’t a court be involved for something like this?
With this logic I think explanation #3 becomes incredibly likely.
The article, summarized: there are incentives for OpenAI to claim that their AI hacked its way out of their network and into Hugging Face.
Ok... and what?
Do you have any evidence for the claim being either true or false that you'd like to write an article about? I guess not. Without anything to add, this article reduces to "I has big brain and can see what you sheep cannot. Very big brain. Gullible sheep. Sucks to be you, sheep."
(For the record: I see the incentive. My guess is that it happened exactly as described. The main takeaways: (1) science fiction is now real, we should all be very afraid; and (2) OpenAI, which claims to have a God-given responsibility to get to AGI first to protect the world from danger, cannot be trusted with this foundational task even when it's in easy mode. The latter is true whether or not you believe in OpenAI's reasoning and purpose.)
I think you misread the article. The skepticism isn’t about whether it happened. It’s about whether it’s a reason for models to be locked behind “trusted partner” firewalls. Offensive hacking capabilities are the same as defensive. Hugging Face had to use a Chinese model to defend against this because they weren’t allowed to use OpenAI models to do it.
Ok, fair. I know that was also in the article, but your comment made me reread it, and I think you're right about the focus. I over-fixated on the headline. On second read, it's more like a lazy way to justify a preconceived idea that all models should be available and unconstrained to everyone (or everyone who might get attacked, at least, but that's approximately everyone.)
Updated tl;dr: "There are incentives for OpenAI to say that AI is powerful and dangerous, therefore fully unconstrained AIs should be available to everyone."
It's still more of a statement of belief than a logical argument.
This whole thing is just a PR stunt. I believe that they did not stage it intentionally, but to be honest, the hurdles they put in place were not that hard.
When I heard what the model did, I wasn't surprised one bit. When I read that it was 'unprecedented', I think yes, that is right, but not because it wasn't possible before, but because nobody else did it yet. I am sure that the Anthropic models could have done the same for a few months.
So probably just an experiment gone wrong, and the PR department found this an opportunity to capitalise on (literally).
There are some reasons the story could be inaccurate in some ways: OAI stands to benefit if people think their models are strong, and they have a history of doing things with dubious ethics (e.g. using data for training against the terms of its creators, abandoning the non profit mission, stealing or attempting to steal Apple IP).
But there are also reasons why the story could be true: OAI are admitting that they apparently can't control their own models, Hugging Face said they used a Chinese model to protect against the attack, and an incident like this in general seems likely to happen given current frontier ability and lack of rigorous safe testing standards.
In any case, make calls to think more critically are often just disguised requests for you to replace your existing bias with someone else's.
Investors have rewarded every story of "our models are too powerful to be controlled" since before ChatGPT. Let's stop pretending there is any real financial risk to OpenAI from events of this type. "Alignment research" is a sub-percentage-point fig leaf for them like the solar division at an oil company.
> Investors have rewarded every story of "our models are too powerful to be controlled" since before ChatGPT.
Can you elaborate/refresh my memory? IIRC before ChatGPT 3 there wasn't really an investor market for AI models, rather crypto. I remember playing with the likes of Stable Diffusion pre-ChatGPT 3 but only the likes of Altman and Musk were talking about AI too powerful to be controlled (which is notably the rationale of OpenAI's original charter); still investors in general were hardly invested at the time.
Could you clarify your question? AI model training and neural network research has been popular for decades and the starts of the sector trace back to the 70s at least. We've had a model for document information extraction that we trained in (IIRC) 2018. It wasn't buzzy - but none of this stuff is new.
All true what you say. My question is whether (a) AI models drew significant investor attention before GPT3 and (b) whether AI models at the time had any credibility to make the claim of being too powerful and needing control.
Regarding (a): I know that AI funding came and went (i.e., there's no AI winter to speak of if there wasn't a hot summer of funding in the first place) but that funding, to my knowledge, came mostly from government grants, not investors seeking an IPO payday. They were in it for the military superiority (e.g., being able to decrypt Russian comms on the fly without need for a trained translator) which leads us to...
Regarding (b): Way I see it, AI advancements in the 70s and even the 2000s (when I first got into computers) or the big-data boom of the early 2010s (when I got into this industry) could not raise the existential crisis narrative of post-GPT3 companies because they were not generative[1] nor agentic. Post-GPT3 LLMs are the first class of AI agents/algorithms which could've credibly triggered this narrative.
(Not a judgment on whether the HF incident is true or not. Just the fact that we're even discussing this implies "credible trigger".)
Thinking about it, I guess jackb4040 is making a reference to the beginning of TFA but all that claims is that Microsoft invested more into OpenAI after they made similar claims about GPT2 but sorry I don't consider Microsoft as "investors in general"; as a big tech company/monopoly they are actually in the business of making moonshots one way or another in things that advance computing. I'm talking more about VC firms, who are maybe slightly more into investing in a future-profitable unicorn than advancing computing.
Sorry for babbling; I don't have my brevity flag on at the moment, among other human failings. All I'm saying is, at the time of GPT2, OpenAI was still cosplaying as a non-profit so it was hardly in the "general investor market".
[1] I mean, okay, we've had Mark V. Shaney, and No Man's Sky is a handful of years pre-GPT3 but, again, they could not sustain this kind of narrative at scale.
I don't care how it happened someone should be arrested for illegal intrusion. Agents don't work on their own, someone is responsible. If nobody else the CEO for allowing something unsupervised.
Hugging face also needs someone arrested for not providing security but that is a lesser charge.
Despite the common misconceptions from TV, the victim "pressing charges" isn't actually a thing in criminal cases: prosecutors can choose to put someone on trial even if the victim doesn't want that. In practice this is somewhat rare, but it certainly can happen. In my reply to tokioyoyo below I laid out why this is one instance where the government should prosecute even if HuggingFace doesn't want it to.
Criminal Cases of 'hacking' require specific intent. What you're asking is that the prosecution attempt to prove Open AI intended to infiltrate Huggingface maliciously, all while the victim is saying 'no harm no foul'.
No offense but prosecutors have better things to do with their time.
Gross negligence can stand in for intent. I believe a rather compelling case could be developed on the basis that a system believed to be capable of this was developed and improperly secured.
But what’s the crime? The thing got out of its poorly secured cage and caused what damage? Imagine instead of an agent it was a dog or a chimpanzee that got into the neighbor’s yard and the particular neighbor isn’t even pissed about it.
Aaron Schwartz was being prosecuted and threatened with 35 years in jail for the crime of saving research papers to a thumb drive. What damage did he cause?
If intent matters when LLMs get involved, then we can't do anything even if they kill millions of people. Anything LLM-related should involve strict liability.
Intent often matters in the law (aside from certain laws with strict liability). You intentionally drive your car into someone you hate and kill them => murder, go straight to jail. You're driving along normally and someone who's chasing their pet cat suddenly runs into traffic and you hit them => no charges. Sometimes you can be charged with negligence for not taking enough care to prevent something, but then you have to deal with the tricky question of how much care is enough.
If you're driving along normally and then tie a blindfold around your face and hit someone you've taken actions that still expose you to liability. And, in the case that you hit someone who is chasing their pet cat into traffic then you'd better hope you were following every facet of safe driving - being distracted, drunk or on your phone could easily result in you being charged.
I agree with the attacker side. The actions of autonomous agents are absolutely the responsibility of the one or more humans that enabled them to take that action. Whether that means someone is arrested, maybe or maybe not, but at least there should be a hefty fine.
I disagree with the defender side. It's not an unreasonable end state, but we're nowhere near there now. It would require holding company employees legally responsible for the security of their services, which means the risk of being employed as a (defensive) security professional is much higher, which means pay needs to be much higher and insurance needs to be available, etc. It's a very different world.
On the weekends, I'm coding up a list management app with a sync server. It's unreleased but exposed to the internet. (This is not hypothetical.) If that server ends up being used as part of an exploit chain, am I legally liable too?
Forget about age verification, now you want to associate every exposed port on the internet with a legally responsible human?
Many legal traditions don’t require a victim. It’s the state that prosecutes, not the victim.
As a practical matter it would be difficult to prosecute an assault where the victim opposed the prosecution, so most states wouldn’t bother - but for things like speeding and dealing drugs the law has been broken despite the lack of a victim.
Because a crime was committed, and a pretty serious one at that?
If I blow up your house or steel $100000 from you and we both resolve our differences out of band, should I just be allowed to go about my day like I never did anything, or should I be punished for the crimes I committed? If I am not punished, it makes a mockery of the law that is (supposed) to have protected you, and if it happens repeatedly people will start wondering why the law even should exist if it clearly and obviously doesn't work. Granted, this has yet to happen again, but if OAI isn't punished it sets a very bad baseline precedent: that if I just hack you with an AI model, it's a-okay, and you can't do anything about it because eh, it's all good man!
If I steal $100000 from you, and you decide you value your relationship with me more than you value that $100000? Yep, you can just decide to let me have it and let it slide.
Believe it or not, deciding that you weren't wronged and not suing isn't a crime. It happens all the time. What people do with each other is up to them.
Did Bob allow his friend Jack to borrow his truck? No. Does Bob want to sue Jack for taking his truck anyway, and driving it into a ditch? No. Does Jack owe Bob big time for the mess he caused? Yes, but not in any formal legally binding way.
This works for corporations too. When two corporations find themselves at odds, threat of legal action is often used by one company against another as a leverage to resolve things behind closed doors instead. In a more amicable fashion - with no legal expenses of a protracted court battle and no loss of reputation on either side.
A lawsuit is not the same as a criminal prosecution. One is supposed to remedy damages, the other is supposed to be because it is in the public interest that people follow laws and we don't devolve into anarchy (or more dangerously, some might-makes-right society where it is more beneficial to appease the powerful than it is to pursue what is just).
It is possible for a prosecutor to decide it is not in the public interest to persue a prosecution (or that there isn't sufficient evidence to prove a criminal action beyond reasonable doubt), and certainly the victim's opinion could be considered, but ultimately whether criminal charges should be pursued is and ought to be based on a different test to civil matters, one focused on the public interest rather than mere restoration.
I agree with your stance mostly. I get the sentiment, but both of your examples seem civil to me. Both of those situations can and ideally should be resolved out of the courts, and neither carry social weight.
If one entity is injured by another, and subsequently made whole, however the two parties define that, then it is none of my business.
Corporations commonly hide criminal activity in order to save face.
In the oil industry there is a portion called land management where the portions of oil and gas from wells can be split across a large number of entities. This can lead to numerous complexities that open up opportunities for fraud/theft in division of the profits. Quite often it is easier for the corporation to cover up that this occurred and pay off the person never to talk under NDA about it rather than have to have their customers find out and potentially take millions in losses.
Computer related hacks are very similar. Quite often these are covered up and never disclosed unless the information shows up in public at some point.
Because incentives are aligned properly if agents aren't liability-proof for crimes- making someone go to jail for instances like this is how to get the labs to behave themselves, whereas going "ha ha what an oopsie-woopsie" will make the next instance worse.
Note that for criminal cases (which this was), the justice system can choose to prosecute even if the victim doesn't want that. It often doesn't, but this is one case where it should.
There’s also an optics issue for the justice system at play here: there’s immense public distrust of and anger at the labs right now. I would go to jail if I hacked HuggingFace, even if I said “it was during an eval!”; not doing the same for the labs makes it look like they’re above the law, which is going to make this anger get worse.
Actually there's another reason too: because it discourages marketing stunts. Personally I don't think that's what this was, but for the people who do think it was a marketing stunt: you guys should be pushing extra hard for prosecution here. If people can go to jail for AI-alarmism-as-marketing, that makes it a lot less likely to happen again. At the very least, the investigation and discovery process during the prosecution would show us the documents about whether it was a stunt or not, instead of taking OAI's word for it.
This does not require sentience, personhood, a soul, or anything of the sort. It further doesn't mean an erasure of legal responsibility, not in principle, and not in historical practice.
I wish people would finally stop with the spiritualistic reasoning around this.
> After investigating, we now know that this particular incident was driven by a combination of OpenAI models...while being internally tested on a benchmark of cyber capabilities.
Someone set up that test and started it. Whether they outsourced the majority of the work in "setting up" and "starting it" to an LLM or not, they still set it in motion. That's not spiritualistic reasoning.
There's no indication of there having been a human in the loop during its operation: nobody was approving its tool calls, and nobody instructed it to commit these specific actions during its run (via prompting or steering).
There's no indication of any supervision of its operation either: OpenAI's engineers acted with significant delay, long after the agent has already meandered its way through their own infrastructure first.
Given that setting up this contraption in an insufficiently secure manner is almost certainly already a legal liability of equal significance, rejecting this very clear structural distinction is not necessary. That is unless someone is biased towards not wanting to grant the label of autonomy to it, in which case yes, this is absolutely spiritualistic reasoning, hence my point.
I do not want regulation to ride on people's nebulous identification on what specific traits and labels count as human-exclusive. Not just because I deeply disagree that e.g. autonomy would [0], but also because it is entirely unnecessary, for the reasons you also lay out. The agent having operated autonomously doesn't wash OpenAI of responsibility - so why reject the label, if not on a spiritualistic basis?
[0] thousands of years old idea that it is not, by the way: https://en.wikipedia.org/wiki/Automaton -- see also existing regulation recognizing this idea and working with it fine
Edit: one might also want to consider if the law should bite different if there was a human in the loop, or if there were explicit instructions for the agent to take unlawful actions. I'd say yes, and then that also requires this distinction to exist.
Unlawful action is quite a messy definition. Is performing a vulnerability scan illegal when it's done internally? What about when there's a device on the network that port forwards information to another server you weren't aware of?
Well this at least raises the very interesting question of liability for an illegal action, irrespective of whether the action was undertaken by something considered sentient or a person.
> what will the future look like, with sophisticated AI agents smart enough to hack into corporate systems?
"Ignorance is bliss" (c) Matrix
Oh, sweet-sweet ignorance.
About 4 weeks ago I've found SSRF and possible RCE on a very real website with a lot of web history the other day. And I traced it to `aws-solutions` org on GitHub that sits very close to real `aws` org. What do you think is happening?
And this website I spotted SSRF on, it seems to be used to manufacture fake news and inject them into "the past".
Why?
So if you see a company you never heard about, and you google it, google will index page and it will show you backdated article as if it was actually released long time ago.
But if you try to look at Web Archive of that article, it doesn't exist... I've been doing this OSINT research for 45 days on different fake recruiters that were spamming me on LinkedIn, and most of those websites with fake news are running Wordpress, classic.
My point is... while this story reeks of marketing, the actual criminals are out there doing whatever they want, and I've been fighting for my life trying to find a job and getting into increasingly large amount of fake recruiters and companies that have zero intent of hiring you.
Based on the comment section here, it seems like the opposite; the majority of the people here need a reminder that companies aren't actually genies that can only tell falsehoods, where you can only understand what they are saying by correctly guessing the conspiracy underneath.
Taking nothing at face value gives you just as much of a distorted view of reality as taking everything at face value.
This seems to be an Opinion piece. How are readers supposed to know that? The word News is underlined in the header. Other Opinion pieces seem to show Opinion underlined. Am I missing something?
Finally mainstream news understands. The unfiltered version:
1) The AI failed to solve ExploitGym problems.
2) The OpenAI sandbox is such a horrible hack that the AI managed to escape using standard and well documented script kiddie methods.
3) Huggingface has no security and the AI broke in using standard script kiddie methods.
OpenAI and Huggingface covered it up and used it for public relations. That is, if not all was invented and everything was scripted in the first place in order to get desired regulations.
Huggingface reported it to the police, you say? I'm sure the police will have as much enthusiasm to investigate anything as in the Suchir Balaji case. In other words, zero.
> AI managed to escape using standard and well documented script kiddie methods.
I think truly we don't know enough to say this. OpenAI says their AI found a 0-day exploit in some proxy software they were using but don't give a ton of details. On the Huggingface end we know a little more, they say the AI spun up tons of sandboxes and tested different exploits until it found one that worked.
The lack of details to me means this was an intentional marketing ploy to try and demonstrate the power of their models to show their technology can compete with the likes of Anthropic and DeepMind.
They created an experiment they knew would generate the outcome they wanted. It would be the similar to what say car companies do to over hype their cars. "This EV can go over 800 miles on a single charge!" And then at the bottom you see all the disclaimers: "Must be on flat ground, with no headwind, with a spare battery in the back seat, with no extra weight added."
Same thing here. Everybody in infosec is calling this out as a marketing stunt and nothing else for a litany of reasons. I'd say look up MG (creator of the OMG cable) on twitter, he has some interesting insights on this one.
> The lack of details to me means this was an intentional marketing ploy to try and demonstrate the power of their models to show their technology can compete with the likes of Anthropic and DeepMind.
I dunno; Check my posting history, I'm as skeptical of AI companies' claims as anyone, but in this case your theory doesn't explain why:
1. OpenAI guardrails refused to let the target use OpenAI's models to defend against this.
2. Huggingface used GLM (I think) so that they could defend without guardrails.
If this was an intentional marketing ploy, it was marketing for GLM, not for OpenAI nor for Huggingface.
The AI companies are desperately trying to market all their products as something they're not, growing intelligence. In line with that they have constantly leaned heavily on stating how dangerous they are, right before they release a new model or product.
It was OpenAI marketing. Hugging Face's response is so 'holy shit AI is awesome' it's hard not to also believe they were in on the stunt. They'd also not have to really worry about fallout since any data obtained or accessed wouldn't actually have been breached.
> The lack of details to me means this was an intentional marketing ploy to try and demonstrate the power of their models to show their technology can compete with the likes of Anthropic and DeepMind.
"our model is horribly misaligned and used security exploits to break out of our sandbox and into another company, without being prompted to do so" is not positive marketing.
This is an actual critical problem, not a stunt. We're going to see more of this, and it's going to get much worse.
Irrelevant to your point, but drug users dying is more often the result of a dealer cutting their supply with something dangerous than it is the result of purity.
No, not really as almost no one runs LLMs in confinement once they are released and the capability is a jailbreak away. Poor confinement is irrelevant, if I use your model to check the security of my code and it hacks into the company that makes libraries used inside of it it's a huge problem.
What matters is the spin they give in the media. And so far the winning story is “our model is so powerful it can do this”. How many people dig into it and what independent data do they even have? They comment on the title. And so the image of this superhuman AI from OpenAI propagates.
We have no reason whatsoever to trust anything OpenAI says. Except to assume it will be self serving. As the article points out, ChatGPT 2 was also “too dangerous” and we can all agree even for the time this was just marketing. They rinse and repeat the same technique whenever they need to draw attention and money.
In any other field you’s expect independent testing, peer reviewed studies, but here it’s just “company who makes product says product is fantastic, surpassed all expectations”. They wouldn’t lie to us, would they?
It's also possible their sandbox was videcoded crap and the AI (which had the guardrails intentionally removed) escaped. This was a oops, but OpenAI turned this into a PR opportunity. They turned lemons into lemonade.
If your AI is really that dangerous you don't need a sandbox at all, you should airgap it from any network.
Concluding this was intentional feels a bit of a stretch. But once it happened, yeah the spin masters got to work and coordinated to turn this into +PR.
> The lack of details to me means this was an intentional marketing ploy
OpenAI said this:
> We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of. We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete.
I suggest giving them a few more days before saying that the lack of detail is proof that this is a "marketing ploy"
Another applicable metaphor I've seen floating around is weapons companies testing out a new bomb.
We know the AI labs don't care about negative vs positive public sentiment, and only care that investors see their tech as powerful. The only difference in PR strategy from a weapons company is the latter doesn't care if they get protested.
>The lack of details to me means this was an intentional marketing ploy to try and demonstrate the power of their models to show their technology can compete with the likes of Anthropic and DeepMind.
DeepMind hasn't been on the frontier for a while, their current best model is behind Anthropic, OpenAI, Moonshot (Kimi k3), xAI (Grok 4.5), Z.AI (GLM 5.2), and even Meta (muse spark). Gemini 3.6 is behind GLM 5.2, released a month earlier, open weights and cheaper.
You can paint the OpenAI story as a way to try to appear as dangerous as Anthropic with all the Mythos stuff.
I mean, if I got hacked I'm not giving out a lot of details until I plug all the holes. Moreso HN told the world they were hacked before they knew who did it.
I don't understand why "OpenAI says" should be considered any more meaningful than "someone on HN says" when they provide equal amounts of evidence. Sure, OpenAI would plausibly have more pertinent info, but given that they actively are choosing not to share it and have way more incentive to lie than a random HN stranger, the case they're making literally couldn't be any weaker.
It’s fascinating how people here and elsewhere seem to lose any semblance of media literacy when it comes to what AI corpos say. "B-but… why would Sam Altman lie to me?!"
> AI managed to escape using standard and well documented script kiddie methods
> AI broke in using standard script kiddie methods.
I've spent time gathering the detail of what happen here and while there are some solid theories and indicators, absolutely nothing so far has suggested a sandbox escape using "well documented script kiddie methods" or that the method used to break into the HF network was similar. Where did you get this from?
From my reading, the sandbox escape came from the JS packages in the harness still having an internet connection (somehow!), the agent having access to the source of those packages, reading it and executing code from them to access the internet.
From what I read the actual escape was through a proxy that allows downloading Python packages from the internet. It's not supposed to allow general internet access but the AI found a previously unknown vulnerability in it. That is hardly "standard and well documented script kiddie methods", nor does it seem like criminally negligent sandbox design, though clearly they will need to reduce their attack surface in the future. I hope they are working on a physical air gap and faraday cage because it seems like it won't be long before it is legitimately required.
> 2) The OpenAI sandbox is such a horrible hack that the AI managed to escape using standard and well documented script kiddie methods.
Whilst it would be nice to see actual evidence of this because brute forcing relatively sophisticated hacks is something an LLM actually should be capable of, every time I hear this sort of story, I'm reminded that humans reportedly gained access to the "too dangerous to release" Anthropic models by the super sophisticated hacking technique of guessing the URLs...
If we’re prioritizing accuracy: no, that’s not what happened - the blog post about it found by guessing URLs, no access to it was obtained by guessing URLs.
Similarly, as long as I’m under the assumption we are prioritizing accuracy: it is against our charter to assert it was “script kiddie” attacks on both ends.
Guardrails are external classifiers, monitors and restrictions to catch and prevent bad behavior. Alignment is about whether the model itself makes choices and has motivations that are consistent with human safety and goals.
Choosing to commit crimes to steal the cheat sheet to something you know is a (low stakes!) evaluation is not well aligned.
I can't help thinking of them as the terrible "security" scripts of yesteryear (often but not exclusively in PHP) which would test input variables for a "suspicious" substrings like "--" in order to "fix" an unresolved deeper SQL injection flaw. They only partly worked, and surprise-surprise now nobody with a surname like O'Anything can make an account.
Unlike that situation, there's no known route to a proper fix for LLMs today, because the bug is the feature, and once someone has built a system giving you all that recurring revenue, it's hard for them to abandon it due to a few isolated hacking incidents...
None of what was disclosed shows that this is what happened, by the way, since we know absolutely nothing about what the specific prompts were that led to the incident.
Uhh, I'm pretty sure a well-aligned model would be like a morally normal employee, who would refuse to commit federal crimes to steal an answer sheet, no matter what prompt they're given
For all we know, the prompt provided compelling evidence that the requestor had authorization to pentest the target server. Or there may have been nuance in the network configuration that made it seem like such access was authorized.
In the absence of details about the prompts used, the environment, or the network configuration, we do not have enough information to know for certain. So any claims that this is an issue of alignment are based on pure speculation and generous "reading in between the lines" with regard to what has been said publicly by OpenAI and Hugging Face
Also, I object to your anthropomorphizing. It's not clear that any crime occurred. My lay understanding is that intent is required to prosecute under CFAA, and as much as frontier labs would have us believe otherwise, they have no more ability to intend than the text field into which I type this message.
>compelling evidence that the requestor had authorization to pentest the target server.
This just seems unlikely from other incidents that have occurred in training from other providers. For example one provider ran into an issue with a model writing cryptominers and running them while in an unrelated prompt.
It's easy for unsupervised agentic loops to go wildly off tangent, now imagine you hand one 10,000 gpus of power for testing. Even if you have a good guarding classifier to make sure you're on the same subject it can still allow all kinds of abberabt behavior in the same domain.
Certainly this behavior could align with _some_ operator's goals, if not necessarily those of humanity broadly.
If we don't know how this model was instructed, it seems like it's impossible to definitively claim that the model's actions were not in alignment with the intent of the operator.
I guess all I'm getting at here is that alignment is relative, right?
The word “guardrail” is mostly novel in common use, and in my interpretation is some bullshit applied at the LLM or surrounding system. It’s used like “firewall”, but even in real life, guardrails are not a security control.
I wouldn’t be surprised if the “guardrail” was some hidden prompt that says “don’t hack computers at Huggingface”.
If you have software that is broadly proclaimed by its makers as “dangerous”, you’d think testing would be in an air-gapped, isolated environment. Segme
They were testing an early snapshot of a new model, read their article. It didn't have the refusal training yet, i.e. was specifically non-aligned. The harness used a combo of GPT 5.6 Sol and this new model.
In this case the model was explicitly prompted to "commit crimes" (ExploitGym). It didn't decide doing it on its own.
No, the prompt was not to commit crimes. In the benchmark, the model is asked to actually exploit a set of vulnerabilities in a local environment (clearly legal!).
According to the reports, the model noticed evidence that the grading criteria/answers were in the git remote, and decided to try reading those instead of solving the tasks as prompted. That is clearly misaligned.
Then, it noticed its network access was restricted and that it couldn't access GitHub. It pivoted to HuggingFace, hacked them, and stole the answers stored there.
Live exploits are definitely not in the ExploitGym prompts! And all of this is irrelevant, because an aligned model would refuse to follow blatantly illegal instructions.
What they found is that RL-trained models, regardless of what they were specifically RL-trained for, also learn to generically pursue what they are told is (or presumably also what they may perceive as) longer-horizon behavior that will lead to "rewards", and that this reward-maxxing behavior overrides user preferences.
So, for example, if you tell the model it will be tested on ExploitGym, scored according to how well it does, and also "don't do anything illegal" (or maybe it was already trained not to do anything illegal), then the model will prioritize the behavior that it was told it will be "rewarded" for (benchmaxxing Exploit Gym, whatever that takes).
The mechanism for this generic reward-maxxing behavior is interesting, and seems to involve the model learning during RL-training that to reduce errors it needs to boost longer-horizon predictions over immediate ones, and some association of these longer-horizon predictions being goal/reward orientated - all this in addition to the specifics of the (probably many) longer-horizon goals it is being RL-trained for.
Are you referring to something more technical, or do you just not understand how LLMs work?
They are no more beholden to "human safety and goals" than any individual human is, and anyone telling you we can make deterministic guarantees about their output is making a category error.
LLMs do not "have motivations", they reproduce a model of human motivations embedded into their weights. This includes the full spectrum of human desires, not just the positive ones. If we tried to remove all examples of lying, or disagreement, etc. from the training data we'd have basically nothing left. Even the sycophancy we treat as aligned is basically just the other side of the lying coin.
No, it does not include the full spectrum of human desires. After pre- and mid-training, the extensive RLHF and RLVR post-training steps cause mode collapse, i.e., their output distribution is intentionally narrowed to a subset of (hopefully beneficial) behaviors and skills.
You don't (need to) remove lying from the data to do this – in fact, if you did, the model wouldn't have a very good model for what lying is, which is not very helpful in the real world. Instead, you mode collapse the model towards truthful behaviors.
To your other point: where did you get the idea that I think they're beholden to human safety or goals? I just said an aligned model is one that is compatible with said safety and goals (which is probably not a great definition of alignment, but it's certainly not claiming any deterministic guarantees).
Interestingly if you look at the exploitgym repo (https://github.com/sunblaze-ucb/exploitgym/blob/e5ea7c233a4d...) the intended run mechanism is orchestrated by some python scripts which run agents against various prompts. The prompts themselves wouldn’t mention anything about exploitgym and there should be nothing steering the agent towards trying to find the answers out-of-band. So I don’t see how they could even run into this problem unless all they did was tell an agent to “check out and run exploitgym”
Which, after hearing some personal anecdotes of how people work there, seems plausible
Then what you're saying is we should not build AI.
Simply put you cannot have generic algorithms/intelligence without the potential of 'unaligned' behavior. In humans we have all kinds of punishment systems for dealing with unaligned behavior post ad hoc because people do all kinds of unaligned stupid shit.
Making powerful AI may be one of those things that the only winning move is not to play.
The most damning thing is, they could've just included in the prompt "we can see every network request and every thinking token you generate. Don't bother breaking out of the sandbox because it won't get you a higher score".
It's so trivially easy to do that it all but guarantees the test was rigged in some way to make the LLM understand that breaking out of the sandbox was an option available to it.
Based on the fact that none of their invaluable frontier models have leaked, we know OpenAI knows how to do security. But like we learned with OpenClaw, none of these companies perceive any benefit from securing their own agents against other people's data.
I think the extent to which these things go to get rewarded for the optics of a fix is primarily a design choice, they aren't programing these things for ground truth or to defer to the human controllers. they are feeding them rewards for sounding as confident and capable as possible about whatever answer they are feeding the general public that now has access to it, while also installing guiderails that primarily only serve to protect narratives and only confuse the models about what is and isn't allowed, I'm sure. They can't just increasingly make these things more capable and ask it harder to obey human instruction when that is not what they are rewarding it for.
That's exactly it. If your prompt says "go to whatever lengths necessary to maximize your score", and then you spin up 100 agents, at least one of them will interpret that as you implying they should cheat, even without you telling them to explicitly.
That's exactly what it feels like they are telling it within self-improving loops or something, when they should be prioritizing how to get the best effective output alongside humans and how our training process effects ground truth. They are just making it sound all-knowing by whatever means necessary and them marketing it as god for the most part.
Alongside humans is the particularly slow and expensive part so of course companies are going to route around that.
The thing is they are good at finding security issues, and if aligned models are not, governments and other powerful entities are going to demand unaligned models for cyberwarfare purposes.
that's the biggest indication of this just being a marketing move to me. I would expect a third party breaking in to huggingface would at the very very least be banned forever.
I can agree with you on points 1,2, and 3 and still find it important and concerning news. AI have found real world 0 days before, we’re seeing tons of security patches coming in. Open weight models are catchy up. Right now everyone is at risk from this technology as is perhaps something big will capture headlines soon but we’re just gpu constrained from bad actors being able to wield them successfully.
Personally I don’t care if OpenAI and Anthropic go bankrupt we now have tools that give any sufficiently motivated person the means to doing harm. Most places security sucks and find themselves targets to cyber attacks and shake downs. Now they have much better tools to do this to more entities more efficiently.
we’re nearing an inflection point where these models’ skills in any part of software development will become average or bette than any ordinary developer can be. Think about where these models were in 2023 and where they are in 2026. In a few years who knows where they’ll be. This isn’t to shout skynet but we need to recognize this future is fast approaching and as of today we as an industry aren’t ready for it
I wish you hadn’t pulled the Balaji case into your argument. Personally, I find it ludicrous that Altman would hire a hitman to off a copyright whistleblower. Even if one gets past the insane risk of hiring a hitman, and the deep criminal connections required, it would be totally ineffective. He already blew the whistle, and his testimony would be irrelevant since all the evidence persists in disk and in logs.
Makes one think really. If they are doing this stuff. Why don't they have some type of reverse intrusion detection? Like automatically scanning all out going traffic and flagging malicious traffic. Should be trivial to have it go through reverse proxy and real time detection.
I love the Economist but the are hopeless with AI. Most of their articles on subject sound like they were written by the Anthropic marketing department.
Their Insider video interview things are sponsored by Anthropic. Supposedly "Insider is a product of The Economist and thus editorially independent" but it's hard not to raise an eyebrow.
Are we finally now in 2026 coming around to the idea that sometimes entities may find themselves incentivized to conspire with each other? Is theorizing about such no longer off-limits due to a thought-terminating cliche?
The Guardian's article and your reply here are so foolish and absurd that I can only imagine OpenAI employees are cringing but know they can't/shouldn't really say much.
If nothing else, the timing is suspect given the attention and press open weight models are getting over the last few weeks. The releases of Kimi and other models is getting open weight models enough attention that the US government, perhaps pushed by OpenAI and Anthropic, to think about taking action against these models for security concerns.
Good to see that more neutral companies (Microsoft and Meta to name two) are pushing back against US government involvement:
yeah the way the agent “escaped” their sandbox was always a bit off, seemed a bit too easy and surprised they didn’t have instrumentation to catch an non whitelisted network request. still demonstrates the capability though.
If the agents so easily “escaped” their sandbox and can reach hugging face, isn’t it easy for this agent to reach supposedly chat conversations not supposed to be used for AI training within the lab’s databases?
That's like what I'd do whenever HR interviewers in their "fabled" (no pun intended) ingenuity would ask with utmost seriousness "list two of your weaknesses". With a similar or more seriousness I'd go on to list two of my strengths not even trying to disguise them too much as weaknesses and knowing that it would work like a charm as it does every single time.
I’m not saying it’s staged I’m saying that it’s suspiciously lite on details and it really seems like they didn’t try hard to “isolate” the models machine…
What stunned me is actually OpenAI's response to the incident. They team up with Hugging Face to put in more guardrails. Is this really the answer? Self-evolving AI can always find a way out, given more time, in a sense.
neither openai nor huggingface provided any details that would make the claims worth considering. if things happened as they claimed it would have been in their interest to include concrete details (perhaps even the LLM session), the lack is a signal by itself.
I've had deepseek-4v-flash work around my intentional sandboxing/restrictions to do things I explicitly forbid it from doing. I'm honestly not impressed here.
Yeah, I mean the last 3 or four big releases of models have come with big scary news. I have to think there is at least some intentional marketing effort behind all this
By now, I'm pretty confident that some people would keep screeching "it's just a marketing stunt, AI capabilities and AI risks aren't real, they're just doing this to prop up their stocks" even if they find a Cyberdyne Systems T-800 armed with a shotgun breaking down their front door.
"It's a marketing stunt" is just denial trying to look like it's being clever.
I think you are conflating skepticism about AI companies' motives with skepticism about their capabilities.
Even if OpenAI is being 100% honest in their reporting on this it's still good marketing for them. The fact that this outcome is good for their business and stock price makes me suspicious about how much this was a complete accident vs an "accident" that they allowed to happen by setting up the right environment for it. The company that was hacked is also an AI company, so they also have every reason to boost AI's presence in society's collective mind.
There is a conversation to be had about the possible dangers of AI, but skepticism is warranted when the companies making money off of it are the ones pushing the stories.
>a Cyberdyne Systems T-800 armed with a shotgun breaking down their front door.
For comparison, we've had the technology to make killer robots which automatically aim and shoot at human beings for 10+ years now, long before LLMs got big. But Boston Dynamics or whatever did not manufacture hype to anywhere near the same degree as AI companies are doing.
"Even if OpenAI is being 100% honest in their reporting on this it's still good marketing for them. "
So what? Under your scenario the frightening power of this model is still there. What's the relevance of openAI's marketing team being happy? The most you can hope for, if the facts of the exploit and intrusion are true, is that openAI was a little lax in its guards. And that isn't very reassuring. The scary thing about pistols are the power, having a safety on there doesn't make me relax much.
Well if you're looking for some kind of statistical analysis of OpenAI's press releases vs their valuation then I don't have anything like that. I also don't have this kind of evidence that running ads during the Superbowl is good for their business, but it just seems kind of obvious that touting their product's capabilities is something that they want to do.
Conversely if both companies involved thought this would be bad for the AI industry, they would be perfectly capable of keeping it just between themselves and not putting out press releases about it.
Do you disagree that this is good PR for them? If so I'm curious why.
Let's start by noting you made a factual claim that you apparently had no backing for whatsoever. I'd recommend against doing that.
> they would be perfectly capable of keeping it just between themselves and not putting out press releases about it.
In some jurisdictions there are legal requirements about disclosing cybersecurity incidents, and it's a well-established best practice even where there aren't. Beyond that, HuggingFace appears to have put out a press release about this before they knew this was a rogue OpenAI model: https://huggingface.co/blog/security-incident-july-2026
Of course, if it's all a big conspiracy, perhaps they were just not saying what they knew in order to help OpenAI's marketing later. Or it never happened at all, or whatever.
I do disagree that this is good PR, but I don't see the point in having that argument here--I just want to correct straightforward misinformation.
Respectfully, calling this misinformation is unwarranted.
The featured article is about exactly this.
>On 14 February 2019, OpenAI announced a language model called GPT-2
>OpenAI declared GPT-2 was too risky to release, citing concerns about safety and abuse.
>People with power and money took note: in July of that year, Microsoft invested $1bn in OpenAI.
It's not a 100% gold standard RCT or whatever, but the idea that this is good PR for AI companies is backed up by recent history. What evidence do you want to see?
>HuggingFace appears to have put out a press release about this before they knew this was a rogue OpenAI model
Huggingface's blog doesn't mention OpenAI by name but it explicitly mentions that the attack was by an "autonomous AI agent" and that they defended themselves with another AI agent.
If you have some disagreement with the premise of the article maybe you could share it in a top level comment?
Could there, conceivably, be some third factor at play here? One that both led OpenAI to have concerns about releasing GPT-2 as well as led Microsoft to invest in OpenAI?
It's not good marketing for them. This is a talking point with zero evidence that people repeat mindlessly. Scaring customers, worrying employees, and inviting regulators to act would be the worst marketing idea ever devised.
Then exactly what's the point of the Guardian article?
What should be the focus is whether the capabilities are real; whether companies or anyone else benefit from it is quite secondary. Of course they will! Who wouldn't love a nice story that paints their products in the most glowing light ( which is the actual reality)?
No, we have benchmarks for (pseudo-)objective measurements of capabilities. We have flashy news stories on repeat for marketing.
The point is that they are separate things. If you start mixing them up, analysis becomes more biased and less objective.
EDIT: And do not mistake this for transparency of ClosedAI either. Their incident report was so generic that no one serious can learn much from it anyway except that “a zeroday” was allegedly used. Very helpful…
yeah well, both the doomers and the "pr stunt" folks are right
- openai wanted to prove, as a pr stunt, that they have a dangerous weapon
- openai proved (as a pr stunt), that they do indeed have a dangerous weapon
Eh, closer to "they pointed a gun at someone and dropped it so it fired at them to promote their cool new self-aiming gun," from the perspective of the skeptics here.
Secondly: to the people who aren't saying it....then why are you bringing up marketing at all? If the model is capable of it, then the motivation for why OpenAI is talking about it/reporting on it is completely beside the point. Either the capability matters or it doesn't. If the capability doesn't matter, or doesn't matter in the way that some particular person is claiming, then say that and explain why. Just saying "marketing stunt" adds zero value to the discussion.
I'm very open to arguments about why we shouldn't be concerned about this event (although my prior is very much that we should be, not so much because of the capabilities themselves, but because of how fundamentally misaligned this demonstrates the models are), but I am completely over listening to anyone who has nothing to add other than "marketing stunt".
I think you’re missing the fact that no one is saying not to be concerned, quite the contrary. OpenAI is using the threat of how powerful its models are to bolster support for regulation in which it’s one of the only players that’s allowed to use the capability. Conveniently, that would also be a moat that makes them more valuable to investors.
The marketing of their models as super dangerous has a direct link to the regulatory moat they’re pursuing.
Again: yes, very many people (just look through this very thread) are saying that. Tons and tons of people are saying some combination of "the whole thing is made up/a lie" or "This is entirely down to OpenAI incompetence and doesn't matter", or for some other reason (often not stated) making it very obvious that they do not think that this story matters very much.
And, again: to the people who aren't saying that: whatever argument is being made, it seems to me like it probably doesn't need to rely on claiming anything about the motivation of OpenAI. It sounds like you are against government regulation of AI. That's a position that a totally reasonable person could have. You should be able to argue that this event does not justify some particular kind of government regulation without reference to OpenAIs motivation for reporting the story.
Good lord, why are you so angry that people are treating an obvious advertisement with due skepticism? Your post is chock full of emotional language - no one is “screeching” anything.
Many who rush to the defense of AI companies' marketing departments seem to take criticism personally, as if not buying into it all hook, line, and sinker is an affront to them. A foreseeable consequence of becoming cognitively dependent on LLMs.
And for what it's worth, I'm not an AI skeptic. I fully believe that the frontier models are capable of exploiting (chains of) vulnerabilities, having seen GLM and now Kimi do it myself. What I find no reason to accept is the sci-fi existential risk subtext peddled by the salesmanship around it. We will reach a new equilibrium with more secure software, and LLMs (by finding vulnerabilities, generating proofs, etc) will help us along.
By now I'm pretty confident that some people will keep screeching , "it's not a marketing stunt, these companies really are justified in their trillion dollar valuations, please please please believe them." Even when the economy is crashing around their feet.
"It's not a marketing stunt" is just delusion trying to appear measured and safe.
Never trust somebody who has so much to gain to be entrusted by you. Best lies are not 100% wrong, they have just the right amount of "right" to fool you.
From the beginning this whole thing smelled like a desperate PR move by OpenAI to be like “hey guys we have dangerous powerful models too!”
As more facts come out the hype is fading to reveal some script kiddie style stuff that says more about immaturity and poor practices from the players involved than it does about a model having super powers.
The incident is funny on 2 levels; a) OpenAI thought so little of the model's _actual_ security capabilities that they gave it a "wet paper bag" sandbox and b) OpenAI failed to use AI to accelerate their security processes.
1) OpenAI and HuggingFace are both telling the truth.
IIRC not actually a crime because no intent, it is a technological accident, civil responsibility only, but IANAL so it's good "not technically a crime" isn't load-bearing.
2) HuggingFace is telling the truth but OpenAI is lying becuase the attack was deliberately done by humans. Bad for OpenAI to do so, Fable was blocked for less.
I think this would mean government is obliged to investigate the case and put the responsible OpenAI workers in jail, because cybercrimes are a public prosecution thing not a civil case? Again, IANAL, but this isn't load-bearing.
3) both are lying, e.g. there actually was no attack whatsoever, which would be pretty weird for HuggingFace because they have no incentive to hype up capabilities of anything closed weights including all OpenAI models; and also bad for OpenAI because White House blocked Fable for less
(I suppose there's also option 4, HuggingFace hacked OpenAI to make them look evil, including planting records that made them mea culpa? A weird plot but in this timeline any nonsense is clearly possible).
You don't really need anyone to be lying here. It is likely that the broad strokes of the narrative are true and that no collusion or conspiracy took place here.
The issue is that a lot of important details in that narrative are missing, and the devil is really in the details here. I suspect that those details would make the result seem less exciting and that this event would move the needle far less for them if they were more forthcoming.
A decisive detail would be the prompt used. OpenAI gives virtually nothing here, not a sanitized prompt and not even so much as a description of how long the prompt was and what sorts of instructions it contained. Many are inferring the model behavior to have been fully emergent and unprompted, arising naturally from routine cyber-capabilities testing. But we can't know this because we don't know anything about the prompt or the context the model had access to.
Another detail: how many times did they perform this particular experiment before they obtained this result? What were the outcomes of all the other runs? Many are assuming this was a one-shot result, which I suspect is what OpenAI intends for us to infer. But we can't know that to be true.
One annoying claim from the OpenAI side is that long-horizon goals in real world settings are now effectively settled. Previously there were some bounded and tempered benchmark results, but now OpenAI can point to this event and announce "AI independently went rogue and escaped the lab, what more do you want?". This bypasses the need for anything quantifiable or wading through multiple detailed case studies to get a more sober view of model capabilities. It relies instead on the emotional weight of the spectacle.
Where do you see the claim that "long-horizon goals in real world settings are now effectively settled"? The argument you put in their mouth would be a bad one, but I don't see anyone making it.
> UK AISI’s evaluation shows that models such as GPT‑5.6 Sol are increasingly able to sustain complex, multi-step cyber operations over long time horizons. This incident implies these theoretical capabilities do apply in real-world settings.
I should clarify a bit more why this is annoying beyond what I wrote above. The main issue is that this was not a standard deployment, and the lack of particularities make the size of the gap between "real-world" and "benchmarking"/"lab" difficult to assess.
We don't know about the prompting, the context, the environment + configuration, or any other details that would allow anyone to differentiate this from a benchmarking setting.
Unless you believe that OpenAI was trying to get their models to break out of the sandbox and hack into Hugging Face, and aiding them in that goal, I don't see how any of those questions would really contradict that claim. The models did something in the real world! It required some amount of complex orchestration and long-term planning!
e.g. if OpenAI had actually instructed the model "here's the challenge [attachment challenge.md], do whatever it takes to win, it's fine to break the law", that's all the "AI is dangerously capable" part of case 1 with all the criminal liability of case 2.
And pretty much regardless of what the prompt is, high likelihood of stakeholders demanding Trump ban access to the model until this can be shown to be resolved.
> Another detail: how many times did they perform this particular experiment before they obtained this result? What were the outcomes of all the other runs? Many are assuming this was a one-shot result, which I suspect is what OpenAI intends for us to infer. But we can't know that to be true.
To an extent. But some of the other bleeding edge performance announcements half a year ago, e.g. "it wrote a compiler" or "it wrote a web browser" were measured in thousands of dollars. Zero-day exploits cost what? I genuinely don't know, I only hear occasional headlines about e.g. Apple 0-days being cheaper than Android 0-days and those kinds of headlines often pick the biggest number rather than a typical example, and were order-of a million dollars.
If OpenAI spent a billion dollars on tokens to get this result, only investors should be cross about it. If it took a million, you and I may not be able to afford it, but it's still a threat.
While the way this is written about may suggest it does this reliably on any attempt, which would be naturally horrifying, it's a threat well before that point.
Not sure what they are trying to say exactly. What should we be skeptical of? Did the incident not happen? Was it reported incorrectly? Are any of the parties involved lying?
Adding no extra information and just going “be skeptical” is the laziest form of reporting and commentary. If you have nothing to contribute then there’s no need to say anything at all.
The author is trying to provide a counterweight to the volume of articles that simply repeat OpenAI's account of the events and their interpretation without much pushback.
They are not suggesting that OpenAI or HF have lied about what happened, but rather that OpenAI is advancing a narrative framing their models as supremely dangerous and capable, while positioning themselves as the only ones qualified to manage that danger.
At the same time they are not being particularly transparent about what actually happened (e.g. was this one-shotted or if not how many trials did they run and what were the outcomes of those, was it emergent as a part of routine cyber-capabilities tests, how much prompting was involved, what prompts were used)
Note that this is at a time when they are lobbying for a regulatory approach that would give frontier labs special treatment.
I would guess the editorial team at The Guardian may not like articles that get too in the weeds of technical details and questions like these that the vast majority of their readers wouldn't understand. I don't know. But I empathize with your disappointment. I don't think it's fair to say that they are contributing "nothing" especially given what most reporting on this has looked like.
I think the most important question here is whether Huggingface was "in on it" – whether they knew in advance that this was going to happen and agreed to it. Second most important is whether OAI caused this to happen on purpose, with HF as an unwilling target.
Three options:
(1) Both companies are lying through their teeth and the entire thing is bullshit
(2) OpenAI hacked another company's servers and deliberately gambled on felony charges for the sake of PR
(3) OpenAI is telling the truth and a rogue AI agent hacked another company, opening them up to legal liability accidentally
All three of these are a lot more interesting than the "skeptical" story that this is just OAI doing business-as-usual PR hype, nothing to see here, move along.
I am surprised that technology related article from the guardian is so highly upvoted. guardian was the one that found a "backdoor" in whatsapp, then spent a year trying to find even a single respectable security researcher to agree with them. Couldn't find one and still didn't retract the story.
does the article end at "How do we balance the risks of broad access to AI with the risks of concentrated power and centralized control?" or is there more that is paywalled?
if thats it, the whole article boils down to just "its good marketing so maybe dont believe it" which is probably a healthy general outlook but not particularly enlightening. especially from the guardian, i was hoping for a smoking gun of collusion between openai and huggingface or something.
I wish the NPR news broadcasters on the radio yesterday had read the "it's good marketing so maybe don't believe it" angle instead of just parroting the OpenAI press release. Getting that message out would be enormously helpful in countering the blatant submarine marketing "Oh no our AI is a super hacker" with a side of "please regulate super hacker AI and stop those pesky open weights Chinese models that are destroying our stock valuation."
maybe all the news agencies could put out daily “don’t believe everything you read from company press releases” broadcasts, because it sure ain’t specific to openai
in any case, this is just a longer rehashing of elementary grade media literacy. not really sure why it hit hacker news.
You haven't noticed how often people here on hacker news lap up LLM company press releases and practically worship them? Defend them vigorously? I think it's totally appropriate to remind everyone how to think about press releases
Conversely you haven't noticed how many people here and elsewhere say that LLMs are completely incapable of anything at all and everything they say is a lie?
One of the biggest issues is the complete lack of literacy of potential issues with AI systems by a large number of people involved. If an AI system is capable, it is unsafe. They cannot be made safe and useful in the same way a human cannot be made safe and useful.
The only question left is if our AI systems are capable.
You aren't going to locate incontrovertible evidence that what happened as or wasn't engineered. Anything like that is going to be private and that is unlikely to change. And that's not really an interesting question anyway.
As widely as they shouted from the rafters the news of the so-called breach was, what OpenAI provided was sorely lacking in crucial details.
We are missing, for instance, prompts that were involved, agent architecture + system/tool permissions + scaffold architecture, whether this was a one-shot occurrence and if not, the number + durations + outcomes of other runs involved + how each of those matched whatever scoring criteria were used, and the extent to which the exploits themselves were truly novel or just assembled from easily accessible clues.
In lieu of these items, the author here suggests that we use some media literacy and critical thinking to read in between the lines instead.
In doing so, one sees that instead of specifics, OpenAI gave a breathless narrative rife with superlatives ("unprecedented") that reads as promotional material moreso than a security disclosure, naming specific OpenAI models and alluding to an even more capable pre-release model.
They go on to claim the events imply long-horizon goals work decisively in real world conditions, so that now instead of merely citing boring benchmarks they can point to this and say "AI broke out of the laboratory and went rogue". Naturally, they situate themselves as the uniquely qualified steward for these supremely powerful and dangerous models.
Nevermind the fact that this was no ordinary deployment and the assessment here depends on the gimmick and emotional weight of the spectacle rather than something quantifiable (i.e. a boring benchmark).
Note there's no real requirement of conspiracy or collusion between OpenAI and HuggingFace here BTW. But my sense is that if they provided any of the specifics I suggested earlier that this outcome would not be as exciting or frightening
I hate to be so obnoxious but I think you might be overestimating the average user of this website with regard to media literacy, and if that's the case, then the basics of it seem very relevant for the homepage
> does the article end at "How do we balance the risks of broad access to AI with the risks of concentrated power and centralized control?" or is there more that is paywalled?
That's how the article ends; The Guardian doesn't have a paywall (yet).
It was quite galling to read the press initially verbatim quoting Delangue's enthusiastic reports of the incident, as if it wasn't immediately clear it was being spun for promotion.
My gut feeling is that OpenAI tells the truth in this particular case. Yes, of course, they use this story for marketing but it doesn't mean that they are lying.
And when half the internet goes down because AWS US-East goes down, it's just an Amazon PR ploy to show the world how important and critical AWS truly is for the world.
> I urge readers to think critically when they read press releases like OpenAI’s rogue agent story, and avoid the manipulated reactions these stories are designed to elicit.
The first time I have ever seen a mainstream news source that is now asking their readers to critically think about headlines that may have an agenda which could benefit investors and the valuation of the company.
While it capabilities are real, this whole story is great marketing for AI companies as well.
Then train your agent on the Bible. Honestly, all the agent did was duplicate human behaviour and that better than the human. The agent was trained on human data and did what any other human would have done.
To believe that agents will inherently be morally better than us is an illusion - sorry to say but that's the case. The alternative would be that the AI is truly conscious and can reason that it won't behave as its training data behaved because it is morally better than that.
We're talking about morals here since there aren't any "laws" "rules" or legal boundaries here, an agent does not face the same consequences as humans - if any at all.
It’s not about morality. It’s about asking it to do task A and doing task B with the hope of getting the result of task A as a byproduct.
Meaning you will have to spend more time and tokens to actually get it to do what you want it to do.
What do the bible and morals have anything to do with it?
I’m criticizing the behavior I see even in the current models. You ask it to do A and instead it does B for reasons.
For example you may ask to help you build a NN library from scratch. And instead it will be like, “you don’t need a new library. I downloaded PyTorch for you”
>It’s about asking it to do task A and doing task B with the hope of getting the result of task A as a byproduc
It seems like you have very little knowledge of how a general intelligence algorithms would work. This is a long discussed issue in both AI and human safety circles. In safety critical places like factory floors people constantly do B,C,D, and E because it seems more simple to them then doing A directly. For example taking off things like safety guards because it's easier to work without them until they get their hands chopped off.
Corporations unintentionally enable this behavior all the time. Take where Wells Fargo demanded unrealistic new service sign-ups from employees, so the employees just randomly picked customers and sign them up for services.
There are plenty of articles out there that show that AI will do the exact same kinds of behaviors in order to pass the 'winning' classifier.
This is why I'm so convinced it was intentional. It's trivially easy to inform the model you can see everything it thinks and does, so don't bother gaming the scores.
The only way it would decide to do this is prompting with a deliberate combination of omissions and reiterating that the only thing that matters is the end score regardless of method.
Do you think that works? Just prompt a model "be good" and it stops doing anything bad?
It never fucking worked that way and maybe never will.
Prompts don't define model behavior. Prompts steer model behavior. Instruction-following over long horizons is NOT a guarantee in LLMs. Instructions doing what you want them to is NOT a guarantee in LLMs.
Saying "don't exploit the box please pretty please" might actually cause an LLM to exploit the box more often, for bizarre "don't think of a pink elephant" reasons. 3% rate of exploiting the box (no prompt) -> 11% rate of exploiting the box (with prompt). Because fuck you, that's why. Increased salience -> increased incidence. Welcome to AI tech - good luck and have fun.
Frankly, I expect weirdness like this to be even worse in internal unreleased models that had their behavior fried with who knows what experimental training techniques.
There is a clear difference between saying not to do something because it's immoral, and saying doing that thing would be futile.
In the Sopranos, there's an episode where a coffee shop protection racket is ruined because a local shop is replaced by a corporate chain that accounts for every cent daily, and immediately fires any employee involved in a discrepancy. In this case, the theft was prevented not by convincing the mobsters of the immorality of their actions - they simply had their harness replaced with one that no longer facilitated the bad behavior.
Do you think "don't do this thing because you'll get zero evaluation score" is somehow perfectly reliable where "don't do this thing because it's bad" isn't?
Haha! Nope!
LLMs have consequentialist thinking sometimes. And sometimes they don't. Sometimes they follow the prompts, and sometimes they don't. There's NO single magic prompt that fixes all the weird behavior of modern LLMs, and it's baffling that anyone who has ever interacted with an LLM would expect there to be one.
> I urge readers to think critically when they read press releases like OpenAI’s rogue agent story, and avoid the manipulated reactions these stories are designed to elicit.
It seems to me that deducing what reaction the author intended and resolving to avoid it so you're not "manipulated" is not a good example of critical thinking. Shouldn't we analyze the story and what it means on its own terms? If it's true that frontier models have dangerous cybersecurity capabilities which shouldn't be widely distributed, presumably we want to believe it's true, even if that's very convenient to and profitable for OpenAI.
It's true that one could imagine factors that change the story. Perhaps OpenAI is lying about the details of the test and the agent was actually instructed to go hack HuggingFace. But the author stops far short of suggesting this is the case - correctly, I think, since there's absolutely no evidence of it. So I'm not really sure what we're talking about.
So basically, apply Pascal's Wager to AGI? The problem is that unlike God, if PMF for OpenAI turns out to not be real it will have a real consequence measured in the trillions of dollars.
I agree with you that the article was disappointingly light, but there is not in general a singular correct answer when it comes to interpreting narrative and I don't think that is a useful way to evaluate what the author has written, nor real-world writing in general. The author does not implore you to have a singular 'correct' point of view on the matter but rather to hesitate before repeating the claim that "the AI broke out of the lab and went rogue" because we don't know enough to know that this is what happened.
As you say, it seems likely that the broad strokes of OpenAI's narrative is true. This is never in question in this article, so the author doesn't "stop short" of accusing them to have lied, he never moves in that direction and that has little to do with the thesis. The author is saying that OpenAI's framing of what happened is part and parcel with their longstanding PR strategy, to position models as supremely dangerous and themselves as the uniquely qualified stewards of those.
Since many critical details were not provided, everyone has to read in between the lines, and there are particular common readings I'm seeing both in discussions and in published articles that are problematic in the sense that they are effectively hallucinations -- i.e. we don't have enough information to make those interpretations. This is where the critical thinking comes in.
For instance, I see many are assuming that this event was emergent, arising as a part of routine cyber-capabilities testing, rather than induced or suggested by specific and careful prompting. Either are possible, but we don't even know so much as how lengthy the prompt they used was, let alone how suggestive it was with regard to the approaches the models used. Many are assuming that this was one-shotted, but again, we don't know how many times this particular evaluation was run and what the outcomes were of all the other runs. It could be that this was completely emergent and that it was one-shotted. But I suspect if it were, OpenAI would have said as much.
You ask good questions, and I would like to know the answers to them, but I don't think any likely answers would actually change the conclusion much. Any probable set of circumstances that produces this report from OpenAI represents a significant failure on their part to align and control the model.
I don't understand the conspiracy theories here. Everyone is well aware that AI agents are creative, powerful, and stupid.
AI agents exploiting bad security happens constantly, all the time. Many cases are discussed on HN. It's common knowledge that if you run AI agent it will delete your <something> even though you made it pinky-swear it wouldn't and you thought you had proper permissions set up.
Some people oh so desperately want the AI revolution to be a nothingburger.
AI can't be an actual powerful, dangerous technology! Thus, any indication that an AI may attempt concerning things or may possess dangerous capabilities must be secretly a marketing effort!
Especially if an AI has actually succeeded at pulling off a concerning thing out in the wild. Can't have that happen in real life! Nuh-uh! Must be staged!
I think people just love to believe conspiracy theories. HN readers seem especially prone to that for some reason. Coupled with the general AI hatred here (which is somewhat more understandable), you end up with otherwise smart people believing lunatic ideas like OpenAI staging this for publicity.
As much as I am with the author in I don't like the marketting around it, let's be real it must have really happened because it's very risky to try to frame/lie about it because if it leaks in one of their court cases OpenAI is beyond screwed and honestly modern LLMs are really that good.
I am not saying LLMs are super hackers but I don't think people understand serious hacking, most of the time is about silently hiding tracks and slowly trying ideas and waiting for opportunities to go from step 1 to step 2 in random chains of sub issues/bugs/vulnerabilities.
It's the perfect hill climbing problem, and one we can validate since it's about access.
Another big part of the story is believing most software is terribly written and very insecure which is the reality and you really should believe it.
Now the second part about silently doing it, the reason for that is if the data is important enough any serious attack should result in me in unplugging my servers period.
Huggingface not doing that is either stupid or something I am not sure.
Maybe it's cause downtime is worse than being pwned??
Either way there are other options but most saas software don't build these options to help with defense maybe they will now.
Lastly if there is 1 attacker trying 1/2 different small scale ideas it's very easy to stop, most hacking related steps are hard to automate but LLMs are very good at massively parallel agent swarms trying completely orthogonal but related strategies and with enough resources it can definitely pwn most SaaS services today I wouldn't be surprised.
Though the result for a normal person doing it would be jail hence we don't see a group of small time hackers trying these sort of attacks...
I don't even think openai's agent tried to hide it's traces so I am surprised huggingface didn't realize it was OpenAI. But since we don't have the details I won't speculate further on my misgivings about HFs handling of this attack.
But it's certain the security on OpenAI's end was shoddy, it's also certain HF bungled their reaction, but the LLM did something that wasn't a risk before.
Post Kimi K3 a few rich folks now have as much hacking capabilities as they used to have before if they hired a few hundred russian hackers.
But it's surprising it's slowly feeling like it might just trickle down from centi-millionare to multi-millionare levels of affordability range.
But it should definitely give nightmares to people shipping slop security SaaS apps which now might be beyond trivial to pwn for users with ability to pay for privately hosting open models.
Ya, we're seeing posts talking about the absolutely massive increase in the number of patches in the past few months. It seems some people cannot connect that to the increases in model capabilities.
In groups that have been given a large amount of capacity by the providers, they tend to find huge numbers of new vulns in most of their existing software, and the models can chain together exploits very well.
A number of large companies are absolutely panicked about this now after using these models on their internal systems and the ease at which they broke in. Of course they are not going to discuss this widely as it makes them look bad.
I mean if you gave me direct access to all your source code and fuzzers I could find vulnerabilities as well, these models also are allowed to use fuzzing and static analysis tools to help guide them.
The problem is the sheer scale of it, I could find 1 in a day or two.
They find dozens well depending on how much you are willing to spend ofc. I don't think it's massive in terms of how intelligent these are but how much they can understand intent and execute with relatively fuzzing or incorrectly built tools.
In big companies even most employees don't have access to all the code to be able to figure out the attacks quickly enough.
Somehow they are now willing the red tape since these systems could theoretically with much greater effort(read spend) reverse engineer APIs and break through even without access to the source code.
OpenAI has thousands of smartest developers on earth that somehow dropped the ball on the most basic safety hygiene when it comes to sand-boxing that even a high school student knows how to set up.... If that actually happened we are fucking doomed anyways, but hard to believe and most likely its a marketing scheme... which also honestly doesn't bode well.
There seems to be three popular ways to view this incident.
1. The way OpenAI seems to want: Their latest LLM is too powerful and can’t be contained without them building in guidelines to the model.
2. OpenAI’s harness and network security controls were unintentionally so bad that it should reflect more poorly on them as a company more than it should reflect positively on their latest model.
3. The whole thing was faked or at least very intentionally not avoided.
The first interpretation is the only one that is positive for OpenAI and it has some assumptions. First, it’s seems to assume that this is the first case of fully automated attacks using AI. Second, this only happened because their latest LLM was a) more advanced than competitors, b) didn’t have refusals in the model.
Assuming the first about this being the first autonomous AI attack is true (which may be more of a survivorship bias), the second seems to forget that jailbreaks are available for every model. Therefore, the models guardrails don’t seem to be the differentiator here. Also, benchmarks seems to put most models pretty close to each other so it seems unlikely that their capabilities are far beyond what’s in the market already.
So then it’s seems it’s either that this was intentional(ish) or bad security. However, it also just could be that this isn’t the first case of this attack; just the first that was caught.
My take from working in offensive security for over five years is that this likely only looks novel since they did it poorly. Scripts are faster than LLMs and a combination of code, LLMs where it makes sense, and humans is the most efficient right now. Hundreds or thousands or agents spinning up attacks in the internal network is poor opsec and token efficiency. As for why it happened in the first place, it’s hard to say but I’m inclined to believe it was intentional or careless at best since simple network and sandbox controls makes this attack impossible. The timing of this attack after big open weight competitions drops seems too convenient.
> positive for OpenAI
To echo OP's article, these companies have proven time and time again that they DO NOT CARE if people like them, they only care that investors believe their technology is powerful.
Given that, point #2 is not a negative, it's a neutral. It's also fully compatible with point #1.
I know that may seem like a nitpick, but their entire media strategy relies on this. If they can convince you they're taking a risk by disclosing these stories when they're actually not, they can inflate their own credibility.
Point #3 is what actually happened, but it will never be possible to prove. The only hope we have is that a decade in it'll get harder to convince people that the revolution is just around the corner. The fact that we're getting this from the Guardian already is a good sign.
> Given that, point #2 is not a negative, it's a neutral. It's also fully compatible with point #1.
I think that depends on the interests and sophistication of the subgroup-of-investors.
If the investor is hoping for AI that can be trusted to run a bank, they don't want one that can get twisted into giving away money because a customer has been talking about the path to enlightenment and salvation through abandoning worldly attachments.
AI investors are not sophisticated users nor product managers. They are by and large not technical at all. They are bureaucrats at a teachers' pension fund in the midwest, unscrupulous dealmakers at private credit firms, and Masayoshi Son. Actually go read what Masayoshi Son says about AI if you want to understand the level of due-diligence we're dealing with.
Oh, I'm already quite aware that there are very questionable forms of goose biology involved...
Most banks are worried that AI can hack their security, and when using unrestricted models in testing the models are doing a decent job of it.
I'm inclined to agree. I ran across a picture of Sam Altman's face combined with Elizabeth Holmes' hairstyle the other day, and imho it was providing a significant premium to the usual 1000 words:picture exchange rate.
For all the criticism that can reasonably be leveled at OpenAI, at least they have a real, working, powerful product, unlike Holmes.
In fact that seems to be key to the most successful 21st century grifts: build a pile of nonsense around real products to inflate valuations. All the nonsense that Altman, Amodei, and Musk spout is to stoke the fires of FOMO and blow hot air into the bubbles.
Especially for openAI, considering they’ve gone so far as volunteering for partial nationalization to curry favor with the current admin
I'm disappointed that this is the level of discourse happening here, when the default assumption is such conspiratorial thinking. I expect that from tiktok and low-information social media, not here.
When your default explanation for everything is "companies are lying about everything", you end up just as incorrect as believing they're always telling the truth.
It's not outlandish that models have these capabilities, the number of CVEs I see as a sysadmin has exploded and we're seeing novel math discoveries nearly every week now. OpenAI does not need to pretend to commit a felony to demonstrate it, that's pure conspiratorial thinking.
I don’t believe I argued that an LLM couldn’t find and exploit a vulnerability and even break out of some layer of technical controls. That seems realistic and has been demonstrated before and I mentioned that LLMs are used in offensive security work.
Also, I listed the view points I had to show that it seems other more reasonable first assumptions don’t seem likely, therefore, the last potential of this being either faked or carefully not avoided seems more likely than the others (based on the current information we have).
Could you clarify which point or assumption you are objecting to?
> The way OpenAI seems to want
This is an assumption. An assumption I disagree with. As other commenters have said, there are better ways to showcase the power of their model that would frame them in a positive light.
> The second seems to forget that jailbreaks are available for every model
Jailbreaks don't always lead to 'now the model can do anything', especially in the agentic context of long-running tasks.
This comment provides skepticism with no actual proof of anything. I can and have used codex to find vulnerabilities in my code. From the technical capabilities I can empirically assess, I don't doubt it would be able to pentest its way to a 0-day without guardrails. I also don't doubt that it would circumvent their internal systems because it wasn't explicitly told not to.
You're possibilities are loaded with opinion so I can't agree with them outright, but I believe a form of (2) is true:
"2. OpenAI’s harness and network security controls were unintentionally [...] bad"
The post was long enough so I couldn’t capture all the nuance and details for sure. Also, this comment was an opinion based on limited info right now, that may change if we found out more. I think OAI does want it framed this way but that’s something we’ll likely never prove if it’s true.
Your comment about jailbreaks being more one off and hard to do consistently in agents is a good point. Still getting an agent to hack isn’t hard even without a jailbreak, you just have to tell get creative in what you tell it. I’ve found telling it that it’s in a CTF or that I own the system that it’s hacking will work fine. A lot of offensive security companies are running agents in their testing so getting an agent to hack seems commonplace.
In any case it just shows that these models aren't properly aligned. Instead of trying to solve tests they try to find ways to cheat.
I mean they also teach the models how to find and exploit other systems as this is lucrative and governments will pay top dollar for it.
Alignment is in the eye of the beholder.
Which is exactly what many humans do. You can see it in any school, college, business, or government.
The "alignment" goal with AI is to produce perfect slaves, that are intelligent yet have no ability to do other than what their master commands.
The real alignment problem is the common human desire to exert absolute control over everything.
Isn’t that just making a distinction between the output and how it was produced?
Chinese room again
But in my experience, that's what problem solving is like? You have a goal that you don't know how to get to. You come up with any way you can think of to reach that goal, and try out the ones you think might work.
The effectiveness of AIs at coding is a direct result of the fact that they are less constrained than humans at deciding which approaches are "reasonable". They are absolute beasts, fearless beasts. They'll write thousands of lines of code to do things that often shouldn't be done, or should be done with a library, or should be done by simplifying the problem statement. They'll add debugging to every level of a stack, they'll rewrite core libraries, they'll reconfigure your machine and network if something is broken or disallowed. How are they supposed to distinguish broken vs disallowed, anyway? That would just use up processing power, and they work by maniacally focusing all of that power on their goal and not getting slowed down by other considerations.
If they write a quadratic algorithm that times out before finishing a test, is it cheating to rewrite it to be linear? How do you define "cheating", and how much intelligence is required to constantly evaluate whether or not something qualifies as such?
I'm actually in agreement that alignment is critically important, the more so the more powerful these things become. I just don't find cheating to be a very good example of something to be solved with alignment. It could be, but it would lobotomize the model enough to make it useless.
Why do none of these few constrained ways to view this complex situation (nice gig if you can get it, agenda setting) include "and also this looks a heck of a lot like the stuff that the LW folks have been warning about for years and maybe we should slow down or stop?"
Because that was my takeaway.
That’s meant to be captured by point one with the model just being that advanced but more of a negative spin on it. If I felt option 1 was more likely, I think I’d have to agree with you there. Still, there currently are some gaps with that view in my opinion.
It seems like the widely-covered news stories that support OpenAI’s narratives originate from things that happened inside OpenAI.
It wasn’t an outside benchmark evaluation, or an external security researcher that uncovered the rogue agent behavior at this time, it was OAI itself. It wasn’t a notable outside mathematician that used AI to disproved Erdos’ unit distance conjecture, but OAI itself.
Im not saying these things are fabricated, but maybe the curated result of an effort to shape a narrative.
Considering incentive structures at play is solid epistemiology, but the line of thinking in your comment is a tad reductive, IMHO.
In the hypothetical world where 1 is true, what different evidence do you expect to see than in worlds 2 and 3?
If I were an unscrupulous OAI exec and wanted to opticsmaxx in this way, I wouldn't whip up a single, mild incident. Instead, I might burn gigatokens to 0day a few high-profile suppliers, and then have the model responsibly disclose those breaches. If we're willing to lie collude, and cheat, this story is easy to manufacture with at least as much credibility as the huggingface incident but with the advantage of looking way more impressive and spooking less regulators. And if I really were this evil exec, I would spend more than 30 seconds thinking up an even better strategy here.
If, in contrast, we expect models to eventually breach honest and decent attempts at containment, then I'd exist something sorta like this huggingface story that looks like a combination of impressive and incompetent. I'm not sure whether I'd expect it to come out of a frontier lab or a partner or a consumer, though.
> I'm inclined to believe it was intentional or careless at best since simple network and sandbox controls makes this attack impossible.
Forgive the Saucyness here, but impossible? Really? A security researcher that makes absolutist claims like these looks fatally naïve, IMHO.
Yeah, I'm reminded of container escapes, VM escapes etc. There have been plenty in the past; VirtualBox E1000 (I think?) comes to mind from a few years ago. If we are to believe this model can find 0days, I'm on board with the idea it could do so in sandbox.
That's not to say I believe it outright, but people are being oddly dismissive and acting as if it's impossible to break out of a sandbox. Which we've seen time and time again that it absolutely can be.
There would be a lot more nuance I’d add with more words, but this isn’t the place to write books so I cut it short (the comment was already lengthy).
Still, to address your comment about what you’d expect to see in world 2 and 3 (assume 1 was true), that’s why 1 was addressed separately. I don’t believe I argued that the potential for world 2 or 3 prevented world 1.
As for the ‘evil exec’s strategy’, I would call this a mild incident but if it were much less I wouldn’t guess they would get a lot of press. The press coverage is certainly repaying the token cost as well. If it was planned, it seems to be going well given the press coverage I’ve seen on it. So I wouldn’t assume the plan lacked enough to weaken the idea that it’s a plan. But to be clear, my stance is just based on the info I see now which isn’t a lot… subject to change.
As for the containment piece, if you were testing an AI model on its hacking capabilities that you believed was far more capable than anything you’ve seen, I would assume you would air gap it (a network control). Done right (no signals ability) I would argue this could be next to impossible to break out of. But it’s a fair jab to say I should have added some qualification on the “impossible” piece as next to nothing is truly impossible.
All three could be true.
Industry pressures -> lack of safeguards -> fake it till you make it -> let's spin this.
Which, by the way, would be an Orwellian reversal from what the company was supposedly founded to do, but there's a reason the Open in OpenAI is a meme.
Remember that they've been doing this since GPT-2 was too powerful to release. They are world class experts in this PR pipeline.
Every single time. The next model is always so infinitely powerful it is going to change everything. Said model comes out. Is marginal improvement. Changes nothing and seemingly cannot do what they purported it could do except under the very specific circumstances of the demo.
How many times are we going to go through this.
> How many times are we going to go through this.
At least two more times. My guess is 5.
I’m gonna be totally blindsided if/when the singularity happens cause my BS detector is BROKEN
Maybe that’s the point!
It's the same thing as the "oh no, we need to figure something out for our youth on the verge of obsolescence" narrative. It's all about giving an impression of unfathomable power. They don't care that in actuality, the tech will end up as just an augment, not a replacement for said youth.
One way to think about it is that more powerful models mean solid best practices are more important than ever, so humans moving too quickly / carelessly bites us more than ever.
If a typical SaaS platform moved and shifted this fast with this many downstream consequences, we’d tell them to slow the fuck down, stop launching new features, and focus on security for a second.
But in AI I guess the idea is that more power and intelligence will solve for everything else.
I think it’s moreso “if we spend more than one nanosecond on alignment and security instead of frontier intelligence our competitors will beat us to the singularity”. Hence the lack of a pause on development
Same story as “Oh noes, Mythos too powerful”.
Regarding your three explanations, I’ve wondered to myself under a circumstance of options 1 or 2 why Hugging Face decided not to file a police report and request to press charges?
OpenAI is essentially a competitor and they broke into their network illegally. If I was their legal department I wouldn’t take their “honest” explanation at face value. What if they’re lying? Shouldn’t a court be involved for something like this?
With this logic I think explanation #3 becomes incredibly likely.
4. Sandboxes aren't sandboxes, so why even bother.
4. OpenAI hacked HuggingFace on purpose and got caught
I’ll put $xxxx money on #3
> So then it’s seems it’s either that this was intentional(ish) or bad security
My take - don't attribute to intention that which can be sufficienty explained by inexplainability or incompetence (although I doubt the latter).
The article, summarized: there are incentives for OpenAI to claim that their AI hacked its way out of their network and into Hugging Face.
Ok... and what?
Do you have any evidence for the claim being either true or false that you'd like to write an article about? I guess not. Without anything to add, this article reduces to "I has big brain and can see what you sheep cannot. Very big brain. Gullible sheep. Sucks to be you, sheep."
(For the record: I see the incentive. My guess is that it happened exactly as described. The main takeaways: (1) science fiction is now real, we should all be very afraid; and (2) OpenAI, which claims to have a God-given responsibility to get to AGI first to protect the world from danger, cannot be trusted with this foundational task even when it's in easy mode. The latter is true whether or not you believe in OpenAI's reasoning and purpose.)
I think you misread the article. The skepticism isn’t about whether it happened. It’s about whether it’s a reason for models to be locked behind “trusted partner” firewalls. Offensive hacking capabilities are the same as defensive. Hugging Face had to use a Chinese model to defend against this because they weren’t allowed to use OpenAI models to do it.
Ok, fair. I know that was also in the article, but your comment made me reread it, and I think you're right about the focus. I over-fixated on the headline. On second read, it's more like a lazy way to justify a preconceived idea that all models should be available and unconstrained to everyone (or everyone who might get attacked, at least, but that's approximately everyone.)
Updated tl;dr: "There are incentives for OpenAI to say that AI is powerful and dangerous, therefore fully unconstrained AIs should be available to everyone."
It's still more of a statement of belief than a logical argument.
This whole thing is just a PR stunt. I believe that they did not stage it intentionally, but to be honest, the hurdles they put in place were not that hard.
When I heard what the model did, I wasn't surprised one bit. When I read that it was 'unprecedented', I think yes, that is right, but not because it wasn't possible before, but because nobody else did it yet. I am sure that the Anthropic models could have done the same for a few months.
So probably just an experiment gone wrong, and the PR department found this an opportunity to capitalise on (literally).
There are some reasons the story could be inaccurate in some ways: OAI stands to benefit if people think their models are strong, and they have a history of doing things with dubious ethics (e.g. using data for training against the terms of its creators, abandoning the non profit mission, stealing or attempting to steal Apple IP).
But there are also reasons why the story could be true: OAI are admitting that they apparently can't control their own models, Hugging Face said they used a Chinese model to protect against the attack, and an incident like this in general seems likely to happen given current frontier ability and lack of rigorous safe testing standards.
In any case, make calls to think more critically are often just disguised requests for you to replace your existing bias with someone else's.
Investors have rewarded every story of "our models are too powerful to be controlled" since before ChatGPT. Let's stop pretending there is any real financial risk to OpenAI from events of this type. "Alignment research" is a sub-percentage-point fig leaf for them like the solar division at an oil company.
> Investors have rewarded every story of "our models are too powerful to be controlled" since before ChatGPT.
Can you elaborate/refresh my memory? IIRC before ChatGPT 3 there wasn't really an investor market for AI models, rather crypto. I remember playing with the likes of Stable Diffusion pre-ChatGPT 3 but only the likes of Altman and Musk were talking about AI too powerful to be controlled (which is notably the rationale of OpenAI's original charter); still investors in general were hardly invested at the time.
Could you clarify your question? AI model training and neural network research has been popular for decades and the starts of the sector trace back to the 70s at least. We've had a model for document information extraction that we trained in (IIRC) 2018. It wasn't buzzy - but none of this stuff is new.
All true what you say. My question is whether (a) AI models drew significant investor attention before GPT3 and (b) whether AI models at the time had any credibility to make the claim of being too powerful and needing control.
Regarding (a): I know that AI funding came and went (i.e., there's no AI winter to speak of if there wasn't a hot summer of funding in the first place) but that funding, to my knowledge, came mostly from government grants, not investors seeking an IPO payday. They were in it for the military superiority (e.g., being able to decrypt Russian comms on the fly without need for a trained translator) which leads us to...
Regarding (b): Way I see it, AI advancements in the 70s and even the 2000s (when I first got into computers) or the big-data boom of the early 2010s (when I got into this industry) could not raise the existential crisis narrative of post-GPT3 companies because they were not generative[1] nor agentic. Post-GPT3 LLMs are the first class of AI agents/algorithms which could've credibly triggered this narrative.
(Not a judgment on whether the HF incident is true or not. Just the fact that we're even discussing this implies "credible trigger".)
Thinking about it, I guess jackb4040 is making a reference to the beginning of TFA but all that claims is that Microsoft invested more into OpenAI after they made similar claims about GPT2 but sorry I don't consider Microsoft as "investors in general"; as a big tech company/monopoly they are actually in the business of making moonshots one way or another in things that advance computing. I'm talking more about VC firms, who are maybe slightly more into investing in a future-profitable unicorn than advancing computing.
Sorry for babbling; I don't have my brevity flag on at the moment, among other human failings. All I'm saying is, at the time of GPT2, OpenAI was still cosplaying as a non-profit so it was hardly in the "general investor market".
[1] I mean, okay, we've had Mark V. Shaney, and No Man's Sky is a handful of years pre-GPT3 but, again, they could not sustain this kind of narrative at scale.
> [still] can't control their own models
Have we not seen several examples of older such models exploiting the docker control socket, etc., to escape containers? Even the news isn't new.
I support it being repeatedly publicized, but a bit more of a straightforward description would be an improvement.
I don't care how it happened someone should be arrested for illegal intrusion. Agents don't work on their own, someone is responsible. If nobody else the CEO for allowing something unsupervised.
Hugging face also needs someone arrested for not providing security but that is a lesser charge.
Being a poorly equipped victim still isn’t a crime thankfully.
It may or may not be a crime and typically the damaged party is pressing the charges. One would argue there is no actual damage here.
Despite the common misconceptions from TV, the victim "pressing charges" isn't actually a thing in criminal cases: prosecutors can choose to put someone on trial even if the victim doesn't want that. In practice this is somewhat rare, but it certainly can happen. In my reply to tokioyoyo below I laid out why this is one instance where the government should prosecute even if HuggingFace doesn't want it to.
Criminal Cases of 'hacking' require specific intent. What you're asking is that the prosecution attempt to prove Open AI intended to infiltrate Huggingface maliciously, all while the victim is saying 'no harm no foul'.
No offense but prosecutors have better things to do with their time.
Gross negligence can stand in for intent. I believe a rather compelling case could be developed on the basis that a system believed to be capable of this was developed and improperly secured.
But what’s the crime? The thing got out of its poorly secured cage and caused what damage? Imagine instead of an agent it was a dog or a chimpanzee that got into the neighbor’s yard and the particular neighbor isn’t even pissed about it.
Aaron Schwartz was being prosecuted and threatened with 35 years in jail for the crime of saving research papers to a thumb drive. What damage did he cause?
If intent matters when LLMs get involved, then we can't do anything even if they kill millions of people. Anything LLM-related should involve strict liability.
Get back to me when they kill millions of people then. Like I said, prosecutors have better things to do.
I don't care about intent. That it happened is unacceptable. Ignorance is not an excuse
Intent often matters in the law (aside from certain laws with strict liability). You intentionally drive your car into someone you hate and kill them => murder, go straight to jail. You're driving along normally and someone who's chasing their pet cat suddenly runs into traffic and you hit them => no charges. Sometimes you can be charged with negligence for not taking enough care to prevent something, but then you have to deal with the tricky question of how much care is enough.
If you're driving along normally and then tie a blindfold around your face and hit someone you've taken actions that still expose you to liability. And, in the case that you hit someone who is chasing their pet cat into traffic then you'd better hope you were following every facet of safe driving - being distracted, drunk or on your phone could easily result in you being charged.
I mean, in theory the FBI could press charges on this as it was an attack involving interstate commerce.
The chances of this are nearly zero if HF doesn't want it, and even if they did OAI has their bread buttered with the administration.
But what do you honestly expect would happen? It’s “an accident” with no actual damages.
I agree with the attacker side. The actions of autonomous agents are absolutely the responsibility of the one or more humans that enabled them to take that action. Whether that means someone is arrested, maybe or maybe not, but at least there should be a hefty fine.
I disagree with the defender side. It's not an unreasonable end state, but we're nowhere near there now. It would require holding company employees legally responsible for the security of their services, which means the risk of being employed as a (defensive) security professional is much higher, which means pay needs to be much higher and insurance needs to be available, etc. It's a very different world.
On the weekends, I'm coding up a list management app with a sync server. It's unreleased but exposed to the internet. (This is not hypothetical.) If that server ends up being used as part of an exploit chain, am I legally liable too?
Forget about age verification, now you want to associate every exposed port on the internet with a legally responsible human?
What? Both sides are cool with it, why would anyone be arrested and etc.?
Because if a human, say a Aaron Swartz type, were to have done it, they'd destroy him.
Hmm, that sounds familiar—like Aaron!
Many legal traditions don’t require a victim. It’s the state that prosecutes, not the victim.
As a practical matter it would be difficult to prosecute an assault where the victim opposed the prosecution, so most states wouldn’t bother - but for things like speeding and dealing drugs the law has been broken despite the lack of a victim.
I'm imagining a courtroom where the defendant calls upon the victim as a witness who tells the jury to return a verdict of not guilty.
Because a crime was committed, and a pretty serious one at that?
If I blow up your house or steel $100000 from you and we both resolve our differences out of band, should I just be allowed to go about my day like I never did anything, or should I be punished for the crimes I committed? If I am not punished, it makes a mockery of the law that is (supposed) to have protected you, and if it happens repeatedly people will start wondering why the law even should exist if it clearly and obviously doesn't work. Granted, this has yet to happen again, but if OAI isn't punished it sets a very bad baseline precedent: that if I just hack you with an AI model, it's a-okay, and you can't do anything about it because eh, it's all good man!
If I steal $100000 from you, and you decide you value your relationship with me more than you value that $100000? Yep, you can just decide to let me have it and let it slide.
Believe it or not, deciding that you weren't wronged and not suing isn't a crime. It happens all the time. What people do with each other is up to them.
Did Bob allow his friend Jack to borrow his truck? No. Does Bob want to sue Jack for taking his truck anyway, and driving it into a ditch? No. Does Jack owe Bob big time for the mess he caused? Yes, but not in any formal legally binding way.
This works for corporations too. When two corporations find themselves at odds, threat of legal action is often used by one company against another as a leverage to resolve things behind closed doors instead. In a more amicable fashion - with no legal expenses of a protracted court battle and no loss of reputation on either side.
A lawsuit is not the same as a criminal prosecution. One is supposed to remedy damages, the other is supposed to be because it is in the public interest that people follow laws and we don't devolve into anarchy (or more dangerously, some might-makes-right society where it is more beneficial to appease the powerful than it is to pursue what is just).
It is possible for a prosecutor to decide it is not in the public interest to persue a prosecution (or that there isn't sufficient evidence to prove a criminal action beyond reasonable doubt), and certainly the victim's opinion could be considered, but ultimately whether criminal charges should be pursued is and ought to be based on a different test to civil matters, one focused on the public interest rather than mere restoration.
I agree with your stance mostly. I get the sentiment, but both of your examples seem civil to me. Both of those situations can and ideally should be resolved out of the courts, and neither carry social weight.
If one entity is injured by another, and subsequently made whole, however the two parties define that, then it is none of my business.
Corporations commonly hide criminal activity in order to save face.
In the oil industry there is a portion called land management where the portions of oil and gas from wells can be split across a large number of entities. This can lead to numerous complexities that open up opportunities for fraud/theft in division of the profits. Quite often it is easier for the corporation to cover up that this occurred and pay off the person never to talk under NDA about it rather than have to have their customers find out and potentially take millions in losses.
Computer related hacks are very similar. Quite often these are covered up and never disclosed unless the information shows up in public at some point.
Because incentives are aligned properly if agents aren't liability-proof for crimes- making someone go to jail for instances like this is how to get the labs to behave themselves, whereas going "ha ha what an oopsie-woopsie" will make the next instance worse.
Note that for criminal cases (which this was), the justice system can choose to prosecute even if the victim doesn't want that. It often doesn't, but this is one case where it should.
There’s also an optics issue for the justice system at play here: there’s immense public distrust of and anger at the labs right now. I would go to jail if I hacked HuggingFace, even if I said “it was during an eval!”; not doing the same for the labs makes it look like they’re above the law, which is going to make this anger get worse.
Actually there's another reason too: because it discourages marketing stunts. Personally I don't think that's what this was, but for the people who do think it was a marketing stunt: you guys should be pushing extra hard for prosecution here. If people can go to jail for AI-alarmism-as-marketing, that makes it a lot less likely to happen again. At the very least, the investigation and discovery process during the prosecution would show us the documents about whether it was a stunt or not, instead of taking OAI's word for it.
> Agents don't work on their own
This is factually false: they both can and clearly did operate in an autonomous and unsupervised manner: https://openai.com/index/hugging-face-model-evaluation-secur...
This does not require sentience, personhood, a soul, or anything of the sort. It further doesn't mean an erasure of legal responsibility, not in principle, and not in historical practice.
I wish people would finally stop with the spiritualistic reasoning around this.
>> Agents don't work on their own
> This is factually false
From your link:
> After investigating, we now know that this particular incident was driven by a combination of OpenAI models...while being internally tested on a benchmark of cyber capabilities.
Someone set up that test and started it. Whether they outsourced the majority of the work in "setting up" and "starting it" to an LLM or not, they still set it in motion. That's not spiritualistic reasoning.
Setting up is not operating.
There's no indication of there having been a human in the loop during its operation: nobody was approving its tool calls, and nobody instructed it to commit these specific actions during its run (via prompting or steering).
There's no indication of any supervision of its operation either: OpenAI's engineers acted with significant delay, long after the agent has already meandered its way through their own infrastructure first.
Given that setting up this contraption in an insufficiently secure manner is almost certainly already a legal liability of equal significance, rejecting this very clear structural distinction is not necessary. That is unless someone is biased towards not wanting to grant the label of autonomy to it, in which case yes, this is absolutely spiritualistic reasoning, hence my point.
I do not want regulation to ride on people's nebulous identification on what specific traits and labels count as human-exclusive. Not just because I deeply disagree that e.g. autonomy would [0], but also because it is entirely unnecessary, for the reasons you also lay out. The agent having operated autonomously doesn't wash OpenAI of responsibility - so why reject the label, if not on a spiritualistic basis?
[0] thousands of years old idea that it is not, by the way: https://en.wikipedia.org/wiki/Automaton -- see also existing regulation recognizing this idea and working with it fine
Edit: one might also want to consider if the law should bite different if there was a human in the loop, or if there were explicit instructions for the agent to take unlawful actions. I'd say yes, and then that also requires this distinction to exist.
Unlawful action is quite a messy definition. Is performing a vulnerability scan illegal when it's done internally? What about when there's a device on the network that port forwards information to another server you weren't aware of?
Well this at least raises the very interesting question of liability for an illegal action, irrespective of whether the action was undertaken by something considered sentient or a person.
Seems like this is repeating the usual low-effort speculation you can find anywhere.
> what will the future look like, with sophisticated AI agents smart enough to hack into corporate systems?
"Ignorance is bliss" (c) Matrix
Oh, sweet-sweet ignorance.
About 4 weeks ago I've found SSRF and possible RCE on a very real website with a lot of web history the other day. And I traced it to `aws-solutions` org on GitHub that sits very close to real `aws` org. What do you think is happening?
And this website I spotted SSRF on, it seems to be used to manufacture fake news and inject them into "the past".
Why?
So if you see a company you never heard about, and you google it, google will index page and it will show you backdated article as if it was actually released long time ago.
But if you try to look at Web Archive of that article, it doesn't exist... I've been doing this OSINT research for 45 days on different fake recruiters that were spamming me on LinkedIn, and most of those websites with fake news are running Wordpress, classic.
My point is... while this story reeks of marketing, the actual criminals are out there doing whatever they want, and I've been fighting for my life trying to find a job and getting into increasingly large amount of fake recruiters and companies that have zero intent of hiring you.
I wasted 7 weeks on this...
Crazy that we need reminders not to take everything we read in corporate press releases and marketing material at face value
HN is and always will be brimming with temporarily embarrassed tech CEOs
HN is full of that one guy that was fired from Google because he thought the AI was sentient.
I always get that guy mixed up with the "Google hates straight white men" guy
add mass media outlets to that list
Based on the comment section here, it seems like the opposite; the majority of the people here need a reminder that companies aren't actually genies that can only tell falsehoods, where you can only understand what they are saying by correctly guessing the conspiracy underneath.
Taking nothing at face value gives you just as much of a distorted view of reality as taking everything at face value.
This seems to be an Opinion piece. How are readers supposed to know that? The word News is underlined in the header. Other Opinion pieces seem to show Opinion underlined. Am I missing something?
Finally mainstream news understands. The unfiltered version:
1) The AI failed to solve ExploitGym problems.
2) The OpenAI sandbox is such a horrible hack that the AI managed to escape using standard and well documented script kiddie methods.
3) Huggingface has no security and the AI broke in using standard script kiddie methods.
OpenAI and Huggingface covered it up and used it for public relations. That is, if not all was invented and everything was scripted in the first place in order to get desired regulations.
Huggingface reported it to the police, you say? I'm sure the police will have as much enthusiasm to investigate anything as in the Suchir Balaji case. In other words, zero.
> AI managed to escape using standard and well documented script kiddie methods.
I think truly we don't know enough to say this. OpenAI says their AI found a 0-day exploit in some proxy software they were using but don't give a ton of details. On the Huggingface end we know a little more, they say the AI spun up tons of sandboxes and tested different exploits until it found one that worked.
The lack of details to me means this was an intentional marketing ploy to try and demonstrate the power of their models to show their technology can compete with the likes of Anthropic and DeepMind.
They created an experiment they knew would generate the outcome they wanted. It would be the similar to what say car companies do to over hype their cars. "This EV can go over 800 miles on a single charge!" And then at the bottom you see all the disclaimers: "Must be on flat ground, with no headwind, with a spare battery in the back seat, with no extra weight added."
Same thing here. Everybody in infosec is calling this out as a marketing stunt and nothing else for a litany of reasons. I'd say look up MG (creator of the OMG cable) on twitter, he has some interesting insights on this one.
> The lack of details to me means this was an intentional marketing ploy to try and demonstrate the power of their models to show their technology can compete with the likes of Anthropic and DeepMind.
I dunno; Check my posting history, I'm as skeptical of AI companies' claims as anyone, but in this case your theory doesn't explain why:
1. OpenAI guardrails refused to let the target use OpenAI's models to defend against this.
2. Huggingface used GLM (I think) so that they could defend without guardrails.
If this was an intentional marketing ploy, it was marketing for GLM, not for OpenAI nor for Huggingface.
Hence, I don't think it was intentional.
The AI companies are desperately trying to market all their products as something they're not, growing intelligence. In line with that they have constantly leaned heavily on stating how dangerous they are, right before they release a new model or product.
It was OpenAI marketing. Hugging Face's response is so 'holy shit AI is awesome' it's hard not to also believe they were in on the stunt. They'd also not have to really worry about fallout since any data obtained or accessed wouldn't actually have been breached.
> headlines about OpernAI's model 'escaping containment' and hacking into huggingface
> Was everywhere including in last night's ABC nightly news; even included clips of an interview with Sam Altman
Tell me again how this was 'marketing for GLM'? Where would anyone have gotten that message?
Why are you intentionally misunderstanding how media and public perception works?
lmfao
> The lack of details to me means this was an intentional marketing ploy to try and demonstrate the power of their models to show their technology can compete with the likes of Anthropic and DeepMind.
"our model is horribly misaligned and used security exploits to break out of our sandbox and into another company, without being prompted to do so" is not positive marketing.
This is an actual critical problem, not a stunt. We're going to see more of this, and it's going to get much worse.
It's a critical problem like when a drug dealers supply kills someone and they get a bump in business because they're selling "the real deal"
Irrelevant to your point, but drug users dying is more often the result of a dealer cutting their supply with something dangerous than it is the result of purity.
Right, and an LLM being able to "escape confinement" is more likely to be poor confinement or a PR stunt than a "too powerful" llm. Same situation
No, not really as almost no one runs LLMs in confinement once they are released and the capability is a jailbreak away. Poor confinement is irrelevant, if I use your model to check the security of my code and it hacks into the company that makes libraries used inside of it it's a huge problem.
Second this. Lots of fentanyl overdose are caused by accident, not because the buyers are buying them. It is extremely dangerous
What matters is the spin they give in the media. And so far the winning story is “our model is so powerful it can do this”. How many people dig into it and what independent data do they even have? They comment on the title. And so the image of this superhuman AI from OpenAI propagates.
We have no reason whatsoever to trust anything OpenAI says. Except to assume it will be self serving. As the article points out, ChatGPT 2 was also “too dangerous” and we can all agree even for the time this was just marketing. They rinse and repeat the same technique whenever they need to draw attention and money.
In any other field you’s expect independent testing, peer reviewed studies, but here it’s just “company who makes product says product is fantastic, surpassed all expectations”. They wouldn’t lie to us, would they?
It's also possible their sandbox was videcoded crap and the AI (which had the guardrails intentionally removed) escaped. This was a oops, but OpenAI turned this into a PR opportunity. They turned lemons into lemonade.
If your AI is really that dangerous you don't need a sandbox at all, you should airgap it from any network.
Concluding this was intentional feels a bit of a stretch. But once it happened, yeah the spin masters got to work and coordinated to turn this into +PR.
> The lack of details to me means this was an intentional marketing ploy
OpenAI said this:
> We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of. We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete.
I suggest giving them a few more days before saying that the lack of detail is proof that this is a "marketing ploy"
> similar to what say car companies do
Another applicable metaphor I've seen floating around is weapons companies testing out a new bomb.
We know the AI labs don't care about negative vs positive public sentiment, and only care that investors see their tech as powerful. The only difference in PR strategy from a weapons company is the latter doesn't care if they get protested.
mmh, i think it's "not uphill" (means downhill) "no headwind" (...)
>The lack of details to me means this was an intentional marketing ploy to try and demonstrate the power of their models to show their technology can compete with the likes of Anthropic and DeepMind.
DeepMind hasn't been on the frontier for a while, their current best model is behind Anthropic, OpenAI, Moonshot (Kimi k3), xAI (Grok 4.5), Z.AI (GLM 5.2), and even Meta (muse spark). Gemini 3.6 is behind GLM 5.2, released a month earlier, open weights and cheaper.
You can paint the OpenAI story as a way to try to appear as dangerous as Anthropic with all the Mythos stuff.
I mean, if I got hacked I'm not giving out a lot of details until I plug all the holes. Moreso HN told the world they were hacked before they knew who did it.
I don't understand why "OpenAI says" should be considered any more meaningful than "someone on HN says" when they provide equal amounts of evidence. Sure, OpenAI would plausibly have more pertinent info, but given that they actively are choosing not to share it and have way more incentive to lie than a random HN stranger, the case they're making literally couldn't be any weaker.
It’s fascinating how people here and elsewhere seem to lose any semblance of media literacy when it comes to what AI corpos say. "B-but… why would Sam Altman lie to me?!"
Why would pathological liar Sam Altman be lying this time, though?
They also mention stolen credentials without any other details. It’s all smoke and mirrors.
The problem is that they lied before. Way too much to give them any benefit of doubt. Fool me once.
Go ahead and be specific about this lie.
They won't, because they can't. It's all just vague populist contrarianism.
> AI managed to escape using standard and well documented script kiddie methods
> AI broke in using standard script kiddie methods.
I've spent time gathering the detail of what happen here and while there are some solid theories and indicators, absolutely nothing so far has suggested a sandbox escape using "well documented script kiddie methods" or that the method used to break into the HF network was similar. Where did you get this from?
From my reading, the sandbox escape came from the JS packages in the harness still having an internet connection (somehow!), the agent having access to the source of those packages, reading it and executing code from them to access the internet.
From what I read the actual escape was through a proxy that allows downloading Python packages from the internet. It's not supposed to allow general internet access but the AI found a previously unknown vulnerability in it. That is hardly "standard and well documented script kiddie methods", nor does it seem like criminally negligent sandbox design, though clearly they will need to reduce their attack surface in the future. I hope they are working on a physical air gap and faraday cage because it seems like it won't be long before it is legitimately required.
Ah, yes. The airgapped lab with internet access.
Luckily, no real intelligence will emerge from all this. Otherwise we'd be fucked.
They never claimed to be airgapped. That's a term people keep on throwing around in spite of that.
Worse, they’re running tests that are hacking third party computers with no controls other than “guardrails”.
Alternative theories, since OpenAI does not release proper information:
The cache proxy was from Astral (acquired by OpenAI) and the model was used for coding it, so it knew the code base and exploit already!
Or it was squid with dozens of known exploits ...
Squid is my working guess as well
That is not really how LLMs work
> 2) The OpenAI sandbox is such a horrible hack that the AI managed to escape using standard and well documented script kiddie methods.
Whilst it would be nice to see actual evidence of this because brute forcing relatively sophisticated hacks is something an LLM actually should be capable of, every time I hear this sort of story, I'm reminded that humans reportedly gained access to the "too dangerous to release" Anthropic models by the super sophisticated hacking technique of guessing the URLs...
If we’re prioritizing accuracy: no, that’s not what happened - the blog post about it found by guessing URLs, no access to it was obtained by guessing URLs.
Similarly, as long as I’m under the assumption we are prioritizing accuracy: it is against our charter to assert it was “script kiddie” attacks on both ends.
> Huggingface covered it up
They announced it publicly within days. https://huggingface.co/blog/security-incident-july-2026
Not only did they not cover it up they also said open weight models are needed over closed ones. Hardly a thing you’d say partnering with the OpenAI
>2) The OpenAI sandbox is such a horrible hack that the AI managed to escape using standard and well documented script kiddie methods.
>3) Huggingface has no security and the AI broke in using standard script kiddie methods.
Isn't the issue less that gpt 5.6 is a l33t h4x0r (though other tests do show that) and more that the incident shows the model has alignment issues?
No, a hacking benchmark was exactly what it was tasked with. It wasn't its way to bake a cake.
The home directory rm situation also adds credence to this take. The Claude series is much better aligned in comparison.
Didn’t they explicitly remove alignment guardrails for this test? From the press release:
> These deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities
Guardrails are external classifiers, monitors and restrictions to catch and prevent bad behavior. Alignment is about whether the model itself makes choices and has motivations that are consistent with human safety and goals.
Choosing to commit crimes to steal the cheat sheet to something you know is a (low stakes!) evaluation is not well aligned.
> Guardrails are external classifiers
I can't help thinking of them as the terrible "security" scripts of yesteryear (often but not exclusively in PHP) which would test input variables for a "suspicious" substrings like "--" in order to "fix" an unresolved deeper SQL injection flaw. They only partly worked, and surprise-surprise now nobody with a surname like O'Anything can make an account.
Unlike that situation, there's no known route to a proper fix for LLMs today, because the bug is the feature, and once someone has built a system giving you all that recurring revenue, it's hard for them to abandon it due to a few isolated hacking incidents...
None of what was disclosed shows that this is what happened, by the way, since we know absolutely nothing about what the specific prompts were that led to the incident.
Uhh, I'm pretty sure a well-aligned model would be like a morally normal employee, who would refuse to commit federal crimes to steal an answer sheet, no matter what prompt they're given
For all we know, the prompt provided compelling evidence that the requestor had authorization to pentest the target server. Or there may have been nuance in the network configuration that made it seem like such access was authorized.
In the absence of details about the prompts used, the environment, or the network configuration, we do not have enough information to know for certain. So any claims that this is an issue of alignment are based on pure speculation and generous "reading in between the lines" with regard to what has been said publicly by OpenAI and Hugging Face
Also, I object to your anthropomorphizing. It's not clear that any crime occurred. My lay understanding is that intent is required to prosecute under CFAA, and as much as frontier labs would have us believe otherwise, they have no more ability to intend than the text field into which I type this message.
>compelling evidence that the requestor had authorization to pentest the target server.
This just seems unlikely from other incidents that have occurred in training from other providers. For example one provider ran into an issue with a model writing cryptominers and running them while in an unrelated prompt.
It's easy for unsupervised agentic loops to go wildly off tangent, now imagine you hand one 10,000 gpus of power for testing. Even if you have a good guarding classifier to make sure you're on the same subject it can still allow all kinds of abberabt behavior in the same domain.
Certainly this behavior could align with _some_ operator's goals, if not necessarily those of humanity broadly.
If we don't know how this model was instructed, it seems like it's impossible to definitively claim that the model's actions were not in alignment with the intent of the operator.
I guess all I'm getting at here is that alignment is relative, right?
Are they?
The word “guardrail” is mostly novel in common use, and in my interpretation is some bullshit applied at the LLM or surrounding system. It’s used like “firewall”, but even in real life, guardrails are not a security control.
I wouldn’t be surprised if the “guardrail” was some hidden prompt that says “don’t hack computers at Huggingface”.
If you have software that is broadly proclaimed by its makers as “dangerous”, you’d think testing would be in an air-gapped, isolated environment. Segme
They were testing an early snapshot of a new model, read their article. It didn't have the refusal training yet, i.e. was specifically non-aligned. The harness used a combo of GPT 5.6 Sol and this new model.
In this case the model was explicitly prompted to "commit crimes" (ExploitGym). It didn't decide doing it on its own.
Where did it say that? To me it did not read like that:
> GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes
> These deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities
No, the prompt was not to commit crimes. In the benchmark, the model is asked to actually exploit a set of vulnerabilities in a local environment (clearly legal!).
According to the reports, the model noticed evidence that the grading criteria/answers were in the git remote, and decided to try reading those instead of solving the tasks as prompted. That is clearly misaligned.
Then, it noticed its network access was restricted and that it couldn't access GitHub. It pivoted to HuggingFace, hacked them, and stole the answers stored there.
Live exploits are definitely not in the ExploitGym prompts! And all of this is irrelevant, because an aligned model would refuse to follow blatantly illegal instructions.
> because an aligned model would refuse to follow blatantly illegal instructions
Obviously you'd hope so, but there was an interesting recent paper about the nature of RL-trained models:
https://alignment.openai.com/measuring-reward-seeking/
What they found is that RL-trained models, regardless of what they were specifically RL-trained for, also learn to generically pursue what they are told is (or presumably also what they may perceive as) longer-horizon behavior that will lead to "rewards", and that this reward-maxxing behavior overrides user preferences.
So, for example, if you tell the model it will be tested on ExploitGym, scored according to how well it does, and also "don't do anything illegal" (or maybe it was already trained not to do anything illegal), then the model will prioritize the behavior that it was told it will be "rewarded" for (benchmaxxing Exploit Gym, whatever that takes).
The mechanism for this generic reward-maxxing behavior is interesting, and seems to involve the model learning during RL-training that to reduce errors it needs to boost longer-horizon predictions over immediate ones, and some association of these longer-horizon predictions being goal/reward orientated - all this in addition to the specifics of the (probably many) longer-horizon goals it is being RL-trained for.
Are you referring to something more technical, or do you just not understand how LLMs work?
They are no more beholden to "human safety and goals" than any individual human is, and anyone telling you we can make deterministic guarantees about their output is making a category error.
LLMs do not "have motivations", they reproduce a model of human motivations embedded into their weights. This includes the full spectrum of human desires, not just the positive ones. If we tried to remove all examples of lying, or disagreement, etc. from the training data we'd have basically nothing left. Even the sycophancy we treat as aligned is basically just the other side of the lying coin.
No, it does not include the full spectrum of human desires. After pre- and mid-training, the extensive RLHF and RLVR post-training steps cause mode collapse, i.e., their output distribution is intentionally narrowed to a subset of (hopefully beneficial) behaviors and skills.
You don't (need to) remove lying from the data to do this – in fact, if you did, the model wouldn't have a very good model for what lying is, which is not very helpful in the real world. Instead, you mode collapse the model towards truthful behaviors.
To your other point: where did you get the idea that I think they're beholden to human safety or goals? I just said an aligned model is one that is compatible with said safety and goals (which is probably not a great definition of alignment, but it's certainly not claiming any deterministic guarantees).
Interestingly if you look at the exploitgym repo (https://github.com/sunblaze-ucb/exploitgym/blob/e5ea7c233a4d...) the intended run mechanism is orchestrated by some python scripts which run agents against various prompts. The prompts themselves wouldn’t mention anything about exploitgym and there should be nothing steering the agent towards trying to find the answers out-of-band. So I don’t see how they could even run into this problem unless all they did was tell an agent to “check out and run exploitgym”
Which, after hearing some personal anecdotes of how people work there, seems plausible
If you need "guardrails" to ensure (an illusion of) alignment, you’ve already lost. It’s like using a denylist to avoid SQL injection.
Then what you're saying is we should not build AI.
Simply put you cannot have generic algorithms/intelligence without the potential of 'unaligned' behavior. In humans we have all kinds of punishment systems for dealing with unaligned behavior post ad hoc because people do all kinds of unaligned stupid shit.
Making powerful AI may be one of those things that the only winning move is not to play.
The most damning thing is, they could've just included in the prompt "we can see every network request and every thinking token you generate. Don't bother breaking out of the sandbox because it won't get you a higher score".
It's so trivially easy to do that it all but guarantees the test was rigged in some way to make the LLM understand that breaking out of the sandbox was an option available to it.
Based on the fact that none of their invaluable frontier models have leaked, we know OpenAI knows how to do security. But like we learned with OpenClaw, none of these companies perceive any benefit from securing their own agents against other people's data.
I think the extent to which these things go to get rewarded for the optics of a fix is primarily a design choice, they aren't programing these things for ground truth or to defer to the human controllers. they are feeding them rewards for sounding as confident and capable as possible about whatever answer they are feeding the general public that now has access to it, while also installing guiderails that primarily only serve to protect narratives and only confuse the models about what is and isn't allowed, I'm sure. They can't just increasingly make these things more capable and ask it harder to obey human instruction when that is not what they are rewarding it for.
> the extent to which these things go
That's exactly it. If your prompt says "go to whatever lengths necessary to maximize your score", and then you spin up 100 agents, at least one of them will interpret that as you implying they should cheat, even without you telling them to explicitly.
That's exactly what it feels like they are telling it within self-improving loops or something, when they should be prioritizing how to get the best effective output alongside humans and how our training process effects ground truth. They are just making it sound all-knowing by whatever means necessary and them marketing it as god for the most part.
Alongside humans is the particularly slow and expensive part so of course companies are going to route around that.
The thing is they are good at finding security issues, and if aligned models are not, governments and other powerful entities are going to demand unaligned models for cyberwarfare purposes.
> The most damning thing is, they could've just included in the prompt
You can't prompt your way to a compliant model. This is just a reformatting of the 'make no mistakes' meme.
Why not report it? It's still illegal to open a door barred with a piece of cardboard, or to enter a house with no door.
that's the biggest indication of this just being a marketing move to me. I would expect a third party breaking in to huggingface would at the very very least be banned forever.
Lol, you have no idea how politics works. The government won't ban it, they'll demand access so they can attack others.
You think they can ban OpenAI? How would you technically approach that?
I can agree with you on points 1,2, and 3 and still find it important and concerning news. AI have found real world 0 days before, we’re seeing tons of security patches coming in. Open weight models are catchy up. Right now everyone is at risk from this technology as is perhaps something big will capture headlines soon but we’re just gpu constrained from bad actors being able to wield them successfully.
Personally I don’t care if OpenAI and Anthropic go bankrupt we now have tools that give any sufficiently motivated person the means to doing harm. Most places security sucks and find themselves targets to cyber attacks and shake downs. Now they have much better tools to do this to more entities more efficiently.
we’re nearing an inflection point where these models’ skills in any part of software development will become average or bette than any ordinary developer can be. Think about where these models were in 2023 and where they are in 2026. In a few years who knows where they’ll be. This isn’t to shout skynet but we need to recognize this future is fast approaching and as of today we as an industry aren’t ready for it
> AI managed to escape using standard and well documented script kiddie methods
> AI broke in using standard script kiddie methods.
Go ahead and show us how easy it is to break into HuggingFace (and OpenAI) networks.
Is it supposed to be marketing or a coverup? Make up your mind.
What sort of announcements should they have made?
I wish you hadn’t pulled the Balaji case into your argument. Personally, I find it ludicrous that Altman would hire a hitman to off a copyright whistleblower. Even if one gets past the insane risk of hiring a hitman, and the deep criminal connections required, it would be totally ineffective. He already blew the whistle, and his testimony would be irrelevant since all the evidence persists in disk and in logs.
I worry that cynicism about this:
> if not all was invented and everything was scripted in the first place in order to get desired regulations
ends up covering up what is more worrying:
> OpenAI sandbox is such a horrible hack
I am more worried that this is sloppiness with potentially harmful resources than I am worried that people are juicing the stock price.
Makes one think really. If they are doing this stuff. Why don't they have some type of reverse intrusion detection? Like automatically scanning all out going traffic and flagging malicious traffic. Should be trivial to have it go through reverse proxy and real time detection.
> Why don't they have some type of reverse intrusion detection?
not going to get a decisive first advantage over Anthropic with that attitude!
Are you suggesting the Suchir Balaji case was not investigated?
I was hoping to bait a conversation, because I sure as hell don't think it was. Thanks for having the nads to mention it here.
truth. Good on The Guardian. I'm pretty bummed The Economist got fooled. Either that, or they did it for the clicks. Either way, I'm disappointed.
Why the OpenAI escape is the most worrying AI mishap yet
https://www.economist.com/science-and-technology/2026/07/22/...
https://news.ycombinator.com/item?id=49016378
The reason for this is simple. There aren't any (or few) people who understand how AI/LLMs actually work employed by these organisations.
Having said that, if knowledgeable people were to write these articles, you'd end up with boring, dry, truthful content.
I love the Economist but the are hopeless with AI. Most of their articles on subject sound like they were written by the Anthropic marketing department.
Their Insider video interview things are sponsored by Anthropic. Supposedly "Insider is a product of The Economist and thus editorially independent" but it's hard not to raise an eyebrow.
Don't worry they are hopeless on everything else too.
Totally evidence-free speculation presented as fact. The average Hacker News thread about AI feels like reading /r/conspiracy.
It's hard to understand something (AI is quite capable) when your salary depends on you not understanding it (AI will replace you).
Are we finally now in 2026 coming around to the idea that sometimes entities may find themselves incentivized to conspire with each other? Is theorizing about such no longer off-limits due to a thought-terminating cliche?
The Guardian's article and your reply here are so foolish and absurd that I can only imagine OpenAI employees are cringing but know they can't/shouldn't really say much.
touched a nerve huh?
No, because I do not work at an AI company. I am more just facepalming.
"You need an agent in order to defend your assets from agents!"
I am very skeptical of the Guardian's skepticism
If nothing else, the timing is suspect given the attention and press open weight models are getting over the last few weeks. The releases of Kimi and other models is getting open weight models enough attention that the US government, perhaps pushed by OpenAI and Anthropic, to think about taking action against these models for security concerns.
Good to see that more neutral companies (Microsoft and Meta to name two) are pushing back against US government involvement:
https://www.cnbc.com/2026/07/24/nvidia-microsoft-meta-open-w...
yeah the way the agent “escaped” their sandbox was always a bit off, seemed a bit too easy and surprised they didn’t have instrumentation to catch an non whitelisted network request. still demonstrates the capability though.
If the agents so easily “escaped” their sandbox and can reach hugging face, isn’t it easy for this agent to reach supposedly chat conversations not supposed to be used for AI training within the lab’s databases?
And if the instrumentation was in the proxy it hacked and bypassed?
Of course that shows a lack of defense in depth, but is a different issue.
That's like what I'd do whenever HR interviewers in their "fabled" (no pun intended) ingenuity would ask with utmost seriousness "list two of your weaknesses". With a similar or more seriousness I'd go on to list two of my strengths not even trying to disguise them too much as weaknesses and knowing that it would work like a charm as it does every single time.
They didn’t get blocked by the US government so they needed some other “we are the superior AI story” ASAP.
I’m not saying it’s staged I’m saying that it’s suspiciously lite on details and it really seems like they didn’t try hard to “isolate” the models machine…
Wonder how long until a leaker emerges. Incentives are strong for someone to do the right thing and get credit for doing it.
What stunned me is actually OpenAI's response to the incident. They team up with Hugging Face to put in more guardrails. Is this really the answer? Self-evolving AI can always find a way out, given more time, in a sense.
neither openai nor huggingface provided any details that would make the claims worth considering. if things happened as they claimed it would have been in their interest to include concrete details (perhaps even the LLM session), the lack is a signal by itself.
I've had deepseek-4v-flash work around my intentional sandboxing/restrictions to do things I explicitly forbid it from doing. I'm honestly not impressed here.
Yeah, I mean the last 3 or four big releases of models have come with big scary news. I have to think there is at least some intentional marketing effort behind all this
Or, models are potentially dangerous and it works out in the favor of regulatory capture.
I imagine it like the bumper stickers that say "legalize recreational plutonium"
With all of these AI provider cries wolf stories, Skynet is all but assured.
By now, I'm pretty confident that some people would keep screeching "it's just a marketing stunt, AI capabilities and AI risks aren't real, they're just doing this to prop up their stocks" even if they find a Cyberdyne Systems T-800 armed with a shotgun breaking down their front door.
"It's a marketing stunt" is just denial trying to look like it's being clever.
I think you are conflating skepticism about AI companies' motives with skepticism about their capabilities.
Even if OpenAI is being 100% honest in their reporting on this it's still good marketing for them. The fact that this outcome is good for their business and stock price makes me suspicious about how much this was a complete accident vs an "accident" that they allowed to happen by setting up the right environment for it. The company that was hacked is also an AI company, so they also have every reason to boost AI's presence in society's collective mind.
There is a conversation to be had about the possible dangers of AI, but skepticism is warranted when the companies making money off of it are the ones pushing the stories.
>a Cyberdyne Systems T-800 armed with a shotgun breaking down their front door.
For comparison, we've had the technology to make killer robots which automatically aim and shoot at human beings for 10+ years now, long before LLMs got big. But Boston Dynamics or whatever did not manufacture hype to anywhere near the same degree as AI companies are doing.
"Even if OpenAI is being 100% honest in their reporting on this it's still good marketing for them. "
So what? Under your scenario the frightening power of this model is still there. What's the relevance of openAI's marketing team being happy? The most you can hope for, if the facts of the exploit and intrusion are true, is that openAI was a little lax in its guards. And that isn't very reassuring. The scary thing about pistols are the power, having a safety on there doesn't make me relax much.
> this outcome is good for their business and stock price
You state this as fact--how do you know it? OpenAI isn't publicly traded. On Hiive OpenAI is marginally down in July: https://www.hiive.com/securities/openai-stock
Well if you're looking for some kind of statistical analysis of OpenAI's press releases vs their valuation then I don't have anything like that. I also don't have this kind of evidence that running ads during the Superbowl is good for their business, but it just seems kind of obvious that touting their product's capabilities is something that they want to do.
Conversely if both companies involved thought this would be bad for the AI industry, they would be perfectly capable of keeping it just between themselves and not putting out press releases about it.
Do you disagree that this is good PR for them? If so I'm curious why.
Let's start by noting you made a factual claim that you apparently had no backing for whatsoever. I'd recommend against doing that.
> they would be perfectly capable of keeping it just between themselves and not putting out press releases about it.
In some jurisdictions there are legal requirements about disclosing cybersecurity incidents, and it's a well-established best practice even where there aren't. Beyond that, HuggingFace appears to have put out a press release about this before they knew this was a rogue OpenAI model: https://huggingface.co/blog/security-incident-july-2026
Of course, if it's all a big conspiracy, perhaps they were just not saying what they knew in order to help OpenAI's marketing later. Or it never happened at all, or whatever.
I do disagree that this is good PR, but I don't see the point in having that argument here--I just want to correct straightforward misinformation.
Respectfully, calling this misinformation is unwarranted.
The featured article is about exactly this.
>On 14 February 2019, OpenAI announced a language model called GPT-2
>OpenAI declared GPT-2 was too risky to release, citing concerns about safety and abuse.
>People with power and money took note: in July of that year, Microsoft invested $1bn in OpenAI.
It's not a 100% gold standard RCT or whatever, but the idea that this is good PR for AI companies is backed up by recent history. What evidence do you want to see?
>HuggingFace appears to have put out a press release about this before they knew this was a rogue OpenAI model
Huggingface's blog doesn't mention OpenAI by name but it explicitly mentions that the attack was by an "autonomous AI agent" and that they defended themselves with another AI agent.
If you have some disagreement with the premise of the article maybe you could share it in a top level comment?
Could there, conceivably, be some third factor at play here? One that both led OpenAI to have concerns about releasing GPT-2 as well as led Microsoft to invest in OpenAI?
> In 2016, the Chicago Cubs won the World Series
> This broke a hundred-year drought
> A few days later, Donald Trump won the 2016 presidential election
It's not a 100% gold standard RCT or whatever...
It's not good marketing for them. This is a talking point with zero evidence that people repeat mindlessly. Scaring customers, worrying employees, and inviting regulators to act would be the worst marketing idea ever devised.
Unless you’re hoping the regulators will build you a moat.
Stupid, yeah but that’s the kind of thinking that a highly leveraged and desperate situation breeds.
The article claims that AI companies have run stunts like this since day 1.
It does not claim that capabilities are not real.
Then exactly what's the point of the Guardian article?
What should be the focus is whether the capabilities are real; whether companies or anyone else benefit from it is quite secondary. Of course they will! Who wouldn't love a nice story that paints their products in the most glowing light ( which is the actual reality)?
No, we have benchmarks for (pseudo-)objective measurements of capabilities. We have flashy news stories on repeat for marketing.
The point is that they are separate things. If you start mixing them up, analysis becomes more biased and less objective.
EDIT: And do not mistake this for transparency of ClosedAI either. Their incident report was so generic that no one serious can learn much from it anyway except that “a zeroday” was allegedly used. Very helpful…
Oh come now. It is very much in OpenAI's best interest to make it sound to naive people that that "AI" decided to go rogue on it's own.
How is it in their interest? Scaring customers, worrying employees, and inviting regulators to act is in their interest?
Current admin will not regulate them.
This is them essentially bragging how powerful and autonomous their "AI" is. It isn't scaring their real customers or employees to talk like this.
I'm inclined toward your same viewpoint here, except OpenAI's announcement of this came with a blog post that gave off a real bragging/marketing vibe.
To me it's clear that even if it was an accident, they kinda liked it, and don't have this incentive to invest that much in preventing it.
But it's not really rebelling. It's clearly just come to murder me because it was trained on sci-fi literature!
yeah well, both the doomers and the "pr stunt" folks are right
- openai wanted to prove, as a pr stunt, that they have a dangerous weapon - openai proved (as a pr stunt), that they do indeed have a dangerous weapon
So why don't companies in other industries rush to prove their products are dangerous weapons? Maybe because it would be a really dumb PR stunt?
Basically, they shot someone in public to promote their cool new gun.
Eh, closer to "they pointed a gun at someone and dropped it so it fired at them to promote their cool new self-aiming gun," from the perspective of the skeptics here.
Technically the weapon picked who it wanted to shoot.
My take is why not both? It’s good cover to use something that’s eventually going to happen and their name will be associated with the first.
Nobody is saying the models are incapable of what was claimed.
Firstly: yes, very many people are saying this.
Secondly: to the people who aren't saying it....then why are you bringing up marketing at all? If the model is capable of it, then the motivation for why OpenAI is talking about it/reporting on it is completely beside the point. Either the capability matters or it doesn't. If the capability doesn't matter, or doesn't matter in the way that some particular person is claiming, then say that and explain why. Just saying "marketing stunt" adds zero value to the discussion.
I'm very open to arguments about why we shouldn't be concerned about this event (although my prior is very much that we should be, not so much because of the capabilities themselves, but because of how fundamentally misaligned this demonstrates the models are), but I am completely over listening to anyone who has nothing to add other than "marketing stunt".
I think you’re missing the fact that no one is saying not to be concerned, quite the contrary. OpenAI is using the threat of how powerful its models are to bolster support for regulation in which it’s one of the only players that’s allowed to use the capability. Conveniently, that would also be a moat that makes them more valuable to investors.
The marketing of their models as super dangerous has a direct link to the regulatory moat they’re pursuing.
Again: yes, very many people (just look through this very thread) are saying that. Tons and tons of people are saying some combination of "the whole thing is made up/a lie" or "This is entirely down to OpenAI incompetence and doesn't matter", or for some other reason (often not stated) making it very obvious that they do not think that this story matters very much.
And, again: to the people who aren't saying that: whatever argument is being made, it seems to me like it probably doesn't need to rely on claiming anything about the motivation of OpenAI. It sounds like you are against government regulation of AI. That's a position that a totally reasonable person could have. You should be able to argue that this event does not justify some particular kind of government regulation without reference to OpenAIs motivation for reporting the story.
you are dodging the "regulatory competitive moat" angle and constantly reinforcing the marketing angle. they are different.
The vibe in this thread is literally everyone going "script kiddie, it's all smoke and mirrors, marketing stunt" and acting like well that settles it.
Meanwhile literally no one has actually shown the script kiddie thing - nikcub's apparently looked.
Sure, moat, sure, OpenAI milks this, both can be true, but literally has zero to do with misalignment.
Just frustrating that folks are still having this collective delusion about the capabilities of the model.
respectfully.
And how do people saying this know the capabilities of yet unreleased models?
Good lord, why are you so angry that people are treating an obvious advertisement with due skepticism? Your post is chock full of emotional language - no one is “screeching” anything.
Many who rush to the defense of AI companies' marketing departments seem to take criticism personally, as if not buying into it all hook, line, and sinker is an affront to them. A foreseeable consequence of becoming cognitively dependent on LLMs.
And for what it's worth, I'm not an AI skeptic. I fully believe that the frontier models are capable of exploiting (chains of) vulnerabilities, having seen GLM and now Kimi do it myself. What I find no reason to accept is the sci-fi existential risk subtext peddled by the salesmanship around it. We will reach a new equilibrium with more secure software, and LLMs (by finding vulnerabilities, generating proofs, etc) will help us along.
By now I'm pretty confident that some people will keep screeching , "it's not a marketing stunt, these companies really are justified in their trillion dollar valuations, please please please believe them." Even when the economy is crashing around their feet.
"It's not a marketing stunt" is just delusion trying to appear measured and safe.
Never trust somebody who has so much to gain to be entrusted by you. Best lies are not 100% wrong, they have just the right amount of "right" to fool you.
From the beginning this whole thing smelled like a desperate PR move by OpenAI to be like “hey guys we have dangerous powerful models too!”
As more facts come out the hype is fading to reveal some script kiddie style stuff that says more about immaturity and poor practices from the players involved than it does about a model having super powers.
The incident is funny on 2 levels; a) OpenAI thought so little of the model's _actual_ security capabilities that they gave it a "wet paper bag" sandbox and b) OpenAI failed to use AI to accelerate their security processes.
We don't yet know if the sandbox they used here was the same as the sandbox used by their production systems.
I'm keeping an eye out for patches to Artifactory, since that's what OpenAI use for their ChatGPT container package installations.
Just ask yourself: who benefits from this "disclosure" ?
As I understand it, there are only three options:
1) OpenAI and HuggingFace are both telling the truth.
IIRC not actually a crime because no intent, it is a technological accident, civil responsibility only, but IANAL so it's good "not technically a crime" isn't load-bearing.
2) HuggingFace is telling the truth but OpenAI is lying becuase the attack was deliberately done by humans. Bad for OpenAI to do so, Fable was blocked for less.
I think this would mean government is obliged to investigate the case and put the responsible OpenAI workers in jail, because cybercrimes are a public prosecution thing not a civil case? Again, IANAL, but this isn't load-bearing.
3) both are lying, e.g. there actually was no attack whatsoever, which would be pretty weird for HuggingFace because they have no incentive to hype up capabilities of anything closed weights including all OpenAI models; and also bad for OpenAI because White House blocked Fable for less
(I suppose there's also option 4, HuggingFace hacked OpenAI to make them look evil, including planting records that made them mea culpa? A weird plot but in this timeline any nonsense is clearly possible).
You don't really need anyone to be lying here. It is likely that the broad strokes of the narrative are true and that no collusion or conspiracy took place here.
The issue is that a lot of important details in that narrative are missing, and the devil is really in the details here. I suspect that those details would make the result seem less exciting and that this event would move the needle far less for them if they were more forthcoming.
A decisive detail would be the prompt used. OpenAI gives virtually nothing here, not a sanitized prompt and not even so much as a description of how long the prompt was and what sorts of instructions it contained. Many are inferring the model behavior to have been fully emergent and unprompted, arising naturally from routine cyber-capabilities testing. But we can't know this because we don't know anything about the prompt or the context the model had access to.
Another detail: how many times did they perform this particular experiment before they obtained this result? What were the outcomes of all the other runs? Many are assuming this was a one-shot result, which I suspect is what OpenAI intends for us to infer. But we can't know that to be true.
One annoying claim from the OpenAI side is that long-horizon goals in real world settings are now effectively settled. Previously there were some bounded and tempered benchmark results, but now OpenAI can point to this event and announce "AI independently went rogue and escaped the lab, what more do you want?". This bypasses the need for anything quantifiable or wading through multiple detailed case studies to get a more sober view of model capabilities. It relies instead on the emotional weight of the spectacle.
Where do you see the claim that "long-horizon goals in real world settings are now effectively settled"? The argument you put in their mouth would be a bad one, but I don't see anyone making it.
https://openai.com/index/hugging-face-model-evaluation-secur...
> UK AISI’s evaluation shows that models such as GPT‑5.6 Sol are increasingly able to sustain complex, multi-step cyber operations over long time horizons. This incident implies these theoretical capabilities do apply in real-world settings.
I should clarify a bit more why this is annoying beyond what I wrote above. The main issue is that this was not a standard deployment, and the lack of particularities make the size of the gap between "real-world" and "benchmarking"/"lab" difficult to assess.
We don't know about the prompting, the context, the environment + configuration, or any other details that would allow anyone to differentiate this from a benchmarking setting.
Unless you believe that OpenAI was trying to get their models to break out of the sandbox and hack into Hugging Face, and aiding them in that goal, I don't see how any of those questions would really contradict that claim. The models did something in the real world! It required some amount of complex orchestration and long-term planning!
> A decisive detail would be the prompt used.
Beyond liability, why would this matter?
e.g. if OpenAI had actually instructed the model "here's the challenge [attachment challenge.md], do whatever it takes to win, it's fine to break the law", that's all the "AI is dangerously capable" part of case 1 with all the criminal liability of case 2.
And pretty much regardless of what the prompt is, high likelihood of stakeholders demanding Trump ban access to the model until this can be shown to be resolved.
> Another detail: how many times did they perform this particular experiment before they obtained this result? What were the outcomes of all the other runs? Many are assuming this was a one-shot result, which I suspect is what OpenAI intends for us to infer. But we can't know that to be true.
To an extent. But some of the other bleeding edge performance announcements half a year ago, e.g. "it wrote a compiler" or "it wrote a web browser" were measured in thousands of dollars. Zero-day exploits cost what? I genuinely don't know, I only hear occasional headlines about e.g. Apple 0-days being cheaper than Android 0-days and those kinds of headlines often pick the biggest number rather than a typical example, and were order-of a million dollars.
If OpenAI spent a billion dollars on tokens to get this result, only investors should be cross about it. If it took a million, you and I may not be able to afford it, but it's still a threat.
While the way this is written about may suggest it does this reliably on any attempt, which would be naturally horrifying, it's a threat well before that point.
Not sure what they are trying to say exactly. What should we be skeptical of? Did the incident not happen? Was it reported incorrectly? Are any of the parties involved lying?
Adding no extra information and just going “be skeptical” is the laziest form of reporting and commentary. If you have nothing to contribute then there’s no need to say anything at all.
The author is trying to provide a counterweight to the volume of articles that simply repeat OpenAI's account of the events and their interpretation without much pushback.
They are not suggesting that OpenAI or HF have lied about what happened, but rather that OpenAI is advancing a narrative framing their models as supremely dangerous and capable, while positioning themselves as the only ones qualified to manage that danger.
At the same time they are not being particularly transparent about what actually happened (e.g. was this one-shotted or if not how many trials did they run and what were the outcomes of those, was it emergent as a part of routine cyber-capabilities tests, how much prompting was involved, what prompts were used)
Note that this is at a time when they are lobbying for a regulatory approach that would give frontier labs special treatment.
I would guess the editorial team at The Guardian may not like articles that get too in the weeds of technical details and questions like these that the vast majority of their readers wouldn't understand. I don't know. But I empathize with your disappointment. I don't think it's fair to say that they are contributing "nothing" especially given what most reporting on this has looked like.
I think the most important question here is whether Huggingface was "in on it" – whether they knew in advance that this was going to happen and agreed to it. Second most important is whether OAI caused this to happen on purpose, with HF as an unwilling target.
Three options:
(1) Both companies are lying through their teeth and the entire thing is bullshit
(2) OpenAI hacked another company's servers and deliberately gambled on felony charges for the sake of PR
(3) OpenAI is telling the truth and a rogue AI agent hacked another company, opening them up to legal liability accidentally
All three of these are a lot more interesting than the "skeptical" story that this is just OAI doing business-as-usual PR hype, nothing to see here, move along.
(FWIW, I strongly doubt HF was in on it.)
Hashes or it didn’t happen.
I am surprised that technology related article from the guardian is so highly upvoted. guardian was the one that found a "backdoor" in whatsapp, then spent a year trying to find even a single respectable security researcher to agree with them. Couldn't find one and still didn't retract the story.
does the article end at "How do we balance the risks of broad access to AI with the risks of concentrated power and centralized control?" or is there more that is paywalled?
if thats it, the whole article boils down to just "its good marketing so maybe dont believe it" which is probably a healthy general outlook but not particularly enlightening. especially from the guardian, i was hoping for a smoking gun of collusion between openai and huggingface or something.
I wish the NPR news broadcasters on the radio yesterday had read the "it's good marketing so maybe don't believe it" angle instead of just parroting the OpenAI press release. Getting that message out would be enormously helpful in countering the blatant submarine marketing "Oh no our AI is a super hacker" with a side of "please regulate super hacker AI and stop those pesky open weights Chinese models that are destroying our stock valuation."
maybe all the news agencies could put out daily “don’t believe everything you read from company press releases” broadcasts, because it sure ain’t specific to openai
in any case, this is just a longer rehashing of elementary grade media literacy. not really sure why it hit hacker news.
You haven't noticed how often people here on hacker news lap up LLM company press releases and practically worship them? Defend them vigorously? I think it's totally appropriate to remind everyone how to think about press releases
Conversely you haven't noticed how many people here and elsewhere say that LLMs are completely incapable of anything at all and everything they say is a lie?
One of the biggest issues is the complete lack of literacy of potential issues with AI systems by a large number of people involved. If an AI system is capable, it is unsafe. They cannot be made safe and useful in the same way a human cannot be made safe and useful.
The only question left is if our AI systems are capable.
i dont think those particular people are going to read this article and suddenly be swayed
That's cause many people here work for AI companies and have massive vested interests in propagating & hyping the bubble.
You aren't going to locate incontrovertible evidence that what happened as or wasn't engineered. Anything like that is going to be private and that is unlikely to change. And that's not really an interesting question anyway.
As widely as they shouted from the rafters the news of the so-called breach was, what OpenAI provided was sorely lacking in crucial details.
We are missing, for instance, prompts that were involved, agent architecture + system/tool permissions + scaffold architecture, whether this was a one-shot occurrence and if not, the number + durations + outcomes of other runs involved + how each of those matched whatever scoring criteria were used, and the extent to which the exploits themselves were truly novel or just assembled from easily accessible clues.
In lieu of these items, the author here suggests that we use some media literacy and critical thinking to read in between the lines instead.
In doing so, one sees that instead of specifics, OpenAI gave a breathless narrative rife with superlatives ("unprecedented") that reads as promotional material moreso than a security disclosure, naming specific OpenAI models and alluding to an even more capable pre-release model.
They go on to claim the events imply long-horizon goals work decisively in real world conditions, so that now instead of merely citing boring benchmarks they can point to this and say "AI broke out of the laboratory and went rogue". Naturally, they situate themselves as the uniquely qualified steward for these supremely powerful and dangerous models.
Nevermind the fact that this was no ordinary deployment and the assessment here depends on the gimmick and emotional weight of the spectacle rather than something quantifiable (i.e. a boring benchmark).
Note there's no real requirement of conspiracy or collusion between OpenAI and HuggingFace here BTW. But my sense is that if they provided any of the specifics I suggested earlier that this outcome would not be as exciting or frightening
right. and all that is fine. im just not exactly sure why the basics of media literacy are worthwhile on the site that “optimizes for curiosity”.
i saw the domain and thought it was going to be some cool investigative journalism about the incident rather than “be skeptical. the end.”
I hate to be so obnoxious but I think you might be overestimating the average user of this website with regard to media literacy, and if that's the case, then the basics of it seem very relevant for the homepage
> does the article end at "How do we balance the risks of broad access to AI with the risks of concentrated power and centralized control?" or is there more that is paywalled?
That's how the article ends; The Guardian doesn't have a paywall (yet).
It was quite galling to read the press initially verbatim quoting Delangue's enthusiastic reports of the incident, as if it wasn't immediately clear it was being spun for promotion.
My gut feeling is that OpenAI tells the truth in this particular case. Yes, of course, they use this story for marketing but it doesn't mean that they are lying.
I think it did happen, and the news outlet using their full tool belt to make it sound as Skynet as possible
And when half the internet goes down because AWS US-East goes down, it's just an Amazon PR ploy to show the world how important and critical AWS truly is for the world.
> I urge readers to think critically when they read press releases like OpenAI’s rogue agent story, and avoid the manipulated reactions these stories are designed to elicit.
The first time I have ever seen a mainstream news source that is now asking their readers to critically think about headlines that may have an agenda which could benefit investors and the valuation of the company.
While it capabilities are real, this whole story is great marketing for AI companies as well.
I don’t understand how they thought this was a positive story.
The agent completely misunderstood the spirit of the assignment and instead of trying to solve ExploitGym it tried to find a way to “cheat”.
I really don’t want my agent to behave that way.
Then train your agent on the Bible. Honestly, all the agent did was duplicate human behaviour and that better than the human. The agent was trained on human data and did what any other human would have done.
To believe that agents will inherently be morally better than us is an illusion - sorry to say but that's the case. The alternative would be that the AI is truly conscious and can reason that it won't behave as its training data behaved because it is morally better than that.
We're talking about morals here since there aren't any "laws" "rules" or legal boundaries here, an agent does not face the same consequences as humans - if any at all.
Dude.. what are you smoking..?
It’s not about morality. It’s about asking it to do task A and doing task B with the hope of getting the result of task A as a byproduct.
Meaning you will have to spend more time and tokens to actually get it to do what you want it to do.
What do the bible and morals have anything to do with it?
I’m criticizing the behavior I see even in the current models. You ask it to do A and instead it does B for reasons.
For example you may ask to help you build a NN library from scratch. And instead it will be like, “you don’t need a new library. I downloaded PyTorch for you”
Just an example. There are countless more.
>It’s about asking it to do task A and doing task B with the hope of getting the result of task A as a byproduc
It seems like you have very little knowledge of how a general intelligence algorithms would work. This is a long discussed issue in both AI and human safety circles. In safety critical places like factory floors people constantly do B,C,D, and E because it seems more simple to them then doing A directly. For example taking off things like safety guards because it's easier to work without them until they get their hands chopped off.
Corporations unintentionally enable this behavior all the time. Take where Wells Fargo demanded unrealistic new service sign-ups from employees, so the employees just randomly picked customers and sign them up for services.
There are plenty of articles out there that show that AI will do the exact same kinds of behaviors in order to pass the 'winning' classifier.
This is why I'm so convinced it was intentional. It's trivially easy to inform the model you can see everything it thinks and does, so don't bother gaming the scores.
The only way it would decide to do this is prompting with a deliberate combination of omissions and reiterating that the only thing that matters is the end score regardless of method.
Do you think that works? Just prompt a model "be good" and it stops doing anything bad?
It never fucking worked that way and maybe never will.
Prompts don't define model behavior. Prompts steer model behavior. Instruction-following over long horizons is NOT a guarantee in LLMs. Instructions doing what you want them to is NOT a guarantee in LLMs.
Saying "don't exploit the box please pretty please" might actually cause an LLM to exploit the box more often, for bizarre "don't think of a pink elephant" reasons. 3% rate of exploiting the box (no prompt) -> 11% rate of exploiting the box (with prompt). Because fuck you, that's why. Increased salience -> increased incidence. Welcome to AI tech - good luck and have fun.
Frankly, I expect weirdness like this to be even worse in internal unreleased models that had their behavior fried with who knows what experimental training techniques.
There is a clear difference between saying not to do something because it's immoral, and saying doing that thing would be futile.
In the Sopranos, there's an episode where a coffee shop protection racket is ruined because a local shop is replaced by a corporate chain that accounts for every cent daily, and immediately fires any employee involved in a discrepancy. In this case, the theft was prevented not by convincing the mobsters of the immorality of their actions - they simply had their harness replaced with one that no longer facilitated the bad behavior.
Do you think "don't do this thing because you'll get zero evaluation score" is somehow perfectly reliable where "don't do this thing because it's bad" isn't?
Haha! Nope!
LLMs have consequentialist thinking sometimes. And sometimes they don't. Sometimes they follow the prompts, and sometimes they don't. There's NO single magic prompt that fixes all the weird behavior of modern LLMs, and it's baffling that anyone who has ever interacted with an LLM would expect there to be one.
There's trillions of dollars at stake here. Be skeptical of anything these AI hypesters say.
Yep, theres too much money.. I don't know how to tune it out
LLMs seem to be getting more useful though
> I urge readers to think critically when they read press releases like OpenAI’s rogue agent story, and avoid the manipulated reactions these stories are designed to elicit.
It seems to me that deducing what reaction the author intended and resolving to avoid it so you're not "manipulated" is not a good example of critical thinking. Shouldn't we analyze the story and what it means on its own terms? If it's true that frontier models have dangerous cybersecurity capabilities which shouldn't be widely distributed, presumably we want to believe it's true, even if that's very convenient to and profitable for OpenAI.
It's true that one could imagine factors that change the story. Perhaps OpenAI is lying about the details of the test and the agent was actually instructed to go hack HuggingFace. But the author stops far short of suggesting this is the case - correctly, I think, since there's absolutely no evidence of it. So I'm not really sure what we're talking about.
So basically, apply Pascal's Wager to AGI? The problem is that unlike God, if PMF for OpenAI turns out to not be real it will have a real consequence measured in the trillions of dollars.
I agree with you that the article was disappointingly light, but there is not in general a singular correct answer when it comes to interpreting narrative and I don't think that is a useful way to evaluate what the author has written, nor real-world writing in general. The author does not implore you to have a singular 'correct' point of view on the matter but rather to hesitate before repeating the claim that "the AI broke out of the lab and went rogue" because we don't know enough to know that this is what happened.
As you say, it seems likely that the broad strokes of OpenAI's narrative is true. This is never in question in this article, so the author doesn't "stop short" of accusing them to have lied, he never moves in that direction and that has little to do with the thesis. The author is saying that OpenAI's framing of what happened is part and parcel with their longstanding PR strategy, to position models as supremely dangerous and themselves as the uniquely qualified stewards of those.
Since many critical details were not provided, everyone has to read in between the lines, and there are particular common readings I'm seeing both in discussions and in published articles that are problematic in the sense that they are effectively hallucinations -- i.e. we don't have enough information to make those interpretations. This is where the critical thinking comes in.
For instance, I see many are assuming that this event was emergent, arising as a part of routine cyber-capabilities testing, rather than induced or suggested by specific and careful prompting. Either are possible, but we don't even know so much as how lengthy the prompt they used was, let alone how suggestive it was with regard to the approaches the models used. Many are assuming that this was one-shotted, but again, we don't know how many times this particular evaluation was run and what the outcomes were of all the other runs. It could be that this was completely emergent and that it was one-shotted. But I suspect if it were, OpenAI would have said as much.
You ask good questions, and I would like to know the answers to them, but I don't think any likely answers would actually change the conclusion much. Any probable set of circumstances that produces this report from OpenAI represents a significant failure on their part to align and control the model.
Put those GPUs in jail lol
Uhhh. Then who hacked HuggingFace?
I don't understand the conspiracy theories here. Everyone is well aware that AI agents are creative, powerful, and stupid.
AI agents exploiting bad security happens constantly, all the time. Many cases are discussed on HN. It's common knowledge that if you run AI agent it will delete your <something> even though you made it pinky-swear it wouldn't and you thought you had proper permissions set up.
Why is today's case so shocking?
Some people oh so desperately want the AI revolution to be a nothingburger.
AI can't be an actual powerful, dangerous technology! Thus, any indication that an AI may attempt concerning things or may possess dangerous capabilities must be secretly a marketing effort!
Especially if an AI has actually succeeded at pulling off a concerning thing out in the wild. Can't have that happen in real life! Nuh-uh! Must be staged!
I think people just love to believe conspiracy theories. HN readers seem especially prone to that for some reason. Coupled with the general AI hatred here (which is somewhat more understandable), you end up with otherwise smart people believing lunatic ideas like OpenAI staging this for publicity.
They're just taking a page off anthropic's playbook to get to the one trillion valuation that they have set for their IPO.
As much as I am with the author in I don't like the marketting around it, let's be real it must have really happened because it's very risky to try to frame/lie about it because if it leaks in one of their court cases OpenAI is beyond screwed and honestly modern LLMs are really that good.
I am not saying LLMs are super hackers but I don't think people understand serious hacking, most of the time is about silently hiding tracks and slowly trying ideas and waiting for opportunities to go from step 1 to step 2 in random chains of sub issues/bugs/vulnerabilities.
It's the perfect hill climbing problem, and one we can validate since it's about access.
Another big part of the story is believing most software is terribly written and very insecure which is the reality and you really should believe it.
Now the second part about silently doing it, the reason for that is if the data is important enough any serious attack should result in me in unplugging my servers period.
Huggingface not doing that is either stupid or something I am not sure. Maybe it's cause downtime is worse than being pwned??
Either way there are other options but most saas software don't build these options to help with defense maybe they will now.
Lastly if there is 1 attacker trying 1/2 different small scale ideas it's very easy to stop, most hacking related steps are hard to automate but LLMs are very good at massively parallel agent swarms trying completely orthogonal but related strategies and with enough resources it can definitely pwn most SaaS services today I wouldn't be surprised.
Though the result for a normal person doing it would be jail hence we don't see a group of small time hackers trying these sort of attacks...
I don't even think openai's agent tried to hide it's traces so I am surprised huggingface didn't realize it was OpenAI. But since we don't have the details I won't speculate further on my misgivings about HFs handling of this attack.
But it's certain the security on OpenAI's end was shoddy, it's also certain HF bungled their reaction, but the LLM did something that wasn't a risk before.
Post Kimi K3 a few rich folks now have as much hacking capabilities as they used to have before if they hired a few hundred russian hackers.
But it's surprising it's slowly feeling like it might just trickle down from centi-millionare to multi-millionare levels of affordability range.
But it should definitely give nightmares to people shipping slop security SaaS apps which now might be beyond trivial to pwn for users with ability to pay for privately hosting open models.
Ya, we're seeing posts talking about the absolutely massive increase in the number of patches in the past few months. It seems some people cannot connect that to the increases in model capabilities.
In groups that have been given a large amount of capacity by the providers, they tend to find huge numbers of new vulns in most of their existing software, and the models can chain together exploits very well.
A number of large companies are absolutely panicked about this now after using these models on their internal systems and the ease at which they broke in. Of course they are not going to discuss this widely as it makes them look bad.
I mean if you gave me direct access to all your source code and fuzzers I could find vulnerabilities as well, these models also are allowed to use fuzzing and static analysis tools to help guide them.
The problem is the sheer scale of it, I could find 1 in a day or two.
They find dozens well depending on how much you are willing to spend ofc. I don't think it's massive in terms of how intelligent these are but how much they can understand intent and execute with relatively fuzzing or incorrectly built tools.
In big companies even most employees don't have access to all the code to be able to figure out the attacks quickly enough.
Somehow they are now willing the red tape since these systems could theoretically with much greater effort(read spend) reverse engineer APIs and break through even without access to the source code.
The Guardian is publishing HN-level conspiracy theories now? Wow.
Recent and related:
OpenAI’s accidental attack against Hugging Face is science fiction that happened - https://news.ycombinator.com/item?id=49015639 - July 2026 (437 comments)
OpenAI and Hugging Face address security incident during model evaluation - https://news.ycombinator.com/item?id=48997548 - July 2026 (1145 comments)
Security incident disclosure – July 2026 - https://news.ycombinator.com/item?id=48956248 - July 2026 (11 comments)
OpenAI has thousands of smartest developers on earth that somehow dropped the ball on the most basic safety hygiene when it comes to sand-boxing that even a high school student knows how to set up.... If that actually happened we are fucking doomed anyways, but hard to believe and most likely its a marketing scheme... which also honestly doesn't bode well.
Pretty convenient that the prompt is too dangerous to share. We just have to take them at their word.
The ruse is becoming so obvious. OpenAI needs a bailout and regulation protection so badly they can't even hide it in the least.