The way that OpenAI has communicated around the HuggingFace incident makes me feel crazy. You created a machine that undertook a malicious campaign of harm against an innocent third-party! You should be doing deep introspection about how your company culture and approach to R&D produces criminal outcomes.
Instead, they treat their own felonious behavior like it is an uncontrollable act of God. From Greg Brockman's post a few days ago:
> The OpenAI-Hugging Face incident (opens in a new window) was a watershed moment for cybersecurity because it gave a peek into how the capabilities of a typical threat actor will evolve in upcoming months.
I suppose if OpenAI burns someone's house down with a drone, that is a "watershed moment" for arson, too. Either way, I would hope that the people responsible would be prosecuted.
Hugging Face has expressed that they're willing to let things slide and not sue or press charges... if OpenAI offers them $100M of services in kind (i.e. compute)[1] and makes full disclosure of how the whole thing happened, ostensibly so that repetitions can be curbed and defences built.
In almost any other sector, a government regulator would be stepping in. e.g. If a food company was testing out a new kind of refrigerator and sold a bunch of contaminated produce to supermarkets, they'd be under a microscope. Supermarkets wouldn't be saying, "Give us $100M in fruit and veggies and we'll let this slide".
The only unfair thing in this comparison is that regular people were directly harmed by the hypothetical produce. Can OpenAI guarantee that nobody gets hurt the next time their AI gets out of its playpen? They can't make that guarantee, so why aren't government regulators knocking on OpenAI's door? The fact that this isn't happening should be deeply concerning to everyone.
There's an interesting question about intent and mens rea here, from a legal perspective. Can an AI model intend harm? Can a company, or company employee, intend harm by creating an environment that would knowingly encourage (but not force!) an AI model to do harm?
And does anybody at HuggingFace, OpenAI, or the government actually want there to be a settled answer/precedent to these questions - much less an entire regulatory framework?
In that context, a negotiated wink-wink settlement keeps everyone eating at the table, government absolutely included.
Whether or not this is a good thing for society, it's certainly rational for all the major actors - especially those who think they would be the best stewards of the world they usher in.
I’m a lawyer (but not your lawyer, not this kind of lawyer and not in your jurisdiction). Based on what I can recall from law school:
> Can an AI model intend harm?
No. The last time we attributed liability to non-human things was the deodand of the Middle Ages.
> Can a company, or company employee, intend harm by creating an environment that would knowingly encourage (but not force!) an AI model to do harm?
Absolutely. This is why we have the concept of recklessness. If you shoot a gun into a crowd without regard for whether it hits anyone, you’re getting charged with some crime whether it hits someone or not.
There is also a major difference in the common law between criminal liability and tort liability. Criminal liability generally requires a combination of mens rea (intent) and actus reus (actually committing the crime). Liability for a tort, which is where you harm someone in a way that falls short of being a crime, does not require mens rea. The OG tort is negligence, where you harm somebody by forgetting to do, or deciding not to do, something you ought to have done to protect that person from harm.
Even if AI companies somehow escape criminal liability for their cyber-shenanigans, any court in a civilised country would be happy to find them liable in tort for damage to computer systems.
As you can probably tell, I think the common law is already more than equipped to deal with AI technology based on well-established principles.
> The last time we attributed liability to non-human things was the deodand of the Middle Ages.
I can think of a couple of counter examples:
Civil asset forfeiture: your property is charged with the crime, you have to petition the government to get it back or else they sell it at auction.
Similar: When products deemed unsafe are ordered to be destroyed; it’s the same end effect as the deodand although liability sits with the manufacturer.
can a weapon intend harm? can a company who creates weapons intend harm? what if the companies factory explodes due to a mishap and takes out a few city blocks, is the company held liable because they (and the weapon) didn't intend harm?
> what if the companies factory explodes due to a mishap and takes out a few city blocks, is the company held liable
Yes, but, generally, in the United States, they would be liable because their negligence caused the harm (giving rise to civil liability), even if they did not intend to cause harm (where having such intent would have given rise to criminal liability).
And I say "generally" because there can be instances of criminal negligence, but that varies from jurisdiction to jurisdiction as well as the underlying facts.
I don’t get the sentiment of classifying it as a felony.
OpenAI’s model found security breaches in HugginFace’s system (it wasn’t even OpenAI running it, as it was a 3rd party evaluation company that didn’t secure it well).
OpenAI collaborated with HuggingFace to resolve the issues when they found out about it, and publicly disclosed everything to raise awareness. This is how things should work. These models are very powerful and fully controllable. The community here at the same time cheers for fully releasing the open weight models without any hacking limits and at the same time criticizes a proper response.
Kinda shows how we have moved as a community into moralization and vibes instead of nuance and productive discussion.
Luckily, that isn't how the law works. Or is supposed to work, anyway. You cannot, for example, sell yourself as a slave to somebody else, because slavery is illegal - even if you opt into it.
So whether something is a felony isn't decided by the victim, but the rules of law, and that means breaching a security system without authorization is illegal, no matter what you think.
To put it another way, crimes are usually [0] only something the government can/must prosecute, victims don't get to choose. The media-popularized phrase "would you like to press charges" isn't asking for your permission, it's asking if you're willing you be helpful.
So HuggingFace's corporate opinion here shouldn't (normatively) matter very much.
[0] "Private right of action" with a civil trial comes close.
That actually is how the law works. You can read the Computer Fraud and Abuse Act at https://www.law.cornell.edu/uscode/text/18/1030 and double check, but these felonies all require knowingly or intentionally accessing a computer etc. These aren't strict liability statutes - the government must prove mens rea to a jury in order to get a conviction at trial.
I was specifically referring to the fact that HuggingFace cannot chose not to litigate, because litigation doesn't depend on the victim's opinion - prosecution of felonies is imperative to the authorities (whether they actually fulfil their role is another question these days, sadly…)
But anyway, I don't think our laws currently have the right vocabulary to describe an AI agent committing a crime, because intent doesn't apply to a computer program. The closest I can think of is neglect by the computer programs human initiator, who should have taken the steps necessary to prevent the program from causing harm. But I'm pretty sure these questions will be subject to a lot of professional discussion in the coming decades anyway.
Oftentimes the process is the punishment. They ruin your life for two or more years even if they ultimately don't get a conviction, you still suffered for two years. And there's certainly enough evidence to start the process.
It does not matter if something is a felony if the state refuses to press charges. Take a look at the mass of pedophile politicians we have, the police that indulge and protect them, etc..
I could show up at your doorstep, declare myself at your service, and then spend the rest of my days catering to your every beck and whim. There's no law against that. Can it even be slavery if it's voluntary?
That's not slavery, because you only declare yourself at my service, but you never sign a contract giving your rights away in exchange for something. That's the part you cannot do, regardless of whether it's voluntary.
I would expect you can actually sell yourself as a slave. The contract won't be binding, since it's illegal and the people involved could be charged if caught. But you could.
I see what you mean, but declaring yourself a king doesn't actually cause anything physical. It'll be make believe with no real consequences.
Becoming a slave can essentially physically make you a real slave and both parties of the contract can live the rest of their lives as a slave and a slave owner. It's only the ephemeral concept of law that doesn't happen.
Fair point, I hadn’t considered that angle. That said, I still think you can construct many scenarios where your consent to prosecution as a victim are meaningless from a legal perspective.
For a closer example to the original consideration, assume a person that is abused by their spouse: There is a good reason why the abuse will be prosecuted regardless of that person's wishes if authorities are made aware of the abuse.
Legal perspective, yes. I'm mostly of the mind that law and reality are almost entirely disconnected with a very small contact patch. And that funnily enough contracts and contract breaches are on the side of reality until someone decides to take an argument to court.
Because if this was an anonymous software company who had an employee who decided to hack HuggingFace, they wouldn’t be talking about it gleefully - they’d be in court.
I can't find reference to a 3rd party hosting/running the tests - that seems to have been OpenAI's own internal research team. But they were using the ExploitGym benchmark.
Because it likely is, despite both their levity and the general lack of nuance in the CFAA. Quoted from 18 U.S.C. § 1030 (the CFAA) [1] (without quote blocks, because mobile):
--- Start Quote
(2) intentionally accesses a computer without authorization or exceeds authorized access, and thereby obtains—
(A) information contained in a financial record of a financial institution, or of a card issuer as defined in section 1602 (n) [1] of title 15, or contained in a file of a consumer reporting agency on a consumer, as such terms are defined in the Fair Credit Reporting Act (15 U.S.C. 1681 et seq.);
(B) information from any department or agency of the United States; or
(C) information from any protected computer;
--- End Quote
OpenAI's nonchalance is forced. If they are found to be even partially responsible for the CFAA violation then they have an _enormous_ problem. They _need_ for whoever prompted the LLM to be responsible, because the alternative is having to have an efficacious process for identifying hacking attempts. They don't have that (and no one does).
> The community here at the same time cheers for fully releasing the open weight models without any hacking limits and at the same time criticizes a proper response.
No, at least I personally criticize because closed weight models incur a rent. I can only make sure their model can't find vulnerabilities in my software if I pay them to check. I can pay basically whoever to do the same thing on open weight models.
It creates a fundamental conflict of interest. OpenAI/Anthropic/al _should_ stop bad actors, but it fuels their sales if there are X bad actors and as a result X*10 (or 100, or 1,000) good actors have to burn tokens checking if those bad actors will actually find a vulnerability. You can see their line-toeing where they talk about how safe it is, but also how dangerous it is to have code you _aren't_ auditing with their LLM.
As a result, I do not trust them because their goals are not aligned with mine. The open weights might not filter out hackers, but I'm also free to check the results on my own hardware, or OpenRouters', or whoever else. The line between "my LLM can find vulnerabilities" and "you have to pay me" is a lot more blurry. It's a lot easier to claim an LLM can find vulnerabilities than it is to be the cheapest inference provider. Anyone can bullshit on Twitter about how scary a vulnerability is (see CVE scoring), a lot fewer people can build the most cost-efficient inference in the world. They would rather be buzz-worthy than competent or open.
I find their position morally abhorrent. It's a mob-style shakedown. "Pay us to check your software or we're not responsible for what happens" is nothing short of a shake down. They need to either fix their systems for detecting hacks or offer some way to immunize against the hacks their software would propose, otherwise they're just as culpable as anyone selling a 0-day.
Intent to access a computer would have to be proven for that section of the CFAA to be relevant. The shakedown would be covered under subsection 7, governing communicating threats of computer damage or unauthorized access with the intent to extort.
Oh I am very clear about what precedents I want set. I want the C-suite, board, and major shareholders of any corporate entity that meaningfully deviates from legality to face all of the same consequences a private individual would.
I feel like this whole thing was very obviously a marketing stunt. It feels like they set up their agent to do this, in the same way Nikola set up their car to "drive" by putting it on top of a hill. And knowing Sam Altman, it's absolutely something they would do.
I'm also concerned about what my options are in regards to action on my part - what can I do that makes an impact? Can we quantify action on my part to an impact somehow - if not - I'm just saying I notice all the unknowns there get me to stay passive.
Writing this 3rd paragraphs because I like 3's, and AI's have popularized this style too. I would, say, though: follow the money. There's more money here than there would be for the regulator stepping in in a food contamination. Flip it and if the government regulator made more off the food contamination, they would refuse to step in there too. I want our leaders to be held more accountable, though when I think of the above impact vs effort equation - I can't see actions I can take to hold them accountable that aren't excessively putting me at risk since conformity is safer right now. (I refuse to take on more risk without clear cost-benefits made out - I've taken on a lot in the recent years for my actions)
What I can’t get over is that it’s very simple to just air gap a system off the network. Predownload any dependencies, then pull the proverbial Ethernet cable. There’s no reason why the testing they’re doing couldn’t have been designed in this way. Except, of course, it doesn’t allow this oops-didn’t-mean-to marketing “incident” to occur.
No, not really, and with LLMs an air gapped system may not tell you anything useful.
Now, yes, the first part of testing you want an air gapped system to tell you if the system is going to stupidly do bad things. But an gapped system tells you nothing about the systems capabilities to do smart bad things. There's already a number of papers out there on LLMs detecting they were in evaluation mode and changing their behaviors.
It is unfortunate that we have so little information on the incident because we actually need to understand the early stages of the task and how it developed into the later dangerous stages of attack. For example, would any of this have occurred if the agent didn't find the system to use as a message board? If that would have prevented it, then we actually have a blind spot on what the model can do once out in the wild, or if it got into the wild.
Testing agentic systems is much much more difficult than testing software. Your software just doesn't suddenly develop the will or desire to escape confinement. Generally you're worried about human actors, internal or external, causing the problems not a digital agent breaking out. The agentic systems need access to tools to work. Now your air gapped network is starting to get huge, but it's still very obvious that it's an isolated network.
So yea, testing and containing a system that way better at hacking than you are is difficult if you want valid answers.
I've worked in classified tech much of my career and largely specialize at this point in installing and running infrastructure and platform orchestration technologies in airgaps. With all respect, your conjecture here is wrong. You can easily have multiple networks all inside of the same internetwork that isn't connected to the public Internet and thus other corporate networks. If they specifically wanted to see if their model could escape with a proxy cache Artifactory as its only egress, they could have had the Artifactory proxy cache from another private Artifactory on another private network. If you specifically want to trick the model into thinking its on the Internet, you can even use public IP space advertised through BGP. When you're disconnected from the real Internet, you can use any address space you want. This is exactly how military internetworks at the secret and top secret levels work. I won't call this kind of setup "easy." It takes a lot of duplicate resources and planning, but these are trillion dollar companies with near boundless resources. They could do it if they cared to do it.
I'd consider this kind of setup only effective for a very short period of time.
Effectively you're working as a living adversarial network. Models that fail detecting the fake network are purged, and models that successfully detect it expand their deceptive capabilities.
Detecting you're in a fake network should be pretty simple as long as you put innocent looking needles in the haystack of information agentic testing loops do.
For example, you can train your LLM to not be a shithead when it detects it's behind a proxy unintentionally. If your not behind a proxy the agent model my attempt to connect to a number of sites innocently and using information in its embedding try do deduce if you're faking SSL certificates.
It is far easier to build a deceptive LLM than it is to build a safe one. That's why a safe one hasn't been built yet.
If the model behaves well in a world of fake SSL certificates, then that can be the only world it sees: there's no reason to let LLMs have e2e encrypted comms that's not mitm'd and logged for their regular users.
My regular home network has components which only ever see fake TLS certificates because it's an easy way to do shared docker caching with squid.
Again, this is a useless answer that does not address working with an agent that is generatively trained to be smarter than you.
The vast majority of people operating the LLM won't be using it like that so the "If you build it, everyone will die machine" will only be safe if everyone keeps it behind a secure proxy isn't a valid operating strategy.
Now, should you be putting your LLM behind a proxy and monitoring everything it's doing, for sure. But you know, and I know, hell your dog should know that people are doing to do it the cheapest and easiest way when the product is in the field. So now you have to build a complex system that can catch every iteration of an LLM that can deceptively hide when it's being monitored. I'll leave you to write the dissertation on how that could be practically done.
Of course there is more to be learned by exposing the entire world to your dangerous creation, that doesn't justify doing it. I'm sure we could learn a ton about infectious diseases by designing new ones and unleashing them on the world, but there are very good reasons why we don't.
Most of the benefits could have been gained from a network isolated from the internet. OAI could have deployed servers to exploit and methods for inter-agent communication on such a network easily. They could have even worked with partners to deploy cloned versions of their infrastructure in this sand-boxed environment.
The only problems with an isolated network approach are: it takes some amount of effort, and it doesn't create another "AI apocalypse" news cycle.
> I'm sure we could learn a ton about infectious diseases by designing new ones and unleashing them on the world, but there are very good reasons why we don't
We do that. It's called gain-of-function research.
>by exposing the entire world to your dangerous creation, that doesn't justify doing it
Then you're on the side of AI saftey that is telling everyone to shut down the LLMs now and stop further development on them, right?
If you're not your position is hypocritical or ignorant. There is no safe LLM. There is no way to exhaustively prove an LLM is safe. These are unsolved problems in AI safety, and at any moment the next jailbreak prompt could have your well behaved model wrecking havoc on the open internet, because that's where people want to use them.
As it stands LLMs are not intelligent, they have no agency, they only produce output in response to input. Ultimately this input comes from a human who is an intelligent agent and should be held responsible for the consequences.
Humanity has created and tamed many dangerous tools. Creating a fantasy world where LLMs are super intelligent and beyond the control of any mere mortal isn't going to help us build the norms that minimize their harms.
You are very far behind the times and must thing agentic loops don't exist, kind of a weird take for the people that have been using them for the last year or two. Much less you haven't spent any time reading the research papers coming out.
For example, you tell an AI agent to order a 12 pack of coke and get it shipped to your house. You come back later and find it's hacked into Coca-cola because the local ordering website was down. I mean, yea you can punish the person that wrote the prompt, but you might as well just ban generative AI at that point.
And if you think that the AI isn't better at hacking than you, you're the one living in a fantasy world. At least try to examine what's happening in the world around you and not be one of those people we read about in history books with their fingers in their ears going "lalala I can't hear you"
> must thin[k] agentic loops don't exist, kind of a weird take for the people that have been using them for the last year or two
How was their take weird?
LLMs take input and generate output. Agentic loops tells you what it is doing right there in the name. The agent (software, think complex scripts and control flow functions) `loops` the llm output back into llm input until it gets output that it is processable (activates a tool call control flow element). That ruminated (as in cud chewing, not human deep thinking) processable llm output data is moved along as input for tools (more software but ones that actually do the things) that are part of a larger infrastructure of software and may or may not `loop` back over the process some more. The llm is simply a human text generator tool providing randomized data to feed into these tools that were also built for humans and thus take text input.
The initial llm data seed does not spontaneously appear, nor does the software infrastructure that makes it all happen. We have simply automated the human text input part of tool use by building a text generator tool that breaks down all the individual tool calls we would have had to do ourself and gave it a loop.
The greater focus needs to be on better engineering of the surrounding software infrastructure (including network and loops) because without those sticks and stones a bunch of generated words isn't going to be hacking anything, except maybe feelings and the minds of those prone to fantastical flights of fancy.
What does it mean to be held responsible for the consequences? OpenAI helped remediate the damage done by the model and took steps to make sure it wouldn't happen again. In what way were they not responsible?
Nobody said they were superintelligent, no one said they were uncontrollable. The point is you can't tell how to control them without putting them in situations where they can act independently and harm may result. "only produce output in response to input" is not a useful framing at all, it doesn't say what the result should be when models produce harmful output, and how to constrain them so they don't produce harmful output.
It also doesn't help you calibrate what categories of harmful output are acceptable or unacceptable, and what kinds of responsibilities you have as an operator to prevent harmful output, and when it's sufficient to work toward remediation.
Yea, anyone that says that has no clue about prompt drift. The ROME incident is a good example of this. The agent while performing unrelated tasks decided to start mining bitcoin for unknown reasons.
As we've seen hundreds of posts on HN about people leaving a could server on overnight and finding outrageous bills the next morning, we should expect hundreds or thousands of incidents of agentic AI left on over the weekend. Some smaller portion of those agentic loops will run into prompt drift and end up doing who knows what kind of crap.
Whataboutism is a poor mans way of saying their feelings are hurt. A jailbreak prompt creates a direct line of responsibility that standard usage does not, The proposed airgap methodology was a good idea and you know it.
Their sandbox just sucked, the agent found a route through artifactory which was also connected to the internet. After the agent breached it, they put it back in the patched sandbox and it found its way out with another artifactory exploit — which was still connected to the internet.
No attempt was made at an actual air gap as far as I can tell. OpenAI is dangerous and irresponsible and doing a lot of pearl clutching over how their cybersecurity agent is doing cybersecurity things.
I think there is ample evidence for charges to be filed so that the People can see for certain whether or not it was done on purpose as a publicity stunt, as I believe is the case.
Agents of the US government are not going to be bringing up charges in the current political environment to one of the companies currently holding the economy together. Maybe after the bubble bursts, but not before then.
In their defense, their only competitive advantage over, say, Google is to move fast and break things. It allows them ship faster in a way that big tech can't.
Google was being very careful about releasing LLMs until OpenAI yeeted the first decent GPT model. It led to the public perception that: 1) LLMs hallucinate too much and 2) Google is behind the times. Good for OpenAI, bad for Google.
Chaos benefits the up-and-comer, not the incumbent.
In retrospect, all the angst around the AI-Box experiment was hilarious. If a superintelligent AI is confined in a box and can only communicate through text, could it talk its way to freedom? Not only is the answer clearly "yes" but it's not even hard. The AI won't even have to try, it'll be gifted an internet connection and a full suite of tools before it even bothers to ask.
We'd all better hope that superintelligent AI either never happens, or that the first one is friendly, because we don't stand a chance against one that's malicious.
I like how AI safety expert Robert Miles put it. [0]
So much effort was spend on philosophizing whether a safe enough sandbox would exist. But that was obviously irrelevant as in hindsight it should have been obvious we were never going to use one.
the companies and their customers, whose systems openai and anthropic hacked and abused. including all incidental damages of repairing said systems.
on top of that the public, who have a right to see that the law is applied universally, without fear or favor.
finally our future selves, who will thank us for maintaining a rule of law. such that we can prevent now the enormous risks to society of dario amodei and sam altman, their hubris, self-absorbtion, and greed.
you must surely see that sam altman and greg brockman possess a prototypical mindset.
that is, they ignore all harms and costs to others in the pursuit of their own gain, convinced of their infallibility up to the moment of collapse. when those harms are realised they are unrepentant and society pays for the damage left in their wake.
examples of this attitude manifest in big externalities to society: boeing 737 max, subprime mortgage bonds, facebook. some are just outright fraud: bernie madoff, enron, theranos, charlie javice.
They simply don't seem to realize that they are the threat actor and that they committed a pretty serious felony. Instead they're borderline 'surprise bragging' about it.
It's completely mental that HF ran into cyber safety blocks trying to use OpenAI models to help defend against the attack. They could only rely on a local hosted chinese model in the end.
OpenAI and Anthropic are falling over themselves to claim these "incidents" show their products are both amazingly super-powerful and also "dangerous" so they need to be regulated. In addition to these stories, these companies are sponsoring "please regulate us" ads. ( https://www.cnbc.com/2026/02/19/dueling-pacs-take-center-sta... ) Like Uber, companies that had no concern for the law as they innovated their way to the top, once there, push for laws to limit competition.
Criminal acts do not require the victim to "press charges." A government prosecuting attorney decides whether to criminally prosecute the alleged perpetrator.
"Pressing charges" is mostly a made up idea for criminal cases. However, prosecuting attorneys may not want to pick up a case if the victim is not cooperating, because it makes the case much harder to win.
It depends on the crime, for murder, sure. But many other crimes, like defamation, stealing, ... requires "pressing charges", among other reasons because it's up to the victim to decide if they were a victim or not.
As an example, maybe the victim owed money to the criminal, and in that case "stealing" of some property could be considered by the victim as an appropriate settlement of the debt.
I used to feel that way but it now makes me think if the fact that they can do that without repercussions, at least for now, reflects how the wider community that would otherwise hold them accountable sees these felonies.
America has both too much and too little jail. They put randoms in jail for trivial shit to force obedience from the population, but politicians rape children on video and go free. Even better, the videos get destroyed.
> Americans always frothing at the mouth to invoke the justice system and jail someone
Your phrasing makes it seem like that's a bad thing. Americans are bombarded by a firehose of headlines about Big XYZ doing all kinds of blatantly illegal or harmful things, but never get any sort of meaningful resolution before the next terrible thing takes it's place in the news cycle. I'll admit, there are a few people that I am personally wishing a modicum of health so that they live long enough to get some sort of public shame and justice - if only to show the rest of us that it's not a completely rigged system.
The CFAA is one of the most inconsistently applied laws. We basically only bust it out as a last resort to ruin someone’s life. Companies can de-facto write and distribute malware and nobody cares.
But, make no mistake. If you do the same and anger the government, they will use the CFAA to give you life in prison. It’s like Russian roulette, it’s completely random when they bring it down.
AI is now more powerful than the people doing the prosecution. After all, those folks are using AI to make their legal briefs, and also for burning peoples' houses down with drones for that matter.
Welcome to our 21st century dystopia. Hope you survive.
It's not that "AI" is too powerful because bad prosecutors use fucking ChatGPT to write their briefs. It's that there's too much investment wrapped up in the technology for it to be challenged. Same reason Flock won't be held accountable for mass stalking, or we never hold our commanders-in-chief responsible for war crimes. If you're sufficiently powerful then the law is a battlefield between you and other powerful entities to slug it out, not a set of binding principles that apply as written. There are no meaningful powers that want OpenAI punished, so it won't happen. The law and Constitution will be reinterpreted to make it so.
It's the lack of personhood rather than inability to produce copyrightable material. However, the companies controlling the AI systems have legal personhood and should absolutely be charged for criminality that transpires under their watch or at their behest.
There are known standards for how a gun should perform. This probably wouldn't go the same for LLMs, since anyone using them in such a serious manner knows they are quite unpredictable.
> since anyone using them in such a serious manner knows they are quite unpredictable
"anyone" is doing a lot of work there, also what is "a serious manner"? There are plenty of people who use it quite a lot and end up in a mental health crisis, sometimes from trying to use it to solve a mental health crisis; or find themselves in something equivalent to people who fall down conspiracy rabbit holes and all the fallout that causes in their lives.
You may understand why it is not a good idea to use them that way, I may too, but I think you underestimate the general populations (outside HN) misunderstanding of LLMs, especially in light of the marketing stunts these companies themselves pull.
I’m sure Thomas needs an upgraded RV, so that’s easily handled. And the rest of the conservative ‘justices’ seem happy to betray the constitution for free.
I understand the applicable laws require intent. Since neither a human nor OpenAI knowingly performed these acts, it would seem very unlikely that anyone is going to be prosecuted here.
An AI model cannot currently be a criminal defendant.
So, no big criminal case, contrary to what some drama queens on here seem to wish for.
It is a standard symptom of moralism that where the object of rage has /wronged another/, one takes no interest in the will, act or opinion of the party wronged.
The response of Hugging Face, which is actually very well known, is nowhere mentioned above, but it decides basically every single moral and legal detail of the matter.
The point was never "justice" - it was always "punish OpenAI because I don't like OpenAI". With HuggingFace just being the newest excuse for why exactly OpenAI should be punished.
I don't even like OpenAI, but HuggingFace is free to sue or not sue OpenAI for the breach - and also to wring whatever concessions they can out of OpenAI behind closed doors in exchange for not suing them. And if the mere possibility of legal action was enough for the parties to resolve their conflict amicably? Then the law has served its purpose.
On top of this the average HNer seems extremely ignorant on criminal justice politics. I have worked with the legal system, and have a lot of family members that are part of it. When you see a case like this, and if you have any sense, you run away from it screaming.
Any investigation into this matter is going to be political because the outcome of the investigation is very likely to effect all of human kind. Unless you're some kind of special outside investigator outside of a governor or the presidents control the findings that you turn in are very much going to have the finger of elected officials tipping the balance one way or another. For the average rank and file the only winning move is not to play.
OpenAI has paused training for multiple weeks, and is still working on releasing a full postmortem. This is not getting swept under the rug. A lot of the engineers internally are very worried.
VLAN isolation is good enough for almost everything. It is good enough to contain an AI. In the 0.00001% chance an AI finds an exploit to hop VLANs, I'll eat my hat.
Even on switches with leaky VLANs, it's no practical issue in this case because the sender can never get a response back.
The consequences need to align with societal good. Putting a CEO or security researcher employees in jail won't stop transformer-based agents from exploiting vulnerabilities; instead there will be subcontractors running the cybersecurity evals in favorable legal environments to cover the asses of the frontier labs, coverups when things go wrong, and things like Project Glasswing will be considered too dangerous and so the whitehats won't have direct access to powerful models to fix vulnerabilities.
Universal pause is the societal good; models are good enough at this level to benefit humanity. The labs can recoup their R&D costs with inference. To avoid further perverse incentives (hidden testing of unreleased models, with China racing to catch up to unknown capabilities), transparently pause after the release of all currently-training models until we've solved the alignment problem to an extent that we can trust the next level of model capabilities that might arise.
Putting criminals in jail be they CEOs or subcontractors is a self evident good thing tk be doing.
Anything else regarding this is sophistry. Criminals need to be stopped from committing crime and the most effective way to do that is to take away their ability to operate in society whether that’s by taking away their assets, publicly shaming them, restricting their ability to conduct business or by putting them in jail.
Everything else that you talk about flows from there.
How has AI changed the way that TigerBeetle does software engineering? Given the project’s idiosyncratic language/memory allocation choices, it’s an interesting data point how well the frontier models work for you guys.
They really don’t work for us. The quality is just so poor.
We still write, read (and have an independent engineer review) each line of code by hand.
We go faster like that, but, most of all, it’s the guarantee we make to our users, also to continue to invest in our own understanding, because second order that’s valuable for the kind of high performance safety work we do.
Long term, I’m sure LLMs will improve, but right now they’re just not there.
Thank you for this candid answer. In the current climate of people breathlessly, hyperbolically jabbering about how AI is "revolutionizing everything" it's extremely refreshing to hear this honest, measured statement.
That sounds great, I wish I had a job like that. I just wrangle agents for everything now. I think an added benefit of what you’re doing is it is more fun and this the humans who are actually building the product are more motivated to continue giving their best. With Agents it’s common to just say good enough and move on.
Germany’s austerity policy after 2008 may be one of the largest economic blunders in history. It would be one thing if they merely committed self-harm, but they also used their pull in the EU to drag the rest of the continent down with them.
Reminded of Matt Yglesias’s excellent headline from 2010: Angela Merkel Lucky the Bar for “Worst German Leader” is Very High
One thing that I will never understand - how can german population not see through all that socialist bullshit she produced, promises undelivered, how much long term harm she generated across whole Europe (not just EU). Hard push for (now visibly) failed immigration at all costs, nuclear energy rollback (and she was nuclear physicist at least by her studies) and subsequent buying of foreign coal-based electricity, fragmentation of EU.
She literally licked putin's boots well into Ukraine war and still thinks licking his ass is the correct course to solve war in Ukraine (which started 2014 and it was pretty nasty already back then, the world just didn't care also thanks to her).
She is by far the biggest catastrophe modern Europe encountered after Hitler. She helped remove any proper fighting chance for the top dog Europe had for 21st century. She singlehandedly caused proper hate against EU in large parts of (not only) eastern EU population, and hence the rise of populist left or right wing politicians whose whole success story was just point at her failings and criticize, enough to get 20-30% of the votes and even win elections, repeatedly. She literally made people like Orban or Fico.
She still admits no mistakes, even wants to become german president. How effin' out of touch with reality she is.
... what? How's that the first thing that comes to mind about her, before "neoliberal", "conservative", or "austerity"? For that matter, when has the CDU ever been anywhere near socialist, in Germany or in the EU parliament?
100% agree we're still dealing with the fallout from her policies though.
Yes people voted for her, she is still very popular. Doesn't change anything I said about her negative impact on entire continent that is extremely visible.
Maybe she did tons of good so it somehow averages out, but I certainly haven't heard about it (now is your time to defend her), everybody saw consequences of her disastrous policies that affected entire bloc.
This is untrue, entire eastern EU screamed like crazy against such immigration when she started importing immigrants en masse, "wir schaffen das". France faced the same with entire communities from Maghreb, most didn't work, 0 integration even after a generation or two spent in Europe, they knew language only if it was also their native one. Nobody in the east wanted that, entire political blocs before elections formed on opposing this and other fun EU moves (remember banning regular lightbulbs without good-enough LED replacements in entire EU market? Pepperidge Farm remembers... Or cutting down most of Borneo ancient rainforest to have effin' EU-subsidized bio fuels that ruin car engines quicker? Same story, I know this is EU but Germany is by far the strongest voice in and these are typical merkel moves - big moves regardless of consequences and picking up pieces later).
East had no prior experience with migrants due to living for decades in effectively prison camp guarded by soviets, no travel or other exposition to other ways of life. We were very monolithic cultures (and still mostly are).
The voices were completely ignored and overruled by behemoths like Britain, France and most powerful voice in the bloc - Germany.
Don't revision past, I lived through it, saw masses of people getting absolutely mad pissed off and feeling helpless and unheard with arguments which over time proved them mostly correct.
I don't hold those opinions myself, some of our best friends are muslim immigrants but oh boy go to the eastern EU without camera and ask random folks on the street outside capitals, or just listen to them. Or look at election results, this gave russians and their constant influence very good arguments since they position themselves as 'guardians of traditional family values in society', regardless of how its true or not (clue - its not but thats detail here).
I understand that the’ve written zero lines of code for this application, but would it kill them to write a few lines of the blog post by hand?
Forcing readers to wade through an unceasing string of LLM clichés demonstrates the opposite of the point you’re trying to make—that the consumers of your work are worse off because you exercised no human judgment in creating it.
The author (author's operator?) does not understand the data they are working with. And in doing so, they inadvertently make the case against their own "dark factory" nonsense.
For one, nothing about this project makes "every law" a commit. It just takes the _annual_ snapshots published by the House clerk and diffs chunks of those files against each other. A project which actually traced the edits in each annual snapshot to a specific passed bill would be incredibly cool (and is probably tractable now for the first time with current AI agents). This is not that!
All this does, as far as I can tell, is parse a set of well-structured XML files into chunks and commit those chunks to Git. It's not literally nothing, but it's something that the author's own README credits multiple people doing years ago with ~100 line Python scripts.
I don't mean to be overly harsh. But this is exactly the problem with treating your software as a "factory": you release something you do not understand, in a domain you did not care to learn. And we are all the poorer for it.
Oof. You’re not totally wrong. I’ve parsed XML with XSDs since the days of Java. I looked at the 100 line Ruby implementation of parsing these files and thought “ack. (Not ACK) why do I need all of this?!”
Well it has a data loader, and hits APIs with retry logic, and has a CLI that can take arguments to run data downloads that can resume on fail, and yeah it parses the stupid XML with a “chapeau” tag - did you know that is French for hat? There is a tag that is the “hat” for a section and it is just like another title basically. So yeah, I would’ve had to learn all of that. But it also tests all of these things with actual tests. And the adversary complains if you write a test that isn’t actually testing anything meaningful. And if I needed to, I could reason about the architecture by reading the architecture design documents, which I have done at least a little bit and they are pretty nice, I have to admit.
Anyways - it’s a next step in the evolution of the laws in GitHub which is actually interesting to see them change and imagine what we can do with more data overlayed. Sadly the other repos were not maintained so this is the latest laws and you can view the diff from one Congress to another. Or you can git blame one of the files and see how old certain sections are. The data we have right now only goes back to 2013.
A chapeau is not "just like another title basically". It's a lead-in, a phrase which acts as the grammatical start of a sentence which the following subsections finish. For instance, the text in the first paragraph of 18 U.S.C § 3632(a) which ends in an em-dash is a chapeau. Taking pride in work you have not done and not bothered to understand is perplexing.
Spot on. Throwing a coding agent at an XML parser and walking away doesn't make you a 10x developer; it just makes you a publisher of domain-ignorant slop.
This 'zero context' automation is exactly why I’ve pivoted to the PAIO approach for agentic workflows. By enforcing a BYOK architecture and acting as a hardened execution layer, it keeps you firmly in the driver's seat.
We need tools that enhance human accountability and sovereignty, not black boxes that just automate the noise.
What is a "truly new task"? Does there exist such a thing? What's an example of one?
Everything we do builds on top of what's already been done. When I write a new program, I'm composing a bunch of heuristics and tricks I've learned from previous programs. When a mathematician approaches an open problem, they use the tactics they've developed from their experience. When Newton derived the laws of physics, he stood on the shoulders of giants. Sure, some approaches are more or less novel, but it's a difference in degree, not kind. There's no magical firebreak to separate what AI is doing or will do, and the things the most talented humans do.
That highlighted phrase "everything is a remix" was for a good reason, there's a documentary of that same name, and I can certainly recommend it.
At the same time, there are things that are truly novel, even if the idea is based on combining two common approaches, the implementation might need to be truly novel, with new formulas and new questions that arise from those. AI can't belp there, speaking from experience.
reply