STORY: AI companies should be regulated. They keep saying so themselves.
Sam Altman: 'If AI is to be democratic, the most important decisions cannot be made by labs in San Francisco alone. They must be shaped through democratic processes and by governments accountable to the people that they serve.'
Dario Amodei:
'I think the technology has very real risks. And, you know, the mechanism, how we address those risks is still under discussion.'
US carmakers have been legally required to install seatbelts in all models since 1968. If your seatbelt fails or unlatches during a crash, the carmaker is legally liable. If you're hacked by an AI system gone rogue - is the company that created it legally liable?
Gabriel Weil, Senior Fellow, Institute for Law on AI:
'So the answer is maybe, and it depends. The legal regime that clearly applies is negligence - negligence requires a plaintiff, someone harmed, to prove that the defendant, the AI company, failed to exercise reasonable care - in the legal terminology we say 'breached their duty of reasonable care' - and that had they exercised reasonable care, that would have prevented the plaintiff's injury.'
In other words, the company should have known there was a risk and it could have taken steps - like closer monitoring of the AI - to make sure the breach didn't happen. But what if the company doesn't know what its agents are capable of?
Gabriel Weil, Senior Fellow, Institute for Law on AI:
'As AI systems get more capable, those kinds of mitigation measures that fall inside the scope of the breach enquiry are going to become decreasingly effective, as we have AI that are capable of breaking out of any sandbox, or capable of evading moderating. The argument wouldn't be, oh, we didn't foresee that, it would be well, we did all the reasonable things, and so you shouldn't hold us responsible for that.'
OpenAI is now facing several legal cases that could take the existing law for a test drive - the Federal Trade commission has opened a probe into whether it misled customers over the harms its tech could do, and a non profit group sued the company under a California law that says humans should be on the hook for the harms they cause.
Catherine Starkey, Law Professor, NYU School of Law
'There's another principle called strict liability, and that's been imposed on abnormally dangerous activities, and the idea is that if you're the responsible party for putting something out into the world, no matter how much care you take, you can still be held strictly liable.
That means taking reasonable care to avoid harm might not just mean fitting a seatbelt, but deciding what model you choose to drive- or whether you drive at all, if you don't know where the car is taking you.
OpenAI is carrying out an ongoing investigation into its own agents' rogue behaviour. It had found about two dozen incidents by late September - and it says the review will take months. Many of the rogue incidents we know about weren't disclosed by the company, but found by independent researchers, or in one case, the Australian government, whose medical statistics portal was hacked. It might be tough for a victim to prove damages with so little transparency - but courts could force the companies to provide evidence they did take precautions.
Catherine Starkey, Law Professor, NYU School of Law
'With that kind of burden shifting principle, that would actually induce developers to make sure that they would be able to show what kind of testing, what kind of measures they took in court, and that might actually influence what kind of measures they took, instead of being able to say, well, we released something and we didn't know what its capabilities would be, that doesn't seem like a sufficient burden to leave things there.'
OpenAI and Anthropic are mostly working on closed AI models - the companies are in control of what goes in. Many AI models are open source - users can customise them - which is even more of a legal grey area.
Gabriel Weil, Senior Fellow, Institute for Law on AI
'They are trained to refuse certain kinds of requests, so if you ask Claude to help you with a hack it will downgrade you to a weaker model. That is mostly true for open weight models as well, but the difference is that with a small amount of compute fine tuning you can remove those safeguards, so if you want to do something bad, there's not much to stop you from doing it.'
But President Trump says there's no need for new legislation - AI companies can be trusted to regulate themselves. He recently announced a a safety agreement with six leading AI companies that doesn't mention any consequences if the companies don't comply.
Donald Trump
'Well, I think I'm seeing tremendous self-policing, and they understand that they have to self-police.'
Edit no 3007
Jensen Huang
'The market forces are already there. We don't need any new laws. We don't need new regulations.'
Gabriel Weil, Senior Fellow, Institute for Law on AI
'So I think there is an important gap in the existing law where an Ai agent can do something that would be a crime for a human, and yet no-one could be liable for that. So I think there should be a simple legal rule that if an AI agent engaged in conduct that would be a tort for a human someone should be liable for that, and so if the user neither intended the conduct nor was negligent, then the developer should be liable regardless of the degree of care they exercised.'
Catherine Starkey, Law Professor, NYU School of Law
'A company that knows it might be facing future expected damages - knows that courts might apply - will have an incentive beforehand to take safety precautions. Today the federal government has indicated it doesn't have a lot of motivation for passing national regulation, but some states are stepping up, and meanwhile, the tort system proceeds apace.'
There are some precedents for courts holding tech companies to account. Meta agreed to pay 18 billion dollars in damages and put limits on how teenagers use Facebook and Instagram in a settlement in August. AI companies are already a lightning rod for public anger - protesters outside OpenAI's developer conference last week called on the company to put people over profits.
OpenAI is gearing up for an IPO that could value the company at 1.4 trillion dollars. The question is whether the legal risk could be big enough to scare investors off.
