There are probably few things more personal than grief.
So, when Dolly Parton’s sister asked people to stop posting AI-generated tributes following the singer’s death, it was difficult not to wonder how we had arrived here.
On 7 September, Stella Parton pleaded with people to stop posting what she described as AI-generated “garbage” about her sister. The family, she said, had already been subjected to enough insensitivity following Dolly’s death. It was a remarkably human reaction to a remarkably unhuman problem.
The technology did not know Dolly Parton. It did not know her family. It did not understand grief, respect or where the boundary between tribute and intrusion might lie. It simply did what it was asked to do, and that, in a nutshell, is becoming the problem with AI.
Not necessarily what it can do but what happens when we give it the ability to do things we have not properly thought through.
The warnings are getting louder
Dolly Parton’s AI tributes are hardly an existential threat to humanity. But they are a useful reminder that technology can create consequences before society has decided whether those consequences are acceptable.
On the same day as the Dolly story, OpenAI chief scientist Jakub Pachocki warned that nobody was prepared for the consequences of a continued rapid rise in machine intelligence. He called for “extreme caution” and said more intervention might be needed to ensure humans remain in control of the future.
Two days later, on 9 September, the temperature rose considerably.
Evan Hubinger, a senior safety researcher at Anthropic, said he believed there was a greater than 10% chance that AI could kill all humans within the next decade.
Importantly, he was not suggesting today’s AI models are about to wipe out humanity. He said the risk from existing models was low. His concern was what could happen if AI systems become capable of improving themselves to the point where they pose an existential risk.
Ten per cent is an extraordinary number, and it’s an extraordinary claim, so how seriously should we take it?
That is a reasonable question. These are expert judgements about a deeply uncertain future, not scientific predictions with a known probability. Other experts are considerably less alarmed about the prospect of AI-driven extinction.
But there is another reason the warnings are difficult to dismiss. They are increasingly coming from people who actually build the technology..
The problem isn’t intelligence. It’s control.
The debate around AI often gets reduced to a question of whether machines will eventually become more intelligent than humans.
That may not be the most useful question. The more important one is whether humans remain in control.
A highly capable AI system that answers questions is one thing. A highly capable system that can write and execute code, access systems, make decisions, replicate tasks and pursue an objective with increasing autonomy is something else entirely.
On 9 September, AI safety concerns were being voiced not only by researchers but by politicians and former industry insiders. Anthropic’s Hubinger warned about the possibility of future systems becoming sufficiently capable to create an existential risk, while former Anthropic researcher Jacob Coxon resigned amid concerns that leading AI companies were racing towards increasingly powerful, potentially self-improving systems without adequate safety measures.
The concern isn’t that today’s chatbot has suddenly developed a secret plan. It is that the direction of travel may eventually produce systems capable of taking actions that their human creators cannot reliably predict, understand or stop.
And that presents regulators with a rather awkward problem.
How do you regulate something when the fundamental risk is that you might eventually lose control of it?
Westminster starts to worry
On 9 September, while Hubinger was voicing those concerns, Labour MP Alex Sobel introduced the Artificial Superintelligence Security Bill into Parliament. The Bill has been drawn up by ControlAI, a non-profit organisation that warns against the “extinction risk” posed by superintelligent AI systems.
It is important to be clear about what this is – and what it isn’t. It is a private member’s bill, not new legislation. Private members’ bills rarely make it onto the statute book without government backing.
But the proposal itself is remarkable. Its stated purpose is to prohibit the development, deployment and operation of artificial superintelligence systems, while establishing monitoring and control powers around such systems.
In other words, this isn’t a proposal to put a warning label on ChatGPT, it’s an attempt to draw a line around a form of AI that does not yet exist.
The bill defines the problem in terms of systems capable of effectively replacing and outperforming humans across tasks. Its proponents argue that such systems could improve themselves, replicate themselves and resist attempts to shut them down.
That is quite a leap from where AI regulation began and also illustrates the central difficulty.
You are trying to write legislation today for a technology whose capabilities tomorrow are not yet known.
Meanwhile, AI is already in healthcare
If superintelligence sounds like a problem for some hypothetical future, healthcare brings the argument firmly back into the present.
And one day later, on 10 September, the UK’s Medicines and Healthcare products Regulatory Agency published recommendations calling for new approaches to the regulation of AI in healthcare.
The National Commission into the Regulation of AI in Healthcare has recommended staged authorisation for new AI models, similar to “L-plates” for learner drivers. New systems could be deployed under close supervision and tight safeguards before being granted fuller authorisation.
It also recommends continuous real-world monitoring throughout the life of an AI-enabled medical device. That is significant because conventional regulation tends to work on the assumption that something can be tested, approved and then monitored.
AI doesn’t necessarily behave like that.
Its performance can change after deployment. Models can be updated. Their environments change. They can encounter circumstances that were not present during testing.
The MHRA’s commission has therefore been looking at something closer to lifecycle regulation: don’t simply approve the technology once and walk away. Keep watching it.
The commission gathered evidence from more than 12,000 people, including patients, clinicians, healthcare leaders, developers and industry. Its findings point towards a fairly sensible public expectation: AI can be used, but there must be strong safety standards, meaningful human oversight and transparency about when it is being used in someone’s care.
And there is the phrase that keeps appearing throughout this debate: human oversight.
Because ultimately that is what regulation is trying to preserve.
The UK doesn’t have an AI Act
So where does that leave the UK?
Unlike the EU, which has developed a comprehensive AI Act, the UK has not created one overarching piece of legislation governing artificial intelligence as a technology.
Instead, it has deliberately taken a pro-innovation, sector-specific approach.
Existing laws and regulators deal with AI according to where and how it is being used. Data protection law may apply to personal information. Consumer law may apply to misleading AI-generated content. Product and professional liability can apply where AI causes harm. Sector regulators can impose their own requirements.
It is a pragmatic approach, and for relatively narrow applications of AI, it makes sense, but it starts to become less comfortable when AI stops fitting neatly inside one particular sector. What happens when an autonomous AI system moves between finance, cybersecurity, healthcare and critical infrastructure? Which regulator owns the problem, who is responsible when an AI system makes a decision nobody anticipated, and who has the authority to stop it?
The law may already be behind the technology
These aren’t simple theoretical questions. Pinsent Masons reported that existing English law can already make businesses and individuals liable for harm caused by AI, even where the harm wasn’t deliberately caused.
That could include employers being liable for AI use by employees, professionals being held to standards of reasonable skill and care when using AI, and circumstances in which product liability rules could apply regardless of fault.
So, businesses shouldn’t assume they can simply point at the AI when something goes wrong.
“The AI did it” isn’t going to be a particularly useful legal defence.
And that is perhaps one of the more important messages to come out of all this for business. Regulation doesn’t have to arrive in the form of a shiny new AI Bill before organisations have responsibilities, they already do. The question is whether those responsibilities are sufficient for what comes next.
And what comes next?
This is where things become much harder.
In a recent report on the matter, WIRED described UK lawmakers as “freaking out” over what it called AI’s “summer of chaos“, pointing to growing concern about autonomous AI agents and the possibility of superintelligence.
The language may be dramatic but the underlying issue isn’t. We are moving from AI that generates information towards AI that can take action.
That distinction matters enormously. A system that writes a report for you is one thing. A system that decides what needs doing, writes the code to do it, accesses the systems required to execute it and then evaluates the result is something else entirely. The more autonomy we give AI, the more important human control becomes and the harder it is to rely solely on traditional, sector-by-sector regulation.
There is another awkward reality. AI doesn’t respect borders. A UK law can regulate companies operating in Britain, impose responsibilities on UK organisations and give powers to UK regulators. It cannot, however, determine what an American, Chinese or other international company develops elsewhere. That is why the debate is already moving beyond Westminster. The Artificial Superintelligence Security Bill calls for the UK government to seek international agreement aimed at securing a global ban on the development, deployment and operation of superintelligent AI, while the Guardian has reported growing calls for international regulation and even a multinational treaty covering AI safety.
Technology moves globally. Regulation moves nationally. And, for the moment at least, the technology is moving faster.
The dam is starting to crack
Look at what has happened in just four days.
On the 7th, Dolly Parton’s family was asking people to stop using AI to manufacture tributes to her, while OpenAI’s chief scientist was warning that nobody was prepared for the consequences of rapidly increasing machine intelligence.
On 9 September, an Anthropic researcher put a greater than 10% probability on future AI potentially killing all humans, while a UK MP introduced legislation aimed at preventing the development of artificial superintelligence.
And on the 10th September, the UK’s medicines regulator said new approaches to AI regulation in healthcare are needed, including continuous monitoring and staged approval.
That is quite a sequence. It is also why the argument about whether AI needs regulation is beginning to feel rather dated. The argument now is about what regulation should look like, how quickly it needs to arrive and whether it can keep pace with the technology it is intended to control. Because writing a law is relatively easy but writing one that still works when the technology has changed is considerably harder.
And events over the past few days have only strengthened that argument. Over the weekend (12/13 September), a report from the Joint Committee on Human Rights warned that the UK is currently unprepared for the potentially serious consequences of AI, while concerns around privacy and the ability to challenge AI-driven decisions are growing. At the same time, Anthropic chief executive Dario Amodei called for the industry to slow the pace of AI development so that safety measures can catch up, a call subsequently backed by OpenAI chief executive Sam Altman and Elon Musk. When people building the technology are themselves arguing that its development may need to be slowed, the question of how, and how quickly, governments respond becomes rather harder to ignore.
So, will the AI Bill work?
The UK will almost certainly need more specific legislation around AI. But it doesn’t necessarily need to regulate everything in the same way. A medical AI system that helps identify cancer is not the same problem as an autonomous AI agent operating across a corporate network. Neither is the same as a theoretical superintelligence capable of outperforming humans across almost every task.
The challenge is finding the point at which existing regulation stops being enough. Move too quickly and regulation could restrict useful innovation, particularly in areas such as healthcare, scientific research and cybersecurity. Move too slowly and society could find itself dealing with the consequences after the event. And take years to write rules that are too specific, and they may already be out of date by the time they come into force.
That may be the defining challenge of AI regulation. Not deciding whether AI is good or bad, or whether we should embrace it or fear it, but deciding how to maintain meaningful human control while the technology becomes increasingly capable of operating without us.
The warnings may prove exaggerated and the 10% figure may turn out to be wildly wrong. Superintelligence may arrive much later than its most enthusiastic proponents predict or it may never arrive at all. But we don’t need certainty before asking sensible questions about safety. We didn’t wait for every possible consequence of aviation, pharmaceuticals, nuclear power or financial markets to become certain before creating rules around them.
AI is different in one crucial respect: the thing we are trying to regulate is changing while we are still working out how to regulate it.
And that leaves us with the question that matters most.
The AI Bill is coming, but will it work?
Image: Publicity shot of American singer Dolly Parton, 1977 RCA Records, Public domain, via Wikimedia Commons
If this caught your interest, you may also like our earlier piece: Madonna, the Pope and the AI Reckoning