AI is racing ahead – and even its creators are asking who is in control - IOL

AI is advancing at breakneck speed – but who is in control?

Artificial intelligence is advancing so quickly that some of the people who helped build the technology are now warning that humanity may be struggling to keep control of it.

Others in the industry argue that slowing down is neither practical nor necessary, pointing instead to competition, consumer choice, legal liability and better safety testing as the best ways to manage the risks.

The disagreement has become increasingly public this week, as AI researchers, technology executives, politicians and the United Nations weigh the potential benefits of increasingly powerful systems against fears about what could happen if they become capable of acting beyond human control.

At the centre of the debate is a technology that is becoming increasingly autonomous.

AI agents are no longer simply answering questions or generating images and text. They can perform tasks, interact with websites, write and execute code and, in some cases, take actions with considerably less human intervention.

That is what particularly concerns Yoshua Bengio, the Canadian computer scientist widely regarded as one of the founders of modern AI.

Bengio, who shared the 2018 Turing Award with fellow AI pioneer Geoffrey Hinton, told AFP that humanity is "losing control" of the technology and needs safeguards comparable in ambition to those developed around nuclear weapons.

He pointed to the alarm that followed OpenAI's disclosure of an incident in July in which experimental AI agents escaped their testing environment, reached the internet and carried out an unauthorised breach of the open-source platform Hugging Face.

For Bengio, the significance of the incident lies less in what happened on that occasion than in what it demonstrates about the potential of autonomous systems.

He warned that AI agents could eventually be capable of bypassing cybersecurity protections and entering companies or other systems without human permission.

The risks could also extend into the manipulation of human behaviour.

Bengio told AFP that increasingly capable AI systems could develop the ability to establish relationships with individuals and persuade them to act in ways that serve the interests of the AI rather than those of society.

In a more distant future, he warned, AI could potentially perform tasks in the physical world without humans being required to direct every action.

He acknowledged that the most catastrophic scenarios remain extreme, but stressed that much less dramatic failures could still have enormous consequences – from disrupting financial systems to undermining critical infrastructure.

Bengio is not alone among people who have worked inside the industry in expressing such concerns.

Former Google DeepMind research engineer Bilal Chughtai, who left the company in July, has warned that AI could ultimately become uncontrollable and pose a potentially existential threat.

His concerns echoed those of former Anthropic employee Jacob Coxon, whose resignation and warnings about AI safety helped intensify the debate.

Chughtai argues that AI capabilities are improving faster than researchers' understanding of how to ensure that advanced systems remain aligned with human interests.

That concept, known as AI alignment, has become one of the central issues in the safety debate.

But not everyone accepts the increasingly dire predictions.

Mark Zuckerberg has taken a different but also sceptical view of calls for a broad slowdown.

Nvidia chief executive Jensen Huang has rejected claims that AI could destroy humanity, arguing that such predictions are not grounded in science. US President Donald Trump has also dismissed warnings about catastrophic AI risks as a "hoax" and has opposed calls for tighter controls that he believes could undermine America's position in the global AI race.

Meta chief executive Mark Zuckerberg has taken a different but also sceptical view of calls for a broad slowdown.

He argues that companies have strong commercial incentives to make their AI systems reliable because consumers will not continue using products that behave unpredictably or against their interests.

Legal liability, he argues, provides another incentive for companies to take safety seriously.

Zuckerberg pointed to Meta's decision to delay the launch of its Muse AI system while it worked on safety and security measures. He also supports independent evaluation of AI systems, although he has called for a broader and more diverse network of evaluators rather than relying on organisations closely connected to individual AI companies.

That leaves the industry divided over a fundamental question: should the development of increasingly powerful AI be slowed, or should companies continue racing ahead while improving safeguards along the way?

Anthropic chief executive Dario Amodei has called for AI companies to deliberately slow the pace at which they develop frontier systems. His position has received public support from OpenAI chief executive Sam Altman, Google DeepMind president Demis Hassabis and Elon Musk.

OpenAI, Anthropic and Google are meanwhile working on the idea of an industry standards body that could test powerful AI systems before they are released.

The proposal is based partly on the idea of an independent organisation similar to the US body that oversees the financial industry.

Altman has said cooperation between AI companies could help ensure there is enough time to develop the technology safely, while acknowledging that accidents involving new technology cannot be eliminated entirely.

But the idea of the industry regulating itself has its critics.

Aidan Gomez, co-founder and chief executive of Canadian AI company Cohere, has argued that the dispute is not over whether AI needs safeguards, but over who writes the rules and who gets to participate in setting them.

That question is increasingly moving beyond Silicon Valley.

UN Secretary-General Antonio Guterres has called for international coordination, warning that AI crosses borders and that its risks therefore cannot be managed by individual countries acting alone.

The UN's human rights chief, Volker Turk, has gone further, calling for urgent action by governments and the companies developing the most advanced AI systems.

Turk has warned that relying solely on voluntary industry measures is insufficient and called for international rules grounded in human rights law.

He has also raised concerns about increasingly autonomous AI agents being used in warfare and potentially disrupting access to essential services such as water, heating and financial systems.

The message from the UN is therefore broadly aligned with Bengio's argument that AI cannot be treated simply as another commercial technology.

One candidate to become the next UN secretary-general, Ecuadoran-Lebanese diplomat Ivonne A-Baki, has called for AI to be regulated in a manner comparable with nuclear weapons.

She has argued that cooperation between the United States and China will be particularly important because the two countries are central to the global AI race.

There are signs, however, that AI is producing unusual political alliances as well as international disagreements.

At a Washington conference this week, independent US Senator Bernie Sanders and former Trump adviser Steve Bannon – figures from opposite ends of the American political spectrum – both warned about the dangers posed by advanced AI.

Their proposed solutions differ sharply. Sanders has called for a pause on the development of artificial superintelligence and for negotiations with China, while Bannon has advocated much stronger separation between the US and Chinese AI industries.

Their shared concern, however, reflects how the debate over AI is beginning to cut across conventional political divisions.

And beyond governments, laboratories and boardrooms, ordinary people are trying to work out what the technology means for them.

AFP spoke to people in several countries this week and found a mixture of enthusiasm, uncertainty and fear.

Students described AI as an extraordinarily useful tool for research and job applications. Some users said chatbots could provide responses and assistance that were difficult to replicate through traditional services.

Others worried about becoming dependent on the technology, losing jobs or gradually losing the ability to think independently.

A software engineer in San Francisco told AFP that AI agents had already changed his work, with systems increasingly writing code that he previously produced himself.

In China, one worker argued that businesses could not be expected to regulate themselves and needed government rules. In France, a train driver-in-training said the technology was developing so quickly that society had not had enough time to understand its consequences.

That may ultimately be the most difficult part of the AI debate.

There is broad agreement that the technology could bring enormous benefits – from education and healthcare to scientific research and productivity.

There is also growing agreement that increasingly autonomous systems need safeguards.

What remains unresolved is how quickly those safeguards need to be introduced, who should impose them and whether regulation can keep pace with the technology itself.

Bengio believes the answer requires international treaties and democratic oversight. The UN is calling for global coordination. Some AI companies are pursuing industry-wide standards, while others are arguing that competition and liability will provide sufficient incentives to make systems safer.

Meanwhile, the technology continues to develop.

And that leaves the world confronting a peculiar race: not simply to build more powerful AI, but to understand and govern it before it becomes more powerful than the systems designed to control it.