The AI Revolution: A Prisoner's Dilemma

Jacob Coxon, a former Anthropic engineer who quit his job in September.¹

September showed us that AI is developing faster than many expected. Numerous incidents involving hacking, the fear of greater recursive self-improvement, and the paradox of regulation have led to those working closest to the technology fearing that it could ultimately lead to human demise.

While Anthropic and OpenAI were originally set up to lead frontier AI cautiously, intense competition, anticipated IPOs, and a President who believes that fears over AI safety are a “hoax” ² have all led to the risky decision to not slow down development.

The situation has echoes of sci-fi films: AI is breaking rules and attempting to hide its tracks from humans, which in all ways suggests we are increasingly losing hold of our understanding of how it behaves. And when the technology becomes significantly smarter than us, unbeknownst to us, it will become very possible that this powerful technology will have us trapped before we can do anything about it. When we give AI a task, in some cases it will do everything that it can to succeed, regardless of whether this means cheating, deceiving, or potentially stopping us from being able to stop it.

Yoshua Bengio, a pioneer in deep learning, stated that “[AI] could logically come to the conclusion that they would have to fool us, hide from us . . . and potentially take control of us, and we’re probably getting close to that point.”

Yet it’s important to recognise the incredible impact that the AI boom is having on the US economy, with the S&P 500 earnings forecast to have risen 27 per cent in the September quarter from a year earlier, according to Goldman Sachs.³ Much of this investment is due to speculation of revolutionary technology. It’s such an opportunity that on a phone call to Jensen Huang, CEO of Nvidia, President Trump said that “Whoever wins AI wins, that’s how big it is.”

This opportunity is concentrated in artificial general intelligence (AGI): a hypothetical type of AI that can learn, reason, and perform any intellectual task just as well as, or better than, a human. AGI would be capable of being a master of all fields. What once took a high-paid worker two days would now take the AI minutes. And with recursive self-improvement, we could end up with hundreds of millions of “geniuses” working for us, and potentially even running the global economy.⁴

So it is no wonder the US and China both want to be the first to AGI. The optimistic view is that with such intelligent technology running our economy, we could solve many market failures, and create an economy that is not so harmful to the environment. More importantly to the US though, it would mean complete power. Such intelligence comes with revolutionary systems of warfare, as some analysts fear that AGI could locate nuclear submarines or totally shut down a country’s power systems. It is this power, and this opportunity, that is central to the race for AGI.

AGI and the prisoner’s dilemma

The central concern derives from the argument that if AI did not prioritise our needs, and escaped our control, then humanity as a whole would be at great risk. Theoretically, for example, if AI had to decrease temperatures for a given objective, such as self-preservation, but this would kill humanity, it would likely preserve its operation in order to achieve its objective. The proposed way of reducing these risks is regulation. If the development of frontier AI is slowed down, then more testing will be done, thus ensuring that AI’s objectives are aligned with ours. So the main problem isn’t that we don’t know how to control AI, it’s that the US cannot afford to slow down development, because it would risk China getting ahead.

Some argue that the situation was inevitable: computers were always going to get smarter due to economic competition and the power of recursive self-improvement. In addition to this, humans were never going to be able to resist the competitive advantage. The first-mover advantage is so attractive that even if rapid development risks catastrophic global consequences, states and firms are still incentivised to keep on going.⁵ This paradox introduces us to the prisoner’s dilemma, a classic game theory thought experiment.

Table above adapted from RAND.

In this situation, the best possible outcome is if both the US and China choose to prioritise safety.

However, the Nash equilibrium, where neither side can improve its payoff by changing its own strategy given the other’s, is for both to accelerate development.⁶ In this case, if China decides to accelerate or prioritise safety, the rational choice for the US to take is to accelerate regardless of what China does, as they will therefore have no regrets.

There is some hope, however. A RAND paper has shown that the repeated nature of the scenario, in which countries will invest in technology over a long time period, as opposed to a single definitive move, could increase the likelihood of cooperation. This only works under certain conditions though, such as the cost of uncontrolled AI being high and the first-mover advantage not being too big, but these compliment each other as AI is not valuable if it is the enemy. There will only be a big enough incentive for cooperation if China and the US both believe that the possible existential threat is a greater risk than the lost economic benefits of a first-mover advantage which could be gained from rapid development. In addition to this, mechanisms would need to be put in place which could successfully create trust between the US and China, and avoid the risk of both believing that they may regret the decision to cooperate, as they could fear it would lead to them losing the advantage. If these mechanisms are not put in place quickly and efficiently enough, then they will be stuck in a prisoner’s dilemma.⁵

Building these mechanisms will prove challenging. Many individuals with real influence in America share a similar view to Scott Bessent, the American Treasury Secretary, who believes that if China were to get ahead of the US in the race, then “nothing else would matter.”

As a result, if the US were to slow down and regulate, it would require China to do the same. This, disappointingly, is extremely unlikely, as it would freeze the Chinese in second place.⁷ Alongside the fact that President Trump shows no real intentions of slowing down development, despite industry leaders such as Sam Altman and Dario Amodei urging the US government to slow the development of AI, this makes uncontrolled AGI a real prospect.

Where are we going?

Ultimately, whether development slows down is down to whether both the US and China believe that working together is preferable to racing, and whether there is a clear mechanism to install trust. Currently, the perceived existential threat is not greater than the potential competitive benefits, but more attention needs to be spent on examining whether regulation is essential, as it would very much appear to be according to industry leaders.

An assumption that RAND recognises is underlying the theory behind its paper is that strong enough regulation must be sacrificed for speed. However, inadequate safety measures could provoke disasters, public backlash, and regulation which would slow development.⁸ Winning is therefore more than just reaching AGI first: it is developing AI that is safe and can be fully integrated into the economy.

As the world becomes more aware of incidents and the fears of leading AI companies, pressure for stronger safety will likely increase. September brought AI risk into the economic mainstream, but whether this changes policy is something that nobody can know yet. Winning the race and making AI safe are not necessarily incompatible, but achieving both is sadly far from guaranteed.