Two Years of AI Politics: Past, Present, and Future

Despite early success, the situation has worsened, and it’s probably going to get even worse.

In this article I’ll walk you through the major events in AI politics, policy, and governance of the last two years, interspersed with some of my own takes, where we are now, where we’re heading, and what change is needed. A lot has occurred in two years, and I will no doubt have missed things, but here is my attempt.

2023: Awakening

General purpose AI systems approach human-level, AI as a threat to humanity enters the public consciousness, and consensus is built on the risks of AI and the need to address them.

March

We can pinpoint the moment where AI politics really began as around the middle of March 2023. You could probably push it back to 2022 (e.g. with the CHIPS Act), but this is when it entered the public consciousness.

March saw OpenAI’s launch of GPT-4 on the 14th, a significant advance on their GPT-3.5-turbo model that they launched via ChatGPT only 4 months earlier. GPT-4 was multi-modal, able to take both text and images as an input, and scored 86.4% on the MMLU — at the time, a useful benchmark.

Plotting a straight line on state-of-the-art MMLU scores would project us roughly to this level. Indeed, in a never-published forecasting project, I used the same method and predicted GPT-4’s capabilities quite accurately. Even so, the leap from GPT-3.5 shocked and surprised many, including myself, when I first used it.

For many, including myself, GPT-4 did feel approximately human level. Now having used it for longer, its limitations on reasoning are more evident.

This was shortly followed by The Future of Life Institute’s open letter, a week later, which called for all AI labs to immediately pause training AI systems more powerful than GPT-4 for at least 6 months, also specifying that “If such a pause cannot be enacted quickly, governments should step in and institute a moratorium”. The reasoning being that AI systems are rapidly approaching human-level competency, will soon surpass us, and we currently have no way to ensure that we don’t lose control of our civilization, along with a number of other risks we don’t know how to mitigate.

Notably, this call received tens of thousands of signatures, including by top experts in the field such as Yoshua Bengio, Stuart Russell, and John J Hopfield, along with Elon Musk — and made top news headlines in major outlets.

Some have criticized the specific policy asks of this call. For example, one potential flaw is that due to the cost of the compute curve, it arguably wasn’t really feasible to train AI systems significantly more powerful than GPT-4 for a period longer than 6 months anyway. OpenAI has probably only in the last few months of 2024 trained what would be a GPT-5 scale compute AI model (so, ~66x the amount of training compute spent on GPT-4).

On the other hand, in hindsight there was, and very likely still is, vast potential for algorithmic improvements to enable the training of systems significantly more powerful than GPT-4 with a similar amounts of compute.

I think this open letter served as a very important risk awareness-raising and consensus-building moment, I think it’s good that it happened, and I signed it.

April

The UK founded the Frontier AI Taskforce, with an initial budget of £100 million (ca. $125 million USD), which in November evolved into the UK AI Safety Institute, which is tasked with evaluating and ensuring the safety of the most advanced AI systems.

May

The momentum of FLI’s letter continued, with the Center for AI Safety organizing a short joint statement that read: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” In some ways this could be seen as a slight softening of FLI’s message, since there are no concrete policy asks. However this is offset by the strong consensus that was built on mitigating the risk of extinction from AI, with the statement signed by hundreds of top AI experts, and even the CEOs of the leading AI companies themselves.

This statement was then echoed by world leaders such as then-prime minister of the UK Rishi Sunak, and EU Commission President Ursula von der Leyen. Similar acknowledgements of the existential risk posed by AI to us have been made by other world leaders since.

August

At the 2023 BRICS summit, Chinese President Xi Jinping made some interesting comments as part of his speech, which received very little coverage but which I nevertheless think are notable and worth highlighting here:

BRICS countries have agreed to launch the AI Study Group of BRICS Institute of Future Networks at an early date. We need to enable the Study Group to play its full role, further expand cooperation on AI, and step up information exchange and technological cooperation. We need to jointly fend off risks, and develop AI governance frameworks and standards with broad-based consensus, so as to make AI technologies more secure, reliable, controllable and equitable.

Often a narrative is painted that China would never cooperate on AI risks, and would simply go as fast as possible, but this seems to be without much evidence. In fact, as early as 2018, Xi Jinping said “It is necessary to strengthen the analysis and prevention of potential risks in the development of artificial intelligence, safeguard people's interests and national security, and ensure that artificial intelligence is safe, reliable, and controllable.” [Source (Chinese)]

October

Ahead of the first AI Safety Summit in November, there were a couple more open letters published:

• One by IDAIS, in which top western and Chinese AI scientists called for coordinated global action on AI safety research and governance, recommending the defining of clear red lines on AI development.

• Another letter, led by myself, and signed by hundreds of experts, which called for governments to actively respond to the catastrophic risks of AI. It called for an international AI treaty framework to be developed and agreed, proposing several policy measures, including global compute limits and a global CERN for AI Safety, covered in the MailOnline.

Both of these open letters demonstrated the potential for international consensus-building on global AI governance to address the grave risks posed to us by AI.

Since then, and throughout 2024, there has been a large number of open letters pushing in the same direction.

On the 30th, President Biden signed an AI executive order, which among other measures, introduced reporting requirements on models trained above certain compute thresholds (10^26 FLOP, or 10^23 FLOP for bio-sequence data).

November

On the 1st of November, the US established its own AI Safety Intitute.

Early November saw the world’s first AI Safety Summit, hosted in the UK, where world leaders, AI experts, and leaders from industry came together to discuss AI Safety, with countries agreeing the Bletchley Declaration.

In the Bletchley Declaration, countries recognized the potential from AI for catastrophic harm, and committed to work together to address these risks. Notably, it was signed by all major AI powers, including the United States, China, the United Kingdom, and France.

On the 17th, OpenAI’s non-profit board attempted to oust OpenAI’s CEO Sam Altman, after they lost confidence in him. While they had the legal right to do it, and I think they were justified in trying it and we would be in a better place if they had succeeded, it seems that ultimately they probably didn’t really know what they were getting into when they pulled the trigger, and the attempt failed catastrophically. As a result, Ilya Sutskever, Tasha McCauley, and Helen Toner lost their places on the board. It appears that OpenAI is since under full control of a confluence of Sam Altman and Microsoft.

Later in the month, a struggle began over the EU AI Act, which was making its way through the EU legislative process. The EU AI Act eventually did pass, and placed some requirements on the developers of the most powerful AI systems, but big tech companies fought hard to try to exempt themselves from this. They ultimately failed.

2024: Confusion

One of the biggest stories of 2024 was the constant stream of resignations from OpenAI in the aftermath of November 2023’s board crisis, with key figures such as Ilya Sutskever leaving the company, and half of its AI safety researchers quitting.

The other was the push by some, including Leopold Aschenbrenner, OpenAI CEO Sam Altman, and Anthropic CEO Dario Amodei of a strategy where the US races to superintelligence in a mad dash to secure world domination.

March

The EU AI Act, after much conflict over whether foundation models would be covered, was approved by the EU parliament.

June

Leopold Aschenbrenner published his “Situational Awareness” manifesto. Briefly, it makes the case that artificial superintelligence could be a small number of years away from being developed, and that the US should engage in a Manhattan Project style race to superintelligence, in order for the US to obtain a decisive strategic advantage over rivals, particularly China, and effectively secure full control over the future of the world (and universe) indefinitely.

Aschenbrenner seems not to have seriously considered cooperative solutions to the risks that the development of superintelligence presents, which I consider to be a major flaw of the essay series. The essay series appears to be motivated more by strict ideological faith, showing naivety to the realities of geopolitics, while being overly optimistic on the ability to rapidly build *safe* superintelligence.

That is to say:

• Cooperation between the US and China on managing AI risks seems both from obvious assumptions (neither want to bring about human extinction), and from evidence (Bletchley Declaration, and many other statements and moves by leaders on both sides) to be eminently possible.

• It is far from obvious that if China judged that the US was X number of months from developing artificial superintelligence as part of the suggested strategy that China would not put traditional weapons of mass destruction on the table, though other means of prevention and deterrence are more numerous and I assume would likely be tried first.

• We currently have no way to ensure on a technical level that superintelligence can be built safely (that is to say, that it does not result in our own extinction, or a number of other comparably bad outcomes). It also seems highly unlikely that this problem would be solved in time, and when you are in a racing situation, as proposed by Aschenbrenner, the incentive is always to move as fast as possible, deprioritizing safety in the process.

July

The White House announced that they had secured voluntary commitments to manage AI risks from Amazon, Anthropic, Google, Inflection, Meta, Microsoft, and OpenAI. These included commitments to internal and external security testing of their AI systems before release, and information sharing on managing AI risks.

Sam Altman wrote an article in the Washington Post echoing Leopold Aschenbrenner’s call for the US, or US-led coalition, to race to superintelligence to dominate the world.

September

Ivanka Trump tweeted out Situational Awareness. This clearly and publicly demonstrates the influence this essay series has had on elite opinion.

A Beijing AI Safety Institute was established.

The first AI treaty was agreed, signed by the US, UK, EU, and others, covering human rights, democracy, and rule of law.

OpenAI launched o1-preview, in my opinion a vast improvement on GPT-4, which is capable of usefully performing reasoning, trading off additional inference compute for increased performance.

In September, and the months leading up, there was an intense battle between AI safety advocates and big tech companies over California’s SB-1047 bill, which would have made leading AI companies liable if their technology caused a catastrophe.

The bill was not perfect, proposing a fixed compute threshold above which AI systems would be covered. This is problematic, as compute is only a proxy for capabilities, and over time algorithmic improvements mean that you can squeeze equivalent amounts of capabilities out of lower and lower amounts of compute. There is likely some fairly fixed critical danger threshold of capabilities, above which systems are inherently potentially dangerous in the hands of e.g. biothreat actors, and so a fixed compute threshold means that in perhaps a small number of years potentially dangerous AI systems would not be covered by the legislation. A better approach, in my opinion, would be to task the regulator with lowering the compute thresholds over time to account for algorithmic improvements.

Nevertheless, the bill seemed to be a good step in the right direction, and would have set an important precedent on legislating to mitigate the catastrophic risks of AI development.

SB-1047 faced significant lobbying efforts from big tech companies and venture capital, including by OpenAI, Meta, Anthropic, Y Combinator, and Andreessen Horowitz, with Anthropic changing their position later on after the bill was watered down. The bill was supported by a broad coalition of AI experts, civil society, artists, and public figures. Ultimately, Governor Gavin Newsom vetoed the bill.

October

ControlAI published a concrete policy plan for humanity to survive AI and flourish “A Narrow Path”. I worked on this as part of the team, mainly on the design of the international AI governance framework that is proposed — I think it’s a good plan.

Anthropic CEO Dario Amodei published his manifesto “Machines of Loving Grace”. Notable was the public shift towards advocating that, as Aschenbrenner and Altman do, a US-led coalition of democracies race to superintelligence to control the future.

Max Tegmark wrote a critical article in response to this, “The Hopium Wars: the AGI Entente Delusion”, where he identified Amodei and Aschenbrenner as proposing a suicide race.

December

OpenAI launched o1, having previously launched o1-preview in September, and announced o3, its next model. o3 shows tremendous gains on the ARC-AGI-1 benchmark, and other benchmarks. I haven’t got to use it yet, and it seems likely to be launched in January, but it looks as though it could be another phase-change in AI capabilities, similar to the launch of GPT-4 back in ‘23.

The benchmarking and communications surrounding the announcement have received criticism from some, including from Gary Marcus. I haven’t spent a long time digging into the details yet, but currently I’m basically persuaded that OpenAI probably did do a bad science. On the other hand, numbers continue to go up, a lot, and this seems like a compelling argument in favor of o3 still being a significantly more powerful AI system than e.g. March 2023 GPT-4.

2025: Racing?

We are in a bad position. Although a broad scientific and political consensus that AI is potentially a very dangerous technology has been obtained, the fact remains: the narrative that the US should race to superintelligence is growing in strength — I expect that to continue.

Furthermore, as the AI industry continues to grow, I expect that ever greater amounts of cash will be dumped into influencing politics towards an accelerationist tendency, and those advocating to slow down and develop AI in a way that doesn’t risk our own extinction will be increasingly out-spent, and uncompetitive. This is a strong reason to push on policy now, or rather yesterday.

Notably, it seems as though Silicon Valley will have an unprecedented influence on the incoming Trump administration. Some of these actors are openly accelerationists, others like Jacob Helberg (a close ally of Sam Altman), have publicly called for a dangerous AGI Manhattan Project — and Helberg will be serving in the administration.

In mitigation, others, such as Elon Musk, are much more reasonable, with Musk having consistently expressed concern about the existential risks of AI over many years. I recently attended a conference where I met dozens of people in this milieu, and my main general learning was that on the whole these kinds of people seem far less accelerationist, and far more concerned about the risks of AI and its intersection with geopolitics than I had anticipated.

Nevertheless, the racing narrative is gaining traction, it has the heads of both OpenAI and Anthropic advocating for it, and will have political actuators inside the Trump administration. Once the US is locked into an AGI Manhattan Project, it could be very difficult to climb down from it, leaving us stuck in a suicide race.

Of course, Trump could be just the man to prevent this disaster. He is a negotiator and deal-maker, and a US-China deal on managing the risks of AI would probably literally be the biggest deal in the history of deals, and of the future too.

What next?

The clock is ticking on our survival. AI development is rapidly advancing, we have no technical way to ensure that superhuman AI systems do not cause catastrophic consequences. Meanwhile, the politics of the issue continue to move in the wrong direction.

While we have acknowledgement from scientists and political leaders of the risks of AI, this isn’t enough. We need to make addressing the risks of AI a top policy priority for every government with relevant AI development in their jurisdictions, but in particular the United States and China.

We need to combat the racing narrative, and avoid locking ourselves into a suicide race. In practice, this means facilitating US-China cooperation on AI risk and associated geopolitical risk.

We should massively grow the field of AI policy and governance. This is not just a problem for tech dudes, there is a vast untapped pool of talent that have spent their lives dedicated to the study of international relations, history, society, politics, and many other fields which may be crucial to getting it right. Everyone has a stake here, and there are plenty that could meaningfully contribute. There are a lot of smart tech dudes, and they are making important contributions, but there is also a notably large contingent that scarcely understand the politics of their own countries, let alone those of others and the interactions between them, and may in some cases as a result be prone to conjuring up nightmarish fever-dreams like Situational Awareness. So, help is needed!

I consider that part of the reason we are in this position now, and at such a late hour, is because pre-2023 those concerned about the risks of AI largely kept quiet and to themselves. It seems that they thought they could avoid politics, and perhaps quietly solve the problem on a technical level. In my judgment, time has run out for technical solutions, and if we are to get them, we need policy to buy more time.

As a goal: I still stand behind the AITreaty.org open letter I led in October 2023, that is to say, we should move on getting an international AI treaty agreed, with broad international support, that addresses the catastrophic risks of AI. I still consider that the policy recommendations it makes are appropriate, most prominently:

• Global Compute Thresholds: Internationally upheld thresholds on the amount of compute used to train any given AI model, with a procedure to lower these over time to account for algorithmic improvements.

• CERN for AI Safety: A collaborative AI safety laboratory akin to CERN for pooling resources, expertise, and knowledge in the service of AI safety, and acting as a cooperative platform for safe AI development and safety research.

• Compliance Commission: An international commission responsible for monitoring treaty compliance.

There are steps that need to be taken along the way to this, and I have a lot more to say about what should be done, but this article is already longer than I had intended, so I will save it for another time.