Vitalik: Humanity stuck between 'naive and naive squared' choice in the ASI transition

Ethereum co-founder Vitalik Buterin argues that a lot of the public argument over advanced AI comes from both sides holding assumptions they never actually share. 

Buterin’s kill switch proposal for all AI applications has been met with criticism as the tech community engages in yet another debate regarding how fast AI will progress and what that means for the workforce. 

What is driving the AI debate? 

In a post on X, Ethereum co-founder Vitalik Buterin said the clash between supporters of the “AI 2040” scenario and its critics comes down to how fast and how significant AI progress will be. 

The AI 2040 framing assumes superintelligence of some kind will appear by 2040 unless strong measures stop it. Meanwhile, the critics believe that AI 2040 supporters are underestimating the capacity of human coordination and threatening freedom, but they do not see superintelligence itself as a power concentration risk.

Buterin admitted he does not know which scenario is closer to reality. “If I was confident that (present-day-style) AI is normal technology, I would be in the detractor camp. If I was confident that superintelligence is coming in 2030 by default, I would be closer to the AI 2040 camp,” he wrote. 

But the cofounder remains open to slowing or pausing AI development if risks become significant.

The debate pulled in AI researcher Yann LeCun, author Daniel Jeffries, and policy analyst Adam Thierer. 

Yann LeCun, Meta’s (NASDAQ: META) chief AI scientist, argues that AI safety is fundamentally an engineering problem that can be solved through careful iterative design, much like jet engines were made reliable. 

LeCun pointed out that “it took 50 years” to make aircraft truly safe, and that the fear of AI is premature when we are yet to create a system capable of human-level intelligence. He has consistently argued that large language models are limited “autocomplete machines” that lack reasoning and causal understanding. 

Harry Hawk, posting as @hhawk, said he aligns with Yann LeCun and believes future AI systems would be engineered for safety like aircraft are. He also said he does not believe AI and robots will do everything, leaving no work or jobs.

Buterin replied that that perspective denied the existence of an “AI so powerful that AI alone can perform any task,” which he calls ASI.

How can powerful AI systems be controlled?

Buterin suggested a “plan A,” which proposes a wide-reaching rule that forces everyone to be open about what they are building, plus an emergency off-switch that can slow down or stop large AI training if things get dangerous.

He added that “naive well-meaning intellectuals” who think they can pick and choose which AI uses are okay and which are not will push back against this plan. 

Romeo Dean, who prompted part of Buterin’s thread, called the approach “pretty reasonable” but said its triggers would arrive too late under his worldview. He added that he does not grasp the “massive downsides” critics attach to plan A.

Buterin admitted there is no perfect solution. “I see zero plans for how to deal with an ASI transition that are not naive,” he wrote. “Perhaps humanity is stuck with a choice between naive and naive squared.”

The AI 2040: Plan A report came from former OpenAI employee Daniel Kokotajlo’s AI Futures Project. 

The report says the US and China should work together to push back superintelligence until 2040. Both countries would have to share all their research openly. It also includes a system based on nuclear war logic, where both sides can destroy each other’s computing power if needed. They call this “mutually assured compute destruction.”

Richard Ngo, an AI researcher, said the report is too worried about AI arriving soon. He also said it does not think enough about how much political trouble AI could cause inside each country.

Are open source models the solution?

Running underneath the whole debate is the status of open source models. LeCun wrote on July 9 that AI’s biggest risk is the “concentration of power” in a few dominant companies. He also wrote that “the only solution to AI sovereignty is open source foundation models.” His post drew more than 2,900 likes and over 430 reposts.

Author Daniel Jeffries, writing the same day, said open source models underpin American technology and warned against “short-sighted safetyists and hawks” seeking to restrict it. 

Policy analyst Adam Thierer, a senior fellow at the R Street Institute and author of a prominent House AI Task Force report submission, has warned that US AI governance is at a “critical crossroads.” 

He pointed out that Congress currently employs a messy, random, and secretive process for reviewing AI. He warned that if this informal system grows and progress is blocked behind special approvals, it will destroy open source AI. 

Instead of heavy rules, Thierer proposes a “permissionless innovation” approach, where people will be allowed to build and release AI freely. He suggests that existing laws should be used to punish harm when necessary, and also supports things like testing zones for new AI, requiring some models to stay open, and putting more money into AI research. 

If you're reading this, you’re already ahead. Stay there with our newsletter.