To understand how we change the incentives that lead to the intelligence curse, we should understand the current social contract, and how it is set to break.

Under the current social contract, powerful actors require people to sustain their power.

This is true for governments. We’ve seen how richest states are those that have the richest, best-educated, most-productive citizens, and that grant those citizens the most extensive freedoms to work and think as they please. This gives citizens bargaining power—if their needs aren’t met, citizens can remove their leaders from power, since after all, the leaders cannot have an economy or a military without the people.

It’s true as well for companies. We’ve seen how organizations that need high-caliber talent have to offer higher salaries and lucrative benefits. If a company fails to uphold this relationship, employees can join other ones or start their own.

It’s also true for the market as a whole. Humans are the consumers, and they buy things that either 1) they want or 2) that assist them in earning more to buy the things they want. Humans are also the producers, which means that market forces incentivize investing in humans. Humans might need to serve the market, but the market serves them back.

As AI replaces the need for humans, this could be upended. The keys to power might run entirely through non-human factors of production like land, capital, resources—and control over AI. Regular people could be economically displaced, and powerful actors could lose their incentive to invest in them.

We’ve been calling this the intelligence curse.

The intelligence curse incentivizes a breakdown in the social contract that is antithetical to democracy and human-oriented capitalism—the systems that have produced more value for mankind than any other in history. At their core, both capitalism and democracy uplift some humans as a reward for them uplifting other humans, and create incentives on institutions to care about people being well-off and capable. Everyone—including the powerful—wants regular people to do better. Both systems explicitly encourage the new to supplant the old, creating a force for progress and social mobility.

If states follow its incentives, the intelligence curse will pull them towards entrenched authoritarianism, ushering in unparalleled concentration of wealth and power and a closing of the door for regular people to get ahead. But will that actually happen?

Is society really that exposed to economic incentives?

The existence of an incentive for society to move in some direction does not mean that it will certainly happen. Cultures, governments, and institutions are also incredibly strong forces with lots of inertia.

However, there are strong historical reasons to think that the pull of incentives, while not absolute, has the foundational influence on social structures in the long run.

We’ve discussed the resource curse, a class of examples much-studied by economists. Countries rich in resources do tend to end up with worse institutions and governance. But the resource curse is far from the only historical comparison.

In Foragers, Farmers, and Fossil Fuels, historian Ian Morris argues that the social structures and the values of societies undergo changes during technological revolutions. Almost all farming societies—unlike the foraging societies before them—tended towards hierarchically-regimented, patriarchal societies¹. During the industrial era, the incentives shifted, and suddenly it was important for a state to have efficient markets, an educated workforce, wealthy consumers, and sufficient freedom to enable its scientists and entrepreneurs.

Growth alone shifts incentives too. It’s also true that the Enlightenment mattered, but our drift towards liberal democracy and unprecedentedly free and empowered humans was greatly boosted by the alignment of these things with material incentives.

As McInnes et. al. note in Anarchy as Architect, states are not free to pick their structure—they must pick structures that are competitive with other states. The competitive requirements change with new technologies. In particular, if a new technology allows some highly competitive social structure to exist, states might be forced to adopt welfare-degrading policies in response.

In addition to the big examples of industrialization and agriculture that Morris discusses, McInnes et. al. give specific examples going back to ancient times. For example, bronzeworking led to centralization of power. Bronze weapons were highly effective but bronze was very scarce. This meant that only a small chariot-riding warrior elite could be equipped with bronze, but this then let them dominate the battlefield. Then, developments in metallurgy made ironworking possible. Now entire armies comprising a significant portion of the population could be armed with iron weapons. The small warrior elite gave way to massed infantry armies. Power decentralized, and the historical record shows that economic inequality fell then too.

Technology is mostly good because it expands human capabilities, and humans prefer to use those capabilities for good. Most technologies look like the first diagram above. But we should work to accelerate the technologies that uplift humans and continue to bind competitiveness and welfare.

The great blessing of our time is that competitiveness is remarkably correlated with what we value—liberal democracy, the rule of law, and human freedom, education, and prosperity. But it is not a rule of nature that this correlation will continue. A liberal democracy at war must make tradeoffs between human welfare and its continued competitiveness—and existence. The fundamental tradeoff is between what we wish to do and what we must do.

Might AI free us from competitive pressures?

Some expect that AI will obsolete such material competitive pressures. There are three arguments given for this:

First, abundance: AI might usher in massive levels of abundance that wash away all other issues. Abundance may weaken the harshness of competitive pressures, and give cushioning such that much can happen to people in terms of inequality, turmoil, and loss of power, while still leaving them alive and materially well-off. This is good.

However, neither competitive pressures nor human greed have any intrinsic stopping point—consider how geopolitical tensions have rocketed up recently despite history’s greatest level of wealth. You should, as common sense tells you, be worried about the long-run stability of any arrangement where you have no power. And even if you would survive in material comfort anyway, there may be much greater and more fulfilling futures available if we break the intelligence curse than if we don’t.

Second, domination: AI might lead to a single actor taking over the world, for example through recursive self-improvement of an AI system creating a godlike superintelligence, or through some company or country achieving a decisive advantage over the rest of the world that lets them impose their will over everyone else.

We expect the AI balance to be more multipolar and arrive more slowly than some of the more aggressive scenarios predict², making this path less feasible. More fundamentally, we are extremely concerned about the massive risk that this strategy entails. If the single actor is corrupt, or if any subsequent transfer of power fails or corruption arises, there is no recourse. Arguing for personal power based on prosocial motives, and then diverting that power to serving your selfish interests, is one of the most prototypical human failures.

Third, coordination: AI might enable radical new coordination technology. We might experience the “Choice Transition”, where aggregate competitive pressures stop driving history, and instead deliberate long-horizon steering by humanity’s collective preferences always has the deciding vote. New types of institutions and new coordination technology might let us steer around competitive traps, without requiring a single centralized actor, much as markets let us plan economic activity without central planning.

Radical levels of beneficial coordination without any power concentration risk or any of the standard failures of central planning would certainly be great, if it were possible. There’s some chance that AI-powered coordination technology eventually takes us further towards this than we can currently imagine, but we do not expect the technology or the institutional readiness and coordination to arrive before the intelligence curse starts to bite. By default, we expect to unlock the labor-replacing impacts of AI before its other transformational impacts. And so, while we are excited about technology for better coordination and institutions—in fact, we propose several ideas for this in the next essay—we feel deeply wary about betting the future of humanity on only that.

Differential technological development

While we’re pessimistic about coordination-based solutions fully solving the problem, and while materialist explanations of society may sound demoralizing, they tell us something very powerful: by building different technologies, we can durably push society in a better direction.

Differential technological development was advocated by Vitalik Buterin in his essay “My techno-optimism”. Like Buterin, we reject the idea that technology is inexorably driving us towards either doom or utopia, and that all we need to do is either slow down or speed up the rate of technological progress.

We can’t decide which technologies are possible, but we can decide which ones we build. And by deciding what to build, we shape the incentives that in turn shape society.

The good outcome

We want to live in a world where:

1. Humans can create economic value for themselves and can disrupt existing elites well after AGI.
2. Everyone has an unprecedentedly high standard of living, both to meet their needs and to keep money flowing in the human economy.
3. No single actor or oligarchy—whether that be governments, companies, or a handful of individuals—monopolizes AGI. By extension, no single actor monopolizes power.
4. Regular people are in control of their destiny. We hold as a self-evident truth that humans should be the masters of their own futures.

To do that, we should build technologies that help people remain economically relevant, that reduce concentration of power risks, that improve our institutions, and that guard against catastrophes.