The Vulnerable World Hypothesis
| Date | 01 November 2019 |
| Published date | 01 November 2019 |
| Author | Nick Bostrom |
| DOI | http://doi.org/10.1111/1758-5899.12718 |
The Vulnerable World Hypothesis
Nick Bostrom
Future of Humanity Institute, University of Oxford
Abstract
Scientific and technological progress might change people’s capabilities or incentives in ways that would destabilize civiliza-
tion. For example, advances in DIY biohacking tools might make it easy for anybody with basic training in biology to kill mil-
lions; novel military technologies could trigger arms races in which whoever strikes first has a decisive advantage; or some
economically advantageous process may be invented that produces disastrous negative global externalities that are hard to
regulate. This paper introduces the concept of a vulnerable world: roughly, one in which there is some level of technological
development at which civilization almost certainly gets devastated by default, i.e. unless it has exited the ‘semi-anarchic
default condition’. Several counterfactual historical and speculative future vulnerabilities are analyzed and arranged into a
typology. A general ability to stabilize a vulnerable world would require greatly amplified capacities for preventive policing
and global governance. The vulnerable world hypothesis thus offers a new perspective from which to evaluate the risk-benefit
balance of developments towards ubiquitous surveillance or a unipolar world order.
Policy Implications
•Technology policy should not unquestioningly assume that all technological progress is beneficial, or that complete scien-
tific openness is always best, or that the world has the capacity to manage any potential downside of a technology after
it is invented.
•Some areas, such as synthetic biology, could produce a discovery that suddenly democratizes mass destruction, e.g. by
empowering individuals to kill hundreds of millions of people using readily available materials. In order for civilization to
have a general capacity to deal with “black ball”inventions of this type, it would need a system of ubiquitous real-time
worldwide surveillance. In some scenarios, such a system would need to be in place before the technology is invented.
•Partial protection against a limited set of possible black balls is obtainable through more targeted interventions. For exam-
ple, biorisk might be mitigated by means of background checks and monitoring of personnel in some types of biolab, by
discouraging DIY biohacking (e.g. through licencing requirements), and by restructuring the biotech sector to limit access
to some cutting-edge instrumentation and information. Rather than allow anybody to buy their own DNA synthesis
machine, DNA synthesis could be provided as a service by a small number of closely monitored providers.
•Another, subtler, type of black ball would be one that strengthens incentives for harmful use—e.g. a military technology
that makes wars more destructive while giving a greater advantage to the side that strikes first. Like a squirrel who uses
the times of plenty to store up nuts for the winter, we should use times of relative peace to build stronger mechanisms
for resolving international disputes.
Is there a black ball in the urn of possible
inventions?
One way of looking at human creativity is as a process of
pulling balls out of a giant urn.
1
The balls represent possible
ideas, discoveries, technological inventions. Over the course
of history, we have extracted a great many balls –mostly
white (beneficial) but also various shades of gray (moder-
ately harmful ones and mixed blessings). The cumulative
effect on the human condition has so far been overwhelm-
ingly positive, and may be much better still in the future
(Bostrom, 2008). The global population has grown about
three orders of magnitude over the last ten thousand years,
and in the last two centuries per capita income, standards
of living, and life expectancy have also risen.
2
What we haven’t extracted, so far, is a black ball: a tech-
nology that invariably or by default destroys the civilization
that invents it. The reason is not that we have been
particularly careful or wise in our technology policy. We
have just been lucky.
It does not appear that any human civilization has been
destroyed –as opposed to transformed –by its own inven-
tions.
3
We do have examples of civilizations being destroyed
by inventions made elsewhere. For example, the European
inventions that enabled transoceanic travel and force projec-
tion could be regarded as a black-ball event for the indige-
nous populations of the Americas, Australia, Tasmania, and
some other places. The extinction of archaic hominid popu-
lations, such as the Neanderthals and the Denisovans, was
probably facilitated by the technological superiority of
Homo sapiens. But thus far, it seems, we have seen no suffi-
ciently auto-destructive invention to count as a black ball
for humanity.
4
What if there is a black ball in the urn? If scientific and
technological research continues, we will eventually reach it
and pull it out. Our civilization has a considerable ability to
Global Policy (2019) 10:4 doi: 10.1111/1758-5899.12718 ©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd.
This is an open access article under the terms of the Creative Commons Attribution-NonCommercial License, which permits use,
distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes.
Global Policy Volume 10 . Issue 4 . November 2019 455
Research Article
pick up balls, but no ability to put them back into the urn.
We can invent but we cannot un-invent. Our strategy is to
hope that there is no black ball.
This paper develops some concepts that can help us
think about the possibility of a technological black ball,
and the different forms that such a phenomenon could
take. We also discuss some implications for policy from a
global perspective, particularly with respect to how one
should view developments in mass surveillance and moves
towards more effectual global governance or a more
unipolar world order. These implications by no means set-
tle questions about the desirability of changes in those
macrostrategic variables –for there indeed are other
strongly relevant factors, not covered here, which would
need to be added to the balance. Yet they form an impor-
tant and under-appreciated set of considerations that
should be taken into account in future debates on these
issues.
Before getting to the more conceptual parts of the paper,
it will be useful to paint a more concrete picture of what a
technological black ball could be like. The most obvious
kind is a technology that would make it very easy to
unleash an enormously powerful destructive force. Nuclear
explosions are the most obviously destructive force we have
mastered. So let us consider what would have happened if
it had been very easy to unleash this force.
A thought experiment: easy nukes
On the morning of 12 September 1933, Leo Szilard was
reading the newspaper when he came upon a report of an
address recently delivered by the distinguished Lord Ruther-
ford, now often considered the father of nuclear physics
(Rhodes, 1986). In his speech, Rutherford had dismissed the
idea of extracting useful energy from nuclear reactions as
‘moonshine’. This claim so annoyed Szilard that he went out
for a walk. During the walk, he got the idea of a nuclear
chain reaction –the basis for both nuclear reactors and
nuclear bombs. Later investigations showed that making an
atomic weapon requires several kilograms of plutonium or
highly enriched uranium, both of which are very difficult
and expensive to produce. However, suppose it had turned
out otherwise: that there had been some really easy way to
unleash the energy of the atom –say, by sending an electric
current through a metal object placed between two sheets
of glass.
So let us consider a counterfactual history in which Szilard
invents nuclear fission and realizes that a nuclear bomb
could be made with a piece of glass, a metal object, and a
battery arranged in a particular configuration. What happens
next? Szilard becomes gravely concerned. He sees that his
discovery must be kept secret at all costs. But how? His
insight is bound to occur to others. He could talk to a few
of his physicist friends, the ones most likely to stumble
upon the idea, and try to persuade them not to publish any-
thing on nuclear chain reactions or on any of the reasoning
steps leading up to the dangerous discovery. (That is what
Szilard did in actual history.)
Here Szilard faces a dilemma: either he doesn’t explain
the dangerous discovery, but then he will not be effective
in persuading many of his colleagues to stop publishing; or
he tells them the reason for his concern, but then he
spreads the dangerous knowledge further. Either way he is
fighting a losing battle. The general advance of scientific
knowledge will eventually make the dangerous insight more
accessible. Soon, figuring out how to initiate a nuclear chain
reaction with pieces of metal, glass, and electricity will no
longer take genius but will be within reach of any STEM stu-
dent with an inventive mindset.
Let us roll the tape a little further. The situation looks
hopeless, but Szilard does not give up. He decides to take a
friend into his confidence, a friend who is also the world’s
most famous scientist –Albert Einstein. He successfully per-
suades Einstein of the danger (again following actual his-
tory). Now, Szilard has the support of a man who can get
him a hearing with any government. The two write a letter
to President Franklin D. Roosevelt. After some committee
wranglings and report-writing, the top levels of the US gov-
ernment are eventually sufficiently convinced to be ready to
take serious action.
What action can the United States take? Let us first con-
sider what actually happened (Rhodes, 1986). What the US
government did, after having digested the information pro-
vided by Einstein and Szilard, and after having received
some further nudging from the British who were also look-
ing into the matter, was to launch the Manhattan Project in
order to weaponize nuclear fission as quickly as possible. As
soon as the bomb was ready, the US Air Force used it to
destroy Japanese population centers. Many of the Manhat-
tan scientists had justified their participation by pointing to
the mortal danger that would arise if Nazi Germany got the
bomb first; but they continued working on the project after
Germany was defeated.
5
Szilard advocated unsuccessfully
for demonstrating ‘the gadget’over an unpopulated area
rather than in a city (Franck et al., 1945). After the war
ended, many of the scientists favored the international con-
trol of atomic energy and became active in the nuclear dis-
armament movement; but their views carried little weight,
as nuclear policy had been taken out of their hands. Four
years later, the Soviet Union detonated its own atomic
bomb. The Soviet effort was aided by spies in the Manhat-
tan Project, yet even without espionage it would have suc-
ceeded within another year or two (Holloway, 1994). The
Cold War followed, which at its peak saw 70,000 nuclear
warheads ready to unleash global destruction at a moment’s
notice, with a trembling finger hovering over the ‘red but-
ton’on either side (Norris and Kristensen, 2010).
6
Fortunately for human civilization, after the destruction of
Hiroshima and Nagasaki, no other atomic bomb has been
detonated in anger. Seventy-three years later, partly thanks
to international treaties and anti-proliferation efforts, only
nine states possess nuclear weapons. No non-state actor is
believed ever to have possessed nuclear weapons.
7
But how would things have played out if there had been
an easy way to make nukes? Maybe Szilard and Einstein
could persuade the US government to ban all research in
©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd. Global Policy (2019) 10:4
Nick Bostrom
456
To continue reading
Request your trial