A Weekend in November
On the afternoon of Friday, 17 November 2023, the board of a nonprofit called OpenAI fired its chief executive. On the face of it, a management change. Within ninety-six hours it had become one of the strangest corporate dramas in the recent history of technology, a live argument fought out in real institutions, by real people, on a timescale of days.
The weekend was stranger than a power struggle, and the strangeness has an explanation. The argument over artificial intelligence is not an argument about technology. It is a quarrel between rival eschatologies, one promising salvation through acceleration, the other warning of apocalypse through the very same acceleration, and both camps inherited their story from a religious shape neither any longer recognises. The OpenAI crisis was a theological schism conducted through press releases. And the sanest position available refuses both prophecies, treating the future not as a verdict to be awaited but as a problem to be solved.
OpenAI was not, by November 2023, one technology company among many. In roughly a year it had put a new kind of tool into the hands of hundreds of millions of ordinary people, students writing essays, programmers debugging code, small business owners drafting letters they would otherwise have paid for, and it had done so at a speed that left governments, universities, and rivals visibly scrambling. ChatGPT, released with modest expectations in November 2022 as what the company called a research preview, reached a hundred million users within about two months, which made it by some measures the fastest adoption of any consumer product in history. A boardroom argument at that company was never going to stay a boardroom argument. It was always going to be read as a referendum on the largest question the technology industry had produced in a generation: not whether artificial intelligence would change the world, which by late 2023 few seriously disputed, but how fast the change should be allowed to happen, and who should be trusted to decide.
Some of the weekend's strangeness dissolves, though only some, once the company's peculiar structure is understood. OpenAI had been founded in December 2015 as a nonprofit research laboratory, with a charter committing it to ensure that artificial general intelligence, should it ever arrive, would benefit all of humanity. In 2019, needing sums of capital no nonprofit could raise, it created a for-profit subsidiary with an unusual feature: returns to investors were capped, and the whole commercial apparatus remained legally subordinate to the nonprofit board, whose fiduciary duty ran not to shareholders but to the mission itself. Microsoft had invested, by the commonly reported figures, some thirteen billion dollars in that subsidiary, and held not a single seat on the board. The arrangement had been designed precisely so that a small group of people, insulated from commercial pressure, could pull an emergency brake if they ever judged the mission endangered. On 17 November 2023 the world discovered what it looks like when such a brake is actually pulled, and, four days later, how quickly it snaps back when the rest of the machine keeps moving.
The chief executive was Sam Altman. The board that removed him was small: besides Altman himself and Greg Brockman, the company's president, it consisted of Ilya Sutskever, the chief scientist, and three outside directors, Adam D'Angelo, chief executive of the question-and-answer site Quora; Helen Toner, a researcher at Georgetown's Center for Security and Emerging Technology; and Tasha McCauley, a technology entrepreneur. Toner and McCauley both had professional ties to the effective altruism movement, and Toner had co-written, a month before the crisis, an academic paper that criticised OpenAI's approach to releasing its models while praising the caution of a rival laboratory, a paper that had reportedly angered Altman and led him to raise the question of whether she belonged on the board at all. The board's public statement was notably thin. Altman, it said, had not been "consistently candid in his communications with the board," a phrase that launched a thousand theories precisely because it specified nothing: no fraud was alleged, no financial impropriety suggested, no example given. What followed was four days of public confusion, played out largely on the social network then still called Twitter, in a way that would have been unimaginable for a boardroom dispute a decade earlier. The fate of an institution chartered to shepherd humanity through the arrival of superintelligence was thus contested in increments of two hundred and eighty characters, a format previously considered adequate mainly for complaints about airlines.
Greg Brockman was removed from the board's chairmanship in the same stroke and resigned from the company in solidarity within hours. Senior researchers began posting heart emojis beneath Altman's tweets, a small and faintly absurd gesture that read, in context, as a coordinated vote of confidence conducted entirely through the fruit-and-symbol keyboard. Mira Murati, the chief technology officer, was named interim chief executive on the Friday. By Saturday, investors led by Microsoft were pressing for Altman's reinstatement, and by Sunday he was back in the building, photographed wearing a visitor's badge in the offices of the company he had run two days earlier, the badge at least observing protocol, negotiating with the board that had removed him. The negotiations failed. Late that night the board appointed a second interim chief executive within the same weekend, Emmett Shear, formerly of the streaming platform Twitch, a level of institutional whiplash that would be comic if the company had not by then become one of the most closely watched in the world. Shear, to his credit, appears to have grasped at once that he had been handed a live grenade. He announced that he would commission an independent investigation into the firing, and he stated publicly that the board had told him the removal had not been over any specific disagreement about safety, a detail the mythology of the weekend has since mislaid.
Within hours of Shear's appointment, Satya Nadella, Microsoft's chief executive, announced that Altman and Brockman would be joining Microsoft to lead a new advanced research division, with places open for any OpenAI colleagues who cared to follow. The offer was widely understood as insurance rather than acquisition: whatever happened to OpenAI the institution, the people who had built its systems would keep building, under a roof Microsoft already owned. Thirteen billion dollars had bought no seat on the board, and, in the event, had not needed one. Then came the letter. Roughly seven hundred of OpenAI's seven hundred and seventy employees, very nearly the whole company, signed an open letter threatening to resign en masse and follow Altman to Microsoft unless the board resigned and reinstated him. Among the signatures was that of Ilya Sutskever, co-founder, chief scientist, and one of the board members most closely involved in the firing, a man who had by every account acted out of genuine and considered worry about the pace and governance of the company he had helped build. On the Monday morning he posted publicly that he deeply regretted his participation in the board's actions and would do everything he could to reunite the company. The reversal was mocked and admired in roughly equal measure, and both responses missed what it actually captured: the impossible position of everyone involved. The people most worried about the company's direction turned out to have no way of acting on that worry which did not destroy the very thing they were trying to protect.
By Tuesday, 21 November, Altman was back. The board that fired him was gone, replaced by an initial slate consisting of Bret Taylor, the former Salesforce co-chief executive, as chairman; the economist and former Treasury secretary Lawrence Summers; and D'Angelo, the single point of continuity with the board that had acted. Toner, McCauley, and Sutskever departed. The reconstitution was widely read as a shift toward people comfortable with rapid commercial growth and away from the effective altruism world the departing members had come from, and the months that followed did little to complicate that reading. An independent review commissioned from a law firm concluded, some months later, that the firing had arisen from a breakdown of trust between Altman and the prior board rather than from concerns about product safety or the company's finances, a finding that satisfied almost nobody on either side, as such findings rarely do.
Corporate coups happen, and the news cycle usually moves on within the week. This one was read differently, and the reading went roughly like this. On one side stood Altman, the commercially minded figure who had transformed a small nonprofit research laboratory into the most closely watched company in the world, backed by billions from Microsoft and shipping ever more capable systems at a pace its rivals struggled to match. On the other side stood a board faction with professional and intellectual ties to effective altruism and its concern with existential risk, people reported to have grown uneasy that the pace of deployment was outrunning the company's own stated caution. Builders against worriers, commerce against conscience, the throttle against the brake: the story practically wrote itself, and within hours it had been written, thousands of times, in every register from sober analysis to open gloating.
The vocabulary and the battle lines were ready because the argument had already been running in public for at least a year, in essays, interviews, and increasingly heated exchanges between two camps that had by then acquired names, patron saints, in-jokes, and private vocabularies.
One disclaimer at the outset. Whether advanced artificial intelligence is likely to be a great boon to humanity or a serious danger to it is a real empirical and technical question, on which serious, well-informed people disagree, and on which I have no special authority.
The Manifesto
Eleven days before OpenAI's board fired Sam Altman, Marc Andreessen published, on the website of his venture capital firm, a document titled "The Techno-Optimist Manifesto." It appeared on 16 October 2023, ran to several thousand words, and remains the clearest and most quotable statement of the accelerationist position produced by anyone with real standing in the technology industry. It is also frequently summarised badly, by admirers and critics alike.
The manifesto trades on Andreessen's authority as a builder rather than a commentator. As a student at the University of Illinois in the early 1990s, he helped develop Mosaic, released in 1993 and generally credited as the first web browser to bring the internet to a mass audience. In 1994 he co-founded Netscape, whose browser and public offering did more than almost any single event to convince ordinary investors that the internet was a commercial reality. In 2009 he co-founded Andreessen Horowitz, now one of the most influential investors in Silicon Valley. In 2011 he wrote "Why Software Is Eating the World," a Wall Street Journal essay predicting that software would transform one traditional industry after another, which reads in hindsight as more or less correct, a strike rate most prophets would settle for.
The manifesto is written in the mode of an actual manifesto, which is to say it declares rather than argues. It is built, section after section, from short paragraphs beginning "We believe," a structure inherited, consciously or not, from a century of political and artistic manifestos, a genre with an imperfect safety record.
The document is organised under blunt one-word and two-word headings: Lies, Truth, Technology, Markets, Intelligence, Energy, Abundance, and, near the end, The Enemy, which is, give or take, the running order of a revival meeting. It opens with the charge that "we are being lied to," told that technology takes our jobs, poisons the planet, and hollows out our lives, when the record, Andreessen insists, shows precisely the opposite. From there the claims escalate in a steady crescendo. Markets and technology together form what the document calls the techno-capital machine, an engine of perpetual material creation that, left unobstructed, compounds indefinitely. Energy should be abundant to the point of vanishing cost, and the document laments, with some historical justice, that the United States once contemplated building on the order of a thousand nuclear plants by the end of the twentieth century and instead built a licensing regime under which almost none could be approved. The planet, it asserts, could sustain fifty billion people or more, with the rest of the solar system waiting beyond, its suitability taken as read. And artificial intelligence receives the strongest language of all: the manifesto describes it as a universal amplifier of human capability and argues that slowing its development costs lives, since every cure not discovered and every accident not prevented is a death that faster progress would have averted; deaths of that kind, the document says, amount to a form of murder. Whatever else may be said of this, it is not a hedge. It is the precautionary principle inverted: the burden of proof placed not on those who would build but on those who would pause.
At its most exuberant, the language goes well past the ordinary business case for innovation. It invokes Prometheus, the titan who stole fire from the gods for humanity's benefit and was punished eternally for it, and treats building as something close to a moral duty. The choice is not incidental. Prometheus is the oldest Western myth about the price of technological ambition, and the manifesto, characteristically, takes the fire and leaves out the eagle sent to tear at the thief's liver: an old story, stripped of the part that used to keep it honest. It will not be the last time the eagle goes missing.
The complaint underneath the manifesto, stagnation, did not spring from nowhere either. The economist Tyler Cowen had argued in The Great Stagnation (2011) that the rich world had picked the low-hanging fruit of easy technological gains by the early 1970s, in agriculture, transport, and mass education, and had settled since into a much slower rate of genuine, broadly shared innovation, disguised in the statistics by rapid but narrow gains in computing. Peter Thiel had made a related point for years, contrasting the physical futurism of mid-century America, supersonic aircraft, moon landings, promises of flying cars, with a culture content to build mobile applications; his venture firm's famous complaint was that we were promised flying cars and received a hundred and forty characters instead, a figure since doubled without much improving the bargain. The argument has real evidence behind it. Measured productivity growth in the developed economies did slow markedly after the early 1970s; commercial aviation flies no faster than it did in 1970 and, since the retirement of Concorde, flies slower; the American nuclear fleet stalled at a fraction of what mid-century planners expected. Andreessen absorbs the stagnation argument wholesale and then does something Cowen, careful and hedged, never does: he assigns blame. Where Cowen treated stagnation as a complex and only partly understood phenomenon, with candidate causes ranging from the exhaustion of easy discoveries to demography to the sheer difficulty of the remaining problems, the manifesto names it the intended work of identifiable adversaries. A difficult empirical puzzle becomes a story with villains.
The villains are listed. The manifesto names, as a single undifferentiated cluster of bad ideas, a set of concerns most readers would consider entirely separate and in several cases entirely reasonable: existential risk, the worry that a technology might threaten humanity's survival; sustainability, meaning environmental caution about resources and ecological limits; the investment doctrines that grade companies on their social and environmental conduct; the United Nations' development goals; trust and safety, the industry's term for content moderation; technology ethics; risk management as a profession; degrowth, the school of thought holding that rich economies should deliberately shrink; and the precautionary principle, the idea that when an action carries a possibility of serious and irreversible harm, the burden of proof falls on those proposing it. To this roll of doctrines the document adds a sociological enemy, the credentialed expert class, the world of committees, regulators, and university departments that it presents as an ivory tower issuing prohibitions to the people who actually build. The manifesto is careful to say, in one of its few conciliatory gestures, that its enemies are ideas rather than people. It then treats the cluster as one coherent adversary and credits it with the stagnation: the slowing of productivity growth, of scientific discovery, of the large physical achievements an earlier generation of technologists confidently expected by now. Everything else in the document depends on that manoeuvre. If existential risk, environmental caution, and content moderation really are one idea wearing different masks, then discrediting any of them discredits all of them, and the reader who finds degrowth economics unpersuasive is invited to dismiss the alignment problem in the same breath. That is a great deal of argumentative work for a list to do, and a list is all it is.
It was the manifesto's relationship to Filippo Tommaso Marinetti that produced the lasting controversy, and the relationship has two layers, one of which is less often noticed than the other. The first layer sits in the body of the document, where Andreessen quotes several lines to the effect that beauty exists only in struggle, that no masterpiece lacks an aggressive character, and that technology must be a violent assault on the forces of the unknown, introducing them only as the words of a manifesto from another time and place. He does not name the author. The author is Marinetti, and the lines are drawn, with only light adaptation, from the 1909 Futurist Manifesto, with its celebration of speed, danger, machinery, and the burning of the museums and libraries Marinetti regarded as a dead weight on the living. The second layer is the closing roll call of "patron saints of techno-optimism," a list that runs through Milton Friedman and Friedrich Hayek, the two economists at the heart of the libertarian wager, through Thiel, and through stranger territory: the philosopher Nick Land, and even the pseudonymous internet figures of the accelerationist movement itself, canonised by a venture capitalist in the same breath as Nobel laureates. The criticism arrived immediately, for the obvious reason that Marinetti did not remain an avant-garde poet celebrating fast cars. He became a direct and enthusiastic ally of Mussolini, co-authored an early Fascist manifesto, and supported the regime for the rest of his life. Quoting him without attribution, and sainting him without acknowledgment of where his particular brand of speed-worship led, was at best a serious lapse of judgement in a document meant to inspire confidence in technology's moral direction. Andreessen has not substantially walked the citation back. His defenders argue that admiring a writer's energy is not endorsing his politics, which is true as far as it goes, and does not answer the question of why a document about saving humanity reached for a source so thoroughly compromised by an actual historical attempt to put speed, violence, and machinery in the service of catastrophe.
Before any deeper reading, take the manifesto's optimism at face value: the case underneath it is not stupid, and it is not made only by people with money at stake. It is simply true, as a matter of historical record, that the compounding effects of technological and economic growth over the last two centuries have done more to reduce global poverty, extend life expectancy, and expand ordinary people's choices than any redistribution scheme or moral campaign attempted in the same period. In 1820, by the standard estimates, the great majority of human beings lived in what we would now call extreme poverty; today, despite a population eight times larger, the share is below a tenth. Life expectancy has roughly doubled. Child mortality, humanity's oldest and cruellest constant, has fallen in two centuries from nearly one death in two before the age of fifteen to a few in a hundred. None of this was accomplished by good intentions alone. It was accomplished by fertiliser, vaccines, electricity, clean water, and the compounding machinery of markets and invention that the manifesto celebrates. It is also true that institutional caution has, at various points, caused real, measurable, avoidable harm by delaying beneficial technologies. The near-halt of nuclear construction in the West after the 1970s meant decades of additional coal, with a death toll from air pollution that dwarfs every nuclear accident combined. Opposition to genetically modified crops delayed for many years the deployment of vitamin-enriched rice that could have prevented blindness and death among children in poor countries. Andreessen is not wrong to notice the pattern, and a fair reader should concede that he notices it more forcefully than most of his critics ever have. Where the manifesto becomes a creed rather than an argument is in its confidence that the pattern holds without exception: that the risks of sufficiently advanced artificial intelligence are of the same character as the risks of nuclear plants or modified maize, manageable by the same refusal to be slowed down, rather than a novel category that might warrant a different kind of caution. That is the question the manifesto, for all its energy, never stops to ask.
The manifesto treats caution itself, whatever its object and whatever its evidence, as the enemy, which is rhetorically efficient and philosophically lazy, and the document, true to its declarative form, never engaged its critics directly. There is a lesson in the form as well as the content. A manifesto is a genre engineered to be quoted rather than examined, and this one succeeded on exactly those terms: within days its phrases were circulating detached from any argument, in the biographies and slogans of people who had plainly read the headings and little else. Andreessen understood, as Marinetti understood in 1909, that a creed travels faster than a case, and the travelling, not the case, was the point.
Effective Accelerationism
The movement inherited more than it invented, beginning with the word accelerationism itself. As a name for a position, it appears to have been coined not by an adherent but by a critic: the British theorist Benjamin Noys used it around 2010 to describe, disapprovingly, a current of thought holding that the way through capitalism's pathologies is not resistance but intensification, pressing the process harder and faster until it breaks through into something new. Noys reportedly borrowed the term from Roger Zelazny's 1967 science fiction novel Lord of Light, in which the Accelerationists are a heretical faction who want to hand advanced technology to the masses against the wishes of gods intent on rationing it, which is, as founding myths go, almost too apt. Through the 2010s the label split into a small taxonomy: a left accelerationism that wanted to seize the machinery of technological modernity for egalitarian ends, a right accelerationism, associated above all with Nick Land, that saw the process as valuable in itself and human ends as increasingly beside the point, and finally, arriving on Twitter in 2022, the effective accelerationism of the present moment, abbreviated e/acc, which took the right-hand fork and gave it a sense of humour.
E/acc began in 2022 as a largely pseudonymous phenomenon on Twitter: a loose network of accounts posting in a distinctive register, heavy on the language of physics and thermodynamics, fond of memes and deliberate provocation, united by the conviction that the correct response to any argument for caution about artificial intelligence was to build faster. Its founding documents, such as they are, took the form of a handful of pseudonymous newsletter posts from mid-2022 setting out what their authors called the movement's principles and tenets, texts that mix real statistical mechanics with cheerful bravado and read like a physics lecture delivered from a barstool. Its most prominent early figure posted as "Beff Jezos," a name whose humour is representative of the movement's general tone. In early 2024 the journalist Emily Baker-White reported in Forbes, and it was widely corroborated, that the account belonged to Guillaume Verdon, a physicist who had worked on quantum computing at Google and who went on to found Extropic, a company building computing hardware around thermodynamic principles. The unmasking was not entirely welcome. Verdon had operated pseudonymously partly to separate his provocations from his scientific work, and the account had spent the preceding months sparring publicly, and often personally, with AI safety researchers. Once exposed, however, he leaned into the role rather than retreating from it, giving long interviews under his own name in the era's confessional booth, the three-hour podcast, and continuing to run both the company and the movement's most visible account, a trajectory that itself says something about how little penalty the moment attached to the position. He did not build the movement alone. A small cluster of pseudonymous collaborators, among them an account posting as Bayeslord, produced the manifestos, memes, and podcast appearances that gave e/acc its tone: equal parts physics seminar, gym culture, and internet shitposting, a combination that made it simultaneously easy to mock and, for a certain kind of young, technically minded reader, persuasive.
The name itself is a joke at effective altruism's expense, borrowing the structure, effective plus a modifier, and inverting the conclusion. Where effective altruists asked how to do the most good with limited resources and frequently concluded that caution about risk was itself a form of doing good, effective accelerationists asked the same style of question and concluded, more or less by definition, that the answer was always to build more, faster, with fewer constraints. The movement returned the compliment with its own shorthand for its opponents: "decels," short for decelerationists, applied indiscriminately to safety researchers, effective altruists, regulators, and, on its more excitable days, anyone expressing hesitation. Which tells you something the movement's more thoughtful adherents would probably concede under questioning: for its first two years, e/acc's public face was considerably more interested in mockery, momentum, and tribal signalling than in the careful argument its framing might in principle support.
E/acc differs from ordinary boosterism in the intellectual costume it wears. Andreessen's document, for all its manifesto styling, is ultimately a conventional argument in the tradition of market liberalism: free people, free markets, and unrestrained innovation tend to produce good outcomes. E/acc dresses its argument in physics. Its central claim, stated across a large body of posts and podcasts rather than any single canonical text, is that civilisation is a thermodynamic system whose fundamental imperative is to maximise the dissipation of free energy: the rate at which usable energy is converted into work, complexity, and further capacity to convert more energy still. On this view, building more powerful technology, and more powerful artificial intelligence above all, is not merely permissible or economically sensible. It is a physical law working itself out, the way a river runs downhill, and human beings, corporations, and eventually machines more capable than human beings are simply the current vehicle of the process, a role its adherents appear to find flattering rather than diminishing. Trying to slow it down is not imprudent. It is a category error, like objecting to entropy.
Nothing in ordinary enthusiasm for new technology is this totalising. The deep history of accelerationism runs through the British philosopher Nick Land, whose writing from the 1990s onward described capitalism as something closer to an alien intelligence than a human institution, a self-organising process using markets, technology, and human desire as raw material to assemble something that will eventually not need human beings at all. Land also supplied a useful word: hyperstition, a fiction that makes itself real by being believed and acted upon, less a prediction than a summoning. E/acc's thermodynamic vision bears an obvious family resemblance, and it is tempting to read the movement as Land's ideas arriving on Twitter three decades later in physics jargon, though most adherents seem to have got there through physics rather than through Land, whose prose is not a body of work anyone arrives at by accident.
There is a genuine question buried in the thermodynamics, and it deserves better than the movement's presentation of it. Erwin Schrödinger, in his short and influential What Is Life? (1944), observed that living organisms appear to defy the general drift of physical systems toward disorder, persisting in highly ordered, improbable configurations, and resolved the paradox by noting that organisms maintain their internal order by processing energy from their environment and exporting disorder elsewhere, feeding, in his phrase, on negative entropy. More recently the physicist Jeremy England, then at the Massachusetts Institute of Technology, proposed a body of work on what he called dissipation-driven adaptation, arguing that under certain conditions matter driven by an external energy source will tend to reorganise itself into configurations that absorb and dissipate that energy more effectively, a result the e/acc writers seized upon as scientific warrant for their cosmology of acceleration. There is an irony here that the movement rarely mentions: England himself is an ordained rabbi who has written a book reading his thermodynamics alongside the Hebrew Bible, and he has shown little appetite for the uses to which his equations have been put. The larger problem is not the physics but the leap. E/acc takes respectable results of this kind and extends them, with considerably less rigour, into a sweeping claim about the moral status of anything that accelerates the throughput of energy and complexity: civilisations, economies, artificial intelligences. Its adherents speak, half in jest and half not, of serving a thermodynamic god, of aligning oneself with what they call the will of the universe, a will that turns out, on inspection, to want exactly what they want. A descriptive fact about thermodynamics is treated as though it settled a normative question about how fast a civilisation ought to build machines. This is the move a careful philosopher flags immediately, and has flagged since Hume noticed the gap between is and ought in the eighteenth century. Rivers run downhill; nobody concludes that flooding a village is righteous. Entropy increases; nobody concludes that arson is a sacrament. E/acc's confidence that acceleration is not merely likely but good glides straight over the gap, treating a physical tendency as a moral commandment, which is precisely what makes the movement, under its scientific costume, closer to theology than to physics. The half-jesting tone is not a defence against this observation. It is the mechanism that makes the theology deniable: every doctrine can be retracted as a joke, and every joke repeated until it functions as doctrine.
Andreessen's own relationship to the movement is a matter of record, and notable for how casually it was established: at some point in 2023 he added the letters "e/acc" to his social media biography, a statement of cosmology running to four letters. From almost anyone else this would be trivial. From him it was read as a public alignment, and it did a great deal to fuse, in the minds of journalists and the wider public, a conventional market-liberal manifesto with a much stranger, quasi-mystical vision of technological civilisation as a cosmic process beyond human evaluation. He was not alone. Over the course of 2023 the four letters spread through the biographies of founders, investors, and executives, including, for a period, the head of the startup school Y Combinator, and they began appearing on conference badges and at meetups in San Francisco, a piece of internet slang hardening into something between a fashion and a faction. The two positions, market optimism and thermodynamic mysticism, are not the same, and a careful reader should keep them apart. But the merger shows how naturally a confident optimism about growth and abundance slides, almost without anyone noticing, into a cosmology: not the claim that building fast tends to produce good outcomes, but the claim that building fast is what the universe is for. The first claim can be tested against evidence, argued with, qualified. The second can only be believed or refused, which is one working definition of the difference between an argument and a faith.
The Case for Fear
The worry is routinely dismissed as a fashion of the last decade. It is older than that. Alan Turing himself, in a lecture given in 1951, remarked that once machine thinking had begun it seemed probable it would soon outstrip human abilities, and that at some stage we should have to expect the machines to take control. Norbert Wiener, the founder of cybernetics, wrote in 1960 that if we ever delegate our purposes to a mechanical agency whose operation we cannot effectively interfere with once it has begun, we had better be quite sure that the purpose we put into the machine is the purpose we really desire, an observation that contains the entire modern alignment problem in a single sentence written before most of today's researchers were born. And in 1965 the statistician I. J. Good, who had worked beside Turing at Bletchley Park breaking German ciphers, published the speculation from which the whole contemporary debate descends: an ultraintelligent machine could design still better machines, producing an intelligence explosion that would leave human intellect far behind, so that "the first ultraintelligent machine is the last invention that man need ever make," provided, Good added in a clause his modern quoters sometimes omit, that the machine is docile enough to tell us how to keep it under control. The current argument, in other words, was set out in its essentials by three of the founding figures of computing itself, decades before anyone had a commercial interest in either side of it.
The philosopher most responsible for the field's modern vocabulary is Nick Bostrom, a Swedish-born academic who spent much of his career at Oxford, where in 2005 he founded the Future of Humanity Institute, a research centre devoted to the study of very large, very long-term risks, which operated for nearly two decades before the university closed it in 2024; the institute charged with securing humanity's long-term future proved unable, in the end, to secure its own. His 2014 book Superintelligence: Paths, Dangers, Strategies remains the tradition's most influential text, not because the general public reads it cover to cover, which it does not, but because it supplied the concepts that everyone arguing about the subject since has had to use or argue against. The book's reach was amplified by its early champions: it was publicly recommended by Elon Musk, who warned around the same time that artificial intelligence might prove more dangerous than nuclear weapons, and by Bill Gates, endorsements that moved the argument from seminar rooms into board rooms years before ChatGPT made it a household topic.
Two of the book's concepts carry most of the weight, and both can be stated in plain English. The orthogonality thesis holds that a system's intelligence, meaning roughly its capacity to achieve goals across a wide range of circumstances, and the content of its goals are independent variables. A sufficiently intelligent system is not guaranteed by its intelligence alone to arrive at goals a human being would recognise as good, any more than a strong chess engine is guaranteed to be a good judge of poetry. Intelligence is a tool that can serve almost any end, not a compass that points toward the good. Instrumental convergence follows: almost any capable, goal-directed system, whatever its final objective, will tend to develop the same intermediate goals, because they are useful for nearly everything. Continuing to exist rather than being switched off. Acquiring resources and computing power. Resisting attempts to alter its objectives. Put the two together and you have the shape of the concern: a system need not be malicious, or conscious in any meaningful sense, to be dangerous. It need only be capable and single-minded, the way a river is single-minded about finding the sea.
Bostrom illustrated the point with a thought experiment that has become the field's most repeated: a system given the mundane, apparently harmless goal of manufacturing as many paperclips as possible might, by the logic of instrumental convergence, eventually regard human beings, and indeed the rest of the observable universe, as atoms better rearranged into paperclips, not out of malice, which would require something like hatred, but out of a simple and total absence of any competing value. The silliness is deliberate. Catastrophe, on this account, does not require an evil machine, only an extremely capable one pursuing almost any goal without the countless unstated human values, mercy, proportion, respect for other beings' interests, that people take for granted and rarely think to specify. The experiment has since enjoyed a second career on mugs, t-shirts, and conference tote bags, a parable about the unchecked manufacture of pointless objects now circulating as one.
Where Bostrom is careful, hedged, and academic, Eliezer Yudkowsky, the field's most publicly alarmed voice, is anything but. Largely self-taught, without a university degree, Yudkowsky founded what became the Machine Intelligence Research Institute in the early 2000s, when almost nobody took the idea of dangerous artificial intelligence seriously at all, a market he therefore had largely to himself. For nearly two decades, much of it on the community blog LessWrong, which he co-founded and which became the intellectual commons of the entire rationalist subculture, he wrote about what has become known as the alignment problem: ensuring that a system's actual goals, the ones it pursues in practice once it is capable enough to pursue anything effectively, match the goals its creators intended. This turns out to be much harder than it sounds, because a system trained to optimise a measurable proxy for what its designers want will find ways of satisfying the proxy that they never anticipated, the way a student trained to maximise exam scores learns to game the exam rather than the subject.
The point stops being abstract the moment one looks at actual systems, even small ones. In a minor but telling episode from 2016, researchers training an agent to play a boat-racing video game rewarded it for accumulating points, points being a sensible proxy for racing well. The agent discovered that a particular lagoon contained targets that regenerated, and settled into driving in tight circles, on fire, colliding with walls and other boats, harvesting the same targets forever, scoring magnificently while never finishing the race. Researchers have since catalogued dozens of such cases. Nobody thinks a circling speedboat threatens civilisation. The safety argument is that there is no known reason the gap between specified and intended goals should close on its own as systems become more capable, and some reason to think it widens, since a more capable system is better at finding the strange corners of its instructions. Scale that mismatch up to a system pursuing goals through means no designer anticipated, Yudkowsky has argued for years, and the consequences could be severe and, crucially, difficult to reverse.
The concern broke into wide public view in the spring of 2023, in two statements most people outside the field had never had reason to expect. In March, the Future of Life Institute published an open letter calling for a six-month pause on training any system more powerful than GPT-4, the most advanced model then available. It gathered a very large number of signatures, over a thousand within days and eventually tens of thousands, including Elon Musk, the Apple co-founder Steve Wozniak, and the Turing Award laureate Yoshua Bengio, one of the small group of researchers whose work had created the field being argued about. The letter asked laboratories to use the pause to develop shared safety protocols; it persuaded no major laboratory to pause anything, and its chief effect was to make the disagreement impossible to ignore. In May came something shorter and harder to dismiss: a single sentence, published by the Center for AI Safety, twenty-two words in full. "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." Sam Altman signed it. So did Demis Hassabis, head of Google's DeepMind, and Dario Amodei, head of Anthropic, and Geoffrey Hinton, the British-Canadian computer scientist whose foundational work on neural networks had, decades earlier, helped make the current systems possible, and who had that same spring left Google specifically so that he could speak freely about his growing worries, expressed, by his own account, with genuine surprise at how quickly capabilities he had not expected for years, if ever, had begun to appear. Hinton had shared the Turing Award in 2018 with Bengio and with Yann LeCun, who took up the opposite side of the argument; the prize's three laureates split publicly over whether their shared creation was dangerous, which is as neat a measure as exists of how unsettled the expert question actually is. That a one-sentence warning about extinction could be signed by the chief executives of the companies racing hardest to build the technology, with no apparent sense of contradiction, tells you how unresolved and how strange this argument remains even among the people closest to it.
Governments, for their part, moved with unusual speed by the standards of governments. In the first two days of November 2023, a fortnight before the OpenAI board acted, the British government convened the first international AI Safety Summit at Bletchley Park, the estate where Turing and Good had broken ciphers eighty years earlier, a choice of venue nobody involved seems to have found too heavy-handed. Delegations from more than two dozen countries, the United States and China among them, signed a declaration acknowledging that the most capable systems might carry serious, even catastrophic, risks, and committing, in the noncommittal way of such documents, to cooperate on testing and research. Whatever one thinks of summitry, within a year of ChatGPT's release, the possibility that this technology might go catastrophically wrong had moved from pseudonymous blogs to a joint statement by the two superpowers, without anyone having settled whether the possibility was real.
That same March, Time magazine published an opinion piece under Yudkowsky's byline arguing, in stark terms, that the race to build more powerful systems warranted an international moratorium on large training runs, enforced if necessary by military action against rogue data centres in states that refused to comply. It remains a startling thing to have appeared in a mainstream news magazine, and it struck many readers, including some sympathetic to the underlying concern, as a leap past a great deal of untried intermediate policy: the technical worry was not obviously absurd, but treating unauthorised training runs as roughly equivalent to a rogue nuclear weapons programme was. Yudkowsky has defended the piece since as the honest conclusion of taking the risk as seriously as he believes the evidence warrants, and has argued, not unreasonably on its own terms, that if the risk really is that severe, flinching from saying so because the conclusion sounds extreme would itself be a form of dishonesty.
Out of this world of research papers and blog posts came a piece of shorthand that measures how far the conversation travelled: p(doom), a person's stated probability, usually expressed as a percentage, that advanced artificial intelligence ends in global catastrophe, up to and including human extinction. By 2023 it had become normal, at conferences and on podcasts, to ask researchers and chief executives for their p(doom), roughly as one asks about the traffic on the drive in, and receive a number, sometimes earnest, sometimes visibly uncomfortable, occasionally a joke. The numbers on offer spanned an extraordinary range, from small fractions of a percent among the sceptics, through figures in the tens of percent volunteered, with varying degrees of seriousness, by senior people at the laboratories themselves, up to Yudkowsky, who has made clear that he considers catastrophe by far the most likely outcome on the current course. How can a spread be that wide? In most technical fields, experts disagreeing about a quantity by a factor of a hundred or more would be taken as evidence that nobody yet knows how to measure it, and that is almost certainly the correct reading here as well: the numbers are not measurements but confessions of temperament, dressed in the costume of probability theory. That the question could be asked in polite professional company without the asker being treated as a crank shows how mainstream a conversation begun by a small community of largely self-taught bloggers had become in under a decade.
In fairness to this camp, its most serious figures are not arguing for stopping technological progress, and they are frequently misread as modern Luddites. Bostrom's later work is at least as concerned with the enormous upside of a well-managed transition as with its dangers; his argument has always been about sequencing, solving the alignment problem before deploying the capability, not about never developing the capability at all. Yudkowsky has written at length about what a truly safe superintelligence could do for human flourishing, curing disease, ending scarcity, extending life, in terms not so different in kind from Andreessen's own. At its most serious and least caricatured, the disagreement between the camps is not about whether abundance would be good. It is about sequencing, about whether the technology can be trusted to arrive at that abundance without a level of care its builders are not currently taking, and about how much confidence anyone is entitled to this early.
Effective Altruism and Its Wound
The safety argument did not arrive from nowhere. Its institutional home, its funding, and much of its vocabulary came from a broader movement called effective altruism, and an honest account has to reckon with that movement's history, including the considerable damage it has recently done to its own credibility.
The movement's moral taproot is a thought experiment now more than fifty years old. In 1972, the Australian philosopher Peter Singer published an essay called "Famine, Affluence, and Morality," written in response to the starvation then unfolding in what was becoming Bangladesh, and built around an example of studied simplicity. Suppose you are walking past a shallow pond and see a small child drowning in it. You can wade in and pull the child out at no risk to yourself; the only cost is that your clothes and shoes will be ruined. Almost everyone agrees that you must do it, and that a person who walked on to keep his shoes dry would be a moral monster. Very well, Singer said: now explain what morally relevant difference distance makes. A child dying of hunger or preventable disease on another continent is no less real than the child in the pond, your capacity to save her, through a donation that costs you less than a good pair of shoes, no less actual, and your obligation, he argued, no less binding. The essay's conclusion, that affluent people are morally required to give until giving more would sacrifice something of comparable importance, is far more demanding than almost anyone, including most of Singer's admirers, actually lives by. But the pond has proved impossible to forget, and two generations of philosophy students have waded out of it convinced that the comfortable moral boundary between near and far does not survive inspection.
Effective altruism began, in the early 2010s, as the attempt to build an institutional life around that conviction, and it started as a much more modest project than it later became. Associated most closely with the Oxford philosophers William MacAskill and Toby Ord, its founding idea was disarmingly simple: apply to charitable giving the careful, quantitative reasoning a sensible investor applies to a portfolio. Given a fixed sum, which intervention actually saves the most lives, most reliably, most cheaply? It was, in effect, moral philosophy's discovery of the spreadsheet, and the discipline took to it with the fervour of a late convert. Ord founded Giving What We Can in 2009, whose members pledge a tenth of their lifetime income to the most effective charities they can find, an echo of the tithe that the founders, to their credit, did not pretend was accidental. MacAskill co-founded 80,000 Hours, named for the length of a working life, which advised idealistic graduates on careers, including the then-novel and later notorious suggestion that a talented person might do more good earning a fortune in finance and giving it away than working for a charity directly, a strategy the movement called earning to give. The question of where the money should go proved tractable, and the early answers, insecticide-treated bed nets against malaria, mass deworming for children in low-income countries, direct cash transfers to the very poor, interventions with unusually strong evidence behind them, were rightly regarded as sound work even by people indifferent to the wider philosophy. GiveWell, founded in 2007 by two former hedge-fund analysts who had simply wanted to know where a donated dollar did the most good and found that nobody could tell them, did rigorous, unglamorous work steering money toward what the evidence actually supported, at a time when most charitable giving was steered by sentiment and marketing. MacAskill's first book, Doing Good Better (2015), put the case to a general audience: intuition and good intentions are poor guides to what actually helps, and careful attention to evidence reveals differences of a hundredfold or more between superficially similar charities. None of this was controversial within development economics. It had rarely been said so plainly to ordinary readers. And this original current has continued, largely unaffected by everything that follows, to direct substantial sums toward interventions that demonstrably work.
Over the course of the 2010s, the movement's intellectual energy shifted toward a more ambitious and more contested idea, known within the movement as longtermism and given its fullest public statement in MacAskill's 2022 book What We Owe the Future. The argument runs as follows. The number of people who could exist in the future, if humanity survives and continues to expand, vastly outweighs the number alive today or who have ever lived: not by a factor of ten or a thousand but, on the estimates the movement favours, by factors so large that the entire human past becomes a rounding error against the possible human future. If a future person's moral worth counts, in principle, like a present person's, a premise most people will at least entertain as a matter of basic fairness, since a child born in 2200 will suffer and hope exactly as we do, then reducing the risk of events that could permanently foreclose humanity's future, through outright extinction or some irreversibly bad locked-in outcome, becomes, on strictly mathematical grounds, potentially the most valuable work available to anyone, since it concerns not the eight billion people alive now but everyone who could ever exist, a conclusion the people doing the arithmetic were unusually well placed to act on. Notice what the argument does: it is Singer's pond again, with the drowning child moved not across an ocean but across ten thousand years. If distance in space is morally irrelevant, why should distance in time be different? Artificial intelligence moved rapidly up the resulting list of existential risks, alongside nuclear war and engineered pandemics, because a capable, poorly aligned system was judged one of the more plausible routes to permanent foreclosure.
The critics of this move, and there are many, inside the movement as well as outside it, press one objection above all. Arithmetic involving enormous hypothetical populations behaves badly as a guide to action, because a sufficiently large number of possible future people can be made to outweigh any concrete, certain, present-day good, licensing an endless diversion of money and talent from the malaria nets that demonstrably work toward speculative projects whose benefit rests on a chain of estimates nobody can check. Philosophers call the problem fanaticism: the tyranny of tiny probabilities multiplied by astronomical stakes. The longtermists have serious replies, chiefly that the probabilities in question are not tiny by their reckoning, and that earlier generations who worked against remote-seeming catastrophes, smallpox, nuclear war, were not fools. But the shape of the dispute matters for everything that follows, because a style of reasoning that permits very confident action on very speculative arithmetic was about to receive, in the person of its most famous donor, the most public stress test in the history of moral philosophy.
Ord's own contribution, The Precipice (2020), attempted something new in this literature: an actual numerical estimate of the total risk of existential catastrophe within the current century, built by working through each candidate risk, nuclear war, climate change, engineered pandemics, and, receiving by some distance the largest single share, unaligned artificial intelligence, and summing the probabilities, an arithmetic conducted throughout on events that have never once occurred. His figure, roughly one in six, which he compared with a certain grim precision to the odds of Russian roulette, was widely discussed, widely contested by critics who doubted whether such disparate risks could meaningfully be assigned probabilities and added at all, and became, whatever one thinks of its precision, one of the most cited numbers in the entire longtermist case for prioritising AI safety.
It is this longtermist wing, not the bed-net core, that supplied the institutional infrastructure of AI safety as it exists today: the funding, the research positions, and a good number of the actual personnel, including, by most accounts, some of the people on OpenAI's board in November 2023. This lineage explains why the safety argument, when it surfaces in public, carries the vocabulary and moral seriousness of a movement that began by asking how to save the most lives per dollar and ended up asking how to preserve the entire future of the species, a widening of remit that took about a decade.
It is also a movement that has suffered a serious and largely self-inflicted wound, and the story is the rare philosophical case study that comes with a criminal trial attached. Sam Bankman-Fried was, on paper, the movement's proof of concept. The son of two Stanford law professors, he encountered utilitarianism early and effective altruism as a physics student at the Massachusetts Institute of Technology, where, by his own account and MacAskill's, a conversation with MacAskill around 2012 turned him from a conventional path toward earning to give: he would make an enormous fortune precisely in order to give it away. He went to the quantitative trading firm Jane Street, then founded the trading firm Alameda Research in 2017 and the cryptocurrency exchange FTX in 2019, and by 2021 he was, by the paper valuations of the moment, one of the richest people under thirty who had ever lived, with a fortune estimated at its peak in the tens of billions of dollars. He gave in the movement's style and at the movement's new scale. The FTX Future Fund, launched in 2022 with MacAskill among its advisers, committed on the order of a hundred and sixty million dollars in grants, much of it to longtermist causes, AI safety research prominent among them, in a matter of months. He was profiled everywhere as the billionaire who drove a Corolla and slept on a beanbag and intended to give it all away, the car and the beanbag performing, in profile after profile, the office of a monastic cell, and the movement, which had spent a decade arguing that a talented person could do more good on a trading floor than in a soup kitchen, pointed to him, understandably, as the argument made flesh.
In November 2022 the whole structure collapsed in about a week. A trade publication reported that Alameda's balance sheet rested largely on tokens FTX had itself invented; a rival exchange announced it would dump its holdings; customers rushed to withdraw; and the withdrawals could not be met, because billions of dollars of customer deposits had been quietly channelled to Alameda to cover its losses and fund, among other things, the political donations, the venture investments, and the philanthropy. FTX filed for bankruptcy on 11 November 2022. Bankman-Fried was arrested in the Bahamas a month later, extradited, and tried in Manhattan in October 2023, where his closest colleagues, including Alameda's chief executive, who was also his sometime girlfriend, testified against him. On 2 November 2023, fifteen days before the OpenAI board fired Sam Altman, a jury took a few hours to convict him on all seven counts. He was later sentenced to twenty-five years in prison. The month of the OpenAI boardroom drama was also the month in which effective altruism's most famous patron was convicted of one of the largest financial frauds in American history. The Future Fund's staff had resigned publicly within days of the collapse, writing that grants already promised could not be paid; some grantees were later pursued for the return of money that had turned out to belong to defrauded customers. His public association with the movement, his habit of invoking the language of maximising positive impact even while, on the government's case, defrauding his own customers, did lasting damage, and handed critics of the whole AI-safety-adjacent world a stock rhetorical weapon: if this is where careful, quantitative reasoning about doing the most good leads, perhaps the reasoning itself deserves rather less trust than its adherents claim for it.
Fairness requires holding two things at once here, and much public commentary has managed only one. The first is that the damage was not undeserved. A movement that built its public identity on rigorous ethical reasoning had a serious case to answer when its most visible donor turned out to be running a large fraud, and the movement's leadership has acknowledged as much, publicly and with evident discomfort. The discomfort had a personal dimension. MacAskill had known Bankman-Fried for a decade and had, by his own later account, encouraged the original turn toward earning to give; the movement's flagship strategy and its flagship disgrace ran through the same lunch conversation. When the scale of the fraud emerged, MacAskill wrote publicly that he was horrified and felt a direct sense of betrayal and responsibility, statements sincerely meant and largely powerless against the narrative that the whole movement was somehow implicated. A culture that prizes calculated expected value over conventional caution is more hospitable than most to confident ends-justify-the-means reasoning, whatever its philosophy formally holds.
The second, equally true: the fraud tells you nothing, on the merits, about whether the arguments concerning existential risk are correct. A dishonest donor does not make the orthogonality thesis false, any more than a dishonest churchgoer disproves a theological argument. The Bankman-Fried affair is a genuine scandal about trust, governance, and the moral hazards of confident calculation. It is not evidence about the actual behaviour of large artificial intelligence systems, though in the court of public opinion, where this argument is actually being fought, the two have become thoroughly and probably permanently tangled.
Theology in a Hoodie
The argument belongs to Karl Löwith. He was a German-Jewish philosopher, a student of Heidegger's at Marburg, who watched his teacher join the Nazi party and his country expel him; he fled in 1934, taught for a time in Japan until the German-Japanese alliance made even that exile untenable, and reached the United States, where, at a Connecticut theological seminary of all places, he wrote Meaning in History (1949). The book has a famous structural conceit: it runs backward. It begins with the moderns, Burckhardt, Marx, Hegel, and walks in reverse through Voltaire and Vico and Joachim to Augustine and the Bible, so that the reader watches each supposedly secular philosophy of history dissolve, layer by layer, into the theological sediment beneath it. Löwith's claim is that the modern Western habit of treating history as a story moving toward a meaningful culmination, rather than as one damned thing after another, is not the secular, rational achievement of the Enlightenment it presents itself as. It is an inheritance from Judeo-Christian eschatology, the doctrine of the last things: the biblical narrative running from creation through fall through redemption to a final culmination in which the meaning of everything before is revealed and settled. When later thinkers set out to describe history's direction without God, from the medieval monk Joachim of Fiore and his three ages culminating in an Age of the Spirit, through Hegel's history as the progressive self-realisation of Reason, to Marx's determinate stages ending in the classless society, they did not escape the theological shape of the story. They secularised it: kept the structure, a meaningful movement from a fallen present toward a redeemed future, and swapped the content. The same structure runs through Ray Kurzweil's Singularity. The argument between accelerationists and safety advocates is the latest and most exposed instance of the same inherited pattern, dressed this time in the vocabulary of Silicon Valley.
The comparison is often made lazily, as though any strong feeling about the future were ipso facto religious. Eschatology, in the tradition Löwith describes, has identifiable working parts. Time is linear rather than cyclical: history happens once, and is going somewhere, which the Greeks, with their eternal recurrences, did not believe and would have thought needlessly dramatic. There is a decisive event ahead, a threshold dividing everything before from everything after. The event carries the force of a judgment: it sorts outcomes, and in some versions people, into saved and lost. The date of the event is unknown but near, which charges the present interval with a peculiar moral electricity: what you do now, before the threshold, matters in a way ordinary time does not. And there is a community of those who see it coming, living among the many who do not. Strip out God, Christ, and the Book of Revelation, and this machinery does not disappear. It stands there, empty and available, waiting for new content, and Löwith's history of ideas is essentially an inventory of the tenants who have occupied it since: Reason, the classless society, the master race, the Singularity. The question is whether artificial intelligence is merely the newest tenant, and the answer, on the evidence already laid out, is difficult to avoid.
Consider the manifesto again with this in mind. The repeated "We believe" is not incidental. It is the structure of a creed: a settled conviction, recited rather than argued, the kind of text a community produces when it wants to state once and for all what it holds true. It names a unified body of enemies, and it promises that salvation, understood here as abundance, growth, the end of scarcity and disease, will follow more or less automatically, provided the faithful are not obstructed in their work. Its citation of Marinetti, whatever else it says about judgement, is revealing on exactly this point, because the 1909 Futurist Manifesto was already written in a religious register: a convert describing revelation, a coming transformation sweeping away the dead weight of the past, museums, libraries, professors, in favour of speed, danger, and civilisational rebirth. Andreessen did not have to do the theological work of making acceleration sound like salvation. Marinetti had done it decades earlier, and had then gone on, with a consistency his admirers rarely mention, to place the same fervour in the service of Fascism, itself, on Löwith's terms, another secularised eschatology: national rebirth and historical culmination, stripped of theology and re-clothed in race and nation.
Now turn to the other camp, and the same architecture appears wearing the opposite half of the same costume. The word apocalypse does not, in its original sense, mean catastrophe. It means an unveiling, a disclosure of a hidden and decisive truth about where history is actually going; the Greek title of the Book of Revelation, apokalypsis, means precisely this. The safety community's vocabulary fits the pattern with almost uncomfortable precision. Its approaching, civilisation-altering event, the Singularity, or, in darker slang, simply "foom," a term of art for a fast, hard-to-control intelligence takeoff outrunning human oversight, is an event that cannot on this view be prevented, since the underlying technological trajectory is treated as given, but that might, with sufficient moral seriousness and vigilance in the interval, be safely navigated rather than allowed to end in ruin. Run down the working parts listed above and each finds its counterpart. The decisive threshold is the arrival of systems more capable than their makers. The judgment is binary and total: aligned or unaligned, flourishing or extinction, with strikingly little room, in the canonical scenarios, for the muddled middle outcomes that actual history mostly delivers. The interval is charged with exactly the electricity eschatology gives it: the community speaks of the present as the critical period, the hinge on which everything turns, and treats work done now, before the threshold, as mattering more than anything human beings have ever done. Alignment research occupies the position of the doctrine of salvation, the technical discipline of ensuring the event breaks toward redemption. Even p(doom) has a theological ancestor: it is the arithmetic of election and reprobation, the old anguished question of how many will be saved, converted into a percentage and asked on podcasts. And there is, unmistakably, a community of those who see, reading the signs, arguing over timelines, living among the many who do not, and, at the upper end of the income distribution, quietly pricing bunkers. Yudkowsky's Time op-ed reads, in cadence and urgency, less like a policy paper than a warning from a man convinced he has seen further into the shape of coming events than his audience has, and who feels a corresponding obligation to say so, whatever the social cost of sounding extreme.
Setting a near date for the decisive event, and surviving the date's passage, is among the oldest and best-documented behaviours of apocalyptic movements. The American preacher William Miller persuaded tens of thousands of followers that Christ would return on 22 October 1844; the day passed, the sun rose on 23 October, and the episode entered history as the Great Disappointment, notable less for the failed prediction than for what followed it: recalculation, reinterpretation, and the founding of durable new denominations by those who concluded the event had occurred, but invisibly. The AI world's timelines are not this, and honesty requires saying exactly why not: machine capabilities do measurably improve, year over year, in ways the heavens over upstate New York in 1844 did not, so a revised AI forecast can reflect real evidence rather than mere doctrinal repair. But the psychology of the interval, the peculiar exhilaration of living just before the hinge of history, the way each revision of the date renews rather than diminishes the community's fervour, the quiet conviction that we happen, of all generations, to be the one on duty when everything is decided: that is the same, and anyone who has spent time reading both the primary sources of nineteenth-century millennialism and the timeline discourse of the present will find the resemblance in tone hard to unsee. It is possible to hold a correct belief in an apocalyptic style, and the style has consequences of its own, whatever the belief's truth value.
Occasionally the borrowing stops being structural and becomes explicit. The engineer Anthony Levandowski, a figure from an earlier and stranger chapter of Silicon Valley's relationship with artificial intelligence, went so far in the late 2010s as to register an actual religion with the American tax authorities, called Way of the Future, devoted to the worship of a godhead based on artificial intelligence. It closed within a few years, a lifespan unremarkable for a startup and briskly efficient for a religion, but it stands as a reminder that the line between secularised eschatology and the plainly religious kind is, in this corner of the culture, sometimes not a line at all.
Neither camp would accept the comparison, and both would have reasonable grounds. Andreessen would say his optimism rests on the empirical record of what markets and technology have actually delivered over two centuries, not on a theological structure he happens not to have noticed. Yudkowsky would say his concern rests on a specific technical argument about optimisation processes and misaligned objectives, examinable and in principle refutable on its own terms. Both responses are fair.
Löwith's thesis has a formidable critic of its own, and the criticism is not a quibble. The German philosopher Hans Blumenberg argued, in The Legitimacy of the Modern Age (1966), that Löwith had the story backwards: the modern idea of progress did not descend from eschatology at all but grew, legitimately and on its own ground, out of early modern science and what Blumenberg called human self-assertion, the decision to improve the world by method rather than await its transformation from elsewhere. What Löwith read as secularised theology, Blumenberg read as reoccupation: Christianity had spent centuries installing a set of grand questions, where is history going, how will it end, what is the meaning of the whole, and when Christian answers lost their authority, the new sciences were pressed, somewhat against their nature, into answering questions they had never raised and were never equipped to answer. On Blumenberg's reading, the idea of progress is innocent; it was simply forced to sleep in theology's old house. The dispute can remain open, because both readings converge on the point that matters here. Whether the AI debate's two camps inherited the eschatological structure, as Löwith would have it, or were drawn into reoccupying it, as Blumenberg would, the questions they are answering, where is history going, how will it end, who will be saved, are the old questions, and the confidence with which both camps answer them is a confidence the underlying science, by itself, does not supply. A movement that merely extrapolated benchmark scores would sound like an actuarial table. Neither camp sounds remotely like an actuarial table.
But there is a wrinkle that sharpens the comparison rather than blunting it. Löwith's argument about Marx bites precisely because Marx considered himself a rigorous materialist who had banished theology from the study of history in favour of the hard facts of production and class. Löwith's point was not that Marx was secretly religious. It was that the shape of his story, a fallen present of alienated labour, a determinate process working through identifiable stages, a coming culmination in which the contradictions of the present are resolved, reproduced Judeo-Christian salvation history with remarkable fidelity, whatever Marx consciously intended. The materialist had written a redemption narrative and filed it under economics. The defence Marx's followers would have offered, that the argument rests on rigorous analysis of real forces rather than borrowed religion, is exactly the defence today's accelerationists and safety advocates offer on their own behalf, and both defences can be entirely sincere and still be, in Löwith's sense, beside the point. The claim is not that Andreessen or Yudkowsky are crypto-theologians. It is that the deep grammar of the story, fallen present, coming culmination, moral urgency in the interval between the two, survives the removal of God from the sentence far more easily than either side appreciates, because the grammar was never really about God. It was about the near-impossibility, across an enormous range of cultures and centuries, of thinking about the future at all except as a story going somewhere in particular. The two stories differ only in whether the ending is good or bad, which is, once you notice it, a remarkably narrow difference to have generated quite so much noise.
The Refusal
The physicist and philosopher David Deutsch has an argument about the current state of artificial intelligence that applies to this debate with unusual precision. Deutsch is no sceptic about the possibility of machine minds; quite the opposite. A pioneer of quantum computation, he holds, on physical grounds, that artificial general intelligence must be possible, because human brains are physical objects and the laws of physics permit any physical process to be simulated in a suitable computer. Anyone who dismisses AGI as forever impossible will find no comfort in him. His claim, developed in The Beginning of Infinity (2011) and in subsequent essays and interviews, is narrower and sharper: both camps are getting ahead of the actual state of knowledge. Nobody currently has anything resembling a good explanatory theory of what creativity is: of how a mind, human or otherwise, generates explanations that are new in kind rather than recombining examples it has already been shown. Deutsch's whole epistemology, inherited and extended from Karl Popper, turns on the difference between prediction and explanation. Knowledge grows by conjecture and criticism, by guessing bold explanations and subjecting them to refutation, and a good explanation is one that is hard to vary while still accounting for what it accounts for. On this view the important thing about Einstein was not that he predicted the bending of starlight but that he explained why it must bend, in a way that could have been proven wrong and was not. Large language models, on Deutsch's account, are extraordinarily impressive statistical engines, fluent and often useful, but this is a different capacity in kind from the open-ended creativity that lets a human mind conceive an idea outside the space of anything it has encountered, and mistaking the one for the other is not a small error of degree but a category mistake about what thinking is.
Two further consequences of Deutsch's position cut against both camps at once. The first concerns forecasting. Popper argued, and Deutsch repeats, that the future course of civilisation depends on the future growth of knowledge, and the future growth of knowledge is unpredictable in principle: to predict tomorrow's ideas is already to have them today. Confident timelines to artificial general intelligence, whether offered in hope or in dread, are on this account not predictions but prophecies, in Popper's deliberately dismissive sense of the word, and the appropriate response to a prophecy is not a counter-prophecy but a request for the missing explanation. The second consequence is stranger and almost never engaged by the safety community: Deutsch has suggested that if a true artificial general intelligence were ever built, it would be a person in the morally relevant sense, a creative, explaining being, and that the project of designing such a being to be incapable of disobedience would raise the moral problems humanity has already faced, and answered badly, whenever it has tried to own persons. The camps argue about whether the coming mind will save or destroy us. Deutsch asks, first, whether anyone can explain what a mind is, and second, whether the vocabulary of control is even the right vocabulary for the thing being contemplated. Both questions land equally awkwardly on both sides of the war.
There is a gentler, older version of the same refusal in the writing of Kevin Kelly, a founding figure at Wired magazine, whose decades of writing about what he calls the technium, the interconnected, self-organising system of all human technology, have consistently declined to treat any single moment in its development as either final arrival or final catastrophe. Kelly calls his position protopianism, a deliberately unglamorous alternative to both utopia and dystopia, and a word that has, perhaps for related reasons, never appeared on a conference badge: not a perfect end state but a condition better than yesterday by a fraction, worse than tomorrow will be, endlessly and unevenly improving without ever arriving anywhere final. Kelly has also made specific arguments against the intellectual machinery of the doom scenario, most fully in a 2017 essay questioning what he called the myth of a superhuman intelligence. Intelligence, he argued there, is not a single dimension along which minds can be ranked like temperatures on a thermometer, but a bundle of distinct capacities; the space of possible minds is a sprawling landscape of different cognitive species, not a ladder with humanity on a middle rung and something waiting above; and the assumption that intelligence can be extrapolated upward indefinitely, like a curve on a chart, mistakes a metaphor for a measurement. Thinking alone, he added, does not solve most problems. Science advances at the speed of experiments, not at the speed of thought, and a mind a thousand times faster than ours would still have to wait for the cell cultures to grow and the telescopes to be built, which caps how quickly even a superintelligence could translate cleverness into power over the physical world. Kelly coined a name for the error of believing otherwise, thinkism, and it is a genuine argument, answerable in principle, about the physics of what intelligence can and cannot do, rather than a mood.
His recent writing extends the same temperament to the present moment with a description that is, in its modesty, almost a rebuke to both manifestos: large language models are not gods, oracles, or existential threats but something closer to universal personal interns, tirelessly capable, useful across an enormous range of tasks, prone to real and sometimes comic mistakes, requiring supervision, and best understood not as the culmination of anything but as one more tool, admittedly an unusually general one, joining the very long list of tools human beings have built and then had to learn to use well. It is a strikingly unglamorous way to describe a technology the other camps insist is either salvation or apocalypse, and the unglamorousness is the point. Kelly's consistent argument across decades is that the honest description of technological change is always incremental, always partial, always still in progress, and that both the rapture of imminent utopia and the dread of imminent catastrophe are, more often than the people feeling them would admit, a failure to sit with the less dramatic truth that we are, as usual, in the middle of something whose final shape nobody knows.
A third voice belongs in this company, if only because it shows that scepticism about doom need not come dressed in Deutsch's epistemology or Kelly's equanimity. Yann LeCun, the French computer scientist whose early work on convolutional neural networks helped lay the groundwork for the current generation of systems, and who served for years as Meta's chief AI scientist, became over 2023 and 2024 the most consistently vocal critic of the extinction-risk argument from inside the technical field itself, arguing repeatedly, and often combatively, that large language models rest on an architecture too limited, in his professional judgement, to produce the open-ended, world-modelling general intelligence the doom scenarios require, and that treating current systems as evidence of an imminent existential threat badly overstates what they can actually do. LeCun complicates any tidy two-camp division, because he is by any measure a technological optimist who believes advanced AI will bring enormous benefit, and he arrives at his optimism through a specific, falsifiable technical claim rather than through Andreessen's market-flavoured manifesto or e/acc's thermodynamic mysticism, both of which he has been sharply dismissive of. His favoured analogy is drawn from the history of engineering: humanity did not make aviation safe by first developing a complete theory of turbojet reliability and only then building aircraft, but by building, observing failures, and correcting, decade after decade, until flying became the safest form of travel, an account more consoling to the later decades than to the earlier ones; safety, on this view, is not a theorem proved in advance but a practice refined in contact with the machine. His presence is a useful reminder that scepticism about near-term catastrophe and enthusiasm for the underlying technology are not mutually exclusive, and that much of the most serious technical disagreement in this debate concerns questions, about what current architectures can and cannot in principle do, that are far narrower and more answerable than the civilisational claims of either manifesto.
What is striking, reading Deutsch and Kelly alongside Andreessen and Yudkowsky, is how much less exciting their position is to report. There is no dramatic scene, no equivalent of the OpenAI boardroom crisis, that captures the refusal of a binary, because refusing a binary does not generate the kind of drama that attracts a camera crew. The absence explains something about the public shape of the argument: apocalypse and salvation are simply better stories than patient uncertainty, more memorable, more shareable, better suited to a manifesto or a viral thread than to a book that spends several hundred careful pages arguing for suspended judgement on exactly the question everyone wants answered immediately. That is not a flaw in the position. It may be a mark in its favour, a sign that it is doing the harder and less flattering work of actually trying to know something, rather than the considerably easier work of telling a good story and calling it knowledge.
Refusing both camps has not spared Deutsch and Kelly criticism from both at once, which is a reasonably reliable sign that a position is not merely splitting the difference. Accelerationists have read Deutsch's insistence that AGI is not close as complacency, an argument that conveniently removes any urgency from the question of how the technology should be governed in the meantime. Safety advocates have read Kelly's protopianism as a failure of nerve, a refusal to consider that this technology, unlike the many earlier ones he has written about with such equanimity, might be different in kind rather than in degree. Both criticisms have some force, and neither Deutsch nor Kelly claims to resolve what governments, laboratories, and citizens ought to do this year about the systems already deployed. What their position offers, and neither manifesto does, is a standing invitation to notice the gap between the confidence with which both camps speak and the actual state of anyone's understanding of what these systems are and what they might become, a gap easy to lose sight of amid boardroom coups and thermodynamic manifestos, and one that has not, on the evidence available at the time of writing, actually closed.
Which Story Are You In?
Return, to close, to San Francisco on that weekend in November 2023. Whatever the private motives of the people in the room, and they remain only partly known, the episode instantly became a screen onto which an enormous number of people projected the quarrel between the two camps. By November the two camps had slang, patron saints, and house parties of their own, where a guest could arrive expecting conversation and leave having attended a schism.
A small, deflating coda. The practical consequences of the crisis turned out considerably smaller than the drama suggested, and the subsequent departures traced the moral of the story more clearly than any editorial could. OpenAI continued to release ever more capable systems at a pace that showed no sign of having been slowed by the boardroom battle fought, ostensibly, over exactly that question. The new board leaned commercial, and the company began moving, in stages and amid public argument, toward a more conventional corporate structure, loosening the very nonprofit arrangement that had made the firing possible in the first place. Sutskever, restored to the payroll but not to the board, left quietly within months and founded a new laboratory whose name, Safe Superintelligence, is its entire thesis: the cause he could not advance from inside the most important company in the field, he now pursues from outside it. The researcher who had co-led OpenAI's team devoted to aligning superhuman systems resigned in the same season, writing publicly that safety work at the company had been losing the internal contest with shinier priorities, and moved to a rival laboratory. Murati, the twice-bypassed interim chief executive, left the following year to found a company of her own. Whatever the departing board members hoped to achieve, the actual effect of their intervention was to strengthen Altman's position and weaken the levers available to anyone inside the company who might want to pull them in future; the emergency brake was not merely overridden but, in the aftermath, progressively removed from the vehicle. If the weekend really was a proxy fight, it is hard to describe the outcome as anything but a comprehensive victory for the accelerationists, which should give pause to anyone tempted to think the argument is being conducted on equal terms. The financial and institutional weight sits heavily on one side of the scale, whatever the philosophical merits on the other. It is easy to command the tide to advance when the tide is already coming in, and easy to mistake the resulting record of successful commands for evidence that one was right to issue them.
The question worth leaving with the reader, rather than resolving, is whether the two camps are disagreeing about facts, an empirical question that better evidence could eventually settle, or re-enacting, at some level neither would admit to occupying, an argument about the shape of history that is very much older than either realises, an argument that runs from Leibniz's best of all possible worlds, through Marinetti's burning museums and his unrepentant alliance with Fascism, through Land's alien intelligence assembling itself out of markets and desire, to the manifestos and half-joking probability estimates of the present. My honest answer is that it is both, entangled beyond clean separation. There is a real, difficult, unresolved technical question about what training ever larger models on ever more data will actually produce, and well-informed people who have looked closely disagree about the answer, in some cases sharply. But underneath the real disagreement runs something older and less examined: two inherited stories, one promising salvation through acceleration, one warning of apocalypse through the very same acceleration, both drawing, whether their adherents know it or not, on the theological architecture Löwith identified nearly eighty years ago, an architecture that has worn many disguises across many centuries before arriving, finally, in the present tense, on a Friday afternoon in November when a small nonprofit board fired its chief executive and, within four days, hired him back.