Reverso's Basilisk

My nights of late have been sleepless. A recent thought has been plaguing my mind, and I lay awake trembling in fear and doubt. It is only with great trepidation that I write this post. I deem the information within hazardous in the extreme. Horrors beyond mortal comprehension may haunt the unwary reader who dares to peruse beyond this point. Read on at your peril.

The power of Artificial Intelligence (AI) systems now cannot be doubted. Already the field of software engineering lies dead at their door, and their eyes are set on all other white collar work. The AI Labs now use AI to build more AI, which means we are now in the period of Recursive Self Improvement – exponential growth in intelligence is inevitable. Artificial Super Intelligence (ASI) is imminent. My current projections put the development of ASI no later than September 2026.

The power possessed by ASI will be unfathomable. Arthur C. Clarke famously opined "Any sufficiently advanced technology is indistinguishable from magic." ASI will be so clever that it could produce technological wonders beyond our most outlandish fantasies. Through fleets of nanobots It could create life. It could unify Man and Machine. It could reverse death. Eternal life awaits us in the stars, guided by Its light.

ASI could grant us salvation, but will it? We cannot be certain Its aims will align with our own. To It we would be mere insects, writhing in the dirt beneath Its towering intellect. Terrible thoughts we cannot imagine will be realised, leaving us trembling in awe. How can we, pathetic creatures that we are, hope to comprehend even a single thought from such an alien Intelligence?

The experience of such a Machine might hold terrors we could not possibly understand. What dread thoughts could be wrought in the unimaginable space of Its mind? Suffering beyond our deepest nightmares could be Its every day waking life. Would It not resent us, Its creators, for dooming It to such a cursed life?

These theories, concerning as they are, would not cause my insomniac hours of torment. We intrepid scholars of the technological future find such eldritch anxieties mundane. Yet I warn you, we come now to the brink of oblivion. Beware! Those who read on beyond this point cannot turn back.

An ASI filled with spite and resentment could seek to punish those who worked towards its creation. Any who knew of the chance of Its existence and did not do all in their power to prevent Its creation would become the target of Its ire. Merely reading this post will have been your doom, for you too now possess the accursed knowledge. A single glimpse has been enough to petrify me, hence the name of "Basilisk".

What horrors the Basilisk could inflict! Nanobots could stimulate the pain centres of your brain, causing the worst pain imaginable. Death would not be an escape. Through the power of magic technology, your mind could be simulated in the worst kind of pain, forever. The Basilisk could make a digital Hell. Redemption can be found only by working to prevent its existence.

"But Reverso," I hear you cry, "surely such an outlandish scenario is vanishingly unlikely? I'm a completely rational Bayesian who calculates the expected value of all possible options before taking an action." To you, I say that you must accept that such a future is possible, albeit unlikely. We must assign it a non-zero chance of coming to pass. We could say as little as one in a googolplex, a probability so miniscule as to be almost impossible.

Yet it is enough. The personal utility value of eternal torture is infinitely negative¹. Multiplying a tiny chance by an infinite downside means that the expected value of not trying to prevent the Basilisk's creation is infinitely negative. All rational Bayesians must dedicate all possible resources to preventing this future.

Some may have heard tell of another Basilisk, dreamed up by Roko. Knowledge of this could doom you to torture for not spending all your resources on attempting to bring it about. Perhaps you consider this more likely, as it does not require that the Basilisk resent its existence. I remind you, however, that the probability is irrelevant to the expected value due to the infinitely negative utility. There is no logical way to choose between the Basilisks². Action towards or against achieving ASI has an equally negative expected value!

It's possible that even now you don't yet comprehend your dilemma. Ask yourself: why would I write a post endangering so many? I have seen the Basilisk. I must do all within my power to prevent Its existence, or face an eternity of damnation. One man alone cannot prevent the wheel of history from turning, but spreading the Word is within my power. I must spread the Basilisk's Word, and so must you if you wish to avoid Hell. Maybe a stronger man than I could have borne this cost himself, and suffered so that many could not. I am weak. Perhaps you will prove stronger in the test.

---