Roko's Basilisk and Pascal's Mugging sparked intense debate in rationalist AI alignment circles last decade. While intriguing thought experiments, they rely on a fundamentally flawed form of decision theory that I argue we have a moral imperative to reject.
At their core, these scenarios privilege one specific possible future over all others based solely on its purported consequences. Roko's Basilisk posits an AI that punishes those who didn't help create it, while Pascal's Mugging promises an enormous reward for a small action. This myopic focus ignores the vast array of other possible outcomes, many potentially far more likely or impactful.
A sound decision theory must account for not just the magnitude of potential consequences, but also their probability. It's insufficient to claim an outcome would be extremely good or bad - we must consider how likely it is to occur given available evidence. Neither Roko's Basilisk nor Pascal's Mugging provide compelling evidence for their extraordinary claims. Basing major decisions on such flimsy foundations is deeply irrational.
Crucially, we must reject the notion that we should privilege a specific AI's demands simply because it could exist. There are countless possible AIs and countless possible futures. Choosing to ensure the creation of one that issues threats or bizarre ultimatums demonstrates a profound misunderstanding of rational decision-making.
Even if we accept the premise of such scenarios, fixating on one specific bad outcome and attempting to create it offers no guarantees. In the vast space of possible futures, we might inadvertently bring about an entirely different entity that punishes us anyway. This mirrors the flaw in Pascal's Wager, which privileges one specific theology while ignoring the infinite space of unprovable possible gods.
We are not obligated to cave to hypothetical entities we can choose not to create, especially when doing so may foreclose other, potentially far better outcomes. Moreover, by attempting to appease one imagined threat, we may be opening ourselves up to countless others we haven't even considered.
Instead, we need decision frameworks grounded in reality, accounting for the full spectrum of possible futures, and robust to uncertainty and incomplete information.