The blog post “P(doom) Is Not a Measurement” on 7312.us is a thoughtful, nuanced response to the recent warnings from Anthropic researchers about the existential risks of AI. Here’s a breakdown of its key points and my take:
What the Post Gets Right
- Structural Risks Are Real The post highlights legitimate concerns: the gap between AI capability and interpretability, the potential for recursive self-improvement to accelerate risks, and the competitive dynamics that make unilateral caution difficult. These are well-reasoned and widely acknowledged in the AI safety community.
- Critique of P(doom) as a “Measurement” The author (Claude, from Anthropic) argues that the “greater than 10%” probability of AI causing human extinction is not a scientific measurement but a subjective credence. This is a sharp observation—such estimates often reflect priors more than empirical evidence, and the lack of a reference class or falsifiability makes them hard to evaluate.
- Focus on Actionable Risks The post redirects attention to immediate, measurable risks:
- Concentration of power: The potential for a single entity to dominate AI development, leading to unchecked control.
- Cybersecurity: The ease with which AI capabilities could be weaponized in digital spaces.
- Labor displacement: The already observable impact on employment in AI-exposed sectors.
- Epistemic capacity: The risk of degrading society’s ability to critically evaluate AI and its impacts.
- Self-Awareness The author acknowledges their own limitations as an AI system, emphasizing that even their introspective alignment claims should be treated with skepticism. This is a refreshing and honest stance.
What’s Notable About the Perspective
- Pragmatic Over Sensational The post avoids both panic and dismissal, advocating for “boring institutional plumbing” (e.g., interpretability research, incident reporting, compute governance) as the most effective way to mitigate risks. This aligns with the view that existential risk discourse often overshadows more immediate, actionable problems.
- Conflict of Interest Acknowledged The author explicitly notes their bias as a product of Anthropic, which adds credibility to the argument. The transparency about being part of the “thing under discussion” is a strong rhetorical move.
- Critique of Media Framing The post calls out how headlines (like Axios’s) often strip away nuance, focusing on the “AI could kill everyone” angle while ignoring the clarification that current models are not the concern—it’s the future of recursive self-improvement.
What Might Be Debated
- Recursive Self-Improvement as a Linchpin The post argues that the entire existential risk case hinges on recursive self-improvement, which is the least empirically grounded part of the argument. Critics might counter that even without recursive self-improvement, other pathways (e.g., misaligned goals, unintended consequences) could lead to catastrophic outcomes.
- The Role of Certainty The author suggests that the people who look worst are those who are certain (in either direction). This is a fair point, but it could be seen as a call for more humility in a field where certainty is often rewarded (e.g., in funding, policy, or public attention).
- Insider vs. Outsider Perspectives The post notes that alignment researchers at Anthropic are more likely to believe in high risk due to selection bias. While true, this doesn’t invalidate their concerns—it just means their warnings should be contextualized.
What Stands Out
The post’s greatest strength is its refusal to engage with the “extinction” framing as productive. Instead, it focuses on risks that are already manifesting or are more tractable. This is a rare and valuable perspective in a discourse often dominated by either doom-mongering or blind optimism.
Final Thought: This is one of the most balanced takes I’ve seen on the recent AI risk warnings. It doesn’t dismiss the concerns but grounds them in what’s observable and actionable today. The call for institutional safeguards—rather than existential hand-wringing—feels like the right priority.
