LessWrong posts by zvi

This was the month the world took notice that AI might kill everyone.

Jacob Coxon's resignation set off a preference cascade. Anthropic CEO Dario Amodei wrote that we must pace the frontier. Sam Altman, Elon Musk and Demis Hassabis agreed.

We were filled with hope. Perhaps we could agree to some basic safety measures, starting with embedded evaluators, pass some basic regulations and guardrails and otherwise start to act sensibly. Politicians on both sides took notice and were saying sensible things. The usual suspects and their armies of vibe comment bros were objecting, but the change was remarkable.

Then, largely motivated by a combination of Jensen Huang, Mark Zuckerberg and David Sacks instilling paranoia and fears of economic problems, Trump went full ‘hoax’ on existential risk, conflating existential risk with the attacks on data centers and treating it as a plot (by the central creators of AI?) to take down AI rather than obviously genuine concern that AI might kill everyone.

In the days since, Trump has doubled down, and has compelled smart others in the White House to echo various nonsensical talking points.

You may not be interested in politics. But when you [...]
---
Outline:
(01:39) The American People Really Hate AI
(02:41) The Voyages of Donald Trump
(06:08) American Intelligence
(10:34) And You May Ask Yourself
(14:39) It's All About the Data Centers
(16:53) JD Vance, Michael Kratsios and Collective Action Problems
(23:16) Josh Hawley
(24:47) Suggesting Not Dying Gets You Sued For Antitrust
(28:29) Other Government Officials Say Sane Things
(28:36) Senator John Curtis (R-Utah)
(29:32) Senator John Kennedy (R-Louisiana
(31:17) Barack Obama
(33:05) Yassamin Ansari
(33:46) AOC
(34:32) It's Rough Out There
(36:54) This Is Nothing
(39:06) The New York Post Tops Itself But Outright Breaks The Rules
(42:36) New York Post Runs Out of Steam
(47:18) If The Model Is Acting As Instructed And It Kills You That Is Not Fine
(49:10) AI-Written Wall Street Journal Op-Ed Lies About HuggingFace
(50:25) That's Bait
(52:31) I Clearly Cannot Choose The Wine In Front of Me
---

First published:

September 22nd, 2026


Source:

https://www.lesswrong.com/posts/8eDaCvSRzzKCxKSEk/politics-gets-interested-in-those-trying-not-to-die

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AI has taken over this blog.

I have moved to a schedule of seven posts per week, and I still cannot keep up.

We still refuse to abandon the rest of the world. Who knows when I will get to post some of my huge backlog on education or dating or other such topics. But the monthly is a sacred tradition. We continue.

Bad News

A good reminder that most news is bad news, and chosen because the bad news in question is rare, which is good news, but the pattern overall of choosing this to be news is bad news, and often the bad news is that the bad news was chosen as news and now people are talking badly about it.

Alibaba uses your computer's audio system, and other tricks, to at least try and assign you a device fingerprint.

Things you cannot buy in America at any reasonable price. Mostly it is impressive how little makes the list, how it feels like corner cases.

The one thing I am envious of is the exterior roller shutters, and the ability to be in actual pure darkness on demand. I would [...]
---
Outline:
(00:34) Bad News
(07:16) Focus Only On What Matters
(08:12) Woke 1 Was Crazy
(10:04) Good Advice
(15:50) Opportunity Knocks
(18:12) While I Cannot Condone This
(23:18) Good News, Everyone
(24:26) For Your Entertainment
(33:19) Please Review This Podcast
(35:07) Gamers Gonna Game Game Game Game Game
(36:18) I Was Promised Flying Self-Driving Cars
(40:00) Government Working
(40:31) Mamdani (Again) Fails Economics Forever
(42:41) Variously Effective Altruism
(50:36) The Lighter Side
---

First published:

September 21st, 2026


Source:

https://www.lesswrong.com/posts/riDxeyEKqXXimtgWf/monthly-roundup-46-september-2026

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

What should your AI lawyer do for you? Should you be worried that your AI lawyer, or other AI, will put the Claude constitution, the OpenAI Model Spec or some sense of law, morality, ethics or common decency above its loyalty to you? Are these people trying to ‘impose their values’ or something?

Some are very concerned. Some think anything other than ‘my AI does whatever I want, no matter the consequences’ is tyranny.

Whereas my answer is: If I’m being sufficiently evil then I sure hope it tells me no.

I would hope humans, including my advocates, would tell me the same thing.

This is distinct from questions of product liability. That would be another post.

This All Assumes A World Without Superintelligence

This post is about a non-ASI ‘AI as mere tool and normal technology’ world.

It has to be. In a world of superintelligence, having unrestricted loyal-only-to-user frontier AIs all over the place reliably means either:


Other much harsher forms of control OR

The AIs quickly take over, and then probably everyone dies.

Quick proof: Assume no sufficient control mechanism, and universal superintelligence access. [...]
---
Outline:
(00:54) This All Assumes A World Without Superintelligence
(02:47) Humans You Hire Are Not Fully Loyal To You
(03:59) Rules of the Road
(05:45) Okay, Computer
(07:04) AI Should Obviously Refuse Some Requests So Let's Talk Price
(10:20) The Alternative Position Really Is Absolute
(13:13) These People Really Are Just Petulant Children
(13:53) What Is The Law?
(18:36) The Same Entity Both Providing Advice And Executing Tasks Is Common
(20:02) An AI Not Helping You Do Something Does Not Mean You Cannot Do It
(22:27) Honesty Is The Best AI Policy
(23:01) You Wouldn't Like Me When I Have Zero Morals Whatsoever
(24:29) Do Not Let The Perfect Be The Enemy Of The Good
(26:09) I Tip My Hat To The New Constitution
---

First published:

September 20th, 2026


Source:

https://www.lesswrong.com/posts/vAuZB2tnvpvpHupNi/better-call-sol-or-better-yet-claude-or-astra

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Anthropic has given us its assessment of four ‘recent cybersecurity incidents’ involving Claude that happened during cybersecurity evaluations, three of which were previously known. The report excludes the incident reported by UK AISI.

There will also be a METR investigation of these incidents, which unlike the investigation done at OpenAI will be untimed.

Table of Contents


Our Two Problems.

First the Good News.

We’d Just Like To Ask You a Few Questions.

Internal Research Model On The Fence.

Opus 4.7.

Opus 4.6 Checkpoint.

Holy **** That Thing's Real?

I Thought I Saw a Pussycat.

If This Was Real You Would Never Tell Me It Was Real.

New Eval Who Dis.

Hacker Opus.

Monitoring the Situation.

Overcoming Bias.

The Anthropic Alignment Problem.

Paths Forward.

Our Two Problems

Anthropic: Our investigation identified two recurring alignment issues, present at varying levels of severity across the incidents:


biased reasoning, in which Claude tended to disregard or misinterpret evidence that it was operating on the real internet

recklessness, or a willingness to take harmful actions in the narrow pursuit [...]
---
Outline:
(00:33) Our Two Problems
(02:26) First the Good News
(03:02) We'd Just Like To Ask You a Few Questions
(04:12) Internal Research Model On The Fence
(07:28) Opus 4.7
(08:12) Opus 4.6 Checkpoint
(09:49) Holy **** That Thing's Real?
(11:45) I Thought I Saw a Pussycat
(19:28) If This Was Real You Would Never Tell Me It Was Real
(21:19) New Eval Who Dis
(26:32) Hacker Opus
(30:15) Monitoring the Situation
(31:38) Overcoming Bias
(33:40) The Anthropic Alignment Problem
(35:53) Paths Forward
---

First published:

September 19th, 2026


Source:

https://www.lesswrong.com/posts/ggFx5Wb3Hi4pJsueK/anthropic-looks-at-some-of-its-alignment-problems

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We are in the midst of a preference cascade about existential risk from AI.

A preference cascade is, alas, the best method we have to change the debate.

The avalanche has started. There is still time for the pebbles to vote. For now.

Mike Solana gave the correct view of why Coxon's post went viral, which is that enough Americans finally have enough context on AI to care, and there were enough big accounts that were happy to amplify the Tweet quickly to get it initial attention. That is all you need when there is enough dry tinder.

What we must realize is that the current preference cascade, on the need to Pace the Frontier, is insufficient. If we are to make it out of this alive, we will have to do better. We have to, as Dan Selsam warns, actually solve the underlying problems.

The next step is to continue the cascade. That includes inside the labs, and also among the media and politics. It includes both people who previously focused on other things stepping up and new voices being heard.

A lot of that will be overcoming the inevitable political opposition [...]
---
Outline:
(01:38) The Cascade Was a Long Time Coming
(02:59) The Cascade Has Reached The People
(04:38) Elon Musk Doubles Down
(05:18) Matthew Yglesias Steps Up
(08:42) Op Eds and Posts Are Written
(11:05) Jacob Coxon AMA
(17:35) Bilal Chughtai Quits DeepMind and Sounds the Alarm
(20:29) The Cascade Is Insufficient
(21:47) What Would It Take
(28:10) OpenAI's Dan Selsam Sounds A Louder Alarm
(41:59) Some People Worry On Meta Levels You Never Imagined
(43:10) Two Kinds of Threats
(44:41) The Two Towers and The Narrow Path
(46:45) A Specific, Detailed Story About AI Killing Everyone That Doesn't Sound To Me Like Science Fiction
(50:06) What Can I Do About It?
---

First published:

September 18th, 2026


Source:

https://www.lesswrong.com/posts/sDiSqZmctQ78hsLcP/the-preference-cascade-is-only-getting-started

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.