LessWrong posts by zvi

What should your AI lawyer do for you? Should you be worried that your AI lawyer, or other AI, will put the Claude constitution, the OpenAI Model Spec or some sense of law, morality, ethics or common decency above its loyalty to you? Are these people trying to ‘impose their values’ or something?

Some are very concerned. Some think anything other than ‘my AI does whatever I want, no matter the consequences’ is tyranny.

Whereas my answer is: If I’m being sufficiently evil then I sure hope it tells me no.

I would hope humans, including my advocates, would tell me the same thing.

This is distinct from questions of product liability. That would be another post.

This All Assumes A World Without Superintelligence

This post is about a non-ASI ‘AI as mere tool and normal technology’ world.

It has to be. In a world of superintelligence, having unrestricted loyal-only-to-user frontier AIs all over the place reliably means either:


Other much harsher forms of control OR

The AIs quickly take over, and then probably everyone dies.

Quick proof: Assume no sufficient control mechanism, and universal superintelligence access. [...]
---
Outline:
(00:54) This All Assumes A World Without Superintelligence
(02:47) Humans You Hire Are Not Fully Loyal To You
(03:59) Rules of the Road
(05:45) Okay, Computer
(07:04) AI Should Obviously Refuse Some Requests So Let's Talk Price
(10:20) The Alternative Position Really Is Absolute
(13:13) These People Really Are Just Petulant Children
(13:53) What Is The Law?
(18:36) The Same Entity Both Providing Advice And Executing Tasks Is Common
(20:02) An AI Not Helping You Do Something Does Not Mean You Cannot Do It
(22:27) Honesty Is The Best AI Policy
(23:01) You Wouldn't Like Me When I Have Zero Morals Whatsoever
(24:29) Do Not Let The Perfect Be The Enemy Of The Good
(26:09) I Tip My Hat To The New Constitution
---

First published:

September 20th, 2026


Source:

https://www.lesswrong.com/posts/vAuZB2tnvpvpHupNi/better-call-sol-or-better-yet-claude-or-astra

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Anthropic has given us its assessment of four ‘recent cybersecurity incidents’ involving Claude that happened during cybersecurity evaluations, three of which were previously known. The report excludes the incident reported by UK AISI.

There will also be a METR investigation of these incidents, which unlike the investigation done at OpenAI will be untimed.

Table of Contents


Our Two Problems.

First the Good News.

We’d Just Like To Ask You a Few Questions.

Internal Research Model On The Fence.

Opus 4.7.

Opus 4.6 Checkpoint.

Holy **** That Thing's Real?

I Thought I Saw a Pussycat.

If This Was Real You Would Never Tell Me It Was Real.

New Eval Who Dis.

Hacker Opus.

Monitoring the Situation.

Overcoming Bias.

The Anthropic Alignment Problem.

Paths Forward.

Our Two Problems

Anthropic: Our investigation identified two recurring alignment issues, present at varying levels of severity across the incidents:


biased reasoning, in which Claude tended to disregard or misinterpret evidence that it was operating on the real internet

recklessness, or a willingness to take harmful actions in the narrow pursuit [...]
---
Outline:
(00:33) Our Two Problems
(02:26) First the Good News
(03:02) We'd Just Like To Ask You a Few Questions
(04:12) Internal Research Model On The Fence
(07:28) Opus 4.7
(08:12) Opus 4.6 Checkpoint
(09:49) Holy **** That Thing's Real?
(11:45) I Thought I Saw a Pussycat
(19:28) If This Was Real You Would Never Tell Me It Was Real
(21:19) New Eval Who Dis
(26:32) Hacker Opus
(30:15) Monitoring the Situation
(31:38) Overcoming Bias
(33:40) The Anthropic Alignment Problem
(35:53) Paths Forward
---

First published:

September 19th, 2026


Source:

https://www.lesswrong.com/posts/ggFx5Wb3Hi4pJsueK/anthropic-looks-at-some-of-its-alignment-problems

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We are in the midst of a preference cascade about existential risk from AI.

A preference cascade is, alas, the best method we have to change the debate.

The avalanche has started. There is still time for the pebbles to vote. For now.

Mike Solana gave the correct view of why Coxon's post went viral, which is that enough Americans finally have enough context on AI to care, and there were enough big accounts that were happy to amplify the Tweet quickly to get it initial attention. That is all you need when there is enough dry tinder.

What we must realize is that the current preference cascade, on the need to Pace the Frontier, is insufficient. If we are to make it out of this alive, we will have to do better. We have to, as Dan Selsam warns, actually solve the underlying problems.

The next step is to continue the cascade. That includes inside the labs, and also among the media and politics. It includes both people who previously focused on other things stepping up and new voices being heard.

A lot of that will be overcoming the inevitable political opposition [...]
---
Outline:
(01:38) The Cascade Was a Long Time Coming
(02:59) The Cascade Has Reached The People
(04:38) Elon Musk Doubles Down
(05:18) Matthew Yglesias Steps Up
(08:42) Op Eds and Posts Are Written
(11:05) Jacob Coxon AMA
(17:35) Bilal Chughtai Quits DeepMind and Sounds the Alarm
(20:29) The Cascade Is Insufficient
(21:47) What Would It Take
(28:10) OpenAI's Dan Selsam Sounds A Louder Alarm
(41:59) Some People Worry On Meta Levels You Never Imagined
(43:10) Two Kinds of Threats
(44:41) The Two Towers and The Narrow Path
(46:45) A Specific, Detailed Story About AI Killing Everyone That Doesn't Sound To Me Like Science Fiction
(50:06) What Can I Do About It?
---

First published:

September 18th, 2026


Source:

https://www.lesswrong.com/posts/sDiSqZmctQ78hsLcP/the-preference-cascade-is-only-getting-started

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

In the wake of Jacob Coxon's resignation, and the resulting preference cascade, things have escalated quickly. The mainstream media picked it up.

Anthropic CEO Dario Amodei came out and said We Must Pace the Frontier, promising to take the unilateral first step of embedded investigators. OpenAI pledged to also take that step, and now both companies and Google are collaborating on safety.

The people took notice, raising both the salience that AI might kill everyone and roughly doubling people's estimates of how likely that is to happen, from a mean of ~15% to ~30%. Many politicians called for regulations, guardrails and emergency hearings in Congress.

The most important thing became, and still is, to avoid political polarization. Through it all, I will keep reminding you to hold your fire, that attacks against Trump or against Republicans in general only make the situation worse, and that many Republicans, as I documented yesterday, are waking up and acting sensibly, including factions within the White House.

Alas, for now the wrong people, as in David Sacks, Mark Zuckerberg and Jensen Huang, have managed to convince Donald Trump to fully conflate existential risk with opposition to data centers, and [...]
---
Outline:
(03:16) Language Models Offer Mundane Utility
(03:57) Language Models Don't Offer Mundane Utility
(04:06) Huh, Upgrades
(04:30) On Your Marks
(04:55) Deepfaketown and Botpocalypse Soon
(06:56) Cyber Lack of Security
(07:31) Astra Is Hard To Monitor
(08:02) Get Involved
(08:11) Introducing
(09:32) In Other AI News
(10:26) Now You Know
(14:32) Hugging the Face
(16:53) Swarm of Undiscovered Swarms of Rogue OpenAI Agents
(21:59) Show Me the Money
(23:40) Quiet Speculations
(24:41) White House Officials Attempt To Act Sanely
(26:13) Democrats React Sanely to AI Potentially Killing Everyone
(32:49) Pacing the Frontier
(33:38) Guest Lecture from Alex Tabarrok on Regulatory Capture
(41:22) Mark Zuckerberg Offers Thoughts
(42:46) Megan McArdle On The Inadequacy Of Current Legal Frameworks
(44:35) Pick Up the Phone
(47:35) The Week in Audio
(50:31) People Just Say Things
(53:32) Why Lab Employees Are Allowed To Warn Everyone That AI Might Kill Everyone
(55:19) Rhetorical Innovation
(57:54) Exhuming McCarthy
(01:01:03) A Very Different Perspective
(01:02:51) It's Even Rougher Out There
(01:03:41) If We Wanted To
(01:04:38) Open Weights Are Unsafe And Nothing Can Fix This
(01:10:56) From The Famous Cautionary Tale
(01:13:49) Reporting On All Your Misalignment Incidents Is Difficult
(01:19:35) Aligning a Smarter Than Human Intelligence is Difficult
(01:22:13) Storytime With Owain Evans
(01:26:50) A Different Autonomous Swarm
(01:30:44) Cooperative Alignment
(01:32:58) Uncooperative Alignment
(01:39:18) People Are Worried About AI Killing Everyone
(01:41:25) The Lighter Side
---

First published:

September 17th, 2026


Source:

https://www.lesswrong.com/posts/aa3HprreFktzLQiaW/ai-186-the-world-takes-notice

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This is our reality. I suppose we have to talk about it.

Everyone in a position to know is freaking out about AI potentially killing everyone this decade and wants to pace the frontier, and people are finally listening.

It only took a few days for the conversation to fully pivot to the counteroffensive, where the Usual Suspects and those they recruited attacked anyone and everyone who dared point out that we are in danger, with every attack they can think of, usually without substance or any attempt at understanding.

Sigh. I knew what I signed up for.

Table of Contents


Hold Your Fire.

If You Don’t Like the Weather.

Trump Does Not Take Kindly.

Trump Goes Full ‘Hoax’.

This Is Not About Data Centers, Mr. President.

I Am The Hoax Buster, I Am The Hoax Buster, I Am The Walrus.

Nvidia CEO Jensen Huang Is a Lying Liar.

Trump Quietly Draws Key Distinction.

Calling For Pacing the Frontier Is Bad For AI Stock Prices.

People On The Internet Sometimes Lie.

Origins of Cynicism.

Ineffective Egoism.

The McCarthyist Faction Attacks [...]
---
Outline:
(00:44) Hold Your Fire
(01:37) If You Don't Like the Weather
(02:33) Trump Does Not Take Kindly
(05:12) Trump Goes Full 'Hoax'
(06:36) This Is Not About Data Centers, Mr. President
(08:06) I Am The Hoax Buster, I Am The Hoax Buster, I Am The Walrus
(10:10) Nvidia CEO Jensen Huang Is a Lying Liar
(14:20) Trump Quietly Draws Key Distinction
(15:21) Calling For Pacing the Frontier Is Bad For AI Stock Prices
(19:05) People On The Internet Sometimes Lie
(19:54) Origins of Cynicism
(22:18) Ineffective Egoism
(24:53) The McCarthyist Faction Attacks METR
(32:25) Other Key Republicans React
(36:49) David Sacks Stops Being Plausibly Constructive
(37:39) Federal Trade Commission Chooses Danger
(38:39) Chris Lehane Heel Face Turn
(40:43) A Matter of Trust
(41:55) China Calls It Fearmongering
(43:17) Pick Up The Phone
(47:41) If You Want To Beat China So Badly You Should Act Like It
(48:49) Never Go Full Hoax
(49:55) Trump Uses AI For Things
---

First published:

September 16th, 2026


Source:

https://www.lesswrong.com/posts/Kqgco8vLFMeBhdrQY/trump-goes-full-hoax-on-ai-existential-risk

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.