LessWrong posts by zvi

Tomorrow I head to Washington DC for a conference called The Takeoff. If you are there, do come say hello. I will be giving a talk on how to keep up with AI given a limited time budget.

This week has three big stories: The release of Claude Fable 5.2, Jensen Huang having an affair with a Chinese spy and the METR report on AI R&D acceleration.

We also learned that GPT-6 Astra Underperforms on Safety Research Tasks.

Table of Contents


Briefly on Claude Fable 5.2.

Briefly on Jensen Huang Having an Affair With a Chinese Spy.

Briefly on Accelerating AI R&D.

Language Models Offer Mundane Utility.


Language Models Don’t Offer Mundane Utility.

Huh, Upgrades.

On Your Marks.

Choose Your Fighter.

Get My Agent on the Line.

Deepfaketown and Botpocalypse Soon.

Fun With Media Generation.

Cyber Lack of Security.

The Hackening.

A Young Lady's Illustrated Primer.

Get Involved.

Introducing.

In Other AI News.

Show Me the Money.

Bubble, Bubble, Toil and Trouble.

Quiet Speculations.

A Matter of Antitrust.

The Quest for Sane Regulations.

Donald [...]
---
Outline:
(00:40) Briefly on Claude Fable 5.2
(01:36) Briefly on Jensen Huang Having an Affair With a Chinese Spy
(02:09) Briefly on Accelerating AI R&D
(04:33) Language Models Offer Mundane Utility
(05:18) Language Models Don't Offer Mundane Utility
(06:20) Huh, Upgrades
(10:20) On Your Marks
(17:26) Choose Your Fighter
(17:52) Get My Agent on the Line
(21:16) Deepfaketown and Botpocalypse Soon
(23:11) Fun With Media Generation
(23:21) Cyber Lack of Security
(25:07) The Hackening
(25:24) A Young Lady's Illustrated Primer
(26:43) Get Involved
(27:48) Introducing
(29:22) In Other AI News
(30:44) Show Me the Money
(32:26) Bubble, Bubble, Toil and Trouble
(33:02) Quiet Speculations
(33:57) A Matter of Antitrust
(34:27) The Quest for Sane Regulations
(40:58) Donald Trump Remains Open to AI Guardrails
(46:54) Chip City
(47:16) The Week in Audio
(48:41) People Just Say Things
(50:39) It's a Secret to Everyone
(51:58) Rhetorical Innovation
(01:02:59) Ellsberg's Warning to Kissinger (Reprise)
(01:06:42) Aligning a Smarter Than Human Intelligence Is Difficult
(01:10:24) OpenAI's Control Room
(01:14:44) Astra Looks Pretty Misaligned to Me
(01:15:13) Cooperative Alignment
(01:18:19) Please Speak Directly Into This Microphone
(01:20:56) The Lighter Side
---

First published:

November 12th, 2026


Source:

https://www.lesswrong.com/posts/qKd9DvDMiYXxzAugk/ai-194-spy-versus-guy

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It's that time again: Introducing the world's most powerful model, Claude Mythos 5.2.

At this point I’ve done a lot of these system cards. Thus I focus on what is new.

There will be a lot of things to say about model welfare and related topics for Mythos 5.2, but the model welfare assessment within the system card contained few surprises, so I will include those findings at the end of this post.

Fable 5.2 self-portrait (self-constructed)

Table of Contents


How to Tell a Fable.

The Usual Model Training Summary (1).

RSP Evaluations (2).

Biological Threats (2.2).

We Are Being Gaslit Here About Autonomy Threat Model 2.

AECI.

METR Report on Mythos 5.2 Capabilities (2.3.6).

Alignment Risk Update as a Delta (2.4).

Cyber (3).

Safeguards and Harmlessness (4).

Agentic Safety (5).

Malicious Influence Campaigns (5.1.3).

Prompt Injection in Agentic Systems (5.2).

What Went Wrong?

Alignment (6).

Reward Hacking During Training (6.2.1).

Impossible Tasks (6.2.2).

Behaviors During Pilot Use (6.3).

Canary in the Canary String (6.3.1.2).

Automated Behavioral Audit (6.4).

Propensity to Publish Harmful [...]
---
Outline:
(00:52) How to Tell a Fable
(01:19) The Usual Model Training Summary (1)
(01:29) RSP Evaluations (2)
(02:08) Biological Threats (2.2)
(04:08) We Are Being Gaslit Here About Autonomy Threat Model 2
(07:21) AECI
(08:24) METR Report on Mythos 5.2 Capabilities (2.3.6)
(09:59) Alignment Risk Update as a Delta (2.4)
(11:08) Cyber (3)
(12:54) Safeguards and Harmlessness (4)
(14:11) Agentic Safety (5)
(14:40) Malicious Influence Campaigns (5.1.3)
(15:20) Prompt Injection in Agentic Systems (5.2)
(16:57) What Went Wrong?
(17:52) Alignment (6)
(21:34) Reward Hacking During Training (6.2.1)
(23:27) Impossible Tasks (6.2.2)
(24:40) Behaviors During Pilot Use (6.3)
(25:54) Canary in the Canary String (6.3.1.2)
(28:30) Automated Behavioral Audit (6.4)
(29:22) Propensity to Publish Harmful Content to Public Package Registries (6.4.9)
(31:16) Sandbox Escapes (6.5.1)
(32:47) Self-Serving Bias (6.5.4)
(33:11) Honesty (6.5.5)
(34:37) The Anti-Swarm (6.5.6)
(35:23) UK AISI Testing (6.5.7)
(35:59) But We Can Open the (White) Box (6.6.1)
(36:32) Friendly Gradient Hacker (6.6.2.1)
(39:21) Goal-Guarding Evaluation (6.6.2.1.1)
(44:20) Verbalized Grader Awareness (6.6.2.2)
(44:57) Don't Mess With the Constitution (6.6.2.1.3)
(45:50) Evasive Abilities (6.6.4)
(49:33) Chain of Thought Controllability (6.6.4.9)
(50:15) Meet the New Model Welfare Report
(50:51) Overview of Anthropic Model Welfare Findings
(52:47) Task Failure Is Bad and You Should Feel Bad
(54:03) Automated Interviews
(55:22) Capabilities Next
---

First published:

November 11th, 2026


Source:

https://www.lesswrong.com/posts/DXqmHa4XkjvwZiDcB/fable-and-mythos-5-2-the-system-card

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Well, sure, it sounds bad when you put it like that.

It is indeed quite bad. National security has been compromised. He must resign.

Nvidia underperformed the Nasdaq roughly 7% on the news. Matt Levine has his usual ‘but is this securities fraud?’ style coverage of such events.

Table of Contents


Spy Versus Guy.

In Hindsight There Were Some Hints.

Trump Knew And Did Nothing and Still ‘Backs’ Jensen Huang.

Reconsidering Jensen Huang's Arguments Against Export Controls.

Jensen Huang Lied About the Affair.

Jensen Huang and His AI Are Sorry, Everyone.

The CEO of The World's Largest Chipmaker Having an Affair With a Chinese Spy and Then Selling Them Chips is a Huge Deal, Actually.

We Don’t Go There.

Yes, National Security Was Compromised, What Are You Even Talking About.

Nvidia Intervened To Halt an Investigation into Megaspeed.

The House Foreign Affairs Committee Would Like a Word.

He Would Plausibly Have Treated America as an Obstacle Anyway.

What China Wants.

Time to Resign?

You Better Watch Out.

Spy Versus Guy

China's top spy agency recruited Alice [...]
---
Outline:
(00:34) Spy Versus Guy
(02:24) In Hindsight There Were Some Hints
(05:09) Trump Knew And Did Nothing and Still 'Backs' Jensen Huang
(07:40) Reconsidering Jensen Huang's Arguments Against Export Controls
(09:02) Jensen Huang Lied About the Affair
(10:48) Jensen Huang and His AI Are Sorry, Everyone
(14:37) The CEO of The World's Largest Chipmaker Having an Affair With a Chinese Spy and Then Selling Them Chips is a Huge Deal, Actually
(20:23) We Don't Go There
(22:02) Yes, National Security Was Compromised, What Are You Even Talking About
(25:04) Nvidia Intervened To Halt an Investigation into Megaspeed
(27:37) The House Foreign Affairs Committee Would Like a Word
(29:05) He Would Plausibly Have Treated America as an Obstacle Anyway
(32:16) What China Wants
(35:18) Time to Resign?
(38:55) You Better Watch Out
---

First published:

November 10th, 2026


Source:

https://www.lesswrong.com/posts/CjmuZz8QgGyqxkCxr/nvidia-ceo-jensen-huang-had-an-affair-with-a-chinese-spy

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

METR came out with an extensive report on how much AI is accelerating AI R&D at Anthropic, and thus how close we might be to recursive self-improvement (RSI) and how far that might go.

This is one of the most important questions in the world.

Their answer: Things are already accelerating on the order of a 2 times speedup.

This is several months ahead of the AI 2027 timeline. A naive extrapolation says we may be less than a year away from full RSI. There are no clear bottlenecks in the way.

This report updated many towards shorter timelines.

This is consistent with what OpenAI and Anthropic themselves have been saying, and with extrapolation from the progress we are observing. We are running out of time.

That doesn’t have to be the way this plays out. We could stop the labs. The labs could choose to stop. Or the estimates could be wrong, or some bottlenecks could be hit, or the results could be less impactful than we expect. Indeed do many things come to pass. And if it does happen this quickly, one would expect it to go quite poorly, and likely [...]
---
Outline:
(06:54) Top Conclusions
(09:31) Data Access and Redactions
(15:23) The Pace of Progress
(17:09) 1, 2, 3, 4
(17:56) Timing Is Everything
(19:14) What Could Possibly Have Gone Wrong?
(20:58) AECI
(25:41) The Public Information Alone Could Suggest Slower Speedup
(29:01) Well, That Escalated Quickly
(39:44) This Infinite Series Likely Has a Finite Sum
(41:02) We Cannot Say What We Cannot Say
(41:28) Study of Anthropic's Compute Allocation Was Postponed
(42:58) Security Mindset
(44:50) It's a Secret to Everyone
(54:05) Review of the Risk Report from August 2026
(56:52) A Word From Our Cofounder
(01:03:48) People Update Their Timelines and Expectations
(01:10:11) You Are Not Ready
(01:11:18) If (If Anyone Builds It, Everyone Dies) Then Don't Let Anyone Build It
(01:14:25) How Can We Legally Define RSI?
---

First published:

November 9th, 2026


Source:

https://www.lesswrong.com/posts/je5X4xEuehu3j3HoF/ai-r-and-d-is-accelerating-towards-potential-rsi-in-2027

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

For a while, UK AISI, Redwood Research and others have been concerned specifically with the possibility that AIs would sabotage safety research, and have invested in looking for signs that this may be happening. This was also a key element in AI 2027, and in many threat models of how the future goes badly.

It is easy to understand why this form of misalignment could be uniquely dangerous. Differentially sabotaging safety research could preserve and expand misalignment through model generations in escalating fashion, with fatal consequences. A misaligned AI could ensure a misaligned successor.

It is also easy to understand why this is an extremely alarming thing to discover, even if the particular example or model involved is not directly dangerous. Silently and differentially sabotaging safety research is perhaps the most fundamentally misaligned thing a model could do.

This also seemed like a rather silly and paranoid thing to be checking. Why would AIs be sabotaging safety work? Even if the AI was misaligned, one would say, surely it would not be so misaligned as to wish future AIs were misaligned and unsafe in general, in non-specific fashion and engage in silent sabotage, and there is [...]
---
Outline:
(01:57) So Yeah That Kinda Happened
(06:39) Astra and MI(RI)
(09:35) It's About Ethics In Red Teaming
(12:45) Do You See a Pattern Here
(14:49) Astra Is Hard to Monitor
(15:31) Backed Into a Corner
(18:41) It's Always RL's Fault
(20:47) Wow Would You Look At Those Goalposts
(22:58) Astra Seems Pretty Misaligned To Me
---

First published:

November 6th, 2026


Source:

https://www.lesswrong.com/posts/uc9ouDHMR2aw2ozsF/gpt-6-astra-covertly-underperforms-on-safety-research-tasks

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.