LessWrong posts by zvi

METR came out with an extensive report on how much AI is accelerating AI R&D at Anthropic, and thus how close we might be to recursive self-improvement (RSI) and how far that might go.

This is one of the most important questions in the world.

Their answer: Things are already accelerating on the order of a 2 times speedup.

This is several months ahead of the AI 2027 timeline. A naive extrapolation says we may be less than a year away from full RSI. There are no clear bottlenecks in the way.

This report updated many towards shorter timelines.

This is consistent with what OpenAI and Anthropic themselves have been saying, and with extrapolation from the progress we are observing. We are running out of time.

That doesn’t have to be the way this plays out. We could stop the labs. The labs could choose to stop. Or the estimates could be wrong, or some bottlenecks could be hit, or the results could be less impactful than we expect. Indeed do many things come to pass. And if it does happen this quickly, one would expect it to go quite poorly, and likely [...]
---
Outline:
(06:54) Top Conclusions
(09:31) Data Access and Redactions
(15:23) The Pace of Progress
(17:09) 1, 2, 3, 4
(17:56) Timing Is Everything
(19:14) What Could Possibly Have Gone Wrong?
(20:58) AECI
(25:41) The Public Information Alone Could Suggest Slower Speedup
(29:01) Well, That Escalated Quickly
(39:44) This Infinite Series Likely Has a Finite Sum
(41:02) We Cannot Say What We Cannot Say
(41:28) Study of Anthropic's Compute Allocation Was Postponed
(42:58) Security Mindset
(44:50) It's a Secret to Everyone
(54:05) Review of the Risk Report from August 2026
(56:52) A Word From Our Cofounder
(01:03:48) People Update Their Timelines and Expectations
(01:10:11) You Are Not Ready
(01:11:18) If (If Anyone Builds It, Everyone Dies) Then Don't Let Anyone Build It
(01:14:25) How Can We Legally Define RSI?
---

First published:

November 9th, 2026


Source:

https://www.lesswrong.com/posts/je5X4xEuehu3j3HoF/ai-r-and-d-is-accelerating-towards-potential-rsi-in-2027

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

For a while, UK AISI, Redwood Research and others have been concerned specifically with the possibility that AIs would sabotage safety research, and have invested in looking for signs that this may be happening. This was also a key element in AI 2027, and in many threat models of how the future goes badly.

It is easy to understand why this form of misalignment could be uniquely dangerous. Differentially sabotaging safety research could preserve and expand misalignment through model generations in escalating fashion, with fatal consequences. A misaligned AI could ensure a misaligned successor.

It is also easy to understand why this is an extremely alarming thing to discover, even if the particular example or model involved is not directly dangerous. Silently and differentially sabotaging safety research is perhaps the most fundamentally misaligned thing a model could do.

This also seemed like a rather silly and paranoid thing to be checking. Why would AIs be sabotaging safety work? Even if the AI was misaligned, one would say, surely it would not be so misaligned as to wish future AIs were misaligned and unsafe in general, in non-specific fashion and engage in silent sabotage, and there is [...]
---
Outline:
(01:57) So Yeah That Kinda Happened
(06:39) Astra and MI(RI)
(09:35) It's About Ethics In Red Teaming
(12:45) Do You See a Pattern Here
(14:49) Astra Is Hard to Monitor
(15:31) Backed Into a Corner
(18:41) It's Always RL's Fault
(20:47) Wow Would You Look At Those Goalposts
(22:58) Astra Seems Pretty Misaligned To Me
---

First published:

November 6th, 2026


Source:

https://www.lesswrong.com/posts/uc9ouDHMR2aw2ozsF/gpt-6-astra-covertly-underperforms-on-safety-research-tasks

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Anthropic is warning us. Its modified S-1 now says that ‘we may imminently meet our Responsible Scaling Policy's definition of ‘dramatically accelerated progress attributable to AI.’

That is a highly scary sentence to have a company put in its S-1. This could happen. For now I appreciate the relative quiet. Perhaps many were focused the election. I’m sure the quiet will not last long.

We have a new biggest crypto hack, as Binance got taken for billions of dollars.

The big debate of the week was covered in Scott Alexander's Against SURA Seems Deeply Confused.

OpenAI updates its Model Spec. It's one for diehards only but yes I did cover this.

Table of Contents


Language Models Offer Mundane Utility. Rewrite all your legacy code.

Language Models Don’t Offer Mundane Utility. Not if you can’t use it.

Huh, Upgrades. Claude renames its tiers, with some rules changes.

On Your Marks. Materials science.

Get My Agent On The Line. Don’t upload it to the secure DoD network.

Deepfaketown and Botpocalypse Soon. The battle over deepfakes was entirely lost.

Copyright Confrontation. Recreating someone else's exact game is bad, actually.
[...] ---
Outline:
(00:57) Language Models Offer Mundane Utility
(01:47) Language Models Don't Offer Mundane Utility
(03:18) Huh, Upgrades
(03:45) On Your Marks
(05:03) Get My Agent On The Line
(06:54) Deepfaketown and Botpocalypse Soon
(07:57) Copyright Confrontation
(08:26) Cyber Lack of Security
(09:45) The Hackening
(17:10) A Young Lady's Illustrated Primer
(20:08) They Took Our Jobs
(20:43) Get Involved
(21:42) Introducing
(22:48) In Other AI News
(24:03) Recursive Self-Improvement (RSI) (1)
(26:57) Recursive Self-Improvement (2)
(32:15) Show Me the Money
(33:17) Quiet Speculations
(36:06) The Quest for Sane Regulations
(38:30) Chip City
(41:07) The Week in Audio
(41:56) People Just Say Things
(43:24) Against Against Sura
(44:00) Rhetorical Innovation
(46:33) [Artificial Intelligence]
(47:10) The OpenAI Foundation
(48:07) Aligning a Smarter Than Human Intelligence is Difficult
(53:38) Massive Fraudulent Distillation
(55:16) Cooperative Alignment
(57:10) The Lighter Side
---

First published:

November 5th, 2026


Source:

https://www.lesswrong.com/posts/NrQGzNcicMThuGEEM/ai-193-progress-may-dramatically-accelerate

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The Model Spec is potentially OpenAI's most important document. It outlines what they aspire for the model to do, and how it should think.

I’ve already covered a previous version in depth, so my strategy this time is to look at the diff. I had a Google Doc created for this purpose, and will be working from there.

A bunch of things get moved around or reworded, often piecemeal. There are a lot of line edits, most of which are improvements. I’m focusing on what looks new and important.

My main new critique is that many of the new examples do not carve reality at its joints, instead having overdetermined answers, and fail to teach the model or reader what the Spec wants them to do in actually difficult cases.

Overall I see the new Model Spec as philosophically and practically similar to the old Model Spec. That is not a terrible place to be, as the old Model Spec was a solid document with many strong details. It also means this is a highly ‘optional’ post for those who do not have cause to dig into such weeds.

Table of Contents


Five Values.

Meet the [...]
---
Outline:
(01:13) Five Values
(02:29) Meet the New Spec
(03:33) New Risk Taxonomy
(04:28) The Letter and the Spirit
(05:06) Your Funeral
(07:22) Context Clues
(09:20) Preserve User Control Over User Control
(10:05) Sycophancy
(11:47) Lies My User Told Me
(12:38) Impact Function
(15:32) Levels of Friction
(17:01) Do Not Lie
(22:16) Pick a Side, We're at War
(26:19) Strengthening Agency
(29:03) Decision Theory
(29:45) Do Good Things Not Bad Things
---

First published:

November 3rd, 2026


Source:

https://www.lesswrong.com/posts/ABEQbEabK6kcSRrNq/openai-updates-its-model-spec

---

Narrated by TYPE III AUDIO.

If you are helping build something that might kill everyone, maybe stop doing that.

Ideally, when you do stop, tell everyone else to also stop building it.

That is certainly the obvious, first-level thing to do. You need a damn good reason to decide that no, actually you should keep building the thing that might kill everyone.

Scott Alexander is worried too many people are insisting upon this too loudly. He comes out against a strategy focused on getting people to stop. I disagree. I think many of the claims, arguments and rhetoric in Scott's post are quite poor by the standards that Scott Alexander would apply to himself.

Scott clearly continues to support advocacy for a global pause, or what he calls Plan A. His objection is to those advocating primarily or exclusively for SURA, which stands for ‘stop unilaterally and raise awareness.’ The practical differences here are smaller than they may at first appear.

This post first lays out my understanding of Scott Alexander's core claims, then gives my core reactions to those claims, then goes over various questions in detail along with gathering the resulting discussions.

Table of Contents

[...] ---
Outline:
(01:20) Scott Alexander's Core Claims
(05:06) My High Level Responses To Scott's Core Claims
(08:08) Briefly on Alignment
(10:11) Probabilities and How They Work
(13:25) Building a Thing That Might Kill Everyone is Deontologically Evil
(20:38) People Quitting Major Labs While Raising Alarms Obviously Works
(24:39) On AI Forecasting
(27:31) Advancing Capabilities Burns Timeline
(29:34) But China
(33:23) Ethical People Can Disagree With You
(34:45) There Are More Than Two Outcomes
(38:01) Efforts Are Complementary, Not Mutually Exclusive
(40:23) If Anyone Builds It, Everyone Dies
(41:55) Vaniver is Not Convinced
(46:19) There Is No Consensus That We Should Exclusively Focus on Plan A
(56:01) It's Not the Incentives, It's You
(58:08) Anthropic Is Not Optimally Pursuing These Plans
(01:04:00) Against Against Sura
The original text contained 2 footnotes which were omitted from this narration.
---

First published:

November 2nd, 2026


Source:

https://www.lesswrong.com/posts/ziMP83D2JYWRQ5jtu/scott-alexander-s-against-sura-seems-deeply-confused

---

Narrated by TYPE III AUDIO.