LessWrong posts by zvi

Astra is an excellent model. The jump from Sol to Astra is larger than the jump from Fable 5 to Fable 5.1. This is a big deal.

Astra is the best model for what one would broadly call ‘ambitious projects,’ and likely has the highest raw intelligence factor of any model. These are the largest jumps.

It is amazing at doing things in 3D, or anything involving games. Astra also excels at computer use, and at subagent coordination.

Many benchmarks show dramatic jumps from all previous models. Where Astra is good, it can be in a league of its own.

That does not mean Astra is in its own league across the board. Fable 5.1 is still a Claude. Astra is still a GPT. If you have a strong preference for one over the other, that still applies. For many purposes, especially involving back-and-forth discussions, Fable 5.1 is still my top choice. Fable remains my primary editor.

If you want the best answer to your questions, you should ask both models.

Regular coding is getting less of a focus. Astra is not a quantum leap there, but of course it is very good [...]
---
Outline:
(02:34) Meanwhile
(04:38) The Official Pitch
(13:08) Our Price Cheap
(14:06) Unnecessary Overstatement
(15:53) Paced Rollout
(16:32) Official Benchmarks
(22:37) Other People's Benchmarks
(30:47) Thinking, Fast Without Slow
(34:25) How Dare You, Sir
(36:36) PoetryBench
(37:54) In 3D
(39:14) Time to Think
(39:59) I'm Putting Together a Team
(41:02) Reviews and Essays
(41:37) Computer Use
(42:47) Positive Reactions
(50:53) AGI
(54:20) Astra Can Do The Math
(57:27) Astra Can Code
(59:35) I Came to (Change the) Game
(01:02:37) Astra Does Other Cool Things
(01:03:31) Astra Never Quits Except When It Does
(01:05:26) Negative Reactions
(01:07:45) Stop It With the Hedging
(01:08:57) Personality Clash
(01:09:53) Revealed Preference
(01:12:03) Dual Wielding
The original text contained 1 footnote which was omitted from this narration.
---

First published:

September 12th, 2026


Source:

https://www.lesswrong.com/posts/snaKjCwazKcRiS4qs/gpt-6-astra-can-do-ambitious-things

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

CEOs of major AI labs, and employees of major AI labs, including OpenAI and Anthropic, often say they plan to build superintelligence soon, as in within a few years create AIs that are superior to humans at essentially all cognitive tasks.

They often warn that such AIs might kill everyone. Or that AIs might cause mass unemployment, cause cyberattacks across the internet, enable mass surveillance or risk causing any number of other highly bad things.

These warnings are consistently and directly against the interests of the labs. Yet the warnings have recently gotten a lot louder and more frequent. OpenAI has been practically screaming, for those with ears to listen, on many occasions.

A series of events, over two months and especially the last week or so, including internal observations of the pace of progress at OpenAI and also Anthropic, have freaked out everyone involved quite a lot more than they were already freaked out.

After all the events, plus statements by Dean Ball and Jakub Pachocki, we were already seeing the beginnings of a preference cascade.

Then along came Jacob Coxon as the tipping point, and things took off.

Table of Contents
[...] ---
Outline:
(01:22) Jacob Coxon Resigns From Anthropic In Protest And Sounds The Alarm
(05:46) Mainstream Media Finally Pays Attention
(06:47) Preference Cascade at Anthropic
(10:18) Preference Cascade at OpenAI
(13:26) Preference Cascade at Google
(14:43) #NotAllMembersOfTechnicalStaff
(15:23) Why a Preference Cascade Now?
(21:01) This Is What Many Anthropic and OpenAI Employees Actually Believe
(24:08) To Quit Or Not To Quit
(31:29) Quiet Quitting Is A Dominated Option
(32:50) When You Quit, Very Serious People Understand What That Means
(39:29) Jacob Coxon Believes Existential Risk Is High That Is Why He Quit
(41:47) Evan Hubinger Believes Existential Risk Is High That Is Why He Stays
(45:22) Anthropic and OpenAI Have Commercial Incentives To Downplay Existential Risks, Not Advertise Them
(50:54) What Do We Do Now?
(53:03) OK, But How Exactly Would AI Kill Everyone?
(01:01:44) Best Start Believing In Science Fiction Stories Because You Are In One
(01:06:20) Literal Extinction Is Not Much Harder Than Loss of Control
(01:07:42) Conspiracytown Is Always Hiring
(01:20:14) Now You See It
---

First published:

September 11th, 2026


Source:

https://www.lesswrong.com/posts/5MB7KENgEAW6Q4JtJ/jacob-coxon-warns-of-human-extinction-and-triggers-a

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

These are quotes from OpenAI, Anthropic and Google employees, in the wake of Jacob Coxon's warnings, in which the employees confirm that they think AI might soon kill everyone.

If more quotes come in over the next week or so, I will update this post accordingly.

Preference Cascade Statements At OpenAI: Tomek Korbak

Tomek Korbak (OpenAI): i’m late to the party but: from his time at OpenAI I remember Jacob as a very thoughtful researcher and he continues to be so in this thread. neither anthropic nor openai are on track to solve alignment to a degree sufficient for shipping superintelligence and we need to slow down

Vie McCoy

Vie McCoy (OpenAI): I think pacing progress and ensuring human enhancement is the only way that we don’t get out-evolved while retaining the dream of superintelligence.

In this context, I see two paths before us.

In the first, we race towards RSI without embedding human flourishing and human enhancement as a deep value within the models, and by and large either get left behind or suffer catastrophic losses.

In the second, we set the pace of progress, focus on embedding human flourishing [...]
---
Outline:
(00:27) Preference Cascade Statements At OpenAI: Tomek Korbak
(00:55) Vie McCoy
(03:46) Adam Majmudar
(05:05) Aidan Clark
(05:46) Mo Bavarian
(07:32) Boaz Barak
(08:20) Micah Carroll
(09:15) Roon
(12:55) Confirmations At OpenAI: Dean Ball
(14:39) Leo Gao
(14:57) Anthropic's Evan Hubinger Confirms His Stance
(15:50) Preference Cascade at Anthropic: Samuel Marks
(17:36) Anna Wang
(18:22) Ethan Perez
(18:59) Dima Krasheninnikov
(19:27) EigenGender (Anon Account)
(19:58) Joe Benton
(20:07) Confirmation at Anthropic: Drake Thomas
(21:28) Jan Lieke
(22:07) Sluggy
(22:29) Preference Cascade at Google
(22:56) Andreas Kirsch
(23:39) Neel Nanda
(24:15) Victoria Krakovna
(25:26) Vishal Maini
(27:01) Joe (OpenAI, ex-Google)
(28:21) Josh Engels
(28:39) Geoffrey Irving
(29:30) Alex Turner and Geoffrey Hinton: Classic Examples
(29:45) #NotAllMembersOfTechnicalStaff: Ted Sanders
---

First published:

September 11th, 2026


Source:

https://www.lesswrong.com/posts/APGvWZtXEwkinvHDd/the-extinction-risk-preference-cascade-quotes

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The world of AI is inside my OODA loop. Even if I can process all the incoming information and sculpt it into posts, and even using Saturday and Sunday as flex slots, I don’t have enough days of the week to post all the posts that need posting.

That was already true. There was already a preference cascade happening where people finally were admitting that they thought AI might well kill everyone.

Then Jacob Coxon resigned from Anthropic, rang the warning bells and turned that cascade into an avalanche.

Now that is what everyone is talking about. Finally, everyone is actually saying the thing, out loud. I plan to cover that in its own post soon.

There are several things in the weekly that, in a normal week, would get their own coverage. Senator Sanders and Representative Casar introduced an outright ban on superintelligence and I have to remind myself that happened this week. Suddenly it is not so crazy to think such a thing might pass.

So here's what I’ve already posted about so far since the last weekly:


Claude Fable and Mythos 5.1: The System Card.

Claude Fable and [...]
---
Outline:
(04:49) Language Models Offer Mundane Utility
(05:39) Language Models Don't Offer Mundane Utility
(06:53) Huh, Upgrades
(08:25) How To Tell a Fable
(09:35) On Your Marks
(09:55) Deepfaketown and Botpocalypse Soon
(14:16) Levels of Friction
(17:35) Cyber Lack of Security
(23:48) A Young Lady's Illustrated Primer
(25:18) They Took Our Jobs
(29:18) Anthropic Offers Economic Scenarios
(34:12) Get Involved
(37:05) Introducing
(37:47) In Other AI News
(38:13) Show Me the Money
(38:25) Quiet Speculations
(42:49) The Quest for Sane Regulations
(44:09) The OpenAI Policy and Lobbying Department
(50:41) Greetings From the Department of War
(51:58) Hugging The Face
(59:16) Hugging the Question
(01:03:21) The Ban Artificial Superintelligence Act
(01:11:05) Chip City
(01:12:21) The Week in Audio
(01:12:41) People Just Say Things
(01:18:09) PauseAI Global Disendorsed PauseAI US
(01:19:55) Paul Christiano Joins Board of OpenAI Foundation
(01:25:18) Rhetorical Innovation
(01:34:23) Aligning a Smarter Than Human Intelligence is Difficult
(01:37:00) Cooperative Alignment
(01:45:48) Drive to Survive
(01:48:13) People Are Worried About AI Killing Everyone
(01:50:07) Other People Are Not As Worried About AI Killing Everyone
(01:52:30) The Lighter Side
---

First published:

September 10th, 2026


Source:

https://www.lesswrong.com/posts/tFmtz9HW6c2X9dw2B/ai-185-preference-cascade

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

OpenAI claims that Astra is ‘the most intelligent and most aligned [available] model’ in the world. Not the most intelligent and aligned OpenAI model, but the most period.

That is bold talk. It risks overstepping, and by doing so souring the release of what is clearly an excellent model. As do the severe problems with monitorability.

It also raises the question of what they mean by ‘most aligned model.’ How do they define ‘aligned.’ Why do they think it is more aligned than Claude Fable 5.1?

keltan : “a significant step forward in […] alignment.”

Buddy, how tf are you measuring ‘alignment’? Would love to know because being able to measure that would save the fucking world.

roon (OpenAI): low rates of cheating

Rob Miles: *detected cheating

keltan : Thank you for clarifying. But you know what I’m gonna say next, right?

roon (OpenAI): that this metrics are not a full solve of alignment and will break discontinuously

keltan : Yep. But I would have said it in a dumber way. Something like: Low Rates of Cheating ≠ Alignment

roon (OpenAI): I agree but also in some real sense [...]
---
Outline:
(04:12) OpenAI's Safety Claims About Astra (1)
(08:33) Preparedness Capabilities Assessment (10)
(09:00) Biological and Chemical Capability is High
(10:33) Cybersecurity Capability is Critical
(17:05) AI Self-Improvement Capabilities (10.1.3)
(17:45) Astra Is Highly Verbally Eval Aware (from 8.6)
(19:05) Safe Mundane Completions (4.1)
(21:05) Jailbreaks (5.1)
(22:21) Prompt Injection (5.2)
(23:41) Health (6)
(24:22) Hallucinations (7)
(24:57) Alignment (8)
(26:06) Obeying Restrictions (8.2)
(29:19) That's Worse, You Do Get How That's Worse, Right?
(30:45) OpenAI Does Not Understand Why This Is Worse
(34:52) The Alternative Explanation Is Also Worse
(42:39) Metagaming (8.7)
(45:07) Alignment Faking (8.7)
(46:19) Don't Lie to the User (8.3)
(47:25) Misalignment in Realistic Work Environments (8.4)
(48:05) Unintended Agent-to-Agent Communication (8.5)
(49:48) The Three Obviously Monitored Temptations of Astra
(51:26) Severe Issues In Simulated Traffic Are Down By Half
(53:01) UK AISI External Evaluations (8.8)
(57:02) Sabotaging Safety Work
(57:29) What About The July 19 Attacks?
(58:58) Apollo Research External Evaluations (8.8.1)
(01:00:01) The Alignment Verdict
(01:02:04) It Depends What You Mean By Alignment
---

First published:

September 9th, 2026


Source:

https://www.lesswrong.com/posts/AmFJZyeCgvFjNKgNk/gpt-6-astra-the-system-card-alignment-and-what-comes-next

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.