Interconnects

Hill-climbing is easy, knowing what to hill-climb is hard. Making scores go up is expensive, but we have a much clearer playbook for it than we do for creating meaningful benchmarks at the true frontier of AI today.
Independent evaluation of language models is under-appreciated in how hard of a task it is. These groups are not just taking current models and certifying their scores, which is a technically involved task. This is their job if AI progress is slow. When AI progress is as fast as it is, the technical details of the job are the routine work part of trying to put a single number on what makes for a good AI model today. The goal is to have a single number that is representative of current models, matching vibe checks, community sentiment, and scientific analysis, while being forward looking enough so that it’s not outdated in 3 months.
Everyone building a benchmark faces a sort of paradox. You need your benchmark to reduce to one single number, so that it is easy to discuss, use in reports, and compare models, but every method of aggregating scores comes with meaningful trade-offs. The best scientists need to do this, everyone does. It’s a matter of creating aggregation metrics that you think are most fair, knowing there will always be imperfections.
With the amount of money in the AI ecosystem today, the weight of independent evaluation is even higher – you dictate the winners and losers of a trillion dollar market. You also influence some of the models themselves, steering the shape of progress. This is a lot of pressure!
My prompt for this article is Artificial Analysis’s recent update to their Intelligence Index, v5. The AA index is the most prominent independent intelligence metric for the latest frontier models, followed in a distant second by Epoch AI’s Capabilities Index (ECI). To date, all of these have been incredibly useful for the community and massively flawed!
Interconnects AI is a reader-supported publication. Consider becoming a subscriber.

Language models are capable of so much today, and this scope of capabilities is growing quickly. Our era of singular intelligence indices should be phased out soon, but it will take a long time.
A summary of AAv5 is below, from their blog post, where they describe it as “reflecting real-world usage today:”
Intelligence Index v5 weights:
➤ Knowledge Work (25%): AA-Briefcase (15%), AutomationBench-AA (5%), and GDP.pdf (5%)
➤ Coding (25%): FrontierCode 2.0 (15%) and Terminal-Bench 4.0 (10%)
➤ Science (25%): Terminal-Bench-Science (15%) and CritPt v2 hard subset (10%)
➤ General (25%): AA-Omniscience v1.1 15% (10% accuracy, 5% non-hallucination) and HLE-Diamond (10%)
Here’s the new ranking of models:
The problem with trying to benchmark real-world usage today is that it is extremely jagged, just like the models. The most proven use-case is coding, where open models are far more competitive and likely to keep up. In knowledge work, I’m less confident in open models.
At first glance, in aggregate, their model ranking definitely looks reasonable. Still, it shows signs of strain and confusion, as any benchmark will, with GPT 6.1 Terra beating Claude Fable 5.1 on AAv5. Today, some benchmarks measure the absolute frontier of capabilities, playing nicely in open-ended tasks through your favorite harness, which often does take a large model. Some other benchmarks measure certain skills that will be needed of a model in repetitive agentic tasks. I get the feeling that AA is tuning their benchmark more towards the former, a general agentic collaborator, but all of these behaviors overlap.
This takes us to the plot going around that prompted me to write this, where they showed the balance of open vs. closed models on the new benchmark. This plot shows 8 closed models beating the best to-be open-weight model, Kimi K3.1 Pro (which isn’t colored blue). This benchmark is taking the opinion of trying to measure where frontier models are moving into, which grows the gap to open models. This is an okay decision to take, but over time as existing workflows of open models become key factors of the economy, it could become a benchmark that misses where the core economic engine of the industry is.
Open models’ demise from this seems greatly exaggerated, but I do think the previous index was flattering the open weight models too strongly. Consider this plot from the Kimi K3.1 Pro launch — yes, the open models are strong and useful in some cases, but the best OpenAI and Anthropic models have been a major step ahead in their flexibility and reliability.
Some of the evaluations in AAv5 that open models are most behind on are “science” evals like terminal-bench-science and CritPT v2. Open models are behind, but the value proposition was never to compete truly at the frontier. The open model economy will be about
* Reducing domains with high market demand for inference to a commodity price of hardware.
* Allowing ownership of intelligence, specializing it to internal data, and controlling internal agents across many companies in the economy.
Both of these will take a very long time to materialize and are partially represented in benchmarks. The supply of compute is currently so low that open model providers also are raking in solid margins – when this shrinks, we will know the system is starting to work. The same goes for diffusion into enterprises, it’s not clear what the exact ceiling on it is because of the difficult ML expertise that is needed to build evaluations and do post-training today.
At the next major model cycle, say in 2-3 months with GLM 6 or Kimi 3.5, I expect the leading open model labs to make substantial steps on this index, again surprising many observers. This is a cycle we have been on for years now.
I’ve written many times about how open models keep up with the frontier, and a summary is that:
* The best open models do benchmax a little. The worst open models benchmax a lot (but don’t get a ton of attention anyways).
* The best open models are a bit narrower in which capabilities they cover, as seen above with their gap on scientific domains. The open model labs are working to address this, but I think it could be come a strategic error to not focus.
There’s a more nuanced version of explanations of the performance gap today, where so many of the gains in training are coming from having RL environments with distributions close to the evaluations (a longer analysis on this is here). Keeping up with rapid progress across multiple domains requires a sophisticated RL stack that not all the open model labs may possess. Some may be reliant on buying data that has moderate lead times as well.
Another part of this announcement is shifting to “private benchmarks”, where the prompts and test environments are not public, as a sign that open labs are benchmaxxing. This sort of analysis has been done before, and fits with my above theory on how open labs benchmax. It is easier to create training data to support an evaluation when you know the shape of it, but the task isn’t impossible with hidden data. All in, this is a change that directionally I think is defensible, but it’s not unanimous. For example, you’re putting complete trust in the private benchmark builder, as you no longer get the benefit of the community finding issues in the benchmark and pushing the benchmark to improve itself – which happens a lot! There is enough money on the line, where I’d rather see transparency in benchmarking and enabling many eyes to find bugs.
The final point I need to make, which does flatter the closed labs, is that the likes of Anthropic and OpenAI moved on from caring about these indices months, or even years, ago. Their focus is to a large majority on supporting internal uses – like AI R&D automation – and the use-cases of customers who are spending billions of dollars. Yes, they have team members hill climbing on other capabilities to unlock new markets, but these behaviors are to a minority extent driven by public benchmarks. At the same time, it is…


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe

We — Nathan and Florian — wanted to discuss all the messy trade-offs of a busy week in AI and ask all the pressing questions like:
* Why did Kimi K3.1 Pro have a mixed reaction with such incredible benchmarks? Will it be adopted?
* Were we wrong about the risks of open-weight models and cybersecurity?
* Why do cyber experts discuss recent events so differently than AI experts?
* What comes next in the few months? What will change our minds?
We will write more about this subject, but wanted to bring you behind the curtain as we aspire to quickly learn more about the cybersecurity industry and trends across the last few years that have been leading us today. We conclude the podcast with some lighter topics, from Muse-maxing to excitement in still hiring junior technical talent. Please send us questions and topics, as we plan to record more of these podcasts going forward.
Listen on Apple Podcasts, Spotify, and wherever you get your podcasts. For other Interconnects podcasts, go here. YouTube ↓
Chapters
* 00:00 Kimi K3.1 Reaction
* 08:15 Kimi’s open-weight cyber principles
* 11:23 You can hack everything era
* 15:54 PHP2Shell, LiteLLM, and the new internet
* 24:00 Abliterated APIs, safeguards, and trusted access
* 31:41 Anthropic not hiring juniors employees
* 36:16 Florian’s Muse addiction
* 45:30 Wrap-up
Links we discuss
* My thread with Dean Ball about the “boy who cried wolf” story of the frontier labs
* Kimi K3.1 Pro announcement blog
* Kimi K3.1 Pro on Artificial Analysis Intelligence Index
* Kimi K3.1 on Artificial Analysis defensive cyber benchmarks
* Program Bench
* Open-weight Mythos would’ve been fine take
* PHP “hasn’t caused a disaster yet” update
* Abliteration.ai serving abliterated models API
* Morph serving abliterated DeepSeek via API
* Joshua Saxe’s Substack on AI and cybersecurity
* Azeem Azhar, “The internet’s Wuhan moment”
* Anthropic, “How employment is changing at Anthropic”
* LiteLLM security incident report
Transcript
00:00:00 Nathan Lambert: We are back with another Interconnects podcast. We are with Florian, eval guy, recently moved to SF. Florian, is Kimi K3.1 Pro, name is a mouthful, going to live up to the benchmark scores, for example on Artificial Analysis, or where is it going to land?
00:00:17 Florian Brand: In the past, the Kimi models always lived up to the expectations, but this release is kind of weird compared to the others. I think it is also partially because they are becoming more professional, and the previous model drops were kind of out of nowhere. This time, which also got some criticism, they launched a bigger social media campaign, which allegedly also involved paying people to post about the model without disclosing.
00:00:56 Nathan Lambert: I think a lot of companies do this and tell you not to mention it, but it’s a mess. It’s bad form. No company should do it. Essentially, one person tweeted a screenshot of the press instructions for Kimi that said, “Don’t say this is a paid promotion.” And to my knowledge, no one has corroborated it after that.
00:01:15 Florian Brand: Yeah, and there are right now a few of these model launches, including by American companies, that are just very obviously astroturfed. We have seen launches where it’s thousands of retweets and a few hundred likes, because it works so well to get the attention. Some of them disclose that it is a paid ad while others don’t. But in general, we have seen the professionalization of model launches over time, and pretty quickly. A year ago, everyone just dropped the model, then patched vLLM afterwards, so no one was actually able to run any of the open models for a while. Now you get all the inference frameworks on day one, you get all the inference providers on day one. Or, which is also what we see more and more, the model creator has a few weeks of exclusive access before they release the weights. And that’s also what’s happening with K3.1 Pro. I think it’s two weeks of exclusive access before they release the model weights.
00:02:38 Nathan Lambert: They said the weights would come out by November 4th. And the headline things... I don’t know if I’m going to be able to share screen. I don’t have this set up on this computer. But the headline is this Artificial Analysis number where it’s ahead of GPT-6 Astra. And then if you look at the Kimi K3.1 blog, they have their headline numbers, which are really strong, and then they also have some numbers later on which seem a bit more jagged. I think it was Terminal-Bench Science, which is 20 points behind GPT-6 Astra. I think there were a few others, like ProgramBench. Do you know Terminal-Bench Science and ProgramBench? Are any of these things that people think are important benchmarks for frontier models? Or how do you explain the fact that on Artificial Analysis the model is top, what, three, four? It’s the two latest Claude 5.5 models, Muse Wonder Full, and then Kimi K3.1, which looks crazy. It just looks like totally crazy scores, and I think everybody is still surprised by it. But then there’s this long tail of details that you use to evaluate a model now. I had posted a thread on Twitter of, “What are people’s impressions?” And there’s a lot of people that are like, “This seems not great,” and then some people that are like, “Yeah, it’s fine.” I don’t know, there’s a mixed bag. Relative to K3, which I think was pretty unanimously, this feels like a big model and the post-training seemed a little bit undercooked for that. Maybe this is just a tiny bit RL fried. We’ll see where we land.
00:04:26 Florian Brand: For the benchmarks, one of the things is that the benchmarks that are right now used by Artificial Analysis, they will post their new version probably very soon after we release this podcast.
00:04:43 Nathan Lambert: We can’t tell you the exact date, but both of us know.
00:04:48 Florian Brand: In general, the Artificial Analysis benchmarks are rather short in terms of time horizon, whereas something like ProgramBench is really long and also very explorative. Other benchmarks are telling the model, “All right, this is the goal, and I want you to reach it,” whereas ProgramBench is more like, “All right, here’s what I want, but you need to explore a lot.” And I think that might be the reason that we see some differences: their RL training is still more towards the exploitative side of model training versus the explorative side, where you need to find the solution somehow before you actually know what you’re doing.
00:05:41 Nathan Lambert: But if you’re using a three trillion parameter model, I feel like you kind of want it to be pretty robust at exploration. Because if you’re going to want to use a model for pretty known coding tasks, like using DeepSeek-V4.1-Flash-1017, which is not that far behind on these scores, it’s just going to be way faster. It might not be wildly different on cost per task with the actual token numbers. That’s why I think it’s pretty reasonable criticism of this Kimi release: a big model has got to be... Because it’s going to be slower, it’s got to be like Fable, which is just... I still think Fable is really nice if you’re doing very blurry things. I revert to it when checking my slides over sometimes, where I just feel like it kind of gets me a bit better, where Opus is really, really grindy. I think the open-weight models are better in this kind of grindy perspective, because you can just get them to be faster and cheaper.
00:06:44 Nathan Lambert: It seems like Kimi is gearing up for this Flash-style release with their naming change. Previous Kimi models were just K2, K2.6, K3, and now this is K3.1 Pro. So a lot of the company forecasting, investment-y side of interest on Twitter was, a lot hinges on this for Kimi, because it’s kind of true. We’re not the inference experts, but serving the model seems like you’ve got to commit way more resources to serve this three trillion versus this 700 billion parameter model. You’re going to need NVL72 racks, I’m guessing, to do it well.
00:07:21 Florian Brand: To fit the model, you need B300s, one node per …


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe

This has been a week where the balance of American open-weight models took a step change back toward the frontier. Meta took the lead of this charge with Muse Spark Open, which is now the strongest American-made open model by quite some margin, but the addition of Reflection’s first model Beam is also relevant. Without these two models, there would be 20+ open-weight models built and released by Chinese companies better than the best Western models. At the time of writing this, Mistral has also announced a strong model, Mistral Large 4, with an index score of 38 (see below), but its weights are not yet released. There are rumors of a few more strong American open-weight models coming from other organizations too!
In many ways, the gap between the best American and Chinese models has been reduced roughly by half. We’ve gone from the best American open models being largely irrelevant to consumers who require the cutting edge of performance, to them being behind but relevant.
This paints the picture of the cut-throat, competitive nature of the frontier model business. Reflection and Mistral announced their model scores and Meta dropped the weights of a smarter model before these other two labs could get their own weights out. All of these models are competing with Chinese models released months ago, where the industry is collectively asking — where have Kimi and Zhipu AI been since the releases of GLM-5.3 and Kimi K3 on Aug. 18th and Jul. 16th, respectively?
That is a very nuanced question, which I’ll dig into another time. Today’s piece is focused on Meta’s turnaround, what Muse Spark Open means for the ecosystem, and the balance of power across Western frontier labs.
Why Muse Spark Open should take off — and why it may not
One thing we see again and again at Interconnects through our open model analysis is that Western open-weight models get astonishingly more adoption than Chinese models with the same performance-to-size footprint (even though the Chinese models usually come out first)! Muse Spark Open is landing into this environment, at the same time as when open-ended LLM agents are finally real (~GLM 5.2 capabilities and on).
Riding off a rapid increase in performance of Chinese models, the open ecosystem has been living through an acceleration in demand. The inference providers are rapidly scaling, the open post-training economy is forming, and countless enterprises are looking to own their intelligence. Muse Spark Open’s performance falls somewhere above GLM 5.2, near DeepSeek V4.1 Flash, and behind GLM 5.3 or Kimi K3. It should be squarely “good enough” to slot right into this.
For some background, from the Muse Spark Open model card on Hugging Face:
Muse Spark Open is an 816B total/42B active parameter open weights model with a context window of 1M tokens. Built for coding and agentic work, Muse Spark Open is natively multimodal. This open weights version has additional safety post-training to make it more robust against misuse as a standalone model without system guardrails. We recommend deploying Muse Spark Open as part of a broader AI system, with additional guardrails suited to the specific use case.
Here’s the distribution of the various Muse models on the Artificial Analysis Intelligence Index.
This is included to loop back to the original model card snippet. It was shared that this open-weight model underwent additional safety training to prepare for this open release. Many observers saw this release as potentially a model that was “broken” during the safety training, as it came out after Muse Spark 1.3.
In reality, this was much more likely a safety tune of Muse Spark 1.2, which would only be a 1 point drop on Artificial Analysis to shore up safety behaviors. When Meta released Muse Glimmer, the 30B agentic model, Alex Wang said that they were preparing to release the Muse Spark 1.2 model as open weights, which fits with the above.
If anything will hold Muse Spark Open back, it’ll be the specific license they designed for it. In summary, this custom license has pretty much all of the restrictive license behaviors we’ve seen (and criticized here) in the last few years of building with open models. They are as follows:
* Users of the models in products must display “Built with Muse” on their products. Meta did this with Llama, and Kimi has tried it, but almost no one complies.
* There is a revenue-gating factor, where companies that make more than $20M per month across their entire business deploying the model need a separate license with Meta. Other prominent open model providers are doing this too, in order to set up revenue share agreements for re-selling inference.
* Muse Spark Open makes you agree to an acceptable use policy, which in the true “open source software” corner of the world makes this model extremely not open-source.
* All of the above applies to outputs from the model as well, and derivatives, so there is legal ambiguity if you can safely use the model for synthetic data.
The license does permit using Muse Spark Open for internal research and applications.
We’re at a weird equilibrium where this license is complex and unproven in court, but many players will likely ignore it. Many, many people will build on this model from various inference providers, and Meta will only likely go after egregious violations.
For now, licenses haven’t been the determining factor of AI adoption, but this could change in the future. This is why we think it’s an unnecessary strategic risk for labs trying to compete and regain adoption to have complex, non-standard open licenses for their models.
Interconnects AI is a reader-supported publication. Consider becoming a subscriber.

Is Meta a tier 1 (OpenAI/Anthropic) or tier 2 (Gemini/xAI) frontier lab?
Just a few days before Muse Spark Open, Meta unveiled their closed frontier models, Muse Wonder and Muse Wonder Full. On traditional benchmarks, these are very strong models only behind Anthropic’s Opus 5.5 and similar to OpenAI’s GPT-6-Astra or Google’s Gemini-4-Argon.
These major releases are coming just over a year out from Meta’s massive AI redirection, characterized by CEO Mark Zuckerberg’s Personal Superintelligence post, which got a lot of criticism at the time. Meta is the only American company with frontier open and closed models. Alibaba’s Qwen has followed a similar strategy, but with a far more open-leaning approach.
The vibe checks across all of these models have been solid, but the expectations were for them to not immediately challenge the level of refinement of the most recent Anthropic and OpenAI models in real-world usage. I’m excited to try them more — time to sign up for another AI subscription.
I like this recent run of form from Meta, driving their own products and directions in the popular Muse agent, taking wins like the leading US open-weight crown when useful. Whether or not it’s financially successful, it’s a unique perspective that fits their history as a business.
It often feels like there’s a revolving door of which lab is nipping at the heels of the Anthropic-OpenAI leading duo. It’s not clear that in the long run Gemini and xAI are out of it, but those two are much more muddled in their offerings than Meta today. If you listen to Zuck and Alexandr Wang on The Cheeky Pint, they clearly had a very specific vision of a high-ownership, high-compute per researcher lab that I would expect to pay off. Many labs die due to layers of bureaucracy and misaligned incentives. Meanwhile Alex Wang has over 200 direct reports — i.e. it’s a very flat org with minimal management — and they’re releasing very consistent model and product improvements.
Much of my worldview of building models is that you need the resources, the compute, the data, and the talent (all downstream of large sums of money), but organizing them is very hard. Gemini is sometimes described as a place where it’s impossible to get any work done or grow — which is embellishing the issue, as many people love it there, but there are kernels of …


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe

I’ve had one too many discussions on the cyber risks of open models — where someone assumes that either banning open models happens in a silo (and bad actors will somehow actually be stopped) or that China is a threat and doesn’t care about AI safety — that I feel we’re going to end up making policy decisions that both limit American AI competitiveness and increases long-term cyber risk. I feel like we’re locked in on a path of lose-lose situation, unless people embrace nuance, trade-offs, and scary realities.
Much has been written about the views of various players in this debate, which I summarize as follows:
* Open weight models pose an untenable risk to society’s functionality, occupied by frontier lab leadership and the national security community in the U.S.
* Western voices saying open weight models are necessary for defense and banning them will make the world less safe, occupied by AI risk moderates to different extremes. I include myself in this bucket, along with Hugging Face’s perspective after the OpenAI incident, and others like Joshua Saxe.
* Chinese companies continuing to release open-weight models with strong cyber capabilities, who are making decisions based on the risk assessments of their society and government.
This list is also organized by volume or clarity of message in the Western AI ecosystem. The risk-focused discussions have been disproportionately visible, including pieces of work which I view as very detrimental to potential of engaging across assumed party lines on this issue. There’s also very little substantive engagement trying to understand how Chinese companies assess risk — rather, mostly arguments that border on ad hominem attacks like “Chinese labs don’t care about safety.”
Interconnects AI is a reader-supported publication. Consider becoming a subscriber.

The delusions of the anti open-weight alliance
The most recent publication from the “open-weights are dangerous” crowd was the report by Anthropic on the risks of GLM-5.3 as an offensive cyber tool. The problem with this is not the technical research they did, which is largely reasonable, but the failure to engage on more cross cutting questions like: What happens if we ban open models due to cyber risks or why do Chinese companies deem these models safe to release?
These questions, which I’ll come back to, are the ones we need to answer to understand the ecosystem perspective on current AI risks. All policy actions happen in a multi-piece puzzle, where the shape of risk can be changed, but it is rare that you get a universally better outcome. They are all trade-offs.
This sort of solipsistic writing by Anthropic fits very closely to another type of discourse I’m hearing about regularly that is determining the fate of open models and AI broadly — classified briefings. I’ve had many discussions with people who say something like “Look, I’m pro open models, but if you’re seeing what I’m seeing, it’s an onslaught out there due to [XYZ Chinese open-weight model] and we need to act.”
This perspective flies in the face of public information, where to date closed models have been documented as the cause of most existing cyber attacks. As someone who tries to make grounded predictions, the evidence of numerous attacks from OpenAI models is the only data I have on the shape of cyber risks (or, you can broaden the scope to include all of FelonyBench). There are a few possible explanations, and it’ll take years to get to the true answer:
* Maybe open model weights and closed model APIs (with some safe guards) are both far closer to being easy to mis-use, rather than API models being closer to safe. The trope "Open Dangerous, Closed Safe" may be closer to "Open Unsafe, Closed Unsafe."
* Maybe there are far fewer bad actors who are willing to use AI models for “loud” cyber attacks that target crucial infrastructure in the US or powerful and underprepared institutions. Closed models have stronger capabilities and stronger safeguards (in theory), but the stronger capabilities part could matter more in net harm if the safeguards on both are porous.
* I have a lot more to learn about how the cybersecurity ecosystem works, so I won’t support this full-heartedly as one of my current recommendations, but I think it’s at least a reasonable argument that open weight models as a diffusion tool for cyber defense could be the best tool we have to prevent harm over the next few years. There are many domains, e.g. sensitive government agencies, where open-weight models are the only tools that can be deployed in the near-term on air-gapped networks.
Lots of this situation seems like the messy reality being hard to pin down, where the logical limits are clearer. I may agree with the simple reality that closed models are far safer in the limit, as a technical tool with more layers of protection, but they may actually be causing more harm in the near term because of how accessible model APIs are. Still, much of the impact of the tools comes down to how they’re used and controlled. A duopoly on such a crucial tool may be the defining, risky part. This is the debate we’re having, and over time, as intelligence diffuses, many things may change.
These together make me come to the conclusion that if you think the latest open-weight models need to be banned to slow the diffusion of cyber risks, you probably also need to make public-facing APIs for the frontier closed-models illegal. The current stack of safeguards on closed models is stronger than open-weight models, but far from perfect. It is very likely that cyber capabilities of closed models increase much faster than guardrail performance, and a world where open models are banned while closed models continue to progress would be increasing the offense-defense cyber gap. It will take meaningful time to build mitigations that allow defenders to use strong AI models on their private infrastructure if we remove access to strong open-weight models.
Explaining China’s AI risk posture
China cares about AI safety. They have come to this discourse in a complete, endogenous manner relating to Chinese culture and power structures. There are some big picture pieces of this puzzle that reflect in other ways, such as Chinese society’s techno-optimism that extends into AI and the Chinese government’s complete focus on political stability. These things do, of course, interface with emerging AI risks like cybersecurity.
They are in their own AI risk dynamic, which hasn’t risen to the prominence of the political dance between the White House and the frontier labs which we have followed in the U.S. for a few months. My understanding is that the Chinese companies have to register every major model release with the Chinese government. This includes evaluations, which originated around information control for banned government information. It is unclear if this framework has expanded meaningfully into any risks like cyber or bio. The Chinese government is very decentralized and information from the labs needs to go through existing structures to make it to the top.
At the same time, there is a much stronger air of social risk in China, where leading AI researchers aren’t allowed to leave the country and the industry is closing off to foreign investment. On balance I would guess the personal risks facing people by unleashing any domestic harms would be much higher than in the U.S., where the worst case is that your company gets deleted.
Overall, the political oversight seems much earlier than in the U.S. The Western AI companies’ core principle is close to “the best way to stop a bad guy with AI is to be a good guy with AI.” In China, these companies are much more practical about building a wonderful tool, and in many cases building a tool that complements their existing businesses.
I do think that the companies consider the global implications of their work. These labs also clearly are incentivized, and often pushed via the government (at least in public messaging) to compete as much as t…


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe

This episode is with Jean-Stanislas “JS” Denain of Epoch AI, who leads their Insights Team and is one of the people I find myself debating the state and trajectory of AI with more and more. We’ve had follow-on discussions of many of my favorite recent posts online and/or in private, so I wanted to dig into the nuance in a public episode.
A big takeaway of this podcast is how JS and I both have so much uncertainty with exactly where we are heading, and this was our best effort at stating our observations today.
Chapters / topics include:
* 00:00 Predictions for RSI
* 18:15 The role of robotics in an AI acceleration
* 24:20 How far behind are Chinese models?
* 27:39 Does distillation explain the gap?
* 40:58 What Chinese job postings reveal about their labs
* 48:13 Are open or closed models safer?
* 58:10 How Epoch AI ticks
* 1:00:55 What a frontier post-training recipe looks like
Enjoy!
More from JS: Epoch AI profile and writing, X, LinkedIn
Listen on Apple Podcasts, Spotify, and where ever you get your podcasts. For other Interconnects interviews, go here.
Transcript
00:00:00 Nathan Lambert: I’m here with JS Denain, who is a senior researcher at Epoch AI. He leads the insights team. He is one of the people who I feel like I get the best feedback on my writing from, whether it’s from US-China AI capabilities, now RSI. And I just wanted to open this discussion and honestly go deeper with him, trying to understand how he thinks about these various things. And I think you have a very useful, moderate point of view, which I feel like you’re probably a step further into what would be called faster scenarios for AI progress. But let’s get into this, and it’s like, what measurements do you think OpenAI and Anthropic are seeing when we get all these proclamations on RSI happening very imminently?
00:00:47 JS Denain: Yeah. I think, so there’s the measurements they’ve published, right? So, OpenAI and Anthropic both had blog posts, I mean, Anthropic two at least, on the effect AI has on accelerating AI progress. I think at least the things they publish, I don’t think are super strong evidence of imminent self-sustaining acceleration AI capabilities, or full automation of the job of AI researcher. But I think the kinds of things we see are, I think probably the most striking thing I saw in the OpenAI blog post was increasing usage of AI systems in model deployment, like the increase in spending on Codex that we saw. And it’s kind of unclear how exactly to interpret this, because maybe it’s a measurement artifact where they’re only looking at Codex, but in fact, there was a bunch of ChatGPT usage before from the researchers. But overall, that plot, for example, just shows a 2X a month increase in Codex spending by researchers, and that does seem to me to be some evidence of they’re getting a lot of value out of this probably. I don’t think this is strong evidence that in six months we have a software intelligence explosion.
00:01:57 Nathan Lambert: Do you think this is the same? So, what is the information they have internally relative to what we have? And this is obviously hypothetical. We don’t have this internal information. Because I get the sense that a lot of people are more scared in their updates from the labs than the information we have. And I try to take this very seriously of, what will they be seeing that is making the acceleration of risk comments go faster, and how much of this is material evidence versus how cultures evolve over time? And I’m much more interested in evidence.
00:02:29 JS Denain: Yeah. So two things. I think, first of all, I guess I don’t think, I don’t know, right? I don’t have full information here. I don’t currently think that either there’s some specific thing that people at OpenAI or Anthropic are seeing right now that we don’t have access to that warrants being way more freaked out about this. I also don’t think that... I think the public evidence we have right now, more general on AI progress and just a priori case for this being an important dynamic, I think is enough. I think to care about this particular dynamic of AI accelerating AI progress, that being a big deal and worth tracking. And then it’s kind of unclear what the urgency is of when the feedback loop really kicks in. So, basically on the what is there on the inside that people have access to, I could give examples of kinds of metrics, right, they could be looking at. It’s plausible that we have access to the capabilities of AI systems, but internal teams have their KPIs, and maybe they’re seeing compute multipliers in the pre-training team or other kinds of metrics that people are tracking going crazy. And then the combination of this plus some intuitions of how the different outputs of different teams combine yields a prediction on the trend in actual performance of the end AI systems. So that could be an early warning sign. It’s unclear to me that the recent discourse we’ve seen is evidence of things going crazy on those metrics.
00:04:09 Nathan Lambert: And how do you think of the link between RSI and existential risk? So I would posit that you agree. I think that there are very real risks of AI, and I’m curious on how you think these, what I would describe as very, very early measurements change anything on the scope of risk. Because I don’t think if you had asked people six months ago, it would be as immediate to x-risk among people are very reasonable. I think there’s more people that are reasonable talking about x-risk again, which was a little surprising to me.
00:04:43 JS Denain: Yeah. So, okay, my sense is something like... So personally, I feel very uncertain about this, but I do feel, yeah, basically bought into there’s, I know Evan Hubinger was like, at least 10% of x-risk within I don’t know what timeframe. I think I’m like, yeah, I know, and this seems pretty reasonable over a decade-long timeframe. I just feel extremely uncertain about it, but I’m definitely very worried about this. Now, why am I worried about this, and where do I think the disagreements come from? And then how do I relate this to the early sense of RSI? My sense, I’m kind of a capabilities theory of everything person. I think, and I think some people disagree here, but I really think that principal component of disagreement between everyone is how huge do the capabilities get, how soon, of AI systems? And I sort of agree that there’s other factors that come in play for how big your economic growth gets, also depend on the diffusion you get. And you could possibly you could think that capabilities are going to get crazy, but the AI system’s going to be just aligned and benign and stuff, and so there’s no huge risk. But my sense is concretely, when I look at the main kinds of disagreements between people, most of the people who I see who are very skeptical of those most extreme scenarios... I think just expect capabilities to not be as huge or as I think folks who are—
00:06:18 Nathan Lambert: What does being a capabilities maximalist look like in a few years? Because I think I’m probably on the skeptical side, so please continue.
00:06:28 JS Denain: I think it looks, for example, something like the AI 2027 scenario, right? I think it looks like the mechanism for this is AI is automating the AI research process, I think, and that’s a reason to pay attention to it. But in terms of effect on the real world, I think it’s like massive progress on robotics. I think a huge industrial explosion, AI systems are just managing factories. You have this kind of self-sustaining economy that just is able to make a large scientific progress much faster than you would have expected. And so I think concretely, the kinds of disagreements I would expect are on, yeah, if you have AI systems that are both very intelligent in the book smart sense, but also have been trained to have more affordances and use them astutely, have been trained to kind of manage projects in efficient ways and stuff like that. How big are the real-world bottlen…


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe