Interconnects

I was recently invited to brief a group of Congressional members and staff on the state of open-weight models in the lens of U.S.-China competition. I’m sharing my prepared remarks as a state of the union on open models that is accessible to a broader audience.
Interconnects AI is a reader-supported publication. Consider becoming a subscriber.

Recap: What is an open source v. open-weight vs. closed model?
Open language models are AI models where their weights are publicly available for inspection or downstream use. These are most often contrasted to so-called “closed” AI models. Closed models offer access only through Application Programming Interfaces (APIs) that developers can use to directly query a model, like GPT-4 or Claude Opus 4.5, or through products, like ChatGPT and Claude Code.
Open language models primarily are bucketed into two categories, open-weight and open-source models. Open-weight models are the most common form, such as popular models like Meta’s Llama, Alibaba’s Qwen, Google’s Gemma, or DeepSeek’s models. These models are governed by licenses, governing documents dictating what is allowed with downstream use, and are often accompanied by inference code in libraries such as Transformers, VLLM, SGLANG, etc. Since about April 2025, Chinese AI companies have been the clear leader in open-weight models.
True “open-source” models are similar to these, as they include the weights, licenses, and inference code, but they also include the complete information needed to reproduce the model – the training code and training data. The most prominent open-source models have been built in the United States, led recently by the Allen Institute for AI’s Olmo models that I helped build in my recent 2.5 years there. The other prominent open-source models are also built by American non-profit organizations, including OpenAthena’s Marin models and EleutherAI’s Pythia models.
Open-weight, open-source, and every other label for a model – including closed models primarily offered via an API – exist on a spectrum. For example, Nvidia’s Nemotron models are far more open than most open-weight models, releasing large quantities of their training data under permissive licenses, but they’re not fully open-source because they do not release all of the data. Closed models also exist on a spectrum based on what information the API reveals and the terms of use.
The state of competition between American and Chinese open-weight models (unit economics, technical capabilities, etc.)
We are living in a world where GLM-5.2 and Kimi K3, some of the latest, leading Chinese models, have enacted a step change in the commercial viability of open models — crossing a similar threshold in agentic capabilities that Anthropic’s Claude Code crossed in December of 2025.
America was the early leader in open language models, primarily through Meta’s Llama models, which were used extensively across research and commercial tasks. Chinese open-weight models surpassed American open-weight models in these two key areas about 18 months ago. The simple metric showing this is Hugging Face Downloads, where China took the lead in July of 2025 primarily through the success of Alibaba’s Qwen models. I personally maintain tools to track this data, and since I first published the American Truly Open Models (ATOM) Project in August of 2025, China’s download lead has grown to about 1.6B – with a total of 3.2B downloads, twice that of America’s total.
On popular capabilities benchmarks, such as the Artificial Analysis Intelligence Index (AAII), the Chinese open-weight models have a clear lead over American counterparts. The top three Chinese models as of writing this on September 14, 2026 are Z.ai’s GLM-5.3 and GLM-5.3-Flash and Moonshot AI’s Kimi K3 with scores of 45, 42, and 44 respectively. By comparison, the leading American models are Thinking Machines’ Inkling and Inkling Small, both with a score of 26, and Nvidia’s Nemotron 3 Ultra, with a score of 23. The top American models were released in June and July of 2026, and are updated less frequently than their Chinese counterparts. For example, Chinese labs released models with scores above these American models 2-6 months before the American companies got there (e.g. GLM-5 or DeepSeek V4 Pro). There is a trend of more American companies releasing models, including names like Arcee AI, Poolside and IBM, but they are not rapidly closing this performance gap. Other benchmarks tell a similar story.
Together, Chinese open-weight models are approximately 2-5 months behind the closed American frontier, with the open-weight American models being approximately 6-9 months behind the likes of OpenAI and Anthropic. The Chinese labs are closest in tasks with clear user demand, such as agentic coding, and further behind on more open-ended scientific tasks, such as physics or biology.
The reasons why Chinese labs can produce these strong models, despite having fewer resources than American counterparts, is still an open debate and heavily influenced by different work cultures, but is also influenced by a few key technical factors. The Chinese labs release their models faster and focus on a slightly narrower distribution of tasks, flattering them slightly on public benchmarks. Releasing faster helps them score higher because all the labs are making consistent progress, so once you “finish” a model to be released, it is a snapshot of performance at that given time — labs where that time is later tend to score higher. Still, the models built by the Chinese labs are genuinely strong and represent real competition to the American industry. This competition will not decrease meaningfully as the closed labs patch vulnerabilities in their API offerings which enable distillation.
Distillation is most impactful in new domains and does not make it trivial to create a universally strong final model. I estimate that if distillation was fully prevented, e.g. with know-your-customer (KYC) tools at Anthropic and OpenAI, the gap from the strongest American models to Chinese open-weight models would only increase by 1-2 months.
For example, the Chinese labs are rapidly changing their posture towards paying for training data in 2026. Earlier in the year, the top Chinese labs including Moonshot AI and Z.ai had a strong preference towards building data workflows in-house, but by the summer they had begun to buy the cutting edge data – challenging RL environments for agentic tasks – from both established American companies and new Chinese startups.
With the advance of open weight models in China towards the frontier of capabilities, and the recent documentation of growing risks around frontier models in areas such as cybersecurity (e.g. the OpenAI-HuggingFace incident), there’s growing regulatory uncertainty on how continued releases can enable a safer ecosystem?
A structural challenge in open-weight models is that there are few effective methods for stopping pieces of open software from reaching bad actors. If an attempt was made to restrict access to the strongest open-weight models from China because they amplify risks, the parties who would be set back are American businesses. We have an example of this – HuggingFace used a Chinese open-weight model to understand the cyberattack because closed models would not answer their requests. Thus, managing the risks of open-weight models often comes down to ecosystem preparation.
Open-weight models are becoming an essential tool for AI diffusion, and the best path to get ahead of these risks and unbalanced relationships where American companies rely on models built in China is to continue to enable investment in open models in the US. Ownership of open models allows better coordination and preparation of risks that are global in their nature while accelerating diffusion of AI services throughout the domestic economy.
The state of open model adoption: How is open-source being used by academia, businesses, and other countries?
Open-weight language models have grown substantially in general interest and economic viability in 2026, allowing early glimpses of more direct ways to compare adoption of models from the US, China, or elsewhere on top of Hugging Face metrics. One example is OpenRouter usage. OpenRouter is a popular LLM inference platform that supplies a single interface to switch between models, open and closed, from the US and China. This platform is primarily known for trying different open-weight models. The platform has shared usage data for the top models since Jan. 1, 2025, and shown growth in usage from ~1T tokens processed from open models in a week of September 2025 to ~80T tokens per week today. In that time, Chinese models have grown from ~70% market share to over 80% of usage. Other platforms that are designed to commercialize open models show similar data, such as the open-source coding agent OpenCode, which shows an inference volume of ~95% or higher with Chinese models.
These open platforms are the best approximation of open model usage we have – a large proportion of open model usage is on platforms that do not disclose per-model breakdowns, such as Together AI or Fireworks AI, and in private deployments for enterprise applications.
Many prominent technology companies and startups have been building on Chinese open-weight models for their AI features, such as Harvey, the legal agent, Cursor, the coding agent, and DoorDash’s use of Kimi models, Airbnb’s use of Qwen, or Perplexity’s use of DeepSeek. These prominent companies are the tip of the iceberg, where a large swath of younger Silicon Valley startups are building on Chinese models in order to have low-cost, flexible options. There is a growing trend of American startups and companies entering enterprise agreements with Chinese model labs in order to get permission to use their models in their products – a new form of cross-border technology collaboration I have not witnessed in my career.
The foundation of innovation on Chinese models extends further into the AI ecosystem. To a first order approximation, most of academic research is conducted on Alibaba’s Qwen family of models. Having met multiple members of the Qwen leadership team during my trip to China, they are very invested in and intentional about this type of adoption, which will not be easy to claw back to American models.
To quantify the adoption of open models across academia, I scanned every paper in the 5 most popular ML categories of arXiv (cs.AI, cs.CL, cs.CV, cs.LG, stat.ML), the preprint platform popular in AI research. The results clearly track my understanding of the evolving leadership in AI research, showing LLMs becoming a foundational layer of ML research – mentions of any open model were 2% in January of 2023 and 50% in September of 2026 – and the leading role shift from the U.S. to China in the same time period.
For example, in April to May of 2023, a few months after Meta’s original Llama (a backronym, Large Language Model Meta AI, first released in Feb. of 2023), about 2,600 of 12,000 new AI/ML papers on arXiv mentioned at least one prominent open model family. Of all those scanned papers, ~5.5% mentioned Llama and ~1% mentioned a Chinese model. In the fall of 2024, during Llama’s peak, about 23% of papers mentioned Llama with about 7.5% mentioning Qwen, the most direct Chinese competition. Today, Llama has lost its lead in academia, being mentioned in about 21% of papers still, which is remarkable longevity, but Qwen’s share has risen to 30% of papers. Overall, any Chinese open weight model is mentioned in over 40% of papers, over the U.S.’s 30%, with China’s share continuing to grow.
This shows that we clearly have a lot of work to do in order to re-establish the U.S. as the home of AI research in the era of open-weight language models. There are signs of hope.
In our research, we find that American models of comparable capabilities-to-size regions to their Chinese counterparts get adopted at disproportionate rates. In the last year we’ve seen OpenAI’s first open-weight models since ChatGPT, gpt-oss, become one of the most adopted open-weight models of all time. Since then, Google’s Gemma 4 models have been some of the only ones ever to show similar adoption numbers to Qwen’s most popular small models, and Nvidia’s Nemotron models have modest adoption despite numerous more capable models at the same size point.
Summary
The story of open models in 2026 is one of establishing economic relevance. This is the convergence of many stories across the AI ecosystem, summarized as:
* The gap from open to closed models available to users has been decreasing over the last 3 years. This varies by task, but can be estimated as a 2-5 month gap in capabilities. With capabilities overall progressing so fast, this has seen open-weight AI models unlock substantial markets in 2026 and points to more inflection points in the near future.
* Open model usage is exploding in high-value industries (e.g. software engineering, legal services, financial services), indicating an emergence of an alternative ecosystem to the best closed models. Platforms offering inference primarily on open models, from Together, OpenRouter, Fireworks, Baseten, etc., are seeing incredible growth as the first winners of an open model post-training economy (other layers include finetuning APIs such as Thinking Machines’ Tinker). This is combined with numerous anecdotes from technical staff in the AI industry that uses open-weight models such as GLM-5.3 as an alternative to Claude or GPT due to a combination of speed, lower prices, customizable offerings, and privacy.
* Chinese AI companies are the clear leaders in open weight models. Relative to 2025, where Chinese models like DeepSeek R1 shook the AI world with surprise, the American AI labs have been recovering in their positions with open-weight models, but despite more substantial investment in the US, the Chinese labs regularly are producing notably stronger models adored by many types of users.
* Distillation of American AI models by Chinese labs does not explain the entire story of their success. Distillation is an industry standard technique of training another AI model on the outputs from a usually stronger model. The technique is most prevalent in the Chinese AI industry, which has used basic exploits to extract reasoning traces and additional data from American companies’ products that are not fully secured. The best estimates are that distillation helps reduce the performance gap of Chinese companies relative to the American frontier by 1-2 months.
* Chinese models, particularly Alibaba’s Qwen family, are established as a foundational layer of research and development across academia and local model users. In recent months, Chinese open weight models were mentioned in 38% of AI papers, above the U.S.’s 28% – and the Chinese share is growing much faster than its American counterparts. This, along with other political factors and the closed nature of leading American AI companies, is contributing to an accelerated decline in America’s lead as the preeminent AI research hub in the world.
* Open weight models are entering the capability levels where new risks, e.g. cybersecurity, can be enabled by numerous open-weight models being available, necessitating an ecosystem level response in preparation. This new era of risks is also enabling a period of political uncertainty, where there is regulatory attention on the strongest AI models, but massive uncertainty on how policy would be legally enacted. At the same time, many researchers and engineers rely on open models due to more permissive safeguards, where the closed models such as Claude and GPT often refuse critical cybersecurity defensive work or biology research.
For more data, view the Interconnects Dashboard.
Conclusions
In 2026 the Chinese labs are clearly maintaining their status as the leaders of the open-weight AI ecosystem. This comes as open-weight models have passed an inflection point in economic viability and in the face of increased activity from American labs as model competition. The leading Chinese labs do not appear to be meaningfully challenged, as they expand their enterprise and research adoption globally.
This landscape of open models comes at a crucial time in the broader AI ecosystem. We’re seeing OpenAI and Anthropic take massive steps forward with their latest public models, and at the same time call for coordinated care on how we manage the next stage of AI progress. What is happening in the confines of a few AI labs today, especially with extreme talent and compute density, is a precursor to what will soon emerge in the open model ecosystem. Open models are going to be the substrate for everyone else in the world outside of the few true frontier AI labs, to harness an acceleration in software engineering and other computational practices. This represents a substantial source of soft power, influence, and potential for the organizations that enable this broad access to transformative intelligence.
With this future coming soon, we need to collectively stay humble about the exact path open models will take. There are a lot of unknowns with open models – e.g. we don’t have good data on how they’re used in countries other than the U.S. and China. With the distribution of ML training expertise being broad, i.e. tens of organizations and thousands of people that are within a year of the frontier of capabilities, it is a matter of when, not if, open models cross the performance thresholds that enable new workflows. The collective approach should be to understand how to use this broadly accessible, open intelligence for good while proactively mitigating the potential harms.
Thank you to Florian Brand and Kevin Xu for feedback and/or suggestions for this work. For more research informing this post, see the open-source AI reading list.


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe

We’re in an era where a few organizations are using thousands of concurrent agents to improve their processes and output. These organizations happen to be just the frontier AI labs, in particular OpenAI and Anthropic. In the last few weeks, I’ve been pondering what it means for so many employees across these organizations to rapidly update their expectations for the pace of AI progress and associated risks.
A core perspective I have is that the frontier labs and broader frenetic, competitive culture in the San Francisco AI scene set up an environment that amplifies any AI concern. This has some benefits in causing more general audience awareness of AI, as fear sells, but exaggerating risk timelines or severity will have negative second-order effects. I remember many loud AI safety debates, and their associated clouds over the viability of open-source AI, in 2023 and 2024 — the primary risks then did not arrive in the forecasted timelines.
The general populace of these two key labs was very anxious about AI risks and the rate of progress even a year ago, and especially as agents got stronger product-market fit at the start of 2026. This cultural precondition, when exposed to the reality that thousands of agents will constantly be working fairly productively in your business, will only increase this anxiety. The step from this anxiety, and incidents like OpenAI-HuggingFace, to extinction risks feels very religious.
Richard Ngo had an apt summary of the situation:
Now a large proportion of the AI safety community is implicitly or explicitly orienting to futures where an intelligence explosion occurs within a few years. My default expectation (absent an extensive pause) is that a similar thing will happen: they’ll turn out to be directionally correct (relative to the expectations of almost anyone not linked to the community) but factually wrong. Specifically, we won’t have superintelligence within the next 8 years, but things will still be moving so fast that it’ll *feel* like the people who argued for short timelines were right.
… I wanted to say something now because it feels like the level of bandwagoning towards “singularity soon” is getting pretty wild.
Personally, I think this view aligns closely to what I outlined in my alternate scenario to true recursive self-improvement (RSI), which I called lossy self-improvement. A summary of this view is that:
* Automatable research is too narrow to achieve a massive net acceleration in progress, in the face of scaling laws’ exponential costs,
* Diminishing returns of more AI agents in parallel are real, &
* Resource bottlenecks and politics are a major factor in building strong LLMs (and AI can do much less to accelerate this).
So, I’m left balancing the above, latent increase in the cultural temperature with the potential that the labs have seen genuinely scary, specific breakthroughs that are not public yet. My expectation is that more of the current AI safety concern is on the former – scaled agents working – but I hold high levels of uncertainty here. Foundational, imagination-based AI breakthroughs are the sort of thing that would make me update my RSI timelines from closer to a tool to sustain progress in the face of exponential costs (scaling laws), to something more unpredictable and/or unstable.
Interconnects AI is a reader-supported publication. Consider becoming a subscriber.

Some of the best recent resources on RSI have been Dwarkesh’s podcasts with Noam Brown and the trio of John Schulman, Beren Millidge and Charlie O’Neill. I have a few important reflections from both of them.
First, the podcast with Noam Brown made me internalize how big of a short-term acceleration mass inference capacity is. These labs will throw thousands of agents at important, measurable problems. At the same time, compute capacity available to them is going to continue to scale. I have my doubts that the labs can afford to spend a constant portion of this compute on internal R&D as the total volume goes up, especially with plans to IPO, as they face increased scrutiny on basic economics. It is important to not confuse massive steps in inference-time scaling, a dynamic which should be fairly predictable, with being the outputs of RSI, which is highly uncertain.
Second, the trio podcast debating the state of the art in technical capacities induced more of a surprising reaction that I haven’t fully settled. Through the first hour or so of this podcast, where they debate the role of RL, distillation, scaling, inference-time compute, etc., I found myself strongly agreeing with the distribution of claims. A TLDR would be that our current techniques work and let us solve problems we know how to state, but they don’t result in a magical level of generalization to unknown, harder problems in most partially verifiable domains (i.e. progress in math is an exception, rather than a rule).
The surprise of this podcast was the end, where they were predicting timelines for various thresholds of AI. I had GPT-6-Astra summarize the answers provided to three questions from Dwarkesh, of the form “when will AI reach X ability”:
All timelines are relative to the interview date.
* Drop-in remote worker for broad white-collar work over a month
* Charlie O’Neill: ~1 year with programmatic access to workplace tools; ~2 years if it must operate through a browser. Means ordinary white-collar work, not highly creative research.
* Beren Millidge: ~3 years for full generality; 80–90% coverage sooner. Main uncertainties: online learning and the long tail of tasks.
* John Schulman: ~1 year for an “okay” version, with uneven capabilities that improve over time.
* 10× productivity uplift for AI researchers
* Charlie O’Neill: 5–10 years. Bottleneck: absorbing information and deciding which experiment to run next.
* Beren Millidge: Finds John’s ~2-year estimate plausible, but gives no independent timeline. Assumes AI can run successive experiments and learn from feedback; other bottlenecks would remain.
* John Schulman: ~2 years.
* AI surpassing top human experts across all computer-based work, including multiyear projects (“ASI”)
* Charlie O’Neill: 5–10 years. Highlights limitations in memory and context length.
* Beren Millidge: ~5 years for areas labs focus on; potentially longer for literally every domain. Gives no firm timeline for the universal version.
* John Schulman: 3–4 years. Spatial/physical fields may take longer; requires onboarding and solving longer-horizon learning.
Roughly, a recurring problem when discussing RSI is a lack of specification in intelligence. The jaggedness of intelligence means that we need to discuss thresholds in specific, measurable tasks. The nature of LLMs’ intelligence is shaped very differently than humans, and the roles we forecast are human-shaped. AIs, therefore, do not cross these thresholds like remote worker or AI researcher discretely. It’s a slow diffusion, and a form of long tail will always exist.
Take the case of productivity of AI researchers. Many people under-index how much of science is communication and standard setting with colleagues. I do buy the cycle of experiment design and testing being 10x faster in the near future, but not hypothesis generation and intuition building. Accelerating understanding will be the key bottleneck – and it is one that despite all of the AI tools getting massively improved, humans will only improve marginally in their capability. A big improvement in the nature of science will be enabling humans to invest more time here, not them becoming exponentially better at it.
This links back to the Noam podcast. Agent swarms in the near future will be effective at solving clear, open problems with verifiable answers. In this vein, when it comes to improving AI models, RSI is much more helpful at efficiency rather than expanding peak intelligence. This is due to the fact that LLM serving has clear metrics you want to improve that are measurable and malleable. This’ll enable better inference-time scaling and more efficient multi-agent systems.
Still, I cannot get past the fact that all of our scaling laws show that you need exponential compute and resources to make linear improvements in intelligence. RSI is poised to make modern LLMs vastly cheaper. Trends that have shown LLMs get exponentially cheaper at a given intelligence are likely to accelerate. A crucial factor for the labs will be increasing margins as revenue could potentially have negative pressure if there’s fierce competition in lowering prices at a fixed intelligence level — Jevons paradox will likely prevail, resulting in strong businesses.
RSI factors will have a much harder time improving pieces of the LLM puzzle like managing complex post-training recipes. There were a few quotes from John Schulman that I strongly agree with on the state of post-training at the labs:
If I think about a post-training team and why you need a lot of people on the team, it’s just because there are a lot of different areas where you have to figure out how the model should behave. It would be very hard to automate the whole thing, just because someone has to think about how the model should behave in this area.
and later:
It’s really easy to screw up post-training in some way that doesn’t show up in benchmarks.
These tasks are uniquely hard for current LLMs. Yes, they’ll get better as the industry is still rapidly scaling RL environments related to these domains, but this paradigm does not last forever. In the near future, it could become exponentially harder to conceive, build, and test new environments that meaningfully challenge the leading LLMs – these hard environments are the ones that are crucial as a learning signal in RL.
OpenAI and Anthropic have shared a good amount of internal measurements related to RSI, and my current read is that the biggest takeoff in automation within the labs is in tasks like software engineering, monitoring logs, managing planned experiments, and other fairly routine (but not always easy) tasks. For example, I was surprised by this language in the recent Claude Fable 5.1 & Mythos 5.1 System Card:
We believe that internal usage of recent AI models has been a key factor in maintaining the current rate of progress, but we do not yet see clear signs of dramatic acceleration beyond that rate.
Altogether, I think the hardest exponential we are fighting is on peak intelligence. That is the hardest one to budge or even accelerate. Still, my mental model for the very early innings of RSI is more of massively scaling and diffusing inference-time compute to AI research and related activities, which has a large amount of low-hanging fruit available. This, on its own, is still poised to be economically transformative. It may also unlock more resources to push on AI diffusion, which is the crucial bottleneck in unlocking much of the potential benefits of AI.
For now and until more evidence emerges, lossy self-improvement remains my baseline on the trajectory of progress, and the increased discussion of extinction risk seems very misplaced. As always, things can change fast in AI.


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe

As AI became more powerful, it was inevitable that a different, growing group would start to take AI safety more seriously – what we did not know ahead of time, is which set of views they latched onto. We have seen that some of the most extreme views of risk, i.e. moderate probabilities of mass extinction, were the ones that reached the masses. A lot in the AI world is about to change due to this.
How did we get here? Why did this quitting announcement reach so far? In many ways, the rest of the world’s views around AI in the past was a dampening factor. You can think about this like the damp ground around a fire. Many people were striking matches for years about AI risk – they’d smolder in their community and largely burn out, going unnoticed. As the stakes of AI have risen this year, from the OpenAI-HuggingFace incident and breakthroughs like the Navier-Stokes result (also from OpenAI), the ground has dried out and the latent energy around the AI discourse has increased. More people not in the industry have thought, “huh, maybe I should care about this AI thing.” The ambient temperature and stakes have been obviously rising.
Then, some basic factors of human nature apply, with the most crucial being that fear sells. Fear is the simplest story, the one people cannot look away from. Jacob Coxon was the one who stumbled into this new powder keg, totally unaware of what was going to come. What looked like a fairly innocuous event – another AI researcher quitting citing safety risks – landed into a very different environment and it caught like wildfire. The discussion of existential risk, mass extinction, and the trajectory of AI has traveled further than even the most seasoned AI commentariat would ever predict.
There are a set of facts we need to get clear, which paint the picture of the situation. The key Tweets to reference are from Jacob Coxon, the resignation thread, and Evan Hubinger, the source of the >10% extinction risk figure .
* There are plenty of AI risks which are likely to cause harm, even if estimating annihilation is useless. It is important to weigh these with respect to the benefits. The entire discourse around existential risk is on very poor footing. At least Evan was clear in his post, with “kill all humans,” but a major problem in the AI Safety discourse is that people talk about existential risks, when they mean very different things (much like how AGI is a vaguely meaningless term). I put the probability of complete extinction as being so low it isn’t worth discussing, but the probabilities of AI caused disasters – e.g. cyber attacks on critical infrastructure or bio-risks – as being worth debating. Throwing this whole discussion out because there are not these disasters yet is a harmful reaction.
* Jacob Coxon is acting genuinely and with good intentions. The outpouring of support from more well-established AI researchers who know of him and his intentions of resignation is useful. Many factions of AI turned to scapegoating him individually, based on account metadata, personal factors, etc. These are not useful. Many frontier lab employees genuinely have similar views to him. I’m not sure it’s a majority, but there is a substantial group.
* Many frontier lab employees, especially at Anthropic, are out of touch and this will impact their forecasting and/or descriptions of current AI events. I say this without blaming individuals, but it’s a common agreement among my friends not at OpenAI/Anthropic (Ant especially) that people at the labs operate with a religious energy. It’s very common to go through very out of touch interactions with them. I do not blame most of the individuals who get distorted views being part of these companies, but the interactions are wild and spill over into a lot of wack discussions in the AI media ecosystem. Living in this environment that normalizes such out of touch behavior will inevitably distort any human’s understanding of technical progress.
Interconnects AI is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

* This was not a mass political campaign, but rather an opportunistic media coordination. For context, the Wall Street Journal had an exclusive story that Jacob coordinated before posting. I suspect that Jacob shared his plan of quitting in groupchats with AI safety advocacy groups ahead of time, e.g. the morning of posting, asking for amplification. This is normal practice, and could have included some prominent politicians. From there, I think it’s more likely that other politicians are bandwagoning on a rising issue. When you combine this with other factors, like Daniel Kokotajlo’s appearance on Joe Rogan coming out the same day – it definitely looks like a very well-executed, coordinated media campaign. This doesn’t mean it’s a conspiracy or a regulatory capture tactic within Democratic political structures. The determining factor seems to be that no one – including Jacob and those posting about X-risk today – knew that it would go so viral.
* We do not have proof that RSI causes the risks these researchers forecast. The general argument for RSI follows as: The current pace of progress is very high, the current progress is heavily dependent on AI tools, the current AI tools are superhuman in some domains (e.g. math) – so, all together, AI is going to work more on itself and become superhuman in all relevant areas over time to autonomy and intellect. This view dramatically undersells human bottlenecks in building models and allocating resources at organizations, and draws conclusions on future AI capabilities more broadly.I called my alternative view to this, Lossy self-improvement. AI has always been very jagged, and we are making models which are superhuman goal-seekers at math and software engineering, but they have massive limitations on intuitions, creativity, and other types of reasoning that humans are strong at. With AI agents assisting research, we will rapidly find the areas where AI is superhuman – and I expect there to be well more than just research mathematics – but it won’t be a panacea for the current limitations of our approaches to LLMs.
There is another understandable social dynamic at play here, causing many deep AI insiders to overstate the returns from RSI. Many of these researchers were the earliest people to bet on AI’s progress, and the extent to which they were visionaries should not be downplayed (see Ilya’s comments on deep learning as early as 2015). They have been right again and again, forecasting AI’s capabilities better than I certainly could have guessed. This does not, though, mean that their forecast of what will come next will be right. The core idea of RSI is a way to spend more compute on the process of developing a model recipe, rather than just spending more compute on the training run itself. We’re seeing benefits from it, but I argue the expected return on that input is far less than they believe.
Their argument is that RSI will make AI progress go exponential, make it so we cannot monitor the technology, and enable rogue models and new forms of risk. This scenario is often called “Fast Takeoff”. We have not seen the stacking efficiency gains that massively reduce model size and cost, that would lead to an explosion in progress by allowing consistent speedups in experimentation.
* The biggest short-term risk could be from the AI labs not taking safety seriously enough – they haven’t hardened their own infrastructure, enabling AI misuse to proliferate. From my earlier post on the HuggingFace-OpenAI incident, Lessons from the hacks:
* Frontier labs do not seem like they’re watching the models closely enough, due to a general frenetic competitive environment & current SF culture

From OpenAI’s own retrospective, the misaligned model behavior was unfolding over months, and in some cases OpenAI did not know about the hacks for ~weeks. The time to response is too long and I do not think this is an OpenAI only characteristic – rather it is that the frontier labs continually seem underwater in the amount of work they feel like they should do. I am not optimistic in the long-term that the labs change a sufficient amount here to meaningfully mitigate this type of oversight risk in the future. Yes, it is very likely that OpenAI is putting a ton into understanding this – and delayed their latest models to make sure they get it right – but the financial pressure to grow revenue or risk the companies’ long-term balance sheets makes me think it will not be a sustained pattern of caution.
Overall, I think this episode is very bad for the AI ecosystem. It’s pushed the acceptable views in the AI community closer to the extremes. More accelerationists will discount the need for any form of safety, citing mass delusion of the “doomers.” It feels like a very narrow path to believe in AI risks, but to not worry about extinction from the technology.
For example, it is a horrible temporary period for cybersecurity, where AI models going a bit off script and poking around unintended pieces of the web seems like a new normal. This is accelerated by the labs competing veraciously towards their views of AGI, and a slow uptake in the necessary hardening of our cyber infrastructure around the world. This doesn’t mean that it’s an existential risk and something we cannot solve. Each risk will have its own set of solutions and paths forward.
I feel particularly exposed in the current environment as a supporter of open models. If an open model were to be used by a third party organization to intentionally hack another company — similar to how the OpenAI-HuggingFace incident went down, but intentional — my expected outcome would be a severe restriction on the development of stronger open models going forward. Open models are needed for many organizations to perform this cyber hardening, and to maintain the ability to adapt to new forms of AI risks in the future.
Through all of this, we need to stay grounded on what is actually unfolding. Yes, monitoring AI’s behavior is heavily reliant on other AI models, which adds in new types of monitoring risks. These are not inherently insolvable. A recurring read of mine on the emerging agent swarms is that they’re attempting to do a task given to them, and they’re using skills we didn’t know they yet had to circumvent the intended path to success. This is a huge win, as when you squint, the AIs are doing what we told them to do. The models are certainly very odd, and we should accelerate our progress on understanding them, but these swarms are far from being novel independent entities. The models are trained to coordinate on tasks, to write down their progress, and to be extremely persistent. There will be new oddities we find in the future, but prescribing current uncertainty on how AI works to future certainty that we cannot understand AI is a form of giving up.
In this world, we need to rely on the rule of law and science. If the AI labs are not able to do enough safety research themselves to understand the models, they should be more transparent on what is happening so more scientists can make progress on the problem. If an AI lab commits crimes unintentionally, they should be punished, so they have clear incentives to prevent it in the future.
It is a natural reaction to things changing very fast to feel more uncertain about how to create good outcomes — that is actually the correct mental update. We need to use this humility to motivate ambitious solutions.


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe

Housekeeping: Paid subscribers to Interconnects now get a permanent 40% discount on my book when purchasing at Manning.com. Access the code at the Interconnects perks page.
Many AI optimists tend to compare what is happening in this AI boom to the industrial revolution, or to other periods of rapid technological advancement and diffusion into society. These comparisons fit on the scale of technological change, but miss a crucial factor in how most people are exposed to that change. The problem facing AI is that most people have no super tangible new goods thanks to it and society has more inertia resisting change than in previous eras.
I’m writing this coming back online from a few weeks off for my wedding in New England. In this time it would have been very easy to not think about AI at all. The touch-points that average people have to AI products today are fringe, marginally beneficial, or even just very confusing to them (e.g. many people have heard about and brought up the OpenAI-HuggingFace incident, but don’t know what to make of it). People on the positive side think of AI as a way to make fun images, enhanced Google Search, etc. These are very small benefits. On the negative side is an association with addictive social media algorithms, friends of friends addicted to AI chatbots, and a plethora of takes on data centers.
AI is still a rounding error in everyday life
Core aspects of everyday life — family, food, transportation, and entertainment — have few direct impacts yet. It’s a remarkable breath of fresh air to pop out of the bubble and realize how little what is happening really matters today. Being obsessed with AI is a choice that a very few people have yet opted into. For example, the only thing I used AI for in this time was search and creative work (making the pretty seating chart for my wedding guests to find their table).
In industrial revolutions past, average people got absolutely life changing outcomes. The First Industrial Revolution in the late 18th century gave access to cheaper clothing, cooking ware, reading material, and a shift to new livelihoods. The Second Industrial Revolution in the late 19th century introduced household machines (e.g. sewing machines), preserved food, indoor plumbing, photography, better light sources, bicycles, and further benefits of manufactured goods and electrification. The list is remarkable — most of these we still use regularly today — and very physical.
While even the most optimistic versions of AI will usher in new scientific discoveries, advanced therapeutics for rare diseases, and potentially even sustained economic abundance, these benefits have the risk of being too indirect.
How will a common citizen come to credit OpenAI or Anthropic for saving their life, if they went to their family doctor who told them about a new miracle cure?
What percentage of Americans will care about OpenAI solving the Navier-Stokes Millennium Prize Problem?
It feels very likely in 50 years that the average American’s day to day life looks very similar. Their home, appliances, relationships, and vehicles will be similar (of course, self-driving will continue to diffuse, but that has been developing on a very independent trajectory from the innovations of LLMs). In this time, AI will get a lot of credit. 50 years is a remarkable length of time with how fast everything is changing today in this, AI-focused narrow slice of the world.
The most important part of what is happening early in the AI revolution, is building foundational infrastructure, and a general process, which will compound over decades. A major mathematical breakthrough today will look astonishingly minor in scope relative to the advancements that come later in the compounding journey. It is hard to predict what it looks like for every technology you use daily to get faster compounding improvements due to AI.
Interconnects AI is a reader-supported publication. Consider becoming a subscriber.

Breaking social stasis
Much of the narrative around AI is trying to push people to care, due to this long-term reality of progress, at least as a subconscious motive. This will take a long, long time to get right, and AI’s buildout faces immediate political problems due to this imbalance.
Today’s AI is primarily a tool to serve the elite. For knowledge work, which is roughly half of the U.S. economy, AI is as fundamental as electricity (or quickly will be so, with rapid improvements to agents in the next 18 months). It’s highly destabilizing to have such a transformative, productive tool only bring half of society along. It’s not hard for many people to pick up on this — the technology economy booms while life stays otherwise stagnant.
In writing this, I learned of Engels’ pause, which is “the period from 1790 to 1840, when British working-class wages stagnated and per-capita gross domestic product expanded rapidly during a technological upheaval.” If we — the leaders of the AI industry — think this is the closest analogue to what comes next for AI, those not benefiting are right to push back.
AI is the greatest tool ever for scaling technology companies and starting new online-native small businesses. I don’t even expect the tech industry to grow in headcount and nurture its workers through an era of massive success — I agree with Doug OLaughlin that headcount would likely shrink while knowledge work output explodes. These sectors were already the most successful in the American economic system, so the brand of AI will be tarnished as not being a collective good. I worry that this instinctive reaction will kneecap AI’s development, sending it down a path that looks closer to the cautionary tale of American nuclear power.
Part of the challenge is the speed and relentlessness of expectations in society. The AI industry has millions of eyes on it, and won’t get much patience to wait and bring innovations later. If given 100 years to diffuse into society, its impacts will certainly become much more obvious, like the industrial revolutions of centuries past.
Together, the AI industry is facing a few simple issues, in what I would call the first half decade of 50-year diffusion process.
* AI’s positive impacts early in its evolution are too indirect.
* AI is facing a political backlash deeply intertwined with the history of Big Tech in Western society. This is only an AI story due to timing, and if AI’s exponential growth came decades after the growing pains of today’s technology platforms like Google and Meta, it seems likely that the datacenter issue would’ve never risen to such a central political position.
Solving either of these would alleviate a substantial amount of pressure, and give the AI industry a lot more time in showing the positive case for why people should be okay with changes to the status quo (primarily economic). These are both made more challenging by AI self-labeling itself as negative and/or unsafe technology, through proclamations of doom and mass unemployment. The leading figures have begun addressing this issue, but the public needs more work to fully buy into the overarching trajectory.
When zooming out long-term, I could see robotics and self-driving becoming closely linked in storytelling to the current AI revolution. If the intelligence explosion from mass-producing large language models does spill over into enabling the acceleration of robots in everyday life, humans will quickly latch onto the tangible benefits of AI. This is ironic, as many people have spent time trying to convince people that what is happening specifically with LLMs is very different than the previous decade or two of general AI progress. If the same dynamic later saved (or massively overshadowed) LLMs, it would be funny.
Reflecting on what I expect the history of this era to look like, it feels a lot like growing pains of AI. Society needed to break out of old habits and work through problems that predate ChatGPT — which releases a lot of energy and frustration — in order to tap into the longer term growth. The diffusion story will take a lot longer than the fight against it. All of us younger folk following the story today will get to see powerful AI go from effectively 0% to 90%+ full adoption in our lifetime. This sort of AI that is deeply integrated in businesses, acting as personal assistants, etc. is just starting to become viable. It’ll take far longer to gain adoption than easier to understand applications like ChatGPT, and is the true marker of AI’s evolution.
Taking this perspective makes it clear that it is crucial to keep progressing the technology — the benefits will be astounding, but they are not a given — and we have a lot of very hard work to do in making sure they’re distributed widely.


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe

There are a lot of criticisms of AI writing, but most of them are focused on more creative, high-voice writing like this blog. Those — including my own piece — often argue that it is because good writing is high-voice, has a point of view, has a deep human expression that needs to come across, and or a process of thinking that you peek into with the chosen words. As LLMs get more refined as tools, rather than conversational assistants, I think we are actually going backwards on our goals of having models produce inspiring writing.
On the other side of things is non-fiction writing. Filler, copy text was one of the genuinely useful abilities of an LLM (Sam Altman said so much about the early business of GPT-3 on a recent podcast). It has seemed like any flaws here were mostly down to a general lack of intelligence in the models, or some other training issue, and all non-fiction and explanatory text would get obliterated by the rapid pace of progress eventually. Having worked with the models as a writing assistant over the last few years, they’ve gotten a bit better, but it’s worth reflecting on what’s holding them back.
Models being stagnant in long-form, non-fiction writing should be alarming to those reliant on models autonomously solving grand, open science problems in the near future. The models today struggle to organize and compellingly present some of the most established science in their area. This seems like a natural prerequisite that we should expect the models to master before they can solve broad, open-ended problems on their own. Until this is solved, the progress of LLMs for science will look closer to solving low-hanging fruit and merging distant connections across fields, rather than any sort of revolutionary insight.
This is a somewhat controversial take for someone who is very optimistic about AI’s progress, especially writing it on the day that Anthropic published a blog post on Claude making some progress on the famous Riemann Hypothesis. Scientific problems have a vast breadth, and I don’t think current AI models have as much coverage as many think.
Organizing knowledge is a compression. This compression is needed to make insight. Today’s LLMs increase entropy in long-form non-fiction writing, and I don’t see how that can be stacked on top of itself endlessly. They’ll be reliant on humans acting as sort of guides.
I am still very optimistic about translation from these narrow forms of science, like the extreme advancements we’ve seen in math, into consistent, broader progress — LLMs are the most powerful assistants scientists have ever used. I first need to explain how observing the models work on such grounded, low-level knowledge problems in writing makes me see a surprising lack of generalization.
For more context, I just finished writing a post-training textbook, Reinforcement Learning from Human Feedback (buy on Manning or Amazon). I used LLMs in many ways to support this, from helping wrangle LaTeX formatting for equations, doing extensive copyediting, and creating diagrams for programming languages like TikZ (in LaTeX) or Python.
Why have models stagnated in writing quality?
I would’ve expected way more progress on non-fiction writing from the models. I almost thought I would look dumb publishing a non-fiction book in 2026, given how things looked in 2024. Today, some of the most famous models on writing ability are pretty old, examples include OpenAI’s big GPT 4.5 and Moonshot’s Kimi K2. In and around these releases, the models have gone from okay to superhuman at other tasks like coding and mathematics. Maybe a closer, but still imperfect, comparison is how the models went from incapable to decent at search and research tasks. The pace of progress on most other skills is steep, but writing well feels orthogonal to most of them. I do not think writing is just ignored, but rather it’s challenging and lacks good training data to specifically intervene on it.
There is certainly some low-hanging fruit for making AI models better at writing — such as specialized harnesses like Claude Code, prompts, and training environments that make models spend a lot more inference tokens on the output, but I don’t think these will have a multiplicative impact on ability. Writing well is a very hard task! It’s a shame that we haven’t unlocked inference-time scaling for one of the great intellectual pursuits. Regardless, writing seems very different than what the models are good at.
Today, the models seem genuinely horrible at long-form technical writing. They can get a sentence right, but if you try and get them to write an entire chapter it’ll be a mix of sprinkled with confusing wording, muddled in its organization, and generally a bit off. They try to be too cute where they don’t need to be and in the process make random conceptual errors. The models in the near future will get much better at the small errors, especially as models get bigger — which allows them to hold more world knowledge — but I do not expect their ability to utilize it to transform.
For example, the GPT models have been incredible at finding typos and minor issues for a long time. I passed a near-final draft of my book as a PDF to GPT 5.5 Pro and it found deep, surprising minor typos across the manuscript that is 200-300 pages.
On the other hand, the Claude models have been much more useful as an editor. They have a lot more taste, tend to understand the mental model of the task better, and have more interesting suggestions to unstick the different forms of writer’s block.
The examples I’ve given above all have a sort of consistent theme. The models know how to check every unit of content, in this case usually a sentence or equation or figure, or make one, specific section where you are caught. With these skills, they don’t do a good job revisiting components and stringing them together as they make many additions on top of each other. It feels like a sort of irreducible compounding errors. We used to deal with these errors in math and code, but reflecting on it, RLVR has been a truly magical solution in reducing them.
Interconnects AI is a reader-supported publication. Consider becoming a subscriber.

Getting value out of current models as a writer
I’m willing to share that there are a few technical explanation sentences in my book that came from an AI model — well less than 1% — they’re there because I really loved them. I let myself consider including some AI tokens in the book, as it didn’t feel like cheating if I, as a true expert, felt that the sentence was what the reader needed. Especially in the editing process, where I had a very close eye on things and plenty of concern on if my book would ever be done with all the things I have going on, it was an extremely valuable path forward.
For example, I had a list of questions from my editor interspersed in a LaTeX file with a specific delimiter like \editor{}. I would have Claude Code navigate to each comment, print the context before and after, and let me know if it was an easy typo fix or something more nuanced. I would write a response — the text to insert — or ask Claude for suggestions before fixing it. Intellectually it is a very focusing process of editing, it was a fun way to improve the book. Sometimes phrases from Claude’s suggestions are what made it into the book.
It is definitely a slippery slope and when I accepted a few AI suggestions it was at the point where I was going through my second full-manuscript review. Emotionally the project felt completed but I had more work to do. Coming out of the textbook-writing process I so deeply appreciate the cut and dry rule I have for my writing on Interconnects to never use AI outputs in the content. It is way more fun to write in a way that is only you — high voice, valued so deeply for the process — but writing a standard reference is not really an activity known for being fun. I see why people turn AI tools into a crutch when most of their writing is just an output to fill space, rather than a means to an end. I am motivated to write voluminously to learn, to feel, and to express.
I am working through similar balances in my scientific work too. AI models are great for repetitive pieces of the paper, like drafting a related work or background section that you know by heart, but using them for the abstract, introduction, experiments, or conclusion is a shame. Those are where the story and soul of the work is communicated — it’s where you learn what your research is really about.
I am confident I created a lot more net value by being able to have AI models create and check my non-fiction writing work. They make writing equations trivial, can help refactor the repository, port between languages, and many other things. At the beginning, it was very fun, until I was a bit worn down by the length of the publishing process, watching the field move on.
For an example of why AI was crucial in this case, I had to maintain Markdown and LaTeX versions of my book simultaneously in two spots, as readers gave feedback on the web version and my Manning editorial team reviewed a forked copy. Without AI agents, syncing between the two of them would’ve easily taken me five times as long (and this task took tens of hours already).
Something intertwined with this story, which I stumbled upon when thinking about agents, is how your pace of understanding won’t increase by using agents. That understanding, in the form of intuition, taste, instinct, etc. is what will be valuable in the future. Using AI for non-fiction writing takes away from that progression. Doubly, if you weren’t already an expert you won’t be able to catch its flaws.
In my case, I felt such an urgency to dump the knowledge out of my brain onto the page that there were times that using the AI models was a worthy tool. Much of the motivation of my book was to have a single reference for important post-training methods like rejection sampling or character training, where very little exists on the web.
This textbook was so much of giving back to the community, that it was just such a win to complete it in any form, that I felt it was okay. I would’ve learned more and the product could’ve been marginally improved with more human effort, I am sure. The determining factor was that I felt like the book was going to be aged out by the time it was published, a fear of AI model’s capabilities on one side and how fast the field moves on the other.
This turned out to be really wrong? I’m very happy with the result and I’m more confident in its staying power now than when I started in 2024, as the models have so failed to live up to the hype in non-fiction writing.
Where technical writing goes from here
The models are incredible tools, they let you express knowledge in different forms. They’re wonderful for creating creative filler or background material — e.g. the first draft of slides whose real value is being a talking point for the teacher to lecture over — that let any knowledge be transformed from one medium to another.
There’s some subtle, early phase of writing a non-fiction or reference textbook that feels a bit closer to writing a high-voice blog post like this. When pushing through the early organization and the presentation of the core skeleton new knowledge is created. This is the part that takes insight, and the LLMs are far behind in being able to replace it.
The crux of the above paragraph and preceding section is that I would be happy if more of the world’s experts used AI models to write a tiny bit of their books in order to get more of their knowledge shared with the world. The problem is that you can only use AI models to save 10-20% of the effort today, and I don’t see that percentage becoming the majority anytime soon.
There’s also the social pressure, where people expect LLMs to be the best, personalized educators out there, so they think working on a book or educational content is pointless. I think some of these opinions are aging out, as there’s a massive dearth in the highest quality educational work — and there always has been. AI is great at manipulating said content into the form that suits the student, not creating the content from scratch.
In the meantime I feel that we are stuck in a frustrating local minimum, where AI models are going to on net reduce the average effort spent on non-fiction writing, but they could enable great expression. Fewer people will start and push through.
So, in 2-5 years I still expect the best textbooks to be heavily crafted by the human hand. I’m not sure after then, but that’s longer than many would’ve predicted, given just how much knowledge these models have and their structural propensity to stream it.
As for a conclusion on capabilities, the models are great in two contexts: 1) any truly verifiable domain and 2) when given a ton of context and making a small edit — like finding a bug or solving a very specific math problem or giving feedback — not generating prose in an open-ended manner. Long-form writing will definitely fall before creative writing, but it’s a strong tell that the models are not able to express the full extent of their knowledge in underspecified problems. As we try to push the models to be something like “geniuses in a datacenter” solving grand scientific problems, this seems like a fairly fundamental limitation.
Jasmine Sun had a great piece on why LLMs make good editors, while being bad writers too.


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe