エピソード

  • Podcast: Kimi K3.1 Pro, the new cyber reality, and Muse musings
    2026/10/25
    We — Nathan and Florian — wanted to discuss all the messy trade-offs of a busy week in AI and ask all the pressing questions like:* Why did Kimi K3.1 Pro have a mixed reaction with such incredible benchmarks? Will it be adopted?* Were we wrong about the risks of open-weight models and cybersecurity?* Why do cyber experts discuss recent events so differently than AI experts?* What comes next in the few months? What will change our minds?We will write more about this subject, but wanted to bring you behind the curtain as we aspire to quickly learn more about the cybersecurity industry and trends across the last few years that have been leading us today. We conclude the podcast with some lighter topics, from Muse-maxing to excitement in still hiring junior technical talent. Please send us questions and topics, as we plan to record more of these podcasts going forward.Listen on Apple Podcasts, Spotify, and wherever you get your podcasts. For other Interconnects podcasts, go here. YouTube ↓Chapters* 00:00 Kimi K3.1 Reaction* 08:15 Kimi’s open-weight cyber principles* 11:23 You can hack everything era* 15:54 PHP2Shell, LiteLLM, and the new internet* 24:00 Abliterated APIs, safeguards, and trusted access* 31:41 Anthropic not hiring juniors employees* 36:16 Florian’s Muse addiction* 45:30 Wrap-upLinks we discuss* My thread with Dean Ball about the “boy who cried wolf” story of the frontier labs* Kimi K3.1 Pro announcement blog* Kimi K3.1 Pro on Artificial Analysis Intelligence Index* Kimi K3.1 on Artificial Analysis defensive cyber benchmarks* Program Bench* Open-weight Mythos would’ve been fine take* PHP “hasn’t caused a disaster yet” update* Abliteration.ai serving abliterated models API* Morph serving abliterated DeepSeek via API* Joshua Saxe’s Substack on AI and cybersecurity* Azeem Azhar, “The internet’s Wuhan moment”* Anthropic, “How employment is changing at Anthropic”* LiteLLM security incident reportTranscript00:00:00 Nathan Lambert: We are back with another Interconnects podcast. We are with Florian, eval guy, recently moved to SF. Florian, is Kimi K3.1 Pro, name is a mouthful, going to live up to the benchmark scores, for example on Artificial Analysis, or where is it going to land?00:00:17 Florian Brand: In the past, the Kimi models always lived up to the expectations, but this release is kind of weird compared to the others. I think it is also partially because they are becoming more professional, and the previous model drops were kind of out of nowhere. This time, which also got some criticism, they launched a bigger social media campaign, which allegedly also involved paying people to post about the model without disclosing.00:00:56 Nathan Lambert: I think a lot of companies do this and tell you not to mention it, but it’s a mess. It’s bad form. No company should do it. Essentially, one person tweeted a screenshot of the press instructions for Kimi that said, “Don’t say this is a paid promotion.” And to my knowledge, no one has corroborated it after that.00:01:15 Florian Brand: Yeah, and there are right now a few of these model launches, including by American companies, that are just very obviously astroturfed. We have seen launches where it’s thousands of retweets and a few hundred likes, because it works so well to get the attention. Some of them disclose that it is a paid ad while others don’t. But in general, we have seen the professionalization of model launches over time, and pretty quickly. A year ago, everyone just dropped the model, then patched vLLM afterwards, so no one was actually able to run any of the open models for a while. Now you get all the inference frameworks on day one, you get all the inference providers on day one. Or, which is also what we see more and more, the model creator has a few weeks of exclusive access before they release the weights. And that’s also what’s happening with K3.1 Pro. I think it’s two weeks of exclusive access before they release the model weights.00:02:38 Nathan Lambert: They said the weights would come out by November 4th. And the headline things... I don’t know if I’m going to be able to share screen. I don’t have this set up on this computer. But the headline is this Artificial Analysis number where it’s ahead of GPT-6 Astra. And then if you look at the Kimi K3.1 blog, they have their headline numbers, which are really strong, and then they also have some numbers later on which seem a bit more jagged. I think it was Terminal-Bench Science, which is 20 points behind GPT-6 Astra. I think there were a few others, like ProgramBench. Do you know Terminal-Bench Science and ProgramBench? Are any of these things that people think are important benchmarks for frontier models? Or how do you explain the fact that on Artificial Analysis the model is top, what, three, four? It’s the two latest Claude 5.5 models, Muse Wonder Full, and then Kimi K3.1, which looks crazy. It just looks like ...
    続きを読む 一部表示
    47 分
  • Meta shakes up the frontiers, open and closed
    2026/10/17
    This has been a week where the balance of American open-weight models took a step change back toward the frontier. Meta took the lead of this charge with Muse Spark Open, which is now the strongest American-made open model by quite some margin, but the addition of Reflection’s first model Beam is also relevant. Without these two models, there would be 20+ open-weight models built and released by Chinese companies better than the best Western models. At the time of writing this, Mistral has also announced a strong model, Mistral Large 4, with an index score of 38 (see below), but its weights are not yet released. There are rumors of a few more strong American open-weight models coming from other organizations too!In many ways, the gap between the best American and Chinese models has been reduced roughly by half. We’ve gone from the best American open models being largely irrelevant to consumers who require the cutting edge of performance, to them being behind but relevant.This paints the picture of the cut-throat, competitive nature of the frontier model business. Reflection and Mistral announced their model scores and Meta dropped the weights of a smarter model before these other two labs could get their own weights out. All of these models are competing with Chinese models released months ago, where the industry is collectively asking — where have Kimi and Zhipu AI been since the releases of GLM-5.3 and Kimi K3 on Aug. 18th and Jul. 16th, respectively?That is a very nuanced question, which I’ll dig into another time. Today’s piece is focused on Meta’s turnaround, what Muse Spark Open means for the ecosystem, and the balance of power across Western frontier labs.Why Muse Spark Open should take off — and why it may notOne thing we see again and again at Interconnects through our open model analysis is that Western open-weight models get astonishingly more adoption than Chinese models with the same performance-to-size footprint (even though the Chinese models usually come out first)! Muse Spark Open is landing into this environment, at the same time as when open-ended LLM agents are finally real (~GLM 5.2 capabilities and on). Riding off a rapid increase in performance of Chinese models, the open ecosystem has been living through an acceleration in demand. The inference providers are rapidly scaling, the open post-training economy is forming, and countless enterprises are looking to own their intelligence. Muse Spark Open’s performance falls somewhere above GLM 5.2, near DeepSeek V4.1 Flash, and behind GLM 5.3 or Kimi K3. It should be squarely “good enough” to slot right into this.For some background, from the Muse Spark Open model card on Hugging Face:Muse Spark Open is an 816B total/42B active parameter open weights model with a context window of 1M tokens. Built for coding and agentic work, Muse Spark Open is natively multimodal. This open weights version has additional safety post-training to make it more robust against misuse as a standalone model without system guardrails. We recommend deploying Muse Spark Open as part of a broader AI system, with additional guardrails suited to the specific use case.Here’s the distribution of the various Muse models on the Artificial Analysis Intelligence Index.This is included to loop back to the original model card snippet. It was shared that this open-weight model underwent additional safety training to prepare for this open release. Many observers saw this release as potentially a model that was “broken” during the safety training, as it came out after Muse Spark 1.3. In reality, this was much more likely a safety tune of Muse Spark 1.2, which would only be a 1 point drop on Artificial Analysis to shore up safety behaviors. When Meta released Muse Glimmer, the 30B agentic model, Alex Wang said that they were preparing to release the Muse Spark 1.2 model as open weights, which fits with the above.If anything will hold Muse Spark Open back, it’ll be the specific license they designed for it. In summary, this custom license has pretty much all of the restrictive license behaviors we’ve seen (and criticized here) in the last few years of building with open models. They are as follows:* Users of the models in products must display “Built with Muse” on their products. Meta did this with Llama, and Kimi has tried it, but almost no one complies.* There is a revenue-gating factor, where companies that make more than $20M per month across their entire business deploying the model need a separate license with Meta. Other prominent open model providers are doing this too, in order to set up revenue share agreements for re-selling inference.* Muse Spark Open makes you agree to an acceptable use policy, which in the true “open source software” corner of the world makes this model extremely not open-source.* All of the above applies to outputs from the model as well, and derivatives, so there is legal ambiguity if you can safely use the ...
    続きを読む 一部表示
    12 分
  • The Cyber Risk Discourse is Broken
    2026/10/06
    I’ve had one too many discussions on the cyber risks of open models — where someone assumes that either banning open models happens in a silo (and bad actors will somehow actually be stopped) or that China is a threat and doesn’t care about AI safety — that I feel we’re going to end up making policy decisions that both limit American AI competitiveness and increases long-term cyber risk. I feel like we’re locked in on a path of lose-lose situation, unless people embrace nuance, trade-offs, and scary realities.Much has been written about the views of various players in this debate, which I summarize as follows:* Open weight models pose an untenable risk to society’s functionality, occupied by frontier lab leadership and the national security community in the U.S.* Western voices saying open weight models are necessary for defense and banning them will make the world less safe, occupied by AI risk moderates to different extremes. I include myself in this bucket, along with Hugging Face’s perspective after the OpenAI incident, and others like Joshua Saxe.* Chinese companies continuing to release open-weight models with strong cyber capabilities, who are making decisions based on the risk assessments of their society and government.This list is also organized by volume or clarity of message in the Western AI ecosystem. The risk-focused discussions have been disproportionately visible, including pieces of work which I view as very detrimental to potential of engaging across assumed party lines on this issue. There’s also very little substantive engagement trying to understand how Chinese companies assess risk — rather, mostly arguments that border on ad hominem attacks like “Chinese labs don’t care about safety.” Interconnects AI is a reader-supported publication. Consider becoming a subscriber.The delusions of the anti open-weight allianceThe most recent publication from the “open-weights are dangerous” crowd was the report by Anthropic on the risks of GLM-5.3 as an offensive cyber tool. The problem with this is not the technical research they did, which is largely reasonable, but the failure to engage on more cross cutting questions like: What happens if we ban open models due to cyber risks or why do Chinese companies deem these models safe to release?These questions, which I’ll come back to, are the ones we need to answer to understand the ecosystem perspective on current AI risks. All policy actions happen in a multi-piece puzzle, where the shape of risk can be changed, but it is rare that you get a universally better outcome. They are all trade-offs.This sort of solipsistic writing by Anthropic fits very closely to another type of discourse I’m hearing about regularly that is determining the fate of open models and AI broadly — classified briefings. I’ve had many discussions with people who say something like “Look, I’m pro open models, but if you’re seeing what I’m seeing, it’s an onslaught out there due to [XYZ Chinese open-weight model] and we need to act.” This perspective flies in the face of public information, where to date closed models have been documented as the cause of most existing cyber attacks. As someone who tries to make grounded predictions, the evidence of numerous attacks from OpenAI models is the only data I have on the shape of cyber risks (or, you can broaden the scope to include all of FelonyBench). There are a few possible explanations, and it’ll take years to get to the true answer: * Maybe open model weights and closed model APIs (with some safe guards) are both far closer to being easy to mis-use, rather than API models being closer to safe. The trope "Open Dangerous, Closed Safe" may be closer to "Open Unsafe, Closed Unsafe."* Maybe there are far fewer bad actors who are willing to use AI models for “loud” cyber attacks that target crucial infrastructure in the US or powerful and underprepared institutions. Closed models have stronger capabilities and stronger safeguards (in theory), but the stronger capabilities part could matter more in net harm if the safeguards on both are porous. * I have a lot more to learn about how the cybersecurity ecosystem works, so I won’t support this full-heartedly as one of my current recommendations, but I think it’s at least a reasonable argument that open weight models as a diffusion tool for cyber defense could be the best tool we have to prevent harm over the next few years. There are many domains, e.g. sensitive government agencies, where open-weight models are the only tools that can be deployed in the near-term on air-gapped networks.Lots of this situation seems like the messy reality being hard to pin down, where the logical limits are clearer. I may agree with the simple reality that closed models are far safer in the limit, as a technical tool with more layers of protection, but they may actually be causing more harm in the near term because of how accessible model APIs ...
    続きを読む 一部表示
    9 分
  • Debating RSI, the US-China Gap, and Jaggedness with JS Denain of Epoch AI
    2026/09/22
    This episode is with Jean-Stanislas “JS” Denain of Epoch AI, who leads their Insights Team and is one of the people I find myself debating the state and trajectory of AI with more and more. We’ve had follow-on discussions of many of my favorite recent posts online and/or in private, so I wanted to dig into the nuance in a public episode.A big takeaway of this podcast is how JS and I both have so much uncertainty with exactly where we are heading, and this was our best effort at stating our observations today. Chapters / topics include:* 00:00 Predictions for RSI* 18:15 The role of robotics in an AI acceleration* 24:20 How far behind are Chinese models?* 27:39 Does distillation explain the gap?* 40:58 What Chinese job postings reveal about their labs* 48:13 Are open or closed models safer?* 58:10 How Epoch AI ticks* 1:00:55 What a frontier post-training recipe looks likeEnjoy!More from JS: Epoch AI profile and writing, X, LinkedInListen on Apple Podcasts, Spotify, and where ever you get your podcasts. For other Interconnects interviews, go here.Transcript00:00:00 Nathan Lambert: I’m here with JS Denain, who is a senior researcher at Epoch AI. He leads the insights team. He is one of the people who I feel like I get the best feedback on my writing from, whether it’s from US-China AI capabilities, now RSI. And I just wanted to open this discussion and honestly go deeper with him, trying to understand how he thinks about these various things. And I think you have a very useful, moderate point of view, which I feel like you’re probably a step further into what would be called faster scenarios for AI progress. But let’s get into this, and it’s like, what measurements do you think OpenAI and Anthropic are seeing when we get all these proclamations on RSI happening very imminently?00:00:47 JS Denain: Yeah. I think, so there’s the measurements they’ve published, right? So, OpenAI and Anthropic both had blog posts, I mean, Anthropic two at least, on the effect AI has on accelerating AI progress. I think at least the things they publish, I don’t think are super strong evidence of imminent self-sustaining acceleration AI capabilities, or full automation of the job of AI researcher. But I think the kinds of things we see are, I think probably the most striking thing I saw in the OpenAI blog post was increasing usage of AI systems in model deployment, like the increase in spending on Codex that we saw. And it’s kind of unclear how exactly to interpret this, because maybe it’s a measurement artifact where they’re only looking at Codex, but in fact, there was a bunch of ChatGPT usage before from the researchers. But overall, that plot, for example, just shows a 2X a month increase in Codex spending by researchers, and that does seem to me to be some evidence of they’re getting a lot of value out of this probably. I don’t think this is strong evidence that in six months we have a software intelligence explosion.00:01:57 Nathan Lambert: Do you think this is the same? So, what is the information they have internally relative to what we have? And this is obviously hypothetical. We don’t have this internal information. Because I get the sense that a lot of people are more scared in their updates from the labs than the information we have. And I try to take this very seriously of, what will they be seeing that is making the acceleration of risk comments go faster, and how much of this is material evidence versus how cultures evolve over time? And I’m much more interested in evidence.00:02:29 JS Denain: Yeah. So two things. I think, first of all, I guess I don’t think, I don’t know, right? I don’t have full information here. I don’t currently think that either there’s some specific thing that people at OpenAI or Anthropic are seeing right now that we don’t have access to that warrants being way more freaked out about this. I also don’t think that... I think the public evidence we have right now, more general on AI progress and just a priori case for this being an important dynamic, I think is enough. I think to care about this particular dynamic of AI accelerating AI progress, that being a big deal and worth tracking. And then it’s kind of unclear what the urgency is of when the feedback loop really kicks in. So, basically on the what is there on the inside that people have access to, I could give examples of kinds of metrics, right, they could be looking at. It’s plausible that we have access to the capabilities of AI systems, but internal teams have their KPIs, and maybe they’re seeing compute multipliers in the pre-training team or other kinds of metrics that people are tracking going crazy. And then the combination of this plus some intuitions of how the different outputs of different teams combine yields a prediction on the trend in actual performance of the end AI systems. So that could be an early warning sign. It’s unclear to me that the recent discourse we’...
    続きを読む 一部表示
    1 時間 5 分
  • The current balance of power in open models
    2026/09/21
    I was recently invited to brief a group of Congressional members and staff on the state of open-weight models in the lens of U.S.-China competition. I’m sharing my prepared remarks as a state of the union on open models that is accessible to a broader audience.Interconnects AI is a reader-supported publication. Consider becoming a subscriber.Recap: What is an open source v. open-weight vs. closed model?Open language models are AI models where their weights are publicly available for inspection or downstream use. These are most often contrasted to so-called “closed” AI models. Closed models offer access only through Application Programming Interfaces (APIs) that developers can use to directly query a model, like GPT-4 or Claude Opus 4.5, or through products, like ChatGPT and Claude Code.Open language models primarily are bucketed into two categories, open-weight and open-source models. Open-weight models are the most common form, such as popular models like Meta’s Llama, Alibaba’s Qwen, Google’s Gemma, or DeepSeek’s models. These models are governed by licenses, governing documents dictating what is allowed with downstream use, and are often accompanied by inference code in libraries such as Transformers, VLLM, SGLANG, etc. Since about April 2025, Chinese AI companies have been the clear leader in open-weight models.True “open-source” models are similar to these, as they include the weights, licenses, and inference code, but they also include the complete information needed to reproduce the model – the training code and training data. The most prominent open-source models have been built in the United States, led recently by the Allen Institute for AI’s Olmo models that I helped build in my recent 2.5 years there. The other prominent open-source models are also built by American non-profit organizations, including OpenAthena’s Marin models and EleutherAI’s Pythia models.Open-weight, open-source, and every other label for a model – including closed models primarily offered via an API – exist on a spectrum. For example, Nvidia’s Nemotron models are far more open than most open-weight models, releasing large quantities of their training data under permissive licenses, but they’re not fully open-source because they do not release all of the data. Closed models also exist on a spectrum based on what information the API reveals and the terms of use.The state of competition between American and Chinese open-weight models (unit economics, technical capabilities, etc.)We are living in a world where GLM-5.2 and Kimi K3, some of the latest, leading Chinese models, have enacted a step change in the commercial viability of open models — crossing a similar threshold in agentic capabilities that Anthropic’s Claude Code crossed in December of 2025.America was the early leader in open language models, primarily through Meta’s Llama models, which were used extensively across research and commercial tasks. Chinese open-weight models surpassed American open-weight models in these two key areas about 18 months ago. The simple metric showing this is Hugging Face Downloads, where China took the lead in July of 2025 primarily through the success of Alibaba’s Qwen models. I personally maintain tools to track this data, and since I first published the American Truly Open Models (ATOM) Project in August of 2025, China’s download lead has grown to about 1.6B – with a total of 3.2B downloads, twice that of America’s total.On popular capabilities benchmarks, such as the Artificial Analysis Intelligence Index (AAII), the Chinese open-weight models have a clear lead over American counterparts. The top three Chinese models as of writing this on September 14, 2026 are Z.ai’s GLM-5.3 and GLM-5.3-Flash and Moonshot AI’s Kimi K3 with scores of 45, 42, and 44 respectively. By comparison, the leading American models are Thinking Machines’ Inkling and Inkling Small, both with a score of 26, and Nvidia’s Nemotron 3 Ultra, with a score of 23. The top American models were released in June and July of 2026, and are updated less frequently than their Chinese counterparts. For example, Chinese labs released models with scores above these American models 2-6 months before the American companies got there (e.g. GLM-5 or DeepSeek V4 Pro). There is a trend of more American companies releasing models, including names like Arcee AI, Poolside and IBM, but they are not rapidly closing this performance gap. Other benchmarks tell a similar story.Together, Chinese open-weight models are approximately 2-5 months behind the closed American frontier, with the open-weight American models being approximately 6-9 months behind the likes of OpenAI and Anthropic. The Chinese labs are closest in tasks with clear user demand, such as agentic coding, and further behind on more open-ended scientific tasks, such as physics or biology.The reasons why Chinese labs can produce these strong models, despite having fewer ...
    続きを読む 一部表示
    18 分
  • Why I still haven’t bought into true RSI
    2026/09/19
    We’re in an era where a few organizations are using thousands of concurrent agents to improve their processes and output. These organizations happen to be just the frontier AI labs, in particular OpenAI and Anthropic. In the last few weeks, I’ve been pondering what it means for so many employees across these organizations to rapidly update their expectations for the pace of AI progress and associated risks.A core perspective I have is that the frontier labs and broader frenetic, competitive culture in the San Francisco AI scene set up an environment that amplifies any AI concern. This has some benefits in causing more general audience awareness of AI, as fear sells, but exaggerating risk timelines or severity will have negative second-order effects. I remember many loud AI safety debates, and their associated clouds over the viability of open-source AI, in 2023 and 2024 — the primary risks then did not arrive in the forecasted timelines.The general populace of these two key labs was very anxious about AI risks and the rate of progress even a year ago, and especially as agents got stronger product-market fit at the start of 2026. This cultural precondition, when exposed to the reality that thousands of agents will constantly be working fairly productively in your business, will only increase this anxiety. The step from this anxiety, and incidents like OpenAI-HuggingFace, to extinction risks feels very religious.Richard Ngo had an apt summary of the situation:Now a large proportion of the AI safety community is implicitly or explicitly orienting to futures where an intelligence explosion occurs within a few years. My default expectation (absent an extensive pause) is that a similar thing will happen: they’ll turn out to be directionally correct (relative to the expectations of almost anyone not linked to the community) but factually wrong. Specifically, we won’t have superintelligence within the next 8 years, but things will still be moving so fast that it’ll *feel* like the people who argued for short timelines were right.… I wanted to say something now because it feels like the level of bandwagoning towards “singularity soon” is getting pretty wild.Personally, I think this view aligns closely to what I outlined in my alternate scenario to true recursive self-improvement (RSI), which I called lossy self-improvement. A summary of this view is that:* Automatable research is too narrow to achieve a massive net acceleration in progress, in the face of scaling laws’ exponential costs,* Diminishing returns of more AI agents in parallel are real, &* Resource bottlenecks and politics are a major factor in building strong LLMs (and AI can do much less to accelerate this).So, I’m left balancing the above, latent increase in the cultural temperature with the potential that the labs have seen genuinely scary, specific breakthroughs that are not public yet. My expectation is that more of the current AI safety concern is on the former – scaled agents working – but I hold high levels of uncertainty here. Foundational, imagination-based AI breakthroughs are the sort of thing that would make me update my RSI timelines from closer to a tool to sustain progress in the face of exponential costs (scaling laws), to something more unpredictable and/or unstable.Interconnects AI is a reader-supported publication. Consider becoming a subscriber.Some of the best recent resources on RSI have been Dwarkesh’s podcasts with Noam Brown and the trio of John Schulman, Beren Millidge and Charlie O’Neill. I have a few important reflections from both of them.First, the podcast with Noam Brown made me internalize how big of a short-term acceleration mass inference capacity is. These labs will throw thousands of agents at important, measurable problems. At the same time, compute capacity available to them is going to continue to scale. I have my doubts that the labs can afford to spend a constant portion of this compute on internal R&D as the total volume goes up, especially with plans to IPO, as they face increased scrutiny on basic economics. It is important to not confuse massive steps in inference-time scaling, a dynamic which should be fairly predictable, with being the outputs of RSI, which is highly uncertain.Second, the trio podcast debating the state of the art in technical capacities induced more of a surprising reaction that I haven’t fully settled. Through the first hour or so of this podcast, where they debate the role of RL, distillation, scaling, inference-time compute, etc., I found myself strongly agreeing with the distribution of claims. A TLDR would be that our current techniques work and let us solve problems we know how to state, but they don’t result in a magical level of generalization to unknown, harder problems in most partially verifiable domains (i.e. progress in math is an exception, rather than a rule).The surprise of this podcast was the end, where they were predicting timelines...
    続きを読む 一部表示
    10 分
  • One resignation turned the embers of AI fear into a wildfire
    2026/09/10
    As AI became more powerful, it was inevitable that a different, growing group would start to take AI safety more seriously – what we did not know ahead of time, is which set of views they latched onto. We have seen that some of the most extreme views of risk, i.e. moderate probabilities of mass extinction, were the ones that reached the masses. A lot in the AI world is about to change due to this.How did we get here? Why did this quitting announcement reach so far? In many ways, the rest of the world’s views around AI in the past was a dampening factor. You can think about this like the damp ground around a fire. Many people were striking matches for years about AI risk – they’d smolder in their community and largely burn out, going unnoticed. As the stakes of AI have risen this year, from the OpenAI-HuggingFace incident and breakthroughs like the Navier-Stokes result (also from OpenAI), the ground has dried out and the latent energy around the AI discourse has increased. More people not in the industry have thought, “huh, maybe I should care about this AI thing.” The ambient temperature and stakes have been obviously rising.Then, some basic factors of human nature apply, with the most crucial being that fear sells. Fear is the simplest story, the one people cannot look away from. Jacob Coxon was the one who stumbled into this new powder keg, totally unaware of what was going to come. What looked like a fairly innocuous event – another AI researcher quitting citing safety risks – landed into a very different environment and it caught like wildfire. The discussion of existential risk, mass extinction, and the trajectory of AI has traveled further than even the most seasoned AI commentariat would ever predict.There are a set of facts we need to get clear, which paint the picture of the situation. The key Tweets to reference are from Jacob Coxon, the resignation thread, and Evan Hubinger, the source of the >10% extinction risk figure .* There are plenty of AI risks which are likely to cause harm, even if estimating annihilation is useless. It is important to weigh these with respect to the benefits. The entire discourse around existential risk is on very poor footing. At least Evan was clear in his post, with “kill all humans,” but a major problem in the AI Safety discourse is that people talk about existential risks, when they mean very different things (much like how AGI is a vaguely meaningless term). I put the probability of complete extinction as being so low it isn’t worth discussing, but the probabilities of AI caused disasters – e.g. cyber attacks on critical infrastructure or bio-risks – as being worth debating. Throwing this whole discussion out because there are not these disasters yet is a harmful reaction.* Jacob Coxon is acting genuinely and with good intentions. The outpouring of support from more well-established AI researchers who know of him and his intentions of resignation is useful. Many factions of AI turned to scapegoating him individually, based on account metadata, personal factors, etc. These are not useful. Many frontier lab employees genuinely have similar views to him. I’m not sure it’s a majority, but there is a substantial group.* Many frontier lab employees, especially at Anthropic, are out of touch and this will impact their forecasting and/or descriptions of current AI events. I say this without blaming individuals, but it’s a common agreement among my friends not at OpenAI/Anthropic (Ant especially) that people at the labs operate with a religious energy. It’s very common to go through very out of touch interactions with them. I do not blame most of the individuals who get distorted views being part of these companies, but the interactions are wild and spill over into a lot of wack discussions in the AI media ecosystem. Living in this environment that normalizes such out of touch behavior will inevitably distort any human’s understanding of technical progress.Interconnects AI is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.* This was not a mass political campaign, but rather an opportunistic media coordination. For context, the Wall Street Journal had an exclusive story that Jacob coordinated before posting. I suspect that Jacob shared his plan of quitting in groupchats with AI safety advocacy groups ahead of time, e.g. the morning of posting, asking for amplification. This is normal practice, and could have included some prominent politicians. From there, I think it’s more likely that other politicians are bandwagoning on a rising issue. When you combine this with other factors, like Daniel Kokotajlo’s appearance on Joe Rogan coming out the same day – it definitely looks like a very well-executed, coordinated media campaign. This doesn’t mean it’s a conspiracy or a regulatory capture tactic within Democratic political structures. The determining factor ...
    続きを読む 一部表示
    11 分
  • When will average people feel AI’s impact?
    2026/09/09
    Housekeeping: Paid subscribers to Interconnects now get a permanent 40% discount on my book when purchasing at Manning.com. Access the code at the Interconnects perks page.Many AI optimists tend to compare what is happening in this AI boom to the industrial revolution, or to other periods of rapid technological advancement and diffusion into society. These comparisons fit on the scale of technological change, but miss a crucial factor in how most people are exposed to that change. The problem facing AI is that most people have no super tangible new goods thanks to it and society has more inertia resisting change than in previous eras.I’m writing this coming back online from a few weeks off for my wedding in New England. In this time it would have been very easy to not think about AI at all. The touch-points that average people have to AI products today are fringe, marginally beneficial, or even just very confusing to them (e.g. many people have heard about and brought up the OpenAI-HuggingFace incident, but don’t know what to make of it). People on the positive side think of AI as a way to make fun images, enhanced Google Search, etc. These are very small benefits. On the negative side is an association with addictive social media algorithms, friends of friends addicted to AI chatbots, and a plethora of takes on data centers.AI is still a rounding error in everyday lifeCore aspects of everyday life — family, food, transportation, and entertainment — have few direct impacts yet. It’s a remarkable breath of fresh air to pop out of the bubble and realize how little what is happening really matters today. Being obsessed with AI is a choice that a very few people have yet opted into. For example, the only thing I used AI for in this time was search and creative work (making the pretty seating chart for my wedding guests to find their table). In industrial revolutions past, average people got absolutely life changing outcomes. The First Industrial Revolution in the late 18th century gave access to cheaper clothing, cooking ware, reading material, and a shift to new livelihoods. The Second Industrial Revolution in the late 19th century introduced household machines (e.g. sewing machines), preserved food, indoor plumbing, photography, better light sources, bicycles, and further benefits of manufactured goods and electrification. The list is remarkable — most of these we still use regularly today — and very physical.While even the most optimistic versions of AI will usher in new scientific discoveries, advanced therapeutics for rare diseases, and potentially even sustained economic abundance, these benefits have the risk of being too indirect. How will a common citizen come to credit OpenAI or Anthropic for saving their life, if they went to their family doctor who told them about a new miracle cure?What percentage of Americans will care about OpenAI solving the Navier-Stokes Millennium Prize Problem? It feels very likely in 50 years that the average American’s day to day life looks very similar. Their home, appliances, relationships, and vehicles will be similar (of course, self-driving will continue to diffuse, but that has been developing on a very independent trajectory from the innovations of LLMs). In this time, AI will get a lot of credit. 50 years is a remarkable length of time with how fast everything is changing today in this, AI-focused narrow slice of the world.The most important part of what is happening early in the AI revolution, is building foundational infrastructure, and a general process, which will compound over decades. A major mathematical breakthrough today will look astonishingly minor in scope relative to the advancements that come later in the compounding journey. It is hard to predict what it looks like for every technology you use daily to get faster compounding improvements due to AI. Interconnects AI is a reader-supported publication. Consider becoming a subscriber.Breaking social stasisMuch of the narrative around AI is trying to push people to care, due to this long-term reality of progress, at least as a subconscious motive. This will take a long, long time to get right, and AI’s buildout faces immediate political problems due to this imbalance. Today’s AI is primarily a tool to serve the elite. For knowledge work, which is roughly half of the U.S. economy, AI is as fundamental as electricity (or quickly will be so, with rapid improvements to agents in the next 18 months). It’s highly destabilizing to have such a transformative, productive tool only bring half of society along. It’s not hard for many people to pick up on this — the technology economy booms while life stays otherwise stagnant.In writing this, I learned of Engels’ pause, which is “the period from 1790 to 1840, when British working-class wages stagnated and per-capita gross domestic product expanded rapidly during a technological upheaval.” If we — the leaders of the AI industry ...
    続きを読む 一部表示
    9 分