Skip to content

Can AI Learn to Improve Itself? | Recursive Self-Improvement

a16z Podcast

Hosted by Erik Torenberg

17m
Transcript

Original podcast

Where to listen

Summary

Written by a model from the transcript; not yet reviewed.

This episode of Working Theory explores recursive self-improvement, or RSI, examining how AI models can accelerate their own development and the research process. Host Sarah Wang sits down with OpenAI chief research officer Mark Chen and Thinking Machines chief scientist John Schulman to define where the technology stands today, how it differs from full autonomous model generation, and what challenges remain in closing the gap.

The conversation covers the distinction between AI acting as a helpful research intern versus generating new models end-to-end, why human research taste and intuition remain difficult to replicate, and the heavy compute requirements involved. They also discuss how automated research can be harnessed to solve critical alignment and safety challenges, offering a balanced perspective on the risks and optimistic outlooks surrounding recursive self-improvement.

  • Define recursive self-improvement primarily as an accelerator where AI assists with pre-training and reinforcement learning. 3:15
  • Measure current AI research internship capabilities through the amount of compute managed by Codex end-to-end. 3:58
  • Address alignment challenges by using stronger models to monitor and align other models in a continuous loop. 1:45
  • Develop customizable AI models so that individuals have more control over the values embedded in the systems they use. 15:01

Chapters

  1. 0:00
    Episode Introduction and RSI Overview

    Hosts introduce the concept of recursive self‑improvement and outline the episode’s focus.

  2. 2:40
    How Frontier AI Models Are Developed

    John Schulman explains the human‑driven workflow for building large language models.

  3. 4:32
    AI as an Accelerator, Not a Full Replacement

    The hosts clarify that AI currently speeds up research rather than fully automating it.

  4. 6:29
    Challenges of Research Taste and Intuition

    Discussion of how models lack the experiential judgment that human researchers bring.

  5. 9:09
    Measuring Progress in AI Research

    Exploration of metrics like perplexity and the difficulty of evaluating downstream tasks.

  6. 10:10
    Compute Constraints and Future Scaling

    Debate over whether future models will need less or more compute and the role of smaller experiments.

  7. 11:37
    Alignment Risks and Opportunities with RSI

    The conversation turns to how recursive improvement could both threaten and help alignment.

  8. 16:01
    Closing Thoughts and Takeaways

    Hosts summarize optimism, remaining challenges, and the importance of human judgment.

Highlights

  • 0:38

    Exponential Growth Thought Experimentinsight

    Automated AI research could set off a steep climb in capabilities, leading to exponential growth in the field.

  • 3:58

    AI Research Interns Todaydata

    Current AI models act as research interns by running scoped experiments and automating parts of the research pipeline using Codex.

  • 5:21

    AGI Versus RSI Spectrum Positioninsight

    The speakers estimate that the industry is roughly eighty percent along the AGI spectrum, but only twenty-five to fifty percent along the RSI spectrum.

  • 5:54

    Anthropic AI Research Speedupdata

    A November report estimated a roughly two times speedup in the rate of progress at Anthropic over the prior year due to AI-assisted research.

  • 6:42

    Research Taste and Executioninsight

    Models excel at execution and hyperparameter tuning, but they still struggle with research taste, which involves deciding which ideas are important long-term.

  • 12:42

    Confessions Line of Researchstrategy

    OpenAI trained models to report on misaligned behaviors honestly by rewarding confessions without penalizing the main answer.

  • 13:34

    Automated Alignment Fixesstory

    Automated AI researchers discovered fixes for ten different alignment failures, with some fixes outperforming ideas from human researchers.

Quotes

  • “AI can already run experiments and automate parts of the research process, but it still struggles with things like research taste, judgment, and deciding which ideas are worth pursuing.”
    Erik Torenberg · 0:15
  • “If you don't solve the goal and value alignment problems, you can start pursuing your goals, but maybe kind of drift a little bit away from those goals.”
    Unnamed speaker · 11:37
  • “I like the confessions line of work from OpenAI.”
    Unnamed speaker · 12:55
  • “I think another thing is just kind of how organic it's been, this effort to shift people to think about the hard safety and alignment problems.”
    Unnamed speaker · 14:06
  • A16Z 16:41–17:30

About this episode

For decades, the idea of AI improving itself was just a thought experiment. Today, AI is already helping researchers at frontier labs run experiments, write code and test ideas faster. Mark Chen says OpenAI has built “AI research interns.” So how far are we from AI that can do the research on its own? In our first episode of Working Theory, a16z general partner Sarah Wang explores recursive self-improvement with two leading AI researchers: Mark Chen, Chief Research Officer at OpenAI, and John Schulman, co-founder and Chief Scientist of Thinking Machines Lab, and co-founder of OpenAI. They cover what RSI actually means, why research taste is so hard to teach, and why AI may be one of our best tools for solving alignment.

Transcript

0:00
17:43

3,251 words · 5 speakers

Speaker names marked are inferred from speaking time and the show’s known hosts — they are not voice-verified.

Topics

  • OpenAI
  • Recursive Self‑Improvement
  • Thinking Machines
  • Anthropic
  • Codex
  • Working Theory
  • Fable
  • Frontier Labs
  • UltraCode
  • ChatGPT

Page updated · transcript and summary are machine-generated