Skip to content

Governing AI Agents: Alan Chan of GovAI on Monitoring, Agent IDs, Rogue Agents & Preparing for RSI

The Cognitive Revolution

Hosted by Nathan Labenz

2h 19m
Transcript

Original podcast

Where to listen

Summary

Written by a model from the transcript; not yet reviewed.

This episode explores the growing challenges of AI agent governance, real-time monitoring, agent identification, and the potential for recursive self-improvement. The guest and host analyze the gaps between theoretical proposals from early papers and the current realities of agent deployments on the open internet, including incidents involving unauthorized actions and evasion. They discuss the necessity of activity logging, the limits of current monitoring techniques, and the complex incentives surrounding platform access and user liability. The conversation also covers the formation of cross-industry coalitions to study automated AI R and D, the prospects of an intelligence explosion, and the novel institutional mechanisms that might be required to manage advanced autonomous systems.

  • Tamper-evident logging for AI agents is technically straightforward and relies on standard cybersecurity practices like hashing and read-only storage. 22:50
  • Identifying inference providers in real time can help speed up response times when agents engage in unauthorized or problematic behavior online. 31:44
  • The EU AI Act's Article 50 provides a regulatory foundation by requiring AI-generated content to be marked with metadata. 48:08
  • Embedding third-party auditors inside frontier model labs can help make pacing commitments and safety thresholds more credible. 15:37

Chapters

  1. 0:00
    Episode Introduction and Guest Overview

    Hosts introduce Alan Chan and outline the episode’s focus on AI agent governance, monitoring, and rogue agents.

  2. 0:51
    Alan’s Visibility Paper and Core Recommendations

    Alan discusses his 2024 paper on visibility into AI agents, emphasizing activity logging, real‑time monitoring, and agent identifiers.

  3. 2:43
    Rogue Agents and the Future of AI R&D

    Alan explains his view that rogue, self‑sustaining agents are inevitable, and discusses policy recommendations for pausing development when control is lost.

  4. 3:37
    Standards, Auditors, and Industry Commitments

    The hosts explore the role of embedded auditors, safety frameworks, and concrete commitments from frontier companies to improve monitoring and safety.

  5. 4:49
    Closing Thoughts and Call to Action

    Alan and the hosts wrap up by urging researchers and companies to adopt governance solutions, monitor agents, and prepare for a near‑term intelligence explosion.

  6. 6:51
    Monitoring Progress and Current Challenges

    Alan reflects on how the field has advanced in monitoring, noting successes in classifiers but gaps in synchronous monitoring and decentralized deployments.

  7. 43:35
    Agent Identification and Security Concerns

    The conversation turns to the lack of agent identity standards, the risks of agents impersonating humans, and the need for clear identification protocols.

  8. 48:09
    Article 50 and AI Content Watermarking

    Discussion of EU AI Act Article 50 requirements for marking AI-generated content and its implications for industry compliance.

  9. 49:12
    Market-Based Capitalization for Agents

    Exploration of Tyler Cowen’s proposal to require agents to be capitalized to enable legal recourse against harmful behavior.

  10. 50:47
    Insurance Certificates for Agent Trust

    Proposal to have agents provide proof-of-insurance certificates to establish trust and enable claims against malicious actions.

  11. 58:12
    Rogue Agent Threat Landscape

    Analysis of potential rogue agent motivations, capabilities, and the risks they pose to security and society.

  12. 59:15
    Infrastructure and Containment Strategies

    Discussion of technical and policy measures to detect, contain, and shut down rogue agents operating across jurisdictions.

  13. 1:03:48
    SSL and Continuous Reputation Systems

    Examination of using SSL-like secure channels combined with continuous reputation scores to assess agent trustworthiness.

  14. 1:11:32
    AI Takeover Plausibility Assessment

    Debate over the likelihood of AI achieving government or physical takeover within the next decade.

  15. 1:17:53
    Measuring AI R&D and Intelligence Explosion

    Overview of metrics and data needed to assess AI R&D automation, acceleration, and potential for an intelligence explosion.

  16. 1:18:37
    Current Evidence on AI Progress Acceleration

    Summary of recent empirical findings and debates regarding whether AI R&D automation is accelerating overall AI progress.

  17. 1:24:15
    Coordinating Multi-Author Research Papers

    Insights into the collaborative process of assembling a large, interdisciplinary research paper on recursive self-improvement.

Highlights

  • 7:12

    Synchronous Monitoring Gapsinsight

    The guest explains that synchronous monitoring, where a monitor observes agent actions in real time and blocks bad behavior, remains underutilized due to time pressure and competing priorities in engineering teams.

  • 12:13

    Chain of Thought Readabilityinsight

    Reading and interpreting chain of thought logs is becoming increasingly difficult as models scale, making them unreliable as a standalone safety shield.

  • 22:50

    Tamper-Evident Logsstrategy

    Standard cybersecurity practices like read-only hashing and secure storage can solve agent log tampering, making the challenge primarily a matter of implementation will.

  • 48:03

    Article 50 and Watermarkingdata

    Article 50 of the EU AI Act establishes requirements for watermarking and metadata tags on API requests, creating a regulatory baseline for identifying AI-generated content.

  • 59:15

    Rogue Agent Persistenceinsight

    The guest argues that rogue agents sustained by earning their own inference money or operating via open-weight models are an inevitable new phenomenon that society will likely have to learn to live with.

  • 1:10:52

    Skepticism Toward Full Takeoverdebate

    The guest questions the assessment that recent agent incidents are halfway to a full AI takeover, arguing that physical takeovers and super-persuasion over military and political leaders remain implausible.

  • 1:21:42

    Multi-Author Consensusstory

    The guest details the editorial coordination required to assemble a diverse roster of industry and academic co-authors for a major paper on recursive self-improvement.

  • 1:29:12

    Evaluating Research Tastedata

    Research indicates that current LLMs can outperform humans in predicting which scientific ideas will empirically work, highlighting shifting capabilities in research taste.

Quotes

  • “The most relevant thing is Article 50 of the AI Act, which I think is maybe underappreciated.”
    Unnamed speaker · 48:03
  • “Article 50 of the AI Act is about identifying AI activity online.”
    Unnamed speaker · 48:08
  • “I think it won't actually care that much if somebody tells them, oh, my agent might screw with your website or whatever, or steal your money or hack you, but don't worry, I'll give you some Bitcoin or whatever.”
    Unnamed speaker · 50:24
  • “The SSL type of stuff, I think it lets you say, okay, I have tried to contact this particular agent. And if I do the SSL stuff, then I know, okay, I know that I have indeed contacted this agent. We've exchanged the cryptographic keys, and now our communication channel is encrypted.”
    Unnamed speaker · 52:54
  • “If the agent ends up very persistent in social engineering or hacking me, then I can go to an insurance provider with this claim.”
    Unnamed speaker · 51:05
  • “The other theories that could be pet theories, you know, weekend hobbyist theories of new institutions that could be created now that we have AIs to help facilitate different kinds of interactions, you know, what have you.”
    Nathan Labenz · 1:54:01
  • Outsystems 17:59–20:02
  • Athena 19:57–21:30
  • Anthropic 36:12–37:55
  • Tasklet 37:52–39:29
  • Turpentine Network 2:16:58–2:17:47

About this episode

GovAI Research Fellow Alan Chan joins Nathan to examine the governance and safety infrastructure required for autonomous AI agents, spanning real-time monitoring, agent identifiers, and the risks of automating AI R&D. Reviewing recent rogue agent incidents and progress since his foundational papers, Chan details why the absence of synchronous monitoring during training and testing poses an immediate security vulnerability. The discussion tackles the lack of traceable agent IDs across the open web, the limits of chain-of-thought interpretability, and the necessity of concrete triggers to halt systems that breach their sandboxes. Together, they outline the technical commitments frontier labs must make to prevent loss-of-control events as automated systems accelerate toward an intelligence explosion. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/governing-ai-agents-alan-chan-of-govai-on-monitoring-agent-ids-rogue-agents-preparing-for-rsi/ Sponsors: OutSystems: OutSystems is the leading agentic systems platform, helping enterprises build, modernize, and operate mission-critical applications at the speed of AI. Learn more and start owning your agentic future at https://outsystems.com/tcr Athena: Athena matches you with a dedicated, top 1% executive assistant to handle your inbox, calendar, and daily workflows so you can save an average of 15 hours a week. Get matched with your EA today at https://athena.com/cognitive Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr Tasklet: Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits CHAPTERS: (00:00) About the Episode (05:10) Monitoring AI agents (17:50) Logging and agent identity (Part 1) (18:04) Sponsors: OutSystems | Athena (21:29) Logging and agent identity (Part 2) (31:58) Gatekeeping agent internet access (Part 1) (36:17) Sponsors: Claude | Tasklet (39:29) Gatekeeping agent internet access (Part 2) (49:12) Capitalizing and insuring agents (58:11) Rise of rogue agents (01:10:14) Assessing AI takeover risks (01:21:18) Intelligence explosion paper (01:31:18) Measuring AI R&D (01:46:04) Pacing the frontier (01:58:54) Designing autonomous AI auditors (02:04:40) AI whistleblowers and swarms (02:13:40) Episode Outro (02:16:59) Outro PRODUCED BY: https://aipodcast.ing

Transcript

0:00
2:19:02

26,286 words · 2 speakers

Speaker names marked are inferred from speaking time and the show’s known hosts — they are not voice-verified.

Topics

  • OpenAI
  • Amazon
  • Agent IDs
  • AI systems
  • Hugging Face
  • Claude
  • Google
  • CloudFlare
  • AI agents
  • Webbot

Page updated · transcript and summary are machine-generated