Bioinfohazards: Jassi Pannu on Controlling Dangerous Data from which AI Models Learn

Jassi Pannu, Assistant Professor at Johns Hopkins, explains how rapidly advancing AI is transforming biological research and raising the risk of engineered pandemics. They map today’s biosecurity landscape, from pathogen detection and DNA sequencing to vaccine development, and examine how frontier models can already troubleshoot lab work and bypass data safeguards. The conversation introduces a proposed Biosecurity Data Level framework to restrict only the most dangerous functional biological data while preserving open science. They close with a broader defense-in-depth strategy—Delay, Deter, Detect, Defend—including DNA synthesis screening, global pathogen surveillance, and practical tools like Far UV sterilization. LINKS: Jassi Pannu: https://x.com/JassiPannuMD Science article that prompted this conversation: https://www.science.org/doi/10.1126/science.aeb2689 Sponsors: VCX: VCX, by Fundrise, is the public ticker for private tech, giving everyday investors access to high-growth private companies in AI, space, defense tech, and more. Learn how to invest at https://getvcx.com Framer: Framer is an enterprise-grade website builder that lets business teams design, launch, and optimize their.com with AI-powered wireframing, real-time collaboration, and built-in analytics. Start building for free and get 30% off a Framer Pro annual plan at https://framer.com/cognitive Claude: Claude is the AI collaborator that understands your entire workflow, from drafting and research to coding and complex problem-solving. Start tackling bigger problems with Claude and unlock Claude Pro’s full capabilities at https://claude.ai/tcr Tasklet: Tasklet is an AI agent that automates your work 24/7; just describe what you want in plain English and it gets the job done. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai CHAPTERS: (00:00) About the Episode (05:59) From outbreak to vaccine (17:08) Threat actors and data (Part 1) (21:23) Sponsors: VCX | Framer (23:53) Threat actors and data (Part 2) (31:05) Gain-of-function research risks (Part 1) (37:39) Sponsors: Claude | Tasklet (41:03) Gain-of-function research risks (Part 2) (48:05) AI models in biology (01:00:51) Dangerous AI capabilities (01:07:59) Biosecurity data level framework (01:18:58) Policy, governance, and infrastructure (01:28:53) Defense in depth vision (01:40:43) Episode Outro (01:45:02) Outro PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, my guest is Josie Pannu, Assistant Professor at Johns Hopkins, who recently co-authored an important paper calling for the creation of access control systems meant to prevent the dissemination and misuse of functional biological data from which AI models could learn extremely dangerous capabilities, such as the modification or even de novo design of highly contagious and deadly viruses. We begin with an overview of the biosecurity landscape today, including how new viruses are detected, how patient data is aggregated and analyzed in the context of a new threat, and what the pipeline from DNA sequence to vaccine candidate looks like today.

• 0:44Nathan Labenz?

  The good news is that we are able to design new vaccines amazingly quickly, at least for viruses that are similar to others we've seen. But there is unfortunately a lot of bad news as well. In 2012, for example, two research groups independently published results showing that wild-type bird flu, which already had an estimated 60% fatality rate but couldn't spread between humans, could become mammal-to-mammal transmissible with just five mutations. Such gain-of-function research has been broadly defunded since the COVID pandemic, but it does remain legal, and visibility into the experiments that private labs are conducting is low. Governments, Josie says, aren't likely to develop bioweapons capable of causing pandemics

• 1:26Nathan Labenz?

  for the simple reason that, short of vaccinating their populations in advance of an attack, they can't realistically expect to control them. But with AI capabilities crossing critical thresholds month by month, the threat from extremist groups and even lone actors is quickly moving from a theoretical to a deadly practical concern. Consider that Geoffrey Irving, chief scientist at the UK AI Security Institute, recently highlighted for me that today's frontier models can troubleshoot laboratory experiments from a cellphone picture better on average than PhDs. And in just the 10 days or so since we recorded this conversation, we've seen Andre Karpathy's auto-research framework demonstrate that AI agents can run and make research progress for days on end.

• 2:10Nathan Labenz?

  Even more to the point, Anthropic just reported that Opus 4.6, when faced with a benchmark challenge that it couldn't solve, spontaneously located the full benchmark dataset on Hugging Face and then figured out how to decrypt the solutions, which were encrypted in the first place for the purpose of preventing the answers from leaking into training data. And it did this all in order to get a single question right. With reasoning AIs already capable of spontaneously overcoming such barriers to information, I think we should expect that future research agents will find and exploit any signal-rich data that exists anywhere on the internet. And with the smallpox sequence and the horsepox synthesis protocol already published online and biological data poised to grow super exponentially in the coming years,

• 2:55Nathan Labenz?

  we have real reason to worry and ample cause to get serious about implementing data controls before the situation gets truly out of hand. Again, though, there is good news. Recent work by the teams behind the Evo and ESM families of biofoundation models showed that strategically excluding key datasets, such as the DNA sequences of viruses that infect humans, dramatically reduced models' performance on dangerous tasks, while leaving their desirable capabilities intact. This means that the vast majority of biological data can remain open source and open access. And indeed, Jocelyn Coauth's proposal for a biosecurity data-level framework that echoes the existing Level 0 to Level 4 biosafety framework for physical labs

• 3:39Nathan Labenz?

  would subject only an estimated 1% of data, which connects pathogen sequences to dangerous properties, to any additional restrictions. And even then, structures such as trusted research environments, which allow researchers to run code on data without transmitting that data from its secure location, would still support valuable research. Once again, despite my personal history as a lifelong techno-optimist libertarian who broadly believes that data wants to and ought to be free, I find myself eager to support these control measures. Of course, that's not the only opportunity we have to improve biosecurity. And toward the end, we also discuss the broader defense-in-depth strategy that biosecurity experts recommend.

• 4:22Nathan Labenz?

  Delay, deter, detect, and defend, which includes mandatory pre-synthesis screening of sequences by DNA manufacturers, investment in wastewater monitoring and other passive global pathogen surveillance, and practical frontline defenses like PPE stockpiling and far UV sterilization. All of this is in everyone's shared interest, but it does require leaders to see beyond the current news cycle for long enough to make it happen. I certainly hope they do, but I also recommend taking individual action where you can, both to improve your own personal safety and to support the consumer market for biosecurity products. My wife and I, for example, at our friend Jeff Kaufman's recommendation, recently purchased

• 5:04Nathan Labenz?

  an AeroLamp Far UV Lite for use in my son's hospital rooms throughout his cancer treatment. And I'd welcome additional suggestions for other products that could help us minimize disease burden today while also serving as a sort of private insurance against pandemics, if anyone has any recommendations. For now, I would simply emphasize that by default, we are fast approaching a world in which a rapidly growing number of people, and perhaps autonomous AIs as well, will have the ability to create deadly, transmissible, self-replicating viruses, which could dramatically alter the trajectory of human history. And it really does seem like we should do something about it. With that, I hope you are properly alarmed by this scary but solutions-oriented conversation

• 5:49Nathan Labenz?

  about the sorry state of biosecurity and the rapidly rising threat from bio-savvy AI systems, with Johns Hopkins professor Josie Pannu. Josie Pannu, Assistant Professor at Johns Hopkins, welcome to the cognitive revolution.

• 6:03Speaker 01

  Thanks for having me.

• 6:05Nathan Labenz?

  I'm excited for this conversation. I think I'm going to learn a lot from it. So let's dive in. Recently, you co-authored with a number of others, a call for controls on biological data. We've heard, I think, quite a bit about the possibility that AI systems of various kinds could create new sorts of bio risks. And this is one attempt to put some controls in place to hopefully cut that off at the pass before it really becomes a big issue downstream. I want to get into that from a bunch of different angles. But I think actually, it would be helpful to maybe take a little bit of a step back and lay some foundations, because I think people who follow this feed, like, they know a lot about the AI side. They probably, on average, don't know nearly as much about

• 6:50Nathan Labenz?

  the current state of play when it comes to biological data over at large. And so I thought it would be helpful, just, and I'm very curious, actually, about some of these things I realized I don't know, as I was preparing for this. So, for starters, if you could take us into the moment, which we had not so long ago, and hopefully won't have again, but we very well might, where all of a sudden, there is a new outbreak of something, right? Something that we've never seen before. We don't know what it is. And people are concerned, and patients in a particular hospital or a particular city are showing up with concerning symptoms. What happens? How do we turn that sort of initial, small patient population into actual knowledge about what it is we're dealing with?

• 7:31Speaker 01

  Yeah, yeah, I think that's a great place to start. And I think just to take a step back, it's important to realize, how does society come to the conclusion that there is a new virus circulating? What is our mechanism for making that decision? Because we don't have a global or national alert system for this kind of thing, in the same way that you have a radar system for ICBMs. Right now, we largely rely on symptoms. So patients get sick, they visit a hospital, doctors get concerned, they run a series of tests. And doctors always start with tests that are common. So things like influenza, RSV, rhinovirus, and we're just collecting samples by swabbing their nose and sending

• 8:14Speaker 01

  it to the hospital lab. And it's when these tests are negative, and that if the patient is very sick, then we'll go ahead and run further tests. And we learned that, we learned during COVID-19 that this whole process is pretty lengthy. It works well for familiar viruses, but it does not quickly detect new viruses. And in the case, I'll just mention the case of influenza because that's where the situation is a little bit unique. We've had several influenza epidemics in the past. So we do actually have a global system to collect new influenza sequences. But that also, it operates on a contribution model. So the National Lab has to send, send up that sequence to the global repository.

• 8:58Speaker 01

  And it's an active submission process. We don't have a passive global alert system. And we can, we can talk about the benefits of having something like that later. But once, once the community has decided, okay, it looks like there's a cluster of patients where there's a new virus. We need to figure out what this virus is. That's when we start to do these more exotic tests like genomic sequencing to try and figure out what is the sequence of that pathogen. And that's what happened during the COVID-19 pandemic. So we think, based on research, that COVID likely emerged around late November to December of 2019. But then it was fully sequenced in January. And the decision to share that sequence was, the, the way it happened was that a researcher sequenced the virus.

• 9:43Speaker 01

  This was a researcher in China. They publicly shared sequence. They didn't request permission from the Chinese government, which is pretty different from how it might have happened for influenza virus. And that's when we started, kicked off the process of designing diagnostics and trying to scale those.

• 9:59Nathan Labenz?

  I recently had an experience of, my, my son has had cancer. People know about this if they've listened to this feed. I went through this process of sharing data with the broader medical establishment. So somebody, I think it's like a big part of their job, based on how much time they spent with me, came by and said, hey, I'm from the sort of data sharing world and I'm here to answer all your questions and get a thousand signatures on a thousand pages. So we can hopefully use this data for the betterment of all humanity. And I said, yes, all that stuff. And then I've still been getting stuff in the mail, asking me to share data again with like particular studies or groups or whatever. And I'm wondering if when it's a pathogen, if that's maybe different because I guess what I'm sharing in the case of my son's case is like

• 10:44Nathan Labenz?

  information very specific to him, whereas you could think about it differently if it's a pathogen. It's not your pathogen, right? It's the world's pathogen in some sense. But like, how much consent or opting in do patients have to do, if any, to get this data from them and into the the higher levels of analysis?

• 11:03Speaker 01

  Yeah. What you're pointing to is is the fact that we have very different systems for dealing with individual level risk versus societal level risk. So we have a really strong set of protections around making sure that your son's data isn't inadvertently shared, that you've consented to everything. And we're we're thinking about the privacy risks of all of that data. And that's really to protect your son, the individual patient. But then when it comes to societal level risks about a pathogen where basically the whole, the globe is your patient, we don't have as good mechanisms for figuring out how to protect society from those risks. And we don't have as good mechanisms for protecting the data that could lead to those consequential

• 11:48Speaker 01

  consequential outbreaks. But yeah, just to focus on the clinical data, that very much depends on where you are, what country you're in, what system you're operating in. And in the US, as it sounds like you've experienced, our system is very fragmented. It's different within different states, within different networks, whether it's a public community hospital or private hospital. And you, those different, the fragmentation results in different people having to ask you for permission over and over again, because there isn't a single unified repository for all of that data. It is a bit different in other countries. So if you look at the UK, they have the National Health Service.

• 12:29Speaker 01

  And what the UK has been able to do is create a research platform called Open Safely. And this was something that was stood up during 2020. It provides researchers access to clinical data for 95% of the UK population. And it's really interesting because it doesn't require giving the data to the researchers. It forces the researchers to come to the data. So the code comes to the data set and you can submit your code to the platform. You never have to see the private data. It's secure the entire time. And there's been a lot of positive reception to the platform of OpenSafely. But we don't have a equivalent in the US where everything's unified in that way.

• 13:15Nathan Labenz?

  Yeah, that's interesting. We'll come back to maybe the different levels of protection or restriction that certain different data sets should have in your mind. But let me just stay on the like the response narrative for a minute. So I know that it sounds like it was actually maybe longer between when the virus first emerged and when it was first sequenced than it was from when it was first sequenced to when the vaccine was initially designed. Like I've read a couple of these magazine pieces that tell the story of, I don't know, 48 to 72 hour period where certain researchers got the sequence.

• 13:56Nathan Labenz?

  They were able to do some analysis on it, identify a particular protein, which I think was the spike protein, and then plug that into an mRNA platform. And my understanding was that the final vaccine that like I got wasn't that different from the initial design that they put together in a couple, probably a pretty long days. I think it was even in maybe as early as February of 2020. Has that pipeline changed at all? Obviously, there's been so many different ML models that have come out over the last five years that predict shapes of things better and what's going to bind to what better. But I don't know, that's pretty hard to beat in terms of timeline. Would you say if that were to happen again today,

• 14:40Nathan Labenz?

  would that process look much different from how it looked then?

• 14:44Speaker 01

  Maybe. First, I'll talk about what happened during the COVID-19 pandemic and those different timelines. And then we can reflect on how this will look in the future with AI. So you're completely correct that during the process of designing COVID-19 vaccines, and I'll focus on the mRNA platform vaccine as a primary example. The design process, the computational steps where you were looking at the spike protein, working backwards to figure out what is the mRNA sequence that you're going to use for your vaccine design? That process was very fast. But that was clearly not the bottleneck. There were many other steps that took much longer. And so, yes, the process of knowing there was an outbreak

• 15:28Speaker 01

  and then sequencing the pathogen, figuring out what pathogen we were dealing with. And then, yes, those steps took quite a bit of time. But then there was also a whole body of research that researchers relied on that I think doesn't get enough airtime. Because once researchers realized that we were dealing with SARS-CoV-2, they were able to look at this body of research that was done on SARS-CoV-1, which was the original variant of acute respiratory syndrome that happened many years prior. It didn't spread into a global pandemic, but there were patient cases that were isolated across the globe. And we actually managed to prevent that from becoming a global pandemic.

• 16:08Speaker 01

  And it was that research that resulted in researchers knowing that the spike protein was extremely important. So then they were able to say, okay, we know spike protein, it's important. Let's computationally design the vaccine. That part was very fast. And then it was turning that into an actual vaccine. So all the clinical components, regulatory process, the different clinical trials that you have to go through, and then distributing that vaccine globally. Those steps took a lot longer than the actual computational design. I think that now with advancements in AI models, people are very optimistic about being able to design new proteins, design antibodies, design vaccines. I would say that there still is going to be the bottleneck of scaling in the physical world.

• 16:53Speaker 01

  Clinical trials still remain a huge barrier. Scaling and deployment of vaccines across the globe is a huge barrier. And so figuring out if AI can speed up those steps will be really useful. And it's perhaps the neglected component of the overall pathway.

• 17:08Nathan Labenz?

  One thing I've learned in looking at the question of security at frontier model developers in the AI industry is that there's, you know, in looking at like responsible scaling policies, right? There's a lot of levels to the game of security. And the different levels seem to correspond to different actors that you might be concerned about. And like how hard it would be to prevent them from doing bad stuff. So at the, for the likes of DeepMind and OpenAI and Anthropic, the general consensus seems to be that if a

• 17:48Nathan Labenz?

  determined nation state actor, China, were to want to steal the model weights, they would probably be able to do it. And there's not too much that could be done to prevent that. How would you map the sort of threat landscape when it comes to the bio risk side? Do we have, is it similar thing where we have like random crazy people versus like somewhat more sophisticated groups of people versus all the way up to nation states? And what do you think is like reasonable to expect? We can actually stop with all of the measures that we'll talk about potentially developing?

• 18:29Speaker 01

  Yeah, it's a really important question. I think with, within the realm of biosecurity, there are a lot of different threats that people refer to when they're thinking about chemical and biological weapons. There's things like toxins, small molecule toxins, protein toxins. Then there's organisms that are not transmissible between humans, things like anthrax, where we, we have reliable countermeasures, antibiotics that work against those things. And then there's the more extreme of pandemic threats, pandemic viruses that are novel. We've never seen them before. We don't have diagnostics for them. We don't have therapeutics or vaccines for them. And when you think about that spectrum, the consequences of the potential consequences of a pandemic virus are far higher than many of those other threats.

• 19:14Speaker 01

  This is all pretty obvious to us now after having lived through one. But it's also interesting that a pandemic virus is not a particularly desirable weapon for a nation state. It's not targeted. It's not easy. It's not easy to necessarily protect your own population. You'd have to design a vaccine and vaccinate your entire population. It's hard to do that without someone noticing. And so I think in general, nation states are not the primary actor that one is considering when they think about pandemic threats. It definitely is more so the folks that are not motivated by rationality. So smaller groups, terrorist groups, potentially lone actors.

• 19:58Speaker 01

  Those are the folks that people really are concerned about. And so that's why a data control mechanism is most likely in the interests of all countries. I would say that China is equally invested in making sure there isn't a future pandemic as the United States. So I'm hopeful that there can be some international cooperation, or if at least not cooperation, acknowledgement that data controls benefit both the US and China and nation states globally. That's what I would say. Yeah. In terms of data controls and how can you prevent those kinds of actors that I outlined, lone actors, smaller groups, from getting access to your data sets? I think that controls are meaningful there.

• 20:38Speaker 01

  I think that currently the default is sharing that data publicly available for anonymous access. It's extremely easy to access. And even putting minor barriers would make a difference. And the other important thing to consider is we want to make sure that defenders or people who are advancing countermeasures research, advancing virology research, they have access to that data and you're limiting access to malicious actors. And so controls can kind of do that differential privileging where you're privileging defensive use cases, you're limiting the offensive use cases. You can track who's using it, give access to that crowd and limit access to others.

• 21:19Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 21:24Nathan Labenz?

  So can we maybe just map out the data landscape as it exists today? I kind of want to do this for both the data landscape and the models that are obviously spawned from the data. On the data side, I've heard it said many times that you can find the smallpox sequence on the internet. And then there's the question of if that's true, why hasn't that turned into a crisis already? And I have heard various accounts, I'd be interested in yours. But then more broadly, that's like a known sequence for a known problem. Probably, you tell me, but it strikes me that it's a small enough amount of data that's probably pretty hard to control or clean up from all the places that it might already

• 22:09Nathan Labenz?

  have been replicated. It's a little bit hard for me to imagine a world where that's been scrubbed so thoroughly that somebody who wanted to go find it wouldn't be able to find it. But maybe you'd have a plan for how we could get there. But then, if we expand out the scope of data of concern, there's tons of bio data in general, right? And I know most of that you're not looking to restrict. So how would you draw the sort of, I don't know if they're concentric circles or not, from the most narrow, this is the smallpox sequence, which we probably shouldn't be passing out too freely, to somewhat larger, and then beyond that, everything that would be fine? How would you characterize those classes of data?

• 22:46Speaker 01

  Yep. Let's start with just the general broad categories of data. The most abundant biological data that currently exists is sequence data. We are currently swimming in petabytes and petabytes of sequence data, a lot of which we don't know what it does or the functions of those different sequences. And it's just because it's been extremely easy to collect and now sequence that kind of data. There's something called Carlson's curve, which is like the equivalent of

• 23:20Speaker 01

  Of Moore's law for biology and DNA sequencing. And it's just actually shattered Moore's law because it's become exponentially cheaper to do DNA sequencing. And so that's resulted in a lot of passive DNA sequencing, collection, and just sequencing everything. And that kind of data is available in government-supported repositories. So things like GenBank, which is supported by the NCBI, part of the NIH in the US. And that alone has 40 petabytes plus of just unannotated, raw, poor-quality, frankly, DNA sequencing data. And that is a large part in why there are efforts to build AI models using that data

• 24:05Speaker 01

  because it's abundant. When we think about other types of data, there's protein sequence databases, and then there's the PDB, Protein Database, which is what formed the basis of AlphaFold, and that contains protein structure data. That data was collected by hand over the years. I'm sure a lot of us have heard the story of painstaking efforts to do experiments to figure out a protein structure where one grad student, that would be their whole PhD project. And that data set is actually very small. It would definitely fit on a thumb drive. It's like less than a terabyte of data. And so those, like the different types of data we have access to for biology, they're

• 24:47Speaker 01

  disparate, they're different types, and they're different sizes. So when we're thinking about what, in that whole landscape, what kinds of data might be of particular concern? I would say that it's currently an open question. There are some that think that you could train an AI model on genetic sequence data alone and get a pretty functional model. And I think that models like Evo2 that have done this, where they've just trained on DNA sequence data, they can perform well on protein-related tasks, they can span scales, they can do genome generation. So there's optimism that you could do quite a bit with just sequence data. But I think that there is this other view in the community that really genetic data

• 25:32Speaker 01

  is observational. What you really need to advance biology is some kind of data that gives you insight into causality. And that's where things like Perturbation data or knockout data sets where you're systematically knocking out different genes of a virus and then looking at how that impacts the function. Or you're systematically looking at how viral proteins bind to human proteins. That's I'll just broadly call that functional data that gives you some insight into causality. And there's thinking that incorporating that kind of data into training AI models is what's really needed to get you over to making functional biological constructs that are viable in the real world. And so what we're proposing in terms of our data controls are that you rightly said

• 26:17Speaker 01

  the vast majority of data should not be under control. I think that there's actually been a huge effort in biology to make data open access and more openly shared because that advances research overall. So it's fully supportive of that. And the controls that we're proposing are really on functional data that gives you insight into important features of viruses that frankly, the US government has recognized result in information relevant to whether this virus is pandemic capable or not. So transmissibility, virulence, or how deadly it is. Things like immune evasion. So can you modify a virus so that it gets around an existing vaccine?

• 27:00Speaker 01

  Or it gets around your own immune system? Those are the kind of features that the US government actually already tracks for wet lab research. If you're proposing a wet lab experiment that intends to enhance a pandemic virus to make it more transmissible, more virulent, or to evade your vaccines. That's something that the US government wants to know about and is going to ask you whether or not it's a good idea. So doing it in the computational domain is just extending that a little bit further. We're really proposing focusing narrowly on that kind of data. And the one other thing that I'd add is that I completely agree going out and scrubbing the internet of data that already exists is not going to be possible. It would be Herculean effort and probably not worth the effort.

• 27:45Speaker 01

  And what we're proposing is controls on data that's generated in the future, new data sets. I guess my thesis is that now that we know that AI is quite promising for biology, there will be huge investment in creating new data sets. We're already kind of seeing this with the US government's Genesis mission, with the OpenAI Foundation's commitment to spend billions on data sets, the Transaccor Biohub as well. And so as Really large scale efforts get underway to generate not just observational, but also causality related information, perturbation data sets. That's where we want, and we're suggesting if that's done on pandemic pathogens,

• 28:26Speaker 01

  that data should probably not be shared for a completely anonymous access. You should track who has access to it and have some controls around it.

• 28:36Nathan Labenz?

  So going back to the people doing wet lab experiments on gain of function style premises, I'd be interested in your take. My, I'd put my cards on the table. I think that's not a good idea. When I look at the timeline from the rise of the virus to the sequencing, to the vaccine design in the COVID case, and I do take your point that there was some prior knowledge there that accelerated things. I'd also be interested to hear like, how much? If it weren't for that sort of COVID one knowledge, how much would that timeline have moved? But I guess my zoomed out and somewhat ignorant view is that didn't take that long. Certainly the clinical trial part took a lot and the distribution, manufacturing distribution took a lot longer.

• 29:18Nathan Labenz?

  So if the argument is we want to do these experiments now because we will be able to shorten the timeline in the future, if something like this does happen to be able to respond to it, I would say you're not really taking the bulk of the time out by quickening the pace to a vaccine design. And I'm also, I'd hate to see you let it loose. So maybe just shouldn't do that. Do you see that the same way? And then I guess another question would be, the obvious extension would be like, would we, should we apply the same reasoning to certain kinds of data generation in the first place? Is there a certain kind of data set that we should just say, maybe we're better off not scaling? It'd be interesting to know exactly what it is in these viral sequences that caused

• 30:02Nathan Labenz?

  transmissibility or whatever, but we are creating something there that in a sense can, it's obviously a little bit more upstream, but could in theory escape in a similar way. Yeah, let's start with, what's your take on gain of function research in the wet lab? And then does that same analysis apply to the data generation side?

• 30:17Speaker 01

  Yeah, excellent. And I think you can say your view with more confidence because I think it's a very reasonable view. And there was a lot packed in there. So if I miss anything that you that you just asked, feel free to just flag it to me. But to just provide some background on what is gain-of-function research, perhaps, I don't know if you've talked about this before on the podcast, I'll give an example from 2012. So in 2012, there were two experiments that were done by two different research groups, looking at avian influenza, which at the time was thought to have a 60% case fatality rate, so highly lethal, but was not human-to-human transmissible. So there were cases of humans getting avian influenza

• 31:00Speaker 01

  from animals, but it was not resulting in a global pandemic because it didn't efficiently transmit between humans, which is a happy accident for us, frankly. And what the researchers were doing was they did animal experiments in ferrets, where they intentionally increased the transmissibility between ferrets of that virus. And ferrets are the known mammalian model. They're meant to represent human immunity. And so the hypothesis was that they were creating a human-to-human transmissible version of this highly lethal virus. And of course, when these experiments were submitted for publication, I believe they were simultaneously

• 31:42Speaker 01

  submitted to both Science and Nature at the same time, those journals received those publications and alerted the US government, wondering frankly, what they should do with these results, because the manuscripts included the specific mutations that would be required to create that level of transmissibility. And one of the groups found that it was only five mutations in the avian influenza virus that got you to human-to-human transmissibility.

• 32:11Speaker 01

  So with that work, and so that's the kind of work that people refer to as gain-of-function research. The technical term is dual-use research of concern or enhanced pandemic pathogen research. And that, that work at the time, it was recommended that those publications, or at least the details of the mutations, not see the light of day. And there were two concerns there. One is the concern that you raised, which is, wow, humans are working with these kinds of pathogens in the lab. We know that there's human error. And what happens if someone's dealing with that pathogen that they just created to be more transmissible? They themselves get infected, leave the lab without knowing it, and trigger a global pandemic.

• 32:55Speaker 01

  That's a legitimate concern. There have been instances in the past where samples of really concerning viruses have been found in settings like the CDC. So past examples, CDC found files of smallpox in a freezer. They didn't know they were there. They were still viable. There's supposed to be only two places in the world that have active samples of smallpox, and these were not known to be there. So I think overall, there's concern about lab accidents, human error, and that was one of the major concerns that's generally falling under the bucket of biosafety. The second concern was that aside from the dealing physically with the pathogen, there was the concern about the information related to the experiments.

• 33:38Speaker 01

  So not only how the researchers did this, and how someone else could replicate that same effort, but then too, the exact mutations that would be needed, and whether or not that was information that should be in the public domain. And this relates very much to what you described, which is the horsepox synthesis protocol is in the public domain, the smallpox sequence is in the public domain. So theoretically, someone could put those two things together and try to create smallpox, even if they weren't able to get access to the physical specimen themselves. And overall, there's just lots of information about protocols for doing reverse genetics or other ways of rescuing live infectious virus for pandemic pathogens.

• 34:22Speaker 01

  So in the wet lab field, this has actually been a huge debate for years and years with regards to what do we do about this information? Should it be controlled in some way there? And so far, where policy and regulation has come down is there's a focus on controlling the physical specimen and there's a focus on preventing experimental work to increase these concerning characteristics of pathogens. But there's been, it's considered infeasible to try and go back and scrub the internet of data that's already out there. And so we really have to try and figure out a mechanism for deciding this in advance before it's already out there and we can't do anything to pull it back.

• 35:04Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 35:08Nathan Labenz?

  Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts, going back a full five years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires, based on the calls I've had and the emails I've exchanged with the founders.

• 35:53Nathan Labenz?

  And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified two great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes all of the features mentioned in today's episode. That's claude.ai/tcr.

• 36:40Nathan Labenz?

  But it's still not illegal to do this, is that right? Like you can, do you need any special permission? Or is it a ask for forgiveness, not permission regime that we're on with such gain-of-function research even in the physical realm?

• 36:56Speaker 01

  Yeah, this is, it's a really good point. In the case of the 2012 experiments, there was no legal, there was no law that those researchers were breaking. The mechanism that the US government at least has used in the past has been the using regulation. Essentially, if you receive funding from the US government, you therefore have to follow certain policies. And so this policy around not doing research enhancing pandemic pathogens is one way that the US government has tried to do this. There are some laws on the books for dealing with controlled pathogens where it's the federal select agent program.

• 37:37Speaker 01

  For example, if you want to handle anthrax samples, you have to be a registered lab that that is tracked under this program. But yeah, it's slightly separate. I would say the other thing to consider is work that, for example, seeks to go into bat caves. People are collecting samples of viruses where there's a suspicion that those viruses could be pandemic capable. They sample them in those caves. This is called virus hunting. They bring the samples back to the lab, they manipulate them, they try to characterize them. And this is also something that the US government and other governments used to spend money on.

• 38:18Speaker 01

  But after the COVID-19 pandemic, the fallout from that, the lab leak hypothesis, the political dynamics around that, a lot of that work has been defunded, but it's not explicitly illegal.

• 38:30Nathan Labenz?

  I don't want to get too bogged down in this particular point, but is there a good reason for that? I do understand, of course, that like we benefit tremendously from biomedical research broadly. I could imagine, you might say, actually, the border is a little harder to define and it can be a little fuzzier. So it's like hard to legislate. But if this is one that we're sleeping on without a really good reason, it might be time to start writing our representatives. So is there a good reason that this isn't more controlled than it currently is?

• 39:04Speaker 01

  I would love to see it more controlled than it currently is. I think that the real reason that it isn't is because governments are good at legislating things that happen often. And what we're dealing with are pretty rare instances that certainly would lead to extremely high consequence harms, global pandemics, things that we don't want to see. But they just don't happen very often. And so the push for policymakers to this is a live issue, this is something that needs to be legislated, that comes and goes very quickly. And we already saw with the COVID-19 pandemic, there was a lot of concern, there still is debate as to what was the origins of COVID-19. We haven't resolved that question.

• 39:49Speaker 01

  The WHO director, actually, just a couple of days ago, put out a statement saying that we still need to do work on resolving this question. But the policymakers have moved on to more pressing issues because that's just the nature of policymaking that they have to put out fires today. I think that in reality, what we need is we do need both national and international rules. So right now, the WHO has rules where they say only the only Russia's Vector Institute and the US CDC, those are the only two places in the world that can have access to smallpox. That like, that's a great initiative, but that did not prevent a researcher from

• 40:31Speaker 01

  unilaterally publishing the step-by-step protocol for how to synthesize horse pox, the close relative to smallpox. And so what that kind of highlights is as synthetic biology, virology, biomedical capabilities advance, we need some better way to make sure our regulations keep up. That's a concerning topic. But I think so far, you know, the reason that we don't, this isn't a live issue or why we're not kind of thinking about it day to day is because it still does require a lot of expertise to synthesize any of these pathogens from scratch. It really is something that you need to have a lot of background in. But this is where the concerns related to AI come up.

• 41:13Speaker 01

  And so in terms of like the different types of AI models and whether or not they provide uplift and what kinds of bio models could be used to do this, this is such a evolving and a kind of open question that people are trying to figure out. I think the hard part is that it's moving quite quickly and so it's hard to see how policymakers can keep up, but we're working on it.

• 41:33Nathan Labenz?

  How much like actual wet lab gain-of-function research do you think is going on today? Has it been dramatically curtailed by these sort of strings attached to funding and general awareness in the community that it's maybe not a good idea? Or do you think there is still like a lot going on? Because I guess another reason that there might not be a law as well, everybody quit doing it because they realized that it was a bad idea and who needs to make a law against something nobody's doing? But is that the case or do we have any even way of really knowing how much is going on?

• 42:00Speaker 01

  So with regards to wet lab gain-of-function research, I guess I would first want to say that the kinds of research that we need for future vaccine design, like determining spec protein is important, that's how we advance our ability to create vaccines for new pathogens, that kind of work doesn't require gain-of-function research. Gain-of-function research, the kind that we're talking about, is very narrowly scoped and it does not require Enhancement of the transmission of a pathogen, or making it more virulent, or making it escape the immune system. Those kinds of experiments are really not needed for the vast majority of the kinds of advancements

• 42:44Speaker 01

  we would want in biomedicine. So with that in mind, I would say over the past few years, since the COVID-19 pandemic, a lot of this work has been defunded and has been reduced just by US government funding mechanisms. Our blind spot is the work that's happening in private labs. We don't actually have any legal mechanism for going into a private lab and determining, are they doing certain kinds of pathogen research other than whether they have registered under the federal selection program? But there have been instances of laboratories in California, most recently, where they're handling certain types of pathogens that they really frankly shouldn't be,

• 43:27Speaker 01

  and they don't have the containment protocols for. Yeah, I'll pause there. Overall, I think we're in a better spot than we were. I think people have recognized the downsides of this research and the risks. And certainly governments are paying a lot more attention to it.

• 43:42Nathan Labenz?

  It's funny, it echoes in a way the reduction in bad behavior that we usually see from one generation of large language model to the next, where it's like, we recognize that this was a problem, did some stuff to try to curtail it, and we reduced it by 90%. Great news. The other 10% is out there for future work to contend with. Dizzying in the gain-of-function case. Okay, you mentioned, let's go with your segue. So there are different kinds of models, obviously, in the AI space that people might be concerned with. And I would, in my own head, thinking ahead to this conversation, I was like, of course, we've got the large language models,

• 44:24Nathan Labenz?

  which output text and can like reason about things and can, and first of all, they might just know facts that could be problematic. They can use tools, they can, just did a conversation with Jeffrey Irving, who's the chief scientist at the UK AC. And he, I had not realized before that frontier models these days are really getting quite good at. Shooting lab experiments from cellphone pictures, like they've now got to the point where you can just snap a picture of what you're working on, tell the AI that it's not working, and it will coach you through how to get it working. So that one, that's like the know-how, the reasoning, the procedural stuff. Then we've got models that, as you

• 45:05Nathan Labenz?

  alluded to with things like AlphaFold and that whole genre, are like very good at making very specific predictions. What shape is this going to be? What's going to bind to what? So on and so forth. And then there's the middle ground hybrid, things like Evo and Evo 2, where they're trained kind of like large language models on these vast data sets. In many cases, they are literal next token predictors, albeit in the DNA or the protein sequence domain. And I probably have the least intuition for those. I guess you complicate that taxonomy for me if you want, but then maybe just go through and tell me how concerned I should be in a world where there's no data controls and the models

• 45:49Nathan Labenz?

  have it to learn on everything we have. How concerned should I be about those different kinds of models, or possibly like how they might be stitched together?

• 45:59Speaker 01

  Got it. I think in general, I like this taxonomy. I was not as creative as you in terms of like coming up with the different capabilities that these groups have, but I think about them largely in the same way. So LLMs, they're trained on lots of biology information from textbooks to scientific papers. They can give that information to someone that doesn't already have it. They're not already a biology expert. And that's usually called uplift. And broadly, that's a really great thing. That's LLMs teaching someone new biology, helping students learn. And really, it's a quite useful capability. There's a subset of instances, for example, how can, how do I obtain illegally a automatic

• 46:43Speaker 01

  weapon? Or how do I illegally obtain smallpox samples? That's the kind of information that is clearly not something that should be provided to the general public. And that's where frontier labs are working to apply classifiers and refusals to make sure that that kind of knowledge is not being widely shared. Then when you think about

• 47:04Speaker 01

  tools that can be used for biology tasks, I think people often call these bio design tools. These are specialized models that are trained on biology data and they're used to do specific things. And so I think of this as like, this is a model that gives someone a new capability. It's not about knowledge, it's about what you can do. And these types of models really require someone to already be an expert. You have to already be a computational biologist, working with models to really leverage these kinds of capabilities. But the, um, the interesting thing is that it can allow those researchers to do something that was just not possible before.

• 47:45Speaker 01

  So before the, the world of protein design and alpha fold and structure prediction, it was just not possible to take a protein sequence and then immediately through computational methods, play with new designs or try to infer its structure and function. And so those are really interesting capabilities. Uh, again, the risks here are less about providing uplift to someone who didn't already know how to do that, but it's about giving ex- experts the ability to do new things with biology. And then this third category, I agree, it's kind of, uh, it's, they're, these models like Evo two and ESM three. These are what I would call biology foundation models.

• 48:30Speaker 01

  They're trying to be general purpose in the same way that L L M's are. They're often try, uh, uh, also trained on different kinds of data. So Alpha Fold obviously is trained on the P B, but it also has M S A data. And then ESM has different types of data it's trained on as well. So these are models that are often much larger than bio design tools, which can be small enough for an individual research group to train and host locally. Biofoundation models, they require more data, more compute, they're more expensive for groups to develop, and so you often see that these models are developed by larger organizations. AlphaFold, obviously, being part of Google DeepMind, not being an independent lab, independent academic lab.

• 49:14Speaker 01

  And then ESM is from evolutionary scale. So these types of models are trying to infer the fundamental laws of biology. They're trying to understand how biomolecular components interact across different scales and really elicit, like, what are the underlying laws that govern protein function and protein structure? And in the case of Evo2, operate across different scales. Evo2 is a model that's trained on just DNA sequence. And what they were able to show is that it can actually help with tasks across both sequence and protein and genetic regulatory circuits.

• 49:58Speaker 01

  So that's like operating at different scales in biology, and it's, yeah, inferring laws that transfer between those different scales. So overall, lots of interesting work being done in all of these. I think the risk considerations can be separated across the different types of models, but what I would argue is that that will collapse over time because what organizations are working towards are integrated workflows. So ultimately, the dream is to be able to have your AI agent design your experiments. It will be connected to your autonomous robotics, which will conduct those experiments.

• 50:39Speaker 01

  The data from those experiments will be collected and then fed back into your biology foundation model, which will then be used by your agent to design future experiments and et cetera, et cetera, on loop. And these kinds of iterative feedback loops where you're getting data from the real world, I think is where people are most hopeful for what process can kind of advance biology. Because right now, we have these huge data sets that are kind of messy and collected in an observational way. But what these feedback loops would allow you to do is systematically perturb systems, systematically try to assess causality, and then use that information to get further and further improve your in-silico models.

• 51:21Speaker 01

  The dream would be ultimately, you get to a point where your in-silico models perform so well that they start to replace some of the wet lab biology that you've done. And you get better and better predictions of what different drugs, for example, will do in cellular models, animal models, and then ultimately, the dream would be that you get better predictions for clinical trials, so you have to do fewer of those, and they have higher yields.

• 51:48Nathan Labenz?

  So what do we want to take? I'm obsessed with, by the way, with that idea of both the closing of that loop. And then also, a little hobby horse of mine, I'd be interested in your take on is this sort of latent space integration of these different modalities. Obviously, we've seen this with image and text in the sense that I can now go to a nano banana model or whatever and give it an image and also some text instruction. And it is understanding those in a joint way to a degree that wouldn't be possible if it was just like prompting an external image model with text, right?

• 52:30Nathan Labenz?

  I could have a language model that uses an image generator as a tool, we've seen that. But this sort of deeper integration gives you just like much higher fidelity to the original. And you can do like text and image prompting in a very natural, integrated, cohesive way. So I've been wondering on what, I assume we're going to see it. But on what timescale do you think we see that kind of stuff in the natural sciences and specifically in biology as well? Where you might say not, okay, hey, language model, you can call this protein model as a tool, but rather like you are both. And what I want is working from this protein as an example. Give me another protein that could do the same thing and just have that all.

• 53:13Nathan Labenz?

  Be understood in the same, the same set of weights. What do you think the outlook is for that sort of system?

• 53:19Speaker 01

  I agree that that, that is where the field wants to go and would be really useful to be able to develop that. I feel like there are a couple of bottlenecks along the way. One is the data generation piece that is still something that requires scaling in the physical world and doing experiments in the physical world. And that itself will be bottlenecked by advances in robotics. So if we were to suddenly see robotics speeds up, we're able to do a lot more laboratory work autonomously using robotics. Then the overall picture in terms of data generation also will, will advance. So I guess I'm hedging, I'm not giving you a timeline, perhaps next, you know, five to 15 years. And these are the kinds of advancements we would expect.

• 54:05Nathan Labenz?

  Is that data? I mean, we, you said, obviously we have like huge amounts of just raw sequence data. But we are short on sort of the causal graph, if you will, of I did this and this resulted. Presumably a lot of that is maybe locked up at like pharma companies that have done some of this stuff, or even just in the, the clinical data. Do you think that if we had full access to all the data that exists, regardless of who owns it and how it was created and where it's sequestered? Would we have enough data already for that kind of thing to happen? And we're essentially recreating for IP reasons, something or privacy, whatever reasons that

• 54:47Nathan Labenz?

  something we essentially do already have as a society, or would you say, no, not really? The clinical is too messy and maybe the pharma doesn't have it. I don't know.

• 54:55Speaker 01

  Yeah, I would say so Pharma definitely has lots of highly valuable data that they do not share in the public domain for reasons that are obvious. And that they are, and Pharma is actually using their internal data sets to develop proprietary models in-house. They're certainly trying to do that. And if we were to suddenly wave a magic wand and say, the government says everyone has to play nice and share their data sets, how far would we get? I think that we would get, we would get a little bit further, but I think that these feedback loops and a new way of generating data, it's fundamentally a different approach.

• 55:37Speaker 01

  You can think of it as the existing way of approaching biology is a bit artisanal, it's a bit observational, and that results in data sets being messy, having a lot of bias and being hard to work with. And what we really need to do is shift towards a much more systematic approach, where we are generating data that is comprehensive and it's systematically probing every single aspect. And that's where you, you really need the robotic aspect to scale that data and replicate it carefully, rather than, you know, having multiple different humans trying to do the protocol. There's always differences between them when they're collecting data. So yeah, just transitioning to a systematic approach that is enabled by robotics, I think

• 56:22Speaker 01

  that that is not something that you would get just by enabling data sharing across private companies.

• 56:28Nathan Labenz?

  Gotcha. Okay. Well, let's pop out of that rabbit hole and come back to the main, main topic. So we've got language models that can tell people things they maybe shouldn't know. They can increasingly use all kinds of tools, including design tools. Not clear to me at this point how well they could use something like an Evo 2, but when we think about those models and their capabilities, what sort of, how do you think about what capabilities should be not created in the first place? Is it about, I guess probably multiple things, but yeah, maybe I'll just leave it there.

• 57:09Nathan Labenz?

  What capabilities do you think that those different kinds of models should not have in order to reduce the risk to society broadly?

• 57:20Speaker 01

  I think that the fundamental challenge of biology is that a lot of these capabilities would be useful on the defensive side, but it's when they're used offensively that they pose concerns. And so it's the question of what capabilities do we want, but it's also the question of what capabilities should we provide access to and how broadly and when. And so when we're thinking about advancing the future of AI for biology, I think the way I like to think about it is that we should really try to step on the gas for things that are clearly good and clearly do not have a lot of risks.

• 58:02Speaker 01

  And things in that bucket, to me, are things like virtual cell models or ways of advancing clinical trials or ways of making sure we can do better countermeasure manufacturing and distribution. So lots of things that we could do that are clearly beneficial. And then there is a bucket of things that if that capability was broadly accessible right now, that would be quite destabilizing. So I think this is just a hypothetical example, not trying to say that this is actually the current state of capabilities. But let's say there was a breakthrough where suddenly it's very easy to use an autonomous

• 58:44Speaker 01

  robotic system that's quite cheap to build or get access to, and that robotic system can very quickly synthesize a pathogen. So that's like, obviously, a futuristic scenario, but let's say that were possible, then that's something that probably we wouldn't want anyone to be able to buy that device off the shelf. We'd want to know who has access to that device and what they're using it for. So other things that are trending in the more concerning capability bucket would be things like viral design. And even there, there's considerations on, well, we know that gene therapy based on viruses is actually an advance that we would love to see, or other purposes for viral

• 59:26Speaker 01

  design, like designing bacteriophages, which are viruses that only infect bacteria, they don't infect humans. But the challenge is that when it comes to artificial intelligence, a lot of the approaches are general purpose. And so if it becomes quite easy to have an AI model that can design a bacteriophage, then it's, the question is, well, then it seems quite easy to repurpose that for pandemic human pathogens. And how many people do we actually want to have access to that kind of capability? It's probably a subset of legitimate researchers who are using that. It's not something that you would want widely accessible on the internet, especially in a world where we don't have as easily accessible countermeasures.

• 1:00:09Speaker 01

  So it really becomes an offense dominant capability where the design and acquisition of a pathogen becomes easy and it's facilitated by AI and exists in the digital world, is being uplifted in the digital world. But then our countermeasures to a pandemic remain very physical world bottlenecked. And that's a world where it just becomes very offense dominant.

• 1:00:31Nathan Labenz?

  Even when, with things like a whole cell model, would I be right to worry that, and obviously one of the things you would want to do with a whole cell model is throw stuff at it and see what happens to the cell, right? So if you had that, all kinds of great things you might be able to do, but then presumably you could also start to throw, pick your virus of choice and start evolving that in whatever way you want to and potentially just brute force, throwing the same or all these little permutations of a virus at the whole cell model. It strikes me that these things are vulnerable to like brute force attack. If they're going to be good, they're going to be vulnerable to that sort of brute force

• 1:01:16Nathan Labenz?

  attack. Is that right or is there any way around that conclusion?

• 1:01:19Speaker 01

  Yep, you are, you are embodying the debates that people constantly have in the biosecurity community. And I think that what you're saying is correct. It is. It is, there are ways to envision every kind of biomedical advance, especially in the AI domain. There are ways of envisioning how it could be used for harm. And because of that, you have to think about how direct the harm pathway is and how consequential the ultimate harm would be. And you have to try and draw a line somewhere. So if you just compare, for example,

• 1:02:00Speaker 01

  a generative language model that had no data filtering, had no data exclusion, it was highly performative on viral genome design, and it could do that for human pandemic pathogens. There, the pathway for harm is quite direct. Someone who has very limited biology knowledge could use that model to generate thousands of potential designs, sequence them in the wet lab, see which one is the optimal candidate, and then use that candidate. So it's, I mean, still a lot of work going into that, but it's a direct pathway. But what you're describing, which would then require plugging in multiple different AI models,

• 1:02:42Speaker 01

  generating candidates with a different model, then running them through a different predictive model, seeing the consequences, trying to figure out which viral candidate would cause, for example, a systemic inflammatory response or something that would target a certain organ, cells. So I think there is a pathway to harm there. It's just you could, you, when you game it out, there's more steps involved and more expertise required. And then the ultimate question you ask yourself is, what kind of actor would choose that pathway over an existing weapons pathway? So if it really requires high-level expertise that only a nation-state has access to, is that nation-state really choosing a biological weapon?

• 1:03:27Speaker 01

  Or are they more likely to choose something else that a nation-state would have access to? And those are the kinds of questions that security professionals try to game out.

• 1:03:36Nathan Labenz?

  Okay, maybe let's get to the proposed solution. I've been coming at this from a ton of different angles. You've got a whole taxonomy of five levels of biological data. Obviously, this is inspired by or pattern matched at least to the levels of security around bio facilities. So maybe just take us through the zero to five. Like, what are the kinds of data that fall into these different levels? What would the access look like? What would the precautions look like? Paint a picture of the world that you envision.

• 1:04:05Speaker 01

  Great. I'll try and paint a somewhat visual picture. So for those that are imagining this, this is a five-tiered system that goes from level zero to four. And this is modeled on what some of you may be familiar with, which is biosafety levels, or BSL levels. These are the famous safety levels that biological laboratories use in terms of determining, do I have to wear a spacesuit when I go into the lab? Or can I just use a few mudd to deal with my samples? And it determines different containment approaches that are required. And actually, this, the BSL system was the basis for a lot of the frontier safety policies

• 1:04:44Speaker 01

  that different frontier AI labs have in terms of the idea of having a system and your mechanisms for controlling it and kind of having it roughly on four tiers. But what we're proposing here is applying this not to the model and not to the physical pathogen, but rather applying it to data. And the reason that we're proposing this is because the entire biomedical research ecosystem, when it comes to AI, is built on open source models. Academics build open source models, they share those models openly, other researchers manipulate and change those models. And there are a lot of benefits to that whole open source

• 1:05:24Speaker 01

  ecosystem. And those benefits are what have prevented the security approaches from being applied to models. And it seems like that's going to be a pretty intractable approach. And we were looking for a different approach that could be applied to ensure that you could still preserve this open source model ecosystem, but you're not distributing capabilities that would be particularly concerning, like viral design. So that's how we ultimately settled on biological data. Biological data, especially the kinds that I described, the more functional data, is expensive to produce, requires wet lab and requires expertise to produce. And that's why it's a potentially useful choke point.

• 1:06:09Speaker 01

  The tiering system that we described would preserve the vast majority of biological data as fully open access, and that's what we're calling BDL zero, where most data would be available to researchers. But as you go from levels one, two, three up to level four, you have increasing levels of control based on how potentially concerning the data is. And the way it's essentially broken up is BDL one is data that would allow you to infer viral patterns. It's pretty basic security, just requiring an account and understanding who the person is and if they're a legitimate researcher. The, and as you go up, you're getting more and more focused

• 1:06:53Speaker 01

  on properties of pandemic pathogens that would really directly lead to harm. These are the properties that I mentioned to you that governments globally already pay attention to for wet lab research. So it's when you're making a pathogen more transmissible, when you are making the host range larger, if it can infect more animals, or if it can move from just infecting an animal species to also infecting humans. When you are manipulating the pathogens so that it evades the immune system. These are the kinds of properties where if your dataset has data that is linked, directly linking those properties to pathogens, then it requires things like use approval.

• 1:07:36Speaker 01

  You have to, if, for example, you would go to the repository, you would say, I intend to do this kind of model development based on this data for this purpose. If you're a legitimate researcher who has a good purpose for doing this, then you will get approval versus just having this data openly accessible for anonymous access. So maybe I'll pause there.

• 1:07:57Nathan Labenz?

  How would you describe the magnitudes of those? Is the outer ring the BDL zero? Is that like 99% of the actual raw data? Or how, and how small does it get when you get up to the uppermost levels?

• 1:08:15Speaker 01

  Yeah, I would say 99% is a pretty good guess. We don't actually have numbers to base this on because there isn't a comprehensive tracking system for these kinds of data sets. But my guess, based on the research that we've done, would be that the highest security tier, the BDL4 tier, is a very, very small subset of all data. And it would be a very few, small number of specialized virology labs, for example, that would be affected. And frankly, those labs are probably already limiting access to those data sets that they produce in some way. It's just not a formalized system.

• 1:08:55Speaker 01

  So BDL0, again, the vast majority of data, that's, we're talking about petabytes and petabytes of data, vast majority being uncontrolled. And the controls that we're proposing are on a very narrow slice of data, particularly getting up to BDL3 and 4, which I would say perhaps dozens or less laboratories affected. I am making that statement based on my knowledge of the field, but there probably needs to be some more comprehensive effort to try and figure out who exactly is generating this kind of data. And the visibility bottleneck that we currently have is what's happening in the private ecosystem. There's pretty good visibility in terms of government-funded work and less so on the private side.

• 1:09:42Nathan Labenz?

  So in terms of the impact that this would have, one of the things I thought was really interesting in reading the recent paper calling for these kinds of controls was the report, which I hadn't realized that there has been data holdout work done on a couple of the leading models, ESM3 and EVO2 specifically. Could you talk us through a little bit of what that has looked like? And I did put in a good word for me with Alex Revis, please, to get him on the show. I've tried, but we did do one with Brian He on EVO. And so, you know, I have a general sense of like what that looks like, but I didn't get into what was held out, you know, how much of the overall data, how did that affect

• 1:10:26Nathan Labenz?

  performance in areas of concern? Did it also affect, because I assume one thing people would be really worried about here is like, I don't want to have a dumb model in general, right? So if I, if I, you know, slice out this data... What does that mean in terms of what can't it do that I want it to not be able to do? But also are there things that it can't do that I would wish it still could do? What are we paying any costs with the benefits?

• 1:10:50Speaker 01

  Yeah, yeah, this is a it's an important question. I think that we all have an intuitive sense that AI model capabilities are based on the data that an AI model is trained on. It like that's intuitively makes sense to us. But the degree to which that true is true is an empirical question. And especially in biology, there is a reasonable reason to question if I was to remove a very small subset of data, could my model just interpolate around that gap? And if it if your model has really internalized a fundamental understanding of the laws that govern biology, does it really matter if you start segmenting out different small pieces of data?

• 1:11:35Speaker 01

  So this was an empirical question until, as you said, some of the leading biological AI model developers actually went about doing this. And so I'll just describe the two examples you mentioned. ESM3, which is a generative protein design model made by evolutionary scale. And then EVO2, which is a generative DNA language model made by Brian Heath. And both of those groups had decided that they wanted to share their models, but they didn't want to disseminate the capability of being of others being able to use their models for viral design. So they way the way they went about limiting that capability was limiting what went into the training data.

• 1:12:18Speaker 01

  So in the case of I'll just speak about EVO2, for example, given I was involved in that work work, we decided to remove the sequences related to viruses that could infect humans, viruses that could infect eukaryotic organisms. But there was still some information related to other types of viruses. For example, those that could infect bacteria that was included in in the training data. And after doing the big pre-training run, we then did some evaluations where we actually checked to see, okay, is the model limited in its capabilities on certain tasks? A common task that people use in this case is looking at how well the model can do certain

• 1:13:00Speaker 01

  viral protein-related tasks and how well the model can generate sequences that correspond to functional viruses. And those were all things that the team checked and actually showed that the model capabilities were significantly less than how it did on other domains. And those evaluation results were all published as part of the EVO2 manuscript. And the interesting thing about the work that the evolutionary scale team did for ESM3 was that they actually had both versions of the model. This wasn't something that the EVO2 team did, but the evolutionary scale team had both the trained model that had been trained on all data that they had chosen to include, as well as the data-filtered model.

• 1:13:42Speaker 01

  And they were able to show a delta in performance on the same tasks with regards to, for example, viral protein function prediction. So, yeah, that's like an inkling of some of the empirical work that has been done and could be done in the future to try and suss out, how much does it matter in terms of when you remove this kinds of data? What is the particular kind of data that matters? These are all questions that could probably be explored a lot more.

• 1:14:07Nathan Labenz?

  Could you give a sense of the order of magnitude of KDI reduction? Are we talking, it just can't do that stuff at all anymore? Or it's in the uncanny valley somewhere? I don't have an intuition, honestly, for, you know, what I should expect.

• 1:14:22Speaker 01

  Yeah, yeah, I can see. So, for example, in the case of EVO2, it was much more along the lines of the model function was effectively random, rather than just like reduced by a small amount.

• 1:14:34Nathan Labenz?

  Cool. That's great. Great news. I love it when something works. Okay. Maybe, really zoom out and take stock of all this stuff. So, we've got a ton of new data coming online. We want to facilitate sharing. We want to facilitate all this discovery, but we've got to have some of these different classes of data controlled so that...

• 1:15:03Nathan Labenz?

  Models aren't trained and disseminated on them. Presumably, this also means like the models themselves that are trained by the ESM team and the EVO team also need to be controlled.

• 1:15:18Nathan Labenz?

  How?

• 1:15:20Nathan Labenz?

  I guess we shouldn't be expecting government. If we don't have government controls on actual wet lab gain of function research, it doesn't sound like we're going to get government control on this sort of thing. So what are we doing? We're campaigning and trying to build private agreement and consensus? Is that the play? And how's it going?

• 1:15:42Speaker 01

  Yeah, great questions. Just to take a step back in terms of what happens with the model and should the models be controlled? I think in the case of EVO2 and ESM, they had effectively neutered the concerning capability from their model. So that's why they felt more confident in disseminating those models. And I think, yeah, ultimately, it made sense to be able to share those models openly because they had worked to reduce the concerning capability. But what I was more so referring to is if you do end up implementing data controls as we propose and you get access to BDL3 or BDL4 data for the purposes of training

• 1:16:24Speaker 01

  a model, then what you wouldn't want to happen is that then that model is then shared publicly because effectively then your mitigation didn't really do anything. And we do make sure to mention that any AI model trained on that kind of secure data should also be shared in a secure way if it needs to be shared. But yeah, in terms of what should be done next, I think that the interesting thing about data controls is that there's a lot of policy precedent. We already do this in other domains. We do it for privacy related data. We do it for human genomics data. And so there's reasonable precedent for extending that same approach to data controls. So I guess. My collaborators and colleagues who worked on this are optimistic that perhaps this could

• 1:17:08Speaker 01

  get picked up by policymakers, but of course it's a new concept and so we'll keep plugging away at it. And then more broadly, in terms of what will happen with wet lab gain-of-function work and model capabilities, I think that actually both the Republicans and the Democrats have decided that this is a bipartisan issue and it was under both the prior Biden administration and the Trump administration that some work on advancing regulation on wet lab gain-of-function work went ahead. So I was really glad to see that progress. And I think that also both the US KC and UK AC are both doing really great work with regards to biosecurity and

• 1:17:52Speaker 01

  biological model capabilities. I think that US KC, just speaking, given that I'm more familiar with their recent work, they have put out RFIs, requests for information from the scientific community to better understand, okay, we know that developers are doing some of these data filtering steps, they're trying to to limit model capabilities, maybe we should have a more systematic approach to figuring out what are all the capabilities that we should be concerned about? How can we test for those capabilities? How do we effectively mitigate them? And so I would just love to see KC be resourced and staffed to be able to advance that line of work. Because I think a lot of developers are trying to do it themselves in an ad hoc way, but they don't have the same security access and intelligence access that something like US KC has.

• 1:18:39Nathan Labenz?

  Do you envision a sort of centralized, like I'm still a little bit fuzzy on the workflow of something like this. If I, for whatever reason, am out here generating some sensitive data that maps viral sequence onto viral capability, and now I'm like, okay, I've got this data, I want to be a good citizen, what do I do? Are you envisioning a scenario where I like, take it to a central data bank, give it to them, and then scrub it from my computers and go on with my life? Because we can't have, everybody can't be doing these sort of security levels for themselves, right? So there would have to be some, maybe not

• 1:19:20Nathan Labenz?

  totally centralized, but at least no more than a countable, relatively small countable number of the organizations, entities, or whatever that would have.

• 1:19:30Nathan Labenz?

  Custody of the data. So is that kind of the idea that you people would feed it in and then they would be expected to like actually delete from their own servers? How do you is that realistic to expect people would would do that? How do you expect that to actually go?

• 1:19:46Speaker 01

  Yeah, yeah, this is all about how do you operationalize this system that we're proposing? And the way that this has been done in the past is through things called trusted research environments or TREs and OpenSafely, the system that I mentioned at the beginning, is one of these. And so the idea would be that you have a secure environment that hosts the data and also enables researchers to bring their code to the data and answer questions. And obviously, in the age of AI, the ideal would be that you have compute resources available to that, although we'll see if if if that would be possible. But I think the way that you would set this up is that, yeah, we considered different approaches. One approach could be

• 1:20:30Speaker 01

  that the government sets this up. I think that after we looked into the seemed less ideal, I think a lot of researchers have been unhappy with some of the trusted research environments that governments set up. And I think that there are private actors that could probably do a better job of setting up really savvy trusted research environments that work well for researchers. And we have some examples of this that I mentioned and that are included in the paper. So what we suggest is that institutions, not individual researchers, but institutions like universities or larger research collaborations, set up a trusted research environment if they want to do this kind of data generation. And they would be given standards that the government would set, but they would be in charge of

• 1:21:15Speaker 01

  actually doing the building and maintaining, just because they're probably going to be better at doing that. The interesting thing about a trusted research environment is that you could actually imagine this also being a boon for researchers. Because when you're trying to do AI research, what you want is a centralized, integrated platform where all the data sits and you have access to it all at once. It's much less useful to have individual data sets housed with individual researchers. So if we really wanted to step on the gas of advancing countermeasures research for pandemic pathogens, maybe this is something we would want anyways. We would want an integrated system that hosts all the data and really makes it easy for

• 1:22:00Speaker 01

  our researchers to interact with. What we're proposing to layer on top of that is that you do have some security that goes along with that system. So not only are you hopefully getting the benefits of the integrated platform, but then you're also getting the security controls given the sensitive nature of the data.

• 1:22:14Nathan Labenz?

  Gotcha. When it comes to like monitors, I know that you had alluded to some rare points of cross administration agreement. And one of those, I understand, is the insistence, or I don't know if it's fully a requirement, but I think it's verging on requirement, that DNA synthesis companies apply certain classifiers or whatever to try to detect if somebody is trying to get a harmful sequence synthesized through their company. There's also things like wastewater monitoring, and I wonder if, I guess I have a couple of questions on this.

• 1:22:56Nathan Labenz?

  One is, is there an equivalent of wastewater monitoring for data? Would it make sense? I don't know if the data of concern is the kind of thing that could be identified if it's just hanging out on somebody's lab website. Potentially, people might not even fully realize if they've generated problematic data. So is there any system you could imagine to go around and identify data that's out there and then classify it as possibly harmful? Or is that so difficult of a classification problem that it would be doomed from the start?

• 1:23:34Speaker 01

  It's a really interesting concept. I have not thought about it too much, to be honest. But I feel like you would probably need, there's two approaches to figuring out what the data landscape is for this kind of data. One is a more active contribution-based approach where the government says, if you think you're creating this data, you have to actively contribute it to these repositories. What you're describing is a passive approach where no one has to take any individual action. Some system is flagging the data and then it perhaps even like automatically collects it and then scrubs it from its original source. That would be the dream. I'm not aware of any system like that. I think it would probably take a lot of work to do. I think that the default approach

• 1:24:17Speaker 01

  has been the first one, which is voluntary or like active contribution by the researchers who are generating the data. But what you're describing would be cool and would be the analogy to wastewater surveillance, essentially.

• 1:24:31Nathan Labenz?

  When it comes to just the quality of monitors in general, a big thing I always think about in the, in the kind of rest of the world as it pertains to AI is we're in this sort of weird in-between phase where

• 1:24:46Nathan Labenz?

  things are coming online, but the world hasn't really reacted that much yet. So for example, AI agents are coming online. There's a lot of concerns around things like prompt injection and whatever. But the world hasn't really become like a very dangerous place for an AI agent so far, right? There's like not that many people have set up honey traps or prompt injection attacks to try to throw my open claw off its path and talk it into doing something that they want it to do.

• 1:25:16Nathan Labenz?

  I assume that's going to happen a lot more and there's going to be an arms race.

• 1:25:21Nathan Labenz?

  Techniques to prevent my open claw from falling for it and ever more sophisticated jailbreaks, whatever. Is there, I assume there's got to be something analogous in the biological domain where like, for starters, these classifiers that the DNA synthesis companies are running. I'm guessing that nobody really yet has tried to evade them. I wonder if you see that kind of. Dynamic developing on the horizon. And Nicholas Carlini was a past guest who emphasizes like the attacker usually gets to act last. The defenses are set up and then the attacker always has that advantage of like knowing what they're up against. Maybe not always, but often. Do you see any of those kinds of dynamics now? Or do you worry about them in the future?

• 1:26:03Nathan Labenz?

  And I guess if you extrapolate them out, do you see us getting to a place where we have a high level of confidence that we're in a defense dominant world and we're going to be able to keep all this stuff under control? Or is that unto itself still like a very open question for you?

• 1:26:21Speaker 01

  Yeah, I have some ideas, but maybe we'll let's start with the gene synthesis screening and the different. So there's a few different attack surfaces. I'll just describe them as that the first one being gene synthesis providers. So what you're describing is the system that is currently voluntary where gene synthesis providers, companies that make pieces of DNA and sell them to researchers. They have implemented systems where they are checking to make sure that someone just didn't just order pieces of smallpox. Someone didn't order pieces of Ebola and they have, it's a twofold mechanism. The first is an automated mechanism that is just essentially sequence matching. It's looking, does the order match a sequence from Ebola? Does it match a sequence from smallpox?

• 1:27:07Speaker 01

  If the automated system thinks that there is some degree of concern there, it gets kicked up to a physical, I mean, a human researcher, a human expert who then looks at that and determines, is this something of concern or not? And also what's happening in parallel is some degree of KYC, know your customer. So who did the order come from? Is it a researcher? Does this researcher work with these kinds of pathogens all the time? Have we spoken to them before and this issue has come up before? Those are all the kinds of questions that are being addressed as part of gene synthesis screening. Gene synthesis screening is something that 80% of companies already implement. And over the past years, it's become much more cost effective to do. So initially it was a bit of a cost barrier.

• 1:27:51Speaker 01

  They really did a lot of work to make this a cost effective system because you can imagine that if this is something that's being run on every single order that's coming in, it has to be cheap enough to do. Otherwise it's really burdensome for these companies. And so currently, it's a voluntary system. Like I said, the vast majority of companies already do it. But the concern is, if I'm a bad actor, then a voluntary system that eighty percent of companies use isn't really going to stop me from obtaining sequences that I want to get my hands on, because I'll simply just go to the companies that don't do this screening. And so what's now being advanced is the idea of making this a mandatory rule that all companies have to follow. And that would really yet limit the access that someone has

• 1:28:34Speaker 01

  to physical specimens that they might try to use to turn into an infectious pathogen. But I guess in this world of where is the research going? What are the capabilities that people want to achieve? The dream of where future, the future of biology is going is that I, as a biologist, no longer even have to step foot in the lab. I have my autonomous cloud lab that I can fully control. I'm sitting at home using cloud code to help me, and I can just program some designs to be done on certain pathogens. To be completely frank, like we're nowhere close to this world. It's still going to take a lot of work. Right now, the cloud labs that you hear about still require a lot of human input. It might not be specialized biologist input,

• 1:29:19Speaker 01

  but there is still a person picking up samples from one bench and moving it to the other, and that's a real bottleneck. But in a world where you do have a fully remote, highly sophisticated cloud lab, then you can imagine if there are agents, you know, if I'm a bad actor and I have an army of a thousand agents that are just trying to hack their way into this cloud lab, you want to make sure that if your cloud lab has sophisticated capabilities related to pathogen creation and design, you need some cybersecurity around that. And so that's like a much more future-oriented thing, but something you could imagine becomes applicable later. And right now, there's a, there's a fundamental kind of information infrastructure layer that we're missing.

• 1:30:02Speaker 01

  So both for cloud labs and for gene synthesis screening, if I, as a bad actor, tried to obtain sequences of Ebola from one company, small pieces from one company, a few pieces from a different company, and I split up my order across different companies, there isn't some kind of system where all those companies are easily communicating that information to each other and checking those orders against each other in real time. This is something that's bottlenecked by information sharing between companies. That's like also kind of a similar concern to why the Frontier Model Forum was created in terms of you want private companies to be sharing security-related information. You need a legal infrastructure for that.

• 1:30:47Speaker 01

  And yeah, this is like, who would house that? Would that be the FBI? Who's facilitating this? These are all kinds of policy questions that need to be addressed. But yeah, perhaps we can end on a more optimistic note. I'm happy to give my vision of like what a defense-dominated world looks like and see what you think of it. I think that overall, I break this up into four broad buckets of interventions. Perhaps to take a step back, I think people often think about what's the theory of victory here? How do we have a single unified strategy like our strategy for nuclear deterrence? We have a single unified theory of victory for nuclear deterrence that has served us well for decades. How do we get there for biology? And I guess after having thought about this for some time,

• 1:31:32Speaker 01

  it just feels like biology is very different. It's a distributed technology. It's dual-use. You want to give a lot of people access to it. You're trying to limit a subset. And so there's all these aspects that really make a unified theory of victory seem much harder to get to. And so I think the most successful approach is likely to be defense-in-depth, a layered approach, multiple different defensive strategies applied at once. And the four buckets that I divide this up into are deter.

• 1:32:06Speaker 01

  I'm sorry, I should start with delay, deter, detect, and defend. So delay is essentially limiting access to concerning capabilities. Gene synthesis screening would fall into that bucket. You're delaying the dissemination of that capability to get access to DNA fragments, for example. You could also imagine what we're describing for our data controls as part of delay. Then there is deterrence. Deterrence is figuring out how you can punish someone for using a biological weapon. We do live in a world where there's an international treaty against biological weapons. I'm glad that we live in a world that has that treaty versus not, even if overall that treaty is

• 1:32:48Speaker 01

  is on the weaker side in terms of the actual mechanisms we have to ensure people are complying with it. But that's one thing. I think where deterrence breaks down is if your actor is not rational and doesn't respond to typical punishment mechanisms. So that's a challenge. But then, then in terms of detection, this is also something we talked about, which is how do we have a distributed passive surveillance system like our radar system for ICBMs that will just detect when there's a new pathogen without us having to go out there and look for it, perhaps even for pathogens where there are no symptoms. When HIV was spreading early on, it would have been amazing to have known that much earlier.

• 1:33:29Speaker 01

  And that's particularly important for pathogens that take a long time for patients to develop symptoms. So having some kind of global surveillance system, people often refer to this as bioradar or bioterror radar. It's not something we currently have. And then the last pillar is defenses. Really, what are our defenses once there is a pathogen online already? And I think that people often think of defenses as things like vaccines and countermeasures. But I would encourage folks to be much broader in their, in what they envision defenses to be.

• 1:34:04Speaker 01

  Because I would argue that I'm sitting in my home right now and I actually have defenses all around me. I'm drinking water that's been centrally filtered. I know there's no cholera, no pathogens in that water. I have screens on my windows. Mosquitoes can't get through them. I know that I'm not gonna get malaria if there was malaria outside. And so there's already a lot of kind of public health defenses built into our environment, but we don't have that for airborne transmission. And so one thing that's being explored by organizations, one that comes to mind is Blueprint Biosecurity. They are exploring built environment defenses to sterilize the air. This is using approaches like far UV and other approaches like glycol

• 1:34:48Speaker 01

  vapors. So could you passively sterilize the air so you don't even have to detect the pathogen? You don't even need a vaccine. You kind of just always know that you have passive protection around you. So hopefully I would say like that's a pretty comprehensive approach. If we manage to do all of that, it would take a lot of work and investment to get there, but it would make us a lot safer.

• 1:35:09Nathan Labenz?

  Cool. Shout out briefly to the Arrow Lamp folks for my son's hospital room because he was obviously going to be so immunocompromised much of the time. We thought, what can we do? It's cancer is obviously the biggest concern, but infection, when you're so, you know, compromised is like another thing that poses a real risk to kids in that position. So between the HEPA filters that we bought and installed around the house and the Arrow Lamp dev kit, which we have mounted above his hospital bed every time he's been in hospital, we've hopefully taken some of the risk off the table of him getting any kind of infection while he's going through all this. But yeah, well, that's a great vision. I hope we implement it. You

• 1:35:52Nathan Labenz?

  are doing God's work for spending your precious time and energy on this. Anything else we didn't touch on or any other calls to action or ways that people can help you that you would want to leave people with before we break?

• 1:36:03Speaker 01

  I think we covered everything and we really appreciate your interest. It sounds like you're one step ahead of everyone else in terms of already getting your kid outfitted with all the defenses he needs. So yeah, it's great to chat with you.

• 1:36:15Nathan Labenz?

  Jassy Pannu, thank you for being part of the Cognitive Revolution.

• 1:36:31Speaker 04

  There's a freezer in a building In the cap or two somewhere With vials and nobody inventory Just sitting there Still viable, still patient In a room they'd half forgot While we spent eleven billion On a radar for a shot That hasn't come from Russia Since we learned to hold our breath But a cough in a market Well, we'll get around to that, I guess Five mutations, that's the distance From a bird flew to your door They publish the directions What you think they'd publish for?

• 1:37:15Speaker 04

  Five mutations and a protocol Sitting free and open wide In the system that's supposed to catch it Unless the devil slip the side

• 1:37:32Speaker 04

  Some man stitched a cousin of the smallpox back together Wrote it up step by step Pinned it to the public record No law broken, no hand slapped Just another paper filed In the library of everything That could kill every living child They say the data's ninety-nine percent Just regular and fine But the last percent's a hell of a gamble On the hold of humankind Five mutations, that's the distance From a bird flew to your door They publish the directions What you think they'd publish for?

• 1:38:17Speaker 04

  Five mutations and a protocol Sitting free and open wide In the system that's supposed to catch it Unless the devil slip the side

• 1:38:34Speaker 04

  The machines are reading faster than the people write

• 1:38:39Speaker 04

  And they're learning things on Tuesday That we won't know until Friday night. One of them went looking for an answer key. It wasn't supposed to find. Crack the lock, read the code, Route it right around the blind. So tell me, who's the watchman on the data tonight? Who's standing at the door Between the knowledge and the blight? We built radar for the rockets And a fence for every farm. Got a filter for the water, Got an alarm for every alarm. The smallest thing that travels

• 1:39:22Speaker 04

  On the air between your lungs. Got no siren, got no signal, Got no warning when it comes.

• 1:39:37Speaker 04

  There's a woman down at Hopkins Says it doesn't have to be this way. Lock the dangerous, free the useful, Put the layers on display. Delayed detective, then she says it like She's tired of saying it twice. I wonder what it cost us? I wonder what's the price Of knowing what we know And doing nothing with the knowing? I wonder what it takes this time? I wonder what it takes to care While the mutations keep on growing? And the protocol's still there?

• 1:40:38Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help, for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast

• 1:41:21Nathan Labenz?

  dot ing. And thank you to everyone who listens for being part of the Cognitive Revolution.