Lab-in-the-Loop at Compute Speed — Ian Quigley, Leash Bio

A few years ago I stumbled upon Leash Bio’s https://www.leash.bio/ Substack https://leashbio.substack.com/, and was hooked. Be it their Kaggle competition https://leashbio.substack.com/p/introducing-belka that showed that no, we really haven’t solved protein-ligand binding https://leashbio.substack.com/p/belka-results-suggest-computers-can, or their crazy discovery that knowing a molecule’s author is perhaps all you need to know to predict how good it is https://leashbio.substack.com/p/ai-for-chemistry-in-2025-is-like. Their substack — or at least Owl Posting https://www.owlposting.com/’s deep dive https://www.owlposting.com/p/an-ml-drug-discovery-startup-trying on them — should be required reading for any scientist trying to do chemistry with AI. So I was super excited to finally be able to sit down with Ian Quigley https://www.linkedin.com/in/iquigley/ from Leash Bio, and pick his brain. Leash specializes in DNA Encoded Libraries https://en.wikipedia.org/wiki/DNA-encoded_chemical_library (aka “DELs”), a way of collecting ridiculously high-throughput interaction data between proteins and small molecules. Before I discovered Leash, I had heard nothing but horror stories from DEL users. And yet, Ian and friends somehow found a way to tame the beast, extracting clean and reproducible data that is just perfect for large scale modeling. Without teasing too much, the answer comes down to epistemic rigor, and an incredible conviction that you just have to Do The Thing. It gets even better! Near the end of this episode, Ian teases something his team was just playing with when we recorded: an AI playground for agents to do verifiable scientific reasoning. They discovered the data they had spent the last few years collecting could provide tens of thousands of “lab-in-the-loop” simulations, just what an agent needs for scientific RL. Two months later, it’s matured into Minos, and today they’re releasing a full preprint https://github.com/Leash-Labs/minos-paper/blob/trunk/Minos_Leash.pdf, blog https://leashbio.substack.com/p/making-reality-replayable, and open dataset https://strelka.leash.bio/. We’ll be releasing a lightning podcast in a few days with an update. Stay tuned! We had a blast with Ian, probably one of our most fun episodes to date. For both complete chemistry novices and expert AI x Bio practitioners, we think you’ll get something out of this. Enjoy! We covered: * Why DELs traumatize everyone who touches them, and how Leash puts dozens of barcodes on each molecule to catch proteins that bind the DNA instead of the drug. * Ian’s rule that biology only scales two ways: taking pictures and sequencing DNA. * The Kaggle competition https://www.kaggle.com/competitions/leash-BELKA: about 4B measurements from a basement and about 2,000 teams, with nobody beating chance on an unseen chemical library. Also why Leash quietly feared co-folding would make them obsolete. * Clever Hans https://en.wikipedia.org/wiki/Clever_Hans in ChEMBL https://www.ebi.ac.uk/chembl/: models can guess a molecule’s chemist from its structure, and “who made it” predicts activity about as well as the structure does. * Hermes https://leashbio.substack.com/p/we-now-know-what-it-will-take, trained only on Leash’s own data, keeping pace with Boltz https://github.com/jwohlwend/boltz on public benchmarks. Plus the BRD4 molecule Boltz had already seen in the PDB. * Breaking the pocket https://leashbio.substack.com/p/taking-away-the-protein-cheat-sheet: engineering protein mutants at scale after catching Hermes memorizing “nicknames” for proteins instead of binding sites. * Lumacaftor https://en.wikipedia.org/wiki/Lumacaftor-like hits as an accidental detector for misfolded proteins. * The agent sandbox https://github.com/Leash-Labs/minos-paper/blob/trunk/Minos_Leash.pdf: LLMs that reason about SAR in plain English, hunting for the best binders among 6.5M molecules on a budget of 100. * The Recursion https://www.recursion.com/ mafia, and why its alumni companies all rhyme. * The Kosuri paradox https://x.com/srikosuri/status/2035866164325437894: if you can collect the data, why do you need the model? * Going from a DEL hit to good mouse PKPD in six months on an inflammation program, and the case for pills over antibodies on cytokine targets. * What stops a better-funded lab from outscaling them, and Ian’s by-fiat wish for massively parallel purification. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

• 0:00Speaker 00

  My co-founder Andrew Blevins and I were at Recursion looking at a small molecule on a screen. And then a medicinal chemist walked by, this guy named Mike Cucureci, and Mike was like, "Hey, looks like a Stuart Schreiber molecule." And then we were like, "Oh, no. You can tell just by walking by? Are you like who made this? That's horrifying." And if the models can identify the chemist by looking at the molecule alone, then maybe the models can infer chemist intent, which means maybe they don't learn anything at all.

• 0:26Speaker 01

  Well, they did learn something.

• 0:27Speaker 00

  They did learn something. Yeah, they don't learn physics. We downloaded all chemicals from all chemists at Kimball, which is the biggest repository of these things and their activities. Every chemist who published more than, I think it was 30 papers and 600 molecules. So you get all the molecules and all the stuff. It's 1,815 chemists. And then we trained a classifier to predict a chemist from looking at the molecule alone. And we got a top five accuracy of 60%. We chose to represent the molecules not with any chemical structures or formulas at all, but just a vector of the probability of creator. So now it's this 1,800 element in a vector, and it's got some probability of who made it. And you can use that representation and then train a model to predict activity from Kimball.

• 1:09Speaker 00

  And it does as well as if you use chemical formulas. We really think that it's learning a lot of sociology. It's not learning ground truth. When you train models on the exhaust, you might be in trouble.

• 1:22Speaker 02

  Welcome to Late in Space AI for Science Podcast. I'm RJ Hanke, CTO of Mir Omex, and I'm joined by my co-host Brandon Anderson. He develops RNA therapeutics at Atomic AI.

• 1:33Speaker 01

  Super excited to have Ian Quigley in the studio with us today. This episode is going to be a treat. Very excited. Ian is the co-founder and CEO of Leash Bio, which in my personal opinion is one of my favorite young tech bios in the last few years. You know, at a high level, Leash is combining epistemic rigor, data collection at like a crazy scale, and has a strong commitment to open science and has an incredible series of blog posts, a lot of which we'll cover today, which I think really tells you how to do machine learning in a very hard domain in kind of the right way. I think I'll have some fun anecdotes hopefully can trickle in through the episode. But yeah, to start with, thanks for being here.

• 2:17Speaker 01

  And you know, what's your, is there anything about us that you want to tell us before we get started?

• 2:22Speaker 00

  No, thanks. That was very kind. Yeah, we founded Leash historically after being confronted by a particular problem of trying to predict small molecules binding to protein targets and then finding that the data that was out there was kind of wanting. And so we went out and built this thing at a reasonable scale to kind of try to go after it.

• 2:40Speaker 01

  Yeah. So reasonable scale. You're founded upon the idea of doing something called a DNA encoded library. Every person I've ever met who's touched a Dell has come out severely traumatized. I don't know any person who has ever said, "Dells, that's a great idea. Let's do hundreds or thousands of these things." So why? What? And also, please explain what a Dell is before you get started.

• 3:02Speaker 00

  Yes. So when you're going after a small molecule drug discovery, the ultimate goal is to get a small molecule or a dinky chemical to bind to the protein that you think is causing the problem in a disease and then go from there. And some of these proteins are pretty easy to find chemical material that sticks to you, and some are harder. One way to go about this is to have a lot of different chemicals in different tubes. And you then dunk the protein in one tube at a time, and it kind of takes forever. And then a different solution that was proposed some years ago now was maybe if you had a lot of chemicals in the tube at once, and then one tube, and then you've got a small DNA barcode that's physically attached to each small molecule. You dunk your protein in the tube, kind of like a fish hook.

• 3:44Speaker 00

  You pull out different ones that bind, and then you sequence the DNA of those. And so the idea is that you get then pool this large number of molecules, and then you wouldn't have the logistical overhead of trying to go across large numbers of tubes or individual wells and plates. And so what's happened downstream of that is that you've got chemical libraries that if you do one tube at a time, the biggest ones in the world are in the low single-digit millions. With Dells, the largest one that has been produced is 40 trillion molecules. There's some problems when they get that big, but that is the kind of scale you could try to attempt with this technology.

• 4:16Speaker 01

  So the way you create these things is you use this combinatorial chemistry where you have these building blocks, and then you build the library piece by piece, but you build it combinatorially. So.

• 4:29Speaker 00

  That is correct. Yes, yes. And so to unpack that a little bit, if you've got, you know, some number of millions of molecules, what this really means is it is a combinatorial mixture of building blocks, and the number of building blocks is going to be a lot smaller than the final product of all of those. So you might be squared or cubed, right? In some different examples. So it's not uncommon for there to be maybe thousands or tens of thousands of building blocks that would then generate a chemical library of many millions of molecules. And this is restrictive in some ways because, you know, you're stuck with a smaller number of building blocks. But one thing that's really nice about it for machine learning is that you've got all these different combinations of things that look very similar to each other, except for maybe one or two changes. And so that, you know, I'm sure we're going to get into this later, but that does really

• 5:13Speaker 00

  enable comparisons so that you can identify, "Oh, it's this part of the small molecule that's driving the binding, because I see a lot of variations in this other part that's not affecting the behavior too much." And that turns out to be incredibly useful.

• 5:25Speaker 01

  The other aspect of this is how you do the readout, which is the DNA sequencing. This has been a big theme on the podcast. We've had many guests who have had some variation of, "We do something crazy, and then we convert the reading into a DNA sequencing readout." So how does this work and why is this so useful?

• 5:44Speaker 00

  Right, yeah. I would definitely say that it is a party line at Leash that if you're going to go off and try to do life science stuff and couple it with machine learning approaches, there's really only, really only one scalable technology in town, and that technology is taking pictures. Sequencing DNA is another way of taking pictures. And so it's basically pictures and DNA sequencing. And if you can shovel your challenging problem into that using one of those approaches as a readout. So like Recursion takes actual pictures of cells and a bunch of other groups do that, like Incedra does that. And then almost everyone else with the spatialomics stuff or other approaches, they'll do some mixture of pictures and sequencing, which is also pictures. Yeah, there'll be some proteomics people out there who are going to, you know, disagree

• 6:29Speaker 00

  with me, but I'd still argue the scale's not there yet.

• 6:32Speaker 01

  Hot take number one. Yeah. And one of the really cool things about sequencing is that it is dirt cheap. You can, you know, extremely scalable, very well-established technique. We're very good at it.

• 6:46Speaker 00

  It is very, very cheap. It is getting cheaper all the time. There was a little while where it kind of plateaued in terms of the price dropping, but now a bunch of new companies have started up and they're driving it down again. And so like we very much look at like the trajectory of sequencing cost dropping, and it has implications for how we think about data collection and the scale that we're doing. Yeah, it's really nice.

• 7:06Speaker 01

  That's cool.

• 7:07Speaker 02

  So you have these components that you are mixing together to form different, this combinatorial mixture of molecules, and each of those components is barcoded. Is that right?

• 7:20Speaker 00

  That's correct.

• 7:21Speaker 02

  And so they, so they kind of bind together because they're these proteins or these amino acid chains that will click onto the ends of each other and fold together, and then they either bind or don't bind to your target. So first of all, how do you know that they bound to your target?

• 7:40Speaker 00

  Yeah, so first of all, it is very much like this sort of fish hook analogy where if you were to, so right now the target, you know, is some protein or a protein complex that you're going to try to find chemical material to bind to. It's attached to a solid support. And so some folks do it a variety of different ways, but the way that we usually do it is it's attached to a magnetic bead. And so you dunk this bead in this complex mixture of all these small molecules. And in a very easy case, let's pretend that the protein has a very distinctively shaped hole in it. You dunk it in there, and then the chemicals in that pool, some tiny fraction of them have the appropriate shape and electronics that they're going to bind into this very distinctively shaped holes. Then you pull this protein back out, and you only pull out the things that are in

• 8:25Speaker 00

  that hole. You rinse it a couple of times in some way, and then you sequence the DNA of those things that come out that are in that hole.

• 8:31Speaker 02

  Okay, and then when you say sequence of DNA, you're basically amplifying the barcodes that are.

• 8:37Speaker 00

  That is correct.

• 8:38Speaker 02

  And so, and then you can, so amplifying meaning PCR or something like that?

• 8:42Speaker 00

  You do a little bit of PCR, and then, yeah, then you toss it into the sequencer. And then one thing that you're probably thinking right now is, okay, well, we need to compare it to something to get a good measure of relative enrichment. And so you do have to engineer a bunch of good negative controls to say, "Oh, well, if I just stuck the bead in there by itself." Or are there pan protein stickiness signals that are going to pull out certain chemical libraries, chemicals from the library? And so you've got to think pretty hard about all those things and make a reasonable comparison to the thing you're actually trying to get binders for.

• 9:12Speaker 02

  For the things that bind, you sort of have these combinations of barcodes that all sort of, okay, if this, this one bound, then I have like these, how many barcodes are we talking about for one, one protein?

• 9:27Speaker 00

  Great question.

• 9:28Speaker 02

  One small molecule.

• 9:29Speaker 00

  For one small molecule. So there's a couple of things. One thing, if we want to get into the kind of PCR-based weeds, and so for the audience here, if you're making lots and lots of copies of DNA, there's little subtle biases in the DNA that might make them amplify faster than others, and so you want to control for that. So you put in a little bit of unique sequence in there. So when you have lots of copies, you can then collapse it so that they all came from this one that you know very much, you know, it's very much just from this one event. And then there's a bunch of other things about barcodes that we worry about at Leash, which is that when you are trying to stick proteins in there, some of those proteins are going to stick to DNA because that's their job. Right? There's lots of proteins whose job it is to interact with the genome and cells and turn things on or turn things off or affect them in some other way.

• 10:14Speaker 00

  And then you've got this assay where you've got a ton of DNA and you're throwing them in there.

• 10:18Speaker 02

  Yeah.

• 10:19Speaker 00

  And so that can be risky.

• 10:21Speaker 02

  And that can cause false positives, right?

• 10:23Speaker 00

  Very much so.

• 10:24Speaker 02

  Yeah.

• 10:24Speaker 00

  A lot of people complain about this.

• 10:25Speaker 02

  Yeah.

• 10:26Speaker 00

  This is kind of to the earlier comment about folks having Dell nightmares. This is one of those problems. There's a variety of ways to address it. One thing that we do is we have large numbers of barcodes that each encode one particular type of small molecule. So here's one small molecule. That's great. Now we have 32 or 48 different barcodes, all encoding this one small molecule.

• 10:50Speaker 02

  Oh, I see.

• 10:51Speaker 00

  Right? And so then, you know, so like, you know, there'll be a single example, one single molecule has got this barcode and then its neighbor, which is the same chemical formula, will have a different barcode and then so on.

• 11:01Speaker 02

  So if there's any differential there, then you know that you

• 11:04Speaker 00

  That is correct.

• 11:05Speaker 02

  You're binding to the DNA instead of the

• 11:07Speaker 00

  That's right.

• 11:08Speaker 02

  The small molecule.

• 11:09Speaker 00

  And so, and so for like the, the DNA protein aficionados in the audience, one really nice thing about this is that because there's DNA in there, you can see if there's a single DNA encoding barcode that's like way binds way better than everything else, even though it's all the same small molecule. And then you can look at the sequence that you have enriched there and you can recover the described DNA binding preferences of those proteins. It's like, oh, here's a transcription factor. It binds this series of nucleotides that is very well described. You know, it's like six or eight of them. And then you look at the, you recover the DNA sequences back from this experiment. It's like, oh, great. We found the consensus binding motif.

• 11:49Speaker 02

  Right.

• 11:49Speaker 00

  So I think our protein probably got made right.

• 11:51Speaker 02

  You have this barcode that you've identified and you've kind of gotten rid of the false positives using the technique that you just said. And, and so now I know that for a given combination, so for a given small molecule that I'm in, I bind to my target and, and it's, but don't you have all these DNA tags, barcode tags, like hanging off your molecule? Doesn't that impact the

• 12:19Speaker 00

  Yes, they are hefty. They are hefty. This is another complaint that people bring up about Dells. There's a couple of different answers here. I would say, but like the, the very like 10,000 foot philosophical answer is that there are two different, or there are multiple different reasons why you might want to run one of these experiments. One reason is that you're trying to get a small molecule material that behaves a certain way that you're going to progress into drug programs. And so if that is your main goal, then you want to make sure that you're getting some kind of fidelity between having the DNA barcode on there and then having it off there. And that's true. And when we try to pursue interesting chemical material, at least, like you do have to go through the work of manufacturing the chemical off DNA, you've got to run them in these functional assays, there's a certain level of attrition for things that are easy in our hands, about

• 13:04Speaker 00

  90% of them work. For things that are hard is about 50. So there's some loss. Other people have different experiences. But I would also say that if you're thinking about doing this for machine learning training purposes, all these chemicals with their DNA barcodes are in the tube. You ran the experiment. Some of them bound and some of them didn't. And that is real signal that you were physically measuring from the world. And so then you just condition your models on the existence of this barcode and then you just keep moving on with your model training. And so in our hands, when we rerun experiments, so make fresh protein batches, make a new batch of the chemical library over many, many months of time, we get back the same hits in the same order for the most part. And so it's extremely reproducible and robust. And so in terms of like ground truth signal, we feel quite good about it.

• 13:45Speaker 02

  Okay. You could be blocking the pocket or something with your

• 13:48Speaker 00

  You could.

• 13:49Speaker 02

  So then you get a false negative there.

• 13:51Speaker 00

  But you don't because you know that the DNA is there. It did not bind. The way that you're thinking about it is like, oh, I'm thinking about how the molecule will behave missing the DNA, but we know it's there and we don't pretend that it's not.

• 14:02Speaker 02

  Oh, I see. So it, does your hit include the DNA then?

• 14:06Speaker 00

  Yeah, like when we train models, we, we put, we very much put in a placeholder that that marks like the position of the DNA.

• 14:12Speaker 02

  If you're going to synthesize this for a drug, will you leave the

• 14:18Speaker 00

  No, like I said, that's, that's where you get those attrition rates, right?

• 14:20Speaker 02

  Okay.

• 14:20Speaker 00

  Then you've got to make it and sometimes they behave differently.

• 14:23Speaker 02

  Okay.

• 14:23Speaker 00

  Not always, but sometimes.

• 14:25Speaker 02

  Got it. Got it.

• 14:26Speaker 00

  So No, no, no.

• 14:27Speaker 02

  Just needed to understand that.

• 14:29Speaker 00

  Those are really good questions.

• 14:30Speaker 02

  Those are really good.

• 14:30Speaker 00

  Yeah. Like there, there are lots of objections and like there's one, these are many of the reasons why folks aren't trying to do the thing that we're trying to do.

• 14:38Speaker 01

  Yeah, I was going to say, going back to my original statement, I, I again have never met someone who has worked on Dells and said, we should do more of these.

• 14:46Speaker 02

  Oh.

• 14:46Speaker 01

  So

• 14:48Speaker 01

  So I, I mean, I, I, I think, I mean, hopefully we will get to, I think, why I'm actually excited that this route seems to be working. You know, hopefully the audience will, will agree with me before this is over. In fact, actually maybe even a personal anecdote. We have a mutual friend who was, I think, an angel investor in Leash. And when he told me he was an angel investor in a Dell company, I jokingly said, okay, well, that's money you'll never see again. I'm like here in front of our entire audience to say, I was, I'm, I was wrong. I wish I had, I wish I had listened. I'm curious about, first of all, why are Dells so hard? I think we just covered a lot of these.

• 15:29Speaker 01

  Like what in the historical community of people doing Dells was making it so traumatic as an experience? And then why did you think you could overcome that?

• 15:40Speaker 00

  One thing I do want to say is that when Dells were invented, it was definitely at a time when sequencing was kind of more expensive. And it really felt like taking, doing sequencing experiments is kind of like a big logistical and financial swing. And then when you built the technology at the time and then pointed it at the problems that it was pointed at, it was very much, oh, we want to find new chemical material binding to targets that are very challenging. And so that brings up a couple, kind of tactical issues. One of them is that classically folks don't screen large numbers of protein targets. They're usually interested in finding chemical matter for something that's like really hard. And, and, and they have maybe tried a bunch of other techniques and they haven't worked

• 16:25Speaker 00

  out, so okay, let's try a Dell screen because we don't have anything. Then they'll go off and make the protein that they're going to try to screen and they'll put a lot of work into it because it feels like kind of a lift to do a Dell screen. And so maybe they might spend two to six months growing big batches of the protein that they care about, doing a bunch of purifications. Maybe they're going to run some functional assays to convince themselves that, oh, it's really got to be this good for us to go off and run this screen. And then when they run the screen, they're going to try to maximize their chances of getting pretty good chemical material. And so then the incentive drives into a place where we want, we want a big Dell. Like we want a lot of diversity. And so if you go looking at folks who are offering CRO services or internal Dell groups, a lot of times they'll push into these very large Dells in hopes.

• 17:09Speaker 00

  And that, and that's their motivation and it makes a lot of sense. What we have done that's very, very different is we, you know, we screen tons and tons and tons of proteins and we are less driven by trying to find individual binders for a particular protein target that might be of high interest for a program. We just want to get tons and tons of, you know, active chemical material across large numbers of proteins. And so there we can identify how our Dells behave very, very well, because every one of our chemical libraries gets screened against many hundreds of targets.

• 17:41Speaker 02

  So you're characterizing the

• 17:43Speaker 00

  The chemical library.

• 17:43Speaker 02

  The chemical library better.

• 17:45Speaker 00

  We know very, very well. And so we started to encounter lots of these motifs that are quirks of the chemistry that we designed or maybe the manufacturing process. Like, oh, well, even, I mean, a great example is we have one chemical library that we've had made a couple of times. And even though it's the same chemical library, it does behave differently. And we did not know that until we screened hundreds of proteins against it. And then you have to stare at those results and say, oh, well, now we're seeing these motifs show up a little bit more. And then as you characterize the library more and more, it really helps you evaluate and troubleshoot even your protein production process. And so the one reason that we felt that this is pretty tractable overall is that I and a couple of people at the company come from real like genomics methods development backgrounds as compared to chemistry backgrounds.

• 18:30Speaker 00

  And so we knew what it might take to tweak experimental protocols to really get good signal out of these funky assays. And I would say it was that plus many years at recursion, that's where I was last with my co-founder, this guy Andrew Blevins, and a number of other people on our team. And we all made, you know, honestly, like a lot of mistakes in high dimensional biology, bumping around in the dark, figuring out what different modes, failure modes there might have been that would allow models to cheat on, you know, the high dimensional imaging data that recursion produces. And so we had this real eye for thinking about, oh, well, you know, there's these sources of noise this way, we're going to have to chase those down. And oh, can you predict, you know, the instrument that this was run on? And if you can, that's really bad. And so we incorporated all that thinking into going after this particular assay.

• 19:13Speaker 00

  And we had a feeling that folks hadn't really stared at it, you know, in that amount of detail before.

• 19:19Speaker 01

  Yeah, this goes back to the Ron Alpha Noetic episode where I found it.

• 19:24Speaker 02

  Recursion Mafia?

• 19:25Speaker 01

  Yeah, Recursion Mafia. I found there's some real spiritual parallels between both of your companies, this like real epistemic rigor, checking very carefully everything you're doing, doing things in different ways and sort of like semi-overlapping ways. And I'm just curious, we asked Ron this, yeah, very data focused, like at scale.

• 19:46Speaker 02

  Automation.

• 19:47Speaker 01

  And yeah, and I'm curious, like, was there something like at Recursion, like, was this just like quarter Recursion's culture that is sort of?

• 19:55Speaker 00

  It is, yes, yes, especially in the early years. I would say that there are four that I, I think, I think there's four ex like Recursion Mafia companies and they all kind of rhyme. So there's us, there's Noetic with Ron Alpha, there's Invada with Vishwa Kaluru and August Allen and Hannah Gordon and all these great people, Dave Healy. And then there's also Known Medicine that was run by Kitty Rieskele that got picked up by PETHUS a couple of years ago. And so all these companies are basically looking at different interesting biology problems with extremely noisy measurements. And then they pulled in people from Recursion to figure out like, oh, we're pretty good at noisy measurements. Let's try to see if we can get consign a lot of this particular corner. You know, so

• 20:36Speaker 00

  for us, it's a small molecule chemistry problem. For Invada, it's, it's mixtures of natural products. They use, they do do a lot of like, you know, kind of mass spec kind of stuff. And then there's Ron and then there's Kitty Ries's group. So yeah, we're all very similar.

• 20:51Speaker 02

  So let me, let's pause for just a second. What, what do you guys actually do?

• 20:58Speaker 01

  That's great. Sorry, I like, I was so excited about this. I just jumped in.

• 21:03Speaker 00

  Yeah, we, we are a machine learning small molecule company. And so what that means is we go off and we collect massive numbers of small molecules binding to protein targets. And we use that to drive advancements and machine learning, understanding of this kind of stuff. And we also push, you know, small molecule material into partners and move it forward towards, you know, towards patients.

• 21:27Speaker 02

  Just for the audience that's not familiar, what, what is a small molecule? Why do we care? What, how is it different than other therapeutic?

• 21:34Speaker 00

  This is great. This is great. So when you're trying to drug something, I talked a little bit earlier about how you've got a protein of interest in your body and you're going to try to interact, affect its behavior in some way. There are a couple of different things that are currently on the market that people use. One thing that's common is antibodies, which are made by cells, they're large molecules. And by large, they're, they're big enough that they have a really hard time getting inside cells. They also have a really hard time being in a pill. So if you have like antibody treatment or one of these protein treatments like GLP, like GLP-1 treatments, you're going to have to have kind of an ejection. It can affect certain proteins that are easy for them to get to. But there's other ones inside the cell that drive a lot of disease that are, that are quite challenging for

• 22:19Speaker 00

  those. It is kind of more appealing in a lot of ways to instead have this big thing that's a protein to have a little tiny chemical. And tiny chemicals, small molecules here are things that can be in pills, they can go inside your cells and affect behaviors. They do make up most of drugs. If you've got any pills in your house, they're almost certainly small molecules. And of the different machine learning drug discovery groups that are going after treatments in a general sense, we have made, we as in all of, you know, life scientists, you know, in the 24th century, have made pretty good progress on being able to predict and enable the behaviors of protein type drugs. So you've got a bunch of antibody companies like Isomorphic does this, or like Chai or Novla, there's all these groups of

• 23:02Speaker 00

  very good, very smart people pushing on these particular problems. And I think part of that is because we understand or have enough data for, for protein behaviors that make it reasonably effective, you know, for these companies to do these designs. Our thesis is that those capabilities are really not here for small molecule approaches. And so that was one reason why we built this company is we felt that we could go after it in a meaningful way.

• 23:24Speaker 02

  And small molecules, one of the key characteristics, they go everywhere, right? In your body. So that's good and bad.

• 23:31Speaker 00

  Yep.

• 23:31Speaker 02

  That is good and bad. They, they can penetrate into the cell and they can also get through defenses of tumors and cross the blood brain barrier more easily. But at the same time, they do all those things and therefore they can be more prone to toxicity and other side effects.

• 23:47Speaker 00

  Very true. So the, the kind of path through the regulatory barriers for small molecules can be a little bit more fraught than those for antibodies. And so antibodies, we know pretty well right now about how they behave in the body and what they're likely to cause problems thing and what they're not. And for small molecules, there's a bunch of well-described problems that some of them might affect. They might cause liver problems, for example, they might cause heart arrhythmias. And as a medicinal chemist, a person whose job it is to develop these molecules, to get them from something that looks okay in a tube to something that you would feel confident enough to give to a human being, there are a lot of these pitfalls and they have to do a lot of work to avoid these pitfalls. So make sure that my molecule doesn't cause heart arrhythmias. And then if I change it

• 24:32Speaker 00

  to stop the, causing those problems, does it affect it doing the thing that I want it to do?

• 24:37Speaker 02

  How do your models help with this larger problem of developing these and then, you know, maybe with, you know, sort of these downstream problems?

• 24:45Speaker 00

  So one way to think about it is that a hand, some number of these problems, these sort of off-target toxicities are going to be driven by your small molecule binding to other proteins in ways that you might be able to predict or identify. And so that's something that we are very much pushing forward to is, oh, let's also make those proteins. Let's also screen those proteins with our chemistry.

• 25:05Speaker 02

  It seems useful if you have a giant assay of

• 25:08Speaker 00

  That's right.

• 25:09Speaker 02

  Different proteins and a library that you can.

• 25:11Speaker 00

  Yeah, it is, it is not at all an exaggeration to say that it is our aspiration to have screened every protein and then also move on to those and outside of humans, both for reasons to get variation in the proteins, but also to identify, maybe you don't want to run a mouse model, maybe you should run a rat one, because your chemical material behaves differently in those different species, because those proteins might be slightly different.

• 25:32Speaker 02

  Interesting.

• 25:32Speaker 00

  Right. So you want to do that stuff. But yes, going after large numbers of these things allows you to think, all right, well, now I've got my chemical material and then I know of all these chemicals, which ones and what variations or ones likely to bind these things will cause other toxicity problems.

• 25:46Speaker 02

  Got it.

• 25:46Speaker 00

  Yep. Yep. And then downstream of that, there's a bunch of other assays that lots of people do that are more kind of emergent behaviors, where it's maybe not a single protein, but it's how well it gets across cell membranes, how it interacts in a more complex way that you require a cell to evaluate, or it might be like a cardiac cell, for example, for this heart arrhythmia stuff. And so those things, you know, right now, there are lots of groups that are pushing on technologies to measure these at the cellular level, which is, you know, one level of biological abstraction away from what we're focusing on right now. But we like to think that models that have a good understanding of the biophysics themselves can be useful and maybe fine tuned on smaller data sets in the future.

• 26:22Speaker 02

  Popping back one more, maybe in, maybe getting a little into the model. Why, why Dells?

• 26:28Speaker 00

  Dells really are, at least in our view, the most scalable technology for measuring small molecule interactions because you can put many millions of them in a tube. Now, one thing that, that we haven't touched on in terms of like these chemical libraries is that when you make them in a pooled process, there are probably going to be some that react imperfectly and that the label translation of what you think you have might be, it might be a little bit off. And so we've designed a lot of controls into constructing our chemical libraries to mitigate this. When you're using these, you know, the word might be arrayed libraries or libraries that are in one chemical at a time at a tube or in a well, in a plate.

• 27:08Speaker 00

  When folks study to figure out how good those labels are, like you hope that they're good, but when folks examine this, they're typically off by 20 to 30%. It's an enormous mislabeling problem. We think it's probably comparable to mislabeling and Dells maybe worse. And Dells themselves are just enormously, enormously scalable. And maybe I can, like, I'll tell a quick story that I'm sure that we'll get into at some point. Anyway, we ran a Kaggle competition a couple of years ago, and we decided to generate the data for the Kaggle competition and we released it. There's existing, there's a database that was at the time, two years ago called Binding DP, which was the biggest aggregated collection of small molecules binding to protein targets as compared to your cells or something else. It was about two, a little under 3 million physical interactions. And so we released

• 27:53Speaker 00

  a data set that was about 4 billion physical interactions and it was all measured by a single person in the basement of my house.

• 28:01Speaker 01

  Yeah, the early blog posts are really funny. You like literal picture, you have an act. It's not quite a garage startup, but it's in a basement.

• 28:08Speaker 00

  It is a basement apartment with like a seven foot ceiling. Yeah.

• 28:12Speaker 01

  It looks claustrophobic. Yeah, I've never seen a biotech startup started in a basement.

• 28:18Speaker 00

  Oh, me done.

• 28:19Speaker 02

  Christopher ones are like that, right?

• 28:21Speaker 01

  Okay, so the so this Kaggle competition, I think, is just a fascinating story because you had such high hopes going into this.

• 28:29Speaker 00

  We did.

• 28:31Speaker 01

  You were like, and I think this is one of those cases of like very nice episodic rigor, where you didn't just provide, oh, like, you know, here's a train test blow with random split or something. You you thought carefully about how to organize this Kaggle competition, how to run it and how to kind of evaluate fairly. Can you talk about that process, the decisions you made, and then also the result?

• 28:54Speaker 00

  Yes, happy to do that. So we were not by a long shot, the first group to try to evaluate whether you could use computers to make reasonable predictions of small molecules binding to chemicals and then test those predictions. There are lots of papers that are doing this, like this is a real classic paper that came out of X chem and and Google some years before before we founded Leash that that does this. We had run a lot of our own experiments on public data and we had suspected that no one was really very good at it. And so we wanted to put together this contest and and design the evals in particular in a way that could really test it across a range of eval difficulties where because we were thinking about, okay, we're going to have a lot of people, hopefully a lot of people compete in this, some are going to be very good, some are probably going to be beginners.

• 29:39Speaker 00

  We want to make sure that there are some nice, easy wins, then like a bunch of stuff in the middle that's like kind of hard, and then some things at the very at the very top end of what would be difficulty or what we imagine difficulty might be. And so to kind of unpack that, we've talked a little bit already about how these libraries are combinatorial. So you're going to have some smaller number of building blocks, but a large number of combinations. We got our hands on a chemical library from a wonderful sponsor, this guy named Shaoji Liu, who runs a shop called Alpha Ma in Shanghai. And the chemical library was about 130 million molecules. And the center part of the molecule, you can imagine it kind of like a hub with three spokes. And then the center hub was the same for every molecule. It's a it's a particular chemical motif called a triazine.

• 30:23Speaker 00

  And so what we did was we said, okay, so we got the center hub, then we got these three spokes for the evaluation data set. Let's have a couple of different ways to split this. One way is that we could say, all right, the chemicals might have one shared building block, one shared spoke, and then the other two spokes are different. Okay, that's one thing we can do. So let's do that for some, and that was the easy stuff. Another one we can say is, all right, well, maybe we're not going to have any shared building blocks, but we do know that if they're going to come from this chemical library, there's going to be the shared hub. And so we can imagine that some of the spokes might be similar, but they won't be exactly the same, and they won't be shared. So we think this is going to be a little bit harder. Sounds great. And then for the third most difficult eval, we had a completely different chemical library that we had designed at Leash.

• 31:07Speaker 00

  There was no common hub, no common building blocks, different attachment chemistries, different everything. And so we thought, okay, we're going to do, we're going to have these three in different proportions. We're not going to tell anyone that's what we did. We're just going to say here that evals and let's go see how it goes. And so we launched the competition. It was two years ago, and you guys can look it up if you want. We've written a lot about it. And then after 2,000 teams competed, the winner did, you know, okay, on the stuff that where there were shared building blocks. Depending on the target, we saw some boosts in performance for where there were no shared building blocks, but the core hub was still the same, the triazine was still there. And then when you go to the most challenging eval, no one did better than chance.

• 31:46Speaker 01

  Yeah, the like winner was like the like 1,200 just random person who submitted a Jupiter notebook, right?

• 31:54Speaker 00

  Yeah, so that so the winner was this guy, Victor Shlapov, who put together a slightly different approach, but and his was good, but again, like everyone else, no better than chance. Three of the top five were the noisy tutorial notebook.

• 32:06Speaker 02

  So but just to be clear, there usually with Kaggle competition, there's like a a public eval for you to train to to test your algorithm, I guess. And then there's the the private one that's held out.

• 32:17Speaker 00

  That's right.

• 32:18Speaker 02

  And the competition runs that and all you know is the score.

• 32:20Speaker 00

  Yep.

• 32:21Speaker 02

  So was your library included in the public eval?

• 32:24Speaker 00

  Okay. It was it was hidden back to be to make it spicy.

• 32:28Speaker 02

  Right, okay.

• 32:28Speaker 00

  Yeah.

• 32:28Speaker 02

  And it was spicy.

• 32:29Speaker 00

  It was very spicy.

• 32:31Speaker 01

  So so what did you take from this? What was the lesson? And I mean, first of all, what did you hope to get from this? I kind of touched about that. But then how did that sort of affect your thinking about like, how do you proceed forward?

• 32:42Speaker 00

  Yeah, yeah, we had a couple of hopes and some and some were fears. One thing I will be candid about is, you know, we had released this data set that was 1,000 times bigger than anything anyone had ever seen. And we gave it about a 5% chance that recent advancements and kind of like coal folding approaches would mean that it would obviate our company entirely.

• 33:01Speaker 01

  Like, didn't happen.

• 33:02Speaker 00

  No, didn't happen, no. But we were like, you know, there's like a small chance, like we are really messing up here. And that didn't happen. We suspected more likely that folks weren't very good at it and that the data might help. And so we were hoping to see that. One thing that

• 33:16Speaker 02

  Anyway, you'd want to know that.

• 33:17Speaker 00

  You'd want to know that.

• 33:18Speaker 02

  Before he spends 10 years of your life.

• 33:20Speaker 00

  Absolutely. And like another thing that we wanted to know is there's lots of different ways to represent molecules and give them to computers. You could use these things called smiles that do it some way. You could do 3D graphs. There's like weird spherical like spherical convolutional nets. Like people have tried kind of goofy things to try to represent molecules in a useful way. And there'll be lots of people in the field who will argue that like, oh, I think this way is better than this way or whatever. And we did really encourage competitors, especially those who had a chemistry background, like, let's try like some of these different representations. So let's see how it goes. And we were grateful to find that like the thing that we were doing, which are just basic smiles, kind of beat everything else. Like, it wasn't, it wasn't like a resounding thumping, but it was better enough for us to continue to using that approach.

• 34:02Speaker 02

  And smiles, just if I recall correctly, is just a basically a sequence, a textual sequence of strings. It doesn't, it doesn't encode any 3D structure.

• 34:11Speaker 00

  It does not.

• 34:12Speaker 02

  It's just, it's all emergent.

• 34:15Speaker 00

  It is a graph traversal.

• 34:16Speaker 02

  Yeah.

• 34:16Speaker 00

  Translated into a text string. And so you can start the graph at different points. And so there's what they call canonical and non-canonical smiles, depending on where you start. But yeah, it's an extremely simple representation. And so, yeah, we were thinking that, you know, or, you know, we wondered if if some of these fancier ways of representing molecules might beat at least this particular problem, and they super didn't.

• 34:35Speaker 01

  So was your takeaway that you should go full force into this? Was this encouraging to you or was this discouraging you to do like, what was the what was the collective emotional reaction at Leash after that? I always, I've always wondered that. I mean, you kind of wrote about, you had several follow ups, but I I'm just curious to hear.

• 34:54Speaker 00

  Yeah, yeah, this is a great question. So like, we, we are extremely bitter lesson pilled over here. And so our thought was, some of it was kind of delight, which is like, wow, this is much harder than we thought. That's kind of great.

• 35:09Speaker 00

  That's kind of great.

• 35:10Speaker 02

  Yeah.

• 35:10Speaker 00

  Because if this problem is like all the other ones that we're aware of, if you can get pretty good signal out of a scalable measurement regime, then you should just try to go get that.

• 35:18Speaker 02

  Yeah.

• 35:19Speaker 00

  Yeah, no, we, I mean, for us, it was, it was pretty validating, like, oh, man, like, we, this is an opportunity here.

• 35:24Speaker 02

  Especially if your scalable measurement regime is hard and you have specialized knowledge to do that.

• 35:30Speaker 00

  Absolutely, absolutely. It was like, oh, man, like, this is the time to like pour gas on this. This is great.

• 35:36Speaker 01

  Wow, yeah, I mean, that's awesome. So then you were just like, okay, let's full force. We're going to just start collecting data, collecting data, collecting data.

• 35:43Speaker 00

  Correct.

• 35:44Speaker 01

  Yeah. And so you did that for a while. And then what came, what came next? And maybe first, also, what were the lessons you learned in the process? And then building to kind of your, your first useful, like?

• 35:58Speaker 00

  Yeah, yeah. There are a couple of things that were from the process that we felt were very, was very, very useful. One thing that we realized is that the competition was based on three protein targets that were fairly well studied. It became clear to us that we really wanted to, you know, for lots of reasons, but including these, that, you know, getting performance across novel proteins probably meant that we'd have to do a whole lot more. So, okay, so that, you know, we were always motivated to do that, but that really underscored like, oh, man, we're going to have to go, we're going to have to make a lot of proteins, aren't we here? And so that was part of it. Part of it was thinking that we had this very large number of small molecules, like this 100 million compound library, that didn't help as much to us as better characterizing a smaller one. And so that was also like a strategic choice that we made.

• 36:43Speaker 01

  This is actually quite important because you have a fixed budget. And so you can either, let's say, sequence or sequence, you spend more money on sequencing, getting higher resolution for what you already have for a smaller set, or you can get more molecules and then hope that the noise sort of averages out or something.

• 37:00Speaker 00

  Yeah, that's correct. Like, having a fixed budget really does affect like your strategic choices. And for us to, we wanted to make sure that we're getting a lot of variation across both axes as compared to like one axis and like a handful of proteins.

• 37:11Speaker 02

  Sorry, which two axes?

• 37:12Speaker 00

  So that would be like the number of chemicals against the number of proteins.

• 37:15Speaker 02

  Okay.

• 37:15Speaker 00

  Right? And so, you know, you can imagine like, oh, I'm going to do three proteins in a billion chemicals, or maybe I should do like thousands and single digit millions.

• 37:24Speaker 02

  Yeah.

• 37:24Speaker 00

  You know, or some, some other combination of those.

• 37:26Speaker 01

  Single digit millions being probably one of the tiniest cells ever constructed.

• 37:29Speaker 00

  It's pretty small. The smallest one I've seen is 50,000 molecules.

• 37:32Speaker 01

  Oh, okay.

• 37:32Speaker 00

  Yeah.

• 37:33Speaker 01

  I guess you can make a custom Dell if you want to do like SAR.

• 37:36Speaker 00

  You can do it every one.

• 37:38Speaker 01

  Yeah, it's a, when you have the technology, just do it.

• 37:40Speaker 00

  Yeah.

• 37:40Speaker 01

  Two building blocks. Yeah.

• 37:42Speaker 00

  Yeah, right. You can do whatever you want. So that's the nice thing is you have enormous control over, over like, you know, both like the library size and the characteristics and like this barcode issue that I talked about, you can expand that kind of stuff. And we've put a lot of thought into, you know, kind of bells and whistles in our own Dells to, to really boost, you know, kind of like a performance and signal over noise and all this stuff. But getting back to the, kind of like the Kaggle competition lessons, you know, was also clear to us that models could think about memorizing individual building blocks and that would boost their performance. That was something that seemed pretty clear. It seemed pretty clear that they might be able to get an idea through, probably through memorization, that if you have a very similar building block, it might behave similarly.

• 38:23Speaker 00

  So that also some of those performance metrics, and then also that, you know, when I said that it's pretty far away, they, I mean, at the current regime, at that regime, they simply could not do it. And when we released these results, basically saying, hey, you know, no one did better than chance, we did get some blowback where folks were like, well, did anyone good compete, you know, in your Dell competition? Like, maybe everyone's just bad. Maybe it's all first year grad students.

• 38:47Speaker 00

  And so as a result, we just released all of the data in entirety. And so if anyone out there thinks they're awesome, then like, please show us, because like, we want this problem solved. But we haven't seen anyone do great, you know, perform great on that stuff yet.

• 38:58Speaker 01

  And when you say all the data, you have, including all like the raw read counts and everything.

• 39:03Speaker 00

  That's true.

• 39:03Speaker 01

  So, so you can go through and say, oh, it's not just, you know, a processing error. Like, you can go from the raw data and use it.

• 39:09Speaker 00

  You could take, yeah, you could take our binary labels, you could do it yourself, all the evals, everything. It is, it is all on Polaris, which is like a nonprofit group that was run by Recursion. It's, it's there for anyone to use. Please use it, you know, like, we intentionally released it as, as, you know, hoping the community would, would make good use of it.

• 39:25Speaker 01

  Do you think, though, the structure of this dataset was fundamentally limited in a way that maybe without some external augmentation, you're not going to be able to do any better than what people have pretty much already done?

• 39:35Speaker 00

  I think if you had really perfect structural understanding or like really good structural models, then you ought to be able to do okay on this. Right? Because if we, we all have models that really understand, truly understand, like, the physics of the interactions between the amino acids and the small molecules, then it doesn't really matter what the distribution of the small molecule training set would look like, if they're really good at this, you know? And I think when we see, when we see structural model predictions and folks say, oh, well, you know, I've got this time split on the PDB, for example. And, you know, you know, our model's never seen these molecules or this protein before, and then hopefully we can, you know, kind of use this understanding to make reasonable folds. Like, that's, that's very much in the spirit of being able to make the predictions on this stuff, because no one has seen these molecules, especially our internal molecules, like, no one's seen those before.

• 40:15Speaker 01

  That's cool. So Kaggle competition, you went ahead and you learned, you learned some lessons, you went full force data collecting, regrouped, committed to scaling.

• 40:25Speaker 00

  Yep.

• 40:26Speaker 01

  All right. So then you, I think something like six months later, you released, or you had a blog post about your model Hermes.

• 40:34Speaker 00

  Yep.

• 40:34Speaker 01

  Which is, yeah, can you tell us about that?

• 40:37Speaker 00

  Sure.

• 40:38Speaker 01

  Some of the learnings there, and I think both, and especially how you evaluated that, I think, and all the care you, which went into that model.

• 40:46Speaker 00

  Hermes is a model that is the original brainchild of a data scientist at Lich named Max Kleinsasser, Max Kleinsasser, and also Sean Francis Lyon, and Andrew Blevins, of course, is the CTO there. What the thinking was originally was that we had a bunch of data and we were thinking about how to split it and different like training regimes. And so we wanted to come up with a model that was really, really dead simple, so that we could do lots of potential experimental iterations. It wasn't, the purpose of it wasn't to be a state of the art model. It was like, ah, you know, we want to try stuff fast. And so we built Hermes, and then the basic architecture of Hermes is two different embeddings. One of them is evolutionary scale modeling, yes, embedding, that's a protein embedding that comes from a group that was at Meta and then went off to start their own company. And then they got, and they got picked up by Chen Sucker for more recently.

• 41:31Speaker 01

  And so we had Alex Reeves on the podcast.

• 41:33Speaker 00

  Yeah.

• 41:34Speaker 01

  Throwback to that episode. If you want to hear more, go back a few weeks.

• 41:38Speaker 00

  Lots of detail. Thank you, Alex, if you're listening. So we did that for the protein side. And then we had a different embedding called Kemberda, which is basically like a next token predictor chemistry, kind of embedding to this side.

• 41:49Speaker 01

  Something which predicts smiles or?

• 41:50Speaker 00

  That's right. Yeah. Just predicts, yeah, smiles tokens. And so that's these very, very two simple embeddings. And then we just put them together with cross attention, and that was it. And then we used it to predict a binary label.

• 42:00Speaker 01

  Simplest possible thing.

• 42:00Speaker 00

  It is incredibly simple, very, very small. And then we trained some of our own data on it.

• 42:05Speaker 02

  Actually, you know, not the simplest. You could have done a MLP after those, right, instead of cross attention? So why attention?

• 42:13Speaker 00

  The thinking with the attention was, had to do with some other experiences that we'd had, kind of with graph convolutional, like graph models for protein structure. And one thing that we'd noticed is if you've got these sort of graphs and there's a couple of models that do this, and other work that we had done, we had scrambled the positional coordinates of those. And performance for some of those models more or less stayed the same. And that made us think that like attention was super critical. And so we ended up just making sure that the model was at least, at least in our, at least what we hoped, is that it would be looking at the different parts of the proteins, the small molecule structures that could be useful here.

• 42:47Speaker 01

  So this is, I think, oftentimes known as the Picasso problem in like image generation, where, let's say, CNNs, for example, would, were famous for a while, where you could scramble just the features of a picture, like Picasso, like flip the nose around or just like rotate the eyes or something. And the CNN would still depict, oh, this is a Picasso, because it just picked up on individual local features rather than some sort of global structure. And this has been a problem in small molecule drug discovery for a while, where oftentimes it picks up on local motifs and then it doesn't have any global understanding of the topology.

• 43:22Speaker 00

  Yep.

• 43:23Speaker 01

  So especially for something like a Dell, where you have this combinatorial structure of individual building blocks, you could imagine the model learning each building block independently. And then essentially just decomposing into the sum of three building blocks or something. Or, I mean, is a simple concept.

• 43:38Speaker 00

  Yeah.

• 43:38Speaker 01

  Yeah. Yeah. So cross attention, in some sense, forces certain molecules to interact with like certain residues in our protein.

• 43:45Speaker 00

  Yep.

• 43:46Speaker 01

  Or at least maybe not forces, but provides the opportunity.

• 43:49Speaker 00

  Thinking about it.

• 43:50Speaker 01

  It's, it's an addictive bias. Yeah.

• 43:51Speaker 00

  Yeah. Kind of what you're getting at is, you know, we were thinking about how to evaluate this. And so we had done some of the similar tricks that we did over with the Kaggle competition, where we had, you know, training sets on data that we generated internally. And then we'd looked at chemical libraries that were further away. But then also we wanted to think about splits to kind of probe Hermes behavior. And so because Dells have this combinatorial approach, you could do the same thing where you might share certain building blocks. And so we do that. And of course, performance is better, which we suspect is from memorization. But the other thing is you can imagine that there are building blocks that are very similar to one another, that if you're just trying to split on the identity of those, and then you might have things that are like, are close enough that that memorization, memorization or some other, you know, cheap mode might, you know, confer predictive power.

• 44:36Speaker 00

  And so we came up with a scheme where we then did basically clustering of these different building blocks. We had building block clusters on similarity, and that way we would just avoid entire clusters from being overlapped between test and train. And so when you do that, then, you know, the performance does drop. And so one thing that you might be alluding to is we have, we use lots of, lots of reasonable baselines here that we're really worried about. And so for almost everything that we do, we'll use like XGBoost, right? You should do something like that, just to see how it's going.

• 45:09Speaker 01

  Yeah, yeah, yeah. Every, the classic young machine learning practitioner, they spend six months optimizing their favorite model, and then someone walks by and does like, you know,

• 45:19Speaker 00

  XGBoost.

• 45:19Speaker 01

  XGBoost or random forest, and they're like, oh, yeah, yeah, that does beat like six months of work. Oh, yeah.

• 45:26Speaker 00

  Yeah. And so we, and so we did these different like split regimes to try to evaluate how well things would do relative to like XGBoost's performance. And so one thing that was really gratifying to see is that you see XGBoost drop steadily as you get further and further away with these building block cluster approaches. It's like if you share two building blocks, it's a certain level. If you only share one, it gets worse. If you share one, you know, only none, it's worse. And if you do the building block cluster, it's even worse than that. And then we compared that to the Hermes performance, which was pretty steady across all these different regimes.

• 45:58Speaker 02

  I see.

• 45:58Speaker 00

  Yeah. So it made us think like, oh, yeah, you know, you've really got to get pretty far out of distribution for XGBoost to really tank. Yeah. And that's something that we haven't, you know, we haven't, it's in a couple of plots and some work that we published, we haven't yelled about it too much.

• 46:10Speaker 01

  Wait, I thought you, you didn't have a blog post like XGBoost is all you?

• 46:13Speaker 00

  Yeah, yeah, we did do that. Yeah, that's where we did that. Yeah. Running those splits is something that we've put a lot of thought into, into trying to figure out, like, okay, how do you stop your models from really, you know, memorizing and remembering things?

• 46:24Speaker 01

  This might be a good time for a little bit of a detour, but referencing another one of your blog posts, which I found absolutely hilarious from my own personal experience. I have seen machine learning models fail, like, pick up on so many weird detractors. And your one, your blog post about molecule ownership or authorship was a, I think, a really fun story.

• 46:45Speaker 00

  The clever Hans post, yeah. So many years ago, my co-founder Andrew Blevins and I were at Recursion looking at a small molecule on a screen. And then a medicinal chemist walked by, this guy, this guy named Mike Cucarice, and Mike was like, hey, looks like a Stuart Schreiber molecule. And then we were like, oh, no, you can tell just by walking by? Are you like, who made this? That's horrifying. And then that haunted Andrew in particular for like many years, like it was like six or seven years. And we joked, you know, I don't know, every couple of months we're out for beers, like, man, we really should figure out like how bad this problem is.

• 47:19Speaker 02

  Why is that a problem?

• 47:21Speaker 00

  Okay. So it's a problem where it's very easy to imagine you're going to try to train models to understand chemistry. So you're going to get the data from lots of groups, because it's very tempting to get data from lots of groups. And if the models can identify the chemist by looking at the molecule alone. Then maybe the models can infer chemist intent, which means maybe they don't learn anything at all.

• 47:44Speaker 01

  Well, they did learn something.

• 47:46Speaker 00

  Yeah, they don't learn physics, right? They're not learning the thing they're trying to get to learn. And so we were really afraid of this.

• 47:51Speaker 01

  They learned that certain chemists are better than other ones.

• 47:53Speaker 00

  Yeah, we learned all kinds of stuff, but maybe to set this up for the audience. So we went out and like we published this over Christmas. You could look it up. We have a leaderboard. We have a goofy online game you can play. But we downloaded all chemicals from all chemists at Kimball, which is the biggest repository of these things and their activities. Every chemist who published more than, I think it was 30 papers and 600 molecules. So you get on molecules and all the stuff. It's 1815 chemists. And then we trained a classifier to predict a chemist from looking at the molecule alone. And we got a top five accuracy of 60%. It's a very, very easy problem.

• 48:30Speaker 01

  Yeah.

• 48:30Speaker 00

  Yeah, it's super easy for the models to do. Where they get confused is where you have two chemists who were on the same publications and were at the same institution. And so we think it's very likely is that there is a distinctive chemical signature style that individuals have when they make molecules. And then when those individuals have students or trainees, then they adopt some fraction of that, you know? And so that was, that seemed pretty, pretty clear to us. And so that was horrifying enough up on its own.

• 48:58Speaker 01

  And this is a data set that many, many academic projects, many PhDs have trained on, tried to predict on, find patterns. And this is really like a major data set for the field.

• 49:12Speaker 00

  Yeah, like if you are not generating your own data, you're using this. And then if you are generating your own data, then you are probably victim to this because your organization probably has multiple chemists who have their own distinctive chemical styles and that they are using their styles to go after different types of projects.

• 49:28Speaker 02

  But the key here, I think, is that those chemists also tend to solve similar problems. Like I might have a career here and I have a style, but I also am working on a class of problems.

• 49:42Speaker 00

  Right.

• 49:43Speaker 02

  And so that your, your style is also, you're learning the style instead of the problem.

• 49:48Speaker 00

  You're, you're learning the style, you're learning the sociology of the problem. And the other thing that you're learning is that when you're a chemist at these organizations, you know, ordering a molecule to be manufactured and tested is kind of a pain. Right? So what will happen is you will make many, many choices of things you will not pursue and you do not write those down. Oh yeah, it would be dumb to do that. And it's like, yeah, that's their motivation.

• 50:11Speaker 01

  Like, yeah. Our team couldn't just, just wouldn't make it. Like I could say, like, we could do this, but then they would, they would cost too much. Or the, the lab team will say, the chemistry team will say, no, we can't make it now. Or like, we don't have the building blocks on.

• 50:23Speaker 00

  I don't think it's going to work. Like, there's all these reasons why you might not make a choice. And then not making that choice is a fingerprint that, that is going to land on what does get made. And then, you know, when you talk to some of this little chemists, some of them are, you know, bless them, they're superstitious people. And some of them will say like, oh.

• 50:43Speaker 01

  Yeah, I bring this up like every single time. Medicinal chemists are some of the most superstitious people I've ever met.

• 50:48Speaker 00

  Right. And so, and so it's like, oh, well, this building block worked in this, in this program that made it to the clinic. I'm really fond of it. Maybe it helps a side building, so I'm going to put it in things that I like. And so then your model will learn like, oh, the chemist likes this because they put, so that means it's probably working. And so it learns all this weird stuff. And so like, we were very afraid of this. And so we took, you know, we ran this, this, uh, this model to identify who the chemist was. And then we chose to represent the molecules, not with any chemical structures or formulas at all, but just a vector of the probability of creator. So now it's this 1800 element, you know, vector, and it's got some probability of who made it. And you can use that representation and then train a model to predict activity from Kimball.

• 51:33Speaker 00

  And it does as well as if you use chemical formulas. And if you combine them.

• 51:39Speaker 02

  You're quoting the sociology only.

• 51:40Speaker 00

  Yes. If you combine them, it doesn't, you're meaningfully better. And so, yeah, we, we are, you know, we really think that it's learning a lot of sociology. It's not learning ground truth. It's just learning like the choices and the fingerprints of the people who were trying to develop these, these molecules. And it's not a knock on them, right? I mean, that's, that's not what they're trying to do. They're trying to, you know, get to drugs by making the fewest number of chemical series possible. Of course they're going to do it like that. But when you train models on the, on the exhaust, you might be in trouble.

• 52:05Speaker 02

  So this means that this is not a great machine learning training set.

• 52:09Speaker 00

  We do not believe so.

• 52:12Speaker 01

  It's funny, in my, my own, in my own life, you know, I've had many of these similar lessons. It's well known that things like molecular weight will just, you know, all you have to do is sort by molecular weight. That's like one of the single, single biggest predictors you can possibly have. Like, wasn't it CAS 16?

• 52:26Speaker 02

  It was CAS 16 where it was tied for first place.

• 52:29Speaker 01

  Yeah.

• 52:29Speaker 02

  For second place, like it was really close.

• 52:31Speaker 01

  It was really close.

• 52:32Speaker 02

  Yeah.

• 52:32Speaker 01

  Yeah. So all these people pulling out their fancy machine learning models, you just sort the molecules by heaviest molecules, better binders. Yeah. Yeah. Yeah. So.

• 52:41Speaker 02

  Yeah. But this is not an intrinsic property to chemistry. This is a property of the data set.

• 52:47Speaker 01

  Yes, yes. This is a property of the sociology of what gets made, what gets tested.

• 52:51Speaker 00

  Yep.

• 52:51Speaker 01

  And like the way medicinal chemists work.

• 52:53Speaker 02

  Heavier molecules, more likely to bind.

• 52:56Speaker 00

  A lot of times it's because they're seeing improvements to affect other properties. That's part of it. And you probably got some parts to you.

• 53:02Speaker 01

  It's like, basically you have a chemist to, they find a hit and then they start optimizing it. And they're like, oh, I have a hypothesis that this is the binding site. So let me add this functional group to bind here.

• 53:11Speaker 02

  Okay.

• 53:11Speaker 01

  And then they're like, okay, well, that, that was a stronger binder. And then so they're like, oh, well, what if we add this other functional group to bind to this thing? And what ends up happening is they just keep adding things.

• 53:20Speaker 02

  Until it gets really good.

• 53:22Speaker 01

  And then, yeah, until it's sufficiently good. And then, you know, then.

• 53:26Speaker 02

  Instead of using a combinatorial approach.

• 53:28Speaker 01

  As Ian said, also, then you have these other properties. You're like, okay, well now we have this really good binder, but it's not soluble. So let me throw on, you know, some, you know, hydrophilic functional or, uh, some functional groups on the other side of the molecule, which is not in the binding pocket to like make it, you know, and you just sort of, you just keep tacking these things on. And next thing you know, you get this big, good binder. And so you just sort by molecular weight. And it's like the single biggest predictor that you can have in most public data sets. But that has nothing to do with chemistry.

• 53:59Speaker 00

  No, like, I mean, I think a good way of intuition is like, if a thing's working, they're going to do more stuff to it because they like it.

• 54:07Speaker 02

  Yeah.

• 54:08Speaker 00

  So you just do more stuff to it. It gets more baroque or you have more like complicated, expensive reactions. Or, you know, you get like weird building blocks that are hard to get. Yeah. You know, things that you wouldn't do if you didn't think it was working. And so it's all these signatures that the models would just kind of pick up. And so, yeah, it's quite terrifying.

• 54:24Speaker 02

  So this, this is like the horse, something.

• 54:27Speaker 01

  Clever Hans. Can you tell the story about clever Hans?

• 54:31Speaker 00

  Yes, yes. So for, uh, so a little history lesson here is there was a, um, there was a horse trainer in Germany. I think it was like the late 1800s. I mean, I read about this, I should remember it. And there's a horse named Clever Hans. And what Clever Hans seemed to be able to do was complicated math. So you would show the horse, you know, like a, like a chalkboard that's got a math problem. And then the horse would stamp the horse's foot on the ground and then stop when the correct answer, you know, was, was arrived at. And everyone was amazed that this horse is brilliant. And then someone had the good idea to go get a questioner who did not know the answer to the math problem.

• 55:08Speaker 01

  Yeah.

• 55:09Speaker 00

  And then the horse completely fails. And then it turns out that the horse is just looking at like the posture and the micro expressions, mostly of the owner. So that when it got to, you know, let's say the answer is seven, then the owner gets like real giddy when it's at six, right? And the horse stops stamping. And there's this like wonderful paper put out by, by a German psychologist who actually evaluates the, the actual micro expressions and postural changes that he had observed on the horse's owner is quite extensive. Okay. Yeah, like if you're dying for a silly read, you can go look it up.

• 55:38Speaker 02

  So then the analogy is that you're reading the face and not the.

• 55:42Speaker 00

  That's right. Yes, you do not understand math. The models do not understand chemistry at all. They are just, you know, remembering the sociology of, of the creators.

• 55:50Speaker 02

  Got it.

• 55:50Speaker 01

  Going back to Hermes and your epistemic rigor and, you know, you're in trying to build something useful out of all the data you trained.

• 55:59Speaker 00

  Yeah.

• 55:59Speaker 01

  So you find, you know, public data sets are untrustworthy. And you have a bunch of data which you trust internally, but you don't really have a good way of, or, but now you need to understand, how do we accurately and carefully use this to understand, to do something useful in the real world?

• 56:17Speaker 00

  Yeah.

• 56:17Speaker 01

  So how do you, how did you bridge that gap?

• 56:20Speaker 00

  For us to think about this, like we did a bunch of these splits, like I kind of described earlier. But then the next thing that seemed very obvious for us to do is like, well, you know, we've trained models on internal data and we kind of don't like to train models on external data. So maybe we should try to predict on external data instead.

• 56:35Speaker 01

  But without ever training on.

• 56:37Speaker 00

  Without ever seeing it. And so that does put us in a nice position because I haven't seen any other group train models for this task exclusively on internally generated data. I think it's very tempting to go get these other sources, especially because there's so many possible protein targets out there. Because people don't screen as many as we do typically. And so we took large collections of, of external molecules. And then we predicted them to see how well we would do across, you know, many different proteins, lots of different positives and negatives that we chose very carefully. It was, I want to say it was like the total number of tests in the Hermes paper. I think it's like 13 million or something across like many, it's like 500 protein targets. I think it's some huge number. And then we can, and we benchmarked it against the existing, what was at the time, like the state of the art structural model, which is done by the Bolts team.

• 57:22Speaker 00

  And so, which is a very good model and like ours cannot do that, a lot of the things that that model can do. But what we were able to do was do about as well as those guys on those public data sets in terms of like this binding problem prediction.

• 57:33Speaker 01

  When you compare to Bolts, it required some, some sort of tricks to, to kind of have a fairish comparison. I think Bolts also trained on this data. Was that right?

• 57:42Speaker 00

  Um, sort of. So there are a couple of large collections that we, we weren't totally clear on like the different subsets that they had trained on. And so there, I could think of one example in particular where it was very, very clear. And I'll talk about that in a minute. But a lot of the evals that we used were ones that were also surfaced by the Bolts team. Only we put a lot more, a lot more molecules in there to go see how it would go. So, yeah, like I think, I think there, and also if you look at the different performances between Hermes and Bolts, you know, like we're better on some proteins and protein families that, that Bolts is worse at and vice versa. And we suspect this is because of like different distributions of those in the training sets. So that they understand a little bit more. Or, you know, honestly, maybe like memorize a little bit more and they're able to extend that a little bit.

• 58:23Speaker 00

  One thing that was very validating for us is one of our test sets was like another orthogonal chemical library that our models had not seen that we had used internally. But it was a copy of a chemical library that had been out in the literature. And so there was one target in particular. It was called, it's a protein, it's called BRD4, where there's an identical molecule from our screen that was also in the PDB, the Protein Data Bank training set that Bolt had almost certainly seen. Our models did kind of bad on that molecule. It was a far away, it was a distant chemical library. They weren't very good at it. And Bolt was perfect at it. And it's because we suspect that Bolt had seen this protein and this individual molecule in a crystal structure already as part of the training set. So they were awesome at it. Oh yeah, that checks out. Feels pretty good.

• 59:07Speaker 01

  So does this mean that, I mean, in your opinion, this validates your scaling hypothesis? Add more proteins, more diverse chemical libraries, and now you can basically fill in in this space more broadly?

• 59:19Speaker 00

  Or? We hope so. Yes, we hope so. Yeah, like, we think that going after large amounts of diversity, and in particular, going after intentionally chosen, what you might call activity cliffs, is the way to do this. And kind of unpack that a little bit, it's either on the protein side or the small molecule side, do you have two things that look very, very similar, except for a tiny change, and that tiny change drives like outsized changes in behavior? So like a real classic example of this that medicinal chemists will talk about is this idea, and it's kind of mythical, it's not exactly real, but this thing called a magic methyl, and the idea is that you've got a molecule, and then maybe you've got like this one like little thorn that is or is not there, and that presence or absence of

• 1:00:03Speaker 00

  that thorn means like either it works great, or it doesn't work at all. When you have the situations, then maybe your model will start to learn like, man, like, having a thorn could be really meaningful, and so maybe I need to think about like these very small changes. And so one thing that we do a lot, and it's baked into Dells, because you've got, you know, two out of three building blocks might be similar across large numbers of examples and have variations of behaviors. And then on the protein side, we very intentionally engineer mutations in those proteins to do the same kind of thing.

• 1:00:33Speaker 02

  I'm a little confused here, because if you have these building blocks, and the size of your variation is multiple atoms, right, many, how big is like a building block in terms of atoms?

• 1:00:44Speaker 00

  Right, right, it might be, it might be like six to like 30 or something, if we're not ignoring like, if we're ignoring like the hydrogens, but you can, you can imagine that there are building blocks that themselves are very similar with like a single atom substitution.

• 1:00:54Speaker 02

  Oh, I see, okay, so maybe there's two building blocks that have one, one a thorn and not a thorn, and then you, and then, so you can see that in the, in the sequencing?

• 1:01:04Speaker 00

  That's right, that's right, and maybe that thorn just changes position.

• 1:01:07Speaker 02

  Okay.

• 1:01:08Speaker 00

  And then, as you're, and as you're thinking about this, you're like, okay, well, I might have tension between my goals, I might want lots of like rougher, more coarse diversity, because I'm trying to find binders across large numbers of protein targets that may be very different, or I may want like super fine grained changes that are very, very subtle, and you know, to try to like really teach the models like these like very, very subtle differences. And so as we think about designing our own chemical material, like we have to, you know, span this kind of tension. So yeah, we think about it a lot.

• 1:01:37Speaker 02

  And both of those matter?

• 1:01:38Speaker 00

  Both of those matter.

• 1:01:39Speaker 02

  Both of those matter.

• 1:01:40Speaker 00

  Both of those matter.

• 1:01:40Speaker 01

  And it also matters on the protein side, too.

• 1:01:43Speaker 00

  It matters very much the protein side, and so as you look at current like sort of co-folding models, a couple of different groups have shown this, you can take a protein, and it's got this amino acid sequence, and it codes some protein that's got a big pocket in it, and maybe that protein's represented in the training set, so the model has a pretty good idea about what it looks like when there's a molecule in there, a small molecule chemical in there. So you've got this pocket, the thing that goes in the pocket. You can just change the amino acid sequence that you give to the model to consider, and you can use that to break the pocket, you could fill the pocket in like you're filling it with cement, or you could remove it entirely so that there's nothing there, or it's gone, or it's much bigger. You do any of these things, and you get at least what we've seen so far, if you use current state-of-the-art models to go see what the model will do, it will shove the small molecule in where the pocket used to be.

• 1:02:29Speaker 00

  Because it just remembers where it is.

• 1:02:30Speaker 02

  Yeah.

• 1:02:31Speaker 00

  And that's bad.

• 1:02:32Speaker 02

  So it's like looking at the large scale structural things, not some, or like the hypothesis anyway, is that it's doing this?

• 1:02:38Speaker 00

  Yeah, the hypothesis is that it doesn't quite understand, you know, when you, when you have a pocket and a protein, there's going to be a small number of amino acids that will have hydrogen bond positions that are going, they're sticking out, and there's going to be maybe a little bit of pie stacking, these other things that, you know, sort of drive this chemistry, and those things really ought to be very, very important. And, and they are very important in the, in the physical world when you're measuring them. And so if you break them, and the model understands the physics, then it probably should know that it can't shove a molecule on there. And that's the thing, is like, a lot of these structural training sets, like, there aren't great negatives, because what makes it into there is

• 1:03:12Speaker 02

  Yeah, I was gonna ask.

• 1:03:13Speaker 00

  The thing, right? You know, they don't have a lot of these variations that really, you know, disrupt these behaviors. And no one's going to make crystal structures with a bunch of these mutants, like we, you know, like we're currently doing for us.

• 1:03:22Speaker 02

  But how do you get negatives?

• 1:03:24Speaker 00

  The way that we get negatives is that we have a really good understanding of the distribution of the number of those molecules in a baseline condition.

• 1:03:32Speaker 02

  Okay.

• 1:03:33Speaker 00

  So like, when we do the screen, we know, oh, okay, these bound a lot, because now we're enriching for them when we pull the protein out, and now let's break the pocket, and they are no longer enriched. Okay, they are no longer going, that one amino acid substitution that we made is making a bunch of these chemical series drop out.

• 1:03:50Speaker 02

  But this is a one-off experiment, this is not a Dell.

• 1:03:53Speaker 00

  No, this is a Dell.

• 1:03:53Speaker 02

  Oh, no, so you're just, you're not saying you broke it, you're saying, I find the case where it was broken?

• 1:03:58Speaker 00

  No, we break it, so we break, so we take the, so let's say we screen a protein, and the, and the pocket is intact.

• 1:04:04Speaker 02

  Yeah.

• 1:04:04Speaker 00

  Sounds great, okay. We get a bunch of hits.

• 1:04:06Speaker 02

  Yeah.

• 1:04:07Speaker 00

  Let's, let's go even further here, let's say that we know someone else has already identified a chemical that goes in that pocket. Let's buy it, let's stick it in there, run the screen again.

• 1:04:17Speaker 02

  Yeah.

• 1:04:18Speaker 00

  That means all the chemicals that normally go in the pocket will be displaced by a high concentration of the

• 1:04:23Speaker 02

  Definitely not.

• 1:04:23Speaker 01

  Oh, you have a competitive assay?

• 1:04:25Speaker 00

  Oh, we do, all the time.

• 1:04:26Speaker 01

  Okay, okay.

• 1:04:27Speaker 00

  Do this all the time.

• 1:04:28Speaker 02

  Yeah.

• 1:04:28Speaker 00

  Okay, and then you say, okay, well, that looks pretty good, I believe that. And then you say, all right, now, we know that this chemical that someone else has characterized goes in this pocket, because maybe they made a crystal or some other thing.

• 1:04:37Speaker 02

  Yeah.

• 1:04:37Speaker 00

  Great, let's look at the, the amino acids in that pocket, and then we will change the DNA construct that makes the protein for us. We will now mutate physically this one residue, so now like this hydrogen, this, you know, this, this particular amino acid, now we put a big bulky one in there, and the pocket's full.

• 1:04:54Speaker 01

  Or like the different side chain or something?

• 1:04:55Speaker 00

  Different side, all these things you can do.

• 1:04:56Speaker 01

  Yeah, yeah. And this is really easy now, because we have Alpha Fold, and, you know, Bolt, or all these structure models, you can, you can use this to design your proteins as you

• 1:05:05Speaker 00

  You can use this, you can use, and for things that aren't as well characterized, you know, like we use conservation, you know, like we stare at, we have a structural biologist on staff, and kind of look at these things like, oh, yeah, you know, this guy Edward Kraft is amazing. And so, all right, so you know, now we're going to make this one mutation, let's do yet another screen, and, oh, look, all the chemicals that got booted out when we hit it with the, with that one thing that's blocking the hole, they're also gone when we broke the pocket. So now we have really good evidence that they're all going into that pocket.

• 1:05:31Speaker 02

  Right.

• 1:05:32Speaker 00

  Right, and then here's the point where, you know, like the protein chemists come in and they say, well, if you made a mutation, how do you know that you didn't just break the whole thing, right? Reasonable question. You say, okay, well, we only lose the chemical series that got kicked out by that competitor. All the other ones that are binding everywhere else in the protein are intact, and they come back.

• 1:05:49Speaker 02

  Yeah, so you didn't just break the protein.

• 1:05:51Speaker 00

  You didn't just break it, right? And so, and then you think, okay, well, if you're going to be out screening 100 protein constructs a week, which is what we do on big weeks, like

• 1:05:59Speaker 02

  Wait, sorry, so you've broken the pocket somehow?

• 1:06:02Speaker 00

  Oh, yeah.

• 1:06:03Speaker 02

  Then, do you also find that there are other things that do, that start to connect, to bind to the protein?

• 1:06:10Speaker 00

  So when you change things around, you might change the shape in a way that does encourage other things to bind. We have definitely seen it.

• 1:06:16Speaker 02

  And is that an information source for you?

• 1:06:18Speaker 00

  It is, it is. And so another thing that's really nice about this, and the nice part about Dells, is you can, you can imagine, if you're not thinking in large numbers, that, oh, I've got this one chemical that was binding, and now I broke the pocket, and now it's gone.

• 1:06:31Speaker 02

  Yeah.

• 1:06:31Speaker 00

  But because of the combinatorial nature of the Dells, maybe I have five different chemical series, it's like, it's like hundreds of molecules, and there's another group that's the same hundreds of molecules with a single change in the building block. So like, you have these repeated examples, where it's like, oh, this one tiny thing shifted the population behavior enormously, and it's a big population. And so it's like, oh, I am now really convinced, I can show you guys a plot that we, I don't know if you, you know, put pictures on the podcast or whatever, but it's very obvious, like, oh, this one side group on this building block is clearly interacting with this one amino acid residue, because we have like five independent examples of population behavior shifting because of it. It's beautiful, it's very, very clear.

• 1:07:12Speaker 02

  Yeah, go ahead, tell us about this plot.

• 1:07:14Speaker 00

  We pulled this thing up, and the reason that we pulled it up is, we think it's really important for there to be a lot of variation in the proteins, as well as the small molecules. And a reason that we think this is, you want to be able to get the, you know, a model trying to learn medicinal chemistry to really start to put together, you're having these hydrogen bond interactions between like this part of the small molecule and this part of the protein. We've seen other work that I talked about earlier, about how you can, if you digitally break a pocket and you use the current state-of-the-art models, that they have a really hard time understanding the physics, and they'll just shove the molecule back in that thing anyway, so it makes us suggest that they're memorizing. And so we thought one way to go around, get around this, is to experimentally engineer mutations in binding regions of these proteins, and then re-screening.

• 1:07:58Speaker 00

  And so what that looks like

• 1:08:00Speaker 02

  Physically doing that.

• 1:08:01Speaker 00

  Physically doing that. So you take, you take the DNA of the wild type protein that you've already screened, and then you use microbiology techniques to change one or more than one amino acids in that protein, so that the functional part where the small molecules are binding is no longer the same. It's different in some way. We have a different methods to try to identify which amino acid to the smart was to break. Sometimes you break ones that you're confident will do things. Sometimes you don't know. And so you, but we can do this at a pretty big scale. So it's not a big problem to us to take a couple, you know, wingers at each one. And so then you run the screen. And now when the pocket is broken, where the molecules go, what do the molecules do? Do they still go in there? Do some of them go in there? Is there something common among the ones that stick around? Is there something common about the ones

• 1:08:43Speaker 00

  that get dislodged or pitched out? And then you have lots of different examples of these where you might have three to five ish amino acids in a given pocket that are driving the binding behaviors. Maybe you try to break all of them independently and then you do redo the screens each time and get a good feel for what part of the small molecule is interacting with the part that you have changed. And then when you do this with a large Dell, then you'll have lots and lots of repeated examples of similar molecules. They're all following the same behavioral trajectories. And that way you get a really good feel for, oh, I've got lots of examples. They're all pretty similar, but the one thing that's super common is this one little bit on the small molecule. So I have a suspicion that is interacting with the pocket that I just broke.

• 1:09:24Speaker 02

  What are we looking at on this diagram?

• 1:09:26Speaker 00

  So what we're looking at on this plot is all those little dots are small molecules from a chemical library that we have designed and made. Along the axes are how well the small molecule is bound. So if it's like really down the axis, it's binding well. If it's more towards the origin, it's not really binding at all. And then we've colored a couple of different blobs and those blobs are different chemical series. The chemicals look very similar to each other. But the difference is where we pointed that kind of chemical looking structure. And so if you're squinting and you look at these, they're all pretty similar, but there's a couple subtle differences. And that whole population, we believe, is driven by interactions of those differences to the one mutant that we made.

• 1:10:05Speaker 02

  So in this plot, there's on the bottom, the wild type target. And then on the, on the horizontal or the vertical axis, you have the mutant type target. And so there's this red one. And the red one is pointing basically up.

• 1:10:22Speaker 00

  It is binding better.

• 1:10:23Speaker 02

  It is binding better. So that particular one is like the mutation made that particular one bind better, which is speaking to the question that I asked a few minutes ago.

• 1:10:32Speaker 00

  Yep.

• 1:10:33Speaker 02

  And then maybe some of the, most of these seem to be above the axis. So most of this mutation caused most of these to bind better.

• 1:10:42Speaker 00

  That's correct. Yeah. There's a second mutation in that pocket that I can also share with you guys, where all of the, all of those populations, all those blobs get kicked out entirely and they do not bind. So then the pocket was broken in a meaningful way. And then here the binding is increased.

• 1:10:56Speaker 02

  And so that basically everything will be along the bottom.

• 1:10:58Speaker 00

  That's correct.

• 1:10:58Speaker 02

  Okay.

• 1:10:59Speaker 00

  That's great.

• 1:10:59Speaker 02

  So you can give us for post product?

• 1:11:01Speaker 00

  Yeah.

• 1:11:01Speaker 02

  Yeah.

• 1:11:02Speaker 00

  Yep.

• 1:11:02Speaker 02

  Awesome.

• 1:11:03Speaker 01

  So doing this, you can map out both, well, you can, you can treat the activity cliff issue, both in compound space and in protein space.

• 1:11:12Speaker 00

  That is correct.

• 1:11:13Speaker 01

  And this is particularly useful if you're, let's say, working at kinases where there's like 100 kinases, which differ only by like, and the important binding site, you might differ by one residue at a time or something. So this can be incredibly powerful for certain classes like that, in addition to your own sanity.

• 1:11:29Speaker 00

  Yep. That's right. Yeah. And we do this, you know, as I mentioned, we do this, we do this a lot. Yeah. Many, many hundreds of these types of screens.

• 1:11:36Speaker 01

  One of the things you can do with this is Hermes sees this cross attention. So you can actually interpret where Hermes thinks the binding pocket is. So it's learning the binding pocket through these type of maps.

• 1:11:48Speaker 00

  Yes. So one thing that we've sort of written about is using these attention mapping approaches to have a look at where Hermes is, you know, looking in amino acid space. And when we first did this work, we were expecting to, for it to either look at the binding pockets. That was like the first intuition. And then maybe like a more measured intuition was like, oh, well, maybe what we'd hope it for it to do is maybe look at all those residues because it probably doesn't really know what's going on yet. And then when we looked, it was not looking at either of those things. It was looking at really short sequences of amino acids that were away from the functional part of the protein and also away from part of the protein that looks similar across many proteins. And so for maybe the non biology people in the room, the way a lot of molecular evolution has

• 1:12:32Speaker 00

  worked is that there's a stretch of a protein that does a pretty good job. And then often gets shuffled among other proteins as well. So that you had this conserved regions called a domain or, you know, conserve protein domain. And so that part looks pretty similar across many proteins. It's something that biologists will use to infer protein families and relationships and functions. So if they all the same kind of piece, they're probably doing something kind of similar. And so we expected the models to look at that, but they don't. They look outside that. And they look at these little stretches that people don't think typically do a whole lot. They're like a little bit different. They're non conserve. There's a lot more variation in those. And we realized that Hermes was probably coming up with a nickname for the protein. It's not thinking about, oh, this is the functional part. It's

• 1:13:13Speaker 00

  like, I don't need to learn that. I can remember, you know, 10 or 12 amino acids that are different here than everyone else. And I can just memorize it. And we thought that was bad.

• 1:13:25Speaker 02

  Because maybe many of the, like, if you have variations in the pocket or whatever, but, but they all have these other parts that are similar, then maybe in the variation in the pocket mostly don't matter, but some of them do, then you might, the model might learn, oh, I can just pay attention to these.

• 1:13:43Speaker 00

  Yeah. The concern was that the model again was not really learning any physics. It's like, oh, I could just memorize this stuff. It's very much in keeping with when you try to break the pocket with the co folding models and they shove it in there anyway. It's like, oh, it doesn't know what's going on. It's just made this nickname.

• 1:13:56Speaker 02

  Yeah.

• 1:13:56Speaker 00

  It's like, oh, man. So how do you fix that? And then that's when we hit upon trying to do these, these mutations at scale. And so what happens is if you have a protein that looks almost exactly the same, but there's one tiny difference. And then the behaviors change a lot as a consequence of that one tiny difference. You're kind of encouraging the models to look at all the amino acids again. It can't do that for the section that's non conserve. Cause if it does that with these two things and different behavior, it's going to do a bad job.

• 1:14:20Speaker 02

  Yeah.

• 1:14:21Speaker 00

  And so we went back and did, you know, train models with these additional mutations, like the one on this pot that you guys are seeing. And then, you know, look back at the attention maps. And now Hermes will look at all of these things much more evenly.

• 1:14:31Speaker 02

  So you're basically using the attention map to find these cases where it's memorizing nicknames.

• 1:14:36Speaker 00

  Yep.

• 1:14:37Speaker 02

  And then enrich the mutations in the pocket to kind of encourage it to pay attention to the pocket instead.

• 1:14:44Speaker 00

  Yep. Yep. And it's not necessarily just to be across any one gene. It's more like the general problem of like, oh, I can't use this nickname strategy effectively anymore.

• 1:14:53Speaker 02

  You want to teach the model that, no, don't do that.

• 1:14:56Speaker 00

  You don't do that. No, no, that's, that's lame. Don't do that.

• 1:14:59Speaker 02

  So I love this because we've seen where I think that the most pragmatic use of Mechan Terp has been in biology and in probably others related sciences where, and the reason why I think it's qualitatively different is because you go backwards from, you're learning about biology from the model rather than the, just trying to interpret what, like, what is, like, what is it thinking? In this case, maybe it is actually that for a case. But I've just seen so much Mechan Terp being applied to biology because you're really trying to understand, you don't have the language in these machine learning models.

• 1:15:41Speaker 02

  There is no language to lean on. So you really have no clue what it's thinking unless you're inspecting the model.

• 1:15:47Speaker 00

  Well, it's really interesting. So yes. And one thing that we have found very fruitful in the very recent past is actually transitioning to language models.

• 1:15:56Speaker 02

  Okay.

• 1:15:56Speaker 00

  And what we are currently doing is we're using readable, interpretable English to represent the small molecules and the proteins and their behaviors.

• 1:16:04Speaker 02

  Wow. Okay.

• 1:16:05Speaker 00

  And that turns out to work pretty good.

• 1:16:07Speaker 02

  Yeah. I mean, this is, no, so this is, because I want to just frame this a little bit before you tell us about this because there's actually a really interesting paper in the single cell domain called Gene PT, in which people, so I can't remember the authors. They're from Stanford and it was somebody who was a statistician, not even really in bio. Maybe they are now, but that person, they took the textual embeddings of the descriptions of genes.

• 1:16:41Speaker 00

  Yep.

• 1:16:41Speaker 02

  And then they use those instead, like you're using the ESM embeddings?

• 1:16:44Speaker 00

  Uh, yeah, for Hermes. Yeah.

• 1:16:46Speaker 02

  So you were using the ESM, they were just like, let's just use the textual embeddings. And the beauty of this is because now you have like a gazillion years of human time put into understanding each of these genes. And that's all captured to some extent by these textual embeddings, that knowledge of the embedding. So like, yeah. So please tell us about what you're doing.

• 1:17:06Speaker 00

  Yeah. So we started this project a little while ago. Cause one thing that we realized is that we had this. Like growing collection of activity cliffs and profiles of the same chemical material across large numbers of proteins. And we wanted to know if we could actually use reasoning agents to go explore it and use it in a lab in a loop kind of situation. And so that demanded that we started working with reasoning models and language models in order to kind of let them go play. And so what that looks like is we took some open source language models, like we did some Quinns and GLM and the stuff that other people are using. And we taught them medicinal chemistry in ways where you say, here is a structure, describe it in natural English, give me the diff between two molecules in natural English.

• 1:17:50Speaker 00

  And then we got it to the point of where we say, okay, here are some molecules that are binders, here are some that are not binders. Can you tell us why you think the binders are binding?

• 1:17:59Speaker 01

  Mm hmm.

• 1:18:00Speaker 00

  And then you can also give it hard negatives. So you have these two collections that think, okay, well, I think this chemical series has this, you know, and it will write these things down in readable English, which we think is great. And then we said, okay, now that you sort of know these basic concepts, we're going to give you 10 binders and 10 nonbinders that are closed, ask you to decide why you think the binders are binding. And now we're going to give you the opportunity to run your own experiments. Here is a catalog of millions of molecules. Why don't you pick five or 10 and test your hypothesis about why they're binding? And so then the model will go into the data that we have already collected. It will pull out what it thinks are the most informative choices and will get back the answer instantly, like it's a coding sandbox, and then it will update its hypothesis about the binding. And then the ultimate goal is to get the best binder in as few steps as possible, very

• 1:18:44Speaker 00

  much like a medicinal chemist would do. And then we just

• 1:18:47Speaker 02

  But you're doing this based on your A priori run experiments, but it's looking up in a database.

• 1:18:51Speaker 00

  That's right.

• 1:18:52Speaker 02

  Right? So now, so you're basically created reinforcement rollouts.

• 1:18:56Speaker 00

  It is a reinforcement rollout of a large scale that's growing all the time, that's full of these activity cliffs and full of different protein variations. And so now we have, and so you can use this, and since we know that when you change the molecules in these small activity cliff ways, it is a causal change. Like, we know that when you put this thing on there, the behavior changes in this reproducible, measurable way. And so you can also use these trajectories to come up with reasoning traces kind of at a pretty good scale. And so right now, we have these agents running through the data set. We just give them 10 binders and 10 nonbinders, give them a budget of 100 molecules, and they'll go through, you know, probably like 102 halls, where they're just going through this long, many-step reasoning problem, and then come up with what they think are the answers to the best at the end.

• 1:19:37Speaker 02

  So why do you think this is superior to something like Hermes, or maybe like something, you know, sort of scaling up the Hermes approach where you're, you know, sort of building some, you know, attention thing?

• 1:19:51Speaker 00

  Yeah, we think so. We think it's much more resistant to the memorization problem. And so like, while we are very happy with where the attention goes with Hermes when you throw in all these mutants, like, as you add more parameters to a model, like, they're going to memorize the mutants, too. Right? Like, and we want this thing to really understand, like, these causal changes. And it seems like using natural language is like a pretty good regularizer for the problem. And like, we are unaware of other problems where you have a reinforcement learning loop that is physically grounded, that is this scalable, this fast. Like, you can imagine for, you know, when you're doing lab in the loop stuff, where you're going off and making robots and going off to these problems, they will also necessarily require, like, limited resources. They will still make some choices about what to make in the first place, what not to make

• 1:20:34Speaker 00

  in the first place, and then doing those loops will take a little bit longer. And if there's a scalable way to measure that landscape, and then continually make that bigger and bigger and bigger, it's allow the agents to play in it, we think it's probably going to go a lot faster for this particular problem.

• 1:20:47Speaker 02

  That's interesting, yeah. I mean, I would say it's Excel from Xera is getting there, but they're not actually language there, but they are encoding as a sequence. So it's, in some ways, the perturbation is similar, but this is, yeah, this is different. And then Lila also rhymes with this, but

• 1:21:07Speaker 00

  It does. Like, all these guys are really smart and pushing like these automation things.

• 1:21:11Speaker 02

  Yeah, but they're, they're not doing these pooled screens and then using that as they have, as far as they know, haven't figured out how to make that into just a database lookup, so you can do huge number of rollouts.

• 1:21:26Speaker 00

  Yes, yes. So right now, the way that we like to think about it is if you're a medicinal chemist and you're working at a pharmaceutical company, you do this thing called a structure-activity relationship. And what that means is, oh, when I change the molecule in this way, it changes the behavior, and so I'm trying to optimize for these different properties. And you go through what's called an SAR campaign. That's early drug development, it's making the molecule better. You might see a couple dozen of these, maybe 50 of these in your career. And so right now, you know, last weekend, we pulled out over 10,000 of them, and we're not even trying that hard.

• 1:21:56Speaker 02

  Right.

• 1:21:58Speaker 00

  So, yeah, and so now is the opportunity. Now, we'll be restricted to the space of things that we have screened. And then at the end, you can ask the models to make additional choices from, say, catalogs, and then you can go, you know, in the future, go order those and retest them like you might for like a traditional lab in the loop approach. But right now, we give these agents 10 binders, give them a budget of 100, and they'll get the best binders out of six and a half million in that budget. And so it seems like it's working.

• 1:22:23Speaker 02

  Yeah, so like, tell us more about why you think this is, I get the argument about it's a better regularizer, but also, why, why do you believe that this is effective? Right? So like, you can imagine, well, it doesn't get things wrong so badly, but it also doesn't get things that right, but

• 1:22:44Speaker 00

  I think it's because you give it the opportunity to make all those choices. I mean, it's like running these coding sandboxes, you just have like really rapid reinforcement reward signal. But the thing is, you also have to include the choices that a human being would not have made, because we're measuring all the molecules across all the proteins.

• 1:23:00Speaker 02

  Yeah, so it has to be able to make the counterintuitive decision and get a feedback on that.

• 1:23:06Speaker 00

  It's got to have the counterfactuals, it's got to make mistakes. And you need to make sure that the mistakes are, you know, present in the data, or future potential mistakes are present in the data in ways that are not really biased for it to, you know, learn chemist sociology fingerprints.

• 1:23:17Speaker 02

  Yeah, that's, that's really fascinating.

• 1:23:19Speaker 01

  So through all of the, you know, language pre-training corpus, it kind of has an intuition for some basic chemistry that I think most medicinal chemists might have an idea of. And now you're giving it sort of like an infinitely fast lab in the loop.

• 1:23:32Speaker 00

  That's correct.

• 1:23:32Speaker 02

  But maybe reasoning, too. I would say that reasoning skills might even be more important, right? The ability to say, oh, in my reinforcement campaign, I've, learning campaign, I've learned things, and now, now that allows me to reason about the next harder question.

• 1:23:48Speaker 00

  And then just keep making choices.

• 1:23:49Speaker 02

  Yeah.

• 1:23:49Speaker 00

  You know, and, you know, right now, we, like, we're really excited about the early signal. And also, you know, like, we're a tiny shop, right? There's only 10 of us. Like, I'm really looking forward to when, you know, our experimental corpus grows at, you know, like 2X, 10X, you know, like 100X the rate that it currently does, because then like the amount of space for these agents to explore like expands dramatically.

• 1:24:08Speaker 02

  Let's talk about that space, because, so my understanding, and this is from me, like, you know, talking to Claude and Googling and whatever, but my understanding is that Dells are like limited in that they are, they tend to be, create flat molecules or flatter molecules.

• 1:24:24Speaker 00

  It depends on the building blocks.

• 1:24:25Speaker 02

  Okay.

• 1:24:25Speaker 00

  It depends on the building blocks. Yeah, so there are, you can imagine that you will be somewhat constrained by designs in terms of like, oh, I'm going to get some number of building blocks, usually two to three, and then stitch them together in some way. And so like, when we do chemical library design choices, typically, it's like, like, one that we run a lot right now is a forked design, it's like a Y-shaped molecule. And then there's other ones that are linear shaped molecules. There's yet other ones that are kind of like circular, kind of macrocycle shaped things. And then you have to think about, where is the DNA going to go? And like, how is that going to cause me problems? And then also the, the commercial availability of building blocks and the chemistry is required to stitch this together. You can go for custom stuff, takes longer, it's kind of more appealing to us to go faster.

• 1:25:06Speaker 02

  Yeah.

• 1:25:06Speaker 00

  And there's certainly going to be corners of chemical space that we're probably bad at, and then you have to think about, oh, well, future libraries should have this kind of geometry, and then I got to think about the different proteins I'm going to throw at it. And there'll be like a pretty uneven distribution of like, a whole chemical library made this way might be really good for these types of targets, it might be really crappy for these other types of targets. And so then you get to think about, all right, what are my, what corners of protein space am I optimizing for? And those kinds of tensions.

• 1:25:32Speaker 02

  Yeah, so, so it sounds like the argument is, well, we know that we're not optimizing the whole space, but it's a lot more than it was before. So that's better. And then we are, as we get better at this and understand, or we have bigger budget or whatever it is, then we'll expand to these corners that we know that we're lacking in, but, or maybe we don't even know we're lacking, but we will find that out.

• 1:25:54Speaker 00

  Right, exactly, exactly.

• 1:25:55Speaker 01

  Do we want to move on to the last phase?

• 1:25:58Speaker 00

  Yeah, you can, go for it, let's see how it goes.

• 1:26:00Speaker 01

  All right. So for the audience context, Ian requested, I want to confirm that this was his idea and not mine, a, a don't ask me, or a come at me, bro, section, where we, we ask the hardest problems we can come up with.

• 1:26:15Speaker 02

  Which is, as you guys can probably tell, is not very hard for me.

• 1:26:19Speaker 01

  Yeah.

• 1:26:19Speaker 02

  I think I've asked my hard questions already.

• 1:26:22Speaker 01

  We only had about 24 hours to prepare, so I think most of these will be Claude's hardest questions, but anyway, we'll see how this goes.

• 1:26:29Speaker 00

  Yeah, the context for this is I was at, I was at a cancer microtherapeutics meeting last week, and, and the, the format of the meeting is half of the time is for content presentation, and the other half is for very aggressive grilling. And it was, it was, it was horrifying and inspirational, and so here I am, willing to do it in public.

• 1:26:48Speaker 01

  Let's do this. Dells, as RJ was just alluding to, oftentimes have very specific properties, and you can get very good at optimizing in Dell, but ultimately, they frequently fail at certain things which are hard to translate to actual medicine.

• 1:27:03Speaker 00

  Yep.

• 1:27:03Speaker 01

  So certain failure modes being such as, anything which is easy to assemble is easy to disassemble, bad for, you know, admin properties. You only have so much room to kind of build on, because you've already sort of used some of the combinatorial building blocks and that one. Anyway, how is this limiting in terms of, like, long-term out of library generalization?

• 1:27:22Speaker 00

  I think a lot of it is going to be somewhat dependent on like building block availability and like one's willingness to go into custom synthesis. So if we're, especially for the fine-grained mapping stuff, you know, being able to say, move a nitrogen all the way around the ring and all one of those positions is something that's very, very desirable, but it's also something that's pretty unlikely that you're going to get, like, commercially for all sorts of building blocks that you're currently generating. So like, is that going to be a failure mode? For sure, yes, yes. I would, I would love to be able to get more of those and like beef up our custom synthesis, that's something that we're looking at. But right now, trying to get those is challenging. One critique that Claude might be thinking about in terms of molecules easy to put together and disassemble, for many years, it was the most common way to attach building blocks in adult chemistries is amide bonds or protein bonds, and so those are quite digestible,

• 1:28:07Speaker 00

  you know, in, in the body. But in the past few years, there have really, really passed like four-ish years, there've been like a really nice fluorescence of chemistries that are friendly to DNA, that don't destroy it during the attachment process, because that's very important, that will give you carbon-carbon bonds, carbon-nitrogen bonds, these kinds of things. So it's actually pretty good for workaday simple chemistry. Yeah, so like, I'm actually not too worried about that part. I'm more worried about building block diversity than their sources for that.

• 1:28:35Speaker 01

  Yeah. What about admin properties in general? Like, you know, you have been focusing very much on binding.

• 1:28:40Speaker 00

  We have, yeah. So of projects that we've done, trying to look at reasonable admin and like PKPD properties. For what it's worth, we've gone directly from from chemical library hits to good PKPD in a mouse in six months with a very small number of med chem cycles. So at least the chemical material that we currently have, or at least that, you know, those few series that we did that for, you know, they turned out pretty good in our hands. I guess, you know, we will continue to see like how that modeling looks. And also, of course, we want to do more of these like admin target models and then think about how to incorporate other types of, you know, more emergent models along with our designs.

• 1:29:12Speaker 01

  Was that speed driven by your ability to create custom, like local combinatorial libraries, or was that speed driven? Like, what allowed you to get there in six months?

• 1:29:21Speaker 00

  Like, um, the drift, so we, when we designed this chemical libraries at the beginning, we very much optimized that space for drug likeness. And so we think that's part of it is that was like a really good starting point. You know, when we do our chemical library designs, we have this kind of like Thompson sampling level approach where we come up with an optimization function. You say, okay, well, I want the libraries to have these properties. And so some of those properties are going to be, I want to bind these things. I want to avoid these things. And then some of these properties are going to be, oh, there's other admin model approaches that you can use to make predictions about how your chemical material might turn out. And so you put that as part of the optimization function. And then what you do is you say, okay, of all the chemicals that I could make, given my geometry, I don't want to make that many. I want to make a smaller number.

• 1:30:04Speaker 00

  And so you, you could, you basically set up and generate a bunch of synthetic libraries and you score them on this optimization function. And so we did that with this library and the starting material looked pretty good. So I'm sure that we'll continue to do that.

• 1:30:15Speaker 02

  Along those same lines, has anything that you've done, like made it to, so it made it to animals?

• 1:30:22Speaker 00

  Yeah, so PKPD in a mouse is that's, yeah. So what that, what that means is can you dose a mouse with the thing and then see how it gets into the mouse's body and tissues?

• 1:30:31Speaker 02

  Yeah. So PKPD is pharmacokinetics, pharmacodynamics, meaning, yes, how does the body move the drug through the, or how does metabolism move the body, the drug through the body?

• 1:30:42Speaker 00

  Yeah. And then are there biomarkers of the drug's intended activity? And so what that means here is this is like a drug, this is a compound we're working on for a little inflammation. And so we were following cytokine levels in mouse blood and they, and they did what we wanted them to do and expected them to do based on the targets of the drug.

• 1:30:58Speaker 02

  So Sean likes to ask when money, I want to ask when people.

• 1:31:06Speaker 00

  Yeah. I mean, for those we talked to, we have a couple of these things that we're talking to folks about partnering out to spin up new entities to go chase them down.

• 1:31:12Speaker 02

  Okay.

• 1:31:13Speaker 00

  And so, yeah. And of course, like all these things take like sometime, you've got to do, like there are these traditional things for it, which you declare what's called like a development candidate, then you've got to do a bunch of other work about manufacturing and additional safety work before you can get into people. So all these things take a lot longer.

• 1:31:27Speaker 02

  Yeah. And so you're going to spin out, the idea is to spin out a, so people, maybe even internal, they will say like, I really want to take this, this thing to market or try to get it, like take it to the next step.

• 1:31:38Speaker 00

  Yeah. I mean, one way that we kind of think about it is, you know, we built this really great engine for like data generation and model training. And now, as we've talked about, like these reinforcement learning reasoning approaches. And most other domains, when you build an engine to go do that stuff, to go like help on those problems, the subset that you're generating along the way is not itself intrinsically valuable. No one wants to sell reasoning traces about like political theory, but like there are going to be little bits of gold that we have generated as a consequence of doing all of the screening.

• 1:32:11Speaker 02

  Because you're actually reasoning about not only that this model is good, or sorry, this molecule is good, but that this, here is why that's good.

• 1:32:20Speaker 00

  Here is why that's good. And here is chemical material that's proprietary to us that has, you know, activities that are, you know, protein target classes that others might find hard to drug, right? And so you've got starting material for, you know, right now, we have starting material that looks pretty selective to us for, I think, like 700 targets, like it's a norms number. You don't want to chase all those down, but you know, as, as opportunities, they are all opportunities with different levels of, you know, kind of economic value capture and desirability. And so you think, all right, well, we've got all this stuff, like, let's see what we can get out, we'll see what we can get out of it.

• 1:32:51Speaker 02

  Discovery is like a small percentage of the total cost of developing a drug. And you can argue that, well, yeah, well, any given discovery is cheap, maybe, but that, you know, because of the failures, you have a lot of them. So, but even then, you know, to get one drug, one drug, like all the steps of a single drug candidate that is successful is maybe, you know, on the low side, 200 million, on the high side, a billion, depending on who you have to test on and lots of other factors. But, and then, you know, the discovery is a couple million. So, so are you tackling the right problem here?

• 1:33:36Speaker 00

  And so I think there are a number of early discovery problems that are worth that much, you know, like, like reasonable amounts. And so what does that mean? There's a bunch of targets right now with like really good genetic evidence that are very challenging to drug with small molecules.

• 1:33:52Speaker 00

  You know, a pretty straightforward example is for a lot of these kind of cytokine inflammatory indications, folks are using antibodies to drug them and going after a small molecule therapeutic that you could take as a pill instead of an infusion, one that doesn't require like cold chain, you know, transport to get to different patient populations. Like having those things is enormously desirable.

• 1:34:13Speaker 02

  Meaning you have to have refrigerated trucks.

• 1:34:15Speaker 00

  Yeah, right? Yeah. Like, and like, if you want to get these, if you want to get these antibody based drugs to some of these folks, like this is like a very hard problem.

• 1:34:21Speaker 02

  Yeah. And very expensive.

• 1:34:23Speaker 00

  And very expensive. And so, and so folks are like really quite excited about, can you find a small molecule therapeutic that does the same thing than antibody against these?

• 1:34:30Speaker 02

  Right. So there's some antibody, it's, it works.

• 1:34:32Speaker 00

  Yep. And so we know it works.

• 1:34:34Speaker 02

  And so you're like, well, why can't we just go after the same target with a small molecule?

• 1:34:38Speaker 00

  And it turns out that those proteins are very hard to drug for a variety of reasons. But we like to think that, you know, the technology that we're developing can, can get good material. And so we do, in fact, have selective material for a lot of these cytokines. And so that's one example of something that's like a very desirable target where of course, like there's going to be much more capital requirements to like get it across the line to actually get it to market, but the interest is sufficiently high enough.

• 1:34:59Speaker 02

  So you're going to ruin the antibody market, basically.

• 1:35:01Speaker 00

  I mean, if you can, like, and it's good for some stuff, like it's great for some stuff. And like, there's a lot of energy being thrown, you know, at those groups because we're good at protein folding, right? And so like, they're making a lot of progress. And maybe more broadly, I would say is like going after the right problem. It is a little bit of a lamppost problem where the things where we can make enough scalable measurements in the life sciences are these early problems. We can't really run a million mice experiment, right? Like, we can't run many millions of phase one trials. And so we're going to get better at that stuff. We're going to start to like go up these different layers of biological abstraction with like better data and better reasoning and physics models. But like, right now, like the, the technologies that we have to really make the number of measurements that we need are going to be like these early approaches, I think.

• 1:35:44Speaker 02

  Yeah.

• 1:35:44Speaker 00

  Yeah.

• 1:35:45Speaker 02

  And I would argue also that you can, if you have an oracle, some AI oracle that says, oh, this, like all the admin properties will be good and this generalizes to exactly this population. So you have this oracle filter, then all of your failures in clinical trials go away.

• 1:36:01Speaker 00

  Yeah. You can imagine a world where you can come up with an org, you have some new target that, you know, some new evidence that covers, and then the oracle could give you small molecule material instantly that you would feel comfortable running straight to like a mouse study.

• 1:36:11Speaker 02

  Right.

• 1:36:12Speaker 00

  Right?

• 1:36:12Speaker 02

  So that requires training on data from mouse studies, clinical trials, whatever.

• 1:36:19Speaker 00

  I mean, maybe. I mean, I could say definitely like admin properties, if you're confident in the target and you need to make sure that it's not hitting all these other off targets. But even then, like that's, you probably don't need clinical results to feel the confidence to run off and run a mouse study right away.

• 1:36:32Speaker 01

  All right. Well, we're on, you know, everyone's favorite pet question. My favorite pet question is what I call the Kosuri paradox of machine learning and discovery. Yes, yes. Okay. So Shree Kosuri.

• 1:36:43Speaker 00

  Bless him.

• 1:36:43Speaker 01

  Great tweet. I love this tweet.

• 1:36:45Speaker 00

  Love this guy.

• 1:36:45Speaker 01

  It was, yeah, yeah. I mean, he, he was on us a few weeks ago. We hopefully will release his episode in the next few weeks. Sadly, I actually didn't even bring this up on the episode. I'm so disappointed.

• 1:36:57Speaker 01

  But yeah, the tweet is basically what is the business model in machine learning for drug discovery? Because in order to build a good machine learning model, you need lots of data and good, high quality data at scale. You clearly have that. But then if you need that to train the model, like what do you need the model for? Because you just go collect the stuff and you clearly can collect this stuff quite well. So, yeah, we talked a lot about data collection. We talked a lot about machine learning and I'm actually kind of curious in some ways, like where does the actual machine learning, is it necessary for your business model?

• 1:37:29Speaker 00

  So, I mean, of course, it would depend on the business model. I think if your only rationale is to go off and get good chemical material for a particular target, then of course, Ria is right, right? And you, you would have it, you could get started. If you had enough chemical material broadly cast across all these other targets in a way that can, you know, kind of drive a deeper machine understanding of the chemical behaviors, then the way by which you optimize those early materials have the potential to go much faster, especially if you can do it at the scale that we've done it, especially if you can do it at like the low capital infusions that we've been able to do this. We have not raised a lot of money at least, and we've screened it in an enormous amount. And then I think getting even, you know, kind of even stepping back, you know, one of the reasons that I brought up this language model stuff is because we are starting

• 1:38:14Speaker 00

  to wonder if it would enable, you know, a much larger sort of like scope of capabilities than just identifying novel chemical material. Like if we get really good at chemical reasoning, then we can get really good at lots of different industries and their applications without having to physically go and screen in all those different domains.

• 1:38:33Speaker 02

  Like, what's an example of that?

• 1:38:34Speaker 00

  Uh, so let's say we get really good at making these predictions, then we can run off and go work in agriculture, like immediately, right? You could go work in veterinary science, you could work off material science, and you don't have to, if you're really good at it, that you don't have to screen all those proteins too, because you have like a good fundamental understanding of this, right? And the same thing can be argued for like all the protein folding companies, right? It's like now that these models are good, there are teams that are going after protein targets that have not been screened yet or are very challenging to screen and getting like really good starting material. Like the Novel guys did it with this, uh, basically like a receptor on cells that has like a little tiny piece of a mutated cancer gene. That's like really hard to do, and they managed to pull it off. And so you can imagine, oh, there could be a world where we do this with small molecule material as well.

• 1:39:18Speaker 00

  And so like, I don't think Sri is wrong in terms of finding like starting, you know, if you're just trying to drug this one thing, you screen a lot, and then you already had the thing, you don't need a model for the thing, uh, but that's not exactly the same as generalizing across all things. And, you know, we are hopeful that if you have enough of this variation in engineered activity cliffs, that you will actually be at a point where you can do all things, in which case, at which point, you don't need to screen it all ever again.

• 1:39:40Speaker 02

  And arguably, you guys, I don't, I don't know exactly how much you raise. So, I mean, you're, for a campaign, you're talking, that's like a couple of campaigns you, that would cost the same amount.

• 1:39:52Speaker 00

  Uh, yeah, yeah. I mean, maybe to put some real perspective on it, um, so we have, uh, developed and screened about 2,000 different protein constructs, um, a total of probably about 5,000 screens in the history of the company across at least six and a half million molecules each, um, which I don't want to get into like a, like a huge numbers game, but it's certainly almost as much as anyone else on earth, maybe more than everyone else, probably not, there's probably a couple of groups that like have, you know, but like the number of mutations, I think it's probably more than everyone else, that seems likely to me. Uh, and so, yeah, like we just put a huge amount of work into making this like very cheap and very scalable.

• 1:40:29Speaker 02

  And if I was doing one at a time?

• 1:40:32Speaker 00

  Yeah, if you went to a CRO and you did one at a time, it would probably cost you a hundred K each, depending on, you know, how much handholding you got.

• 1:40:37Speaker 02

  Right.

• 1:40:38Speaker 00

  Yeah.

• 1:40:39Speaker 01

  You just said five million, six, was it six and a half million molecules?

• 1:40:41Speaker 00

  Yep.

• 1:40:42Speaker 01

  That's not a lot of molecules, especially since it's combinatorial space, you know, everyone will say 10 to the 40 or something, 10 to 60, uh, big numbers.

• 1:40:49Speaker 00

  Yeah.

• 1:40:50Speaker 01

  I mean, obviously you don't need to understand all the chemical space.

• 1:40:52Speaker 00

  Nope.

• 1:40:53Speaker 01

  You just need to make drugs consistently, repeatedly, that's, that's your goal. But is that actually enough chemical space to meaningfully do anything outside of your chemical space?

• 1:41:03Speaker 00

  Great question. Um, so the, the pleasant answer is, um, you know, I said earlier that we have trained models on internal data only, and that we're able to extend and make generalized predictions into KEMBO, which is very different corners of chemical space. So yes, do we think it's enough? Not remotely. You know, it's so part of our roadmap before it is making, you know, larger numbers of chemical libraries in different corners of chemical space and kind of spanning, kind of what I talked about earlier, where some are like these very fine-grained changes to more appreciated activity cliffs. And then the other side, it's like these big coarse changes to try to capture, uh, new proteins and sort of protein shapes.

• 1:41:36Speaker 01

  This is, sorry, not, not a hard question, just, I'm genuinely curious, are you able to predict like potency, like relative potency meaningfully? Like, can you do KD curves, like where you, you know, titrate like more and more larger libraries or?

• 1:41:49Speaker 00

  So people have wrestled with this a little bit in terms of trying to get the KDs directly out of Dells, doing things like maybe varying protein concentration is something that folks do. One thing that we see is, uh, when we, uh, when we do replicas of screens, we get back the same hits and roughly the same order, which makes us, you know, kind of optimistic that you could get these kinds of things. We started to move more into SPR, biophysics related measurements, try to figure out if we can maybe come up with like a, like a potency ladder that we could then be as part of the chemical material that we're screening, uh, to try to capture those things. But I won't say that like we're good at it yet.

• 1:42:22Speaker 01

  Like you can't use like, you can't just use reads as a proxy for KD or something?

• 1:42:26Speaker 00

  Um, I think the relative ranking, yes, but I think reads for KD, rates, reads for KD, not really. And then you start running into weird stuff. So like, there's lots of proteins that we've encountered where it turns out like the DNA barcode itself, there, there are proteins that are not described as DNA binders that we have discovered are. And so you can imagine that that might affect the KDs of things, and you have to figure out how to mitigate those problems and that kind of stuff. So like, yeah, it's, I think the relative ranking is something that's quite doable. I think translating it into KDs is something that we're actively working on, and we think we might be able to get like some reasonable, you know, kind of like sloppy translation, but I wouldn't say that we're there yet.

• 1:42:58Speaker 01

  Okay. This one, this one was pretty brutal.

• 1:43:01Speaker 00

  Oh, great.

• 1:43:02Speaker 01

  The bitter lesson, yeah, I'm just gonna read, this, this one is pure clod, I had no thought in this.

• 1:43:07Speaker 00

  That's great.

• 1:43:08Speaker 01

  The bitter lesson is really the law, what stops a better funded lab from outscaling your Dell engine?

• 1:43:12Speaker 00

  Oh yeah, of course people are gonna ask us this. Um, so, and the reason, like, you know, here's what I would really say, is that both at leash and at recursion, uh, my team has made so many mistakes in accidentally bumping into the cheating modes that just kind of come out of the consequence of doing laboratory work. There are so many of these, and, and so like you have to chase these down, and a lot of it has to do with tracking batch effects with metadata, for example. And so then you have to build a culture at your organization where it's like, all right, we have to track all these things, you have to build tools to track all these things, we have to make those tools frictionless, that the people doing the work actually will record these things. Uh, and then you have to go back and look at all the data back into this founding history of the company to make sure that you're getting really consistent results.

• 1:43:55Speaker 00

  And then when you see big changes, you have to identify what drove them, then you have to mitigate those. And it's just like, it's like a huge, huge amount of work. And we also rebuilt kind of from scratch the selection process, we rebuilt from scratch a lot of the protein expression process, and then we have all these other proxies that we use to evaluate how well the protein production went, and if it's good or if it's bad, and these other filters. And it's just, um, it takes kind of a unique culture and patience and willing to make these mistakes, and then have the team like really think very deeply, uh, about how to address them. And I think unless you've got folks who, you know, really have a nose for these kind of experimental mistakes, it's gonna be pretty, it's gonna take some time to replicate.

• 1:44:33Speaker 02

  Process mode.

• 1:44:34Speaker 00

  Yeah, I really think so. There's just so many things where it can go sideways, and you have to be paying enough attention all the time to identify them when they're going sideways. And there might be things that you can write automated filters for and maybe not early on. Um, and so like, you know, like one pretty good example I'm happy to talk about is, um, people ask us a lot, how do we know our proteins are any good? We have lots of ways to figure this out. I think one of my favorite ones right now is, um, we have noticed that when proteins misfold, and we're able to identify misfolding in very crude cases, like if you put the protein, you know, run it on a gel, which is this protein trick where you try to figure out how big it is, um, and it misfolds, then you can kind of detect that. But if it misfolds a little bit, it could be hard.

• 1:45:16Speaker 00

  There are drugs whose job it is to refold misfolded proteins. We call them molecular chaperones.

• 1:45:22Speaker 02

  Okay.

• 1:45:23Speaker 00

  You can call it creatures too, that's what Sri would call them.

• 1:45:24Speaker 02

  Yeah.

• 1:45:25Speaker 00

  Uh, there's a very famous one that treats, uh, cystic fibrosis patients called lumocapdoor. We have a bunch of chemicals in our chemical library that looks like lumocapdoor, this thing whose job it is to refold misfolded proteins. Guess what kind of proteins it hits when we screen them?

• 1:45:38Speaker 01

  Cystic fibrosis?

• 1:45:40Speaker 00

  Uh, misfolded proteins. Misfolded proteins. That's what it's for. It's all like, and it's in our libraries. And so, you know, like, and we discovered this by accident, like we saw this, you know, these chemical series keep popping up across all these different protein types, like, man, like, what is that? Is that lumocapdoor? And then you start looking at the protein expression and then like how it went, it's like, oh, it, it goes when, when things aren't going well. And so now we can use it as one filter for if we think, if, if we get a bunch of lumocapdoor hits, that protein probably didn't fall, fold right. We probably don't want to use that as training data. We probably want to rerun it with slightly different conditions and make it more friendly to the protein. Then we'll see what those hits look like. And so like, unless you're paying that kind of attention, like it's really, I mean, it's really hard. Yeah.

• 1:46:25Speaker 00

  And you have to build tools so that you can just look at these things and have people who know to look for these things and recognize them. It's, uh, yeah, it's, it's a hard problem.

• 1:46:32Speaker 01

  Yeah, going back to, uh, everyone who touches Dells comes out traumatized, somehow you've, you've figured it out.

• 1:46:39Speaker 00

  Probably supported this.

• 1:46:40Speaker 01

  Yeah. You're also traumatized, but in a good way.

• 1:46:42Speaker 00

  Yeah, no, it's really hard. That's, you know, and that's why. It's, and, and, you know, when you have the motivation to just finding like one or two really awesome hits to start off a drug program, like, then you're not going to chase down like all of these different aspects of the assay. And so I get it, but like, if you're motivated for it to be machine learning fodder, then like, yeah, you have to pay attention to all this stuff.

• 1:47:00Speaker 02

  One of the questions that we always ask at the end is, if you could buy Fiat, remove a bottleneck, what would that be?

• 1:47:11Speaker 00

  No, when I think about buy Fiat, and it's one of these things that is actually currently happening, which is watching the sequencing costs drop like a rock. Like, I'm thinking about, okay, what do we really want to do? We want more chemicals and we want more measurements. Those are the things that I want to make my life a little bit easier, like the problems that are immediately on the ground.

• 1:47:30Speaker 02

  So you want, sorry, I don't understand.

• 1:47:33Speaker 00

  Cheaper sequencing, more molecules.

• 1:47:34Speaker 02

  Okay.

• 1:47:35Speaker 00

  Yes.

• 1:47:36Speaker 01

  And sequencing, like, can you give some context for how much it's dropped?

• 1:47:38Speaker 00

  Oh my God. Like many, many, many orders of magnitude. It is, uh, I think.

• 1:47:44Speaker 01

  The first genome cost a billion dollars, and now we can do a human genome for a thousand, right?

• 1:47:49Speaker 00

  I think it might be a hundred.

• 1:47:50Speaker 01

  A hundred.

• 1:47:51Speaker 00

  It's really cheap, yeah. There were stretches where it was dropping faster than Moore's law.

• 1:47:54Speaker 01

  Yeah.

• 1:47:55Speaker 00

  Was, was, yeah.

• 1:47:56Speaker 02

  And what, so wait, well, why, then why do you want to even cheaper sequencing?

• 1:47:59Speaker 00

  Oh, I just, I'm just, I'm just happy with the trend.

• 1:48:01Speaker 02

  Okay. What would you remove by Fiat if you, like, that hasn't been removed?

• 1:48:06Speaker 00

  Uh, yeah, here's a good thing, what I would want is, there are different gizmos that you can use to kind of purify small molecules, okay? So when you, uh, right now, when you make them, you, you put some stuff in a pot, and then you, and you heat it up, or do some other conditions, and you get the output, and, and that will work at some level of efficiency. Might be great, might be crappy. And so in either case, especially if it's crappy, there are these other gizmos that you can use to kind of clean it up. So people use like HPLC or some of these other tricks. None of them are super parallelizable that are really, really good. And so being able to like clean up stuff at a massive scale would be really, really nice, because like we, you know, what, what do we want to do? We run all these molecules in this funky assay that we've already described, has given people lots of trauma.

• 1:48:49Speaker 00

  It would be great to then get, you know, large numbers of molecules in a different assay, that'd be really clean and purified and easy to work with, uh, and then go do that next, because then you could really orthogonally validate all those measurements that you're making. So, yeah, that'd be really, really nice. And of course, everyone complains about like GPUs and stuff.

• 1:49:06Speaker 02

  Yeah. Everyone says like, besides GPUs.

• 1:49:11Speaker 00

  Yeah, yeah.

• 1:49:13Speaker 01

  All right, so.

• 1:49:15Speaker 02

  And then the other question is, um, what would you like people to take away from that episode?

• 1:49:19Speaker 00

  I would like people to take away from the episode that like, you know, at least for us, we, we believe that we're building something that's really, really exciting and, you know, like, what feels like physically grounded substrate for all sorts of interesting reasoning and biomedical applications. And so if that's of interest to them, like, please reach out to us. Like, we're, you know, we're hiring machine learning people, we want to talk to partners, and so, you know, we're building this thing that we think will be transformative. So come talk to me.

• 1:49:42Speaker 02

  Amazing.

• 1:49:43Speaker 00

  Cool.

• 1:49:44Speaker 01

  That was great. So much fun.

• 1:49:46Speaker 00

  Thanks, dude.

• 1:49:47Speaker 01

  Thank you for coming.

• 1:49:48Speaker 00

  Thank you.

• 1:49:49Speaker 02

  Yeah.

• 1:49:50Speaker 01

  Yeah.