⚡ An Eval for AI Scientists That Knows the Answer Key - Ian Quigley, Leash Bio

Watch the full show on YouTube: https://www.youtube.com/watch?v=vhPCo-8K78U

• 0:04Speaker 00

  Hey there. Welcome to Lane Space Science. This is Brandon and RJ, joined with Ian Quigley from Leash Bio. We recorded an episode earlier this summer with Ian where he talked about the cool things Leash was doing with DNA encoded libraries and how they could screen millions of compounds against thousands of proteins in a high-throughput way. Around this time, Ian realized that they could do fun things with these data. It essentially allows them to create like a virtual sandbox where agents can explore and make decisions in real time as if they were medicinal chemists and get instant feedback, but with like a huge amount of exploratory rollouts. Since that episode, Ian and Leash have developed this idea much

• 0:48Speaker 00

  more fully. And since then, they have been there now releasing a paper and a dataset and a blog post along with the episode that we just released. We wanted to sit down with Ian, talk a bit more about what he's done and, you know, hear about the updates. Anyway, welcome to the podcast. Thank you, Ian. Yeah, so last time we talked to Ian, you were saying that you had some agents exploring a sandbox and you were like, oh, this is just a fun idea. We don't know what we can do with it, but it's an interesting idea. I think you've since fleshed this out into a full environment for agents to explore, maybe do RL on, you know, scientific reasoning. Can you talk a bit more about that?

• 1:30Speaker 01

  Yes. So the thing that I talked about last time was this idea that we had these large collections of screens and that they were big enough environments and counterfactually dense enough that you could take agents and drop them in there and then give them tasks and they could go do those tasks on a limited budget in this landscape that we had measured completely. And so the flagship problem that we spent a lot of time over the past couple of months on is a structure activity relationship problem. And the idea there is like, okay, I've got some molecules and I know that they bind, maybe they bind to this protein that I want them to. How do I change them to make them better? This is a very common task for medicinal chemists to do. And SAR, as it's sort of abbreviated to, also includes other

• 2:15Speaker 01

  stuff like certain drug-like properties, you know, does it get metabolized, all these things. We're not doing those right now. We're just doing, can we find molecules that bind better to the thing that we're interested in. And so what we do is we take model, we take an agent, like a language model, and then we say, here's a catalog that you could look at. You have a budget of 100 molecules. We're not going to tell you anything except the target and go find the best binder you can. And so different agents try different stuff. They have different search policies. They have different ways of wanting to spend their budgets. And what they'll do is they'll put out in readable language SAR hypotheses about why they think these molecules are working or not working. And then they'll

• 2:56Speaker 01

  go in and with this limited experimental budget, they'll test those hypotheses. And because it's all pre-measured, you can roll out a lot of these. We're up to about 90,000.

• 3:05Speaker 00

  So you started doing these on foundation models. I think you have like a paper or a little a pre-print or something at this point that you can share. Do you want to share that or talk about it?

• 3:13Speaker 01

  Yeah. So the pre-print will probably be out at the same time that this episode comes out. It is pretty ready. And what we, what we show in there is that we get great separation of agents across model families and at the level of that kind of stuff. But another thing that's really interesting is because we have not just the results that you could use as an eval for how the agents do in the job, we also know the results of the entire measured landscape or world. And so then you can start to interrogate, how good is the agent at making scientific decisions? Because maybe it only got to like the thing that's half as good as the best thing in the entire environment. And so there's all

• 3:53Speaker 01

  kinds of interesting behaviors, like agents that give up early, ones buy molecules that don't agree with their SAR hypotheses, ones that keep retesting the same molecules over and over despite having a limited budget. And also when it's good to explore versus exploit. And it turns out that more performant models are much better at abandoning a kind of unproductive neighborhoods. And that was actually a really interesting surprise for us. Yeah. And so when paper comes out, we're going to release a bunch of these environments as harbor packages and then many thousands of transcripts for people to go play with and go see how these are being, you know, how their, how their science is doing.

• 4:30Speaker 02

  That's super cool. So do you have a feel for why the smarter models are able to figure out that they're in a bad landscape and move, or a bad part of the landscape and move?

• 4:41Speaker 01

  We don't know that at all yet. All we know is that they're abandoning these crappy neighborhoods more quickly. And we definitely see a progression of the, like a more willingness to jump around overall for more performant models, but they especially abandon the lousy places fast. And then the really poor ones stick around.

• 5:00Speaker 02

  It's not just that they jump more, it's that actually they're some, they're figuring something out and saying, hey, this is not good. Let's get out of here.

• 5:06Speaker 01

  Exactly. Exactly. Yeah. And what happens with the, the dumber models, it does, they'll just keep running little subtle changes on molecules that aren't ever going to go anywhere. Right? Like, oh, these two, they have a low score, but I feel good about them. And so I'm going to keep trying stuff. And then, and then the smarter models will abandon the entire process. We're going to go some further.

• 5:25Speaker 02

  Have you looked at the rollouts to see if it's thinking about whatever?

• 5:29Speaker 01

  Yeah. Think about all kinds of stuff. They'll say, you know, when, when we release all these transcripts, like everyone should look at, one of my favorites to do is just take, you know, large numbers of transcripts, just dump them in LLMs and just have a look at the different failure modes and stuff like that. But you'll see, yeah, I think that this ring is good. And I think that this particular nitrogen is painful or, you know, whatever. And they'll, they'll say these things. And then they're very plausible, you know, medicinal chemistry narratives. So it's pretty, it's pretty fun.

• 5:57Speaker 02

  Yeah, that's amazing.

• 5:59Speaker 00

  I'm curious, have you given humans these same questions?

• 6:03Speaker 01

  I have to have them look and evaluate, like, not a ton, but because, you know, we have, we have a large number of them. But when we get medicinal chemists to look at it, they're like, no, those are very plausible thoughts about, you know, how to conduct this particular part of the SAR content.

• 6:17Speaker 00

  Maybe I'm, we're thinking, could you make a game out of this? Is this sort of a game, right? And then medicinal chemistry, the video game, and then you can put people in these environments and give them, yeah.

• 6:28Speaker 00

  Medicinal chemistry captcha.

• 6:33Speaker 01

  I like this plan. We did do a game a little bit like this for Name That Chemist. We would have to probably boil it down to have a smaller number of building blocks. Because right now, to have a chemist go in there, you can imagine that if you're a medicinal chemist, you might have this suite of things that you might want to try over time, like put a methyl here, like move this around. And when we have these pre-measured environments, we are going to be restricted to the things that we did actually measure. And so then we have to be asking the medicinal chemist, oh, well, you can't do whatever you want. You can just do a lot of stuffs.

• 7:05Speaker 00

  And they have to know what stuff.

• 7:07Speaker 01

  They have to know what the stuff is. Yeah. So we'd have to figure out a way to surface it to them in a meaningful way, but I kind of like where you're going with this. Yeah. Like you could definitely shrink it down to a smaller number of options. So.

• 7:19Speaker 00

  What model is best? Or the frontier models? What are the best ones?

• 7:23Speaker 01

  Let's see. I was looking at the plot a minute ago. It's, it's kind of embarrassing that I forgot already. Right now, the champion is Astra 6.

• 7:32Speaker 00

  Nice, nice.

• 7:32Speaker 02

  Not, not a big shock.

• 7:34Speaker 01

  Not a big shock. Sonnet 5.5 is actually pretty good, too.

• 7:39Speaker 02

  Okay.

• 7:39Speaker 01

  Yeah. And Sol 6.1 is also pretty good. So.

• 7:42Speaker 00

  And everything 5 is just a raising pile of crap, right?

• 7:49Speaker 01

  Yeah. Sonnet 5 is both very expensive and not very performant.

• 7:52Speaker 00

  Yeah. Yeah. But yeah, I think 5 will be long considered a big mistake by Anthropic. 5.5s are pretty good. Yeah.

• 8:00Speaker 01

  Yeah. No.

• 8:01Speaker 00

  Do you have a plus 5.5?

• 8:02Speaker 01

  We don't yet. No, that's, that's one thing that has actually been a little infuriating as you guys can appreciate. So when we talked last, two months ago, we're starting these rollouts to see what happened. And then there's been a new frontier model drop, what, like once every 11 days?

• 8:16Speaker 00

  11 days. Yeah.

• 8:17Speaker 01

  Yeah. So like every time I'm, oh, we're going to ship this, we need to drop this public stuff in like some other new stupid model shows up. It's like, ah.

• 8:26Speaker 02

  Yeah, they ship faster than Anthropic.

• 8:28Speaker 01

  Yeah. Yeah. Right. Good, good luck, Ian. So yes. So we are excited to put in more in here. And we also have run a large number of, of the open source ones as well. So we've thrown a bunch of smaller quins in there and GOMs and KIMI and Deep Seeks and that kind of stuff. So there's a lot of these.

• 8:48Speaker 00

  How do they do?

• 8:49Speaker 01

  They're, DeepSeek V4 Flash was pretty good. Not that bad. Yeah. Maybe I should just send you guys like the full chart, how things are going.

• 8:57Speaker 00

  Yeah, we can just pull this up on the screen.

• 9:00Speaker 01

  Put it into, pick it up there. I can say, it kind of looks a little bit like, you know, like the Arc AGI 2 plots where you've got some crappy models. Right, right. It does, it does kind of look like that. And that's kind of why we made it like that.

• 9:15Speaker 01

  And then I think one thing I mentioned earlier that has been very interesting to us is about this time last summer, we had started an internal project to do supervised fine tuning on some of these language models to get them to understand smiles because none of the existing frontier models were any good at it. So like none of the anthropic or or open AI models at the time could even convert smiles to non-canonical smiles very effectively.

• 9:43Speaker 02

  So just for non-chemists, smiles is a language that you can use, string language that you can use to describe accurately a chemical, right?

• 9:53Speaker 01

  Yes, yes, that's right. I think we mentioned it a little earlier when we recorded last, but it's a string and molecules can be forked or circles or both put together, right? These different things. And so a smiles is basically a graph traversal across the molecule written down into a one-dimensional string. And so you can imagine starting the graph traversal at a different point in the molecule. And so there's multiples, smiles, ways of representing the same molecule. And so you, it's nice to be fluent in transferring from one to the other. It's a pretty easy, there's a couple open source packages people use like RDkit to do this. But we wanted the language models to do it just because it would go faster and they weren't as good at using tools on their own back then and nobody could do it. And then around February, like a lot of other things,

• 10:38Speaker 01

  it became clear that they could start to actually work with chemistry in a meaningful way, which is a reason we started going in this direction.

• 10:45Speaker 02

  That's really interesting because that means that it's internalizing some understanding of the physical nature of molecules.

• 10:55Speaker 01

  I think a little bit, yep. Like these are the tasks that we're having these things do right now. Like they are iterative tasks where they're looking at molecules and they are learning something about those molecules to make choices to make better molecules next.

• 11:11Speaker 02

  Yeah.

• 11:12Speaker 01

  And we remove the information about the molecules, like these decisions really suffer. So they're, they're learning something. This is very interesting. Or at least they're using it. I guess that's a better, we're not, we're not changing weights here, but they're definitely using the information as to make future choices.

• 11:25Speaker 02

  You mentioned all these, this dual models, right? That are, you've tried. So I wanted to ask, have you just used like one of these Bayesian hyperparameter algorithms to, to, as like a deterministic baseline for your problem?

• 11:42Speaker 01

  We think it's too spiky of a landscape and it's very high dimensional. We're not really, we don't do that, but we mostly think it's not going to work. I think it's a party line. Yeah.

• 11:51Speaker 00

  I know. You plan on releasing all this? So I guess we'll put the, you, wait, what are you releasing? You said you're, you're going to have a pre-print blog and then some data rollouts and everything? So we can put all this in the show notes? Is there anything else?

• 12:03Speaker 01

  Yeah, that's it. Yeah, I think, I think that's the big thing. And that's probably something closer to like, I guess, I guess what I would probably say is we have built this thing where we, if you've got scientific agents and you want to know how they're behaving, what mistakes they might make, what, how they might learn from physical feedback, you know, if a new training recipe improves scientific decision-making, we've got the capabilities to measure these things at scale.

• 12:32Speaker 02

  Yeah, that's really cool. Do you have plans for other similar problems for this same kind of thing?

• 12:38Speaker 01

  We do, yes. We have broadly, the way that we think about this is it is a lab in the loop with amortized latency. So it's like, go out, you record the thing once, and then you can have a huge number of rollouts through that thing, effectively infinite rollouts. And it, we are thinking hard about other types of biological or scientific questions that you could transmute into these kinds of environments. Because if you're doing conventional lab in the loop, it does take a long time. And, and because those things themselves have limited budgets, it's also not counterfactually dense. So like you gotta keep running it and do different choices with every rollout to make the things, to make the agents learn.

• 13:24Speaker 02

  Right. But with your, your multiplexed strategy, then that, then you can just amortize.

• 13:30Speaker 01

  You could just amortize. So it's, it's, it's not quite accurate to say it's a lab in the loop with zero latency, but it's pretty close.

• 13:38Speaker 02

  This is the same thing that Lila is doing.

• 13:41Speaker 01

  Yep.

• 13:41Speaker 02

  I mean, it's not the same thing, but they also do experiments.

• 13:45Speaker 01

  Yeah.

• 13:46Speaker 02

  And then they use that in their rollout, but they're not doing them live.

• 13:49Speaker 01

  Right, right. Yeah, yeah. Doing it, doing it in the rollout, one step at a time, offers a lot more freedom because you can choose almost anything. And here you're restricted to the number of choices that we made when we pre-measured the environment.

• 14:02Speaker 02

  Yeah.

• 14:02Speaker 01

  That's definitely true. But it's like, it's still a pretty big number of choices.

• 14:06Speaker 02

  Yeah.

• 14:07Speaker 01

  Yeah. And then we have, right now, we're, we have, we've screened about a thousand proteins against a couple of these chemical libraries since we started leash. And we're going to start to grow that. I think we're going to try to get to maybe 20 of these new environments a day. That's a plan for us to try to get to the beginning of maybe 26 or 27. We can, we're doing two a day right now. Yeah, I think we could probably get to 20 a day without too much lifting. So we'll, we're pushing. We'll see if we can do it.

• 14:33Speaker 02

  Wow. And so you're going to have this huge RL environment.

• 14:37Speaker 01

  Yeah. It seemed like, it seemed like the thing to do. I mean, kind of informally, it seemed like where a lot of these agents are making the best and the fastest progress is where there's instant feedback that's very scalable.

• 14:52Speaker 02

  Yeah.

• 14:53Speaker 01

  It's really hard to do that in the physical world, but we think that there's probably a handful of biological questions that you could re-rebuild in this way, go after them. And even, yeah. And then we don't yet know if we're able to take reinforcement learning progress from these kinds of environments, move on to new biology yet. But you know, like, we're trying, we're going to find out.

• 15:19Speaker 02

  That's, that's the obvious, like, how, how does this transfer to other biology? You can imagine if you did it for some sort of covering set, right? That you might be able to get things in between though, that you could transfer to pretty well.

• 15:31Speaker 01

  Yep, yep. And then also, one thing that I need to do is talk to some more decision people. Because a lot of this is about how do you spend a limited experimental budget? And then what can you do to do better at that?

• 15:45Speaker 02

  Yeah.

• 15:45Speaker 01

  Yeah. Beyond just mid-just like chemistry, right? Just like, yeah, when do I stay? When do I go? How do I, how do I get better at that?

• 15:50Speaker 02

  Interesting. Yeah. You can imagine it, if it knows how to connect a decision at that point to a knowledge that it has, then it could learn that skill for other problems.

• 16:00Speaker 01

  Yes, yes. So we are, we are incorporating more and more structural tool calls and things like that with this as well. So it's a, it's a pretty fun time right now.

• 16:07Speaker 00

  Yeah, we'll be sure to link to all that cool stuff you have coming out. And yeah, thanks for taking the time to chat.

• 16:13Speaker 01

  Cool, cool.

• 16:13Speaker 02

  Yeah.

• 16:13Speaker 00

  Yeah. Or thanks for taking the time for a follow-up chat. I think this will actually appear back to back with the rest of the episode.

• 16:21Speaker 01

  Okay, this was great. Well, thanks, guys.

• 16:23Speaker 00

  Yeah.

• 16:24Speaker 01

  Always fun. Yeah.

• 16:25Speaker 02

  Bye.