Breaking Boundaries: AI CoScientist to Accelerate Science Research with Gabe Gomes, Professor at CMU

In this episode, Nathan sits down with Gabe Gomes, Assistant Professor at Carnegie Mellon University, and researcher behind Coscientist: the first non-organic, intelligent being to design, plan, and execute a chemistry experiment. They discuss how Coscientist allows scientists to use natural language to control remote experiments on Emerald Cloud Lab, how it will democratize scientific knowledge, and accelerate the pace of research. If you need an ecommerce platform, check out our sponsor Shopify: https://shopify.com/cognitive for a $1/month trial period. We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. LINKS: Autonomous Chemical Research with Large Language Models: https://www.nature.com/articles/s41586-023-06792-0 --- SPONSORS: Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off www.omneky.com NetSuite has 25 years of providing financial software for all your business needs. More than 36,000 businesses have already upgraded to NetSuite by Oracle, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you're looking for an ERP platform ✅ head to NetSuite: http://netsuite.com/cognitive and download your own customized KPI checklist. X/SOCIALS: @labenz (Nathan) @gabepgomes (Gabe) @CogRev_Podcast TIMESTAMPS: (00:00:00) - Episode Intro (00:05:00) - Introduction Gabe Gomes, his research (00:15:02) - Emerald Cloud Labs for remote chemical experimentation (00:15:18) - Sponsor: Shopify (00:22:40) - How AI is addressing the pain point of chemistry research (00:29:12) - Sponsors: Netsuite | Omneky (00:31:00) - Democratizing science and access to chemistry techniques (00:38:00) - Modular CoScientist architecture (00:46:00) - The power of CoScientist to address more than just chemistry (01:17:00) - The explosion of the context window (01:30:00) - AI Safety (01:45:00) - How is Copilot already helping expedite chemistry and scientific research? This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected].

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  At 6:00 AM on the study after that janky prototype, I woke up with this idea. Okay, there is a way for us to go from natural language all the way to the code that allows for running these experiments. And then you sent screenshots and messages those lack saying AGI is here. And I'm like, you cannot joke about this. I came to the office right away to see what he was talking about. He had put together stuff that showed us there is lightning a bottle here.

• 0:30Nathan Labenz?

  And I probably weighed out 4.5 milligrams of palladium acetate, like a thousand times over the course of a year. And I just thought to myself, man, this should be automated. You know, like I am not learning that much. I would just dream of the ability to have a machine do that.

• 0:51Speaker 00

  The idea of cloud labs is that you have not so different from cloud compute, where you have, let's say, a warehouse, where instead of just running CPU clocks or GPU clocks, now you have hundreds of different types of instruments and hundreds of copies of those instruments, as well as technicians and robots that can perform the operations. I am so excited about what we're going to be able to do because we are not encumbered by, you know, having to worry so much about these things that took a lot of time, but you're not fast enough. It's going to be awesome.

• 1:24Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life and society in the coming years. I'm Nathan Labenz, joined by my co-host Erik Torenberg. Hello and welcome back to The Cognitive Revolution. My guest today is Gabe Gomez, professor of chemistry and chemical engineering at Carnegie Mellon University and author of the recent Nature paper, Autonomous Chemical Research with Large Language Models, which describes the pioneering and highly influential co-scientist system

• 2:08Nathan Labenz?

  that he and his graduate students built immediately upon the release of GPT-4.

• 2:14Nathan Labenz?

  This episode was super fun for me for a couple of reasons. On a personal level, I studied chemistry in college. And as an undergraduate research assistant, my job was to optimize a palladium-catalyzed organic chemistry reaction for yield. In other words, I explored the space of possible configurations for the reaction to see which would produce the desired product at the highest rate. It was, to be honest, mostly brute force grunt work. And over the course of that year, I spent most of my time weighing out small amounts of fine powders. I used to joke that I felt more like a low-level drug dealer than a scientist. And only once, over the course of a year, did I observe an unexpected result that led to meaningful new knowledge.

• 2:57Nathan Labenz?

  Often, as I worked, I would daydream about a future in which all of that could be automated. To hear how Gabe and his team have used GPT-4 in combination with a remote-controlled life sciences lab called Emerald Cloud Lab to automate a significant part of this work, and to see how they applied it specifically to the optimization of a palladium-catalyzed organic reaction, was for me uncanny. If this technology had existed back then, I might have followed through on my plan to become a chemist. Far more importantly than my story, though, is how this paper, along with just a handful of others released over the course of 2023, represent the beginning of the AI-powered automation of science.

• 3:39Nathan Labenz?

  While large language model breakthrough insights are still exceedingly rare, with appropriate prompting, scaffolding, and affordances, AI systems can now generate new knowledge. This is already working at a level that can create leverage for graduate students, and it sets the stage, in my view, for the next generation of models to do much more still. However long it may take to develop them, the advent of AI systems that can advance the scientific frontier is generally considered to be a major milestone and a critical threshold, both for the potential upside of accelerated progress in science and medicine, but also for the unpredictable dynamics that may develop as AIs begin to introspect and self-improve.

• 4:22Nathan Labenz?

  Professor Gomez, as you'll hear, is extremely enthusiastic about the benefits and not too concerned about fast-takeoff scenarios. But he is very concerned about the potential for misuse and abuse of these powerful systems. As always, if you're finding value in the show, we appreciate it when you share it with friends. I would suggest sending this episode to a friend in the sciences whose work might potentially be accelerated by a system like Co-Scientists. Now, I hope you enjoy this conversation about using large language models to advance the scientific frontier with Professor Gabe Gomez. Gabe Gomez, Professor of Chemistry and Chemical Engineering at Carnegie Mellon University. Welcome to the Cognitive Revolution.

• 5:03Speaker 00

  Hey, Nathan. Thanks for having me. It's pretty awesome to be here and listen to your podcast. And yeah, let's start some science.

• 5:09Nathan Labenz?

  I'm excited about it. So you have recently had a paper, which I saw the first version of back in April, now officially published in Nature, which is obviously an exciting accomplishment for any academic group. And what you are doing, I think, is really exploring one of the most important frontiers in AI today, which is the question of, to what degree can current AI systems, and obviously, you know, we will think about this a lot for future AI systems as well, to what degree can they, quote unquote, "do science"? This feels like a critical threshold to me, and you have shed some of the most valuable light on the topic. So congratulations on that work. And maybe you want to just

• 5:54Nathan Labenz?

  start off by kind of, tell me how you did it so quickly from GPT-4 release in March to getting the first version of the paper out in April and having one of the papers that really has stood the test of time over the course of 2023.

• 6:06Speaker 00

  Yeah, thanks for that. And, you know, there's lots and lots to say here with regards to where machine learning has been, which role has it been playing in science and engineering and so on, far, far, far before LLMs. You know, with respect to this one in particular, what happens is my group had been working on fine tuning large language models for scientific tasks. We started working on that in October 2022. And, you know, we were having a very hard time. We were trying to, you know, do things with GPT-2 and BERT and so on, and trying to tune in those things for a few tasks that we haven't published yet. But, you know, we're, we're gonna be putting that out, all symbolic

• 6:48Speaker 00

  understanding of scientific experiments and so on. And we were kind of frustrated with that. And then ChatGPT came out, GPT-3.5, and, you know, I started to see, okay, like, there's quite a few things here that are quite interesting. I was at a conference that was very small and one of the amazing people that have been also working on this, Andrew White, now at Future House. He gave a talk. He didn't talk about any of his red teaming work. But he was very, it seemed like he had seen the face of God or something, how much he was interested in this.

• 7:25Speaker 00

  Fast forward to the day that the white paper for GPT-4 came out was March 14th, 2023, if I remember correctly. And that was a super busy day anyway, but during the day, one of my students and coworkers here and co-founder, the new boy called, sent some screenshots of the, of the white paper for GPT-4. And I, I honestly thought, it's like, is this a meme? Is this a joke? Or what is this about? He also sent the PDF and I remember being home like two in the morning on my phone for some reason, just like this, sitting on the couch, reading the, the white paper and having my mind absolutely blown away.

• 8:08Speaker 00

  Specifically with the example of, there was a, a task that the, one of the red teams did that GPT-4 had to try to convince someone to do something with TaskRabbit and it was Soviet capture. And said, oh yeah, I have a visual impairment. Can you help me? And it worked and I was just like, this is insane. And then all the other stuff. I'm now this time knowing that was Andrew with red teaming in terms of chemistry and so on, all of the coding capabilities. And, you know, my group has been working, we are relatively, actually I should say, we are very new group. We started January 2022 at Carnegie Mellon.

• 8:50Speaker 00

  And I, it's a small group, but we have been working a lot on the idea of automation and machine learning applications to developing new chemical reactions and understanding and so on and optimization.

• 9:02Speaker 00

  I guess we were very lucky because we were already with a lot of these things brewing in our minds. And I started to tinker with it with the API for fun. And I had like this absolutely janky prototype of something that is not even what end up becoming a scientist. It will be another paper coming out this month. And I was like, wow, this is really impressive and we may be able to automate a lot of the annoying work that we do. And this, I guess, really comes at the core of it. I was obsessed with a problem that we, I thought would face here, which is the following. CMU has the very first academic cloud lab in the world.

• 9:45Speaker 00

  And it's a facility that you can think very much as AWS, but for scientific experiments over two different types of instruments. That you submit your experiment as code and then it's performed by a mix of technicians and robots. And it really, you know, it's about fifty million initial, fifty million dollars initial investment that is here. This is joining partnership with a startup that started in South San Francisco from two CMU alum called Emerald Cloud Labs. And one of the things about this is that for you to write the experiments, you write as code this fork of mathematics and so on.

• 10:26Speaker 00

  And that was the thing that I was worried about because a lot of physical scientists in, in biology and in chemistry and sometimes also material science, folks there are at the bench running synthesis and experiments. There is a barrier for them to go in code. And especially if you like not talk about stuff that is not as widespread as Python, for example. And I was worried that that would hamper adoption. So at six AM on the Sunday after that Friday janky prototype, I woke up with this idea. Okay, there is a way for us to go from natural language all the way to the code that allows for running these experiments.

• 11:09Speaker 00

  And I sent this message to my trainees, the Neil and Robert, co-authors of this paper.

• 11:16Speaker 00

  And I also sent this janky prototype that they end up not even using because it's too bad. Like, I'm not a good coder. The next day, somewhere else on campus, and Daniel sends screenshots and messages, those like saying, AGI is here. And I'm like, you cannot joke about this. And he's like, come to the office. And I, I, I came to the office right away to see what he was talking about. And, you know, in just this little, less than a day, really, he had put together stuff that showed us there is lightning in a bottle here. Because we now can really leverage how to go from English to experiments.

• 11:59Speaker 00

  Once we saw that, it was non-stop push-through to get this pre-print, this first pre-print ready. We had a lot of challenges. I was traveling at a wedding and submitted a pre-print on like the, like the night before the wedding, to make sure that it was going through and it did, and it went. And my life has completely changed since that. The reason why we were able to be so fast, and especially because I need to emphasize this every time I talk about any of this work. The, the, the new Robert, they are absolutely fantastic, hard-working individuals. And Daniel, it really is, is a genius.

• 12:45Speaker 00

  We didn't have a lot of the amazing tools, like even LangChain back then, right? It was just us putting together a lot of the tools that then would become straightforward things that people use nowadays. But they didn't exist. And we had to make our own things. And because of those constraints, we end up being able to make something that works very well. As you said, it stood the test of time for, for the task that we, we had in mind.

• 13:11Nathan Labenz?

  The time scales on this are incredible. I mean, so March 14th, as you said, original GPT announcement day. You know, at, at that time, most folks didn't even have API access. Your pre-print, I'm looking now, is originally dated April 11th. So you've got less than a month from the announcement to the, the first publication, which is, is pretty incredible. And I think just, you know, kind of a reflection of like, How fast everything in AI is moving? How how much sort of a model advance opens up, in terms of like capabilities that are yet to be explored?

• 13:53Nathan Labenz?

  It was another friend actually of mine who suggested, why don't you see if it can write Emerald Cloud Lab code? So we did try that a little bit back in in the sort of September, October timeframe as part of the red team and reported like, hey, this thing does seem like it can generate pretty reasonable synthesis protocols for at least like common stuff. My own chemistry knowledge stops at the undergrad level. And so I could only kind of push it so far. And the other limitation that we found, I think interestingly, this is kind of increasingly looks like it's to Emerald Cloud Labs credit. I was not able to find any documentation online that I was able to use to inform that work. So I just asked it straight away in that early, early version

• 14:38Nathan Labenz?

  to write Emerald Cloud Lab code. And it just totally hallucinated what that code would look like with like a very realistic syntax. And this was one of my kind of early adjustments moments where it was like, OK, wait, but is this real or not real? You know, like I could tell enough that like the synthesis of aspirin looked right. The, you know, stochiometry or whatever seemed, you know, to a first first read to be correct. But like, was this the actual Emerald Cloud Lab code? I really had no idea and I couldn't find any documentation. At one point in time, you would have thought, well, who cares about that? And now it's like, actually, that might be might have been a prudent move not to put that stuff too far out there in the open.

• 15:17Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 15:21Speaker 00

  At least when we decided to do this, there was quite a bit, if not all of it, documentation was online. But because we have been working with these folks already before. And for example, Robert in my team already had like done work with Emerald. He knew how to navigate this pretty well, but certainly was online. And I remember because that was kind of a little challenge that we had to, you know, parse things in a certain way so that we could do well. It wasn't called RAG then, but like it's called RAG now. Again, we were fortunate that in my lab, we had these robots from Open Trons.

• 16:01Speaker 00

  It's a small startup that makes very affordable liquid handlers from Brooklyn. And all every single thing from Open Trons is open. Open hardware, open software, open APIs in Python. Documentation is amazing and we leverage that to quickly do the initial test that we wanted. Turns out this is quite funny. So GPT four, when it like came out, was able to write some code for Open Trons that would, that would run. But because there had been some updates in between GPT four coming out and the version of Open Trons APIs that we had, some stuff would like would not run.

• 16:46Speaker 00

  And it's like, oh yeah, I just need to fix this, this and that. It's like, wait, how can we update GPT four retrieval? And which, you know, we went to do what now is RAG. But it was that that showed us that's like, okay, we can really do this for Cloud Lab as well.

• 17:04Nathan Labenz?

  Fascinating. Okay, so let's take a minute and just make sure people have a sense of what it is we're talking about on a couple of different dimensions. I want to make sure we sketch in a little bit more detail what Emerald Cloud Lab is and does and why that's valuable. Then we can kind of describe the system that you now call co-scientist that you set up and the different components of that. I want to get into like what it can do. I've got questions about your kind of mental model of like the different thresholds on the way to, you know, more and more AI automated science. And then, you know, we can discuss kind of the likely implication of that for all of us. But starting with just Emerald Cloud Lab, I think this is a huge deal. In its own right, that like, almost nobody has heard of.

• 17:49Nathan Labenz?

  And, you know, one of the things that you do, not to, not to jump ahead, but one of the tasks that you manage to get co-scientists to do is to optimize a palladium catalyzed reaction. And you could tell me more about this particular, you know, palladium catalyzed reaction in a minute, but I'll first just tell you a story, which is that when I was an undergrad, I was a chemistry student and I worked in a lab for a year under Professor M. Christina White. And what we were doing was optimizing a palladium catalyzed reaction. So I'm literally doing, as an undergrad research assistant, the same type of task that you have now got co-scientists to help with.

• 18:33Nathan Labenz?

  What was ultimately the reason I got out of chemistry was how small the ratio of intellectual work to literally weighing out small amounts of fine powders was my time. I sometimes analogize, I'm not a big analogy person, but, you know, to give people an intuition for the sort of work that I was doing, I sometimes analogize it to the, the game Battleship, where you like, you know, say, A2, you got a hit on my ship, right? And then the next thing you gotta do is kind of explore around that hit and, you know, until you can, you know, get the whole ship, right? Well, in, you know, reaction optimization is basically the same thing. When I got there, they had already got that first hit.

• 19:15Nathan Labenz?

  They had discovered that there was a way to use palladium to catalyze this particular oxidation reaction. And that was like, okay, Nathan, you're, you know, just finished your sophomore year, you don't know anything. Here's what we do. We do like, you know, arrays of, basically pick a variable, you know, vary it and run them all at the same time in the same, you know, warm bath and, you know, measure yield and, you know, we gradually grow up our way around through chemical space and optimize. But what was truly insane to me, and I'd sat there dreaming of something like Emerald Cloud Lab, was I had these little tiny vials and these little tiny, you know, spatulas and these containers of powder. And I can still remember it.

• 19:58Nathan Labenz?

  It was 4.5 milligrams of palladium acetate. That was the, that was the first reaction that would go into every little vial. And I probably weighed out 4.5 milligrams of palladium acetate, like a thousand times over the course of a year. And I just thought to myself, man, this should be automated. You know, like I am not learning that much. You know, it was it was economical as in as much as I was like an undergrad. I was just, I would just dream of the ability to have a machine do that work. And, you know, at the time, those machines were maybe starting to happen, but they were not happening in my, you know, they were not ready to displace me.

• 20:40Nathan Labenz?

  And so maybe just tell us a little bit more about Emerald Cod Lab, because I think it is even without the AI component, the ability to go in and program that kind of stuff to happen. The work that I did in a year, I mean, it seems like it's at least an order of magnitude compression that could have happened just with Emerald Cod Lab, you know, no, no AI required.

• 21:03Speaker 00

  This story is really fascinating because it's the it's a very common story for so many people that, you know, go into chemistry, go through that experience and and then, you know, think to themselves, why, like, why are things this way? Or maybe not, right? Because if you look at pictures of most of the things that we had in society of, let's say, last hundred years, right? Cities changed, landscapes changed, how we work changed. And if you look at picture of someone in a lab a hundred years ago, and you look at picture of someone in the lab now, it's not going to be so different, right? That is because so much of it comes to this manual labor

• 21:46Speaker 00

  that is just, as you said, sometimes not intellectually really stimulating. So Cloud Lab, the idea of Cloud Labs is that you have not so different from cloud compute, where you have, let's say, warehouse, where instead of just running CPU clocks or GPU clocks, now you have hundreds of different types of instruments and hundreds of copies of those instruments, as well as technicians and robots that can perform the operations that goes into those instruments. And Emerald is one of the pioneers. Emerald Cloud Labs is one of the pioneers in this, right? They're one of the first. Companies doing cloud labs, there are others.

• 22:30Speaker 00

  And I, I should say for disclosure, I am the AI Secretary for a board for Emerald Cloud Lab. That is the idea that now you can have from anywhere in the world, you write the code for those experiments. And they will be performed by either people for, you know, technicians, the for tests that we do not have really good automation for it yet. For example, weighing solids, unfortunately, weighing solids, solids is a task that's a lot harder than it seems. Things are getting better, but we're not quite there yet. But like dispensing liquids, that's for sure. You know, Sophie, that we do not need people to be doing. We have many, many, many unaffordable robots that can do that.

• 23:13Speaker 00

  This is what really leads to this compression that you mentioned in terms of how much time you would need to take to ask a question. Right. The group that you mentioned, Christina White, is one of the best groups in the world of developing these types of transformations. They are critical to develop better therapies, better materials, a better and more comfortable life for us all. That is strongly rooted in the fundamentals of organic chemistry. And yet so much of it is so slow. And like you said, in the game of Battleship, so much of it is still, you know, got ahead.

• 23:56Speaker 00

  Let me try to do, you know, all that, one factor at a time, you know, all variable at a time and try to optimize around there. This is something that I would say the more modern approaches to reaction optimization no longer go that way. They will use something like Bayesian optimization, right? This is Sophie that the many of us in the field have been doing for a few years now. This is also where machine learning really started to shine on experimental chemistry of a past, let's say, half decade to decade. You really touch on the nerve of the problem here and what has been bothering me so much. Imagine if you cannot go to the lab because you have some.

• 24:41Speaker 00

  You have to take care of a kid or your parents, or you just cannot go to the lab. Does that mean that you cannot do science? Because I don't think so, right? Your intellectual focus here should be on thinking how to push those new reactions for those interactions for not on ways solids, right? And that is what Cloud Lab and automation, I believe, strongly bring into democratizing science for many more people. And I, you know, I'm an organic chemist by training. And one of the, a subfield in organic chemistry

• 25:23Speaker 00

  that exists is total synthesis, which is where people essentially, you know, there's a natural product. They'll go and try to make that because that might have therapeutic properties to, let's say, try to cure cancer, a type of cancer. The amount of work hours at the bench that these folks put in is astronomical. It's really amazing work, but it's extremely labor intensive to the point that it slows down that progress and creates a culture that is just not really healthy or sustainable. So how can we change that? How can we really change that in a way that now we have machines

• 26:06Speaker 00

  that can do a lot of this grunt work? That's the idea of Cloud Labs, in my opinion. And, you know, for the optimization aspects that you were mentioning earlier, machine learning plays a massive role now. Like, it really does. I don't see how one can go and try to optimize a reaction without using these techniques. That would be just a waste of, a waste of time, if I may put it that way.

• 26:29Nathan Labenz?

  I wonder how often, because I saw this too, my, with Christina, Professor, and Christina White, the, the work wasn't on total synthesis, but, you know, the next group over was doing total synthesis. And so I did see some of the, you know, challenging work culture that you mentioned. And sometimes that really was dictated by the chemistry itself. You know, you'd have these sorts of things where it's like this thing needs to be cooled gradually over 12 hours or, you know, there's no law of nature that says that, you know, the, the, the timing of these reactions like needs to line up with your childcare hours or whatever. So it's like, it often doesn't and, you know, people were like running reactions overnight. And, you know, you can imagine when it fails or it doesn't, you know, yield what they want,

• 27:14Nathan Labenz?

  then they gotta do it again. And so it does become a real grind. I saw that up close. I wonder though, in the process of kind of reducing this to something that can be fully explicit with code, I imagine there must be some percentage of the time where for some reason that doesn't work, you know, like there's, there's some aspect of what is going on in the lab with the human, with the hands on, where there's kind of knowledge that, you know, is not even necessarily fully conscious on the part of the human that is allowing them to kind of make the reactions work. And then when you put it into code and you sort of specify this is what's supposed to happen, you know, hey, whatever, it's just, it's not working. How big of a problem is that from what you have seen?

• 27:59Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 28:03Speaker 00

  You're, you're really hitting some really heart touching questions here. And I would say that this one in particular, something that I have been thinking about quite a bit because I, you know, I talk about these things in a way that I want us to democratize access and, you know, imagine if you never, if you didn't experience that lack of intellectual stimuli for ways of this, right? You, you probably would be a chemist still, you know, what is, we have been singing the praises of, you know, manipulating bits and for the past 20 or 30 years, like really worshiping at the altar of Silicon Valley, but man, chemists and chem engineers and

• 28:47Speaker 00

  especially nature has been manipulating atoms, right? That's what catharsis. And when it comes back to.

• 28:57Speaker 00

  Thinking about these tasks that perhaps are not quite automatable, or perhaps there is a lot of intuitive knowledge that you don't quite know how to translate from, you know, the human way of doing something to now an automated fashion. That is fine, right? Like, there will be time that we'll go gradually and be able to implement those things further and further. And this is why I say this, because in the past, folks that were, you know, pushing automation and M L. Chemistry tended to be a little bit too overconfident and push a certain hype that I really dislike.

• 29:42Speaker 00

  We need medicinal chemists that are extremely well versed in what they do, to continue what they do. But we also can simplify a lot of the things that are mundane and routine for them, so that they can be faster at what they do. And with time, we'll develop better hardware and software that can accommodate for this. I do not expect in any way, shape or form that we'll be able to totally automate certain techniques that, or even, you know, certain little aspects that don't go into a supplemental information explaining how chemists actually did this. And in some other ways, you have folks like Professor Lee Cronin, University of Glasgow,

• 30:26Speaker 00

  and the hardware that he developed with computers and so on, that is more of an adaptation of how chemists have been doing this for so many years, that is closer to how what you described as medicinal chemists do those tests would be. So there is a, you know, there is a gradient here. Nothing needs to be black and white, and nothing is. And I do believe that this is important for us to think about. The next steps for coscientists too, right? I am thinking a lot about human machine interactions and a lot about embodied artificial intelligence,

• 31:06Speaker 00

  however you want to describe that, agents, in order to be able to perform more and more complex tests.

• 31:13Nathan Labenz?

  So that's a perfect segue for me to describe what the coscientist project consists of, and you can elaborate on this, but I think for folks who listen to the Cognitive Revolution, the general structure of an AI agent is probably like increasingly familiar at this point. This was an early version of it, but it basically, you know, has become kind of the default structure. You look at the diagram in the paper and, you know, I've seen dozens that look very similar over the last however many months. So basically you have kind of one central, and I'm interested in how you think about it. Is it all one agent or is it like a team of individual agents?

• 31:54Nathan Labenz?

  But you've got essentially one central planner module, and then you've got its kind of surrounding four major affordances. And those are the ability to go on the web and get information, the ability to write and execute code, a lot like an OpenAI code interpreter, the ability to search specifically through the documentation, if I understand correctly, for Emerald Cloud Lab, and then?

• 32:24Speaker 00

  Other hardware as well, not just Emerald, but like really any hardware that we give the documentation for can be implemented in either, it's from minutes to hours to, you know, a few days. Now we are very good at it, so it can be done very quickly.

• 32:43Nathan Labenz?

  Cool. And then finally? The actual access to the APIs to make calls to the physical world. And that's one of the things that I think makes this so interesting, too, is like, it's just a for people who sort of think, you know, robots or killer robots or, you know, it's all very fantasy. Here's a setup where there is already, you know, the kind of API ified physical instantiation that's going to do the bidding of whoever's calling the APIs. And that API doesn't really know whether it's a human or an AI calling it. And, you know, probably, you know, maybe increasingly, but certainly originally, you know, didn't have any need to figure that out, right? Because there were no AIs calling it in the early days.

• 33:25Nathan Labenz?

  So I'm going to just make a guess and you can kind of correct me if I'm wrong here. But as I understand it, basically what happens is the scientist comes to the co-scientist and you're, you know, kind of, I guess, taking inspiration from like co-pilot there, right? So this is something that a scientist can kind of work with in real time and get, you know, help from. And you say, hey, I need a, I want to synthesize X compound. And, you know, I want to, I don't know how much you have to prompt it or how specific the kind of scaffolding already is in advance. But, you know, ultimately I want to send program to Emerald Cloud Lab to synthesize this compound. Your job is to kind of take it from this natural language to those end API calls. Typically, these agent frameworks kind of have a loop.

• 34:07Nathan Labenz?

  So I'm assuming, you know, the first thing is like, well, let's think about that step by step a little bit. Let's maybe go, you know, do a search on the web for known methods of doing this. Write some code to do like the calculations for how much material we're going to require. We know that language models obviously are not great at doing, you know, stoichiometry multiplication. In fact, one of my reports to, as a red teamer to GPT-4 was that, to OpenAI about GPT-4 was that GPT-4 is the world's worst chemistry tutor. Because when I got

• 34:38Nathan Labenz?

  Confused and I was like, you know, I was role playing with it, right? So I would, I could role play it as a confused student and got the stoichiometry wrong. And stoichiometry is basically just like the relative proportions of, you know, reagents or inputs that you need to have to get the right mix on the other side, right? So.

• 34:58Speaker 00

  Relative proportion of inputs for your output here. Like let's, I like that.

• 35:03Nathan Labenz?

  Anyway, when I would get confused, it would get confused. So running code is a really good way to kind of clarify that. Like it understands the logic pretty well. Okay, you've got, you know, I would even just do very basic stuff like CH4 plus 2 equals what? And like trying to get that reaction to balance. It understands the principles, but it can get tripped up in the numbers. And especially if the numbers are gnarly, it's not great at like doing that kind of arithmetic. So what it is good at is writing code to execute that stuff. So, okay, let's like do some analysis, make sure we have the right amounts. Okay, now it's actually time to go, we kind of have, we know the steps, we know the amounts. Now we're gonna, again, this, at each step, right? It gets returned to the planning agent. The planning agent sort of says, okay, cool, we got that update. Now we know how much.

• 35:46Nathan Labenz?

  Now let's go look into the hardware documentation and see what, you know, what machine to use and what they're capable of. And then finally, issue commands to the Emerald Cloud Lab so that it can actually do the thing. What am I missing or what would you want to elaborate on?

• 36:04Speaker 00

  You got it. A couple of things that I would say is that we really thought about this to be in a very modular approach. Our initial intent was for autonomy to, so we really did not want to have something that would be originally too conversational, but really just, you know, here's the, the task, go for it, right? And we went, you know, we went, we really went pretty wild with it, which was a lot of fun. You were correct about everything here. The things that I want to add is that in the execution of experiments, or it doesn't have to be Cloud Lab. In fact, you know, it often in, in our lab here is, it is not.

• 36:47Speaker 00

  It's just whatever hardware you, you have available to run the experiments that you want to run and.

• 36:57Speaker 00

  One of your questions was if I think about it as a collection of agents or one agent. And this is also something that I see people getting constantly confused about the capabilities of large language models, because essentially they think that, you know, should go to ChatBT or Bard or Claude and you ask for this task. And the, you know, chatbot or the LLM that you use cannot perform the whole thing end to end. That is not good. And that's absolutely not the case at all. When you start to think about even how we think in terms of system, system one, system two, so on,

• 37:39Speaker 00

  like we have to have some way also of trying to help the LLMs like process information that way. So when we started this, one of the immediate things, even with my Janki prototype, was that you would have something that would be a planner and would have something else that would do all the tasks. And of course, I'm so happy that you brought up the problem of arithmetic and so on, because that was one of the key ideas that the new brought up, which was giving it the ability to run code in a Docker environment. So one would allow for us to do all, you know, simple math without having to worry about,

• 38:20Speaker 00

  about, you know, the shortcomings of LLMs. So you have the planner, that is, you know, an LLM. Hopefully, well, not hopefully, but ideally, that's going to be the strongest LLM that you, that you have access to. And then the documentation search and the web search, also LLMs. Many of our initial prototypes will try to use GPT 3.5, for example, for those tasks. And, you know, we had some shortcomings. Maybe nowadays with a mixed router, other really powerful open source Llama two, maybe that would not be an issue anymore.

• 39:03Speaker 00

  The other two components, code execution and experiments execution are not LLMs, they're tools, right? In this process, the whole thing is co-site. Now, I would be remiss if I didn't acknowledge the work of many other amazing people working on this, specifically Andrew White and Philip Schaller. Philip Schaller is a professor at EPFL in the Department of Chemistry, is a great friend and collaborator now. Literally today, one of the students from Philip's group arrived at my group and is going to be with us for six months. And Andrew is also a friend, mentor, collaborator. They had ChemCrew, that they saw it as showing the capability of LLMs to be able to use

• 39:47Speaker 00

  tools for chemistry applications, chemical informatics and so on. We released the two pre-prints back to back on Archive together. And you can see that, you know, our approach that then we didn't call co-scientists, all agents or something, was we want something that will be autonomous and to it. Now we have, you know, chat mode and so on, it's many other things. This, this is, you know, how, how I see it. There are many things that you didn't make to either the pre-print or the final paper that is absolutely fascinating to see, especially if you are an experimental, experimentalist, artist, you know.

• 40:31Speaker 00

  For example, when we prompt scientists to perform those complex polymer catalyzed reactions, the same thing that you and I that never have, like may not have never done something is that you will start to search the web. First move, I don't know, Wikipedia, like Reddit, it's like, what is this thing? And it keeps digging, keeps digging. And the planner in, well, let's say, conversation with the web searcher will have that back and forth of like, I need more information and, or no, I'm good now. And once the planner is satisfied, quote unquote here, then it goes, okay, I need to perform the experiments. This is what I have available in terms of hardware. Let's say it's the Open Trans OT2 in Gabe's lab.

• 41:15Speaker 00

  So I need to go and now do the calculations for volumes or whatnot that uses the code interpreter, code execution, I should say. There wasn't code interpreter back then. And We'll do the calculations, and this is the part that I find so funny. Because sometimes we will make little mistakes of variables. We, Python will give an errors like, okay, corrects that, makes a little mistake, corrects that. Once it gets all good, then pass that on to the planner back. And the planner now knows, quote unquote, knows that you should use the documentation searcher or communicate with the documentation searcher about what functions and what you need to write

• 42:00Speaker 00

  the code to execute those experiments with the code execution. That is the workflow for certain tasks that you can have analysis of results on the fly with this. That's where things get really beautiful because now the planner takes those results and can say, oh, you know, this didn't work as, as, as expected. Let's change this, this and that and go try again, right? Not differently from how you did as an undergrad and a lot of scientists do research in their, you know, I, how I did things up until not so long ago.

• 42:44Speaker 00

  So I really do see this massive change now where you don't need to learn the exact nuance of the code that goes into Emerald, Cloud Lab, or know everything about the Python APIs for, for Open Trons or, or Hamilton and so on. No, you, you need to have your idea of what you want to do and that's it, right? That is what is so powerful here, in my opinion.

• 43:16Nathan Labenz?

  So let's talk about the power a little bit and then I want to get into a little bit more of the details. The power meaning like, what can it do? There are six things, six kind of, you know, distinct results listed in the paper of. Things that this co-scientist has been able to do from, if I understand correctly, just kind of a relatively brief natural language prompt like synthesize aspirin or whatever, to on the other end, like aspirin comes to exist in physical form in the world. I'll read the six and you can kind of comment as we go wherever you want. So number one, planning chemical synthesis of known compounds using publicly available data. That would be like your aspirin synthesis is well-known, go find it, calculate the ratios and go.

• 44:01Nathan Labenz?

  I call that text to protocol.

• 44:03Speaker 00

  Nice, yes.

• 44:05Nathan Labenz?

  Okay. Number two, efficiently searching and navigating through extensive hardware documentation. We've kind of covered this, I'd say, already, but just to emphasize, for folks that have not spent time in chemistry labs, there are a lot of different machines and they have all sorts of different buttons on them and all sorts of different details, right? Even for machines that may have conceptual similarity, there's a lot of different, where does the sample go and what are the requirements, what sizes, how much, blah, blah, blah, blah. I had a mercifully limited crash course in that stuff myself, but a lot of machines. Anything to add on that?

• 44:42Speaker 00

  One of the things that when building co-scientists as what we wanted it to be, we never intended it to be something for chemistry only, right? That is a reason why I decided to call it co-scientist, not co-chemist. It's because indeed we are able to do a lot more than just chemistry. It's totally field agnostic, which means that it goes back to these six tasks, let's put it this way, that we demonstrated in the final version of the paper are simply to say that are just examples and we decided to focus on chemistry because we are chemists to show these experiments.

• 45:27Speaker 00

  But I can tell you that we have done a lot of things that are not published, they are in biology or material sciences with co-scientists that we had absolutely no problem with. Now, going back to the hardware documentation, it's not just hardware, but also software that is very specialized software that people will take years to master. And now, like, no, like you just...

• 45:56Speaker 00

  English and it works great.

• 45:59Nathan Labenz?

  Okay, cool. So yeah, let me, let me read the list again, but also this time I'll kind of abstract away from the, the chemistry context a little bit. So planning chemical synthoses of known compounds, that's in general, kind of orienting yourself to like the common, you know, knowledge of, of the literature or the current state of the literature. The next ones are all about these kind of tools, whether it is searching through documentation, hardware, software, issuing high level commands. Notably, you reported in the paper that one of the machines that GPT-4 was able to use was released after the training data cutoff. So you could be confident that it was figuring out how to use that machine based on documentation that was just provided in the

• 46:44Nathan Labenz?

  context window and not relying on its pre-training, which is an interesting distinction at a minimum. Definitely has some implications, I think, for, you know, AI proliferation, you know, as well. And sort of to what degree, you know, these, when you release something now, like to what degree do you have a ability to predict, you know, what that's going to mean in the future? Low level commands to the machines as well. So kind of high level, more conceptual type stuff, but then also like down to the, the micro controlling. Five is kind of putting that all together into complex tasks that basically combine multiple aspects of the first four. And then the sixth is optimizing. And this is the, you know, the battleship type work that

• 47:27Nathan Labenz?

  we've already talked about. That one seems to me like where you actually get to the level where like you're displacing me as the undergrad research assistant.

• 47:37Speaker 00

  Uh, hold on, hold on. Let's, let's, let's talk about the first five and separate from number six, because number six is quite different from the rest. I'll, I'll give up say. So the first five are essentially, in my, in my view is how can you have this very natural language, for the lack of a better term, approach to how we think about doing these tasks that, as we mentioned earlier, are either very labor intensive or require a lot of qualification or a lot of time to be learned, right? And that is, you know, that is all, I would say, in the realm of everything that are.

• 48:21Speaker 00

  Maybe not even state of the art anymore in terms of large language models. And that on its own, to me, is a massive change. Because I can't now, in terms of cognitive revolution, I take a lot of the cognitive load that would be used to, you know, oh man, how do I like do this call to this API in this way and so on? I think, how can I actually think in terms of breaking and forming these bonds? Or how can I actually think in terms of doing this DNA split for whatever task I'm trying? It's true that we were very adamant about making sure

• 49:01Speaker 00

  that it could generalize to things that were not in the training material that goes into GPT-4. And RAG really plays a massive role in this. And it's fun to now have a word for it, which we took augmented generation, but back like that with it. Now, point five, which is the one that combines different aspects and different hardware to develop a workflow, a plan and a workflow that then you can tell cos scientists to do something and will autonomously plan, design, execute and analyze what that, those experiments, that those results come in.

• 49:43Speaker 00

  That is where, you know, that things really gel in terms of it's much more than just, I give English here is code, right? Because this is a part that, let's say, you have a good idea. Let's say you are a chemical biologist and you are working in trying to understand how certain cancer cells grow. And the way for you to do that is by using a fluorescent tag on that. But the chemistry to do that, to do that conjugation, to put the fluorescent tag on the surface of the cancer cell is something that you do not know how to do yourself.

• 50:26Speaker 00

  No bother, because you read the teacher and you know that there's some ideas out there and you can ask cos scientists, hey, this is what I'm trying to do, let's go. And coscientists can either be in a conversational mode that will assist you even in the lab without robots, how to do that, or in an autonomous mode if you have the hardware for that. So now a biologist that otherwise would have to wait until someone to develop a whole new protocol or, you know, kits to do that conjugation can do that instantly, or as this is the experience will allow, to continue their work on tracking those cancer cells and advance their research.

• 51:12Speaker 00

  The time that you already saved is tremendous. And you can think about the flip side, which is, you know, imagine a chemist that knows how to do all of the synthesis, but may not know how to work with cell lines and so on. Training is complicated. Surely they can go and learn this or rely on cloud labs, but they also can use coscientists to assist them in the lab and try to do those experiments. This is what I see also as an important democratization of science that we are not paying attention to in how AI can, modern AI can assist us. And then we have point six, which is solving these optimization tasks. That is something that my group is very focused on, developing and optimizing

• 51:55Speaker 00

  complex chemical reactions with halogens for, you know, developing materials that, or molecules that are part of materials or CRPs and drugs and so on. And this is where a lot of chemists in, let's say in Harvard, for example, will spend a lot of time developing to both make, to scale those reactions and to make them as cheap as possible.

• 52:23Speaker 00

  And don't forget that, you know, there is a lot of variables, as you mentioned earlier, that goes here. So for many years, people tried to do scientific experiments and it's one factor of time and so on, and this grew and grew and grew. And Gaussian processes, of course, Bayesian optimization is one of the most common approaches to this. My group had been working on implementing more and more chemical information about the mechanism of those reactions into the kernel that goes into Bayesian optimization and so on. But we thought, well, what if we gave coscientists, this is a task, but

• 53:05Speaker 00

  You know, in order for us to actually do this well, let's write it as a game. So we really took inspiration from all of the reinforcement learning approaches that folks at DeepMind and others had done about this. The game is the following. There are, you have 20 iterations. Your goal is to maximize something. In this case, yield, and to simplify the metrics, we use normalized advantage, which is just a different way of doing the math for yield. So with that goal, here are your constraints. You have access to this list of chemicals and this list of variables.

• 53:51Speaker 00

  So for one of experiments that we had, or computational experiments, I should say, that we had in the paper, it had over 6,000, about 5,500 possibilities. And if we were to ask a good chemist to do this, they would probably first, you know, see what results there are available, what kinds of things are similar, and then start to poke around and see what is possible. More modern approaches, like we've been developing, and what, for example, the N S F Center for Computer Systems Synthesis, a part of, does is to use those modern techniques, data analysis, Gaussian, Bayesian estimation.

• 54:33Speaker 00

  We thought, well, what if cos scientists can just do this? The task is to optimize the reaction, and here's the list of what you can do. Here's the game. And we let it, we let it go. And we did this many, many, many, many times over. This one figure in the paper, the last figure, it was by far the most expensive experiment, computational experiment that we've done. It was like over I don't even remember the numbers, but it was by far, far, far the most explicit thing we've done. Because we really wanted to show that this was, we wanted to see if it was possible first and foremost.

• 55:13Speaker 00

  And not only it is, but it also has some advantages that, to be quite honest, I didn't think about until we were doing it, which is when we do Bayesian optimization and other, you know, techniques like that, you do not have a very good a priori reasoning for why the model is picking this point in the, in the fitness landscape. And that is fine, you know, like you can retroactively go and understand that if you have enough domain knowledge of the subject. But what Constellate just offers you is as simple as let's do this because of this, this and that.

• 55:56Speaker 00

  Uh, that reasoning that we call, and I'll put in quotation marks, chemical reasoning, is something that's really, really useful because now it says, well, let's choose an additive that has a group that is more electrowithdrawing than the one before and keep everything else the same. What, very similar to what you did as an undergrad, right? And we'll go and perform that and see, okay, the yield went up. So in it, in that moment, the planner now quote unquote knows that this is something that helps in improving the yield. You may hit a bump, a threshold or anything, uh, that very much like you and I were to do this would,

• 56:39Speaker 00

  but it's extremely good at the strategies that it puts forth. Uh, so of course you can think about combining this with some of the techniques we talked about and so on, but on its own, it was quite impressive. And this is again, another thing that I was, that I brought up that you also mentioned. We don't need to be spending so much time at certain tasks that are often not cognitively interesting or intellectually stimulating because we can essentially now put those aside for cos scientists or whatever to do them while we're thinking about other more interesting impactful things. That is to me what will help. So I do not see this

• 57:24Speaker 00

  as, you know, removing anyone from workforce or anything along those lines. I think this is a tool and it's a powerful one that can really be helpful in accelerating how it does.

• 57:36Nathan Labenz?

  Let's talk for a second about just kind of all the different variables that the that the optimization process can include. Then I want to kind of talk about a little bit of just some of the techniques that you use to build this thing and then come back to the sort of, you know, results and kind of, you know, implications. In terms of just optimization, you know, it's a very high dimensional space. I'm just thinking back to the one reaction that I was optimizing. You have like, which chemicals are you going to use? How much? At what point are you going to add them in time? What's the temperature going to be? What's the pressure going to be? What's the light conditions going to be?

• 58:17Nathan Labenz?

  Are you stirring it? When are you stirring it? How much are you stirring it? You know, going back to like what chemicals like that, you know, that's a whole rabbit hole unto itself, right? Like in my thing, you know, I made one very minor contribution to science over the course of that whole year. I guess if you want to give me the Edison credit of finding all the things that didn't work and that I could get maybe, I could claim more discoveries of things that didn't work. But in terms of new knowledge of things that did have useful, you know, that actually moved yield forward and it was like a conceptual discovery, it was one in the whole year. So, you know, just incredibly vast space. And I have to say,

• 58:58Nathan Labenz?

  I would guess that GPT-4 is better than I was at kind of guessing what those next trials should be. I had the help of my advisor at the time, who was like, you know, and specifically, the one thing that I found was, we had a couple of different versions of the reaction. In the one version, it was, had already been kind of discovered that like, more acid would like, push it forward toward higher yield. But then, in the other version, we kind of were working under that same assumption, and tried to, you know, do an experiment to kind of demonstrate that. And the slope of the line was actually the other way. And so, I printed, you know, printed out the result, and I was like, huh,

• 59:40Nathan Labenz?

  it looks like it's going the other way. You know, have we tried less acid? And sure enough, you know, we like, so, my advisor was like, well, here's what we should do. Let's try the least acid that we can, that we can try and see how that goes. So we tried it, and sure enough, that was my one contribution to science, right? Like, in the other version of the reaction, less acid was better. So anyway, it's just a huge, you know, space with all these different variables. Twenty iterations is not a lot. I probably did, you know, I don't know, five times that much over the course of a year. And again, I was just working on this part-time. So a grad student, you know, would probably do hundreds of these over the course of a year. It does seem like GPT-4 probably is better than I was. Is that, is that wrong?

• 1:00:23Speaker 00

  This is great, because, and first of all, what you said about like, the Sony approach that you did a bunch of stuff that did not work, you know, that is a problem with science, because we need negative, quote-unquote, negative data.

• 1:00:35Nathan Labenz?

  Yeah, that stuff didn't make the paper either, by the way, of course.

• 1:00:37Speaker 00

  Exactly. And like, we need to change the culture of physical science and engineering to actually always include, like, negative data. Because, you know, in doing, for example, Bayesian optimization campaigns, negative data is just as important as positive data, right? Like, it tells us, like, hey, you know, that part of the landscape, no good, like, go somewhere else. Twenty iterations, indeed, is not that many. And we did that on purpose. We really wanted to constrain this, right? And we did a comparison with some not-too-too powerful visualization, you know, Cositis was either comparable or better to, to the BO that we, the Bayesian optimization that we used. So...

• 1:01:22Speaker 00

  I have a bit of a hard time saying, oh, you know, it's better than GPT-4, so it's better than

• 1:01:31Speaker 00

  person XYZ at this because, you know, we can't see that, but then

• 1:01:37Speaker 00

  sometimes it just makes mistakes that are just so, so stupid for different things. So these are very powerful, intelligent machines that sometimes make absolutely stupid mistakes. And, you know, I really want to emphasize that what we built here is not a super bullet. It does make mistakes, obviously, and it's far from perfect. And there's a lot of work to be done, right? But it is pretty good at picking, picking variables and how to, to modify them. Going to what you were, you were listening earlier, right? That you said, you know, you have which chemicals and how much of this or what temperature, how long and so on. So you have, you have categorical variables, right?

• 1:02:22Speaker 00

  Like, you know, which of this and so on. You have continuous variables. So like, you know, what's the temperature? You can vary that continuously. And you, you have ways of taking what are categorical variables and transform them to a space that makes it look continuous. And then you can do these, all of these nice optimization strategies that you can take nice gradients through and so on. Right? That's the, the kind of visualization that we do, that then also incorporates information about the reaction mechanism. That is, you know, the differences here. And we, we do not have Cositis doing this.

• 1:03:04Speaker 00

  And if you were to think about something like, I don't know, let's say we had $2 million for no reason that we went to OpenAI and said, let's fine tune GPT-4 for these types of experiments, right? And It probably would become something really, really powerful that you wouldn't even need to do a lot of the other things that we do to make sure that hallucinations are not an issue, that you can retrieve and so on. I like the comparison, the point of Cam Crowe with tools. Then we, you know, we also give a lot of different tools, something that, you know, as a disclaimer,

• 1:03:43Speaker 00

  the new Boyko and I are co-founders of a company that will build, that will be building a version of this that will be available, commercially available with all the guardrails and all the bells and whistles and a very nice interface. And in thinking about this, we start to give it more and more tools that you not make into the paper because of, you know, how much we've been working on. And suffice it to say that we know now how to give it different tools for different problems that on its own, GPT-4 is not super good at. But once you give it access to these tools, then it's no longer an issue.

• 1:04:25Speaker 00

  In the paper, we had examples of how it wasn't super great at certain very simple molecule synthesis that any organic chemistry undergrad would know how to plan them, right? That's okay. It really isn't a problem because there are so many amazing tools out there for retrosynthesis. Some of them developed by Connor Colley at MIT or Philip Schaller again at EPSL that rely on transformers as well, that can be used by cos scientists to do this. So it's not, it's not like a world in its, in its own. And that is the thing that brings back to even your question earlier, is this one agent or many agents?

• 1:05:07Speaker 00

  We really need to take a step back and think about not just one technology.

• 1:05:16Speaker 00

  Say L L M's or another, say automation. But how those two things coming together are much larger than just some of the parts. That is something that I sorely miss in a lot of the conversations that we see online, unfortunately. And what I seek is really like seeing how those things coming together build the next, that give the next step on that. And that, you know, brings two points that you, that are more technical, right? Like, how is prompting? How is scaffolding? How is the tool use and memory?

• 1:05:54Speaker 00

  And so these are all things that you, if you, if you were to take a trip back to March, when G P T four came out and said, hey, by the way, there would be a way that like you can retrieve memory. Like, people would be by gong. And now it's like, yeah, it's fine. Like, yeah, like, you know, no big deal.

• 1:06:16Nathan Labenz?

  I think like the number one profile of a person that listens to this show is the A I engineer. So these are folks who are, you know, working with the different paradigms that you were, you know, an early developer of with agent scaffolding, the loop, the rag, the tool use, the memory. As you said, like, most of that stuff was in either non-existent or very immature form at that time. Now it's come a long way, but I wonder if you have any kind of tips or big lessons learned? You know, like, were you even using a vector database in the early going? Or were you just kind of using keyword search against existing search? And, you know, just anything that you think stands out as a big lesson

• 1:06:58Nathan Labenz?

  or things that have gotten easier, things that have just kind of changed over the last eight or nine months since your, you know, kind of pioneering agent setup to today's agent standards?

• 1:07:09Speaker 00

  Isn't it amazing that like not even a year ago, we did not have what now is a profession, the A I engineer, and now we have it. And it's making such a massive difference in the workforce and how even we think about things, right? Like how, what, what is possible? Uh, I used to think about, you know, We want to push the boundaries, but we always need to know what is feasible, what is possible. I no longer think that. I think that we can do, if we can do something, if it's doable, then it's also feasible. And that, you know, is something that these superpowers that the LLM gave us.

• 1:07:48Speaker 00

  And going back to when we start to work on this, because all of the really nice tooling were not either not available at all or not widespread. And because we really built all of the initial frameworks at that point, we do not rely on many, too many things. It's incredibly vanilla, in some ways, with regards to all of the amazing tools that folks have developed, like LingCheng. And I remember when Baby AGI came out and I was like, this is interesting. Like, there's a lot happening here.

• 1:08:31Speaker 00

  And of course, you know, at that point, folks were very disappointed. It's like, you cannot do everything on this. It's like, yeah, but like, look at the ideas, right? And more recently, if you look at what Microsoft put out with Auto Jam, for example, that's with agents, it really is pretty awesome. I really want to emphasize that knowing your tools is obviously very important, but it is not the end all be all of anything. It's, I really prefer to think more in terms of what are the goals that you're trying to accomplish? We are incredibly agnostic to what the tools we're going to be using. This is something that I, we're really proud of ourselves in. And it doesn't matter, right?

• 1:09:16Speaker 00

  Like, so we didn't have, there was no vector database that were as good as the things that we wanted to use, that you can use now? Okay, let's figure out how we can do this with. Yes, it was like just no pie like and you know you can do this like it's not it. There's a lot of these things too that look much more complicated than they should be. And that is something that I I really watch push on like. You do not need to do the most sophisticated piece of software in order to get stuff done. Right? And I say this honestly as being guilty of as someone that always wants to push for the most, you know,

• 1:10:00Speaker 00

  cutting edge and my my trainees that paradoxically are the ones that are much more pragmatic. Pragmatic than you really is like, no, you don't need that. Just use this simple thing that will do the work and you will work is like, okay, yeah, you have you have a good point. So for the version that's out and so on, of course, scientists, if you were to take a look. Um, we know how to do tool usage and many others then showed how this can be. Very powerful and like mainly benchmarks, et cetera, et cetera. Uh, similarly for RAGs, similarly for memory, RAG indeed was quite the challenge. There was not the name, right?

• 1:10:42Speaker 00

  So but we had a very problematic uh situation in dealing with with uh. One of the documentations that we're dealing with because it was very comprehensive and very wide. And this is where the weirder things come together. So Daniel, uh, when he was doing his masters in organic chemistry at Moscow State University, he was also an ML engineer at VK. And he worked with a lot of different things, but he really did also do work on search and computer vision and so on. And that knowledge really came in handy because he knew how we could do uh searches across the, you know,

• 1:11:27Speaker 00

  what is not massive space in comparison to really anything uh that you'd think in terms of search engines. But in large enough, there was a problem for uh the design of uh our workflow, our platform. So that was one uh. Well, think that he was able to pull together from things that he knew before. We acceptify that with two different algorithms, but in the end, we have one way of doing things that really works best. Prompting is something that is also very, very, very, very powerful.

• 1:12:08Speaker 00

  And there was the rise of prompting in years, of course, we saw that. And what we see here is that prompting comes very, very closely related with both tool usage and like how good or bad that would be. And with retro augmented duration, right? So we did put a lot of effort into that as well as doing lots of A B to that testing. In order to make sure that we would get consistent results to what we needed. Then we also do a little bit of a mixture of experts in some way to make sure that things are smooth. Let's put it this way.

• 1:12:49Speaker 00

  My tip would be think about the goals first and don't get too hung up on the tools. Because if you, actually a bit cynical saying, just because the reality is that a lot of the ideas that came to mind came when G P T four came out and that's a tool. But once you have that, you know, just go for whatever goal you you try to achieve. That's how we we try to do things here and how we've been doing things. And you do not need the most complicated set of tools to do very interesting work.

• 1:13:25Nathan Labenz?

  I mean, that resonates with me in a totally different context. I will tell this in long form again, because people have heard it. But with the Waymark project of trying to write good video scripts. I find and you know, and ultimately create good videos, which is a multimodal challenge. It is almost like you cannot remind yourself often enough to like pull back, come up for error and be like, what exactly was I trying to do here? Again, it is so easy to get lost in the details of the tools and the the sort of evaluations of different techniques. And, you know, obviously, all the evaluations end up kind of being imperfect.

• 1:14:09Nathan Labenz?

  And there is something really important and kind of, and this part is still a little bit beyond the AI's in many cases, you know, which is notable to like, just do that sanity check and make sure you're not like kind of auto-regressing down the path, you know? It's, it's easy to do that if you don't have that kind of system interrupt and just reminder of like what the goal is. That is a, you know, more of a cognitive tip perhaps than a, than a technical tip. But I do think it is a really important one for AI engineers across like any number of different domains. One thing that definitely stands out is that context window has exploded dramatically since the early work. Has this changed or would this change how you would think about designing a system?

• 1:14:52Nathan Labenz?

  Even at the level of like one agent versus, you know, five sub-agents making up an agent? I sort of imagine a chunk of that division is a function of kind of context limitations and the need to manage context. Maybe that's wrong, but it's at least like part of it, right?

• 1:15:09Speaker 00

  Yeah, so context windows are stuff that, you know, there's some good paper showing that, you know, depending on the models I remember showing that, you know, there's too much attention being paid to what's the beginning at the end. What's in the middle gets kind of like, yeah, whatever. And what I can say is the following, it would not, as of like today, it would not change how we design in terms of this multi-lab of blocks. And so what has changed is for something that we haven't published yet, but it will be coming very shortly. That imagine that you want to now leverage the power of quantum mechanics with computational chemistry for designing new materials, right?

• 1:15:52Speaker 00

  And a lot of those softwares are incredibly tedious to learn and so on. That is where I started and that is my original field to computational physics. And we're gonna be showing soon how this all can be simplified significantly. And that's where context windows played a much more important role than with what we did here for Google Scientists. I do want to emphasize the fact that we did this distribution of work. A lot thinking in terms of how, well, you know, Khan has told us in terms of system of thinking fast and slow and so on.

• 1:16:33Speaker 00

  Take that into consideration to break things down, to simplify how you tackle problems and also try to help your agents to tackle problems, like trying to cram everything into one instance. Why? You know, many cases you do need that, but in many cases you do not. And that's, it turns out that when you can break things into planning and execution and so on, turns out you work a lot better than if you were to try to cram all your needs into one massive context window. The context window stuff is fun for, you know, very large corporates, right? But I don't know, like you have vector databases that can also assist you with that and it works pretty great.

• 1:17:19Nathan Labenz?

  Well, then let's get to finally the implications of all of this. I think one question I have is how you just conceptualize it and I'll offer my mental model and then you can, you know, give me yours. But I think of AI very often in terms of threshold effects, meaning when an AI gets above a certain threshold, the situation is qualitatively different than when it's below that same threshold. I sometimes call that the can-can't boundary. And that's also kind of bound up with notions of emergence, which have been certainly in the air again recently as the emergence paper won the NERIP's paper of the year.

• 1:18:04Nathan Labenz?

  There's one threshold here that we've like clearly crossed from the results of the paper. It seems pretty clear that GPT-4 can do this optimization.

• 1:18:12Nathan Labenz?

  Whereas 3.5 cannot, right? Like, that's just a, and I've seen that in a couple of different cases. I talked on a research roundup episode about the self-improving, like the software improver that then, you know, acts on itself and becomes the self-improving software improver. And that had a similar thing where GPT-4 could improve, but 3.5 actually just got worse, you know, iteration after iteration. They both kind of leveled off, but like GPT-4 was leveling off above and 3.5 was leveling off below. So it seems like there is this kind of, you know, sort of divide where it's like, if you're on the one side of it, things can kind of roll downhill in a, you know, improving way and on the other side of it, you know, it's kind of, it's not happening.

• 1:18:55Nathan Labenz?

  I guess, how, what mental models do you bring to this whole scenario of AI in science? Another one that I have in mind, by the way, I think you kind of alluded to is like, when can a general AI train a narrow AI to solve, you know, problems that it itself cannot solve? So we know, for example, that like AlphaFold can do things with protein prediction that GPT-4 cannot do and that no human can do. But at what point can a GPT-N train, you know, design and train an AlphaFold to solve like whatever problems it needs to solve? I don't think we're there yet, but that certainly seems like another big threshold. Anyway, you can explore that any way you want.

• 1:19:40Nathan Labenz?

  You can challenge my threshold model. You can list out some thresholds you think are important, but I'd love to hear how you think about this kind of escalation ladder that we seem to be on.

• 1:19:49Speaker 00

  You know, if a tool already exists, then yeah, GPT-4 can go in. It will be able to use it. Like AlphaFold, this is something that we are doing now very, very, very routinely. Let me just stop there, okay? We have a project that we call Netflix for AI. So that's what, that's what I'm gonna start with. When it comes to the differences between 3.5 and 4 with what, you know, we immediately saw here, there might be something about smaller, you know, models that would be better at following instructions. Like, one of the things I saw that was quite a bit of a problem was JSON for 3.5 that, you know, for 4 it was not, not a big, not a problem at all.

• 1:20:31Speaker 00

  Yeah, this is the thing with extreme scaling and emerging capabilities, right? Like, a lot of this, we don't know until you have the model. And it makes the whole field to be fascinating because, you know, like what is going to come out of this? Sure, there's a lot of talk about, you know, are we at the limits of the architecture? Maybe that's not the point. The point is for physical sciences, we are absolutely nowhere near having anything that has extreme scaling the same way that large language models. And I say physical sciences, I'm biased, I'm really talking about chemistry. Think about the cost of one image, for example, for, you know, vision, vision tasks

• 1:21:17Speaker 00

  and computer vision models and generative imaging and visions and so on. What is the cost of one image out of being the whole internet versus the cost of one experiment that you do for, you know, chemistry? That gap is enormous. We are trying to solve that gap nowadays by trying to pull together what we think would be an image that moment for chemistry. If we're gonna get there, I don't know, but we're trying to. Because I do see that not only with very general models versus narrow ones.

• 1:21:58Speaker 00

  At some point, we're gonna be able to have GPT whatever to, like you said, just build a enough of folds from scratch if you have enough compute and resources and use that to perform tasks that that would be useful for. We do have that in very, very small early days for very, very small models that are like regression or classification models, first things that we care about in, in what we do in my group. But I would never say that, you know, similar to what you mentioned earlier, that's like this

• 1:22:39Speaker 00

  software that improves itself and continues to self-improve. Nothing like that at all. This is awesome, right? Like, it really is. I see no limits for what we can do. Now, the question really is, what are we going to do with this? Because all of a sudden, we have this alien technology, it's Subway's, that gives us superpowers. Why are we using this to, you know, make things that might be entertaining, but are not life-changing? Why are we not using Claude to help develop the next therapy for whatever, right?

• 1:23:22Speaker 00

  I'm sure there are people working on this, but I want to see much more of that and much less of yet another AI girlfriend or whatever like that. Which is important, too. People need companionship. But I'm trying to say is that there is so much that can be done and needs to be done that AI engineers, please, like, come and like, help us, physical scientists and engineers that do not know enough, that need your help, to make sure that we can do the best applications of physical sciences, biology, chemistry, and material science, and the chemical engineering, electrical engineering, so on, with AI applications that are fantastic. So,

• 1:24:06Speaker 00

  here's what I talked a lot about democratizing knowledge that otherwise would be difficult if you're not a specialist. And something that we had in the blueprint very clearly, boldly, given how things have moved on, I am very optimistic about, is on also the potential for misuse and woods. It may not be something that people like to hear. But we must be responsible and transparent and honest about the fact that these technologies are enabling and it's a tool.

• 1:24:47Speaker 00

  And you know, it's not the fault of the tool that can be used for, you know, unsavory purposes. But we must be future attented, we must be responsible to acknowledge that and make sure that we mitigate the possibilities for harmful applications, not only in the context of what my work is, but also in societal applications and that we are not discussing in the, in the context of this episode. And when we had the A O, the executive order from President Biden in October,

• 1:25:25Speaker 00

  that's, you know, what I, I saw as being a very good path forward as well as the UK Security Summit, and the one that will happen in France and South Korea. And all of the movement around this is really nice. I'm not saying in any way she performed, I really want to emphasize that the only way for us to go forward is for us to develop these, the technologies fast, really. Like the technologies that we're developing here, that OpenAI and so and the other big AI labs, will help us mitigate the possible dangers. I have no question about that. All I'm saying is let's do it responsibly.

• 1:26:09Nathan Labenz?

  Yeah, this is definitely, you know, I'm very much of two minds on this, you know, where I'm an, I'm a major enthusiast and love, you know, the upside. And then I also am like, geez, this does seem like we're getting potentially close to some thresholds beyond which it seems to become very difficult to. You know, to predict what might happen. One, you know, it's affectionately referred to as PASTA, the process for automating science and technology advancement, you know, is is one that like the AI safety community has held out for a long time, I mean, a long time in AI years, anyway, as sort of a threshold beyond which not only like do things change just generally,

• 1:26:54Nathan Labenz?

  potentially very quickly, which is, you know, potentially good, but also can be challenging, but also that if systems get past the point where they are now advancing state of the art knowledge autonomously, then that can also be kind of folded back on itself and creates the possibility for these sort of, you know, positive feedback loop runaway type scenarios. It's like GPT four is not going to run away, but a system that can improve chip design and improve architecture and kind of, you know, establish its own like TikTok type of dynamic, you know, at some point, it certainly seems plausible that it it could.

• 1:27:36Speaker 00

  I'm not worried about that, at least in the now at all. You know, a lot of those count with what ifs that we we as a community simply do not know what I I worry much more about things that are already out in the world that can be used. And I also want to really strongly emphasize that there is. It's my strong opinion that we we should not try to curtail any any developments or or or really thoughtful ideas in large language models and and any of the other Puff technologies or all these, because it's all about intent.

• 1:28:18Speaker 00

  Nothing on its own is bad or good, tools are not, but it's about intent. Right? There is a lot to be compared here, not to the usual thing, in my opinion, that folks will say, this is not like Oppenheimer and folks back in in those days. I see this more similar to the idea of, for example, gain of function research that is very problematic in certain circles, but it's something that can assist with understanding.

• 1:28:52Speaker 00

  What is possible in terms of dangers? Not saying this is what we should do. I'm just saying that there are frameworks that can be done. And by the way, I'm not a policy person, right? Like, there are policy people that are very smart, that work really hard. They are educated to this. There are AI safety people. There are AI ethics people. There are ethics philosophers. Those are the specialists. All I can hope to do is to provide information for those folks in the cleanest and most approachable way so that they can be informed about what needs to be done.

• 1:29:29Nathan Labenz?

  I conceive of my role pretty similarly in that, you know, the scout notion is kind of predicated on the existence of generals, you know, somewhere who hopefully I can help inform. So help me understand kind of what you're, what you're allocating. I mean, I, I think the whole spectrum of AI risks is, is worth thinking seriously about. In the original paper, as you said, there was like a very prominent sort of highlighted statement of, we're not releasing this, you know, prompts, code, et cetera, because there's no guardrails and we are not really comfortable with that essentially, right? And that is in like a smaller font in the final paper, but it's basically still there, right?

• 1:30:08Nathan Labenz?

  As I understand it, like there, the, the details of this, the like underlying implementation has not been released. Do you have a sort of scenario in mind or kind of conditions in mind at which point you would say, okay, now it is cool to release it? Or do you, I mean, I, I'd love to hear more about how you think about, because there's obvious tension, right? Between like democratizing on the one hand and releasing and, you know, under what conditions? Do you have like a scenario in mind at which point you would say, okay, cool. Now I'm like satisfied we can release this and it'll be, it'll be fine?

• 1:30:43Speaker 00

  In the development of the original that we thought to be the footprint, we saw these possibilities and kind of, you know, like you could see a hint from the red team work done by Andrew White. And we pushed that to the very limit. And we saw that we ought to be responsible, but not alarmist about the possibilities here. And a lot has changed since then with respect to how companies, the big AI labs are looking at this, how governments are looking at this. But I will say that for quite a few months, he had a really hard time sleeping because I was worried that,

• 1:31:29Speaker 00

  and I really worried about this, to be quite frank here. And I think that a lot needed to move a lot faster than it has, and it hasn't, and a lot, a lot of it has, like especially when it comes to government policy. I moved a lot faster than I could have thought, really. And I wish there was a stronger discussion with some of these companies. Well, we, you know, we have, we really love these models and the capabilities.

• 1:32:13Speaker 00

  And it's precisely because the upsides are so strong for everyone that we need to be responsible about the potential downside. So that, you know, like imagine if one unsavory case happened and we were to just push away all of the good sides. I don't like that at all, right? So how do we mitigate that? I don't know. And

• 1:32:44Speaker 00

  You may seem like I'm avoiding your question, but I'm not. The point is that I do not know how to answer your question in a way that I would be comfortable with that. We'd have a version of it on GitHub and it's open. Similarly, Chemcrol also has a version of what they did on their GitHub and it's open. But a lot of the things that would make me most worried are certainly not available. Anyway, this is something I take very seriously, that we take very seriously. I am happy to see a lot of the discussions that happen around this. I wish folks also were not so dismissive about certain possibilities, simply because

• 1:33:29Speaker 00

  they make comparisons such as, I've seen X, Y, Z in cybersecurity in the past and this is exactly the same. It is not, or also people saying that this is absolutely the doomsday and it's also not that, like no way. But I, I am not comfortable having some of these things to be out without the types of card rails that we put in place. And it's very nice actually to see even Chemcrol does have a function or a module, I don't remember exactly now, that is about, you know, safety and so on explicitly there.

• 1:34:09Speaker 00

  Similarly, I need to mention, Cloud Labs paradoxically become a very useful way of thinking about this because since everything is traced, the amount of metadata that you have is enormous

• 1:34:25Speaker 00

  that you know the provenance of what's going on. And the folks at Emeralds do have their own implementations for safety and checkings and so on that give a good aspect of this, but I will say definitely not enough. Definitely not enough, far from it. I really would like much more push from AI engineers out there trying to make this work from the, from the big AI labs and.

• 1:34:58Speaker 00

  Massive kudos to the UK AI Safety Institute that is trying to make this a little bit more formal, right? So, and I know that some folks like to criticize Anthropic and Claude for being very careful, but like the reality is that like, yeah, we, we need to be careful. And, you know, many also about Google not for right away releasing these things. Folks, like, let's take a pause here and think about consequences, right? You, we want to push the technology as fast as possible. I want that. We all want that. We want the revolution. But, you know, don't always be so cynical. Think about that.

• 1:35:43Speaker 00

  There's a lot that happens that you may not know about.

• 1:35:47Nathan Labenz?

  That's a big part of why I do this show is I think that there's a strong reason, you know, on both sides of the, the analysis, you know, the, the opportunity to accelerate science, you know, upstream of medicine, obviously incredible. You can't, can't be denied. And yet, at the same time, I do think the potential for things to go a little haywire is, is very, very real. There were a couple of changes that I noticed from the original paper to the final one. One was that you dropped the use of the word emergence, which is, you know, kind of been through the wringer. I'm interested in your thoughts on emergence.

• 1:36:31Nathan Labenz?

  And then another is that you dropped the section on generating ideas about new cancer drugs. And that I think is, is also, it is like super interesting. I'm curious as to why that got dropped. And I'm also curious as to kind of how you would characterize.

• 1:36:51Nathan Labenz?

  Just how good, we talked about this a little bit earlier when you were like, it's, you know, in the context of reaction optimization, it can match the Bayesian optimizer. But I'm wondering, like, in terms of like outside the box thinking, you know, like the kind of question was like, you know, can you come up with a new cancer drug? There's something there that, you know, you're kind of, we're looking for these like eureka moments. And I used to say we haven't seen any of them. And now I feel like we've seen like a few, but precious few. So emergence and kind of eureka moments are my two questions. And I, you know, specifically noticed those through the, the changes from first to nature versions.

• 1:37:30Speaker 00

  I think that the, the eureka moment that came here was like really being able to do a lot of the tasks that, you know, otherwise would take a lot of time. And we, we have this video, unfortunately, it's not public, but it could be comparing a very, a very good researcher against coscientist on a given task. And coscientist finished the task in about four minutes with code that, you know, is ready to be run. And this very good researcher, after about 15 minutes, had code that will not even run. So that I see as like, it's a, you know, a lot of people are like, okay, what did you discover?

• 1:38:14Speaker 00

  Well, what discover is what we are going to discover, like, you know, like it's a different way of doing what I've been doing for centuries. That's, that's what it is, first of all. The main reason why we decided to remove that section is because, you know, preprints are drafts. And that part end up being a little bit out of focus with the overall story of what the paper is trying to, to explain and tell and inform.

• 1:38:48Speaker 00

  And we had to prioritize certain things. The development of, say, you know, asking for

• 1:38:57Speaker 00

  Cancel drugs or whatever. It was a little bit too open question that in retrospect, I think that we it can have a lot of very useful ideas. But I remember laughing pretty hard about like the agent at that point to just going for carabinoids. Because there's so much about it on the internet, I guess. And it's like that kind of sub bias in GPT four that unfortunately leaks through in this. This not explicitly, but that that was my my feel at that point. And then, you know, this is on all that.

• 1:39:41Speaker 00

  And we really wanted to do something that was much more focused. So it was like, OK, let's let's remove this. And that's also a reason why I see the two works almost different, like different works. So a margins. Yeah, so I am not a purist of that word. To to say that I know that there are folks out there that believe that margins is only something that you can talk about in the biological context. Or that others come to say is like, there's no real emergence of whatever capabilities in in L L M's and so on.

• 1:40:24Speaker 00

  I mean, come on, let's just use the word as can we just use the word as a as a proxy for we didn't train focused on this. And now we have these capabilities. And so that's, that's how I see it, right? And that comes, of course, from scaling and so on, the different types of data, different quality of data that goes into into building these models. You see five two like tiny model can do some really amazing things. Right? And are we going to say, oh, you know, five two does not have a merge itself like I don't like capability of solving pretty nice quality level physics problems? No, like, it's fine.

• 1:41:04Speaker 00

  I sometimes I feel like we get caught up in certain words too much without actually trying to think about.

• 1:41:16Speaker 00

  The overall message. And this was one of the words that I will say during the overall, you know, rounds and rounds of peer review and many online and not were pretty, for lack of a better word, upset about it. So I don't care enough to to fight for it. I think that it's more important to say, okay, the work is this, it's a tool, it makes mistakes. I don't think we like no one trained for to be a co-scientist and here we are. So is it an emergent capability? I'll let people decide.

• 1:41:52Nathan Labenz?

  Yeah, exercise for the reader. Okay, so maybe last question. This is a good positive note to end on. You know, toward the beginning, obviously, we talked about my experience as a research assistant and just how much grunt work it really was. Now we have co-scientists, always worth reminding ourselves that this is the worst that this technology is ever going to be. So let's say I am, you know, considering a PhD in chemistry or chemical engineering now. What is my PhD going to look like as I start to have co-scientists, you know, as kind of the new normal?

• 1:42:34Nathan Labenz?

  And I'm interested there in terms of like, how much faster can I go? Maybe also like, how much co-scientist budget do I need, you know, but and how much more can I accomplish in the course of a couple few years of research compared to the before? I'd love to, you know, this could be your pitch for your lab or for the hard sciences in general, but I'd love to hear kind of how you think this will be different now that we have these enabling tools.

• 1:42:59Speaker 00

  Definitely, a lot of things will change. We'll be able to pay less attention on certain mechanics that are time consuming, but not really pushing knowledge forward. So I cannot speak for PhD in chemistry or chemical engineering anywhere and everywhere. I can speak for the groups that I know that are using these things like again, Philip Schaller at PFL, Andrew White, and many others. I should really mention Alana Spuriguzic in Toronto, but my postdoc advisor that is one of the pioneers in. Machine learning for chemistry, period. And they are pushing extremely hard there for massive innovations.

• 1:43:45Speaker 00

  And the PhD experience will change, definitely. And, you know, I see, for example, I hope to see in biology become shorter so that both people can go to the workforce faster because they don't, you know, don't biology, there are the problems, biology is hard because biology, but you get the idea. In chemistry, I, what I want from people that come to my group and, and work here is, what are the big questions that you're trying to solve? What are, what like keeps you so, you know, focused on, on a, on a problem that you want to tackle, right? So there are people in my work, in my group, working on things that are basically

• 1:44:27Speaker 00

  applied math, information theory, and there are people that are really pushing the others in biochemistry. And for all of them, what we have is now a way that dramatically diminishes the amount of time that they need to spend doing things that do not help them actually answer that question. And the questions can become more, more complex and more general. I am so excited about what we're going to be able to do going forward because we are not encumbered by, you know, having to worry so much about these things that took a lot of time, but you not push, did not advance the knowledge. It's going to be awesome, right?

• 1:45:10Speaker 00

  It really is, we need to embrace it and it's going to be really great. And by the way, one thing that I really want to see, and I do not know how long it's going to take, is that, hey, this is not just chemistry or computer science or bioengineering anymore.

• 1:45:26Speaker 00

  Forget, let's just solve these boundaries.

• 1:45:30Speaker 00

  Because the main discoveries and solutions really occur at these interfaces between different fields, right? That's how I really like to see things. And my group were interested in developing new reactions, for example, to tackle glycans and sugars that in the body, they are yet another piece of biology that has been severely underexplored because the chemistry of it is so difficult, right? So we had Carolyn Bertozzi get the Nobel Prize in chemistry two years ago, showing that you could do chemical, like click chemistry.

• 1:46:09Speaker 00

  Okay, now let's go and follow her footsteps and explore those things by developing the technologies that will allow for us to modify sugars. That's the kind of thing that I'm super interested in. And I want, I just want people to really ask hard questions and be able to go into it fearlessly without needing to worry about the things that don't really help them or like writing codes that is just, you know, an operation in the process. I

• 1:46:46Speaker 00

  think we are in for a revolution in how education works and how discoveries will be made much faster. We are already there, not because of coscientists at all. Like just look at Alfred Ford. That was, you know, amazing. One of definitely one of the most important discoveries of the last decade. Where are we going to go next? I don't know, but I know it's going to be fast and it's going to be powerful.

• 1:47:13Nathan Labenz?

  I think that is a great note to end on. You have given us a glimpse of the future with coscientists at a minimum. Gabe Gomez, professor of chemistry and chemical engineering at Carnegie Mellon University, thank you for being part of the cognitive revolution.

• 1:47:29Speaker 00

  Lisa, thank you so much. This was awesome.

• 1:47:31Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.