Material Progress: Developing AI's Scientific Intuition, with Orbital Materials' Jonathan Godwin & Tim Duignan

Jonathan Godwin, founder and CEO of Orbital Materials, alongside researcher Tim Duignan, discuss the transformative potential of AI in material science on the Cognitive Revolution podcast. They explore foundational concepts, the integration of computational simulations, and the development of new materials for various applications such as data centers and combating climate change. They also delve into the latest advancements, including a groundbreaking study on the potassium ion channel, and speculate on the future of AI in scientific discovery and material synthesis. Check out some of Tim's work: Google Colab to run you own simulation: https://colab.research.google.com/github/timduignan/orb-models/blob/main/examples/OrbMDTut.ipynb GitHub repository "Orb force fields": https://github.com/orbital-materials/orb-models Preprint "A potassium ion channel simulated with a universal neural network potential": https://arxiv.org/abs/2411.18931 Help shape our show by taking our quick listener survey at https://bit.ly/TurpentinePulse SPONSORS: Oracle Cloud Infrastructure (OCI): Oracle's next-generation cloud platform delivers blazing-fast AI and ML performance with 50% less for compute and 80% less for outbound networking compared to other cloud providers. OCI powers industry leaders like Vodafone and Thomson Reuters with secure infrastructure and application development capabilities. New U. S. customers can get their cloud bill cut in half by switching to OCI before March 31, 2024 at https://oracle.com/cognitive NetSuite: Over 41,000 businesses trust NetSuite by Oracle, the #1 cloud ERP, to future-proof their operations. With a unified platform for accounting, financial management, inventory, and HR, NetSuite provides real-time insights and forecasting to help you make quick, informed decisions. Whether you're earning millions or hundreds of millions, NetSuite empowers you to tackle challenges and seize opportunities. Download the free CFO's guide to AI and machine learning at https://netsuite.com/cognitive Shopify: Dreaming of starting your own business? Shopify makes it easier than ever. With customizable templates, shoppable social media posts, and their new AI sidekick, Shopify Magic, you can focus on creating great products while delegating the rest. Manage everything from shipping to payments in one place. Start your journey with a $1/month trial at https://shopify.com/cognitive and turn your 2025 dreams into reality. Vanta: Vanta simplifies security and compliance for businesses of all sizes. Automate compliance across 35+ frameworks like SOC 2 and ISO 27001, streamline security workflows, and complete questionnaires up to 5x faster. Trusted by over 9,000 companies, Vanta helps you manage risk and prove security in real time. Get $1,000 off at https://vanta.com/revolution CHAPTERS: (00:00) Teaser (01:05) About the Episode (05:10) Welcome to Orbital (06:15) Semiconductors (07:44) Material Science Today (09:22) Experimental Cycle (12:06) Orbital’s Founding (14:51) AI in Materials (Part 1) (21:05) Sponsors: OCI | NetSuite (23:45) AI in Materials (Part 2) (35:00) Sponsors: Shopify | Vanta (38:15) Generative Models (38:16) Diffusion Models (50:50) Orbital Applications (58:19) Perfect Sponge (59:43) AI Simulations (01:01:27) Natural Language (01:02:35) Compute Needs (01:05:05) Human Electrical Nature (01:06:11) Potassium Channels (01:15:51) Scaling Simulations (01:23:56) Roadmap: Carbon Removal (01:30:37) AI & Job Satisfaction (01:36:14) LLMs & Potentials (01:37:19) AGI & Discovery (01:39:58) Outro

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Finding efficient ways of keeping track of the important information and losing the unimportant information is just a central problem in a lot of physical modeling. And I think AI, machine learning algorithms are just very good at doing that.

• 0:13Speaker 01

  The thing that completely blew my mind was training on small inorganic crystals, like 20 atom systems, to then simulate protein through out of the box generalization. That's telling you that we're learning something really fundamental at that small scale, which I don't think anyone had ever expected.

• 0:32Speaker 00

  The hope is that if you can simulate a ton of things, you start to make important and new discoveries and find new things out just by looking at them, right? Which is very exciting.

• 0:39Speaker 01

  By the time we get to making a decision about what to make, we've answered 90% of the questions that we need to in order to feel confident that we're going to have that sort of material.

• 0:50Speaker 00

  The real key challenge there had been these potassium ion channels, which no one has really been able to fully understand, unfortunately, using experimental techniques or traditional computation. And in fact, we don't even know some of the most basic questions about it.

• 1:06Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, my guests are Jonathan Godwin, founder and CEO of Orbital Materials, which is pioneering the application of AI to material science, and Tim Duignan, who was previously here to discuss his work on the simulation of electrolyte solutions and who's since joined Orbital Materials as a researcher.

• 1:26Nathan Labenz?

  Material science underpins virtually every aspect of modern life, from the semiconductors that power our devices to the batteries and solar panels driving the clean energy transition. Advances in materials have been at the heart of human progress for the last century at least. The challenge has been that discovering and developing new materials has always been painstakingly slow. Traditionally relying on trial and error and scientists' hard-won intuitions developed over decades. And more recently, with the shift to computer simulation, still requiring huge computing power to simulate even small molecular systems for short time intervals. Orbital Materials aims to dramatically accelerate this process with, of course, AI.

• 2:08Nathan Labenz?

  Their immediate focus is on developing novel materials for data centers, both to improve efficiency and to capture carbon emissions, but their technical breakthroughs could unlock advances across clean energy, electronics, medicine, and beyond. Their technical approach is really quite fascinating. Using an architecture called message-passing neural networks, which are trained on small crystal structures and which, because they don't use positional embeddings like large language models do, are capable of scaling up indefinitely with computing power. They can design new materials with specific target properties via a diffusion process and also predict the forces between atoms orders of magnitude faster than numerical methods can, which allows them to simulate larger systems for longer.

• 2:51Nathan Labenz?

  They recently demonstrated the power of this approach by simulating a potassium ion channel, a critical protein that controls electrical signaling in our cells by selectively allowing potassium ions to pass through cell membranes. Despite its importance in everything from heartbeats to brain functions, fundamental questions about how this channel works have remained unanswered for decades. And while Tim's recent work still needs to be experimentally confirmed by the broader research community, his simulations were able to show a level of detail in the mechanism that was never before seen and which does help explain previously inexplicable data. The implications for biology and medicine are significant. But perhaps more important still, this work illustrates a critical phenomenon that we are seeing time and again

• 3:35Nathan Labenz?

  as AI is applied to the different branches of science. Namely, that neural networks seem to have the ability to develop a sort of intuitive physics in virtually any problem space. Just as humans can catch a ball without explicitly calculating its trajectory, these AI systems are developing efficient shortcuts for predicting complex physical phenomena, whether that's material properties, protein folding and interactions, weather forecasts, single-cell transcriptomes, or even the evolution of human brain states. For me, this is the clearest reason to believe that superhuman intelligence is not just possible, but increasingly likely. For human scientists, meanwhile, this means a shift away from hypothesis generation and

• 4:18Nathan Labenz?

  toward more validation and implementation work. And while that might mean lower job satisfaction, the potential to dramatically accelerate scientific progress and more effectively address critical global challenges, for me, makes it a worthy trade off. And again, reminds us that we might soon need to look beyond our work for meaning. As always, if you're finding meaning in the show, we'd appreciate it if you'd share it with friends. You can also write us a review on Apple or Spotify, and we love to read your comments on YouTube. We value your feedback and suggestions, too, and encourage you to leave them either via our website, cognitiverevolution.ai, or you can always DM me on your favorite social network. For now, I hope you enjoy this look at how AI is transforming material science, and by

• 5:02Nathan Labenz?

  extension, how it might fundamentally reshape and accelerate scientific progress in general, with Jonathan Godwin and Tim Duignan of Orbital Materials. Jonathan Godwin, founder and CEO of Orbital Materials and returning guest and now researcher at Orbital Materials, Tim Duignan, welcome to the Cognitive Revolution. Hi.

• 5:21Speaker 01

  Thanks for having me.

• 5:22Nathan Labenz?

  I'm excited for this, guys. I'm on quite a journey to try to figure out what is going on at the intersection of AI and science. And obviously, there's a lot of different sciences and a lot of different intersections and a lot going on. I feel like life right now for me is one big crash course. You guys are advancing the frontier when it comes to material science and the use of AI in material science, and I'm excited to get into all that. I thought maybe just for starters, because listeners to this feed are obviously plugged into the AI moment, but probably most don't have much background in material science, maybe just give us the basic foundational pitch for materials. Like, why do they matter so much?

• 6:05Nathan Labenz?

  Maybe a couple of highlights from recent times where an advance in the materials directly led to a change in the way we live, just to get the juices flowing and get people motivated to understand all this stuff.

• 6:18Speaker 01

  I mean, the most obvious one for people with all the AI is semiconductors. You have this wafer, the silicon wafer. And you etch a bunch of stuff, your circuits onto this silicon wafer for your chip. Um, but of course, those etches are incredibly, incredibly thin, incredibly, incredibly small. And if a piece of dust or some debris gets stuck in that edge, then your whole chip fails. So you've got to find materials that don't conduct electricity because conducting electricity across the etches in your wafer will screw up your chip. So it can't conduct electricity. But it's got to be small enough to fit in between

• 7:00Speaker 01

  all of the different bits in your wafer in order to protect your chip from any sort of debris or any sort of pollution that comes on. So these things are called low K materials. And without significant advances, improvements in the performance of these materials in a year, you don't get more transistors on your chip and you don't get AI. So material science, it underpins literally everything in our lives, the entirety of the Silicon Valley revolution. It's given rise to, you know, the tech industry in the West is based upon material science. And material science is the original tech startup. And we've just kind of forgotten that these companies built a lot of the industry in America, in the UK, where I'm from, but all around the world.

• 7:46Speaker 01

  And I don't think the 21st century is going to be any different from the 20th century. The 20th century was the century kind of of material science. 21st century is going to be even more so. We're going to have more chips, AI is even more important. We're going to go to Mars, we're going to have to have incredible spaceships built out of new materials to do that. We're going to have hollow lenses in virtual reality, we're going to have new optical materials to do all of that. So I think material science is the fundamental missing piece in achieving the science fiction future. And AI is our most powerful tool. It's certainly something that we want to be doing to achieve that. So that was just one example, I think, of all the things, but Tim, you've been working on materials for longer than me, so you must see many more.

• 8:30Speaker 00

  No, I think that's an excellent one and you don't think about a lot, but the ideas factory about Bell Labs is fascinating about the birth of the transistor and it's just all material science, right? Trying different elements. But the other big one, obviously, is climate change, right? Where essentially, all of the tools we need to fight climate change and material science problems are broadly defined. And the most successful tools we've built so far have come out of material science. So batteries and solar panels, leading to massive reductions in CO2, both came from breakthroughs in material science. It's so universal that I often like to play the inverse problem of a name of problem, and then I can show you how it's related to material science in one way or another. It's just at the foundation of so many things.

• 9:14Nathan Labenz?

  I'm excited. So maybe a little bit of the status quo would also be helpful in terms of how material science advances today. We've done versions of this for biology, for example, and have contrasted like, somebody sitting there with a pipette and what that throughput can look like. And obviously, some of that is starting to get automated in the physical world. But then there's starting to move more and more of that into simulation and even into AI models. You guys are basically doing the same thing, as I understand it. I saw the motto on the website, engineer in silicon, materialize in reality. But for folks who don't know, before we get to that, engineering in silicon, what is the sort

• 9:55Nathan Labenz?

  of life of a material scientist look like today? Obviously, it's such a big field, there's no single answer to that. But just a little flavor for sort of what it takes and what the experimental cycle looks like to make advances pre-AI.

• 10:12Speaker 00

  Sure. So for an experimental material science scientist, right, there's still a huge amount of experimental and trial and error and guesswork involved, right? So if you talk to an average experimentalist trying to make a new material, you know, they'll have hypotheses, they'll have ideas about what's going on and what they want to do. But there's a huge amount of luck involved, trial and error and intuition as well, like built up over years of what's going to work and what's not. I like to make the analogy to bridge building a hundred years ago, where we didn't have highly accurate models of what bridges were going to be stable and what weren't. And so there was a lot of guesswork and intuition about how you would build these bridges and things built on experience. And they could build incredible things, obviously, but it wasn't nearly as efficient. They couldn't push

• 10:55Speaker 00

  things to the limit and they couldn't do it nearly as quickly as we can today, where to design a modern bridge, you would plug it into some kind of computational modeling software and every kind of bolt and beam, you would know the exact forces and stresses and tenses, you would know the limits of that material and you could design everything really optimally. So if you look back at like Brooklyn Bridge, for instance, massively overdesigned versus what it needs to be, because they needed to make it safe so it wouldn't fall over. And you did used to get new bridges falling down. There's a famous one where it had a resonance because they hadn't tested it computationally and it collapsed after they built it, right? Whereas today, we model these bridges incredibly carefully. We know before we build them exactly what they're going to do. I think we want to transition material science to that kind of world where we know in advance, if we can make this material, we're very confident it

• 11:39Speaker 00

  will have the properties we want and it will do the things we want. We also want to be able to answer questions much more quickly that material scientists have. So they might say, oh, this isn't working for some reason, or I'm not getting the yield I expected, or the performance is bad. We want to be able to go to the computer and say, okay, understand this from the ground up and say, okay, here's exactly what's going wrong. And here's some suggestions of how to fix it. And here's what's actually happening at the molecular scale is a big problem often. We can't see what's going on.

• 12:05Nathan Labenz?

  So where are we on the path to that dream? And that's maybe a great opportunity to introduce the company and kind of tell a little bit of the founding story if you want and give the big vision.

• 12:16Speaker 01

  Yeah, so Orbital came about, I was working at DeepMind and leading a team building large scale AI models for material science. And, you know, I thought that it was going to take us a decade to reach the level of performance we're able to reach into. The speed at which this field is changing is exactly like every other area of AI. It's just changing and developing at unbelievable pace and new capabilities that, you know, we had dreamed about in our reality.

• 12:55Speaker 01

  You know, some of the things that Tim, I'm sure is going to talk about in more detail later on, they just weren't possible on the world's best supercomputers just ten years ago. And we're now doing them on our laptops. And so the goal of our company was to take and continue to develop those models and those techniques, but combine that with a wet lab with a with a team of extraordinary experimental material scientists to translate those AI capabilities into real materials. And we felt that by creating a team that had both of those skills, we're going to be able to move faster than anybody else was and bring those materials to market and commercialize them better.

• 13:41Speaker 01

  And we're going to ground the AI that we develop in the real problems and what's next associated with making breakthroughs in material science. And that team wasn't going to exist at D-Mine, it wasn't going to exist as Google, those are AI companies. It wasn't going to come out of a traditional chemistry or material science company. You know, they've been saying no to computational tools for the last 25 years. They're not going to suddenly turn around and be building frontier tech. So it had to be a new company if we wanted to achieve that goal. So I think we're we're so much further along than we were just a couple of years ago. The things that we're doing now are completely different. I think it's only going to be a matter of a couple more years where we're achieving this dream.

• 14:23Speaker 01

  We need a computer computer to design materials, just get a GUI in front of you with an LLM co-pilot and you'll get to new battery in a couple of iterations just on your laptop and have a real high confidence that when you go and make that, it's going to go and have those properties. And all the things that I spoke about right at the beginning, you know, spaceships, better chips, better semiconductors, more GPUs, more AGI, solving climate change. All of that is going to happen through the technology that we're building.

• 14:54Nathan Labenz?

  One thing you said that caught my ear that I want to dig into for a minute is this idea of simulation. So it seems like we have kind of three modalities for pushing things forward. The original is hands-on with actual physical materials, doing stuff, measuring properties, and ultimately, everything still has to ground out to the ability to do that successfully to achieve the results we want to achieve. And then there's been this sort of middle time of simulation. And you said that some of these things like we couldn't possibly simulate this even on the world's greatest supercomputer. And now we've got this third wave, which is the AI wave. And this was a big theme of the conversation

• 15:37Nathan Labenz?

  I had with Tim last time when we were talking about the crystallization discovery was basically that what distinguishes the AI era for this sort of work from the simulation era, maybe above all, is it runs orders of magnitude faster. So I've been really tuned on that quite a bit. And I'm coming to a worldview that basically amounts to a belief that today's AI architectures seem to be capable of learning what I might describe as an intuitive physics in basically any space that you might want to have them learn that. We obviously have

• 16:17Nathan Labenz?

  an intuitive physics where we can play sports and react to things in sub-second time, which if we were actually using our feeble brains to do all the calculations and simulate step by step, we would never be able to catch the ball. But we're quickly sort of grocking the situation and not calculating in the way a Wolfram would suggest that we might have to. We seem to be taking a sort of intuitive shortcut. And it seems like that is popping up in all of these critical domains, whether it's protein folding, or material science, or solving the wave equation for a bunch of atoms in a configuration or whatever. Is that how you see it too? Would you edit my picture of that? Because that seems like

• 16:58Nathan Labenz?

  a pretty foundational update to my worldview that I kind of want to go shout from the rooftops, but I want to make sure I have it right before I repeat it too often.

• 17:08Speaker 00

  I think that's right, and I don't think we fully understand why yet, or at least I don't. But it's very exciting, right, that this seems like a universal tool. So it's like revolutionizing climate simulations as well, it seems to be, right? And weather modeling as well, like it's across the board, physical simulation. My theory is it's something to do with dimensionality reduction, right? That often you don't need to treat every single part of a system. And so in our case, we're kind of, this is called coarse graining in physics and statistical mechanics. There's a great theories of it, Moritz Zouzzi, it's all based on logevon dynamics, actually. So it's intimately related to what a lot of the AI algorithms are doing. But the basic idea is that you get these universal things across different scales, and they have repeating physical properties. And the key one is that you don't need to treat

• 17:52Speaker 00

  every single piece of the system to predict what it's going to do. So in our case, we're ignoring the electrons. In a lot of biology, you might want to ignore the solvent, say. And then when you get to larger scales, is obviously when you're looking at a ball moving through the air, it's just one object you're tracking, and you're ignoring this vast amount of other information going on. And so finding efficient ways of keeping track of the important information and losing the unimportant information is just a central problem in a lot of physical modeling. And I think AI, machine learning algorithms are just very good at doing that. But what do you think, Johnny?

• 18:23Speaker 01

  Yeah, I think you're absolutely right. You know, what's been extraordinary to me has been not just the kind of universality of the sort of AI models that we use. So, you know, the stuff that we develop is very similar to the stuff that's used in weather forecasting, similar architectures, almost identical.

• 18:42Speaker 01

  But also just how much implicit physics knowledge can be contained in a very small amount of space.

• 18:51Speaker 01

  When you look at some of these world models, right, like you know, some of the genie tube, you've seen that from DeepMind, or, you know, a couple of other startups have released something similar. They have these single models that simulate physics, because they're simulating water dynamics and reflections of lights in puddles. They understand gravity, they're able to deal with very small, you know, tiny effects that would, just a couple of years ago, take in really expensive computer rendering, you know, CGI type computer software in order to produce.

• 19:32Speaker 01

  But this only has, I mean, I don't know the precise number for these models, but these must only have a couple of, a few billion parameters, right? These are going to be diffusion models. Diffusion models are not parameter heavy. They're very data intensive, but they require far less compute during your inference pass or for your, for your, for learning these dynamics than a large language model does. So there's this implicit, as you say, this intuitive knowledge, right? Because we know that the bottoms up calculation of all of these effects and the effects that we model when we think about, you know, atomic scale, quantum effects, which the stuff that we really care about, calculating from bottom up requires a huge amount of floating point operations. So we're taking a lot of jumps.

• 20:18Speaker 01

  Whatever's going on, you can call it intuitive, but it's shortcuts that preserve the fidelity of that simulation in the ways that we care about. It is just quite extraordinary that we're able to condense that information in a way that has that level of fidelity into something just so incredibly small. I mean, the, the big compared to neural networks for a couple of years ago, but they're tiny compared to our brain and they're tiny compared to the amount of compute we're throwing at it with bottoms up physics simulations. So it is quite incredible. And I think that intuitive physics explanation is a really nice way of thinking about what's going on because it's just in the same way that you get extraordinary sports people who can score incredible goals and do incredible feats, but not know any maths.

• 21:02Speaker 01

  That's exactly what these AIs are doing when they're doing these simulations.

• 21:07Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 21:13Nathan Labenz?

  Before we get more into the, the practical details of how you've gone about all this, one other philosophical question kind of relates to, I don't want to be too literal about Grokking, but you know, there's, there are these debates in language models where we get very confused and we're like, you hear people say things like, well, sure, you know, they might learn the concepts that we have learned and encoded in language, but they're not going to learn anything beyond that because, you know, how would they? They can only learn what's already encoded in language. And I sort of find, you know, discussions around that sort of topic can easily get circular.

• 21:55Nathan Labenz?

  But then I move over to a different domain, and we've now seen multiple examples of this out of the intersection of AI and biology, where you can take like interpretability techniques and apply them to say a protein language model and find out that, oh, like I recently had a professor from Stanford, James Zhaoan, and he was like, they discovered a new motif by looking at the, applying interpretability techniques to a protein language model. And so it seems like there's something there where it's pretty hard to deny that this thing has like learned a concept that we did not know, and it didn't get it from us, right? Because it's in that case, it's just training on sequence data. But I don't think I work that way. So this is where the analogy to intuitive physics maybe breaks down, right?

• 22:39Nathan Labenz?

  I don't think, maybe I have, just can't articulate it. I really have no idea, I guess, what's going inside my brain. But it seems like there is some sort of conceptual grocking that that is happening in, for example, the protein language models that are allowing them to do these like shortcuts. It's these higher order concepts. Is that also what you think is going on in material science? And do we know what those concepts are? I have a sense, you know, for a motif and active site or whatever, but I don't even really know what those like higher order concepts would be in material science that these models might be grocking out of the raw data that they're trained on.

• 23:22Speaker 00

  Yeah, I mean, one thing is I think that may be going on in scientists' brains, right? Where like a lot of science, I mean, people don't think of it this way, but it's really intuitive, right? And it's really gut instinct driven, right? Like, you rely on your gut all the time. And it's possible that even though we don't rationally use these ways of discovering things, that it could be that scientists are doing this intuitively. And definitely in material science, I think that's very common is based, you know, people call it intuition, right? Or, you know, chemical intuition is a very important concept, and people rely on it. And they can't, often they'll give you a story, but it's a very powerful tool. So it's very possible that the AIs are doing something similar.

• 24:02Nathan Labenz?

  So what does that look like? Just to calibrate myself, like, does a good material scientist just sort of put a finger to the wind and say, yeah, let's throw in a little more acid?

• 24:12Speaker 01

  I've got a little bit of story about this. I'm stealing someone else's story. Right? So exactly what happens to semiconductors. The people basically spend 30 years of their life on this very tiny thing. I was talking about etching before, right? It's like, what gases do you use to clean, you etch silicon wafer? And you need to de-gassy, you need to clean it with another gas. Which gases do you use? How do you change that slightly? How is it going to affect the silicon? And people just spend 30 years of their life in each one of these. I think there are about 10,000 steps or something, process steps in making a chip. You know, the

• 24:51Speaker 01

  packaging, the etching, the lithography. And it's just, it is like, you know, these are craftspeople, artisanal. They've done so many experiments and gained such intuition that, and it's not written down anywhere. You can't go and learn this stuff. It's not easily deducible from first principles, chemical knowledge, and it's passed down in this sort of master and apprentice type way. And it's one of the reasons why people just can't recreate TCMC, you know, however, however much money the US goes and throws at new chip fabs or China, how much espionage they go and do to try and build their own fabs in China. You can't, you can't beat TCMC because

• 25:34Speaker 01

  they've just spent all of that time and that like intrinsic knowledge that sits within their chemists and material scientists. So it really is this kind of deep sense of intuition. This won't work, this will work. You know, one of the goals of AI is to break that, really, to say, well, AI should be able to, without having to go through all, ultimately, it's still physics. So AI should be able to get it right. If they can get physics correct, and it should be able to democratize that level of knowledge. But yeah, it's like a sage person who's done this 30 years pronounces their views. And it's almost

• 26:15Speaker 01

  The prophetically correct in many cases, you think, no, that can't be right, but it actually is. So my level of admiration for the human own practice and intuition, which is something that's kind of not really captured in something like a reasoning model, but there's, you know, intuitive movement through the search space as scientists in chemistry and material science that that has grown exponentially through my experience also. So you've got some incredible people that just know what will and what won't work. But anyway, Tim, whether your experience with chemistry and material sciences have been the same.

• 26:54Speaker 00

  Yeah, I think so. And it's sad because it's a beautiful thing, right? Like this built up body of amazing knowledge and it's sad to kind of be trying to replace it. But I think there are fundamental limits, you know, to the human brain. And we want to push beyond that basically, right? So we need ways of doing similar stuff computationally. And I think it's very exciting if we can start to get there and then potentially as well, like combining it with the human intuition. I think is very powerful, as Johnny was saying, you know, because they've obviously reached it from very different ways. They're going to be able to correct problems that each other makes. So I think it'll be a powerful combination when we can get them working together really well.

• 27:30Nathan Labenz?

  Another thing that caught my ear was you said that the diffusion models are not parameter heavy, but they're data intensive.

• 27:39Nathan Labenz?

  We talked last time with Tim about the training on molecular simulation data. So if I remember that accurately, it was like, you know, it costs a lot of compute. And I think it was 10 to the minus 15th second time intervals that were being simulated. And then train the model on that. And then you get a model that can do essentially the same thing, but order of magnitude faster. Alpha Fold, if I recall correctly, was only trained on maybe five figures, maybe into low six figures worth of protein structures that were known at that time, which I think was basically everything that was known at that time. What does the data universe look like in a materials science context?

• 28:20Nathan Labenz?

  Because I can imagine it could be like a simulation thing, or it could be a, you know, big catalog of all, you know, materials that we've characterized to date, or maybe even the both of those like.

• 28:33Speaker 00

  So yeah, this was a big issue in chemistry, right? For a long time, I was quite skeptical of AI for chemistry, just for the problems I wanted. There's just vanishingly little data, right? So experimentally, like, I was interested in electrolyte solutions. And there's like some databases out there, but there'll be like a few parameters per electrolyte. And so I really thought like, oh, this is going to be really tough to do anything. Now, there's other areas of chemistry where there is large data sets. But in my particular area, I wanted to apply AI, I just couldn't see how to do it. Obviously, yeah, there's the PDB and biology of these huge data sets. But the thing that really clicked for me was, well, we can automatically generate these training data sets using first principles computation, right? So it's kind of trivial. I realize I have huge data sets.

• 29:15Speaker 00

  If you take into account the massive simulations I've been running based on first principles physics, you know, these are generating, you know, millions of data points, pretty trivially. Because the data is generated computationally, you can do all sorts of nice tricks like active learning and automatically generating the data in loops where you need it and things like that. So that's why I think first principles generated data is a big area of future growth in science.

• 29:40Speaker 01

  Somehow you can think of this as like a, where AI general as well. Because the data that you can generate quickly is often pretty small data. Maybe I can give you like an example. You know, you are simulating what Tim's just simulated, you know, a protein complex interaction with a metal. Now, you're simulating that thing in isolation. It's a small part of something far bigger. And all of the water molecules that surround it, all of the rest of the cells, all the rest of the proteins affect the dynamics of that in some small way. Now, hopefully for the protein situation, it doesn't affect it so much that the protein stops functioning. But for something like a catalytic reaction, if you're in green chemistry and you want

• 30:25Speaker 01

  to convert CO2 into sustainable aviation fuel, in order to simulate that correctly, you need to simulate thousands, if not millions of atoms, and you need to simulate that over very, very, very long time scales. And so the thing that I think is extraordinary about AI is you could take just a tiny piece of that simulation, maybe just a hundred atoms, and you can run that simulation, which isn't going to tell you very much about chemistry, but it's going to give you rules and insights into physics. And you can learn your algorithm on lots and lots of examples of very small systems. And then that's going to generalize to very, very big systems when you then go

• 31:05Speaker 01

  to inference time. And I think that's the crucial observation that really accelerates and is the game changer. Because what you're able to do is take something which is too simplistic to be useful in prospective design of materials, but you can learn the rules from that thing. And then you can then apply that into the large scale systems that do tell you something useful that would be completely impossible to be able to be tractable with traditional approaches, but work with AI. So that generalization capacity of these new AI algorithms, I think of what has been just so game changing. And I think Tim has

• 31:46Speaker 01

  been really leading the way here with some of the all models. Um, you know, the thing that really just blew, completely blew my mind was, you know, training on small inorganic crystals, like 20 atom systems, to then simulate a protein through out of the box generalization. That's, that's telling you that we're learning something really fundamental at that small scale, which I don't think anyone had ever expected.

• 32:14Speaker 00

  Yeah, that's a great point. There might be like kind of a finite amount of complexity that you have to capture. And then once you've captured that, you can kind of do a huge amount of other things. So there's only so much data you may need for some problems, I think.

• 32:27Speaker 03

  Yeah. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 32:32Nathan Labenz?

  Okay, so you mentioned these orb models. I was doing my homework, of course, preparing for this. And I understand that there's sort of a constellation of models, but I need your help to understand the hierarchy of how these things relate to each other. So I gather from the website, there's a model called Linus. That's kind of the internal, most foundational foundation model. And then those are getting kind of fine tuned for different purposes.

• 33:00Speaker 01

  Yeah.

• 33:00Nathan Labenz?

  Maybe it would actually be most useful just to talk about like some of the tasks. We've alluded to one, which is like simulate a system through time. And you alluded earlier to like same architecture as the weather simulation models. That those tasks seem to line up, right? Given some initial weather conditions, propagate that through time. Given some initial setup of atoms, propagate that through time. But I've also heard like other task types described. For example, like, can you come up with a material that will have these properties? And that sounds like a very different sort of input output. So maybe just give us a sense for like what task types we're trying to tackle here.

• 33:40Speaker 01

  Yeah, so here's where Tim's saying everything is long term dynamics becomes really epic. We start off with a generative model of materials. So what we mean by that is to miss like a generator model of a protein or a generator model of an image. So that's probably the right way to think about it rather than a large language model. And these are what are known as diffusion models or score based models. And I was about to give you a description of the physical interpretation of these, but Tim will get the maths will correct than me. So maybe Tim, you can explain how these things are actually all just physics all along.

• 34:21Speaker 00

  Yeah, so the interesting thing about the diffusion models is that if you look into the inspiration, it comes from this non equilibrium statistical mechanics. It comes from these kind of generating these molecular simulation algorithms, long term dynamics designed to generate trajectories of atoms moving through time. So there's kind of a deep connection between this task of generating new molecules and making these simulations of things through time where the mapping is the score, which is this vector field that the atoms move on in a diffusion model that you generate the atoms in is also. It basically analogous to the force field, which is the thing that pushes all the atoms in a molecular simulation. And you can actually show, there's a lovely paper from Cecili Clementi,

• 35:03Speaker 00

  showing that if you train a diffusion model on one of these simulations, right? This time evolution, the score is mathematically identical to the force field. So there's deep mathematical connections between these things, which is very exciting. And so the generative model, the way to think about it is it's doing essentially a molecular simulation. You can change the atom types as well, right? It's not just the positions of the atoms which you allow to move. You can also change the atom types as well. So there's actually deep connections between the molecular simulations and the diffusion models.

• 35:36Speaker 01

  Yeah, it's an extraordinary thing and it allows us to do this pre-training fine-tuning step. So the way I think about it is these diffusion models are learning a simplified physics. The great thing about the diffusion model is that you get to sample infinitely many points. You've got almost infinite training data in a diffusion model. Because the way you train a diffusion model, in the case of an image, you take an image and then you gradually corrupt it with noise. And then you predict, you can do your reverse step where you start with noise and then you predict how to de-noise that image. And that de-noising process is the generative process. But you can sample infinite many noise corruptions. And so you've got infinite amount of training data that you can train on.

• 36:20Speaker 01

  So you've got a really high data density, right? Like you've got infinite samples of your force field. And so that is a great way for doing pre-training. Pre-training, you need a lot of data. So the way I think about our generative model is that it does the generative side. So we get to generate materials according to functional properties of that material. But it's also a great way for us to learn a simplified physics. Like you've got infinite data for a simplified physics. And that is a great pre-training step for us to then learn the real physics and fine-tuning when we're learning the real forces computed through quantum calculations.

• 37:00Speaker 01

  So the Linus is the foundational foundational model, learning a simplified physics, which we then adjust to give the correct physics that we can use in real simulations in that fine-tuning step. And that fine-tuning step gives us orb, which is basically calculating the interactions that happen at a quantum level between atoms. And once you've got that right, then you've basically got physics right. You can then move on to simulating whatever you like. You can simulate nuclear reaction. You can simulate what happens in the body and you can simulate all the incredible materials that we use every day.

• 37:38Nathan Labenz?

  So I get the image one. Let me just stay on this for a minute because I want to understand this better than I do. I've given a very similar explanation to many people many times as you just gave about the image version of this where it's like, okay, I've got an image. I can add noise to it. I have a pretty good intuition for what that means. And I also have a pretty good intuition for how if I just keep adding a bit of noise over and over again, eventually I get to something that for all intents and purposes is just noise. And then I can imagine, you know, reversing that and training to predict, you know, the way I always phrase the question is, what would this image look like if it were a little less noisy? And then you also have the steered version of that, which is,

• 38:20Nathan Labenz?

  what would this look like if it were a little less noisy and a little bit more like this text prompt that was aligned to image space via like clip or whatever. Now, in the context of like proteins or peptides, I've learned, but not super well understood that there are like custom noising processes because it's just more complicated. I guess you end up, if you just add pure noise, you end up with something like basically incoherent and it doesn't really work in the same way as you can just sort of add kind of pure noise to an image. And then there's also probably issues with like discreteness where you're talking about like atom type changing and that seems like not something that is like super differentiable. So, and then there's also this aligning question too, right of like,

• 39:05Nathan Labenz?

  How did these sort of low level detailed configurations of atoms translate to qualitative macroscopic properties that we care about? So, you know, if that was the 101, give me the 201 on what noising means in this context and how you're aligning, you know, low level things to properties we care about.

• 39:27Speaker 01

  So noise is super straightforward in materials. It's very similar to images. You don't have to treat it as especially you take a crystal and then you just jiggle the atom positions around. And you jiggle them more and more until it becomes totally incoherent. And then there are techniques that you can use for discrete diffusion as well. And they're kind of pioneered in images and they work pretty well when you transfer them over to materials. So it's actually more similar to images than it is to proteins in some ways. The only thing that is super weird in a crystal structure that you have with a material is that if here we've got a box and you've got your

• 40:09Speaker 01

  crystal inside that box and atom moves across one face of that box, it appears in the opposite face again. And so noising means something slightly different because, you know, you don't have an opportunity to go all the way off into the distance. You're always going to be sitting inside one of these small boxes. And that has a little bit of challenges to the maths, but in practice, you just treat it like you would an image. So it's pretty straightforward in that sense. When you think about guiding towards property, it actually ends up being very similar. In an image, you will train, I mean, there are a bunch of ways in which people do this, but the way you call it sort of guidance. So the way you do that is you create an image classifier,

• 40:53Speaker 01

  basically. And that creates an energy function, right? Something that says, is this close to a cat? And as that number goes down, it becomes closer and closer and closer to a cat. We just do the same thing. So we will take a crystal structure or a material and we'll Train a simple supervised learning algorithm to predict a property. And we'll set that property to what we want it to be. And we'll try and just get nudge it towards that property through that differentiable machine learning classifier. So your description is basically entirely the same as what we do in materials. You know, the thing that we have to do differently is we can't use the same machine learning architectures.

• 41:38Speaker 01

  We may need to make adaptations to our underlying machine learning architecture. In order to get really accurate results within material science. And that's the big difference between materials and images. If you were to look at the maths, the maths looks pretty similar.

• 41:57Nathan Labenz?

  Sorry, could you say again what the big difference was?

• 41:59Speaker 01

  Say, what's really important, you know, if you have two, if you have two atoms. And they get closer together, then the forces that are acting on those atoms increases. And then if atoms go really far apart. And you get no forces applying on them whatsoever. Um, but there might be a sort of Goldilocks zone where they're attracted to each other.

• 42:28Speaker 01

  Um, and so the distance between two atoms or the vector, if you're thinking in three D space. Is like a first class system in representing an atomic system. Um, it's not a first class system in an image because all the distance, if you think of a pixel as an atom, all the distances are the same. So, you know, you don't need to, you don't need to represent it in your architecture. Or, you know, all tokens are, you know, the same distance apart. So you don't need to represent distance in a transformer. But for three D. Point clouds, you know, for atomic configurations, distances are first-class

• 43:09Speaker 01

  citizens. You don't get to predict the property unless you have a really good, high-fidelity representation of distances. And so that means that you have a different type of modality as an input to your machine learning model. You need to have distances of which there are quadratic number, and you need to have, you know, the atoms themselves. So that gives you a kind of very, very different type of input modality. And so you need to have a different type of machine learning architecture called a message passing neural network, which is basically just like a transformer, but a transformer that takes distances inputs

• 43:49Speaker 01

  as well. And that is what we use. And that's the major difference. If you were to kind of be training, you know, look at the code for our models, and you just look at the code for a vision transformer or a language model, that's going to be where the major differences are.

• 44:07Nathan Labenz?

  And I would love to have a little bit better intuition for how we make those leaps from like atom type to atom type.

• 44:15Speaker 01

  Yeah, so you basically treat categorical prediction, so you're predicting the atom species from zero to one hundred and seventeen as a continuous function. So instead of having a cross entropy loss on it, you treat it as a continuous target, and then you cast that back into a categorical one. So this is a technique, I think it's called binou analog bits. It came out like twenty twenty-two as a way of doing discrete diffusion. You basically just cast it as something that has a gradient in it as a kind of softening of that discrete distribution. And then it's a pretty elegant and pretty simple technique that works pretty well. Yeah.

• 45:01Speaker 01

  It's a hint in paper, so you know it must be good, right?

• 45:04Nathan Labenz?

  Okay, so can we go back to tasks then? And say with all that in mind, you know, when I do a image diffusion model, I might show up with just a text prompt and say, you know, make me a picture of a scientist in a material science lab. Or I might say, here is a picture and I want you to make it, you know, more like a dinosaur working in a material science lab. What do I show up to the different orb models with? And what do I then get back?

• 45:36Speaker 01

  Yeah, so for Linus, you'll show up with something like, you know, I want to have a band gap of X. Give me 10 materials that have this band gap, and it will generate a material that has that property. Very similar to an image generating model. Start off with a description of the image, and you'll generate an image that fits that description. It's exactly the same. That's what Linus does, our foundation, foundation model.

• 46:02Speaker 01

  And then orb, the orb models that we've released, the ones that are publicly available have a different task. And that's the simulation task. So that is, I've got a system, but I want to know what's going on. Like, I want to know if this, like, from a first principles perspective, how is this reaction going to be taking place?

• 46:21Speaker 00

  So the main thing they're doing is predicting forces from coordinates. So you give them the positions of all of the atoms and the atom types, and they'll print out the forces and the energies. It is essentially all they're doing, but that actually unlocks a huge amount of other things you can do, right? Because people have been running these simulations for decades and developed all of this theory and code and know how about how to use this ability to run simulations with these forces, that we can then leverage that to compute a whole host of other properties. So you can compute things like diffusivity, so how quickly atoms will move, their chemical equilibrium. So like how strongly they stick to each other and stick to other atoms. Essentially, all of the really fundamental stuff you want to know about materials,

• 47:04Speaker 00

  you can then extract from these molecular simulations, which is a really exciting thing. So There's then kind of two ways you could go about generating new materials that have the properties you want. One is you could try and just directly simulate a ton of them and see if they have the properties you want. And then you could try and bypass that with a direct generative model. And you can combine them in different ways, right? So coming up with hypotheses with the generative model and then feeding them into the simulation to check that you do see what you want with these more exact simulations.

• 47:34Nathan Labenz?

  Yeah, I kind of just want to say, keep going there because the sort of outer loop is really interesting. I mean, we talked about that in the context of your work with the solutions. And it was like, calculate all the forces, apply a time step increment, everything moves a little bit, recalculate the forces. Next thing you know, you're seeing crystallization popping up out of nowhere. I'd love to, you know, yeah, kind of keep zooming out internal tasks and talk about the scaffolding that you're building around them.

• 48:05Speaker 00

  So in terms of molecular simulation, there's really three key things you can do with it, which is exciting. One is, I think of as a computational microscope, right? So the idea is you just directly see what's happening, right? So you can make a movie and you see what the atoms are doing. This is a very powerful and useful tool, right? Because in particular, it gives us access to a temporal and spatial scale we haven't had access to previously. And that's particularly the nano scale, right? Like nanometers, nanoseconds, kind of a billionth of a meter, a billionth of a second. This is a really important scale because a ton of interesting stuff happens there. All sorts of chemistry, chemical bonds are formed and broke at that kind of time scale and spatial scale. And so we'd really like to be able to see what happens. Very hard to do it experimentally

• 48:46Speaker 00

  because these things are much smaller, you know, they're atom sized, right? So you need something similar size to kind of observe them experimentally, but we can potentially get it computationally, which is exciting. And Amelie, your previous guest talked about this in the protein folding context, right? Where the statics problem is, you know, maybe being solved, you can now predict the static structure of all these proteins. What we're really interested in now is how do they move? What are their dynamics? And particularly, you know, these very short time scales, which we can't get to experimentally. Below, you know, milli- a microsecond. And that's where these techniques are really useful. This idea of of a microscope is really powerful, I think, because it it's revolutionized science many times, right? When Levenhoek built the first microscope, everything he looked at, he made new discoveries, right?

• 49:30Speaker 00

  Like, just looked at some pond water, right? Found bacteria, looked at his blood, and he found red blood cells, just discovery after discovery. And so the hope is that if you can simulate a ton of things, you start to make important and new discoveries and find new things out just by looking at them, right? Which is very exciting. The second one you can do is, I think of as in silico experiments, right? So you don't now have to go and order the chemicals and synthesize the new molecule and do all this work to try and then see it in action and then use microscopy to try and understand what's happening. You can just change the atom type in your input file and rerun the simulation, right? So you can do simulations much more quickly, hopefully, than you could do them experimentally. This is a while off, but eventually you could think about doing high throughput things with this, right?

• 50:13Speaker 00

  So trying, you know, thousands of simulations and just trying to brute force, find out which material works best. The third thing you could do, and this is actually long term, the most exciting one, is actually use those simulations to extract data from them to then use for larger scale models. And I can think of this kind of as providing a bedrock for for physical simulation, right? So for many, we simulate all sorts of physical systems, right? The climate, rockets, whatever, you simulate these things with all sorts of modeling approaches. But these all need to be parametrized, right? They need to have numbers plugged into them. And it's often very hard to work out what those numbers should be. You have to do experiments, very hard to do those often.

• 50:52Speaker 00

  Whereas if you could use lower scale simulation to extract that information, you could plug it into these models, which would be very useful. And then the other thing you could do is you could generate training data sets this way, which you could then go and train your generative models on, which would be great as well. And then there's also this coarse graining thing I've mentioned, which is what I was doing with the crystallization. And this is a statistically mechanically rigorous, well-defined procedure where you can show that you can ignore certain aspects of the system and keep other ones. You could ignore the solvent, for instance. So one day, you could think about simulating how proteins fold with time. And in fact, people are already starting to do this, uh, using some of the approaches that Amelie was talking about, uh, training on all atom molecular dynamic simulations.

• 51:34Speaker 00

  To then enable you to look at how proteins fold dynamically, which is very exciting. And that will come and it's very similar in material science. You want to know both the static structures, these things are forming, but also the dynamics, how the chemical reactions, what are the transition states, all things is very, very hard to get at experimentally now.

• 51:51Speaker 01

  Maybe we can say all these things link together, like what's up, really, how do we go from idea to material using these technologies? That'd be helpful.

• 52:01Nathan Labenz?

  Yeah, please.

• 52:04Speaker 01

  So these technologies are used in a bunch of different ways. We pay a lot of attention to try and make them really useful in the prospect of designing materials. So you often start off with, you know, a functional property that you care about. You'll decompose a chemistry problem into a subset of things that you care about in the material. So maybe you're saying something like, I want to create a metal granite framework, which is a type of material, it's a big sponge-like material. And you say, I want to have a very big pore size, you know, I want to have like each bubble in my sponge needs to be really big because it needs to contain a lot of stuff. I'm going to absorb some things. So I need a large pore

• 52:44Speaker 01

  size and I might want to attach little tendrils, like little ligands to this metal gunny framework. And so I need to have lots of sites, active sites for me to attach these little ligands in, which are going to act like little tendrils that'll pick up the things that I want to absorb from my weird mixture that I'm going to pass this whole thing through. So you'll use a generative model to say, okay, well, generate me a material that has a large pore size and has lots of sites for me to attach ligands to. And that's what the generator model does. It will, it will give you maybe 10, 15, 20 different candidate materials that have some of these properties. But, you know, making 20 of these materials is still going to take

• 53:26Speaker 01

  you quite a long time. And there might be a bunch of things that are really difficult to use as generative prompts, but are really important. We know we can access through simulation. Maybe you want to understand, you know, what are the kinetics going to be like, how fast is this absorption thing going to take place? Am I going to block my entire pore space with these little tendrils and say I won't be able to get access to any things I want to catch? All these sorts of questions, which are really hard to put down in like a prompt, really hard to condition against, but simulation can answer really, really, really well. So then we'll go through quite a, you know, in-depth process of using our AI accelerated simulation tools to answer these questions at unprecedented levels of detail and accuracy.

• 54:11Speaker 01

  So by the time we get to making a decision about what to make, we've answered ninety percent of the questions that we need to in order to feel confident that we're going to have that sort of material. And so, you know, we're using the generative side to have that creativity. And then we use an AI side for qualification and for giving great insight into what's going on. And that stimulates more creativity a lot of the time because you see something unusual. Tim is always coming up with, I hadn't expected this thing to happen. I saw this simulation and this was not at all what the literature had said, but I actually think this agrees better with the experimental data. You know, you see these new things and it stimulates creativity, stimulates,

• 54:53Speaker 01

  stimulates new ideas and new types of materials. So you might go through that process two or three times. And then you'll end up with two or three things that you've like really fully qualified that probably slightly outside of the scope of what you'd come up with by yourself, the creativity of the AI and the generative processes, putting you in a different direction. And then you go into the lab. Your success rate when you go into the lab at that point is just way higher. You're pretty confident it's going to have those sorts of properties. You're doing something more novel than you would otherwise do. And that novelty point is just as important as the success rate, because you're going into new areas of chemical space and that's often where the really exciting materials and chemistries are. So that's the kind of process that we use and we use these

• 55:36Speaker 01

  different tools and functionalities at different stages of the materials design, R and D, functional, but so.

• 55:44Nathan Labenz?

  Go back to the very beginning of that, when you say you want something with a large pore size.

• 55:48Speaker 01

  Yeah.

• 55:49Nathan Labenz?

  And lots of active sites. Is that something you're saying in natural language or is there sort of a well-defined like property space that you are specifying that in?

• 56:00Speaker 01

  Those are things that you could have a world of fine property space on. I think we generally found that.

• 56:08Speaker 01

  Language in some ways is not the best conditioner for a general small model materials. Cause you're often looking for quite specific scale of values rather than a textual description. You know, text is great for describing an image because you don't have a scale of value with which to condition on. But you know, if you wanted a specific number for your pool size, but let's just give it as a number than it is to give it as a text, free text box.

• 56:38Speaker 00

  Yeah, because a lot of things you're trying to hit a sweet spot, right? So you don't want, yeah, and it can be quite quantitative. You want like the right, but not too much Goldilocks, right?

• 56:48Nathan Labenz?

  That makes sense. Well, maybe one more just detail question on the models themselves, and then we'll get into the application that is kind of the new headline. How big are these things? And what is the sort of compute requirements that go into them? Are there, are we like talking, you know, Biden executive order qualifying 10 to the 23? If we wanted to call this more of a, you know, bio type of thing, or yeah, where are we in terms of just resources into these models?

• 57:17Speaker 01

  I think your mental model should be like an image model, like a CNN based diffusion model, right? Like these U net models, which I guess most image sharing models are still probably units. So I think you should think of it like that, similar levels of data, similar levels of compute. So a lot smaller than the large language model. That's one of the reasons why we can deliver so much, you know, world leading models and the lab and developing materials. All, um, you know, less than 40 million raised. That's obviously still a huge amount of money, but you know, we wouldn't be able to train a large language model for that much money.

• 57:58Speaker 00

  Yeah, I mean, the exciting thing is to think that we are at that kind of GPT one state, right? If you really scaled these, I mean, what you could do with them, particularly on the training data side, or if you through some of these huge clusters, it generated really high quality training data sets. There's probably a lot of room to improve these things.

• 58:16Nathan Labenz?

  Yeah, okay. I remember we had two guys, Jonathan Frankel and Abhi from Mosaic to talk about their, among other things, efficiency-driving work. They took stable diffusion training under $50,000, last I heard. So I don't know if that's exactly where you're at, but it just as a general order of magnitude for the audience, you know, those are certainly way less than your trillion-dollar data centers. So with that, let's talk about this paper that you graciously shared a preview of with me, where you are modeling no less than the potassium ion channel that sits in the membranes of our cells and lets potassium

• 58:58Nathan Labenz?

  ions in and out in an ordered fashion. So maybe, again, we could start with like a, I think people will probably be familiar with like ion channels in cells, but why does this matter so much? It seems like it had been something for you that was a longtime dream to try to tackle a challenge like this. I think people would benefit from hearing why this was such a prominent mountain in your mind that you wanted to climb.

• 59:23Speaker 00

  Yeah, so there's always a problem that's fascinated me, as you say. And the basic idea is that the body is really an electrical device, right? We don't think of it this way, but it is. It's just like a computer uses electrons and metal to carry electrical currents around, but the body uses ions, right? Things like sodium chloride, you know, sodium is positive, chloride is negative, potassium ions as well, critically important. And the way it carries these signals is it creates, you know, the cells which have like a high concentration of ions, potassium ions in this case, and then when it wants to send an electrical signal, it will open up these things called potassium ion channels, you know, and the potassium ions can flow down those, and that creates a change in

• 1:00:03Speaker 00

  voltage, which then triggers follow-on processes. So it's really operating kind of in a way that you can think of it as an electrical device, and it's how just a huge number of the body's functions work, right? It's how your muscles contract, it's how your heart beats, all of the neurons firing in your brain, carried by these electrical signals, carried by the flow of these potassium ions in and out of cells. So it's incredibly important. And when Orb, this model first dropped, I wasn't working for Orbital and I saw it and I thought, wow, this looks really exciting. Really good accuracy, very, very fast was the exciting thing. And I started to try it out for many different things and it was doing a fantastic job, always giving me nice, stable results, simulated many electrolyte solutions.

• 1:00:46Speaker 00

  Which is what we talked about last time, which is what I was very interested in. But the fundamental reason I had always been interested in electrolyte solutions is apart from the fact that they're very important for climate change. They're also central to biology, right, to this, this electrical signaling function. And the real key challenge there had been these potassium ion channels, which no one has really been able to fully understand. Unfortunately, using experimental techniques or traditional computation. And in fact, we don't even know some of the most basic questions about it. So like down these channels. Is it just potassium ions flowing or is there water going down it as well? Like that's an unresolved question. There's papers out coming out on both sides and top journals like every few months.

• 1:01:28Speaker 00

  And it's kind of what I think of as a keystone problem or a kind of problem that demonstrates a fundamental lack of some capability we're missing. And so if a new tool came along that could solve it, then I think it would be really important. And so it's always been the dream to try and use the things I'm working on to address it. And because Orb was doing so well, I started to think, maybe I'm thinking about this wrong, right? Like maybe this is the universal force field, the thing that can simulate anything that computational chemists have been dreaming about for decades. And I thought, maybe this thing is finally here or some form of it. And I thought, well, if it was, what would I do, right? This is the problem that I would go and try and solve. So I set up the simulation and at the time I literally thought like. This is such a silly idea, like this is just not going to work. It's going to crash straight away.

• 1:02:11Speaker 00

  The models I had been building myself previously, you know, quite unstable as soon as you got out of distribution. So I thought. It's going to crash and do unphysical things. But in fact, it didn't. It started running stably. I started to see interesting things and it was just so exciting. I literally started jumping up and down. It was very cool. And the main thing is that like the training data looks nothing like these things, right? It's small crystals, like highly periodic repetitive structures. So it's never seen anything really like this. So the fact that it even just gave something reasonable and showing interesting things was really exciting.

• 1:02:44Nathan Labenz?

  Eureka moments.

• 1:02:46Speaker 00

  Yeah.

• 1:02:46Nathan Labenz?

  Increasingly coming from AI systems. Multiple follow-ups there, I guess. One is like, to what do we attribute the generalization from the small crystal training data set to this like quite wonky and in the picture, you know, there's papers and people can just Google a picture of the potassium ion channel to at least get it a rough approximation of what the thing looks like. You've got a cell membrane and then there's this sort of wonky thing that has certain symmetry to it that kind of, you know, protrudes through both ends of the cell membrane and through that channel, the ions can flow. That's way different from a like, there's nothing crystalline about it for starters.

• 1:03:30Nathan Labenz?

  Is this like a divine benevolence sort of thing? Or like, do we have an explanation for why it would generalize so well?

• 1:03:38Speaker 00

  Yeah, I mean, Johnny, my head attack, I don't know. Like, as I said, I didn't expect it to work. So my general philosophy is I started seeing, I don't trust, you know, a lot of AI things at first because they can hallucinate and do things. But I've tried to change my philosophy because I think you can err too far in the wrong direction, right? Of not trusting them enough. So I think sometimes it's worth trying and testing to see these things. And so it was definitely surprising. I wouldn't have predicted it. If I had to give an answer, I think I would say something about, I think it's what Johnny was saying earlier, right? Like, there's only a fundamentally small, you know, some finite amount of information about molecular interactions that you need to know.

• 1:04:17Speaker 00

  There's things like the dipole moment of each atom and how those interactions, how they spatially decay. And those can change in different circumstances, but there's just a finite set of information that you need. And maybe the crystal structures are enough to get enough of that information out that you can then generalize to simulate like a much broader space of materials. I don't know. Do you think that's right, Tim?

• 1:04:41Speaker 01

  Yeah, I think that's right. I think the thing that's remarkable and continues to amaze me there is the effects of scale. We take scale. Certainly data scale, engineering excellence, incredibly seriously orbital. It's part of our pedigree as an organisation. I think it's without casting too much shade on the academic community. I think we're kind of bringing that engineering, AI engineering into this discipline. And that has led to extraordinary results in natural language processing images. And to a certain extent, I think it's part of the result of that, that we're really bringing the best out of these very flexible networks now to train them. And just like you've seen these unbelievable results in so many other areas.

• 1:05:25Speaker 01

  Applying those same sort of techniques and abilities to materials has given something truly extraordinary. But I really agree with Tim that all the information is there, right? Like, there's a limited amount. You just, you're learning interactions between different types of atoms. If you've got some small. Even if those interactions are the same, whether they happen in a protein or they happen in a small crystal. As long as they're there in your crystal structure dating, training data. Then if you've got a network that learns very well and very self efficiently, then it should extrapolate really well. So I think it's extraordinary, but half of the things I've seen in AI over the past few years. You kind of learn to expect these things.

• 1:06:11Nathan Labenz?

  Yeah, expect the unexpected, expect to be surprised, expect to be amazed, definitely unavoidable these days. What's at stake with this? Is this more like a sort of AlphaGo moment where it's like? Holy shit, but it's not going to change the economy all that much in the immediate term? Or are there things that get unlocked very practically as a result of a better understanding? And I guess also, what did you learn about the potassium ion channel? Like, you know, how has our state of knowledge advanced due to this?

• 1:06:44Speaker 00

  Yes. So I mean, the first thing I need to caveat it with is that this hasn't been peer reviewed yet. I've sent it to some potassium ion channel experts who say, you know, it all looks very interesting and plausible and consistent with what we know experimentally. But, you know, ideally, we'd like to see some direct experimental confirmation of the things that we saw. But the exciting things that I saw were that water molecules did enter the, what we call the selectivity filter, like the narrow part of this channel. And that isn't normally seen in classical simulations, in the classical approach to simulating these things, where they tend to think that the potassium's going through alone. But there's some, although it's mixed experimental evidence that water is going through. So that was very exciting. It looks like something different there. And in particular,

• 1:07:26Speaker 00

  what we saw was there was a particular hydrogen bond where this little hydroxyl group on the protein was reaching out and kind of grabbing the water molecules and pulling it into the channel, which was very cool. No one's seen that before, as far as I'm aware. And it just makes a lot of sense, right? Okay, there would be some new kind of qualitative phenomena which the other methods don't pick up, that explains why they didn't see water going in, but we do. And we know if you delete that hydroxyl group, which they've done through mutation studies, the conductivity drops by almost an order of magnitude. So it really is doing something important experimentally. No one's really known what. This seems like a pretty likely candidate to me. So although it still needs to be confirmed, I'm very optimistic that this now explains this longstanding dispute

• 1:08:11Speaker 00

  in the field about what might be missing there. And it's something we can go out and experimentally try and validate now, which is exciting. In terms of why this matters economically and these kind of things, well, this is very important. Actually, a lot of diseases are caused by mutations in the potassium ion channel, right? So all sorts of heart diseases, a long QT syndrome, heart palpitations, things in my family, right? If your potassium ion channels aren't working properly, it will stop these electrical signals flowing in the right rhythm, and it'll impact the beating of your heart. And so if we want to understand, like, how do we fix those mutations? We may need to understand first, how does it work in the non-mutated form? And then we could simulate those mutations and see how they impair the function.

• 1:08:53Speaker 00

  And then even maybe one day start thinking about designing drugs to restore that function. And that sounds a little sci-fi, but this is actually already being done for chloride ion channels. Vertex have small drug molecules that can restore the function of chloride ion channels. So I think there is definitely biological applications for these kind of things in terms of real world impact.

• 1:09:17Nathan Labenz?

  Cool. Just to make sure I understood the piece correctly about the hydroxyl group. Previously observed was that if you make a cell that has a mutation such that that group is not there, then the channel works order of magnitude less.

• 1:09:37Speaker 00

  Exactly, yeah.

• 1:09:38Nathan Labenz?

  And what you saw that was new is presumably without feeding any of that kind of background information into it, you saw a mechanism that made you say, well, that's probably why that's happening.

• 1:09:51Speaker 00

  Yeah, I mean, if I had to bet, I would think it definitely is. It seems very plausible. It was just doing something new that made a lot of sense that I haven't seen reported in the literature anywhere else. So at the very least, it's like a very good hypothesis. And if I had to bet, I think it'll be confirmed.

• 1:10:08Nathan Labenz?

  So how far do you think this goes in terms of? And there could be like a really obvious, like, hard limit that relates to sort of the structure of the models that have been trained or, you know? And then maybe there's more like a conceptual answer, but we've got these like very small crystals, training model that low and behold can simulate something as complicated as the potassium ion channel. It was interesting to read how you sort of fixed essentially a sort of superstructure. You've got this whole cell, right? This whole thing is in a cell and that's in a body and that's in a world. But you got to draw a box around somewhere to, you know, bound your simulation. So you basically said, we're going to, beyond a certain box, we're going to kind of treat

• 1:10:52Nathan Labenz?

  everything outside of that as fixed in place. And so if I understand correctly, there's like some more atoms that are represented in the simulation, but they're just not allowed to move in order to kind of focus the simulation in the zone of interest.

• 1:11:10Speaker 00

  Yeah, exactly. So this, if you go back to early ways, we did simulation of these things, they would use these kind of tricks. And so I thought, you know, I could use some of them here with this new technique. What you do now in a, in a modern simulation, and we will do this with orb now that, you know, if we could scout the computer. At the time I was doing this, I only had one V one hundred, right? So very restricted in my computational resources. And the basic idea is you want to simulate this, what's called the selectivity filter, this like narrow part where everything interesting happens. But you actually need the surrounding atoms, because they help stabilize that, that central part, right? Where the interesting stuff happens. And then you have a problem, because now you need the outer atoms around those. And so you have a kind of infinite regress, and you end up needing a huge system. The way we

• 1:11:52Speaker 00

  handle this normally is a thing called periodic boundary conditions, which is what Johnny was talking about earlier, where if one leaves the, the edge, it comes back on the other side. And so you run very big simulations is what people do now with the classical techniques. I just wanted to have a quick look to see, like, is there anything interesting going on with the selectivity filter? So I had to kind of cut that infinite regress off at some point. So I did it about twenty angstroms away. So it's, I don't know, four or five layers of atoms out. And I just froze that outer layer to give it a kind of thing that would keep it nice and stable for the, and long enough to kind of see the interesting dynamics in this activity filter. So it's a little bit of a hack. We'll fix it up eventually, and do the full simulation of the entire system in the membrane is the other thing. So because this potassium ion

• 1:12:35Speaker 00

  channel sits in the cell membrane, which is like these fatty layers, you actually need a very big system to simulate the whole thing. And that should be possible, you know, with bigger GPUs and a bit more time.

• 1:12:47Nathan Labenz?

  Yeah, it's insane that this was done on one V one hundred, you said?

• 1:12:51Speaker 00

  Yeah.

• 1:12:53Nathan Labenz?

  Yeah, that's wild. So is it just a matter of scaling up your inference infrastructure? I mean, the, the model, I presume, has some sort of structural limits too, right? You couldn't take the orb model and like simulate a whole cell in it if you just had enough inference compute, right? What, what sort of constraints are we gonna hit as we try to do bigger and bigger systems?

• 1:13:14Speaker 00

  Yeah, it's very interesting. I don't know. We'll have to wait and see. I did have a go at simulating a larger one. I saw it, it got a little. And stable, but it's very tricky setting these simulations up. Obviously you need to get that the cell membrane has to be the right thickness and all of these things. So there's a lot to do there to work out how to scale these up to larger systems. I mean, the dream is that you can just continually scale them, right? It's an open question as to whether the current architectures we're using can do that. It's possible you may need a kind of longer range models that can treat these longer range, particularly electrostatics. In these systems, you get charges which have very long range interactions because they

• 1:13:55Speaker 00

  decay as one on R. So which is the same as gravity, right? So they can go to the moon. So you can have these very long range effects. So it's hard to say what will happen when we go to very large systems, but it'll be interesting to find out.

• 1:14:09Nathan Labenz?

  But there's not like a equivalent of a context window or some sort of hard limit built into the current models that?

• 1:14:16Speaker 00

  Yeah, you would hit the memory limits of the GPU if you tried to scale it too much, although that's not really our bottleneck at the moment. We can do huge ones. The problem is it gets a bit slow when you do huge systems, right? So then you would have to start scaling to mini GPUs. People have done this with a neural network, simulated a whole HIV capsid, and it looked stable, not with all, but with with another force field. So you can simulate huge systems with it. People have done it. It looks pretty stable. It's hard to validate it because the time scales are quite short, right? Because it's so computationally expensive. So that's really the problem at the moment is how to get these longer time scales. And I think there this coarse graining idea will be key. We will run all atom simulations, and this is what I did for the salt crystallization. But then you say, we're going to ignore the

• 1:14:58Speaker 00

  solvent molecules and train a new model, which just predicts what's called, well, technically is a free energy now. And but it does the same thing as a as an all atom simulation, but now you're just simulating a subset of the system, just the pieces you care about. And that will get us to these very large scales. So I'm optimistic that we can start to think about doing these kind of simulating whole cell type stuff with this approach. It will just involve some hard work. But I think the theory is all there. The tools are all there. It's just a matter of really investing in it.

• 1:15:28Nathan Labenz?

  And there's nothing about the current models, you know, obviously, large language models have like just a finite number of tokens that they can handle at any given time. Beyond that, they just can't do it structurally. Is there, I guess, there's not a similar hard limit here to like a certain number of atoms or whatever?

• 1:15:45Speaker 01

  So what happens in a language model is that when you have a finite size of your context window, that's because you have this positional encoding that is a feature of your input. And so during training, because you only can train from a memory perspective at training time, a certain context window length, each position in that context window has a specific token, a specific feature associated with it. So at test time, you can't generalize much further than what you've trained on. That's stuff like Mamba or these state space models, right? They don't have this explicit positional encoding.

• 1:16:27Speaker 01

  And that means that you can have kind of infinite context window. And I think similarly, when you think about the sorts of models that we use, we don't have any explicit positional encoding. And then that means that we can really extend just as far as the compute resources can can take us and basically linearly increase the size of your simulation by the number of chips that you can add on to your simulation when you think about an atomistic model. And so really, the sky's the limit there. You can make as big as you possibly want, as long as you've got the chips. We don't have any fundamental model architecture challenges like you would do with a transformer based language model.

• 1:17:06Speaker 00

  And so we're exploiting the nature of molecular interactions that they tend to be local, right? Which is a nice property that they have, which enables you to kind of do this.

• 1:17:16Nathan Labenz?

  So I have to ask, when do we get our room temperature superconductor?

• 1:17:22Speaker 01

  It'll happen. Give us at least five years, but it may happen. Before that, we're going to be solving carbon removal and scaling AGI energy efficiently and cheaply. So we've got some things we're solving first, but once we've done those, we'll think about the room temperature superconductor.

• 1:17:41Speaker 00

  Yeah, the problem with superconductivity is what we call a highly correlated electrons system. So it it doesn't have this property of very local behaviors. It's just a quite complex quantum mechanical things going on, which our the current level of theory we're training on, density functional theory, i- is not really good enough to describe that. But it could be very useful for finding new crystal structures and, you know, working out how to make new superconductors. Once you know how to induce the kind of electronic properties you want.

• 1:18:11Nathan Labenz?

  Interesting. Okay, cool. Well, maybe talk a little bit more about that sort of roadmap, you know, between here and the room temperature superconductor. You know, what else is on your to do list?

• 1:18:22Speaker 01

  Yes, our focus at the moment is really on data center materials. We think that scaling out AGI is one of the most important things that we can do, and that new materials are going to be a really important part, making that both cost efficient and sustainable. As the rack density increases, as power density increases of the chips that we're using, we need better and better materials for thermal management, better materials for all sorts of different things. You know, we can only imagine that we're going to need better and better chips in order to bring AGI to life. So in that sense, new materials for data centers, new materials for AGI is our focus. We also want it to be sustainable, though.

• 1:19:05Speaker 01

  And so decarbonization is our other focus as well. And we'll see coming forward a pipeline of materials and products in this space. So watch this space. We've released information about one of those products, more becoming over the course of this year.

• 1:19:25Nathan Labenz?

  We're told, but we're not the one that's already out?

• 1:19:28Speaker 01

  So that's our that's our carbon removal material. This is a material that we worked on and will be piloting in partnership with AWS towards the end of this year. We have designed this material to operate and capture CO2 under the operating conditions of a data center. It allows us to utilize the waste heat and waste airflow of the data center to maximize the CO2 capture capacity and ultimately offset some of the scope two, scope three emissions of data center operators. You know, it's perhaps the biggest, largest CapEx build out in history, what we're currently seeing with the hyperscalers and the data centers now. But those are also companies that are very

• 1:20:08Speaker 01

  serious about their climate goals and certainly have the resources to invest in reducing the carbon intensity of their operations. So we see that we're, this is a great opportunity to be deploying really advanced and really effective carbon removal and decarbonization technologies.

• 1:20:26Nathan Labenz?

  Yeah, I think technologies like this are the only thing between the hyperscalers and a lot of broken net neutral promises, I suspect. Because I don't think they're going to not build the AGI data centers. So something's going to have to give. Is that material, were you kind of describing it earlier when you described like a spongy, you know, high pore size with a bunch of active sites to grab onto things? Is that the sort of mental model I should have?

• 1:20:52Speaker 01

  I think, I think that's like, we can't say too much about that because we're currently protecting some of that information, but it's certainly a sponge-like material. That's right. And we're certainly reactive sites that should selectively interact with CO2 over all the other stuff that's going to be in the air. There are things that we care about, like the absorption capacity at the temperatures that the data center is operating on, the, the composition of the air, things like that. And we kind of tailor our materials and develop new materials that are specifically designed and can take a unique advantage of that operating condition. That's a really hard thing to do because developing a new material for CO2 capture, it takes a huge amount of time, years and years and

• 1:21:32Speaker 01

  years and years. We're doing that incredibly fast. And so therefore are really able to maximize the unique opportunities of that deployment situation. It's just one example of, of the impact of AI in some of our work.

• 1:21:48Nathan Labenz?

  Do you have to make? Like you could imagine multiple paradigms for that sort of material. You could imagine a sponge that absorbs a bunch of stuff and then you sort of bury the CO2 with the sponge and you kind of keep bringing more and more absorbent material and it's like itself becomes sort of a waste product. Or you could imagine you could like clean it off somehow and reuse

• 1:22:13Speaker 01

  it. So you're gonna do the latter, so you've got to be, you know, absorbing and disorbing the CO2 and that's a really important part of the overall costs of doing the CO2 capture. If you can bring down the heat at which you are disorbing that CO2 or, you know, tailor that temperature so that you can utilize the waste heat of a data center, then that's going to really massively reduce the energy costs. But the longer that you can reuse that material, the better. I don't think you're ever going to get to a point where disorption of that CO2 is going to be more energy efficient, oh sorry, it's going to be more energy costly than making another, you know, X tons of that material. Like that

• 1:22:54Speaker 01

  making process is super energy intense and is a costly thing. And if that material can then last for 10 years, then that's one mechanism which you can radically reduce the cost of that CO2 capture.

• 1:23:07Speaker 00

  Yeah, like the one-off capture is pretty solved. It's really that regeneration problem that's.

• 1:23:11Speaker 01

  Yeah.

• 1:23:13Nathan Labenz?

  Making stuff in general and just manufacturing a lot of stuff is obviously quite a different business to be in versus coming up with the models that designed the materials in the first place. I know you said earlier that you have traditional material scientists on the team and that helps you like close the loop. I assume you're like licensing the things? You're not like actually building factories and also like scaling up production, are you? But tell me.

• 1:23:38Speaker 01

  I think that's, that's right. You know, exactly, you know, whether it be licensing or whether we have suppliers, you know, Nvidia don't have a fab, but they're still sell chips. You know, we don't have to build up manufacturing of material in order to sell materials. But I don't think that we're going to be, you know, standing up a massive plant somewhere in Jersey. That's true.

• 1:24:03Nathan Labenz?

  Do you need to have a lot of physical capital, though, to even just do the validation experiments? I mean, it would seem like that in and of itself would be a kind of scale game, right?

• 1:24:12Speaker 01

  Yeah, we need more physical capital than a software business does. I think that's right, but we don't, you know, we're not going to be raising money for CapEx. Certainly, our lab is cheaper than all of the GPUs you'd need to run an AI company. It's not beyond the scope, I think, of what venture capital-funded businesses should be able to do.

• 1:24:36Speaker 00

  But I think it's so important to have that internally, right? In terms of being able to get feedback really quickly from experimentalists about what's working and what's not working and what they need and stuff is such an important thing that it would just be so much harder to do it without.

• 1:24:52Nathan Labenz?

  Yeah. Cool. This is amazing stuff. So I'm sure you've seen this paper that kind of made the rounds recently from Aidan Toner Rogers from MIT, where I don't think we know, you know, what models are being used or what company this research was done at, but basically a sort of behavioral industrial economics kind of study of the application of models in this general space, anyway, to the practice of material science. The headline results were significantly more materials discovered, forty-four percent increase, resulting in thirty-nine percent increase in patent filings and seventeen percent increase in downstream product innovation.

• 1:25:37Nathan Labenz?

  Interestingly, the AIs, as characterized in this paper, are doing a lot of the idea generation. And then the scientists were kind of following after and like more focused on the validation. So overall, more output from this system, but one of the kind of... Important caveats was the scientists, eighty-two percent of them, reported basically lower satisfaction with their work because I guess they felt like, you know, they want to be the ideas guys and now the AIs are the ideas guys and they are, you know, sort of left to, you know, to be the hands that like validate the AI's ideas. I guess you could react to that in multiple ways, but I'm interested in sort of how you

• 1:26:21Nathan Labenz?

  think those numbers in terms of increased throughput relate to what you expect or aim for? And then also, is there anything, you know, we got Tim jumping up and down here, so I guess not, you know, you're in the eighteen percent who are finding the joy in, in working with the AIs. I wonder if you have any outlook on like, and maybe it doesn't even really matter. I mean, I'm not one who gets too hung up on like, oh, my job changed. Hey, if it's gonna, you know, save, if it's gonna solve climate change, I'll, I'll accept, you know, a little lower job satisfaction here and there. But I do wonder what you think this implies for, you know, kind of future of work, big picture sort of thing. Are we gonna adjust and find our, you know, joy in the work again? Or is this like a sign of things to come? So, yeah, just multiple reactions to that paper, if you have any.

• 1:27:06Speaker 00

  So, on the enjoyment front, I mean, my take is I'm, I'm a physicist and mathematician by training, right? And I've been trying to get into chemistry because it's, it's so important and I love it, but I don't have that kind of deep knowledge of it. In fact, I didn't actually study it in undergrad or anything. So, if from my point of view, these tools are incredibly empowering because they kind of give me access to that intuition, right? That we were talking about. Cause like now I can just run a simulation, see these bonds forming. And, and so I just look at these simulations and I'm like, oh, okay, that's what's happening, right? And I don't need to know anything about potassium ion channels. I don't need to know the literature that, that hydroxyl group is known it to be important. I don't need to know like a deep intuition about what bonds with what. I just run the simulations and see it. So, I have sympathy with the people who've spent their careers building up this intuition

• 1:27:51Speaker 00

  and now something else is coming in and can do it as well. So, yeah, I, I do think that is a sad thing, but, but I think people, I think there will still be things people can be useful for. And I think the satisfaction of like getting materials out there in the world that are having an impact much quicker and much more faster will more than offset like that, that spark of like, oh, I solved the problem myself, you know, like you can do crosswords for that, whatever, but like, this is too important to like, think about this, you know, the happiness of the scientists working on, is my take on that.

• 1:28:21Speaker 01

  It reminds me, I think, the job satisfaction of a bunch of my friends who work in A I research. It's declined. Yeah, I mean, it's declined significantly because, you know, you used to do really creative stuff. And now it's engineering and there are a limited number of experiments you yourself can run. And you can't go and do your own research project anymore. I've got a great idea. You can't really experiment with it because it's so costly. They're just on the computer resources to test it fully. So people's jobs have radically changed in A I research over the past two and a half years.

• 1:29:01Speaker 01

  To, you know, for for many of those researchers, they've declined despite the fact that their jobs have become more prestigious. They're sort of getting paid more. They've, you know, they're they're seeing A G I take place in front of them.

• 1:29:16Speaker 01

  I think that for a lot of people, that's going to be the case. We're going to get more productive. We'll get richer. But we will have to find other ways in which to feel value from our work and that creativity. You know, these systems are really creative. Creativity is one of the things, first thing that's going to go. You know, it's going to be manual tasks, robotics will come quite a lot after the creative industries have been fully accelerated through the use of A I. I'm not so optimistic that people are going to enjoy A I. I'm though fully convinced that the only way that people are gonna, you know, the future is going to be a happy world. You know, is through A I because we need to produce more stuff for less money.

• 1:30:00Speaker 01

  And that means that we need a greater acceleration of scientific discovery than we've ever had in the history of humanity. And the only way that we're going to achieve that form of acceleration is through A G I. So we've got to have it as fast as possible, deal with the pain over the short term to get to the glorious future, as Sam Altman would put it, of, you know, a large portion of economic activity being fully automated.

• 1:30:26Nathan Labenz?

  That's probably a good note to end on. Anything else you guys want to talk about that we haven't touched on?

• 1:30:31Speaker 00

  One point was just on the the combination of L L M's and neural network potentials, these, you know, universal force fields is quite powerful. The point worth making is if you're worried about, so central challenge, if you're worried about artificial superintelligence and things is how are these things going to get some kind of experimental validation? And then maybe they can design, you know, whatever new materials they want, but they need to be able to validate those things. You know, this combination of L L M's with the force fields, which gives them the kind of ground truth, perhaps enables them to do these things internally without needing to go out into the experiments and measure things experimentally. So it's going to be very interesting. I think it's a powerful combination because you get that kind of grounding, the way to check hallucinations

• 1:31:13Speaker 00

  and correct hallucinations physically. And we're seeing this with these new O three, these models that can solve these incredible math benchmarks, but it needs to be problems where you can validate them somehow, right? And so I think these neural network potentials potentially provide that for science, a way of validating hypotheses automatically in the computer rather than having to go out to the lab. So I think they're going to be a powerful combination moving forward. It'll be very interesting to see.

• 1:31:36Nathan Labenz?

  Do you think we end up with like omni modal models where the sort of A G I is something that with the same weights is like talking to me and running molecular simulations?

• 1:31:51Speaker 01

  Certainly will have a shared embedding space. I don't know, you know, because I think some of these image generation models, not your decoder for the image is kind of trained perhaps like separately or feels like a different model in their structure, but they've got the same same embedding space. I think we'll certainly have that. I think we'll have omni modal embeddings. That's where the concept and the thinking happens. I think that's almost inevitable. I integration of the atomistic stuff into the training of a scientific A G I, I think, would be a really cool thing to do. And I imagine someone's already working on that somewhere.

• 1:32:33Nathan Labenz?

  Yeah, I wouldn't doubt it.

• 1:32:36Nathan Labenz?

  It's going to be hard to make sense of it. I do sort of worry that for multiple different reasons, obviously, there's been deception discourse recently and deception discourse was quickly followed by reasoning in continuous latent space. I am among those who's like, geez, you know, I kind of liked it when I could read the chain of thought. You're telling me now that the chain of thought is going to be all, you know, in latent space. That's, that's going to be a little weird. And yet it does seem like it's a very powerful attractor for multiple reasons. I mean, including that the systems will be able to communicate with each other much, much more effectively if they don't have to, you know, condense everything down to language tokens to send messages back and forth. So I don't know

• 1:33:18Nathan Labenz?

  how we get around that. But it does seem like that vision takes us very quickly to a world where we are sort of just hoping the AIs are doing the right thing for us, right? I mean, is there an alternative vision aside from that?

• 1:33:35Speaker 00

  Yeah, I mean, someone I saw recently said, you just hope moral realism is true, right? That like it's going to find the same, it's going to do the right thing.

• 1:33:45Nathan Labenz?

  All right, sobering notes to end on, but, but inspirational work. I mean, really, really cool stuff. And we'll post links, of course, in the show notes. But, you know, some of the, the images back to the crystallization and also with this potassium ion stuff are really, really impressive stuff. So I appreciate your guys' time today in educating me about this. And I'll give you the official send off. Jonathan Godwin and Tim Duignan of Orbital Materials. Thank you both for being part of the cognitive revolution. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.