Fluid Intelligence: Simulating Solutions with Tim Duignan

Explore the fusion of AI and computational chemistry with University of Queensland's researcher, Tim Duignan. Learn about transforming our understanding of electrolytes, the art of simulating physical processes using neural networks, and the potential of AI in scientific breakthroughs. Discover Tim's journey creating AI models for complex system predictions and join the discussion on advancing AI's role in future scientific discoveries. Apply to join over 400 founders and execs in the Turpentine Network: https://hmplogxqz0y.typeform.com/to/JCkphVqj RECOMMENDED PODCAST: Byrne Hobart, the writer of The Diff, is revered in Silicon Valley. You can get an hour with him each week. See for yourself how his thinking can upgrade yours. Spotify: https://open.spotify.com/show/6rANlV54GCARLgMOtpkzKt Apple: https://podcasts.apple.com/us/podcast/the-riff-with-byrne-hobart-and-erik-torenberg/id1716646486 SPONSORS: Oracle Cloud Infrastructure (OCI) is a single platform for your infrastructure, database, application development, and AI needs. OCI has four to eight times the bandwidth of other clouds; offers one consistent price, and nobody does data better than Oracle. If you want to do more and spend less, take a free test drive of OCI at https://oracle.com/cognitive The Brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference. All while remaining affordable with developer first pricing, integrating the Brave search API into your workflow translates to more ethical data sourcing and more human representative data sets. Try the Brave search API for free for up to 2000 queries per month at https://bit.ly/BraveTCR Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off https://www.omneky.com/ Head to Squad to access global engineering without the headache and at a fraction of the cost: head to https://choosesquad.com/ and mention “Turpentine” to skip the waitlist. CHAPTERS: (00:00:00) About the Show (00:02:47) Introduction (00:04:09) Why electrolyte solutions are important (00:06:53) What properties are we trying to predict with electrolyte solutions? (00:09:33) Battery fires (00:13:38) Molecular dynamics (00:16:40) Time step (00:18:51) Scaling (00:23:09) Sponsors: Oracle | Brave (00:25:17) Decoherence (00:27:23) Neural Network Potentials (00:31:04) How big are the models? (00:35:46) What architecture is used? (00:38:12) Equivariance (Part 1) (00:42:29) Sponsors: Omneky | Squad (00:44:15) Equivariance (Part 2) (00:44:16) AlphaFold3 (00:46:52) Zero-shot latent space communication (00:48:21) What is coarse graining? (00:54:31) How to know if there is no crystallization (00:56:53) What is the role of water in the simulation? (01:01:25) Crystallization (01:05:26) Matching surfaces (01:09:32) Self-ionization of water (01:13:22) Active learning (01:16:41) Temperature (01:19:20) Ice and water (01:21:12) Scaling up the model (01:25:21) Big tech singularity (01:27:54) China is not far behind (01:29:15) Safety concerns (01:31:16) KANs (01:34:36) The future of scientific progress (01:39:28) The art of discovery (01:43:02) One model to rule them all (01:47:46) AGI, its risks and benefits (01:50:21) Power vs. Generality (01:53:01) Generality for Utility (01:53:40) Outro

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, I'm thrilled to share my conversation with Tim Duignan, an applied mathematician at the University of Queensland, Australia, who is using cutting-edge AI techniques for computational chemistry with the goal of deepening and perhaps one day revolutionizing our understanding of electrolyte solutions, from the humble and familiar

• 0:42Nathan Labenz?

  saltwater to the latest lithium batteries. Tim recently shared a short video of salt crystallizing in a water solution. He called this "the most exciting result of his career," and this mesmerizing visualization reached nearly 2 million people on Twitter, sparking widespread fascination. In this conversation, we unpack all the techniques that went into making that magic moment. We explore how Tim simulates fundamental physical processes and how by training neural networks on quantum mechanical simulation data, he's created models that can accurately predict the behavior of systems of atoms and molecules several orders of magnitude faster than conventional approaches. This breakthrough unlocks the potential for much larger and longer-lived simulations at various levels of scale, all with

• 1:27Nathan Labenz?

  compute requirements that are remarkably modest by the standards of the foundation models that we're most familiar with today. Throughout our discussion, Tim offers a masterclass crash course on computational chemistry. He explains concepts like coarse graining, that's a method to simplify complex systems by abstracting away detail thus ultimately averaged out, and also equivariance, a way of representing data that abstracts away from any particular coordinate system, and thus helps models generalize better, all with remarkable clarity. We dive into the technical details of how these neural network potentials work, the surprising behaviors that they can capture, and the exciting potential for integrating these techniques into larger, more general AI systems in the future.

• 2:09Nathan Labenz?

  This conversation is a powerful reminder of how AI is accelerating scientific discovery in ways that we're only beginning to grasp. And for me, it's a compelling reason to believe that AI systems could become meaningfully superhuman in all sorts of weird and unexpected ways over the next couple of years. As always, if you're finding value in the show, we'd appreciate it if you could take a moment and share it with a friend. You can always contact us via our website, cognitiverevolution.ai, where I'm now accepting resumes via a Google form. And I always welcome DMs on any social network. Now, I hope you enjoy this deep dive into the frontiers of computational chemistry and its intersection with AI with Tim Dagman. Tim Dagman, welcome to the Cognitive Revolution. Hi, thanks for having me. I'm a big fan of podcasts.

• 2:53Nathan Labenz?

  That's kind to say. Thank you for getting up super early. We're 14 hours apart, or 10, I guess, if we count the other way. And appreciate you accommodating, and I know it's early there for you. I was attracted to your work in the first place, as I so often discover things on Twitter, and you posted a tweet that probably is the most viral tweet you've ever posted, I imagine. It certainly went, probably traveled a lot farther than you expected, which basically showed a short video of salt crystallizing in water solution, so basically salt water with a little salt crystal. And you said that this is the most exciting result of your career, and that inspired me to dig in and try to learn more.

• 3:38Nathan Labenz?

  Maybe for starters, give us a little bit of the sort of context. I think people, obviously, who tune into the Cognitive Revolution are paying attention to AI, they're probably not paying attention to the sorts of computational chemistry work that you're doing. So, you know, they may not even have any sense for like, we're still studying salt water, like that may be a sort of revelation to some. So can you give us the background of what exactly it is you're studying? What are the sort of fundamental questions you're trying to answer? And then we can get into how you're using some cool AI techniques to approach those questions.

• 4:09Speaker 01

  Yeah, definitely. So I've been fascinated with salt water, or the more technical term is electrolyte solutions, for over a decade now. I've been studying them since my PhD. And I've kind of fallen in love with them, you know, and you talk to people, normal people, and you say I'm studying electrolyte solutions, and they kind of glaze over a bit, and they go like Powerade or Gatorade, and they don't quite get it, you know, they're like, why would you be studying that? Surely we know everything there is to know about them. But remarkably, we really don't. And there's still a ton we need to learn about them and understand them better. And they're actually also incredibly important, right? They're, I would argue, one of the most important substances on the earth, right? The ocean is a salt solution, obviously, but it's, saltiness plays a critical role in all sorts of phenomena. So climate change, CO2 dissolves into the ocean to form carbonate ions, right?

• 4:54Speaker 01

  And that's critical into where our CO2 is going. In all of the technologies we need to fight climate change as well, electrolyte solutions are critically important. Batteries, they determine how fast a battery can charge and how long they can last. You know, they're the thing that connects the positive and the negative terminals in the lithium ion battery. So they're really everywhere, and then biology is the other big one, right? The flow of ions in and out of your cells is what creates all of the electrical signals that make life possible. It's what all of your brain functions on, and the body has developed all of these machines called ion channels and ion pumps to move these ions around. So they're critically important just throughout all of science and technology, really. So we really want to understand them better. And so that's why I've been working on trying to understand them for so long.

• 5:39Speaker 01

  And the remarkable thing about it is that we really can't even predict their most basic properties from what we call first principles, right? Which means, at least in our context, means from fundamental physics, starting from nothing but the equations of physics. And that's the dream I've had really for a decade or now, was could we start from nothing but physics, and particular Schrödinger equation and Newton's laws and these kind of things, and predict the properties of these electrolyte solutions? Because that would be an incredible achievement if you could do it, I think. And then it would be a really nice demonstration, you could take this technique and then start taking it further and simulating larger and more complex systems as well. So that's kind of what we've been working on for so long, and this is why I'm excited about this latest, yeah, paper we put up on Twitter, which, yeah, got this remarkable response, I think.

• 6:20Speaker 01

  It was great to see other people getting excited about it, you know, for a long time I thought, no one cares about this, I'm just working on it kind of on my own, or with a small team of people, you know, and it's very small number of researchers in the world actually working on these salt solutions, right? So it's amazing to see other people can get excited about it as well. So that was really great.

• 6:36Nathan Labenz?

  So I think great job on motivating the the problem in terms of all these different domains, in which it turns out to be important. If you can cover climate change and batteries and life, you can even touch on all those with a single question. It's clearly an important question. When you say that we can't predict the properties of these solutions from first principles, what are the sort of properties that you would want to predict? And I guess those are all just determined experimentally?

• 7:09Speaker 01

  Yes, exactly. So currently, we measure all these properties experimentally, and the main one would be things like solubility. So like, how much can you dissolve in water or solution? That's a really important one, particularly for batteries. That determines you want a battery, an electrolyte that can dissolve a lot of ions in it, to have high concentration ones in many cases. And then there's kind of more technical chemistry things that come along with that, things called activity coefficients and chemical potentials and kind of all of these important quantities that and what you do with those numbers is you actually then plug them into larger scale models that you could then use to model bigger processes. So you might plug those into a climate model, for instance, to predict, you know, the different properties of the ocean and then saltwater, actually aerosols form as well from these electrolyte solutions.

• 7:52Speaker 01

  So there's all sorts of different cases where you would really want to know these properties that you could then plug into the models. And at the moment, we get all that data, yeah, from experiment. And the problem with that is that in many cases, you know, say you have very high temperatures, which very often occurs in many minerals processing instances, you know, where again, these electrolyte solutions pop up at this very high temperatures, it's very hard to get that data experimentally because you need very expensive equipment that can handle those kind of temperatures. Or whenever you have like a new electrolyte solution that isn't well studied, then suddenly it's like, oh, we've got to go back to the lab and we've got to do these exhaustive experiments to characterize these electrolyte solutions as a function of temperature and pressure and get all that data. And that can be very, very expensive. And that means that very often, you know, people just don't bother.

• 8:34Speaker 01

  And in particular, you know, it's very hard as an experimentalist to get funding to just go and do basic measurements of electrolyte solutions, right? There's not, you're not going to get a nature of science publication out of that. And so there's honestly not a lot of this happening really anymore. Most of the data that we have comes from, you know, a long time ago when experimenters were working on building up these databases and we're not really building these databases much more anymore. And so this is a real problem when it comes to machine learning and AI because, you know, databases are key here. So in particular for electrolyte solutions, this is a real problem, the limited size of the databases. And there's people doing amazing jobs, taking the data that's out there in the literature now and taking models that can allow you to extrapolate that data a little bit, but it's still inherently limited to having good measurements relatively close to the systems that you're interested in. So that's really an issue there.

• 9:20Nathan Labenz?

  Interesting. Are there like surprising findings anecdotally that could sort of just give us a little bit more intuition for sort of the kinds of things that you may encounter if a solution doesn't behave the way you expect? And I guess I'm imagining like battery fire is the sort of extreme failure mode, right?

• 9:39Speaker 01

  Mm hmm. Exactly. So that's a big one. So in a battery, you have the positive and negative terminals, and you've got the electrolyte in between. And actually, on that surface, that interface between the positive, the metal terminal, or the crystal terminal and the liquid electrolyte, you can actually get salt precipitating out. And that plays a critical role in determining the lifetime of the battery. And if that salt layer, that crystal salt layer on that surface of that electrode doesn't form properly, you can get real problems. And particularly, you can get things called lithium dendrites forming, which then can cause a battery to short circuit. And, you know, that then destroys the battery and starts these battery fires. So actually, yeah, this process of crystallization can be really important there, for instance, where you want your salts to crystallize out in the right form, essentially, because if you

• 10:21Speaker 01

  get the wrong form, you can get really nasty things. And this is something that we still really don't understand at all. So there's one thing, which is just predicting these kind of, what we call thermodynamic properties, which is what I was talking about before. But then another thing is often you just want to know, like, at the molecular scale, what exactly structures are forming? What are these things? What do these salt crystals look like? How are they arranged? And these kind of things? Because that would give you useful information about, you know, what's going to happen when we have a battery and we give, put it at a certain temperature and pressure, how and with a certain charge on it, say, what's going to happen with that, that layer of salt on that surface of the electrode is a really important question for, you know, batteries. And, you know, things can go badly wrong if you don't understand this properly.

• 11:01Nathan Labenz?

  So for something like a battery, are there sort of additives? I guess I'm thinking about like doping sort of strategies. Because you might say, okay, here's a naive lithium battery with sort of the simplest possible recipe. Now I may be making an array of these if I'm Tesla and I try to operate them under a bunch of different pressure and temperature conditions and characterize like what the viable range is and figure out if that's going to work for me or not. And then presumably the naive approach doesn't really work and then I have to go. Now I start adding in like little admixtures doping type of things. Is that?

• 11:34Speaker 01

  Yes, definitely.

• 11:35Nathan Labenz?

  Help me develop my intuition there as far as it goes.

• 11:38Speaker 01

  So yeah, this is a big field and there's a guy called Jeff Dunn, who's really the leader on this, who's developed, he's doing some really nice work on this, developing the million mile battery, right? Which is the idea that you want to make a battery that can go for a million miles, which is very cool. And the way essentially they do that is they're using fairly standard lithium ion batteries, but they're doing exactly what you're saying, doing different additives to the electrolyte to try and control the formation of what's this layer on that surface of the terminal, the positive terminal, the negative terminal of the battery, which is called the solvent electrolyte interface. So they're trying to control that process by adding different chemicals to the electrolyte before you charge up the battery. Because actually the first time you charge up the battery, which is called the formatting step, that's when this layer forms. It's really critically important for determining how long the battery will last.

• 12:21Speaker 01

  But the critical thing is that the way we do that currently is a huge amount of experimental trial and error, right? So just trying different things, adding them. And this is a real problem if you want a million mile battery, right? Because you have to then add, put the additives in and then cycle it for the equivalent of a million miles, right? Which takes a long time. So there's a huge amount of experimental effort to try and find these right additives to do this. So if we could instead kind of understand it from first principles, get a really deep molecular scale insight and understanding of exactly how these crystals are forming and what effect that's going to have on the lifetime of the battery. That's the dream and that would be really useful, right? And so there's actually some ways of machine learning are coming in here. So people, you know, you can do things like you put some additives in and then cycle the battery for a few cycles and then use machine learning to try and predict what it's going to do after a million.

• 13:04Speaker 01

  So there are other ways you can use machine learning AI here, which is quite interesting. But in the long term, I think to like really crack this problem and solve it once and for all, you'd really want to have this like full understanding of exactly what are all the different species forming at these surfaces of these electrodes and how does that affect the cycling performance of these batteries?

• 13:22Nathan Labenz?

  Gotcha. Okay, cool. So that gives me a good.

• 13:27Nathan Labenz?

  You know, outsider intuition for why you would be excited to see crystallization happening in one of your AI projects. But let's take maybe one step at a time. So in terms of trying to figure this stuff out from a first principles perspective, my understanding is that what kind of comes between experiment and the AI models is super computationally intensive simulation. So that is to say, we're looking basically very small. These are like sub-microscopic, right? You're talking about lithium atoms. It's a single atom. It's lost its, you know, third electron. We're really down to a very small shell of a thing. So this would be very much in the quantum realm.

• 14:11Nathan Labenz?

  And, and you are then forced to compute numerically, I suppose. This is not the closed form stuff, right? But just heavily computationally grinding out time steps in the, in the wave equation?

• 14:24Speaker 01

  Exactly, yes. So we do a thing which is called ab initio molecular dynamics, which is a bit confusing because the acronym is A I M D. But it doesn't have anything to do with AI. So, you know, if you call first principles molecular dynamics as well as another term, I think. And the basic idea is you want to solve Schrodinger's equation to calculate the forces on all of the atoms in your system, right? So you're starting with quantum mechanics with Schrodinger's equation. And the key reason you use it is to compute both the energies and the forces, which are linked of every atom in your system. And if you can do that, then that's really useful because you can then do what's called propagate the equations emotional, basically solve Newton's equations to calculate how all of the atoms will move through time. Make this kind of movie of how the atoms are moving. So that's what a molecular dynamics simulation is. And I did some work at a national lab in the States,

• 15:09Speaker 01

  working on running these simulations on this huge DOE Department of Energy supercomputers they have over there, which is very fun. But yeah, these things are really, really expensive. So, you know, you're talking huge supercomputers and, you know, you'll run these things for months. And you're looking at really tiny systems. So you might be looking at, yeah, something very simple like a lithium chloride electrolyte solution and maybe a hundred water molecules or so. And you can simulate it for, you know, picoseconds, which is, you know. Fractions, tiny, tiny fractions of a second. So this is yeah, far, actually far smaller than you can see really using any experimental technique. Uh, but that kind of makes it on a, as a plus as well, right? Cause it gives you access to kind of a spatial scale and a time scale that we can't access any other way really.

• 15:50Speaker 01

  Cause it's just moving way too fast and much too small to see using, using Y to X rays or anything like that. No, I mean, you can indirectly get at this thing, these things experimentally, but you can't directly kind of observe them. So that's the advantage of it. It's actually kind of good that it can access these tiny, short time scales, but it does mean that it's inherently limited, right? Cause a lot of phenomena happen at larger time scales, larger spatial scales and longer time scales, right? And so that's really what we'd like to be able to do. Take these techniques, which we know work really well for small systems, right? There are certain experiments you can calculate, um, and show that you agree with experiment, uh, to demonstrate that these techniques do work. The problem is they're just so computationally expensive, uh, that you need these huge supercomputers. And so it's not the kind of thing you can scale, right?

• 16:31Speaker 01

  You're not going to take this and simulate every possible different electrolyte solution or look at any kind of bigger, complex, more complex systems really with. And that's where the AI comes in, you know?

• 16:40Nathan Labenz?

  So these simulations are done on picoseconds? That's 10 to the minus 12 seconds?

• 16:45Speaker 01

  Yes. So technically your time step is a femtosecond, uh, but you run it for thousands, many thousands of femtoseconds to get you to picoseconds. Yeah. Which is 10 to the negative 12, you know?

• 16:54Nathan Labenz?

  So your time step is 10 to the minus 15 seconds?

• 16:58Speaker 01

  Yeah.

• 16:58Nathan Labenz?

  And you're running thousands of those. So you're getting a slice of time.

• 17:01Speaker 01

  Yeah. That's the kind of time scale which, um, intermolecular vibrations happen. So you, and you really need to capture those. So that's really, you're forced to use that kind of time scale.

• 17:10Nathan Labenz?

  And is this, um, when you're solving the wave equation, this is a, if I recall correctly, the second order differential equation. And I believe you would solve that by a polynomial approximation, like we're using like Taylor series type stuff?

• 17:25Speaker 01

  Uh, kind of, yeah. So we're solving the time independent Schrödinger equation, which makes things a little easier. So we're not doing the full Schrödinger equation, which is time dependent, which can be very hard to solve. But there's some approximations we can use, particularly a thing called the Born-Oppenheimer, some tricks we use. Basically, the electrons are way faster than the nuclei. So that's a very useful fact of, you know, uh, the world that allows us to use some tricks to find easy ways to solve Schrödinger's equation approximately. So we don't have to calculate how the full wave function kind of evolves with time. In fact, we're just doing a thing called calculating the ground state energy and the ground state wave function. So it makes the problem a little bit easier. We're not trying to calculate, you know, this complicated quantum mechanical evolution of the wave function. It's just, just one kind of wave function that we're trying to get. A static kind of wave function. So that makes it a little easier.

• 18:10Speaker 01

  And yeah, there's all sorts of beautiful mathematics people have developed to kind of solve these things really fast and really efficiently. So the main way people use it is Gaussian basis sets, actually. So you can, rather than polynomials, you use Gaussians. Um, and you put those on the center of your atoms, basically. And those kind of just, you write your wave function as a linear combination of these, um, functions to describe your wave function. Or you can use what's called a plane wave basis and as well. And or you can mix and match these things, which is actually the code we use called CP2K, which is really efficient and fast at doing these things. So yeah, there's a ton of really beautiful mathematics going into, how do you go about solving these equations really fast enough to do some of these simulations? You just get to this kind of picosecond time scales and beyond.

• 18:51Nathan Labenz?

  And this is an all to all forces on all other atoms in the system or all other molecules in the system are calculated to a hundred water molecules? That gives you

• 19:01Speaker 01

  Yeah, so that's the way

• 19:02Nathan Labenz?

  Almost an attention-like, uh, relationship. It goes with the square of the number of

• 19:07Speaker 01

  Oh, yes. The scaling is very interesting. Yeah. So you have to calculate the force on every single water molecule. So you calculate a total energy and you take the gradient of that energy. As a as the function of the position of every atom to get the force on every atom. So and because the force is really what we care about because that determines how they move. So you get a force of every atom. Um, the scaling is interesting, so it depends on which approximation you're using. So we don't, you know, if you use the brute force method, it actually scales exponentially with system size, which is a really nasty thing, so you want to avoid that, right? But with the quantum chemists, so that this is the discipline that tries to do this, they've found all these beautiful ways of solving this Trojan equation really quite efficiently so that it doesn't scale quite so badly with system size. So you can't quite get it down to quadratic, which would be nice, but you can get it down to kind of

• 19:53Speaker 01

  some kind of polynomial times depending on which approximation you're using. And that allows you to push to reasonably large systems such that you can kind of get a decent approximation of what a liquid is like, right? Which is where we really want to get to, so much of what a chemistry and all of biology happens in kind of this liquid phase. And the nice thing about these, you know, all these approximations we have now is they can get decent accuracy, but they can start to approximate the things that look a little bit like liquid or look enough like liquid that we can really start to make predictions about liquids to phases, which is really exciting.

• 20:21Nathan Labenz?

  Interesting, just because I'm so curious, why is it exponential time instead of quite, I mean, my naive intuition for this would be like each molecule or atom has some force relationship with each other one, and that would seem to be.

• 20:35Speaker 01

  Yes, so that, so there are, in fact, we can get linear scaling, um, ways of computing the force when we talk about neural net protection. But when we're solving Schrödinger's equation, that's because you actually have to treat every single electron in the system, right? So you're not even, you're not treating the atoms, the atoms aren't the base, you know, they actually just are static. You just say, okay, the atoms or the cores of my atoms are just fixed in position. Uh, the electrons is what you're calculating, you're calculating the wave function in fact. And the thing about uh, the wave function is when you add electrons, uh, the kind of space that you're in increases kind of combinatorially. So as you add more electrons, an electron can be coupled with every other electron in the system.

• 21:15Speaker 01

  And so this kind of gives you this exponential growth in terms of the computational cost of the calculation. If you were to try to brute force, this is using a thing called full configuration traction. If you're trying to brute force it out, every single electron can couple with every single other electron in the system. And so you get this very nasty scaling behavior, um, when you try and do that. And you can't use, and so what you're saying is, yeah, you could probably use an approximation where, um, you only worry about things locally or you don't have to try and worry about everything being coupled with everything else, which is why you get this nasty scaling.

• 21:45Nathan Labenz?

  Yeah, interesting. So is this kind of a, when you run these simulations, is it sort of a tick tock type of thing where you first, at each step, you compute the forces and then you propagate one step and things move and then you recompute the forces and you're going back and forth through that cycle?

• 22:02Speaker 01

  Yes. Yeah, so it's kind of a combination of two things. We solve Schrodinger's equation first to get the forces and then you go and use what's called a molecular dynamics engine to calculate how the atoms are going to move. So you update the positions of the atoms using those forces and then you will redo the force calculation in that new positions. Okay, so you can only get this loop going back and forth between the energy calculation and updating the positions and that allows you to make this trajectory. And the reason we do that is because it turns out that the atoms are big enough that they behave fairly classically. You know, so they follow Newton's equations, which is like the limit of Schrodinger's equation when things get big and high temperatures. Turns out you could just use Newton's equation. So we don't want to use the full quantum mechanics for how the atoms behave, although there are rare cases where you do have to do that, in particular for hydrogen atoms

• 22:46Speaker 01

  sometimes. But mostly it's okay to treat the movement of these atoms using classical laws, whereas the electrons are inherently quantum mechanical beasts, right? They're just so small, so light that you have to use quantum mechanics to treat the properties. That's why we have to solve Schrodinger's equation to get the forces, because that's all about what the electrons are doing. But then if you want to know how the atoms are moving, there you use Newton's equations because they're more classical.

• 23:08Nathan Labenz?

  Fascinating. Okay.

• 23:09Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 23:14Nathan Labenz?

  So at what point does the wave collapse happen? If I'm thinking of quantum mechanics as being these sort of distributions over possible positions or position momentum combinations or whatever, and then you do this sort of next step like propagation, how should I think about the certain apparent contrast between a wave or a distribution and a discrete, like this thing is moving this much in this direction?

• 23:41Speaker 01

  Yes, so that's a great question. We never really have to worry about wave function collapse, which is quite nice, because that starts to get very complicated, right? And this is why, and the reason essentially is because of this thing called decoherence, which means that at room temperature you get decoherence, which means everything essentially just, the wave function is always collapsed. And so we really only care about the wave function in its ground state, which means it's kind of a very well-defined state, a stationary state, and it's kind of been collapsed into that state. So we're not too worried about things where you get the wave function in superposition, which gets very complicated. That's when you get back to needing these kind of exponential scaling calculations, which get very tricky. So that's a nice, another nice approximation that we can make because we're kind of at room temperature and we're dealing with relatively large

• 24:25Speaker 01

  systems. Though there are cases where this happens, but certainly for a quantum computer or something, they're trying to exploit these kind of phenomena like wave function collapse and put these, put their systems in these unusual quantum superposition states they're called. And so they have to be really careful to kind of isolate their systems and make sure cosmic rays aren't interfering or something, because as soon as anything touches your system, suddenly that wave function could collapse because it starts interacting with its environment. So that's really, all of these unusual quantum mechanical phenomena only really happen when things are isolated completely by themselves. As soon as they start interacting with something at room temperature, then that's all going to immediately kind of get washed out and things are just going to collapse down to their normal states. And so that's really what we're just trying to do, is just find the wave function in

• 25:08Speaker 01

  this kind of ground state, which is kind of almost the easiest problem you could have to do, which is nice in quantum mechanics, just finding technically it's the lowest eigenvalue of the Hamiltonian operator. And it makes things a little easier.

• 25:20Nathan Labenz?

  Interesting. Okay. I'm not sure I fully got that, and I suspect that probably a lot of people don't feel like they have full intuition for it either.

• 25:27Speaker 01

  I don't think I have either, so.

• 25:30Nathan Labenz?

  Yeah. Well, nevertheless, though, I think that's at a minimum, a great motivation for, you know, why it would be nice if we could have an AI that could figure all this stuff out for us. So tell me how you're now training AIs. And I understand, first of all, you're training the AIs based on the simulation data. So we've set up to the point where there's a quantum mechanical approximation solve based on where everything is and the forces that that exerts, and then there's a timestamp. So how does that simulation data turn into training data? And then what exactly is the model learning in terms of what are its inputs and its outputs?

• 26:06Speaker 01

  Yes, great. So basically, the by far the most expensive part of these simulations is the solving Schrodinger's equation. That's very, very expensive. And that's where your computer spends almost all its time. And so that's the piece that if you could speed that up, it would be really potentially transformatively useful. And this is exactly why people have developing this tool called neural network potentials. Okay. And so the idea is that this is basically it's a neural network. So it's just kind of a very, very flexible function with a huge number of parameters. And then you're just going to adjust those parameters to predict what that quantum chemistry calculation is doing. That's solving Schrodinger's equation, right? And essentially, so the input there is the coordinates, the positions of all of your atoms. So you've got an oxygen atom here and a hydrogen here and a sodium here.

• 26:48Speaker 01

  And each of those has a force on it and it and there's a total energy as well. And that's what you're trying to predict. So you're trying to go from the positions of the atoms to the forces on all of the atoms. And that is the expensive quantum calculation that we're trying to replace with a neural network. And then, so you do need to do at least some of these quantum mechanical calculations, right? To generate your training data and then sort of supervised learning on that training data that you've generated from these quantum chemistry calculations. And these neural network potentials then can reproduce this mapping fairly well. As long as you have in your training data, something that's sufficiently similar to the case you're interested in. You then don't have to go back and do the full quantum chemistry calculation again. You can just use the neural network to calculate those energies and forces much more quickly.

• 27:31Speaker 01

  So this allows you this loop between, you know, the quantum chemistry and the classical. You can then massively speed that up because you're not spending so long waiting for the quantum soul to work.

• 27:41Nathan Labenz?

  So you're basically doing the same tick tock kind of loop, except instead of the numerical solution for the wave equation, now you've trained a neural network to make a guess. And it turns out that guess is quite good.

• 27:56Speaker 01

  Pretty good normally. Yeah, exactly. So we keep the. Keep the part where we're solving Newton's equations because that's actually pretty quick. It turns out that updating the positions of all of the atoms based on the forces on them is actually doesn't take too long at all. So the but by far the biggest cost is this quantum part. So if you replace that, then that gives you access to, you know, a whole new range of systems you can look at. So, you know, normally that's the quantum, you can speed that by a factor of a thousand or more. And so suddenly you can look at systems that are a thousand times bigger and a thousand times longer. And that opens up such a range of systems that that alone is a really useful tool to look at a huge number of important things that we haven't been able to look at previously. And this idea has been around for a long time, actually. It goes back to the 90s, people came up with it.

• 28:38Speaker 01

  There's a beautiful paper in 2007, actually, Yehud Baal came up with this with Paranello, this idea of using these neural potentials and they train them. The initial ones were just using a very kind of vanilla, you know, neural networks, you know, multilab septron, just a standard kind of dense feed for neural network.

• 28:53Speaker 01

  Since then, we've started to develop these things a bit further and get a bit more sophisticated in terms of the ways we do this thing. So these tools are really getting a lot more useful now, just exciting as well.

• 29:01Nathan Labenz?

  So I want to, you know, a lot of questions about the the nature of the networks. Maybe for starters, like, how big are they? Like, how much data is going into training? People, you know, I think are at this point most familiar with large language models. We're thinking in terms of like trillions of tokens and 10 to the 24, 25, maybe 26 flops. And many billions of parameters, if not even into the trillions with a mixture of experts. How does the just size of like data sets, compute resources and models compare for for these purposes?

• 29:36Speaker 01

  Yeah, I mean, this is what I'm excited about is really they're very moderate, I think, compared to these things people talk about. Yeah, well, these large language models are kind of hard to fathom, the size of these GPU clusters and things to me, right? So we're talking to generate my training data. So for me, I'm only interested in like one electrolyte solution. But to generate that training data, you need maybe a few GPUs for a day or so, you know, so nothing significant at all. So I don't have access to huge, you know, resources at all, really, just using kind of the standard allocation on our supercomputer here in Australia. So, you know, I don't have access to that. You don't really need these huge clusters or anything, I think. And the training data generation is still probably the most expensive part. And yeah, that's fairly, you can do that in a few days. Then running the simulations is not too bad, but you can do that on, you know, at least for my systems.

• 30:19Speaker 01

  Simple electrolyte solutions, you know, just a few GPUs is really all you need. And you can run these simulations for really long enough to converge everything very well. So the cost of these are really very reasonable. Having said that, people are trying to do much more ambitious things, obviously, now with these neural net potentials. So there was a nice paper from the people who developed the neural net potential that I use. They showed they could simulate an entire HIV capsid, right? So a huge system there, you know, millions of atoms. So that kind of thing, then you do want, you should use one of these huge GPU clusters, which they use to simulate that kind of thing. So you can really push these things to huge system sizes when they start to get really expensive and start to get comparable to some of these large language models and things. But in my opinion, there's a huge number of systems that we don't understand fully yet. They're actually quite small and if you just simulate those, you know, there's a ton of interesting insights you could get.

• 31:04Speaker 01

  So I don't think you need these huge resources to study interesting scientific problems, although there are certainly ways you could use these really huge resources if they were made available to us.

• 31:14Nathan Labenz?

  Cool. So what you're doing, the recent results have come from a couple of days of GPU time. And I guess intuitively, that sounds like hundreds of millions of parameters in a model?

• 31:26Speaker 01

  Oh, yes. So the parameter size, actually about, I think, tens of thousands of parameters is what we use. Yeah, so not too big at all. So that, so it's very small compared to, it's funny because it's very small compared to, yeah, the large language models, obviously. But that's kind of, you know, model the entirety of the internet, right? So whereas ours, in fact, is quite a lot simpler, I think. The distributions essentially that we're trying to model, you know, than all knowledge. So we don't need that, that kind of number of parameters. People are trying to build these things called universal force fields, which maybe gets later, but those will, you know, have much higher parameter counts and getting up closer to that kind of thing that you need for a large language model. But it's actually interesting because there's a, there's a classical method of doing these simulations, obviously, which we haven't talked about yet.

• 32:07Speaker 01

  But the basic idea is you kind of guess at some physical equations about what this force field should look like. And there you have this parameter fitting problem as well. But they actually only use, you know, a few dozen parameters. So in some sense, we've got way more parameters than the kind of classical approach to doing this. There's still a lot less than these very large AI models. So it's an interesting, we're kind of in this in-between zone. But there's a lot of work, people are trying to see, can you get the parameter count down even lower? Which are very interesting. So we'll see where the best models end up at. It's going to be a very interesting question.

• 32:39Nathan Labenz?

  Yeah, I'm really surprised that it's only in the tens of thousands of parameters. That's like strikingly small. That also suggests, I guess, if you're training something that small for days on GPUs, that would sound like overtraining on a standard thing. But is there a, it sounds like there may be some sort of Grokking type phenomenon going on here, right?

• 33:01Speaker 01

  Yeah, interesting. So the training is actually not days. The training is only done within a day, for my systems at least. It's mainly the days would be generating the quantum chemical data because that's the really expensive part.

• 33:11Nathan Labenz?

  And that works on GPUs too, so that's also parallelizable?

• 33:15Speaker 01

  Yeah, technically I did it on CPUs, but you can do it on GPUs now. There's now codes that do it on GPUs even faster. So that's really where things are going. I think we'll be running all these quantum chemistry calculations on GPUs now. And yeah, that's the kind of thing you need a few days probably to generate your training data. But the actual training yet is, I don't know the exact timing, but it's probably a few hours really. Is there a Grokking phenomenon? I don't believe so. I haven't seen any evidence of that yet, but that would be very interesting, yeah, if we saw something like that.

• 33:43Nathan Labenz?

  So how about the architectures? Are we talking like transformers here? Or is other, you know, familiar architectures? Or are these exotic architectures you're using?

• 33:54Speaker 01

  Yeah, no, so the one I use is called an equip developed by the Kaczynski Lab out of Harvard. And that is a graph neural network, an equi-baryne graph neural network. So it's got message passing. So, you know, I'm sure people are familiar with those kind of things. There are also some, there's one called PortMD, I believe, which is a transformer, which has quite good performance as well. So you can use a transformer there. And yeah, so the traditional ones, which some people still use, are also just, you know, a plain multi-layered septon, as I said. So yeah, these are all kind of standard architectures that would be familiar to people working in AI, I think.

• 34:27Nathan Labenz?

  And then there's some interesting stuff around like. How the data is represented, right? There's this, when you, the equivariance concept is critical to this whole thing, right?

• 34:40Speaker 01

  Yeah, so equivariance is this idea, and this was kind of the big advance that we just had in the last couple of years, I think. So, as I said, this idea has been around for a long time, but the issue was always you needed so much training data that I always really struggled to get it to work efficiently and effectively. Basically, you need to keep going back and generating training data constantly to kind of look at the things I was interested in, in particular, kind of these pairing of these ions, how strongly they stick to each other in water. And so that was always a real challenger of frustration I had. But then we saw, you know, this paper that came out in a clip, is the architecture that we use. And the incredible thing in that paper, they got this table and they say, you know, we get lower error for a water box. You know, they're trying to calculate the forces on a system of water.

• 35:21Speaker 01

  We get, I think it was four times lower error, and they got that using a thousand times less data, right? So three orders of magnitude less data. So that kind of blew me away. So as soon as I saw that, I kind of jumped on this, because it's not often in science. You know, you'll see a breakthrough with something goes down by three orders of magnitude. That's the kind of thing that's inevitably going to have an impact, I think. So that's really why I jumped on that quickly and should try to exploit these new tools, which have much more training data requirements. Because then, as I said, the biggest cost is this quantum chemistry calculation you have to do to generate your training data. So if you can get the amount of that data that you need down, that's really exciting. Because then you can start to really scale these things essentially and look at many different systems. So that's what I'm really excited about now, is this tool of equivariance that allows you to kind of have this training data.

• 36:05Nathan Labenz?

  And can you explain a little bit more like what that is? Is it a sort of, like a module of certain symmetries? You're sort of finding a way to represent things that is in some way more general, less contingent on like the specific coordinates, but somehow relative to each other?

• 36:22Speaker 01

  Exactly, yeah. So it's a really, really nice idea. It comes out of group theory and representation theories and beautiful mathematics that actually kind of revolutionized particle physics, you know, a few decades ago. So as a physicist, it's kind of really appealing to me that these same tools now are being transferred and being useful in chemistry and useful in molecular simulation, which is really exciting, and AI. And it's, you know, obviously highly mathematical. I'm not an equivalence expert, so, you know, I could give you kind of a high level of understanding. And the basic idea is that you You don't want to just feed raw coordinates into these neural networks, right? Because the raw coordinates are fairly arbitrary. In fact, if you rotate the molecule, the coordinates, the positions of all the atoms are going to change, but its properties really shouldn't change, right? Because if you rotate a water molecule, it's still a water molecule.

• 37:07Speaker 01

  It's still going to have all the same properties of a water molecule. They're just going to be rotated as well. And so what you really want to build is a neural network that has this rotational symmetries of space embedded into it. So it knows about the fact that you're on three dimensional space. And if you rotate something, it's still the same thing. Okay, the way you do this mathematically is you, one way of doing it is you build tensors into the neural network. So it's not just operating on matrices or big lists of numbers, but it's actually operating with these mathematical into these tensors. And the basic idea of a tensor is just has these nice rotational properties, right? It can keep track of directions. So it has some idea which direction it's pointing in, is the basic idea. And it turns out when you do this, yes, it turns out the training data goes right down. And this was really appealing to me because it has actually got deep parallels

• 37:50Speaker 01

  to the way we model intermolecular interactions, which is what's important for building these simulations. This is the ways these things have been modeled for decades. There's a thing called the multipole expansion, which basically you describe your interactions in terms of these tensors, dipoles, quadrupoles, octopoles. You've heard these terms, but these are actually mathematical tensors. And that's the way we've been doing it for decades. And so it turns out that when you build that fact into the neural network, your train data requirements go way down, obviously because it can exploit the fact that this is the natural mathematical way to describe interactions of molecules. And so that was a really nice example of how you could use fundamental physics ideas and combine it with AI to get you these really advances in performance. Now, but there's a really interesting debate going on at the moment about this, though, actually.

• 38:34Speaker 01

  Because AlphaFold 2 was one of the biggest breakthroughs, obviously, in AI for science. And it used equivalence in the way its architecture worked. But the latest version of AlphaFold, AlphaFold 3, actually got rid of the equivalence. And yet they still managed to get comparable performance to AlphaFold 2. So there's people debating, is this actually critical or is it not? And the trick they used was a thing called data augmentation. So they actually did feed raw coordinates into the model. And some other people have shown this works for neural network potentials too. But they took the data set and then they rotated it in like, you know, a thousand different ways and then fed that big, much bigger data set with all of these rotations into the neural network. And it turns out that when you do that, you can at least get error metrics. So, you know, your loss and things can get as low as the equivariant

• 39:19Speaker 01

  models with a similar sized training data. So it'll be very interesting to see if these neural networks are actually as useful for practical tasks like running M D simulations, which I haven't really seen yet. So that's going to be a very interesting question moving forward. Are there ways to kind of get around building this equivalence into the neural network potentials? Because the disadvantage of it is that it can slow your neural network down quite a lot because it's just a little bit more inherently complicated doing these kind of tenses, having these tenses interact with each other rather than just, you know, normal matrices or lists of numbers. So that's a really interesting thing to keep an eye on how that field advances. And as an example, you know, there's a paper from Apple recently where they said swallowing the bitter pill, right? It's an example perhaps of this bitter lesson, right? That really, there's no way to be smart about how you build your neural

• 40:00Speaker 01

  network. And the answer is just to make it big with lots of parameters and then get as much data as you can, basically, and let the neural network learn everything and don't try and constrain it in an interesting way. So that's an interesting example. I'm kind of rooting for equivalence to hold out because it's, you know, you think if there's anything that's worth building into your network would be like fundamental physical laws and like the symmetries of nature. So we'll see how that turns out. It's very interesting.

• 40:25Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 40:29Nathan Labenz?

  So let me just make sure I understand a couple things. It seems like the big idea is from a principles level, the coordinate system shouldn't matter, right? That's a very common thing in physics. Like you can choose a convenient coordinate system, but in general, the coordinate system shouldn't matter. So you can either find a elegant way to represent your data in this sort of relative to one another form, that is the equivalence form, and that essentially sort of abstracts away from any particular choice of coordinate or any particular choice of origin point in the representation of the data.

• 41:12Nathan Labenz?

  But then your neural network architecture gets more complicated because it has to handle this relative representation. Or you can do what AlphaFold 3 did, and that is just systematically apply a bunch of rotations and translations to the data and then feed that all into a less complicated neural network, but more data points, sort of synthetically generated. All of this is synthetically generated, but there's like the expensive waveform solution generations, but then also the sort of very cheap presumably like translation and rotation generations based on those. Feed that into a comparatively simple neural network, more data points, and it seems like

• 41:54Nathan Labenz?

  we're in a world where either of those can work, and you might even think it could come out in the wash, or sort of, you know, in the limit, they might converge on kind of being the same thing.

• 42:05Speaker 01

  Yeah, exactly. I think that's probably where we're headed, is that you can just, it's kind of, you can impose these, essentially you need to tell the neural network that it's in three-dimensional space and it has these rotational properties, right? But you can do that in different ways. One is you can do, put it on the data side and augment the data, or you can kind of encode it, hard code it into the neural network itself, and tell the neural network that actually, you know, restrict the things the neural network can do so that it makes sure that it keeps track of the orientations of all of these things in space. Yeah, so there's kind of the two alternative ways of doing that, and it'll be interesting to see which one wins out. I suspect perhaps for these simulations, running these molecular dynamics simulations, it may be worth having, keeping that equivariance in the neural network itself, just because

• 42:49Speaker 01

  it may give you better generalizability once you get away from the data set, or once you're trying to do things like, look at this crystallization phenomenon we saw. So I'd be really surprised, I guess, if these neural networks that impose it on the data side could still see this kind of generalizability properties. So that's going to be something interesting to watch. But, yeah, that's essentially it.

• 43:07Nathan Labenz?

  Yeah, okay, cool. One ML paper that I would recommend if anybody's interested in

• 43:14Nathan Labenz?

  Developing their intuition for this sort of equivariance concept, but also just a really interesting finding about neural networks is called relative representations enable zero-shot latent space communication. This is out of a group in Rome and also some folks from Amazon. And this is a paper I kind of always go back to because they basically showed that the latent spaces learned from the same data set across different training runs with different initializations seem to be converging to the same thing. Modulo, some sort of like rotation and maybe like a scaling factor. And with that, if they can like figure out, you know,

• 43:57Nathan Labenz?

  what is the sort of rotation or scaling factor that's applied? Then it creates potential for these model spaces to kind of talk to each other or to potentially like distribute training. There could be implications for that. Model merging, like who knows, right? There's a lot of kind of interesting. If there's this like fundamental convergence, there's a lot of interesting downstream consequences of that. So that paper is relative representations enable zero-shot latent space communication. And aside. Nice. So, okay, going back to this crystallization phenomenon. So now we're all queued up to hopefully appreciate, not on your level, but you know, hopefully much more than we were at the beginning, why this is special. So what you saw was you trained a, you collected a bunch of this data in the computationally

• 44:42Nathan Labenz?

  expensive way of solving the wave equation. You train a neural network on it with positions in, forces out. Now you're able to run this simulation propagating one femtoreg, you know, a couple of femtoseconds at a time in this TikTok sort of way where it's here's the position, calculate the forces, apply the forces, update the position, loop through that, but you're doing this now with the network doing the force predictions. And lo and behold, you see a salt crystal forming in your simulated solution. You had said that it's, there was no crystal in the training data? How do you know that for one really stupid starting question?

• 45:26Speaker 01

  Yeah, so yeah, there's one other piece of the puzzle which we should explain first perhaps, which is that that wasn't just a pure all-atom simulation where we saw the crystallization. So we actually had one other step, which is we used coarse graining to get rid of the solvent molecules, which maybe we can get to that. So that actual simulation was just the ions itself. And we actually used a neural network potential to do that as well. So this is a really exciting, even more recent use of these neural network potentials. Is not only can you go from predicting from the positions of every atom to predicting the forces on every atom, but you can also do this trick called coarse graining where you kind of ignore certain parts of the system. So we actually took a simulation, an all-atom simulation,

• 46:10Speaker 01

  and then used this neural network potential to predict the energies and the forces just on the ions. So that simulation actually didn't have any explicit water in it. It was what we call an implicit water simulation. The AI had learned to kind of implicitly, approximately account for what the water molecules are doing without having to exactly calculate them. So this is kind of exactly analogous to what you do with the normal neural network potential, where you're getting rid of all of the electrons. So with a neural network potential, you can just calculate the energies and forces from the positions of the atoms themselves without ever kind of thinking about the electrons. And then you can do exactly that same procedure again, but getting rid of the solvent molecules. So now you don't have to predict, you don't have to incorporate the position of the solvent molecules

• 46:52Speaker 01

  into the neural network. You just predict the energies and forces from the ions themselves. So this is a very, another exciting way you can use these neural network potentials to kind of push to even larger scales. So you're not having to stuck with this simulations where you're treating every single water molecule in the simulation, because that kind of has inherent data limitations. You know, you've got to deal with, you know, when you want to push to large systems, you've got to keep track of every single water molecule, and you end up just spending a lot of computational time wasted studying how some water molecule out in the bulk is buzzing around and rotating, which you don't actually care about at all. So this is the other exciting thing we did in this page. So people had done neural network potential MD of electrolytes before, and if you do the all-atom way, you won't, probably won't see crystallization, because this crystallization phenomenon normally happens on very, very long time scales.

• 47:37Speaker 01

  And so that's how we're feeling confident that it isn't in this in the simulation, right? Because it's very rare to see these kind of things. It only happens typically if there's something really wrong with your forces. If you're studying a normal salt like sodium chloride that doesn't normally precipitate and crystallize out of solution. But then you can also go back and explicitly check these things. So we can compute a thing called the radial distribution function. It's basically just a histogram of the distances. And you could look at that and it looks totally liquid like, you know, it has all the features of the liquid at low concentration and so on. So there's no evidence in that simulation of any crystals forming. And then you can just, you know, inspect the trajectories more directly as well and just check and make sure that there's no signs of these crystals forming. So we did go back and check for that in the all atom and the full simulation that we trained this neural network potential on.

• 48:19Speaker 01

  And so, but the exciting thing was, yeah, we saw this crystallization using this additional way that you can use neural network potentials. We also coarse grain out the solvent. And that was exciting because it means that you can do something that you can't do with these all atom simulations really, which is that you can look at, very quickly look at things like crystallization with much less compute time, basically. Because if you wanted to look at these kind of crystallization things with the all atom case, it would take a huge amount of computer time. You have to run it for very, very long times and you have to use probably a technique called enhanced sampling, where you kind of force this crystallization to happen. Where here it kind of dropped for free out of the simulation. And the remarkable thing about that was, in particular, was when you do this coarse graining or getting rid of the solvent, is that the solvent is actually really, really

• 49:04Speaker 01

  important in determining the interaction of ions. Right, it actually makes the interactions about a hundred times smaller. Okay, it's the thing called dielectric screening. The water molecules are incredibly important in determining the interaction of the ions in water. So it's quite remarkable, but that doesn't occur in the crystal, right? Because there's no water there. And so, yes, somehow this neural network potential now that's trained to get rid of the water molecules can in fact kind of predict that as you go from this water to this crystal, what the structures are going to look like, even though the strength of these interactions could be changing by almost a hundred times as you go from these two different phases of material. So that, that was what made that even more remarkable is that it's not, it, it, it would have been, you know, nice if you just looked at the longer time scales with the neural network potential and saw crystallization, but we could also use this trick where you

• 49:48Speaker 01

  got rid of the solvent molecules and yet it still works to see these kind of phase transformations. Whereas I would have thought that would never be possible, you know, I've been. And this is actually what I was originally trying to build in my PhD with these continuum solvent models, right, which is these models which ignore the water and just treat the ions, which can be much, much faster, which is really their advantage. But the problem is the water is so critically important that it's not clear at all how to do that. So I spent my whole PhD, literally years, trying to build these things, right? And I didn't really get very far, trying to build them by hand and come up with mathematical equations and fit these parameters and do all these things. Anyway, I built some models that were okay, but they're really inherently limited in their generalizability. But here, the model, you know, this was trained, you know, in just a few hours, I think, automatically kind of realized and learned how to build one of these continuum solver models itself,

• 50:33Speaker 01

  and they could run simulations and they could see these emergent phenomena. So that's really why it blew me away and was so surprising. This is the kind of thing that, you know, I've been trying to do for years, I had kind of given up and thought was impossible, and yet it just dropped out for free from training one of these neural networks, which is kind of what's so exciting about it.

• 50:47Nathan Labenz?

  Okay. I want to dig in on this on a number of different fronts. That is definitely really interesting, and I want to make sure I understand it all. I guess for starters, in terms of how you knew that there wasn't crystallization in the dataset, it's a combination of you're running it for a very short period of time, a shorter timescale than crystallization usually takes, and you also went back and screened the data and said, can we spot any apparent crystals by just looking at, are there things that appear to be bunched together? And if you don't see that, then you can say, okay, there's no crystallization in this dataset.

• 51:29Speaker 01

  Yeah, exactly. So what you do see in the simulations is you see the sodium and chloride ions kind of periodically pairing up, and then they'll come apart, and then occasionally you might get, you know, a triplet forming. But you don't really see any of these bigger clusters that look like the crystal, really. So, yeah, that really indicates that it's not crystallizing the data. It's a bit hard to determine exactly what the biggest cluster that forms is, or what the most similar thing in that liquid simulation is to the crystals. That's something to potentially dig into further. You know, how close is the training data? What is the closest example in that training data to a true crystal? But definitely, there's nothing like this crystal growth phenomena, where it starts growing larger and larger and larger, and kind of phase transforming the whole system.

• 52:09Nathan Labenz?

  Gotcha. Okay. So then, in terms of the coarse graining concept, again, just for intuition's sake...

• 52:18Nathan Labenz?

  When we talk about solutions, I think of these measured in like molarity, right? So a one molar, you can tell me how many molar your solutions are, but a one molar solution of of salt water would be one mole of salt in a liter of water. And so a liter of water is a kilogram and a mole of salt would be whatever, like 30 grams or something. So we're one per in 30, something along those lines.

• 52:51Speaker 01

  Yeah, that's a rough order of magnitude. Yeah, so I think technically we were at four moles. So the molarity of pure water, I believe, is around 55 moles per liter. So if you're at four moles, you have 10 water molecules for every salt ion is probably reasonable.

• 53:06Nathan Labenz?

  Gotcha. Okay. So now we got one ion per 10 water molecules. The idea with the removing of the water portion of the all-atom simulation, so you're running in the simulation, you're you are computing all the water molecules, all the ions. But then is the intuition that the main forces come from the ions because they're actually imbalanced in charge, whereas the water is neutral and it has its polarity, but it's not the because of the fundamental neutrality versus the imbalance in charge? Is that where my intuition should come from, that I might be able to abstract away from the water in the first place?

• 53:45Speaker 01

  Yeah, so the forces from the water molecules are actually very large, though. So they actually play a really important role in determining the the force on the ions. So the technical thing we're doing is it's called integrating out or, you know, probability speak, it's marginalization. So the the water molecules are still implicitly accounted for. It's not like you're ignoring them, although I may have said that technically, they're more kind of their average effect is still accounted for in the simulation through a kind of average effect. And that's the real challenge of it, is that these water molecules do play a big role in determining the interactions of the ions. But it's just you can counter that effect in an average way is the basic idea. So that rather than treating, so they've got these two ions interacting and rather than saying, okay, I need to know how every single water molecule around them is arranged

• 54:30Speaker 01

  to calculate the force on them. I can say, on average, what would the force between these two ions be if I average over all the potential configurations of the water molecules around them? And that's the really challenging thing. So that's why it's so incredibly hard to build these continuum solver models, because the water molecule is actually critically important. So you're right, the water molecules are neutral and the ions are charged. So you get these large charge charging directions. But the water molecules are absolutely critical there because they actually reduce the size of that charge charging direction by, you know, a factor of eighty. So it's absolutely critical. The water is absolutely central there. It's really, really hard to get rid of them. That's why it was so hard in my PhD, trying to find ways of ignoring them, right? It turns out it's just really hard to do. But that's the beauty of this, these neural network potentials, so that they can somehow

• 55:11Speaker 01

  automatically average over all of the effects of all of these water molecules, and yet still get you an accurate prediction of what the forces are on the ions are going to be on average, right? When you average over the behavior of all of the water molecules. And so that's what, yeah, it's really.

• 55:26Nathan Labenz?

  So does that suggest, if I'm envisioning what's going on in solution, the ability to average in this way, which I guess procedurally is like, you're taking this all-atom simulated output and literally just removing the water from it, and now you're training the network on purely the positions of the ions?

• 55:47Speaker 01

  Yeah, that's what you do. It's pretty simple.

• 55:49Nathan Labenz?

  So each ion, then I should guess, I guess I should understand this, like, being surrounded by a lot of water molecules most of the time? It's not like it's randomly interacting with one molecule at a time, because then it would seem really hard to abstract away. It's like in a shell of water molecules.

• 56:05Speaker 01

  Yeah, exactly.

• 56:05Nathan Labenz?

  As kind of its default state.

• 56:07Speaker 01

  Yeah, they're surrounded by many water molecules. And so you have to give the neural network a lot of examples of the positions and forces on the ions alone. And you need to do that so that it can average over the water molecules, right? So if you only gave it a few examples of ions without the water molecules, it wouldn't, it wouldn't be able to do it, because it needs to know all of the different ways that these water molecules could be arranged. Can we average over that? So you actually, to train the coarse grain model, you actually need a lot more data or a lot longer simulations. The data is obviously smaller because you only need the ions, but you need a lot longer simulations to generate the training data for training these coarse grain models because they have to do this averaging over the, all of the water molecules.

• 56:49Speaker 01

  So in a given position of ions, you don't have a unique set of forces because it's going to depend on the waters, but you do have a unique average set of forces. And that's what you're getting the neural network to calculate, is average forces on all of the ions in these positions. Whereas the normal way we use the neural network potential, which we talked about before, there is actually a unique force on every single atom in the system, which comes from solving Schrödinger's equation. So you're kind of using the neural network in two slightly different ways. One is to compute the exact energies and forces on every single atom. And then the second way you can use it is to calculate the average energies and forces on these things. And technically, what we're talking about here is actually free energies. So we're using the neural network to calculate free energies rather than pure energies.

• 57:32Speaker 01

  There's some underlying statistical mechanical theory that kind of makes all of this make sense and makes this rigorous, which is quite nice.

• 57:38Nathan Labenz?

  Okay. Very interesting. So now, through the crystallization. How should we understand this? I mean, the, it is a, for one thing, it's a phase change, right? You talked about like orders of magnitude, different amount of force being exerted by the ions on one another when they're in crystal form versus when they're in solution form. I think you said it was like 80 times or something?

• 58:04Speaker 01

  Yeah, exactly. So you've got this in water, the ions obviously come apart and they're dissolved and they're floating around freely and the water in between them actually reduces their interactions by this factor of 80. It's called dilatory screening. Whereas in the crystal, that's not going to happen at all. There, you know, you have much stronger forces kind of sticking these ions together. So that's why I thought that it would never be possible to to look at crystallization and liquid phase with the same kind of continuum solvent model. So that's why it was kind of blew me away when we saw it happen. Because obviously, you could see these crystallization phenomena in the all atom case. If you kind of drive it to happen, because there, you know, you don't have this problem with the water molecules kind of being very relevant in the solution and not relevant to the crystal. So that's what was so surprising about it, the fact that it was able to do that.

• 58:48Speaker 01

  We still don't have our head around exactly why it can do these things. Is that something we want to dig into kind of further? What I suspect is that because in the solution phase, because you get those kind of pairs forming and then these triplets forming and then possibly larger clusters forming transiently, that the neural network can look at those and kind of extract like, say, oh, I see what's happening. When when these pairs are forming, that water's squeezed out and therefore the forces are getting larger between the ions in those cases. And so it's kind of then learning that information implicitly and then it can take that and extract like to what it's going to look like in the crystal. So that's what's kind of interesting now. But an important caveat is that it cannot do this kind of quantitatively precisely. You know, so I don't think, in effect, we know that it's not going to get the quantitatively

• 59:29Speaker 01

  exactly things like the solubility, you know, how much salt you need to make the crystal precisely right. That's not the kind of thing it'll get right because it doesn't have any training data on the crystal itself. You know, there's going to be some more quantitative errors there. So that's that's an issue, but that can still be fixed, I think, fairly easily. Once you've seen the crystallization, then the obvious next step to do is to take those crystal structures and then you could add that to your training data set and do a thing called active learning and try and get those things quantitatively. So that's kind of where we want to go from here, I think, to start trying, add data on the crystal and see how precise we can get these things in terms of actually making precise predictions of experiment, which would even be even nicer. But yeah, this stage is still a bit of a surprise that it could do this at all. So still just trying to work out exactly what's going on, where it's getting the information

• 1:00:14Speaker 01

  it needs to kind of predict what the crystal is going to look like.

• 1:00:17Nathan Labenz?

  I wonder if we should think about this as the network generalizing out of distribution, or if this is sort of a property of like, the outer loop? Because I mean, in like, language models, you're probably familiar with this. It's pretty well established at this point, and certainly listeners have heard me talk about it many times, that there are these higher order concepts that are being learned, you know, purely as a means to next token prediction, but nevertheless, like, we can now go in and identify these concepts, sparse autoencoder techniques, and representation engineering, and even demonstrate that you can steer a language model by injecting one

• 1:00:59Nathan Labenz?

  of those directions, or activations, or features is a better term than activation. You can inject that at runtime, and you can steer the output of the model based on that. Sounds like that's not quite what's happening here. I don't think from your account, that I should expect that there is a feature that's been learned in the network that is like, the crystallization feature, or concept, but rather, it would seem to be more of just like, as things sort of approach a certain position, the forces are such that they walk into a grid and stay there for at least some amount of time. Does that seem right, or could we even resolve that with what we know?

• 1:01:44Speaker 01

  Yeah, I mean, this is a fascinating question, and I definitely don't have the answer. So, the way to think about it, though, I think, is to think in terms of, you know, matching surfaces, essentially. So when we're trying to compute the energies from the positions, you can think of that as trying to match what we call the potential energy surface, which is the basic idea that as a function of the position, which is just a vector, right? The positions of all of your atoms, that has some energy value, and this is a high-dimensional surface, you can imagine, it's like a single value that you map to from the positions of all of your atoms. And it's exactly the same thing with these coarse grain simulations, where it's now a free energy surface that you're trying to match. And so, in the simplest possible case, so imagine you just had, you know, two positions,

• 1:02:28Speaker 01

  so you have a single atom, and it's in two dimensions, that actually, you can visualize that exactly as a surface, right? Every two points in that two-dimensional plane, there's some energy, which you can imagine is kind of the height of that surface. And then you can generalize this to higher dimensions. And this is what the neural network potential is actually learning, it's learning that surface, right? And it turns out that the neural network potential, you can train it on one region of the surface. And somehow it can extrapolate to what it would look like in other areas of that surface, okay? So then you're only training it where it's in the liquid phase and it's kind of matching the surfaces as best it can. And it's got these kind of equivalence and these various constraints which it's trying to use to match that. And it turns out that somehow that they can also do a good job, at least approximately, of predicting what that surface is going to look like when it's in the crystal structure.

• 1:03:13Speaker 01

  So you can imagine the crystal, there's some like a potential energy well, we call it. There's some like sharp dip in that surface, which means it has a lower energy, which means it's likely to be in that state. And so that's the way to think about what it's learning. It's actually learning that high dimensional energy surface. And then that's what it's using to then do these dynamics. Cause you use M D to kind of move the atoms around on the surface. And the atoms like to be in lower energy states, right? That's what statistical mechanics tells us. And so that's kind of what is, if you want to think about the concept of learning, although this is a very vague idea, what is even is the concept, you know? But that's kind of what intuitively I would think about what it's learning. And now there's interesting parallel with this to large language models, right? Which is that people often talk about large language models as fitting a surface as well, right?

• 1:03:57Speaker 01

  They're just fitting a different kind of surface. In fact, they're fitting, you know, the probability surface of next words, right? So this is what these G P T, these big transformers are doing, they're predicting the next word. And that's just a probability, right? That's every vector that represents your sentence, every word has another probability. So that's a probability surface that you're fitting. And in fact, this actually has deep connections to statistical mechanics. Cause the log of your probability density in statistical mechanics is actually an energy, right? To do a thing called the Boltzmann distribution. So there are actually deep connections between what's going on with these large language models and what's going on with the training of these neural network potentials.

• 1:04:36Speaker 01

  Now, when it comes to, yeah, this idea of these concepts or this, you know, the fact that these large language models seem to have this ability to kind of have some notion of concepts and generalize out of exactly what they've been trained on, which is kind of this surprising phenomenon. I mean, I would speculate that it may be something similar going on. I mean, that in sense that when you learn this kind of probability surface very accurately, that then allows you to extrapolate to other regions of that probability surface and to make predictions there. And so probably, you know, I mean, we don't really fully know what's going on in these things, but there may be some parallels there. And you actually see some evidence for this in the way these large language models work. I mean, they use a softmax layer in them, which means they're, which is the softmax is the normalization layer. That is actually a Boltzmann distribution they're using. It's actually an idea that

• 1:05:21Speaker 01

  comes from statistical mechanics that comes from these M D simulations. And so there are, you know, they always talk about log probabilities, you know, you hear that very often. Well, a log probability in statistical mechanics is a free energy. So there's actually some deep mathematical connections going on here. And I don't have my head fully around it, but I really think this is a direction that's worth pursuing, is thinking about these connections between statistical mechanics and large language models, because I think there may be some kind of deeper connections going on there.

• 1:05:47Nathan Labenz?

  Yeah, okay. Definitely very interesting. So more about the generalization potential. You had another phenomenon that you reported as well, which was self ionization of water. I guess we're now back to the all atom case there. But tell us about that.

• 1:06:08Speaker 01

  Yeah. So that was another kind of thing that really kind of blew me away and surprised me. And now the people have actually seen this as well. And I think no one really knew quite what to make of it, but it's very interesting. So when you run these water simulations, so now you're including all the water molecules. And in this case, yeah, we're just trained on, you know, pure water or maybe some ions in water. When you run these simulations for long enough, you actually see this phenomenon called water self ionization. And basically, what that means is the hydrogen hops off a water molecule onto its neighboring water molecule, and it forms an OH and an H three O. So instead of two H two O's, you have these two separate, what they're now ions. And this is a phenomenon that we know happens in real water. In fact, about one atom degree, I think it's 10 million atoms or 10 million

• 1:06:52Speaker 01

  water molecules will be in this kind of ionized state. And it's very, very important. That's the pH, that's where the pH of water comes.

• 1:06:58Nathan Labenz?

  Ten to the minus seven, yeah.

• 1:06:59Speaker 01

  Exactly. Yeah, yeah. So we know that this happens in real water, right? And so the remarkable thing is the neural network potential is actually kind of predicting that this will happen, right? Saying, oh, you know, possibly what, you know, it's running these simulations and then every now and then occasionally one of these hydrogens is hopping off onto its neighboring atom. And that's not something that it's seen in the training data. And yet somehow it knows that that's kind of a fairly plausible thing that might happen. And then the really impressive thing is it keeps the correct shape of that hydrogenium ion, that H3O ion, it keeps it in this kind of pyramid type shape, which was really surprising. I would have thought it would kind of, once you get outside of where it's seen, I would have thought it would just kind of break and go crazy or do something very unphysical, which you do some, you do see quite often, honestly. So I would have thought of break and not be able to handle that. But it actually keeps running, the simulation stays stable, and then you see the thing called

• 1:07:41Speaker 01

  growth is hopping, which is then another hydrogen will hop off from that H3O to its neighbor and you get these chains of hopping hydrogens, which is actually this really important mechanism, which explains why acid diffuses through water so quickly. So this is actually a really important physical phenomenon. The neural network potential is just kind of guessing that it's going to happen, which I thought was quite impressive. Now, this is very well studied, so it's not really telling us anything new, but it does imply that maybe there will be cases where it could find things or suggest things that are new that we haven't thought of, which is quite an interesting idea. Now, but there's a big caveat, which I need to mention, which is coming back to the same point about crystallization, that it doesn't get these quantitatively precisely, right? So it's not going to predict the rate at which this occurs exactly. In fact, it does it much too fast. So it shouldn't really happen in a normal simulation.

• 1:08:25Speaker 01

  You shouldn't see this happen on the kind of, you know, this picosecond timescale we normally access, or you can get to nanoseconds or much longer with these NMPs, these neural network potentials, but you still shouldn't really see it. You should have to force it to happen. So the fact that it happens is not necessarily physically correct. And so that's an important caveat, that, you know, it can't get things quantitatively precisely outside its training data, but it's almost, you know, it's not necessarily a bad thing if, when it goes outside its training data, it can still make reasonable guesses and suggest interesting phenomena that might occur. And that's something that we can easily fix if we want to. And in fact, people have now fixed this and trained these neural network potentials explicitly to get this process right. So you add training data exactly on this hydrogen ion or hydrogen atom as it comes

• 1:09:07Speaker 01

  off the water molecule, get lots and lots of quantum chemical training data on that, and then simulate the whole thing very well, and you can actually precisely predict now that, that pH seven value of water, which is very nice.

• 1:09:17Nathan Labenz?

  So how does that process work? I guess I'm imagining when you're generating the training data via the simulation, you could like take one initial condition and just run it indefinitely. But I guess you could also set up a bunch of initial conditions intentionally and then run from a bunch of different starting points. I'm guessing something like that is what's happening where people are saying, let's bring two water molecules into the right, you know, form the right configuration for that to happen so we can start to have more of that in the training data. Is that right?

• 1:09:51Speaker 01

  Yes, exactly. So what I normally do is just start with what's called an equilibrium molecular simulation. So you just give the add energy, tell it, you know, the temperature, room temperature, and then you just run the simulation and generate your training data from that. But that won't sample what we call kind of rare events, which are things that happen, you know, very infrequently, things like crystallization or this hydrogen hopping off the water molecule type phenomenon. And so there are various strategies you can use to kind of get some of those more rare events. One is you can run it at higher temperature or you could, yeah, as you say, run a ton of these things kind of in parallel and hope that that explores your face space better with different initial starting coordinates. Another thing you can do is called enhanced sampling, where you kind of drive these systems out of their local minimum, you know, the positions they'd like to sit in, you kind

• 1:10:36Speaker 01

  of force them to different arrangements and things. But then another really nice idea, which is probably the most powerful one, is this idea of active learning, which people have shown now works really well for these. And the idea here is you actually, you train a neural network and you go and run much longer molecular simulations with that neural network. And then you have some method of estimating the uncertainty in the predictions of the neural network. And then whenever that uncertainty spikes, whenever it says, oh, I'm not sure about this one, then you take that data and then you go and add that to your training data set and do new quantum chemistry calculations on it. And then you can expand your data set to cover kind of these rare events that don't normally happen in the simulation when you're just running these short ones to keep the training data. And people are showing now that this is a really powerful technique and you start

• 1:11:21Speaker 01

  to kind of, and you can imagine this is kind of a feedback loop, right? You can start to build your training data set, cover more and more space, get cover more interesting phenomena. And then you can start to think about simulating things, yeah, like chemical reactions, which is really important. So people now are doing some really nice things on this, simulating, you know, catalysis, so the reactions on surfaces using these techniques. And they're getting really powerful. So that's really exciting as well.

• 1:11:42Nathan Labenz?

  Yeah, that's interesting. That honestly reminds me in some ways of just down the fairway LLM app development that I've done at times where there's this sort of bootstrapping loop in a lot of cases where, especially if you're working with a pre-trained model, as we normally are in a language model case, you don't need that many examples to fine tune. And I often recommend like S V U S 10 can be enough to do that initial fine tune. And then it's all about identifying edge cases and supplementing that data set.

• 1:12:12Speaker 01

  Exactly, yeah.

• 1:12:14Nathan Labenz?

  It's a more quantitative way to identify your edge cases than I typically do.

• 1:12:18Speaker 01

  Yeah, because the problem with the chat GPT, you know, the hallucinations and things is there's no just function you can go call and go say, you know, what is the ground truth here? And so, you know, it's very hard to tell or to add to your training data the correct things. Often you'll have to go and get a human to kind of label those things. I guess this is the way you do the same thing for a large language model. But the nice thing here is you can kind of do it all within a computational loop, right? So you can kind of automate it and get it all internally happening on one supercomputer. And so you can really start to get these things scaling up in a kind of automated way, which is quite nice because you've always got that kind of ground truth. You can go and just automatically refer to them and calculate, which makes it a kind of really nice powerful technique, I think.

• 1:12:56Nathan Labenz?

  Yeah, so if we're seeing this level of generalization where we're seeing these self ionization and crystallization phenomena happening with really rather modest compute resources to generate these data sets, then I guess I wonder, how far can this go? And you're probably wondering that too. But has it gone any farther than this yet? And what would be prospects for, I guess there's a lot of different dimensions, right? I guess I don't really know how you're dealing with temperature. You said a second ago that you can, you know, you put your temperature into the simulation. Do you also put the temperature in as like a free parameter into the network? Do these networks have or have the prospect to accept a bunch of other free parameters like temperature, pressure, whatever else?

• 1:13:40Speaker 01

  Yeah, so the really nice thing about this method is that the potential energy surface, you know, these energies and these forces that we're learning, those don't depend on temperature. So when you solve Schrodinger's equation, it doesn't know anything about temperature, right? That's just, give them the positions, give me the energies and forces. The way that temperature comes in is actually in this, this classical motion solver, you know, the solving Newton's equations, where you calculate how the atoms are moving. That's where you add a thing called a thermostat, which just basically controls the temperature, right? Just like a normal thermostat. If it's getting too hot, you'll remove some energy. If it's too cold, you'll add some energy. So it's actually really trivial to run these things at high temperatures. So we've run all these electrolyte solution simulations at different temperatures and they, and they look perfectly stable. So that's quite exciting because it should be, yeah, definitely generalizable to different temperatures and as well. Um,

• 1:14:25Speaker 01

  Microsoft just had a big paper called MatterSim, where they're looking at this and they, they showed, you know, you can simulate over a huge range of temperatures, which is really nice. So definitely, yeah, that's the kind of directions we want to be going in, right? Because this is where it gets very hard experimentally. You can, you're going to measure everything at different temperatures and then when things get hot, you know, your equipment starts to run into issues. So this is a really nice, um, aspect of this computation. Like it's almost trivial to change the temperature. Now you may, in some cases, you probably need to add training data at these higher temperatures, say, or so there may be some cases where things break at different temperatures if you don't have enough training data. But in principle, yes, changing the temperatures is a very trivial thing to do. And this is the nice thing about kind of coupling it with this more physical, um, solver, you know, this kind of physical engine, the physics engine, because that means that you don't really

• 1:15:08Speaker 01

  have, you know, if you were relying on the neural network to learn everything, it would have to learn, like, oh, how does everything change with temperature, right? Um, so that would be a much harder kind of AI problem, I think. Whereas in this case, you can kind of use the fact that you're learning something that's independent of temperature, which is a really nice feature.

• 1:15:23Nathan Labenz?

  So if you have this thermostat and you can turn it up and down so freely, can you turn it down to freezing point and see everything crystallize or turn it up to boiling point and see everything turn to gas?

• 1:15:35Speaker 01

  Yes, yes, you can definitely do that. The issue with just cooling everything down to try and crystallize it is that you get this called amorphous crystallization. So like a glass, you know, if you take silicon and cool it down too quickly, it amorphously crystallizes and forms like a, basically just like a solid liquid. Just a liquid that stops, the atoms stop moving very much. So if you just turn the temperature down too quickly, that's often what you see there. But, um, so it's not, it's not as easy to induce crystallization as you might think, but, um, yeah, that's certainly something, yeah, that we want to explore more. Yeah. Can you do those kind of things? Absolutely.

• 1:16:06Nathan Labenz?

  When I freeze water in my freezer at home, I'm getting an actual crystal, right? Or am I getting a sort of water glass?

• 1:16:14Speaker 01

  Uh, yeah, you can get water glasses, but those have to be done in the lab because you have to freeze it very, very quickly, I think. Um, in, in the normal forms of ice are a type of crystal. Yeah. They have a nice, um, crystalline structure, kind of where you get this hydrogen bonding structures forming. So, yeah, you're getting a crystal when you freeze it at home, but there are certain types of ice. You know, there is actually a whole phase diagram of ice can form all these crazy different phases of different temperatures and pressures and speeds. So if in a simulation, you know, because we're using these picosecond time scales, if you just turn the temperature down too quickly, it will just, um, you're probably just getting an amorphous freezing. But people have, people are now using these new potentials a lot to look at, um, ice phases and ice water equilibrium and doing all sorts of fantastic work there.

• 1:16:54Speaker 01

  So you can definitely probe, uh, the crystallization, crystallization of ice water. And there is actually a paper. So there is some precedent for the salt crystallization we saw. Uh, there's a nice paper, uh, where they looked at, um, simulating water and they showed basically the same thing. You can see little, um, water structures that look a little bit like the ice structures. And so you can actually train a new network on water and then use that to say interesting things about ice as well. So there's probably similar parallels going on with, um, ice crystallization, water crystallization, which is ice, yeah.

• 1:17:22Nathan Labenz?

  So what other dimensions of generalization do you see or expect? Like you could imagine training on two kinds of salt solution and trying to predict what happens with a third.

• 1:17:36Nathan Labenz?

  I mean, obviously you could really run with this thing a lot if you're scaling up the data sets and it sounds like there are. Quite a few orders of magnitude of scale up that would be like start to get expensive, but certainly not out of bounds of the sorts of models that we're seeing trained today. You're talking about a couple of days of compute time here or there. Certainly, you probably have a thousand X room to go that you could like plausibly write a grant for, right? What would you expect to see as you go up these orders of magnitude?

• 1:18:05Speaker 01

  Yeah, definitely. So that's the interesting thing moving forward. And people are already doing this. In fact, ByteDance, actually, you know, the company buying TikTok actually had a paper on electrolyte simulation on exactly this, right? They're trying to do this scaling up, right? So they're not, I'm, I'm trying to get these things really, really accurate and precise. They're trying to scale up and look at many different electrolyte solutions. And they'll build a big database of electrolyte solutions. And they saw exactly this. They could train their model on certain electrolytes and then generalize it to new ones that hadn't been seen in the training data, which is very interesting. So they were simulating organic electrolytes, I think, where it turns out, you know, these things are made up of carbons and hydrogen. So you can kind of extrapolate from one to another, which is quite exciting. So definitely, we're starting to see evidence of the ability of these things to generalize to new cases where they don't have the training data on them,

• 1:18:49Speaker 01

  because there's enough similar examples where you do have the training data. And so that's really the exciting direction to go in. And yeah, so this is starting to become a big field, you know, this and you can, people talk about these things as the universal kind of machine learning potential. And the basic idea is you want to train one model, that then you can put any elements into it and any kind of arrangement of atoms and calculate the forces on every single one of them. And so it's quite different to what I'm doing, where I'm trying to build a model just for one particular case. They're trying to build these universal ones that could handle any arrangement of atoms. And this, the big tech companies are starting to get into this as well. You know, so the ByteDance has the electrolyte one. Microsoft has one called MatterSim just came out. Google had this nature paper recently where they trained one of these called Gnome.

• 1:19:30Speaker 01

  And they showed that they could find millions of new crystal structures, which was very, very cool. Meta are working on one of these as well. So yeah, all of these, this is starting to really take off now. And it's the kind of thing that I think maybe industry is a better place to do, because they have these huge compute resources and they're starting to scale up these models and build these really universal ones. And what we're hoping, and I think we're starting to see evidence of this, certainly these papers are starting to show evidence of this, is that you'll see the same thing. That you saw with the large language models when you scaled them up, right? That yeah, exactly as you were saying earlier, you don't need to add nearly as much additional training data. Once you've got this good, really good foundation to to start with, you know, so it's learning all these kind of general features about the ways atoms interact. And then if you want to get, if you want to study a new system, just adding a little bit of extra data on that new system will then get you a really accurate description of it.

• 1:20:14Speaker 01

  Much more quickly than you would need if you were starting from scratch. And so that's really exciting because you can start to think about, well, you know, this could be a universal tool that we start to use as a kind of first. Estimate to get an estimate of what any system is doing, and you just go straight to one of these universal force fields and simulate it. If they could run them quickly enough and start to get a picture of exactly what's happening at the molecular scale. For all any system you're interested in, which is a really, really exciting idea, and I think could be a profoundly kind of transformative scientific tool. Once you get this thing really up and running efficiently and and get it in people's in the hands of scientists who can really use it and to do exciting things with it. And there will always be cases where it breaks and where it fails, right? Just like ChatGPT hallucinates and goes wrong in different places. You know, there'll be systems where it where it has problems and has issues, and so you'll need to keep um.

• 1:20:58Speaker 01

  In keep it in this kind of interactive loop where you're always adding extra training data, going and doing extra quantum chemistry calculations and then essentially fine tuning it, right? Fine tuning for particular applications is probably what we'll have to keep doing. Uh, but it's a really exciting future, I think, as people start to scale these models up. And there's a nice one called Mace Zero, an open source one recently, which is very cool. And the remarkable thing about that was they trained it on crystal data, so solid data is kind of the inverse of what we saw. It trained on the solution and saw crystallization. They trained it only on crystals and they showed that you could simulate liquids with it. So that was very exciting, too. Uh, so I think these things are are really going to start taking off in the near term future, and that's really exciting.

• 1:21:35Nathan Labenz?

  Yeah, there's a couple of big themes there. One is the possibility of what I've. Borrow a term I borrowed from a friend, the big tech singularity, where it seems like. We we've seen plenty of fiction over the years where like five companies take over the world or whatever, but. This is another dimension on which, man, if you really just have to scale it up and there's only a handful of companies that have the ability to do that, these big tech companies potentially start to come in and dominate material science, right? I mean, is that kind of what you would expect the future of the field to look like?

• 1:22:10Speaker 01

  Yeah, I think it's certainly a possibility and I think it's a risk and I think, you know, the governments and the public sector, you know, needs to get onto this and start saying, okay, this is a serious area. We need to fund, you know, independent groups who aren't, you know, in private companies to build these things as well, build these tools as well, so that we can have experts who are outside these companies who really understand these things. I think that's a really important thing that needs to be done. Yeah, I think, you know, there's no reason, we can build, you know, there's like CERN, you know, the Large Hadron Collider, you know, there's examples of huge science projects, you know, where the public sector has put large amounts of investment and built these big things. So I think there's no reason that we have to leave it to these big companies, right? It's not the kind of thing that's suited to, you know, maybe academia. We have these small groups with one professor and a number of PhD students. Like, I think we

• 1:22:54Speaker 01

  need to move away from that model, but I think there are examples of science, you know, these big interdisciplinary collaborations, you know, we start to think about getting all these people together. And the Mesa Zero paper was a really nice example of that, where they got, you know, a huge team of people together or collaborating to build one of these foundational models. So I think there's hope yet that we can build these things and open source them. But yeah, I think there is really a possibility that these things stay internally in these companies. And so, you know, we saw this big debate around this with Alpha Fold 3, where they're, you know, they're hanging on to, they're keeping those models private, at least for now. And so that'll be a very interesting to see how this plays out. Will they release these models to allow the scientists to use them? Because a big concern I have is they've trained these big models, right? There's so, there's so many incredible simulations you could run with these things. And yet, internally in these companies, you know, there's not that many people to kind of run these simulations.

• 1:23:39Speaker 01

  So I think it would be way more valuable if they released these tools to the public or to other scientists in academia to use them. But having said that, you know, these things take a huge amount of resources to build, you know, if you're spending, you know, huge amounts of GPUs on building these data sets, you know, that's a big investment of money and it's understandable that these companies, you know, want to recoup their investments. So it's a complicated issue and it'll be very interesting to see how it plays out. But I think definitely there's room for kind of the governments and academia and the public sector essentially to get into this area as well.

• 1:24:09Nathan Labenz?

  Yeah. Another big theme is the fact that China is not really all that far behind. I am one who wants to avoid the AI arms race with China almost at all costs. Maybe not at all costs, but you know, in terms of ways the future could go badly. I feel like a throw caution to the wind, AI arms race with China would be a bad one. And one way I think we might trick ourselves into going down that path is if we dilute ourselves into thinking that we have a big lead and we're going to win the race. And so I always note these examples where, you know, Chinese big tech companies or whatever, academic groups in some cases as well, are producing top notch stuff. And as far as I can tell, don't actually seem to be all that far behind.

• 1:24:51Nathan Labenz?

  That could change, but I don't even need to comment on that unless you have one. But it is notable that they're not far behind on this dimension either.

• 1:24:58Speaker 01

  Yeah, there's definitely a lot of fantastic research in this area coming out of China. Yeah. And it, you know, I think it has big, big economic implications. You know, you could start to really do materials design, understand biology. You know, there's so much, you know, whenever you get big scientific breakthroughs, it has massive technological and economic implications as well. So yeah, I think it's something that we should really be thinking about a lot more. And yeah, how to avoid the arms race, I'm not sure. That's a very tricky question that's above my pay grade, unfortunately.

• 1:25:25Nathan Labenz?

  Unfortunately, I don't think there's any designated adults either that we can question too. Regardless of pay grade, we might all have to take that one out. Yeah, that's true. Our little part on it. Is there like a safety concern with these sorts of models? I mean, obviously the people talk a lot about highly generalist models and are they going to do what we want them to do and not. And then with biology models, you have sort of inherent dual use of like, if you can cure a disease, you can probably also cause a disease. Is there something similar here or is this more of a one way bet type of technology? Like I have a little bit of harder time imagining the sort of harmful materials, science use case, but maybe it's that I just have limited imagination.

• 1:26:07Speaker 01

  No, no. So I mean, one point is that these neural network potentials are not at all just limited to material science, right? People are also using them in biology to really interesting effects. So there's a paper from the David Baker Group in Science recently where they used any one of these neural network potentials to design new drug molecules. And that seemed to work pretty well. And the state of the art, in fact, for this problem of drug binding to proteins, you know, calculating how strongly a drug molecule will bind to a protein, a recent preprint actually showed that the state of the art was using one of these neural network potentials, at least for the core component of it, for that actual drug molecule. So these things definitely have implications for biology as well. So just the same pathway that, you know, biology and AI, you know, is a very powerful tool could be potentially a concern there. You could use these neural network potentials

• 1:26:52Speaker 01

  to design, you know, drugs and things that could potentially be harmful. So yeah, that's definitely a concern. Yeah. In terms of material science, yeah, I don't think so much. It's probably not so much of an issue there, you know, other than to say that, you know, material science is really important for, you know, a lot of defense industries, obviously, if you're trying to make, you know, various new materials, you know, a key limitation of a lot of things is the ability to design new materials. But I'm not too concerned about the safety things at this stage. I wouldn't be too much too worried about it. But yeah, you can imagine at some point when these tools get powerful enough, that they start to have, you know, enough technological implications, that it's something you want to start thinking about quite seriously.

• 1:27:31Nathan Labenz?

  Have you seen the recent paper on cans, instead of the TEGMARK group?

• 1:27:37Speaker 01

  KAIANs, yeah.

• 1:27:38Nathan Labenz?

  Yeah.

• 1:27:40Speaker 01

  That's a very interesting paper.

• 1:27:40Nathan Labenz?

  But it's a basically new architecture, right? Where instead of learning the weights that are edges between nodes in a network, they are learning functions that sit on those edges. So instead of just a single number at each edge, it's a function at each edge. And then instead of having an activation function on a layer by layer basis at the nodes, they're just adding the inputs from the different learned functions on the edges. So this makes more intuitive sense if you look at a diagram, which they have in the paper, and definitely recommend that, and hopefully, again, have an episode coming soon about it. They seem to think that this is going to be most applicable in science because it seems to address the sort of

• 1:28:25Nathan Labenz?

  Composability problem where you can now learn like functions of functions and you know do all sorts of interesting more and it's also like much more interpretable because you can actually just look at the shape of the individual functions that are being learned. Do you think that has promise for the sort of work you're doing or how would you connect the dots between your work and that new potential tool?

• 1:28:49Speaker 01

  Yeah, that's definitely really exciting. I think it certainly has potential. I really like this idea. The problem essentially is a simple words like fitting a surface, right? It's a very high dimensional function. You have some the positions of every single atom and you're trying to predict the energy, just a single number. And so that's exactly what these, you know, KPIs are trying to find good approximations of these multi-dimensional, high dimensional functions. And so that, yeah, it makes sense to me as an approach that could really perhaps help with the generalizability and build these things that are generalizable. And I like, I think they showed in the paper that you could improve the generalizability without spoiling the what they'd already been trained on, whereas with some of the normal multi-layered septon type approaches, perhaps you would lose some of what it had been trained on previously when you added additional data and fine-tuning these things. So yeah, there's

• 1:29:34Speaker 01

  potential advantages there. I definitely think these things should be explored. I think a big issue though is obviously that the computational limitations, what's going to be computationally most efficient, you know, because in the end we are looking to run these things as fast as possible. And so you really need to demonstrate, you know, that you get some really solid advances in order to justify kind of switching to a new architecture. But I definitely hope someone is trying out building a neural network with these cans. I think that would be really cool and really interesting to see how they perform. I think that makes a lot of sense. I think, yeah, my kind of philosophy and the thing I love about the machine learning community, which is that it's very empirical, you know, so there's a lot of trial and error and just empirically testing things out, seeing what works well and seeing what doesn't. But then also kind of using kind of fundamental

• 1:30:17Speaker 01

  theory where you need to is inspiration for coming up with new ideas. And so that cans was a nice example of that. But it'll be interesting to see, yeah, how it plays out if they are actually generalizable. I saw some people arguing that they were kind of, you could map them to a simple multi-layered septon in a kind of in a way. So whether they are genuinely give you additional new properties that or new behaviors will be very interesting to see or whether it's just a matter of, yeah, whether they reduce training data requirements. It's always very hard to predict in advance, you know, what what's going to work and what's not. So it'll be great, interesting to see. But I think this just needs to be empirically tested, you know.

• 1:30:51Nathan Labenz?

  There's a definitely a strong trend of some sort of fundamental equivalence between a lot of very apparently different things.

• 1:30:57Speaker 01

  Yeah, sometimes I get frustrated. I feel like there's some deeper theory missing underlying these things, you know, and I think it would be really great if we could get a more rigorous description. And I know a lot of people on this, yeah, of exactly why these neural network potential, you know, neural networks in general work so well. When do they work? When are they not going to work? You know, that's really lacking. And I know that's a big area of research. So it would be very interesting to see what comes out of that.

• 1:31:20Nathan Labenz?

  So just zooming out, like, how do you expect all this to shape the pace of discovery in science over the next few years? It seems like you've touched on a number of, obviously, super high value use cases. And I guess the general expectation, very similar to the, you know, episode we did on the biological applications of this, and this sort of overlaps with that, but is definitely more fine grained. I guess the loop is like, we're going to run a lot of experiments on the computer. We're going to identify things that look good. Then we'll do actual experiments with those to try to verify that they're right.

• 1:32:03Nathan Labenz?

  If they're right, we'll have new discoveries. If they're wrong, we'll kind of feed that back end and patch the models and try to get better on those failure modes. And it seems like in biology, we're looking at potentially one to two orders of magnitude speed up in terms of how much like overall resources, you know, human and and financial maybe would need to go into discovering new things, maybe even more than that. It seems like one is kind of a bear case and, you know, two is maybe like the mainline case and above above two orders of magnitude would be like, obviously, a Super Bowl case. Is that your expectation?

• 1:32:44Nathan Labenz?

  How do you expect the sort of trajectory of technology progress to change based on these tools?

• 1:32:51Speaker 01

  Yeah, I mean, I'm really excited and really optimistic, frankly. And you want to be careful because you don't want to overhype things. And your other people will disagree. Obviously, a lot of senior professors, in fact, you know, are quite skeptical of a lot of these things. But in my opinion, I think they're genuinely a transformably useful new tool, which will have a profound impact on, you know, every discipline of fundamental science, I think. And so I'm really, really excited about them. The central challenge, you know, AI. Is clearly an incredibly powerful tool, but the central challenge of it is the training data requirements, right? You always, you know, these are very data hungry beasts. So you wanna be thinking about where are the problems where you can generate large amounts of data efficiently and increasingly diverse but very high quality amounts of data. And so

• 1:33:35Speaker 01

  I was stuck with, you know, machine learning, AI for science for a long time because for electrolyte solutions, you actually have not that much data, you know, it's fairly limited amounts of data that you can use to train these things. So I didn't really get into this until around Alpha Fold came out quite late. But since then, it's become, you know, really clear that you wanna, that there are cases where you can, yeah, exactly as you're saying, have these stages where you can iteratively generate larger and increasingly diverse data sets. And one way to do this is with kind of automated labs and or having experiments in the loop, you know, where you're constantly generating extra data. And there's an interesting article in the New York Times about, you know, it's a pharmaceutical company where they have these, you know, huge automated labs where they're generating all of this new data, which is really one way to go. And I think that's really promising. And then the other way to go, I think, is to use kind of quantum chemistry, fundamental

• 1:34:20Speaker 01

  physics to generate this data on the fly as well. And I think that's a really nice, exciting avenue. And I honestly don't see like what the limitation is, you know, there's probably things that are unforeseen, you know, that will cause these things to not be useful at some point maybe. And but at this stage, I don't really see what the limits are in terms of like, as you scale these things up, get larger and bigger models, why you won't be able to simulate a really large range of interesting systems and really get a lot of insight into many, many different systems that we've been waiting for a long time to really understand. Because if you look at the way, yeah, traditional experiment works, drug discovery in particular, you know, there's huge amounts of trial and error, you know, so working with experimentalists, there's a lot of just intuition and guesswork and trying different things and then getting some interesting results and then saying,

• 1:35:04Speaker 01

  well, we don't know what's actually happening at the molecular scale and then trying to work out some way to model it. And there's this really just, you know, you can just feel the progress being held back by a lack of ability to just really understand what's happening, particularly at this molecular scale, which is so critical, where so many important processes happen. So if you have this ability to simulate these processes really fast and really accurately, I think that could be really transformative and useful. And I'm critically excited about it, both for biology and for, you know, material science and chemical engineering and even things like climate change as well, where there's a lot of these important molecular scale processes go on. So I think it's a, yeah, really exciting new time to be involved in science.

• 1:35:43Nathan Labenz?

  You had a couple of interesting thoughts on the importance of or the opportunity of taking tools from one domain and applying them to another. And I guess you can comment on that in any number of ways, but I'm curious if you see like untapped opportunity right now of tools you'd like to see move from one place to another. We talked a little bit offline about the diffusion models roots in physical processes and how that's obviously now gone kind of horizontal and is in use in a lot of different areas. But tell us more about this sort of borrowing tools from one domain to another, present and future.

• 1:36:20Speaker 01

  Yeah, so there's this brilliant book I was reading recently called The Incomplete Guide to the Art of Discovery by Jake Oliver, who was a scientist involved in discovering plate tectonics, you know, which is this big paradigm shift in geology. We realized that the Earth's not static. It's made of these tectonic plates, which were all moving around. And he made this fantastic point that there's a really easy kind of hack to do really exciting science, and that's to find tools from one field or discipline and take them and apply them in your own. So this is what they did to discover plate tectonics, particularly taking a ton of tools from physics and things and applying them to discover that these plates were moving around. And I think when you start to look at science, you see this cropping up everywhere, you know, historically. So the classic example is, you know, physics itself, right? When Newton realized that if you took, you know,

• 1:37:02Speaker 01

  tools from mathematics, calculus, and then applied them to physics, you get this transformatively useful tool, right? You can suddenly explain gravity and huge amounts of physics, which was incredibly exciting. And that's so kind of ingrained in physicists now, but we don't really even think of that as being kind of interdisciplinary, but it really still is a type of interdisciplinary. And then there are many other examples of this, right? X-ray crystalography, for instance, or that you use to get crystal structures of proteins, you know, that came out of physics and you're applying it to biology. So it's a very, very common feature of important science, taking these tools from one field and applying them to another. And yet diffusion models are actually a fantastic example of this. Where if you go back and look at the original papers that motivated these diffusion models, they're actually explicitly citing non-equilibrium statistical mechanics as inspiration for these models.

• 1:37:46Speaker 01

  And in some sense, using actually the mathematical tools of it. So using thermal annealing and using, I think, longevon dynamics, you know, which is actually the equations or the algorithms that we use to do molecular simulations. Invented by physicists to simulate molecules moving around. And then, you know, people thought, oh, we could take this and adapt this to generating new images. And then you've got these fantastic results, you know, we can make these incredible images now using these algorithms. So that's another example of this more recently. And then the coolest thing now, though, is that we're starting to go back in the other direction and kind of complete the loop. And you had this fantastic guest on the other week talking about, you know, all the ways these diffusion models and things were being used to impact biology. And problems that we would traditionally have been tackled with normal molecular simulations. So you're starting to complete this loop where you're using these same tools again now back in biology, which is very exciting.

• 1:38:31Speaker 01

  And doing molecular simulation, which is very, very cool. Although you want to be a bit careful because then it turns out that you may end up just doing molecular simulation again without realising it. Which I think there's a little bit of that going on there. You're running longevon dynamics and you're using it, this thing you call the score. But there's a great paper out on Microsoft recently where they showed that if you train a diffusion model on molecular simulations, the score, which is the thing the diffusion model is learning, is actually mathematically identical to the force field. And so you've actually come back full circle and you're just doing molecular simulation again. But now you've got a bunch of extra exciting nice tricks you can use to make it really more efficient and make it more generalisable, which is exciting. So, yeah, I think this is a really profound example of this importance of interdisciplinarity, right?

• 1:39:12Speaker 01

  And thinking across different fields and trying to take tools from one field and use them for another. And I think looking forward, another example of this is this idea of taking tools and, I think, understandings from statistical mechanics and actually applying them to understand the theory of large, you know, of AI itself, of machine learning, of what's actually going on in stochastic gradient descent. What are things learning? What does it mean for things to learn? And I think there's a ton of tools from statistical mechanics which could really be brought to bear on this. And we're starting to see some examples of this. So the paper just the other day showing that the the distribution your parameters go to when you do when you're training a model go to a Boltzmann distribution, right, which is a is a tool from statistical mechanics. It's also the distribution of molecules go to this Boltzmann distribution. So we're starting to see really interesting parallels emerge.

• 1:39:57Speaker 01

  And then another really lovely example of this was a Japanese group actually looking at the mathematics of phase transformations. So this is a we have beautiful theories which can explain phase transformations, which is exactly like crystallization. And what they actually showed is when you're training these neural networks, you start to see some of these same mathematical features emerge in this training of these neural networks, which is a very nice idea. So then you could start to think, oh, a dropping, you know, this phenomenon that seems so mysterious, actually it has a lot of parallels to something like a phase transformation in physics, right? And when you look at something like a learning curve, it actually looks a lot like an energy curve when you're equilibrating a in D simulation. You know, it starts high, it drops down to a low value, and it'll be jumping around some plateau with some noise. And then you may get a phase transition where it suddenly

• 1:40:40Speaker 01

  drops lower, right? And that's exactly what you see when you're trying to training a model and you see these rocking phenomena emerge. So this is a really exciting direction, I think, as well, using tools again from different fields to try and understand things within your own field.

• 1:40:53Nathan Labenz?

  You see something like Sora, and the OpenAI folks behind that are less interested in the video generation and more interested in the like world simulation. And, you know, it's obviously like macroscopic world simulation. What do you think is the outlook for like one model to rule them all across all these orders of magnitude versus when you see this coarse graining? Is that something that sort of inherently suggests that we're going to have a hierarchy of models that handle these different orders of magnitude of system? Or can that all be integrated into one thing if you just dump everything into one one data set, one training run, one network?

• 1:41:37Speaker 01

  Yeah, that's a fascinating question, right? Yeah, that would be the dream if you just have one neural network to rule them all, yeah. I don't know that we'll get there. I mean, the way I think in the future, you know, thinking long term, how we'll do these molecular simulations is you'll use kind of a hierarchical coarse graining approach. Perhaps you could incorporate this into one model, but where so you would have a neural network that ignores the parts of the systems that aren't relevant, but then it will have, you know, higher resolution pictures of the system that are relevant. So you think about something like a drug binding to a protein, right? You at the local region where that that's a really important process, right? You probably need an atomistic description of that. You need to know where every atom is as a function of time, something like that. But further away from that protein, you could probably use a coarse grain description, where you're not treating every water molecule,

• 1:42:21Speaker 01

  and you're only kind of treating them a kind of average way, using this kind of coarse graining technique. And so I think that's really exciting long term, thinking about how you can build models that kind of combine different scales, kind of multi-scale modeling, all within one neural network, that both study the kind of atomistic detail where it's important, but then where it's not important, and you can kind of average it out, it can do that kind of on the fly. And so that's really exciting. We can start to think about potentially, yeah, building models that can do both scales at once, or look at multiple scales, because there's a huge range of phenomena where it's not just happening at one scale, right? Where there's molecular scale processes that then determine larger macroscopic things. And so you need to have this kind of multi-scale approach. And that's a really challenging problem. It's going to take us some time to work out. But I think, again, there, yeah,

• 1:43:02Speaker 01

  machine learning and AI will be a critical tool to kind of try and help us do that efficiently.

• 1:43:07Nathan Labenz?

  Would it be like one, would the first step in your world toward that be training a single network on the all-atom versus and also the coarse grain as opposed to like separately? Have you tried that?

• 1:43:21Speaker 01

  Yeah. No, I haven't tried that, but it's a great idea. And then having it or allowing it to kind of automatically jump between descriptions where, you know, it thinks it's best, you know, maybe I need an all-atom description here, so I'll go back to a higher resolution picture of it here would be very cool. So that's, yeah, something I think, yeah, people should be thinking about more long term to try and do something more ambitious, yeah. Because this is the real key problem is that you want to use this coarse grain picture so you can look at bigger systems, but then also these microscopic phenomena are really important in many cases. So having some way that can couple these things in one neural network would be very cool. But yeah, that's a tricky technical problem. I'm not too sure about how to go about.

• 1:43:57Nathan Labenz?

  Cool, there's always more to do, at least for now, until the AIs are doing it. For a few years at least, there's always more to do.

• 1:44:04Speaker 01

  I mean, that's one other point I would I would make is that, you know, I'd really encourage, you know, there seems to be a real excitement and attention around making an AGI or artificial superintelligence, but, you know, I just encourage people that there are so many important, incredibly exciting problems you could tackle with machine learning, you know. And I I think this this goal of kind of creating something that's smarter than any human is maybe something that, you know, it comes along with all these kind of risks and uncertainties. And I know it's an appealing idea, but it's maybe not the kind of thing that society actually wants us to do, you know. So I encourage, you know, machine learning researchers to think, is this actually a goal that that that we should be pursuing? So, you know, because we're throwing billions and billions of dollars at it, it seems at the moment, right? But I think if you go out and ask the average person, like, do you want something that's smarter than a human in every single way, they'd probably say no, right?

• 1:44:49Speaker 01

  They'd say, no, I want something that, you know, can do my dishes and do my chores, but not something that can make art and music better than I can, right? So and similarly for scientists, you know, I don't want an AI bot that can come along and do hypothesis generation and come up with new ideas and read the literature. And like, those are the things I love doing and solving problems, you know, that's what I like doing. So what I would kind of prefer is people to build tools that I can use to do things like massively accelerate simulations, for instance. Take the kind of really hard problems that we're stuck on in science and give us new powerful tools that we could then use to solve problems. So that's if I there's one thing, a message I could give to that community would be kind of just try and find tools to solve problems for humans to do these things.

• 1:45:30Speaker 01

  Because building something that's smarter than any human is not really something that that I think any of us really want to do. And so, you know, like geneticists could be, you know, using CRISPR and genetic editing to make the humans that are smarter than anyone. But it's it's not something that they're trying to do because it's not something that society has decided that we want to do, you know. So I think it's we want to take a step back maybe and sometimes think about at a more fundamental level, what are we trying to do and why are we trying to do it, you know? So Sam Altman talks about this idea of. You know, wants to build an AGI that can discover the grand unified theory of physics, you know, unifying gravity and quantum mechanics and then explain it to them. But like, I think that's something that we should save for a human, right? And we should build tools that help us do it. But I don't think we just want to have an AI explain it to us, right? That feels to me like just looking in the back of the book and just reading the answers, right?

• 1:46:14Speaker 01

  I mean, there's a source of meaning for so many people. And I think it's maybe something that we should think about trying to preserve. And I understand that, you know, an AGI would be incredibly practically useful. You know, it allows to help solve cancer and things. But I think there's ways you could use AI to help solve, you know, these problems by developing really powerful tools, rather than just by developing something that just tells us all the answer.

• 1:46:34Nathan Labenz?

  Yeah. Yeah, amen to that. I'm very largely with you. I think there's an interesting distinction there too between is it the power? Or is it the generality that should concern us more? I tend to fall more on the fear of generality side. And it's an odd feature of the current reality that like the path to a lot of the best performance on even narrow tasks is like through pretty generalist systems. But I do share your concern that, first of all, that I'm like, not sure society wants an AGI or certainly a superintelligence.

• 1:47:14Nathan Labenz?

  Obviously, I definitely do think the risks of that are not to be dismissed. And it does feel to me like a kind of fundamentally ideological project. It's not necessarily framed that way, but I do think that it, you know, it probably should be understood that way. But when I, I think the only point of divergence, I think, from what I understood you saying just now is if I could have a narrow system that could do some of these like frontier things, but couldn't do anything else, you know, sort of the alpha go of like understanding physics.

• 1:47:51Nathan Labenz?

  I think I would be on board for that, even if that meant that the human who counterfactually would have made that discovery, like gets deprived of that, you know, eureka moment, the joy of that eureka moment. I still think I would probably be inclined to make those trades. But the part for me where it becomes like very concerning is when that physics solver is also like a general purpose intelligence that can do anything better than a human. And then we have all these sort of control problems along the lines of the research coming out of Anthropic, but you have these sort of, they generalize in pretty nasty ways, it seems like too, where they just showed one where simple flattery or sycophancy toward the user starts to generalize toward like much more problematic forms of deception.

• 1:48:36Nathan Labenz?

  And it's like, yikes. I think the path through this like superhuman intelligence to a good outcome is definitely fraught, to say the least. If we could find a path to a narrow physics solver or material science discoverer that was better than humans at only that narrow task, I think I would be on board with that. But unfortunately, right now, like that path is not, not super obvious.

• 1:49:03Speaker 01

  Yeah, no, fair enough. I understand if we could build one of these things, yeah, it would probably inevitably get built. I just think, I don't know if we want to throw that many resources at it. It's a question of where we direct our resources to. You're building a narrow AI physics solver. Yeah. If it's easier, then sure, yeah, then doing it, you know, building tools for humans to use, then yeah, maybe that is the way to go. And yeah, the other, yeah, massive obvious counterargument is obviously the only way to get something that's really useful is to build this generality into it. So maybe there's no way to build things that are narrowly really good at individual past. And the only way to really go is to build this kind of general intelligence. Yeah, which may be just an unfortunate fact of the way reality works. But yeah, it will be interesting to see how it plays out. It's gonna be a fascinating time.

• 1:49:44Nathan Labenz?

  Yeah, no doubt about that. All right, I'll say it again. Thank you for getting up early in the morning and thank you, Tim Dagman, for being part of the cognitive revolution.

• 1:49:54Speaker 01

  No problem. Thanks for having me. It's a pleasure.

• 1:49:55Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.