AlphaFold is revolutionizing biology

AlphaFold is an AI system developed by DeepMind that predicts a protein’s 3D structure from its amino acid sequence. It regularly achieves accuracy competitive with experiment, and is accelerating research in nearly every field of biology. Daniel and Chris delve into protein folding, and explore the implications of this revolutionary and hugely impactful application of AI. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: AlphaFold https://www.deepmind.com/research/highlighted-research/alphafoldAlphaFold reveals the structure of the protein universe https://www.deepmind.com/blog/alphafold-reveals-the-structure-of-the-protein-universeAlphaFold: Timeline of a breakthrough https://www.deepmind.com/research/highlighted-research/alphafold/timeline-of-a-breakthroughAlphaFold Protein Structure Database https://alphafold.ebi.ac.ukGitHub: deepmind / alphafold https://github.com/deepmind/alphafoldOxford Protein Informatics Group: AlphaFold 2 is here: what’s behind the structure prediction miracle https://www.blopig.com/blog/2021/07/alphafold-2-is-here-whats-behind-the-structure-prediction-miracleNature: How AlphaFold can realize AI’s full potential in structural biology https://www.nature.com/articles/d41586-022-02088-xNature: ‘The entire protein universe’: AI predicts shape of nearly every known protein https://www.nature.com/articles/d41586-022-02083-2Nature: Highly accurate protein structure prediction with AlphaFold https://www.nature.com/articles/s41586-021-03819-2Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Chris Benson?

  I do know that proteins are the foundation of all life. They can be incredibly complex. Many of our longtime listeners will know that I'm really into animal welfare causes, and particularly, I handle venomous snakes quite often. But a friend of mine, Dr. Brent Segal, he and I will often talk about snake venom. And Brent, with his expertise in chemistry, he'll go and check on the protein makeup of snake venom, and then he'll look at the protein molecules and the folds and where they're at. And he can just tell me exactly how those proteins are affecting and if it gets, if someone's bitten. Protein folding may sound really esoteric to those of us who are not in biology professionally,

• 0:42Chris Benson?

  but it's, it's crucial to understanding chemistry and life itself.

• 0:57Speaker 01

  Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive, and accessible to everyone. Subscribe now if you haven't already. Head to practicalai.fm for all the ways. Special thanks to our partners at Fastly for delivering our shows super fast to wherever you listen. Check them out at fastly.com. And to our friends at fly.io. We deploy our app servers close to our users, and you can too. Learn more at fly.io.

• 1:33Daniel Whitenack?

  Welcome to another Fully Connected episode of the Practical AI Podcast. In these Fully Connected episodes, Chris and I keep you fully connected with everything that's happening in the AI community. We'll take some time to dissect a little bit of the latest AI news and dig into a few learning resources to help you level up your machine learning game. I'm Daniel Whitenack. I'm a data scientist with S I L International. I'm joined, as always, by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris? Doing really well today. Excited.

• 2:10Chris Benson?

  About the, the thing that you're about to tell our audience we're going to talk about. And I just wanted to put a tiny bit of, of context around.

• 2:18Daniel Whitenack?

  Sure.

• 2:18Chris Benson?

  We've gone through the, the pandemic and there's, you know, there are major wars that we've talked about, you know, ongoing as we record this. And, you know, monkey pox is now out.

• 2:29Daniel Whitenack?

  Wasn't it just called, uh, I don't know the designation, they're just designated as an emergency status somehow.

• 2:35Chris Benson?

  Yeah, both who and then now the United States has declared it such. And, uh, as we, as of yesterday, as we record this, and so we're going to be talking about a topic today. That reminds me that we live in the most interesting time in human history. And things are changing faster than they ever have. And there's actually a lot of reason to have hope in the world. As we talk about the possibilities that we're going to talk about today. I just want to kind of remind people that, that, uh, there's a lot of things that are really worth being positive about. And I think today's topic is frankly, one of them.

• 3:12Daniel Whitenack?

  Yeah, a lot of, a lot of people really doing things with tech that are beneficial. Or at least the intention is that they would be overwhelmingly beneficial, right?

• 3:23Chris Benson?

  Yes.

• 3:23Daniel Whitenack?

  So I, I think that this factors in. The topic that we'll be talking about today is, is AlphaFold and the corresponding database that they've released of protein structures. This came up and, you know, I was seeing, I don't know about you, Chris, but I've seen it pop up in my news feeds. Various times over the past couple of years. And most recently, just this week, I think was coming up in the news because of some of the things that they've, that they've released. I think in particular, the sort of recent news is that they have this database of protein structures. And we can talk about kind of.

• 4:04Daniel Whitenack?

  What that means and how it was generated, et cetera, here, you know, over the course of the podcast, but this, this database of protein structures, and they've just released and expanded that from 1 million structures to 200 million structures. So it's a pretty big, uh, pretty big increase in terms of the size of, the size of this database. And I don't know, we, we were just talking even before the episode, Chris, about proteins and maybe like how, how those can be important for the study of various things. I don't know if you, you want to chat about that

• 4:46Daniel Whitenack?

  at all, but it was definitely interesting to look at this project and understand a little bit more about that field, which I'm not, I'm not actively participating in.

• 4:56Chris Benson?

  And it's important that we note that we're exploring this as non-experts.

• 5:01Daniel Whitenack?

  Yeah. Come with us along our journey of learning about AlphaFold.

• 5:05Chris Benson?

  So, so, you know, obviously we're here with our listeners because we all love AI and we're exploring things, but often the use cases are things that we don't have expertise in. And this is one of those episodes that we call fully connected where we're just, we're just exploring and we're bringing people along on the journey as we talk about this. And I have no particular expert. I took some biology in, in high school and, and college, but I have no particular expertise, but I do know that, that proteins are the foundation of all life and is incredibly important to understanding how they can be used in their application, their 3D structure.

• 5:45Chris Benson?

  They can be incredibly complex. It's kind of, I'll actually give a, I know I've relayed this to you privately, but I'll give a quick setting on, on kind of why 3D structure is so important. Many of our longtime listeners will know that I'm really into animal welfare causes and particularly I handle venomous snakes quite often with appropriate safety gear and such. But a friend of mine named Dr. Brent Segal, uh, who has a chemistry PhD from Harvard,

• 6:14Chris Benson?

  I will often talk about, just for fun, it's not what either one of us is primarily do, we'll talk about snake venom. As, as a, just a fun two guys chat chatting thing. And Brent, with his expertise in chemistry, can literally look, we'll, we'll be talking, comparing two species. And he will be able to pick up, he'll go and check on the, uh, look at the, the, the protein makeup of snake venom. And then he'll look at them, at the protein, uh, molecules and the folds and where they're at. And right there, off the cuff, he can, he can just tell me exactly how those proteins are affecting. And if it gets, if someone's bitten, uh, what, what that will do and what that particular combination of proteins.

• 6:57Chris Benson?

  And so protein folding may sound really esoteric to those of us who are not in biology professionally. But it's crucial to understanding chemistry and life itself. It really gave me an appreciation for this topic before we got to this episode. And so, um, I'm pretty excited about the possibility and I think it's going to really revolutionize medicine.

• 7:17Daniel Whitenack?

  Yeah. And I think in this, in this episode, at least what we're going to try to do is, is kind of talk through how the context Ralph will fold, the data, how it sort of works and what the implications are. And so getting in the weeds a little bit with how this is actually operating, um, we'll, we'll get there at a certain point. But yeah, I think setting that context is good. I was looking through some articles again, because I'm not a, I'm not a chemist or a biologist, but looking through some articles that we'll link in our show notes as also good learning resources for you. Talking about the sort of reason why proteins and protein folding is useful.

• 7:59Daniel Whitenack?

  This is from the National Library of Medicine, which sounds very official. I don't actually know a lot about the National Library of Medicine. They talk about how the proteins are basic building blocks of all cells and in our body and living creatures. And that we kind of often think of DNA as being at the core or DNA and genes as sort of being at. The core of of the information needed for life, which is true. But then the sort of dynamic processes of life, like the things that happen in our bodies that like the functions and the processes, defense mechanisms and reproduction of certain things in our bodies.

• 8:43Daniel Whitenack?

  All of those sort of dynamic processes are carried out by proteins, which, you know, do this kind of folding and assembly into all of these complexes to actually perform functions. Right. So it's like the functional process.

• 8:59Chris Benson?

  Exactly. I mean, and to really get that tangible, I mean, and these are examples we've seen in many of these articles on this is, you know, the, the fact that your eye and the retina can receive light and process that light to your brain. The mere fact that it can do that is protein based. The fact that right now you're probably, even if you're sitting down, you're probably moving some part of your body. And that movement that you're engaged in right now is based on proteins. It's just impossible to escape that fundamental, you know, kind of function that proteins provide times, you know, a billion different things. And so this, this kind of technology, it's going to be really fundamental to, to, to life going forward.

• 9:43Chris Benson?

  And I know I was joking to you that earlier that I wish I was younger than I am now, not because, not just from an age standpoint, but because then these kinds of technologies could positively influence me for more years than they, than they're currently going to be able to. It's, it's like I, every time I see these great advances coming out and I'm in my early 50s and I'm looking at it kind of going, God, why couldn't that have happened in my twenties or something like that? So it is pretty cool stuff here.

• 10:11Daniel Whitenack?

  One of the interesting things to me is like they're releasing, so we kind of talked about how protein structure. Is important and how it sort of is tied to the the basic functions of life and why that, like you're saying, is important for advances in medicine and other things. What's interesting is that all of this complicated function and process that are carried out by proteins are fundamentally driven by sequences of what's called amino acids. Yes. And there's 20 of these amino acids. And so I was trying to think of like a metaphor and I don't know if this has been used, I'm probably stealing it from someone, but when I was going through it and looking at this

• 10:53Daniel Whitenack?

  stuff, the sequences of amino acids, there's 20 of them. You know, you can think about how much complexity we can see formed out of 26 letters of the, you know, Roman alphabet in all sorts of languages. And there's, you know, you can express, you know, innumerable things with that kind of small set of characters. Here we have this sort of sequence of amino acids. There's 20 of these acids and that's what forms proteins and drives how they fold and how they assemble and how they do all these functions. And so when we're thinking about like, how does this intersect with AI?

• 11:33Daniel Whitenack?

  The process or the the data transformation that we can think about is like, on one end, you have sequences of amino acids that you might know about. And then on the other end, you have the folds and the assemblies and the geometric structures, the 3D structures that are driven, the protein structures that are driven by these sequences of amino acids or that you could predict from these. So an AI model, as we've talked about many times in this show, is, you know, at its core, it's a data transformation, right? You take an image in and then you get a label out or, you know, something like that. Here, you're taking these sequences of amino acids in.

• 12:16Daniel Whitenack?

  And out of it, you're predicting a 3D structure of one of these proteins. That's really the fundamental kind of data transformation that we're talking about, which is what AlphaFold is addressing is sequences to 3D structure. That's, that's at the main core of what we're talking about.

• 12:34Chris Benson?

  And I think in some of the, the materials that we reviewed ahead of time, if I'm understanding them correctly, you know, those different amino acids, the folding itself is kind of amino acid to amino acid. So even though we're talking about sequences and, and you tend to think about a line of amino acids with the word sequence, but it's being folded in 3D with those different amino acids connecting to each other in different ways and lots of different shapes. So even one sequence can have many, many different possibilities there. Going back to your point, you know, even different folds with the same amino acids is, is, is the impression I'm taking away. So there's a lot to, to happen there. And as I'm, is kind of referencing back what I talked before about my friend Brent,

• 13:16Chris Benson?

  he can look at that and see a functional kind of what it will do after that. So it's, it's very, very practical AI that we're talking about here. We're talking about something that, that is something that, that the output is, it can be put in the hands of an expert who can immediately see in many cases where this is going and what the, what the effect will be. So super practical medicine we're talking about here.

• 13:42Daniel Whitenack?

  Yeah, definitely. And I guess to kind of bring home the importance of the methods that we're about to go into previously. I mean, it has been known that knowing these structures and the folding process is important. And so people have done experiments over time and you can find out the structures via experiment. I, I don't know all the details of that. Maybe we can find a link to share in our show notes, but experimentally, you can find these things out. But of course, anything that involves, you know, chemistry and biology experiment is going to be limited in terms of the pace and capacity that you can do as we've all learned in terms of lab testing, you know, COVID results and that sort of thing in recent years.

• 14:27Daniel Whitenack?

  So there's a limiting factor on that, which means that. Were you to be able to predict protein structures with a computer, which is maybe not, it still has a cost, right? In terms of computational cost and environmental cost and other things. But were you to do it, you're not, you're no longer constrained by your sort of experimental capacity. You're constrained maybe by your computational capacity and that sort of thing. And so the scaling mechanism is quite different.

• 14:59Chris Benson?

  I think into that point, I believe there was roughly, correct me if I'm, if I'm not remembering this accurately, but I think that the, that it was trained on roughly 150,000 known protein folds that had all been human determined. You know, this was before the AI was applied. So that was the baseline. And to talk about the, the leap that we're describing here, what was announced on July 28th, which was just a few days ago as we record this, was the fact that from that training set of 150,000, they went to 200 million, which describes nearly the entire universe of known folds. And I'm sure that there are more that they're going to continue to work on,

• 15:44Chris Benson?

  but kind of that's everything that we currently know for all practical purposes. So that's, you know, you're going from a fairly small subset to most everything in this one big release that we'll talk about with the database and everything. So I'm pretty excited about what comes next.

• 16:23Daniel Whitenack?

  Okay, well, let's maybe give. Just a little bit of context for AlphaFold and then talk about the database that they've released a little bit. So my understanding is that AlphaFold kind of, it first started getting notoriety because of these shared tasks that were really, like what I would think of in the AI world, a shared task, maybe they're called something different in the biology world, but there's these shared tasks within a certain community, critical assessment of techniques for protein structure prediction, or the CASP, I guess, assuming I'm saying that correctly,

• 17:01Daniel Whitenack?

  CASP. And they had these over time, you know, over the years, and CASP 14 was one of those shared tasks where AlphaFold really kind of stood out from the rest of the pack in terms of what it was providing and really showed the ability to very closely replicate the accuracy that you could achieve via experiment with predicting these structures, right? Because experiment in and of itself also has error related to it, right? So when you do an experiment to get these structures, you also don't get like 100% accuracy. There's error bars and all

• 17:43Daniel Whitenack?

  of those things. And so what they were showing, which is quite extraordinary, is that this AlphaFold thing, which we'll talk about more and get into the weeds of, is able to take these sequences and a sort of database of sequences in, and output structures that are of the same kind of level of quality as experiment in many cases, which means, hey, well, now you have a sort of choice. You could run experiments, but if you're getting about the same, same accuracy out of the simulation, then that, that scales, like you were talking about, the scale that you can achieve with that is, is something wildly different.

• 18:20Chris Benson?

  Yeah, I think all of the outputs are obviously being from an AI model. They're all predictions. The accuracy of those predictions has proven to be something that is significant enough to where further research based on those outputs can, can proceed rather than a lot of Kind of going back and trying to figure out if, if the, if the output of the model is, is sufficient and in terms of accuracy to be able to base further research on it. So it's not just turning out a lot of outputs. It's also the fact that they're, they're very high quality. And those two features together are what's going to really propel things forward in the, in the larger biology and chemistry space here to drive medicine forward for all of us.

• 19:04Daniel Whitenack?

  The method that they're doing has created these predictions. And so it's really this bank of predictions that is part of this release that has been, you know, getting a lot of attention. We'll link to a blog post about the release in our show notes, but one of the things that I thought was really interesting, Chris, I don't know if you saw this, was there is a figure of like one circle, which was the experiment today. Like how many, how many structures do we have in our database of experiments? And then the database, when it was originally released, because they originally released the AlphaFold database with about a million structures. And then they have kind of the circle of AlphaFold database today. And the scale just

• 19:47Daniel Whitenack?

  sort of like for our listeners who aren't seeing this in front of them right now, it's like one big circle, which is the database today. And experiment is sort of like a little dot within that in terms of what it represents. Cause experimental structures in a database, one of these I understand is called PDB, has about 190K structures. And Chris, that's what you're saying, these sort of supervised examples that they used in training. And then AlphaFold today, the database has 200 million plus. So that's, that's pretty crazy. They also give these, these circles representing how much is from different places. And you've got kind of a circle for

• 20:28Daniel Whitenack?

  animals and plants and bacteria and fungi and other animals is the biggest category. But then you have plants, bacteria, fungi and other things. So it's pretty interesting, both the diversity and the size of this, I would say. And again, I am near the field, but my understanding in terms of what's offered here is actually, you know, 3D structures. So you can look up, AlphaFold itself is open sourced. So the inference pipeline is open sourced. As far as I know, that the training pipeline isn't, but the inference pipeline is open sourced and you can look. Kind of in 3D at the structures that are coming out. So it's like 3D Cartesian coordinates that are coming out. You put this sequence of amino acids in,

• 21:12Daniel Whitenack?

  you get this 3D Cartesian coordinates out, which are really just this 3D structure representing the structure, 3D structure of the proteins.

• 21:22Chris Benson?

  Yeah, you know, as a data set, the ability to do that and then combine with with previous technologies, you know, so if you if you go back a few years and you talk about how big it was to release the human genome.

• 21:35Daniel Whitenack?

  Yeah.

• 21:35Chris Benson?

  And that provides a different set of capabilities, you know, in terms of understanding, you know, what what our genetic predispositions are and all sorts of different use cases. But now with the protein folding, to be able to, you know, to maybe start with the genome and understand what's likely to happen and what your predispositions are. And then you can go use protein folding from this database and be able to solve for some of those issues is is pretty remarkable.

• 22:03Daniel Whitenack?

  Yeah, I think also it's like, like when you think of the scale 200 million, one of the other things that I comes to my mind, and I'm sure people are exploring this and I'm, you know, our listeners, please share links with us in our Slack or Twitter or LinkedIn or wherever of studies that you know about. But you have this now this data set of 200 million. I'm thinking like, oh, what does it look like to do clustering sort of techniques on top of that? Can you learn about the sort of structures now that like all of the proteins are kind of mapped to these 3D structures? What can you learn at a more aggregate level about like clusters of of folding patterns or

• 22:46Daniel Whitenack?

  structures? What can you kind of post process this data set into and maybe build models off of these 3D structures? We all know that like the graph neural networks now are, you know, are a huge thing that's that's coming up and people are exploring that more and more. So obviously, these are 3D sort of spatial graphs. And it would be interesting to know what are people doing with these structures on the on the back end after their after their form? I think that's an interesting direction to to study as well.

• 23:19Chris Benson?

  Yeah, I'm looking at these same, these same documents that you are, and I can't help but think about the fact that is hopefully this is unleashing this revolution in this type of research. And you talk about that. I'm wondering how many high school and college kids today who have an interest that cross over might, might leap into this. I think, I think this is a moment we're going to remember, just like the release of the human genome was.

• 23:44Daniel Whitenack?

  Yeah, yeah. And they already, I mean, they already talk about the impact that AlphaFold is having even just a couple of months after this sort of release. I see here that after they open sourced AlphaFold and the database, it's already been cited more than 4,000 times in academic research. And there's, you know, things related to here, they do AlphaFold predictions referenced in publications. You know, there's a large complex that acts as a gateway in and out of the cell nucleus. There's from something having to do with malaria, which is a protein for including in vaccines.

• 24:26Daniel Whitenack?

  There's something having to do with the rate of mRNA degradation, which I think a wider audience is now more familiar with mRNA after all of the vaccine stuff.

• 24:39Chris Benson?

  For COVID, yes.

• 24:40Daniel Whitenack?

  Yeah, yeah. There is something having to do with causing frost damaged plants, which is obviously an agricultural thing. So even outside of medicine, you could think about agriculture and other things.

• 24:53Chris Benson?

  That's a really good point you're making because I think we're, we're focused in our conversation very much on medicine. But agriculture, food supplies, there are so many different areas. Pretty much, you know, everything in life, not just us walking around are impacted by this. And so.

• 25:10Daniel Whitenack?

  And I know with your interest, Chris, I had noticed this one too about something involved in the immune system of egg laying animals, including honeybees. And of course, we, you know, you're probably even more familiar than I am with sort of how, how honeybees and, you know, bee populations are in. Decline and having crisis.

• 25:30Chris Benson?

  It's a huge, yeah, it's a huge crisis that we're in.

• 25:33Daniel Whitenack?

  Yeah. So who, who knows how this could, uh, could impact many of those things. Well, maybe we could jump now a little bit and start talking about how does AlphaFold do this? So I think that we've established, hey, it's caught the attention of many people because it does a really good job at this. They've open sourced the inference pipeline so people can use it, but what does, what does AlphaFold do? I mean, this is practical AI, so we could probably all learn, even if we're not all doing protein folding. Maybe there's elements of the way that they're processing this data that are useful in our, in our own creativity and our own problems.

• 26:12Daniel Whitenack?

  And I think it's interesting that in their processing pipeline, you see sort of a number of really interesting things popping up from other, other domains. So the transformer architecture pops up within this. There's a, what they're calling an EVO former, which we can get into why it's maybe EVO evolution related in terms of how, how it is also iterative. But there's this EVO former architecture. There's this element of like joint embeddings. And also there's in the training, they use sort of supervised and like semi-supervised methods.

• 26:57Daniel Whitenack?

  They also use these like BERT style, not in a pre-training way, but they use a BERT style masking in their training as well. Which all of those things, I think we talked about this on a similar episode. This sort of innovation is built off of a number of things that have just been sweeping across the whole AI world, including, you know, you're thinking about transformers, these joint embeddings, semi-supervised methods, mass language models. All of these elements kind of contribute somehow to how the data is processed in this pipeline.

• 27:31Chris Benson?

  Yeah, a few episodes back, we had a, we had quite a conversation about that. And the fact that, you know, as an analogy, if you think about these different approaches that you just, that you just enumerated and think of them almost as Legos and the creativity, then of scientists and researchers being able to say, well, I'm going to try this one. I'm going to try this one and then combine it with that one and, and maybe do it in a completely different domain. And then, and you're getting these interesting outputs. And I think, I think I was before this episode, I was kind of thinking about the fact that it's almost like about a year ago, we almost entered, I think, looking back, kind of a new era of AI. There was kind of the development of, of those models for a while, but now we're

• 28:15Chris Benson?

  seeing the, the mixing and matching of them and such. And I think that this is one of the, the outputs of that. And so, yeah, cool stuff.

• 28:22Daniel Whitenack?

  Yeah, definitely.

• 28:47Daniel Whitenack?

  Okay, Chris, so I think if I'm understanding this right, and you know, we've looked through a bunch of things here, even just you and I are learning about AlphaFold. But it seems like that the network or the architecture that's driving AlphaFold is, is kind of split up into a few different main components. The first of those kind of takes an input sequence and then develops two, two kind of encodings of, of that input sequence, one which is called a multiple sequence alignment and one which is a pair embedding or pair representation. So there's this first stage,

• 29:28Daniel Whitenack?

  which is input sequence to encoding, then, or encoding or embedding. And then there's a second stage, which takes that initial representation.

• 29:38Daniel Whitenack?

  Through a transformer-inspired architecture to develop a sort of hidden representation. And then those hidden representations are then fed into a last stage, which is a structure model, which outputs the actual kind of predicted Cartesian coordinates of the protein. So we've got kind of encoding this transformer-based architecture, which produces a different representation or embedding. And then we've got a structure module, which produces the Cartesian coordinates. And what's interesting, and one of the reasons why I think they've used some terms related to evolutionary

• 30:20Daniel Whitenack?

  algorithms, Evoformer and stuff, is this is actually an iterative piece of this. So those last two stages, kind of putting the representations through the transformer-based architecture, and then out the other end to generate the structure, those actually cycle. So they, at least in their paper, they say that they do that three times. So they kind of refine, they make an initial prediction of the structure, and then refine that by passing it back through the network. So that it kind of goes through this loop a few times, and then outputs a refined protein structure.

• 30:57Chris Benson?

  It kind of has a recurrent network aspect to it there in the in the diagrams that they show there.

• 31:03Daniel Whitenack?

  Yeah, exactly. There's this kind of looping that happens. And from what I was reading, it's, you know, using deep neural networks to predict protein structure in and of itself is not an innovation of this work. So people have tried this for quite a while. But I think that there's two kind of main pieces here that are really kind of set this apart. One of this is this Evoformer architecture, which is unique to what they've done. And the second is this kind of iterative process, which kind of helps the network learn across these representations and the predicted structure.

• 31:48Daniel Whitenack?

  Powerful way. So yeah, it's interesting in this first, we can kind of dive into a couple of these things. But the the first one, it kind of reminded me a lot of some NLP things in to some degree, because you've got this input sequence, which again is just the sequence of amino acids. And they generate two representations from this. So like, maybe people are more familiar with NLP. You might have a sequence of characters, right? And you might assign like a number to each of these characters. Because you have to represent text as numbers to a computer, because a computer knows how to calculate numbers. Right? So here they're in some ways doing a similar thing. They're taking this input sequence and they're representing it by numbers.

• 32:32Daniel Whitenack?

  But in two kind of really interesting ways, one which kind of tries to identify not identical, but other sequences that have been identified in living organisms. And it kind of creates this, what they're calling this multiple sequence alignment. So it's actually an alignment of this sequence with other sequences, a multi sequence alignment. And then they have this pair representation where they're actually trying to identify proteins that have a similar structure and construct an initial representation. That's kind of a pair representation of these two things, thinking that there's similar things maybe in the whole database

• 33:14Daniel Whitenack?

  that we've we've learned about and similar proteins. So maybe we can learn from those things. So the initial sequence goes in these two representations, the multiple sequence or alignment. And then this pair embedding. So one, which is kind of a matrix of sequences. And one, which is a pair representation of one sequence with another.

• 33:38Chris Benson?

  Let me ask you a question that's more from your NLP background than this. But do you think it would be fair to say going through that two step process? It's sort of like pursuing the probabilities iteratively as it goes and kind of constantly working on where it's more likely to be between having the multiple versions.

• 33:58Chris Benson?

  That it's producing in that intermediate step and then looking for other proteins that may have exhibited the same sequence and therefore you already have a sense of what that folding might look like.

• 34:09Daniel Whitenack?

  So in NLP, we leverage a lot of pre-training, which isn't leveraged here and to some degree learn like, hey, a language behaves in a certain way. So I can learn, kind of pre-train some things and learn some things that I can transfer in. I think the idea is slightly similar here in that I think what they're trying to say is, you know, proteins are different one from the other. But if you have similar sequences or similar templates of your protein, they're not going to be quite the same. But some fragments and structure is going to be conserved across them.

• 34:46Daniel Whitenack?

  So I think they're leveraging this existing database of knowledge and sort of these paired representations to kind of understand that, yeah, there's, there's, there's something unique about the single inference. But we also know a lot about other, you know, protein structure and nothing's completely sort of new. So they're likely to be, the, the contact between proteins or amino acids. Yeah, the contact between amino acids, if that's similar in this case to another case, it's likely that some of these fragments of structure will be preserved as well.

• 35:21Chris Benson?

  I gotta say, Dr. Whitenack, for someone who is not trained in this field, that is quite a good explanation.

• 35:27Daniel Whitenack?

  I'll let our listeners who have some type of chemistry and biology background, correct me in our Slack channel or something. But I am very thankful to, I should give a shout out actually to, uh, there's a series of, of blogs that I looked at from the Oxford Protein Informatics Group. So if you're listening out there, if we got any listeners from that group, thank you for your blog posts and your work in explaining many of these things because they're very useful. We'll make sure and link those in the, in the show notes as well. But yeah, you, you sort of got this representation, this initial representation. And then that, as we've learned, is useful basically everywhere, whether we're talking about.

• 36:08Daniel Whitenack?

  About images or text or whatever, these initial representations, the MSA or multiple sequence alignment, and then this pair embedding are passed through a transformer based architecture, which is this Evo former, which is a unique architecture. And you can read more about kind of some of their choices that they made with that architecture in their paper and nature. But it passes through this Evo former architecture, which exchanges information between the two representations, so between the multiple sequence alignment and the pair embedding, and then outputs a kind of updated representation of both the multi-sequence alignment and the pair embedding, the sort of hidden state of the model.

• 36:53Daniel Whitenack?

  And then that's what's passed into this third stage of the structure model, which takes those embeddings, takes that hidden representation, and then maps it to 3D coordinates, 3D Cartesian coordinates, which is the output structure. And then, like we say, there's a looping thing that goes along. So actually, this structure is fed back into the front end of the second step, the transformer step, and you do this a couple of times where, you know, after generating one's structure, it's passed back and that information is passed back to refine the structure.

• 37:32Chris Benson?

  I'm curious, and I'm going to throw you another tough question to you, and it's fine to say too far, Chris.

• 37:40Daniel Whitenack?

  Yeah.

• 37:40Chris Benson?

  As you looked at the Evo former and kind of how it's approaching, do you have any thoughts on, as we're talking about this era of using these different components in different ways and combining them and going across domain, any thoughts on what an Evo former might be used for in other contexts? Do you have any, and I know that's getting out there a bit.

• 38:05Daniel Whitenack?

  It's a, yeah, it's a very interesting question. I do, I do wonder, like, one sort of random idea, and, you know, this is a random idea that I haven't thought about until this moment, so it's probably not. Not, there's probably flaws in it, but I wonder if certain things like this could be used for, you know, multilingual models and that sort of thing, because you're taking these sort of multi-sequence alignments, which are sequences of different proteins, right? And I wonder, and they're, they're kind of labeled accordingly. I wonder if you could have this sort of multi-language alignment between different languages and then, you know, factor that in.

• 38:48Daniel Whitenack?

  I don't know. That's a, that's a random thought, but I definitely think that this sort of idea that you would take a single input and represent it in two initial representations that have a slightly different character and represent different things about kind of your problem space. And then combining the information of both of those representations in the transformer, that could be applied in a number of different ways, you know, whether it's text input or image input, you could represent that in a couple of different ways that are useful and then mix those representations in this sort of Evo former type architecture. So I'm sure that even after AlphaFold, some of those 4,000 citations do a much better job at

• 39:32Daniel Whitenack?

  postulating possibilities than myself. So maybe- That wasn't too bad for off the cuff. Yeah, maybe one, one homework assignment for all of us would be to, uh, look at semantic scholar or something and look at the 4,000 citations and see which, which are the ones popping up that are related to reuse of the Evo former architecture. I'm sure there's a few things that have already come out. I think it is interesting that, we can just say something briefly, maybe about the training of this before we close out, because I think that is an interesting bit of this. We are practical AI after all, and I think we can learn, maybe learn a little bit from the general training structure that they set up for AlphaFold.

• 40:16Daniel Whitenack?

  And that is that they have this initial set of supervised examples from this P, I was gonna say PBR, but that, that's definitely not the right domain. What is it? Uh, PB something? The protein DB, PDB, that's it. So PDB, not Pap's Blue Ribbon, but PDB is this like 175, 190, whatever it was, set of existing protein structures, right? So they have supervised examples, but what they did was actually train, sort of they trained the AlphaFold architecture on these supervised examples,

• 40:57Daniel Whitenack?

  and then use the trained model to generate the new structure of sort of like a bunch of different guesses that they had. And for the high confidence ones, they took, they took 350,000 of those generated samples and combined them back in with the supervised, the gold standard samples to create this mixed data set, which they then retrained AlphaFold on. And so you have this mix of like supervised learning with what, what they're calling this noisy student self-distillation, which is basically this process of, hey, I'm going to use my model to generate new things and then I'm going to add them back into,

• 41:41Daniel Whitenack?

  I'm going to add the high confidence ones back into my data set, which is a really interesting, I think, structure that a lot of people could use. And you know, you don't have to be using AlphaFold to use that idea, right? You can do that when you need to augment your data set somehow. And so I think that that, that's maybe another learning to be taken away here that they're using some creative elements in the training as well, which help, help them kind of boost the performance.

• 42:11Chris Benson?

  So as, as we wind up, I'd like to challenge, we have so many practitioners in our audience. I would love to hear about some of the novel ways that they're taking these techniques and using them across other domains and combining them as that, that has really been fascinating in recent months to see some of the creativity in the space across different types of use cases. So I'm looking forward to hear what people are doing with Evoformers and some of the other combinations that are present in the architecture here to do completely new things that particularly those things that benefit the world at large.

• 42:53Daniel Whitenack?

  Yeah, yeah, definitely excited to hear about that. I've kind of already mentioned some learning resources for people. And we have a bunch of links we'll add into our show notes that people can explore. But if you're looking for something to start with, DeepMind does have a really good brief explainer video about protein folding and AlphaFold and how that fits together. So we'll include that. That's a really good starting point. And if that sparks your curiosity, they actually do have published a collab version of the inference pipeline. So you can actually spin up Google Collab and try to predict some structures yourself. I think that would be maybe the best way to learn about this is just to try it.

• 43:36Daniel Whitenack?

  So we'll link the GitHub to AlphaFold and then yeah, you can try that collab on your own.

• 43:43Chris Benson?

  Okay. Well,

• 43:44Daniel Whitenack?

  Awesome.

• 43:45Chris Benson?

  I'll finish with this. You might share, I started with the idea that there's a lot of reason to be optimistic about the world and the future.

• 43:53Daniel Whitenack?

  Yeah.

• 43:54Chris Benson?

  Despite the fact that there are plenty of things to bring us down. If you've enjoyed this episode, you might go share some of this with the people in your life, whether they're into AI or not, just because it's worth knowing, it's worth knowing that the world is still moving forward in a really positive way, even when other things are a bit challenging. So share this with people who you might not otherwise think about.

• 44:17Daniel Whitenack?

  For sure.

• 44:17Chris Benson?

  And that'll be it. I'll talk to you next week, Daniel.

• 44:20Daniel Whitenack?

  Been good to chat, Chris. See you soon.

• 44:31Speaker 01

  All right, that is our show for this week. If you dig it, don't forget to subscribe. Head to practicalai.fm for all the ways. And if Practical AI has benefited your life, pay it forward by sharing the show with a friend or a colleague. Word of mouth is the number one way people find shows like ours. Thanks again to Fastly for fronting our static assets, to fly.io for backing our dynamic requests, to Breakmaster Cylinder for the beats, and to you for listening. We appreciate you. That's all for now. We'll talk to you again on the next one.