AI's role in reprogramming immunity

Drausin Wulsin, Director of ML at Immunai https://www.immunai.com, joins Daniel & Chris to talk about the role of AI in immunotherapy, and why it is proving to be the foremost approach in fighting cancer, autoimmune disease, and infectious diseases. The large amount of high dimensional biological data that is available today, combined with advanced machine learning techniques, creates unique opportunities to push the boundaries of what is possible in biology. To that end, Immunai has built the largest immune database called AMICA that contains tens of millions of cells. The company uses cutting-edge transfer learning techniques to transfer knowledge across different cell types, studies, and even species. Featuring: Drausin Wulsin – LinkedIn https://www.linkedin.com/in/drausin-wulsin-3a3a8239Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Immunai https://www.immunai.comImmunai Fact Sheet https://drive.google.com/file/d/1qVzRs0yf_Mfbh5p85sYGO8pwNxFjGQ7T/viewDrausin Wulsin | LinkedIn https://www.linkedin.com/in/drausin-wulsin-3a3a8239Drausin Wulsin | Google Scholar https://scholar.google.com/citations?user=_PXVUn4AAAAJUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  One of the things that is really challenging about using AI in this biospace is there is so much that you could do because now there is just so much data that like the space of possible things that we could do is just ginormous. And so you have to think about what is the thing you really care about. And even once you've identified the question or the problem, a highly trained immunologist can't look at a 20,000 dimensional sparse vector and tell you what kind of cell type it is. We can all look at a picture of a cat and a picture of a dog to tell, did the algorithm do a good job or a bad job? And so I think in sort of the bio AI combination, you're sort of in the hardest of both of these regimes and that the data is hard to understand

• 0:43Speaker 00

  because also biology is super messy and the problem space is so large and unbounded that it's really easy to sort of get lost in the woods.

• 1:02Speaker 01

  Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join us at practicalai.fm/community and follow the show on Twitter. We're at Practical AI FM. Thank you to our partners at Fastly for shipping our pods super fast all around the world. Check them out at fastly.com.

• 1:35Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I'm joined as always by my co-host Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

• 1:50Chris Benson?

  I'm doing very well. We are now, we've just come across summer solstice. We're into summer. We got hot weather all over the United States and so I'm just figuring we need some hot AI topics to go with that.

• 2:03Daniel Whitenack?

  Some hot AI topics.

• 2:04Chris Benson?

  If we're burning up 24 hours a day, we gotta burn up with some AI too here.

• 2:10Daniel Whitenack?

  Good one, good one. I don't know many things that are maybe hotter than sort of AI applied within healthcare, or within pharma, or within genomics. I just saw, actually today, I saw a tweet of someone who just released the first open source version of AlphaFold, the protein folding thing, and that's pretty cool. And so, yeah, we're really privileged this week to have someone who's an expert in the field of AI as applied to immunotherapy and genomics. We have with us Drazen Wolson, who is the Director of Machine Learning at Immunai. Welcome, Drazen.

• 2:54Speaker 00

  Thank you. So good to be here. Thanks for having me.

• 2:56Daniel Whitenack?

  Yeah, for sure. Well, maybe before we sort of jump into your specific work and some of the things that you're doing at Immunai, maybe it would be good, from an expert in the field, to just hear a little bit about like, how have you seen AI sort of creeping its way into immunotherapy, or maybe applications to like genomics, or this sort of field? How have you seen that history progress, and where are we at right now?

• 3:30Speaker 00

  Yeah, so it's actually sort of crazy to think about it. So it's my first introduction of AI and genomics was about a decade ago, when I was in graduate school, and I was studying statistical methods for understanding epilepsy. So something very different than immunotherapy, but I saw a bunch of papers starting to explore microarray data. What microarrays are, is basically you want to run an experiment on a 96-well plate, often. And so you get to profile 96 genes, and all of a sudden, we were able to measure 96 genes in a single experiment, in a single little well. And the AI people were going crazy

• 4:13Speaker 00

  about this, because it was this brand-new bio, you know, big-ish, now we would call it like small data, but back then it was big data. This is sort of maybe... Two thousand and ten or so, and people were using this really rich data to start to, both to test new, you know, statistical algorithms and M L models, and also to try to understand, you know, associations between genes, which, which finally we were able to measure enough genes and proteins that we could actually sort of start to peel these back. And I just always thought it was very, so I was like, kind of wish that I, I could like, get into that area of like getting to microarrays. And it's been that I sort of, my, my path took me elsewhere.

• 4:56Speaker 00

  And, um, and then, you know, fast forward eight or 10 years, and all of a sudden, no one uses microarrays anymore because they're, you know, you only get 96 genes. And, you know, meanwhile, now we can get, you know, 20,000, you know, all 20,000 genes on the human genome. And so I, I think the thing that's, and the sort of high level story of A I helping understand biology over the last decade is this sort of intertwined story between algorithms getting better and more capable. And also are critically our experimental techniques getting much better. And I mean, actually, I would say the experimental techniques, you know, might even be getting better faster than the algorithms.

• 5:37Speaker 00

  Because we're just able to profile so many more cells and genes and aspects of biology. And here I'm just talking about one small area of the larger biological understanding landscape. And like, we can talk a little, a little bit more about others. But I, I think that sort of, uh, this is the exciting piece about our field is this sort of racetrack of, of both, both the experimental techniques and also the computational techniques are like racing against each other.

• 6:07Chris Benson?

  You know, it's interesting that you say that. And we've heard that from other people in other fields. But, you know, just the acceleration you get from kind of like, this is a whole new toolset, you know, we've, you know, we've been doing this for a while in, in this, but if you look at the scheme of these fields that it's being applied to, it's a getting new tools and figuring out how to use them effectively. I find it interesting because we love talking about how fast the A I field is, is evolving. And you're actually saying that you're getting a bigger impact just from the fact that you're, you're learning how to best use those tools, which I think is a great lesson for folks.

• 6:41Speaker 00

  And so in the last decade, let's say, just sticking with this area that I know a little bit better, from microarrays to where we are now. We've seen basically three revolutions in profiling of individual cells and biology. We went from microarrays to something called bulk RNA sequencing, where you can take a bunch of cells and understand what they're doing across all the genes in the human genome. And then from bulk, maybe five years ago, we started the single cell revolution. And there, instead of looking at what's the average of 1,000 cells or 10,000 cells, all of a sudden, we can look at what's going on in an individual cell, each individual cell.

• 7:25Speaker 00

  And that is tremendously exciting. And there's a lot more that I could go into if we want to go down this route of the single cell profiling opportunities we have. But this is, as I said, just one area of biology that has been revolutionized by just the data capture that we've been able to perform in the last decade. And it's the data that's the thing that's driving it that I'm most excited about. Sort of like, what does the next decade look like? Yes, the algorithms will be right there along, but the data is going to be the leading pack. And that's the thing that's most exciting.

• 8:05Daniel Whitenack?

  So on that front, I love how you brought up this side of the data-centric side of what you're doing. I'm wondering, like, you brought up, okay, we can go down to the single cell level. We can get a bunch of data about this single cell. I'm wondering if you could describe, like, for us who don't have this experience with biology and such, like, what is measured or what does the data look like for a single cell? And like, why is that data important? Why is that connected to anything we would want to care about?

• 8:41Chris Benson?

  And if I can extend that to one other dimension, what's the difference in the single cell versus the bulk in terms of what you're getting out of it as well?

• 8:48Speaker 00

  Oh, oh, man, I could tell you I could spend a whole chat just on this.

• 8:54Daniel Whitenack?

  You can tell you piqued our interest.

• 8:56Speaker 00

  Yeah, okay. Let's first cover just what is the data? And then we'll get to the what's the difference between individual cells versus what's sometimes called bulk, which is the average of, let's say, many, many, many cells. So what is the data? You have to actually recall your high school biology, which is how does a cell work, right? Okay, so, so, so let's, let's.

• 9:16Daniel Whitenack?

  I remember the walls and.

• 9:19Speaker 00

  Exactly.

• 9:21Daniel Whitenack?

  Different ways, yeah.

• 9:22Speaker 00

  It's, this is why it's wonderful is like, actually, all of this is relevant. So, you know, you've got DNA and the DNA is like your master set of instructions for a cell. But you don't crack the master set of instructions every time, right? So, so normally when a cell needs to do its thing, it just needs a page here, a page there, needs to make a bunch of copies from your master set of instructions. Okay. So you can think about, you know, there's a process that's making a bunch of individual copies of individual pages and then it's giving those copies of, okay, you know, here's how to make this protein or here's how to make that protein to some factories. These factories are called ribosomes. And the ribosomes are the things that actually take the copies of instructions for individual proteins and actually like make the proteins. So with that, so just covering

• 10:05Speaker 00

  that, there are three things you can measure right there. You can measure one, what pages are you copying in the master set of instructions or like what, you know, think about it like, you know, what page is the, is the master book open to? So that's, that, that you're, you know, about to make some copies on. Imagine that you could then say like, okay, what are all the loose, you know, loose pieces of paper floating around that, that are encoding proteins that we care about? And that's going to tell you something about what a cell is doing, but basically like what, what instructions is it passing around to its factories? And then you can also observe, okay, you know, what's the, what's the product of these ribosome factories, the proteins, like what's coming out? How can we measure how many of this kind of protein or that kind of protein? And so you can measure all of these and they're

• 10:49Speaker 00

  all complementary. So, you know, you can measure some proteins a little bit better, but you're, you're, you know, not every RNA molecule. So basically those, these, the middle thing, which are these individual pages that you're sort of copying from your master book, this is what's called messenger RNA. And messenger RNA are these little pieces of instruction that tells ribosomes, like, here's how to make this protein. They code for a specific protein. And so messenger RNA is, so you can imagine reaching into a cell and grabbing a bunch of random, you know, like whatever pages are on the floor, so to speak, right? And like sweeping those up and then like tabulating them and counting them and saying, okay, this is a page for this protein. This is a page for this protein.

• 11:32Speaker 00

  And, and you, and that's one way to profile what a cell is doing, is by sort of counting the number of mRNA molecules. And literally with some of these techniques, we are counting individual molecules of mRNA. And that allows us to sort of, if we can see like, what is the cell, what things is the cell doing? It gives us a sense of what's the function of the cell, where has it been, where is it going? But each of these data, sometimes we call them modalities, it has its biases and its problems and its weaknesses. And, and the idea of a lot of modern genomics profiling is sort of capturing multiple of these modalities.

• 12:12Speaker 00

  So, Amy and I, where I work, we actually have a, a technique that sort of can capture all three at the same time, and they sort of complement each other well.

• 12:21Chris Benson?

  Can you describe a little bit about what you're, since you had started off kind of enumerating the three, and we've kind of, as we've kind of gone through, can you tell us what you're getting? Like, why would you do all three at the same time? Why isn't the most recent one the best then? You know, like, what are you getting from each one?

• 12:37Speaker 00

  So each has its trade-offs and benefits, right? So, so let me, let's take the simpler example of what's called RNASeq, which is basically these like individual copies of RNA. So in this, you can actually count up the molecules of RNA, and you can see, you, you can basically get an expression profile for every gene in the human genome. Twenty, usually about 20,000 genes actually are, have, have activity here. And so you have amazing scope in covering the entire human genome in a, you know, in a single readout. But the trade-off for that is that 90% of the sort of counts, basically what you get

• 13:21Speaker 00

  here is, you know, for each, now I'm talking about, let's say, the single cell, you get for each individual cell, you get a count. How many molecules of RNA for a particular gene did I observe? And the problem with RNA-seq, at least single-cell RNA-seq, is that ninety percent of the genes have zero counts. Because you just, you just can't observe all of, all of these molecules of RNA. And so, you know, you got a lot of zeros you gotta contend with. It's a very sparse readout. So let's contrast that with proteins. So proteins, which basically can exist both within the cell, but also, but especially on the surface of the cell, are what traditionally, in sort of the last fifty years,

• 14:04Speaker 00

  biologists have used these proteins to characterize and identify biology, certain cell types. So in the modern era, we can, I can say, choose a hundred proteins to profile, maybe two hundred. And I have, I can, I sort of choose them carefully and, you know, often I choose them so they capture all the biology that I would want to, to, to measure. And for each of these, I get a pretty, you know, high-quality readout. You know, the, the data is high-quality, but it doesn't cover all the possible proteins. I just have to, like, place my bet. And so sometimes what we'll do is we'll say, okay, I want to use the proteins to identify, let's say, cell types, or to identify which cells are dead or alive, or, you know, there are many things you can do.

• 14:48Speaker 00

  And then I'm gonna use the RNA to identify what's going on, like, under the hood in the cell. And that, that may not be, I may not be able to as clearly see with all the proteins. But sometimes I can say, okay, I really don't care about all the stuff that's going under the hood. I just wanna know, you know, is the cell really energized to go kill some tumors? Or is it feeling exhausted, you know, from, you know, just coming off the field of killing a bunch of tumors? And, you know, sometimes there are a couple of individual surface proteins that, you know, that we know about that will indicate the sort of state of the cell. So it's sort of like you have this menu of things you can read out from a cell. And you have to be very thoughtful about choosing, okay, what are you gonna read out here or there? And there are lots of other very interesting trade-offs

• 15:33Speaker 00

  because, you know, each of these modalities has a cost. It has a cost in money, it has a cost in human effort, it has a cost in time. So I could say, okay, would I rather have, you know, a million cells of this sort of single cell RNA-seq data, or, you know, 200 proteins of worth of data for each individual cell, or would I rather have five million cells with just 10 proteins, right? And depending on what I'm trying to do, I may say, but and that five million cells with 10 proteins, I can get that data tomorrow, right after I did the experiment in the lab. Whereas the stuff where I have all the RNA molecules, or I just have, you know, I have, you know,

• 16:15Speaker 00

  200 surface proteins, that maybe I have to wait a couple of weeks because it has to go get sequenced and stuff like that. So there's, I think, one of the things that's really hard, but also fun and interesting about biology is and biological data, is that there are so many options. You have to think about trade offs a lot more, I think, than if you're just like, working with vision or text or some other sort of like, modality that, or some of those like, type of data that many of us in the AI world are used to thinking about.

• 17:10Daniel Whitenack?

  So we've talked a bit about cells, we've talked about genes, we've talked about this sort of like, measurements within a single cell or bulk. I'm wondering if you could kind of connect this to what I'm learning about on your website, which is more sort of related to immune profiling or immunotherapy. How do the cells and what we know about the cells connect to kind of immunotherapy? And what exactly does immunotherapy mean, maybe, for like, some people that are new to that?

• 17:48Speaker 00

  Great. No, no, it's a, it's a great question. I'll be honest, before I started working here, like, I had no idea what immunotherapy, or I only sort of knew at the very highest level. So I can walk you through sort of my own learning process as well. So, but first, let's even before we get to immunotherapy, let's talk about what is the immune system? Okay, so the immune system is, think of it as like a combination of the, the security guard force, it's the, it's the police force, and it's the army, and it's the air force.

• 18:16Daniel Whitenack?

  Speaking in terms Chris can understand.

• 18:18Chris Benson?

  Yeah, I work in the defense industry. I'm, I'm, I'm all on board with this. Keep going, I'm sorry to interrupt.

• 18:23Speaker 00

  So your immune system is the defense industry for your body, right? And it is insanely good at its job because 99.99% of bad things that happen in your body are crushed, right? The immune system dispatches them no problem. And like, this is the product of, you know, hundreds of millions of years of evolution and, you know, lots of predecessors of ours, like, you know, dying in order to naturally select for this beautiful thing that is the immune system. It is actually like a defense force where you have different specialized players that are, that are good at different things.

• 19:03Speaker 00

  You know, you're not going to send, you know, a SEAL team to monitor your apartment building. You have a security guard for that, and, and the security guard is maybe even better than a SEAL team in some cases for various reasons. And so, the immune system has evolved to have all these really good, different players that work together really well to sort of crush viruses and bacteria when they sort of come into your, into your body, and also to crush things that like, that you're, that happen inside your body. So the, the two biggest problems that happen in, that the immune system is, is balancing between are cancer and autoimmune issues.

• 19:43Speaker 00

  So cancer is when basically the, there's some mutation in your body that is like a runaway train. All of a sudden, some, some of your own cells start replicating and, and all of the brakes and, and emergency brakes that usually keep this from happening have, you know, have broken. And so there's this sort of like out of control growth that's happening. Almost all cancer, again, ninety nine point nine nine percent of cancers like are dispatched summarily by your immune system. Your immune system sees what's going on, it goes and it kills those cells. You'll never know. At the same time, in autoimmune issues, what's happening here is the immune system is is sort of like going overboard, right? It thinks there's

• 20:25Speaker 00

  a problem, but there's actually not a problem. So it's just attacking sort of like civilians, like healthy cells. And this is this is also like very like a problematic. So I'd say the immune system is this wonderful ballot, like a checks and balances. And there are lots of cell types that sort of like work and signal to each other to like keep each other in control. So, okay, so what is immunotherapy? So immunotherapy is this wonderful sort of revolution really in oncology treatment that's really taken place blossomed in the last decade, where we basically have realized that with just some very minor coaching, we can get the immune system to be way more potent at killing some types of cancer. And

• 21:06Speaker 00

  the coaching that I'm talking about is basically just binding, you know, one little antibody, one little kind of antibody to a certain type of immune cell. And when you do this, all of a sudden, the immune cell is it sort of has like a force field on, right? And the cancer cells can't sort of turn it off in the way that that they've evolved to be able to do. And so immunotherapy is really the art of coaching the immune system to be better than it already is. It's not anything, you know, it's sort of using an existing tool just better and sort of, you know, giving it the pep talk that it needs to go and fight the cancer.

• 21:45Chris Benson?

  So I'm just curious, and I don't know if I'm going to ask this the right way, but, you know, by putting in that antibody on that, you know, which is essentially plugging it in right there, that's the light switch that the cancer cell would be flipping, you know, on or off to to have its impact. And you're just kind of taking that away. You're using your force field, or you're putting the kid's safety cover over the light switch, so to speak.

• 22:09Speaker 00

  Exactly. I like to think about it's like it's like putting that piece of tape over the light switch.

• 22:12Chris Benson?

  There you go.

• 22:14Speaker 00

  So in the sort of technical term here in the immunotherapy literature is called immune checkpoint blockade.

• 22:19Chris Benson?

  Okay.

• 22:20Speaker 00

  And so what you're basically doing is sort of Putting that tape over the light switch that the tumor has evolved to be able to like turn off the your T cell. Your T cells are your main fighters. They're like your Marines or your, you know, your SEAL team. They're they're like super elite and they go and they're killers. There are a lot of other immune cells that are involved that, but you know, your Marines, they need an off switch, right? You don't want them running wild, you know, throughout your body. So that's why there is this off switch and tumors evolve because in their ability to like, you know, sneak by and actually sort of like turn the off switch off when the T cell doesn't realize what's going on. It's kind of crazy that it really immunotherapy, they're in the the first five years from let's say 2010 to 2015,

• 23:04Speaker 00

  they're really just a couple of targets. PD1 is one and CTLA4 is another. And just between the two of these, like, I can't tell you how many lives have been have been saved because of the cancer treatments that just these single two targets have enabled.

• 23:20Daniel Whitenack?

  I'm kind of thinking about like in my mind over you while you've been talking about like what these things are, what the data looks like. I've been thinking of in the back of my mind about kind of how I would expect AI to fit into this. And, you know, back or when I teach like a workshop or something, often like a common question is like, when should you apply AI to a problem and when shouldn't you? And I often put that in terms of like scale and complexity. So like, it's very easy for a human to identify a cat in an image. It's very hard for a human to identify a cat in a million images, right? Just time wise, even though that's a simple task for a human. But there's other

• 24:04Daniel Whitenack?

  things like, you know, maybe it's certain time series forecasting, or maybe it's like things where the the data is very complex, language related things or something like that where it's very hard, like the the data, the problem is very complex for a human to even make a single inference, right? And so that doesn't even necessarily require scale. And here you've kind of talked about like there's this, it seems like there's complexity around the data on one side in all of these different things that you can measure selectively against single cells or bulk cells.

• 24:41Daniel Whitenack?

  But then there's like a whole complexity on the other side of like all the different ways the immune system works and its different, you know, targets within the immune system. And I'm wondering if if that's a good way to represent it or like how you would kind of like make the case that like, or for the place, I guess, that AI fits within this problem set.

• 25:03Speaker 00

  So one of the things that is that is really challenging about using AI in this biospace is there is so much that you could do because now there is just so much data that like the space of possible things that we could do is just ginormous. And so you have to think about what is the thing you really care about? What is the what is the question or the problem that really matters to you? And like, go solve that. And even once you've identified the question or the problem, as you said, Daniel, like an immune, a highly trained immunologist can't look at a 20,000 dimensional sparse vector and tell you what kind of cell type it is, you know? They can look at it, you know, we can all look at a picture of a cat and a picture of a dog

• 25:47Speaker 00

  to tell or a face, you know, bounding box and see what, you know, did the algorithm did it do a good job or a bad job? And so I think in in sort of the bio AI combination, you have, you're sort of in the hardest of both of these regimes and that the data is hard to understand and hard to sort of know what the ground truth is sometimes because also biology is super messy. And the problem space is so large and unbounded that it's really easy to sort of get lost in the woods.

• 26:16Daniel Whitenack?

  Yeah, and I know in the in your website and some of the things I've read about, some of the things you're involved with, some of that involves like this sort of like transfer learning. I know like some some parallels were drawn between like large language models, things going on in the NLP space. How do you sort of like chisel down to, like you said, there's so many things you could do. Immuni is specifically interested in immunotherapy. How do you go about like saying, okay, here's like the space of what we're interested in? Here's this really complex. Data. Here's a bunch of things that are going on in the AI industry more broadly to other types of problems.

• 26:58Daniel Whitenack?

  How did you kind of come into the place where you understood? How to connect certain of these things, whether it's transformers or whatever, to the specific problems that you're thinking about? And maybe you could give us an example of one of those problems.

• 27:13Speaker 00

  Absolutely. So actually, sort of, this was one of the founding insights that Lewis and Noam, our co-founders, identified is that there's a lot of data being generated and a lot of problems being posed in the sort of immune system space. But none of those data sets and none of those problems are benefiting from other people posing really similar problems on using really similar data sets, right? Each person is sort of doing in their own narrow little lane. And I think one of the things that, especially with transformers and this idea of a foundation model, but it is sort of in the earlier part of, even back in 2010, 2013, people were talking about

• 27:57Speaker 00

  semi-supervised learning, where you train, you know, an unsupervised model and then you fine tune it for specific tasks. And so what our founders realized is that basically there is this opportunity to do this transfer across tasks. And, and there's lots of ways that, that, and actually like, this is essential in our world because there are, because of the multiplicity of data modalities and readouts and different experimental conditions and contexts, right? So, again, in vision and text, I like to, I mean, I'm sure the people who do vision and text would laugh at me, but like, I like to say that they have it easy, right?

• 28:39Speaker 00

  Because, you know, like, you know, all the images are, you know, you got, you know, they're all RGB and maybe they have some different, you know, focus or out of focus or, you know, different sizes, but basically it's all the same. And text, you've got maybe some, some language differences, but fundamentally you can go and scrape all of Wikipedia and all of Reddit and get a pretty good data set from those two sources. And, and this is actually like what enabled a lot of the AI revolution of the last decade is just. Pulling data from the web. Whereas in bio, instead of having a couple of really high volume sources of data, you've got many tens or maybe even hundreds of smaller pieces of data. And so if you want to benefit from all of those smaller blocks of data,

• 29:24Speaker 00

  you have to have models that can, that are more like Legos, right? That you can sort of like learn an embedding from, you know, this data set over here and bring it to some new model that can benefit from, let's say, that embedding or that inference that you learned. And like, we have not, you know, Ed, Emi and I, I think we're, we're doing a lot of work on this and we're probably at the front of the pack, I would say, but we have not figured it out and no one else has. But, but I think this is the thing that's exciting over the next decade is like, we will, as a, as a community, we will figure out how to do this. And, and it's really helpful to, to be an AI person in the biospace, to be able to point to the successes that, let's say, in natural language understanding, we've been able to have using this transfer learning approach.

• 30:08Speaker 00

  Because, you know, before transformers or, you know, GPT, you know, one, two, three, you had 25 years of, you know, computational linguistics people, like building really finely crafted models and context-free grammars and things like this. And it, it actually took a certain amount of data, a certain scale of data and a certain class of algorithm to, to sort of trample all of that work. And, and I think we're sort of all, you know, just about there in the bio, bio world. But there's still a lot of, a lot of earlier things that I think are, are around. But an example, let me give you like a really specific example of a transfer learning task.

• 30:50Speaker 00

  So Ed, Emi and I, we're a single, you know, primarily a single cell company, which means that we take, let's say, you know, a tumor of yours or a vial of blood, we process it and then we're profiling each individual cell that we're, that we're, that we're getting. Okay? And when I do this, I want to know, you know, are, is this cell, you know, a SEAL Team Six cell? Or is this a, you know, apartment security guard cell? Right? Because it, it depends on how I think about, you know, what is the cell doing or not doing? And is this good or bad? So this is, you know, something called like a cell type annotation and usually involves an expert immunologist who understands what the all the immune cell types, profiles we should see are, but ultimately it can be boiled into a classification problem.

• 31:33Speaker 00

  But so we can do this classification problem for, let's say, cells that we would, immune cells we would see in the blood. But what about immune cells we would see in the bone marrow or a tumor? Those are, you know, they're related, but they're not exactly the same, different profiles. And so is this a separate problem? Or is this just a sort of very similar flavor of the blood profiling problem? And so there's this problem. So traditionally, people think of it as a completely separate problem, completely separate tools and approaches and atlases. But I think the reality is, it's just another instantiation of a similar problem. And then there are a bunch of other technical problems of like, how do I know whether this cell that I'm observing is one cell or two cells?

• 32:15Speaker 00

  Because sometimes the technical hardware, you can actually, there can be two cells that sort of get smooshed together and you can't, you can't resolve them. So this is actually an interesting, like, technical, very in the weeds technical problem that you have to solve if you're going to do with a single cell work. And this is also like a simple binary classification problem. And so, and so like when you think about all these individual, and here I'm talking about just single cell problems, but there are analogues for bulk and, you know, sample level. There are lots of very related problems that up until, let's say, the last year or two, people have been solving independently and, and now we're working to solve them together.

• 33:16Daniel Whitenack?

  I'm pretty intrigued by this, this whole idea of a sort of a generalist approach to applying this kind of pre-training and transfer in like multiple domains with multiple modalities of data, something I'm personally really fascinated by right now. I'm wondering if you could describe this sort of like, like how, how you might approach pre-training and self-supervision for, for like biological systems. Uh, cause I could think like in, in the natural language processing space, for example, I have like, like I know I have this text. I know this word goes here and I can just remove it and then I have a blank,

• 33:58Daniel Whitenack?

  right? Is that some of the inspiration for, for what you're doing in terms of pre-training? Or how do you think maybe about self-supervision and what might be like the more relevant things to think about in the biological side?

• 34:12Speaker 00

  So in many ways, the, uh, like the, the single cell, and again, I'm sort of, we're going deep on the single cell world because especially it's like what I know the best and it's what Amy and I, uh, focuses on. In the single cell world, the sort of you look at the trajectory of, of models and techniques, let's say over the last five years. Like the field was like literally didn't exist five years ago, or like it barely existed five years ago. Five years ago, people published papers on 200 cells. Now we're publishing on 2 million. So just like the scale of the data in the last five years has enabled a certain new flavors of models that just didn't exist, uh, we couldn't do before. But early on in the first couple of, you know, two, three years, people trained auto encoders, paralleling sort of like, uh, the earlier work in both in vision and in text. And so initially it's just, uh, you know, you know, train your auto encoders.

• 34:56Speaker 00

  Basically each cell is an observation and you're, you have, uh, you know, maybe you select the 500 genes or 5 million, uh, 5,000 genes that are the most variable or the most active for a cell, or maybe you do all 20,000. Depending on how much data you have and you run that through a bottleneck and where you're just trying to reconstruct the gene expression. And then you take that that middle bottleneck layer and you you do something with it. Maybe you you fine tune it for a specific task. Often people use it for data exploration, like in an unsupervised way, the maybe like visualization or clustering and things like this. So this is sort of where it started. It's just now in the last the last year starting to happen where people are like, huh. Now we have enough data, maybe we can like, and there's this like fancy transformer thing

• 35:41Speaker 00

  that I've been hearing so much about, like, maybe we can start building some of those. And in that, the task, you know, can vary a lot, but probably one of the most sort of analogous to the to the language world is just masking, right? So instead of masking words, as we do often in the in the language or parts of sentences or like the second sentence after the first sentence, you're masking individual genes. And you say, hey, model, like, I've masked this 15% or 25% or 50% of the genes. And I'm I'm gonna give you the other genes and I want you to tell me, I want you to reconstruct. So that's probably the the simplest formulation, but there are a lot of alternatives that you can do.

• 36:22Speaker 00

  And the cool thing now is that, you know, two years ago, if you wanted to build some, you know, big transformer or big foundation model, you sort of had BERT, right? And to as your as your as your template. But now, you know, the even just the transformer world is completely blown up. And so we have BERT and GPT-3 and the perceber and like lots of options to sort of choose from and customize, which is just, you know, we're not, at Emi and I, we're not, you know, leading the the edge on the the brand new transformer architecture. We're benefiting from other people doing this, like OpenAI and, you know, Facebook and Google and DeepMind. And we sort of get to be like, okay, yes, this is the one I think that that most benefits our application.

• 37:07Chris Benson?

  So I'm curious because you kind of downplayed it a little bit at the end, but these are pretty fascinating approaches, maybe because we're talking about it outside of some of the more common topics that you tend to have in the ML space in this way. And so, you know, but at the end of the day, you're still having to run a pipeline and there's all these kind of practical ML tasks that you're going to be engaging in. But with this interesting dynamic about the thing that you're addressing specifically, both similarities and differences. So what have you learned about running a practical machine learning pipeline in a company that's doing these kind of interesting techniques?

• 37:49Chris Benson?

  It's definitely a realm that most people are not thinking about machine learning in. And that has some of the uniqueness of that. I'm wondering if that's given you some insights that might benefit all of us.

• 38:01Speaker 00

  Well, I can say that, so in my own personal journey, I had a period of time where I did a lot of ML, then I got burned out of ML, and then I did no ML for like four years, just software and data engineering, building pipelines and getting data. You know, I was a data plumber and I loved it. Then I sort of missed the research and the ML stuff, and so I got back into it and here I am. But I think that the biggest thing that I've learned is that when you have a problem that you want to solve, you always solve it without ML first, right? So you solve it with getting data and you solve it by analyzing the data. And then, once you do this, then you solve it with something really simple, like a logistic regression or an XGBoost. Like for classification tasks, these two things probably cover 80% of all the things

• 38:45Speaker 00

  you would want to do reasonably. I think like it's really hard, even where at our current place, to sort of do the responsible thing sometimes. Because you know, you want to go train the fancy transformer, like play with lots of data, but... But the problem is you can just burn so much time, you know, unless you're one of these huge companies. It's so easy to burn so much time getting infrastructure, training infrastructure, GPU, you know, parallel GPU training, like up and running to support these big models. When often, you know, maybe you're solving the wrong problem. So solving the problem without models first, or just with data analysis. Actually, like when I joined EMI and I about two years ago, my boss, who's the CTO,

• 39:30Speaker 00

  he said, okay, Drossin, like, I need you to just understand cell type annotation. And so I didn't train a single model for six months. I just like deeply understood the data and analyzed it and like understood the problem domain. And like, it's actually really fun because I got to work with our immunologists. And, and this was, I just sort of didn't, I was sort of like grumpy about not getting to like, get in and train models right away. But in retrospect, it was a really fortuitous thing because now I just have a much richer understanding of the problem. And I think especially in bio, the problems are hard to define. And defining the problem well is the most important thing. And this comes with, with data and analysis before you have to do any, any ML.

• 40:12Daniel Whitenack?

  And along with that, what is the sort of interaction between maybe those with expertise in machine learning or in even software engineering and like expert biology doctors? Like this sort of like other side of things, like how have you found kind of good synergy between kind of the domain experts and like technical experts?

• 40:37Speaker 00

  It's challenging, even when everyone wants it to work. This is another big difference in the bio AI world is you just need a lot more different flavors of expert in order to get, make a therapy, to make something that's going to help people. And so getting the immunologists and the software engineers and the data engineers and the computational biologists and the machine learning people to all communicate effectively is is hard. And the only thing that works is there are two critical things. One, you have to find people who are interested in doing this, right? Find people who like getting out of their discipline areas and and not everyone is interested in doing this. And that's completely fine. But I think for in the in the bio world,

• 41:19Speaker 00

  you have to be interested in like, you know, what the immunologists are doing and like be be excited by those problems rather than just like wanting to make your ETL code, you know, even better and more efficient. So so like finding the right people and the right team is is critical. And also building the teams such that like I like to think about it, my my sort of a quantitative brain thinks about it like a overlapping Gaussian distributions, right? So so like each person or really think of it like a team, each specialty has, you know, a an area of, you know, a dense, a higher density area of specialization. But if those those areas, if those densities don't overlap, then you've got cracks, you've got holes where things get get lost, right?

• 42:02Speaker 00

  So what you need is basically to like get form teams where the the sort of tails of your distribution, I said a Gaussian, but maybe it's a T distribution which have heavier tails, like where they're overlapping and so it's easier to speak each other's language a little bit. And this is essential and it's like one, again, one of the challenges of work doing AI in the biospace, but it's also one of the best parts about it because you get to work with all of these brilliant people who are working together towards a shared mission. It's not just a bunch of engineers. Man, I love engineers. I am an engineer, but like I wanted to work with other people who are not engineers. And so like that's that's been, it's been a learning experience for sure, but But it's like, it's actually like one of my favorite things about the company where I work,

• 42:47Speaker 00

  Amy and I, and also sort of like the field in general. It attracts these, these little hybrid people who like to sort of be in the, in the marshlands or into, into hinter regions of these different specialties.

• 42:57Chris Benson?

  You know, that was a perfect segue because we're kind of coming close to the end and I, I want to ask you kind of where this is going, you know, in, in, in the sense of not only your organization, but, but the field at large. You know, you've said that you are working at least currently on single cell and, but there's the work that you guys are doing and there's the larger field and it's a fascinating topic that's very different from, you know, most of the folks that we talk to in terms of how you're applying it and you're kind of having to pioneer and not only the techniques, but the identity, the identification of the problem sets to begin with. And so like where, where do you see both your organization and the

• 43:40Chris Benson?

  larger field going over the next few years? What's possible here?

• 43:45Speaker 00

  So we're already seeing it and what's, what's, what's going to be happening is think of it like assembly lines, you know, from patients and cells and animal mice to therapies in people, right? So the thing of this is like, this is the beginning and the end of the like AI, like drug discovery, therapeutic, like biotech assembly line. And what's happening is little pieces of this assembly line are being productized and because we're getting enough data, we understand the problem well enough. They're not being solved yet, right? So, and we're far from, you know, commoditization, but they're, they're being, they're sort of

• 44:26Speaker 00

  like coalescing, right? And so you're starting to see some companies saying, oh, we have the first fully AI-generated therapy.

• 44:35Speaker 00

  I will tell you, and I can tell you this, I feel okay saying this as an insider. There's a lot of like AI marketing that happens in the biotech space.

• 44:42Chris Benson?

  And just about every industry, trust me.

• 44:44Speaker 00

  Yeah, yeah, but like especially in the biotech where it's like a white hot. There's a lot of AI, you know, so is it really, you know, AI fully controlled drug? No, there are a lot of people in the in the middle. But certain core components, the the running of of in vitro experiments in a Petri dish. The understanding of which patients are good for which therapies. These things, we are starting to have enough data where we can actually. The problems are blossoming into well founded problems on their own. And one of the things that's exciting that that we're beginning to see is people from outside of the biospace are getting excited by these problems.

• 45:22Speaker 00

  Right? So so we have people and and Emi and I has collaborations with top tier professors and research institutions who are not biologists, but are excited by the data and the mission that we have and wanna wanna get in on it. And to me, this is the like fields that can attract top talent to come and join the the sort of fight to build out the core building blocks of this ultimate pipeline are this is very, very promising. And so what I think we're going to see is just more maturation of the handful of problems that need to be solved in order to generate, develop a therapy. And at Emi and I ourselves, we are on this journey, you know, you know, we are, you know, one biotech company.

• 46:04Speaker 00

  We're about 150 people and which is tiny compared to some of the some of the big guys, but big compared to some, you know, some tiny biotechs. And and we're trying to, you know, be at the head of the pack here of identifying what are the problems that need to be wrapped up and solved cohesively. But there are a lot of people out there, no one company is going to do it. And but I I couldn't be more excited about the next decade. I think a decade from now, like my goal, and I think many people at Emi and I, and probably lots of other people in the biotechs world, you know, you look at HIV now, you know, 30 years ago, a generation ago, HIV killed people like and they had months to live when you've got HIV. And now it's a chronic condition, you know, and in some cases, we've actually been able to cure it.

• 46:50Speaker 00

  And I think that with cancer, we are like this is within striking distance. Like we can do this and you know, AI is not going to solve this. It's not a magic wand, but the combination of, you know, AI, data, good people, amazing experimental methodologies, all of this is going to come together. Our better understanding of the immune system, I think over the, you know, over the next generation, I want to be able to tell my grandkids that like I helped sort of like solve cancer. And I think that it's a, it's this sort of like a grand challenge, but what kind of challenge do you want to work on in your life?

• 47:24Daniel Whitenack?

  Yeah, well, I, I think that's a really encouraging and inspiring way to, to end my day and in this conversation. I'm really excited about the things that you're doing, Drossen and the whole team there. And it's really interesting to get a sort of view into AI within this space and how it's also being influenced by things happening elsewhere in the industry. So thank you so much for sharing, sharing these insights with, with us and looking forward to, to following your work.

• 47:52Speaker 00

  Thank you so much, Chris Dino. It's been a real pleasure.

• 48:03Speaker 01

  All right. That is Practical AI for this week. If this is your first time listening, subscribe now at practicalai.fm or just search for Practical AI in your favorite podcast app. We're in there. And if you're a long time listener, please do share the show with your friends. It is the best way you can help Practical AI succeed. Thanks again to Fastly for shipping our shows super fast all around the world, to Breakmaster Cylinder for the beats, and to you for listening. We appreciate you. That's all for this week. We'll talk to you again next time.