🔬Causal Models Need Causal Data - Xaira’s X-Cell model for Drug Discovery (Bo Wang & Ci Chu, Chief Discovery Officer & Chief AI Scientist)

Bet on information If test loss flatlines after 1.5B parameters while training loss continues to drop as you scale, that tells you that your model is limited by the amount of information in your data. Training on a single, smallish data set exposed an information gap: the 3.1B model falls off the scaling trend. Neither parameters nor compute will improve performance past this wall. For predicting changes to gene expression, you need more information rich data. This is what Chu and Bo’s teams have done, and here is what ~30x the information buys you: Now we can scale with parameters and training compute! We don’t know how much this effort costed, but we can guess that data collection experiments and infrastructure was a few tens of millions, and compute + headcount + research was a few million. The budget looks like a RL rollout budget, rather than a data rich pre-training one. We were lucky enough to have the two central figures in this story on our podcast. Taking the lead from Ci Chu and Bo Wang, Xaira Therapeutics is betting that information rich data is the key to AI-driven drug development. Chu was recently promoted https://www.businesswire.com/news/home/20260706699581/en/Xaira-Therapeutics-Announces-the-Appointment-of-Dr.-Ian-McCaffery-as-SVP-Translational-Science-and-Early-Clinical-Development-and-the-Promotions-of-Dr.-Ci-Chu-to-Chief-Discovery-Officer-and-Dr.-Bo-Wang-to-Chief-AI-Scientist to Chief Discovery Officer and Bo to Chief AI Scientist, underscoring just how strategic Xaira considers this bet. Reverse engineering the human cell If you had to figure out how a human cell works, what would you do? A good place to start might be by documenting what genes are expressed (e.g. what RNA is floating around) in different kinds of cells, in different circumstances. That is CELLxGENE https://cellxgene.cziscience.com/, a database of 168M cells built by Chan Zuckerberg Institute that maps each cell to a count of how many times 20K-30K genes were detected in that cell, plus detailed metadata about every cell. A ~4 trillion-entry matrix. If the Protein Data Bank https://www.rcsb.org/ (PDB) unlocked structural biology models (Boltz Episode https://www.latent.space/p/boltz, ESM/BioHub Episode https://www.latent.space/p/esmfold2), CELLxGENE has done the same thing for Virtual Cell models. Like PDB, CELLxGENE has inspired a zoo of AI models of RNA expression; so much so that RNA expression models have become synonymous with Virtual Cell models. Bo Wang built one of the most influential, scGPT https://www.nature.com/articles/s41592-024-02201-0, that became the starting point for Xaira’s new model. RNA expression ≠ Virtual Cell Models trained on CELLxGENE describe the relationship between cell types and cell states, but they are not good at predicting what will happen if we make changes to RNA expression. Changes in gene expression are highly correlated, and its is difficult (impossible) to figure out what causes what in most cases. If you could “turn the dial down” on one gene at a time, however, then you would be able to observe what is upstream and downstream of a given gene. You could tell if A → B & C or B → A & C or B → A, C → B → … If you did this for all of the genes, then maybe you could train a model that could predict what would happen to a cell if you change a gene (e.g. with a drug or a gene edit). Or maybe you could figure out the least invasive way to change a particular gene’s expression. X-Atlas → X-Cell This is exactly what Chu and Bo’s teams have done. The data set is called X-Atlas and the model is called X-Cell. In this episode, we discuss: * Why the team abandoned autoregression for diffusion * The CRISPR-based experiments that run millions of tests in parallel, and generate the raw data for X-Atlas and X-cell * Generalization to real lab experiments in real human cells * Beating the linear baseline that has outperformed previous models * Justifying a kitchen-sink of priors, and how that stacks up vs. data and architecture Bo also shared with us some of the (major) advantages he has as an academic vs. industry leader, and how his labs keep up with the breakneck pace of AI innovation. Check out the full episode on YouTube, or your favorite podcasting platform! This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

• 0:00Speaker 00

  And what really blew my mind away is when I saw the model make prediction, just print out the heat map of the gene function changes, look at the actual raw data, and line up the linear baseline prediction, the ground truth and XL prediction altogether. It's visually very clear to see that XL prediction is much more similar to ground truth than the linear baseline.

• 0:22Speaker 01

  This is a wow moment I was talking about in the beginning.

• 0:25Speaker 00

  This is the first time that someone can put together not just one perturbancy, but seven genome-wide perturbancy campaigns together. Something that jumped out to us biologists right away is that some of the perturbations are context universal.

• 0:41Speaker 02

  Hi, I'm RJ Hanke, CTO of Miraeomics. This is Brandon Anderson, who builds RNA therapeutics at Atomic AI. And this is the Latent Space AI for Science podcast. One of the themes that has run through the podcast is how the lab and experimentation and the real world have probably the biggest impact and have the most relevance to whether something is AI for science or something like B2B SaaS. We're really happy to have in the studio with us today Bo Wang and Cichu from Zera Therapeutics. At Zera, they're building with a bunch of other people

• 1:24Speaker 02

  a AI drug discovery platform. They're using high-throughput experimentation system to collect very large data sets and then training AI models that can predict the way that your cells in your body will respond to drugs and therapeutics. Really happy to have you, big fan of your work. Why don't you two introduce yourselves to the listeners?

• 1:50Speaker 01

  Hello, everyone. My name is Bo Wang. I'm SVP and Head of Biomedical AI at Zera Therapeutic. Joined Zera about eight months ago. And before that, I was Associate Professor at the University of Toronto in Canada.

• 2:03Speaker 00

  And I'm Cichu. My first name is incredibly difficult to pronounce unless it's spoken Mandarin, so I go by Chu, as in Chewbacca or Pikachu. Favorite fictional character. I'm the SVP of AI-enabled discovery at Zera. I joined about one and two years ago when I was still in stealth mode. And here I lead the high-throughput biology group, generating the kind of data that will feed our AI models and also think about their applications. Before this, I spent about a decade in, at the intersection of AI and big data and biology. So previously I worked at Ince Trill, leading the Inveetra discovery platform there. And before that, I was at Verily, which spun out of Google X.

• 2:48Speaker 03

  Okay, so you, you're at Zera, the company which is on the perito frontier of confusing names and mega rounds. So Zera is, I think, kind of came out of stealth like a few years ago and just really big org, kind of out of nothing. So I'm curious if you can explain a little bit about what is Zera's mission, what is their thesis statement? Like, what is, you know, special about Zera? And, you know, kind of where you're going in the future?

• 3:17Speaker 00

  Yeah, Zera is an AI-enabled drug discovery company. And at the core of our mission, we're using AI platforms to generate better therapeutics to advance patient care. And so we will be making drugs using different AI capabilities. There are three main AI platforms that we're building here. The first one is protein design, work that spun out of our co-founder, Dr. David Baker's group from UW. A lot of the current generation of protein designers are here in the company. So there, the thinking is to use advanced AI technology to develop molecules against previously undruggable targets.

• 4:01Speaker 00

  The second AI platform, I guess we'll spend a lot of time talking about today, is the one that Bo Wang and I have been working on for quite some time and just released a pre-print on. That's the virtual cell or foundation model of biology work. There, the hope is to build an AI model to predict biology, exactly like you said, and predict what genes and drug molecules will affect cell biology. And the third piece, which we're beginning to build now, is patient representation models. And the goal there is to have AI models that can understand which patients will respond to which therapeutics. So hopefully, together, these platform technology will help us make better drugs faster and with a higher success rate than previous technologies

• 4:44Speaker 00

  to transform what is used to be artisanal, trial and error in the past, into more and more into an engineering discipline.

• 4:52Speaker 01

  I think what sets Zera different is not just the 1 billion drugs around, but also I think Zera is one of the very few AI-native companies for drug discovery that works from end to end of all sections of drug discovery. From as early as, you know, target ID and the protein design, small molecules, and to phase one, two, three clinical trials, we aim to use AI to accelerate every part of the drug discovery. So that not only we increase the success rate of developing drugs, but also greatly, you know, reduce the cycle time so that we can have, you know, new drugs instead of every 10, 20 years.

• 5:36Speaker 01

  So hopefully, we can have the cycle times that we have more useful drugs for patients.

• 5:41Speaker 02

  That's really interesting. I know there's a lot of interest right now in that third thing, maybe called translation from the lab to the clinic. Where are the bottlenecks? So you have these three models. What are the bottlenecks that you're addressing? And, and sort of like, how are you doing that? Why are you doing it that way?

• 6:00Speaker 01

  There is an AI-native company. Almost every parts of the sections of drug discovery, we're trying to use AI to revolutionize how we develop drugs. So the early part, we build causal foundation models, or sometimes we call it virtual cell. Proteins, we have, you know, state-of-the-art protein engineering models. And we have also patient representation learning models. And I think what Zera is trying to do is not only we develop AI models, but also we create the right data sets to empower these models. And I think what's really make me excited to work at Zera

• 6:42Speaker 01

  is we always aim to connect three AI models together instead of letting them work individually by their own. So when we design virtual cell models, we look for connections to that, can we find targets that is easier to, to apply the protein engineering models? And then even when we design the cellular causal models, can we connect to patient representations? What are the right patient data to connect to the cellular models so that we have something to show clinical utilities? So I think what really make me excited is, before I joined there, I'm kind of a professor in computational biology department

• 7:26Speaker 01

  or computer science department, where we mostly working on computers. We look at the data, look at arrays, et cetera. But once coming to Zera, what really excites me is that I get to talk to people like Chu, lots of drug hunters, you know, extremely experienced drug hunters, to really understand their pinpoint. So when we design AI models, we think about questions that really excites biologists. So later, maybe we can talk about how one of the rewarding signals I received after we developed XL is that, like, it's a wow moment from biologists that this is the first time biologists actually find the model can predict exactly how these unseen cell lines kind of respond to different perturbations.

• 8:12Speaker 01

  So that's kind of the part really excites me is the integration of kind of dry lab or AI models to wet lab or the biology, or even eventually to the clinical side.

• 8:24Speaker 02

  With this clinical model, I know you guys are aiming to, you know, take a drug all the way to FDA approval and beyond. Where do we stand now? I don't know if you're able to talk about this, but like, are you able to collect data from clinical trials and tie that back yet?

• 8:41Speaker 00

  As Bo said, I think if you think about drug discovery process, it's easy, right? You just need to find the right target, make the right molecule, and find the right patients to give them to. Of course, each of those steps are incredibly difficult to get right. And so far, like I said just now, realized a lot of un-trial and error in guesswork. And the main issue, I think, is that we don't have the right biological data, really to power the training of a predictive model. And in protein design space, I think that's where we have seen the most rapid progress so far.

• 9:15Speaker 02

  Yeah.

• 9:15Speaker 00

  That's partially because we have a lot of data, high-quality data, over 70 years curated by the entire community. People deposit protein structures into a database, right, called PDB. We also have a lot of sequence data that are collected over the years from different genomes that can help inform the model as well. And it's these high-quality data that are collected and accumulated that ushered in this revolution in protein design and alpha fold and other folding models.

• 9:46Speaker 00

  In the other domains such as clinical model prediction such as virtual cell, we are nowhere near the same kind of massive data that are high quality. And I think it's mainly a data limitation issue. So to your question, that's why we're very invested in generating these data, particularly causal data in cell biology in a lab. And that's, I think, what made it possible to innovate on the algorithm side as well, to usher in virtual cell models. On the patient's side, it's a very interesting question. Perhaps that's one of the hardest data to get because getting access to high quality patient samples is difficult in itself. Getting it matched to the right clinical annotation so that you can actually learn

• 10:31Speaker 00

  the difference, the bridge between molecular data and clinical response, that's even harder. And you might be able to do that across different disease severities, but it will be harder to collect the right data to predict which drug treatment will or will not respond in a particular patient or not. And so that takes a lot of thought and a lot of careful curation to generate data out of. So we're beginning to go into that area, but hopefully we'll be able to share more soon.

• 11:02Speaker 02

  Awesome. Maybe we should switch gears now. You just released X cell. Why don't you guys describe? I'll butcher it.

• 11:10Speaker 01

  X cell is Zara's first virtual cell models. It is an AI model that can predict the response to genetic perturbations. Certainly, we can extend it to other type of interventions such as drug perturbations, chemical perturbations, et cetera.

• 11:27Speaker 02

  So can you just describe for the non-biologists in the, that are listening? What is a perturbation? What do you mean by that?

• 11:36Speaker 00

  In our cells, when Bill talk about genetic perturbations, our cell, human cell typically have 20,000 genes. Not all cells express every gene equally. That's why your eye cell, your skin cell, your heart cell, even though they share the same genome, they function very differently. A lot of that is determined by selective gene expression that determine the type and the state of the cell. So what we do is to build a model that you can in silico ablate certain genes from the cell. That is a in silico perturbation. That's to say, if I reduce the expression of this gene in a cell, what is the implication for the rest of the cells?

• 12:21Speaker 00

  What's the biological consequence?

• 12:23Speaker 02

  You basically turned the knob down on one gene?

• 12:25Speaker 00

  That's right.

• 12:26Speaker 02

  And then that, what happens to all the other genes in that cell?

• 12:29Speaker 00

  Correct. And the hope is, of course, to predict the effect on all the other genes, but maybe even more things than gene expression, such as the function of the cell.

• 12:36Speaker 02

  Okay.

• 12:37Speaker 00

  And that's important and that's theoretically relevant because a lot of drugs are inhibitors and they function through exactly that, turning down the activity of a protein or gene. And so if we can start with gene perturbation prediction, the hope is that we can also go to pathway inhibition prediction, so on and so forth.

• 12:55Speaker 02

  So a pathway is just a set of genes that all kind of talk to each other by this gene expresses a protein, that protein has some impact on another gene and so forth and so on. There's this long chain reaction of genes and proteins. And then, so that's called a pathway. And so if you interrupt that or somehow change it, then that has an impact on the larger phenotype of the cell, what the cell looks like does, et cetera.

• 13:23Speaker 00

  That's exactly right.

• 13:23Speaker 02

  Yeah.

• 13:24Speaker 03

  Yeah. So you have what you call the virtual cell or you're creating a virtual cell. And virtual cells are very popular. It's a lot of people are interested in this concept. But I think your approach is somewhat unique or separate from what other people are doing. Can you explain what do broadly people mean when they say virtual cells? What are some of the distinct other strategies? And then like, what is your specific strategy that you're going for?

• 13:50Speaker 01

  Certainly, virtual cell is a very high level term to describe an AI model that is able to predict or describe what cell looks like or predict the cell expressions or cell functions after certain interventions. It's a very high level concept. It was, first of all, it was not a novel idea. We had virtual cell project almost 20 years ago. But back then, sometime we called virtual cell 1.0 is that people are trying to derive differential equations to trying to use mathematics to describe what's the response for certain pathway interventions, as you just mentioned.

• 14:34Speaker 01

  And by feeding these equations to different observations. And largely speaking, that was a failed attempt in the sense that the biology is just way too complicated to write in a few predefined set of differential equations. And moving forward with the rise of language models, I think the idea of using AI models to mimic how cell responds to different interventions by data-driven approach start to get popular. And I think three years ago, almost just four months after Chad was released, our lab at University of Toronto published one of the early foundation model

• 15:18Speaker 01

  of single cell genomics, call it SCGPT. It can kind of interpret it as a GPT-like model for single cells. And it quickly become very popular in the sense that for the first time we have a foundation model that is able to tackle different downstream tasks using the same model, such as we can use the same model to integrate different batches of single cell RNA seek. We can use the same model to predict multi-omic integrations.

• 15:45Speaker 02

  Let's define those things. So batches integrate different batches of RNA seek. So you have different equipment, you're all collecting.

• 15:54Speaker 01

  Of data collected from different labs.

• 15:56Speaker 02

  Yeah, different labs, different time of day.

• 15:59Speaker 01

  Correct.

• 15:59Speaker 02

  Different phase of the moon, whatever. And those actually have a big impact on the data that you collect. And so there's a big problem of how do I even compare this data set to that data set when there's all this other differences that have nothing to do with the gene expression and just how I measured it?

• 16:17Speaker 01

  We call that batch effect. We certainly want to remove the batch effect while preserving the cell types, which are more important biology we want to reserve.

• 16:26Speaker 03

  So this is sort of like analogous to the tank problem in image classifiers, for example, is a sort of the models pick up on these crazy spurious features which have nothing to do with what you actually care about, the underlying biology.

• 16:38Speaker 01

  Exactly. Certainly, the core idea of integrating different batches is to kind of keep the biological signals while removing the batch effect. And before these foundation models, what happens in single cell domain is that for every task, biologists have to choose the so-called specialist state of art approaches. And with foundation models such as SCV or gene formers, what we hope to bring is that one model that solve all the tasks in single cells. And with the popularity of foundation model, lots of researchers come together under CCI, Chenzhou Institute. And we published a perspective paper at Journal Cell

• 17:23Speaker 01

  to coin, for the first time, coin the term virtual cell, almost virtual cell 2.0 in the sense that let's use data-driven approaches. If we cannot describe, let's learn it. So that's the idea of virtual cell. So that generally speaking, can we build a language model or language type of model to predict what the cell types looks like, how the cell respond to different interventions? And eventually, we kind of, we can replace all the cellular experiments by simply running simulations on computer without even running the actual experiments.

• 17:59Speaker 03

  Maybe for a bit more context, you can think about this as, so a virtual cell is just a general concept. But you think cells have 20,000 genes in them. And in most human cells, I think what roughly, you know, 4,000 to 5,000 are usually active at any given time or expressed at reasonable levels. So you look at a normal cell, you might have 4,000 to 5,000 genes doing things. And so your question is, in many cases, the way medicine works is you, you know, you target a protein or you target some sort of, you know, something which makes proteins more common or less common, or they stop the protein from doing something. And your goal is, given this, you know, some number of genes which are in a cell,

• 18:44Speaker 03

  every cell has a different composition of genes. What is going to change? You know, will some pathway die off? Will some pathway grow? And how does this, you know, from this you could predict how medicine is going to work by just understanding how changing one specific gene or some cluster of genes could change everything? Is that a correct understanding?

• 19:08Speaker 01

  Yeah, that's a correct high level understanding about virtual cell. What's happening for this field is that we are lacking a concrete definition of virtual cells. And people almost equate foundation model with virtual cell. But in my view, virtual cell is probably a much broader concept than just foundation models. Foundation models mostly provide a reliable semantic meaningful representations of cells. But I think virtual cell is more dynamic in the sense that can we build AI models even predict the development of different, the cell states across different times? Or even can we even describe the spatial changes at the cells,

• 19:53Speaker 01

  different cellular resolutions? In my understanding is that we are really at the early stage to develop such comprehensive virtual cell models. And the foundation model is really just the starting point.

• 20:06Speaker 02

  AI models always begin with a data. You're building a high throughput experiment or have built and are continuing to develop a high throughput experimentation system. Can you, that sounds really cool and really complicated. Can you tell us what that entails? Like, what are you doing? What are the experiments that you're running? How does that inform the building of an AI model? And why do this rather than pick up the Cell X Gene Database, which is a collection of gene expression data that has been aggregated over the public data sets?

• 20:41Speaker 00

  Yeah, great question. I want to pick up where Bill left off. I think Bill said something pretty profound, going from a representation model, the foundation model of biology to a virtual cell. And the key difference there is perturbation prediction or dynamic, dynamic processes in biology. That's a causal concept. For that, I think we need causal data. And if you look at cell by gene, that's a fantastic dataset that curated in the beginning more than thirty-three million cells, now a lot more than that. And at the time when S G P T was trained on that dataset coming out of Bo's lab in Toronto, that was mostly an observational profiling dataset.

• 21:26Speaker 00

  It's descriptive data, not causal, and mostly profiling healthy human donors. And so the model that was trained on this dataset is very, very good at doing descriptive tasks, such as harmonizing across batch effects, removing effects from different labs, different technologies. But I think Bo, us and many others in the field have found that these models that are trained on descriptive data do not yet outperform linear models on causal tasks, perturbational tasks, what we call counterfactual tasks. If I did this to the cell, then what would happen? That makes intuitive sense to a biologist because the correlation data in the descriptive

• 22:10Speaker 00

  dataset can be fit with many, many possible causal structures. In a very simplistic case, let's say you observe gene A, B, C all go up and down together in your descriptive dataset. You can infer that A regulate B and C, that's why when A goes up, B and C also go up. You might also say that B regulate A and C, and that would be perfectly reasonable as well. You could also say that A regulate B and C is completely regulated by something different. You see the problem there. And there's N number of ways to fit a causal regulatory network into descriptive data. Fundamentally, we believe observational data are underpowered to learn causality truly.

• 22:50Speaker 00

  And this is why we realized pretty early on that we need to really start training, building causal dataset to train a causal model. So what are the ways to do that? I think the field has come of age to do these at-scale technique that we call high-throughput biology. And there are many ways to generate these causal data at scale. The technique that we have focused on is something called perturb-seq. So for the listeners who are not familiar with that technology, it combines high-throughput pooled CRISPR perturbation together with single-cell RNA-seq technology to build 2D datasets. Let me break that down.

• 23:29Speaker 02

  Yeah.

• 23:30Speaker 00

  So we just talked about in a cell, there are at least twenty thousand analytes to measure. These are the genes. These are both the features to measure. These are also the lever to perturb the cells with. So for clarity, let's call them perturbations and gene expressions. For perturbation on one axis and the features that you measure that describe the cell on the other axis. Perturb-seq is a technique that leverages the latest breakthrough in lab biology, CRISPR Cas9. These are bacterially derived enzymes that allows you to disrupt gene expression in

• 24:13Speaker 00

  mammalian cells, in human cells, for example. And we can do so in one-at-a-time fashion, so I can take out one gene at a time. Of course, that would be incredibly difficult to scale if I want to do all twenty thousand gene expression knockout in one single experiment. Probably need a huge factory, a lot of robots to do that. Or you can do them in a pooled fashion, and I love pooled experiments. These are hyperscalable. So we have lab tricks that allows to disrupt one gene per cell, but do all twenty thousand genes across many, many cells in one single pooled experiment. Perfectly scrambled, so there's no batch effects, there's no play-to-play differences. So that's all the perturbation throughput.

• 24:58Speaker 02

  Basically use some sort of combinatorial trick to first perturb all the different genes in different combinations, and then you can read them out and do some math on it, and you basically pull out a whole bunch of different experiments in one experiment.

• 25:12Speaker 00

  Correct. It requires barcoding technology, and that barcode is actually achieved by directly reading out what kind of CRISPR guide RNA is present in which cell. So for CRISPR Cas9, this bacterially derived machinery to work in mammalian cells, you just have to deliver two things to each cell. You have to deliver the protein, the Cas9 protein, that does the job, and you have to deliver an address barcode encoded by a short piece of RNA called a guide RNA. And the guide RNA tells the protein where to go in the cell purely via Watson-Quick base pairing, ATCG.

• 25:49Speaker 02

  So it matches a part of the gene. It's sufficiently long to say this will match the correct gene, and then that guides it to connect to the right and reduce the expression of that particular gene in the cell.

• 26:05Speaker 00

  Correct. We designed this guide to go to the promoter part of a gene. That's the beginning stretch of every gene before the transcription starts. And if we bring the Cas9 protein to there, armed with the right effector, the silencer, that promoter will get shut off, and that gene will never be transcribed out of again. So effectively, we tuned down the expression level of that gene. And so all you have to know is figure out which guide RNA is in which cell, and that can be done using genomic readouts. That's the barcode. And you can then infer which gene is being silenced in which cell. So that's the way you scale throughput on the perturbation side. On the readout side, it's a 2D dataset, right? So we just talk about one of the dimensions.

• 26:47Speaker 00

  On the readout side, we leverage single-cell RNA-seq technologies. So these are also recent technologies in the last decade that have been scaled, that can let you read out the expression level of all twenty thousand genes simultaneously from each cell. So armed with both high-throughput CRISPR perturbation and high-throughput single-cell RNA-seq technologies, all of a sudden, we can generate these 2D datasets where we systematically perturb or knock out, knock down every single gene in the human genome in a cell type, and we read out this impact on every other genes in the same cells. So we generate these 2D rich dataset, not that different than the size, the type of PDB data that trained our phone models, right?

• 27:32Speaker 00

  If you think about that, that's hundreds of thousands of protein entries. If those are the rows, columns are the X, Y, Z coordinate of every single amino acid. That's also a 2D dataset. And I think it's these type of rich 2D datasets that power the training of foundation models of biology.

• 27:49Speaker 03

  I find it really fun how you have turned a fairly straightforward assay in using, this is NGS sequencing, right? Next Generation Sequencing, very high throughput. You've used this to scale a simple perturbation response, which is individually maybe not all that interesting, to this massive scale of over basically arbitrary number of cells. I think you did twenty-five million or something?

• 28:15Speaker 00

  So it's actually a lot more than that. So twenty-five million is what came out of the most stringent quality filtering. It's actually as much of a scientific challenge to figure out how to do CRISPR and single-cell RNA-seq as it is an engineering challenge. In the first part of the experiment, oftentimes we have to harvest tens, if not hundreds of millions of cells, and they go through various quality funnels to arrive to give Bo and team the highest quality data at the end. That's incredibly difficult to do because, as you can imagine, all of these techniques have been published by academia before, and they work very well in small-scale experiments. But when you think about scaling them to a genome-wide perturbation, we're talking about handling hundreds of millions of cells.

• 29:00Speaker 00

  Techniques that are published in academia used to be all about handling fresh cells, cells that are still alive. And that may be okay if your entire experiment takes only an hour or two. It's not quite easy to handle cells across a fourteen-hour day that's hundreds of millions of cells. And so by the end of the day, he's the joke with the team, you can easily detect stress signals from the cells and from your scientists in the lab.

• 29:24Speaker 02

  Yeah.

• 29:25Speaker 00

  And quickly, we realized that's not the way to do these data generation. Machine learning is very quality dependent, and we want to make the highest quality data to our AI teams. So we're putting a lot of engineering thought and industrialize the whole workflow step by step, introduce chemical fixations so that we lock the state of the cells in at the beginning of this experiment, but figure out ways that it doesn't disrupt all of the biology, molecular biology steps afterwards, it doesn't impact data quality, so that we can do all of these data generation in a time-shifted operational manner that's very not prone to batch effects.

• 30:03Speaker 02

  One thing that you didn't mention is that you're using some sort of stem cells. And so, and obviously, like, you don't have, you know, like, brain cells or blood cells, or if you did, then you would have a big combinatorial effect on that. So how are you, why are you convinced that?

• 30:23Speaker 02

  Working on stem cells, which are, you know, my understanding is that there are actually blood cells that have been sort of the stem cell behavior has been unlocked on them. And that causes some, some sort of stress on the cell as well. So you have these like sort of not quite blood cells that are stressed. And then how, why are we convinced that that is a good proxy for a brain cell or a whatever you're studying?

• 30:52Speaker 00

  Yeah, not quite. So we didn't actually start with stem cells. That was more of a later development. When we started data generation, so we put out the method that I talk about, as well as the first two data sets, which is the world's largest perturbative data release at the time, last June in the pre-print, we call it dataset XL as Orion. That was actually generated from two cell lines. A lot of this field's early work started with cell lines. These are

• 31:20Speaker 01

  Cancer cell.

• 31:21Speaker 00

  One of them is a cancer cell line, the other is just a cell line. These are immortalized cells. Some of them are derived from cancers, hence cancer cell lines. Others are just derived from primary cells but have been immortalized, many times grown for many, many years in various labs. People start with these cell lines in the beginning, as you can imagine, because those are easy to do. It's easy to scale, easy to grow a lot of cells out of. Turns out the ability to grow millions of cells is actually critical for doing these large experiments. So we started there first, and they actually still capture the characteristics of the cell types that are derived from colorectal cancer, as well as hematopoietic cells.

• 32:03Speaker 00

  But later on, in the most recent pre-print, we actually expanded to many more cell types. Now, some of these are still cell lines, are T cells, we chose to use cell lines, but some of these have now gone into primary cells. So we did one experiment in IPSC, these are induced pluripotent stem cells, and another experiment in, and we think this is the most ambitious and coolest screening that we've done to date, this is a pen differentiation, multi-cell type stem cell project. So effectively, we differentiated IPSC into 10 different cell types in one single experiment without restriction, and we did a genome scale perturbation across them.

• 32:44Speaker 00

  So you can imagine, instead of just generating 10,000 different biological experiments, we did 10,000 by 10 cell types, so it's almost a library on library experiment. Why are we doing this? We think that in the beginning phase of data collection, as Beau said, I think we're just in the early days of virtual cell building, context and diversity and richness of the data matters. It's not just the total number of cells or total number of sequencing reads, it's about bits per dollar and information content. So we want to scale not only in the genetic perturbation landscape, but we also want to scale across biological context, so that we can give our AI teams the best rich dataset to build a generalizable model on.

• 33:29Speaker 02

  Is there any thinking about, so you say context, but obviously, these cells and these experiments have been sort of the, I forget the term, but they've been separated from their cohorts, right? Is there thinking about using spatial transcriptomics or other, you know, sort of technologies, imaging-based technologies to build models with perturbations, but within, in the context of the cells that it lives near?

• 33:55Speaker 00

  Great question. So we're thinking about that in a couple of ways. Number one, that's actually exactly why we want to build a virtual cell model in the first place. You might think that, well, you can already do exhaustive screening in these cell lines, why do you still need a model? You can just do the experiment and generate the data. Certainly, if your query is just about cell biology in cell lines, you're right, we don't need a model, right? At least if we're genetic screening, we can just do this experiment. But you're also correct that oftentimes, good targets, biological insights are not about cell lines. These are about primary cells, about cells in their native physiological context, in organs, or even multi-organ coming together and have some emergent properties.

• 34:38Speaker 00

  A lot of immunological diseases are that way. You cannot do exhaustive, highly put experimentation in animal systems, or in organs, or in all of these complex translational models. You can do some experiments, and these are expensive and high-stake. The ability to build a model that can be trained on massive data where it is possible to scale, and be trained in a way that can be fine-tuned and transferred to make high-quality causal predictions in these complex models, so that we can go into the lab and have the highest quality hypothesis possible to validate, I think that's the whole point about building a virtual cell model.

• 35:15Speaker 01

  But from AI side, I think you're absolutely right, that I believe the future virtual cell model should be able to incorporate multiple modalities, not just RNA expressions. Spatial single cell RNA sec is already a popular technology, even for SHBT, we actually have an extended version, we call it SHBT Spatial, that is specially designed for spatial single cellomics. And we also have papers on early attempts to trying to take the HNE images, trying to predict the gene expressions. There's already some signals you can find. So eventually, what I predict is that a virtual cell model will be able to integrate, not

• 35:59Speaker 01

  only RNA sec can integrate more functionally related, for example, proteomics, or other regulatory side ofomics such as a TAC sec, to overall combine all your descriptive omics datasets to predict the future states of the cellular functions. I think that's probably the, the future for virtual cell model.

• 36:23Speaker 03

  We had Ron Alpha and Dan Baer from Noetic as guests recently, and viewers who want to hear a little bit more about that, I think they go, we go quite in depth there. So if you want to, so background, you can go to that. But can you explain a little bit about what spatial transcriptomics and spatial proteomics are?

• 36:40Speaker 00

  So maybe a bit of a history lesson here. Before we had singles RNA sec, we had RNA sec, and before that, we have microarray technologies. What RNA sec and microarray used to do is take a chunk of my tissue, grind it all up, put in a blender, imagine, make a smoothie out of it, and take the, all of the RNA from different cells in that piece of tissue, and measure all of their expression levels. It is great, for the first time, you can measure gene expression all 20,000 at a time. And we used to do them, we used to have to do them one at a time. But it is not great in that we don't know which RNA came from which cell. And this is particularly a problem if you're dealing with a multicellular piece of tissue. You want to attribute RNA to the immune cell, to the skin cell, to the fibroblast, to the

• 37:25Speaker 00

  keratinocytes, but you can't, because you've gone everything up in a smoothie. What singles our technology allow you to do is analyze them cell by cell. So now, I can attribute RNA gene expression to the cell that they originate from. But there's still a problem. I don't know spatially where these signal come from. And for many disease, it matters, right? In immunology, for example, you want to know when T cells are close to a tumor cells, or when a T cell is not able to penetrate the solid tumor, what is the difference between them? Or when a T cell is attacking the tumor cell, when the T cell is not, what is the difference about them? And for that, you need spatial information.

• 38:08Speaker 00

  You need to observe cell in situ in their context. And so now there are different technologies that solve that problem. Essentially, take that chunk of tissue, I don't have to grind it up anymore, I just make a cross-section, lay it down in a piece of slide, and I can measure its morphology using standard techniques like HNE staining. I can then also measure many protein expression using multiplex IF assays, immunopheresence assays. Ultimately, I can also look at the gene expression up to genome-wide in all of these cells, in their native spatial coordinates, by using some of the latest spatial omics assays. So you have the X, Y coordinates of every cell, but also all of the molecular analysis

• 38:53Speaker 00

  that we talked about earlier. And that's an exciting new direction for genomics field.

• 38:57Speaker 01

  Firstly, can you imagine the, the spatial omics adds more difficulty to AI modeling? Because you have, instead of looking at the individual cells, you have to look at the neighboring niche cells to better kind of learn the representation that is spatially cohesive. That is a challenge the current spatial foundation model are facing.

• 39:20Speaker 03

  But that context is going to be crucial for

• 39:23Speaker 01

  That is correct.

• 39:24Speaker 03

  I mean, understanding, let's say, cancer, where the interaction of immune cells and cancer cells and non-immune or cancer cells

• 39:31Speaker 01

  Yeah, that is absolutely right. You can extract spatial aware biomarkers to predict some of the clinical response. I think that'd be extremely important to build such models.

• 39:43Speaker 02

  Getting back to XL, this, you know, presumably can inform a spatial model as well, right? Because you, you can, you have like one cell in one place. If you can imagine, okay, I can just throw away the coordinates and just get, do inferences on one cell at a time, and now I can create, I can create a more complicated model that does that, but it also knows who its neighbors are, I think.

• 40:07Speaker 01

  You're absolutely right, but the current version we were releasing, we're not dealing with spatial omics. However, definitely our ongoing work and the next version of XL will be able to infer the spatially aware representations for different cells.

• 40:23Speaker 02

  I see. We've talked about the data collection a bit. Let's talk about the architecture, get some red meat for the, for the AI engineers listening in.

• 40:32Speaker 01

  Sure. Let's get to the history of virtual cell modeling, in particular, virtual cell 2.0. I think our SHBT kind of sets the foundation for most of foundation models of single cells is that we adopted kind of autoregressive training, extremely similar to how CHGPT is trained on languages, right? We use its next word, next token predictions. So we mimic the way how CHGPT is trained on languages to train the single cell foundation model on cells. By doing that, you have to assume an inherent order of genes. The way we assume the order of genes is by attention mechanism. There's many other methods that are using different orders of genes,

• 41:17Speaker 01

  some as simple as just ranking the genes based on the expression values. There's also more complicated kind of methods to rank different genes. But inherently, you have to assume an order of genes.

• 41:29Speaker 02

  And just to be clear, so when you talk about genes, those are intrinsically ordered, right? They're a sentence spelled out in ATGC, right? So genes themselves have this, the nucleotides, and there's this long chain, and that makes a lot of sense to have an order to them. But what we're talking about is something different. That's the expression data, the expression levels. So the expression level means how it's just a count for each gene of how many of these genes did I see when I was measuring?

• 42:03Speaker 01

  For the sequences, the order of ATGC makes total sense to us, right? But for expression data, they're literally just matrices. So it's really hard to assume an inherent order of genes. Even if you shuffle the order of genes, I think the biology doesn't change much. However, because of the way kind of language model is trained, everybody has kind of preset tricks to train such models. So it's easy to adopt. That's how all the foundation models are started for single cells. And then I quickly realized that with diffusion language models, we actually don't need to assume the order of genes. Instead, we can have a bidirectional diffusion process

• 42:47Speaker 01

  to generate such long, high-dimensional gene expression data sets. So just to think about it, what's the difference between autoregressive training versus diffusion language models? Is that you can think of autoregressive training as typing. There's, for example, I like coffee. You have to type I and then I like and coffee. There's inherent orders. But diffusion language model, you can treat it as editing. You iteratively generate a sentence from a very vague, very rough sentence, and then you can iteratively refine it. So same thing with gene expressions. You can generate a very rough representation of the gene expressions, and then iteratively, from noisy representation to more refined representations.

• 43:31Speaker 01

  So you kind of iteratively edit the gene expression predictions until it minimizes the losses. So this is a very different philosophy to generatively predict the response after perturbation. It turns out, it actually fits more to a single-cell RNA sec. So that's why we switched from SGBD-like model to the current Excel model, which is using diffusion language models.

• 43:57Speaker 03

  When I think of transformers, like they're fundamentally objects which operate on sets. The community spends a lot of time trying to make them things which have some sort of causal ordering to them. But if you just naively take a transformer, it's a set operation, right? So given that, why think about this in terms of diffusion or autoregressive LLMs? Why not have your initial prediction strategy be something like, take just a set of genes, each of which has its own kind of one hot encoded identity, and then use that as sort of a prediction? That seems like a much more natural architecture to me. And it's not just, you know, your work. A lot of people work on things like this.

• 44:39Speaker 03

  And I have been somewhat confused why there's this bias in the community about this.

• 44:43Speaker 01

  So I think what you were referring to is more related to representation learning, where you can take sets of genes and try to project to low-dimensional latent space. But what we care about for building generative modeling for virtual cells, because you want to predict the dynamics of cells. So we want to have a generative model. So that's why we're mostly using decoder-only architectures in order to generate the full transcriptomics, instead of just predict a predefined small set of genes, because you want to model the whole gene regulatory networks, which are extremely kind of high-dimensional, right?

• 45:22Speaker 02

  So just to be clear, input is genes plus a perturbation? Output is new gene expression levels? Is that gene expression levels plus perturbation is input?

• 45:33Speaker 01

  Plus times cells, number of cells.

• 45:35Speaker 02

  Like for each cell?

• 45:36Speaker 01

  Correct, that is correct.

• 45:37Speaker 02

  Right, okay. And so the way that I think about, the way I think about diffusion language models, and you can correct me here, because I don't know a lot about them. But the way I think about them is they're like BERT, but you do it over and over again. Is that kind of a good?

• 45:54Speaker 01

  Yeah, that is a rough understanding of how diffusion language model works.

• 45:59Speaker 02

  Yeah, so you just apply the diffusion, the diffusion process is like basically unmasking or editing once over and over again. The similar to how like an image diffusion model kind of refines the image over and over again. But in this case, I'm using BERT. So it is a transformer, basically.

• 46:19Speaker 01

  It is a transformer.

• 46:20Speaker 02

  But it's like repeatedly updating the sort of sentence in this case, which is a bunch of expression levels over and over again.

• 46:31Speaker 01

  That is correct. Actually, in our paper, we show that as the number of diffusion steps goes on, the loss function keeps decreasing, the fitness of the prediction to the ground truth keeps increasing. So this, which means the model starts to understand how iteratively refine the predictions.

• 46:52Speaker 02

  I see. We're talking about diffusion versus autoregressive. What there, I noticed in the paper, there's a bunch of discussion of preconditioning, using a whole bunch of stuff. Can you want to talk a little bit about that?

• 47:04Speaker 01

  Another major innovation we made in Excel is the way we incorporate prior knowledge into the model. So incorporating biological priors has always been a good idea in biology in general, because biologists spend decades to understand some of the biologists already. How do we tell the model some of the prior knowledge, some metadata about the cells? Before Excel, what people do is they're trying to incorporate a single type of priors. For example, GEAR, using gene regulatory network as a prior to predict the perturbations. SGBD is sometimes trying to incorporate PPI as a prior as well.

• 47:46Speaker 01

  Excel, to my knowledge, is one of the first models and trying to incorporate extremely diverse sets of biological priors. So in our pre-print, we incorporated five types of priors, including literatures. As simple as just ask GEBT, tell me everything about this gene. And then we embed the output as an embedding.

• 48:10Speaker 02

  That's a gene PT.

• 48:11Speaker 01

  Exactly, that's a gene PT. And we also incorporate PPI, protein-protein interaction networks. We also incorporate DEPMAP, which is cancer-related essential gene informations, morphology informations. We're even trying to incorporate SGBD embeddings, which is basically cell types. So with a set of prior knowledge as conditions to the model, the model starts to have more accuracy in terms of context-specific predictions. And what's more interesting to us is that by looking at the weights of different priors, we can actually understand which prior knowledge are more important to these particular cell types.

• 48:54Speaker 01

  So it adds more interpretability to the models. So we find that combining diffusion language model plus a very diverse set of prior knowledges, Excel does much better in generalizing to unseen contexts. So this is some of the AI innovations we made for Excel.

• 49:13Speaker 02

  Do you now need to provide all of that context in order for the model to work? Or that those are like preconditioning that it can also do without if you want?

• 49:23Speaker 01

  We don't need to incorporate these prior knowledge anymore, because these are already learnable parameters inside the models. However, what you suggest is more prompt or in-context learning for virtual cells. We can do that as well, basically by adding more conditions into the prior knowledges, so that to prompt the model to predict towards certain directions.

• 49:45Speaker 02

  In other words, you took it, you, the model now takes advantage of the learning using the priors that you provided during training and doesn't need them, but it has some advantage because you provide them during training. But you can even get more advantage if you are able to provide those priors during inference.

• 50:04Speaker 01

  Yeah.

• 50:04Speaker 02

  Okay, wow, nice.

• 50:06Speaker 03

  How much does that matter? I mean, whenever I see big machine learning papers with tons of things thrown in, I'm always wondering, where's like the big alpha and where's the little alpha? How much are, you know, is this just some, are these adding this little bit of incremental performance boost? Or, I mean, are all these actually crucial to generalization?

• 50:24Speaker 01

  So there's multiple factors we have to consider. How much contribution the data contributed? How much of the contribution, the AI architectures contribute? Even for the architecture, what's the delta from switching to autoregressive training to diffusion language model? What's the delta from the prior knowledges? Certainly, all of these needs very specific ablation studies. From empirical experience, we find that the qualities, the amount of the data sets matters the most. This is why we were extremely excited to publish the Pyese's data sets, which has sixteen different cell types and across twenty-five million cells.

• 51:09Speaker 01

  And it's genome-wide. It had the kind of a huge tensor, if you really think about from computer perspective, genome-wide perturbation, genome-wide transcriptomics, plus number of cells, plus times number of conditions. So it's a massive tensor. And those, because of the poor screening technology, we don't have batch effects. So you don't, you don't need the model to climb the hill of batch effect. That's already a advantage. So we find that train on perturbation data sets, high-quality perturbation data sets already gives a big boost to the models. We also did ablation that if we train all the virtual cell models out there, including

• 51:50Speaker 01

  state cell to send this original S G V D on the same data sets, what's the delta we are observing? We reported the results there as well. We find that switching from autoregressive training to deficient language models give a significant improvement over some of the harder tasks, particularly generalized to unseen tasks. And the prior knowledge, more or less condition-specific. For certain cell types, some of the prior knowledge make a huge difference, but for certain cell types, the delta seems to be marginal. We are thinking about, you know, how to better incorporate the prior knowledge.

• 52:34Speaker 01

  We still believe that letting the model know a big chunk of existing biology should be helpful, but maybe it's the way we incorporate the prior knowledge through cross-attention, limited the scope of the metadata, but I think it's certainly a research topic. But overall, if we have to give an order, my order would be the quality among scale of the data sets, and then the architecture, and then the prior knowledge. But certainly, this is only applies to our Excel. I'm sure there's different choices of architecture, have different ranks of contributions.

• 53:13Speaker 02

  First of all, this is really fascinating, very cool model. I hope everyone has a chance to look at the paper. There's a lot of, obviously, a lot of resources that were put into doing this. I don't know if you guys can disclose how much. It's a lot of money, whatever it was, operating wet lab, probably very complicated training runs. I think there's a couple, four billion parameter model, is that right?

• 53:37Speaker 01

  Four point nine.

• 53:38Speaker 02

  Yeah, four or five billion parameter model. So much larger model, probably took a lot of GPUs to train. What's the lift that you get from this effort versus let's just put the money into like wet lab work and the sort of traditional pipeline that, you know, basically has been the status quo up until now?

• 54:00Speaker 00

  Biology is a multi-skill discipline. There are cells, there are DNA sequences on the most fundamental level. There are cells, there are multicellular pieces of tissues, cultures. You have tissues, you have animal systems, and finally you have human. I think we would like to be able to do causal prediction towards the right of the spectrum, ultimately do causal prediction in human, know what drugs will work in which patients, but that's very difficult to collect high-throughput data on. And so the whole vision of virtual cell is to generate data where it is possible so that we can transfer the causality prediction towards the right, towards the more translational, the more complex systems.

• 54:44Speaker 00

  The, certainly you can mine the data already. We generated a lot of data, as Bo said, seven screens, sixteen different biological context, genome-scale perturbation. There's a lot of good ideas in that already. There is a figure that we put out in a pre-print that just look into inactivation of T-cells. We already saw some, you know, we saw known biology, TCR complex. We also saw some putative new biology, which we're very excited to validate in the lab. Some of that were actually also called out in a very recent screen last December, published from Alex Marsan's lab, also in the Bay Area. So very excited to see that.

• 55:24Speaker 00

  But the hope is to not just mine the existing data, the hope is that the model can generalize, and we will be able to do in silical experiment into the future. Nobody knows before how much data and what kind of data are needed to do that. With the whole field is waiting for the demonstration that the model can beat linear baseline in perturbation prediction, and it can generalize out of context, not just within the cell line you have training data on, but out of that context. That's where, that's why you need a model. So what's very exciting for us is that in this pre-print, we saw that generalization capability, a few demonstrations.

• 56:05Speaker 00

  We first did in T-cells, we actually generated the data expressly for this purpose. We generated the resting T-cell perturbation screen. So these are T-cells in their baseline condition, not activated. And now we have a activated T-cell perturb seek.

• 56:24Speaker 02

  So just T-cell activation means I'm going, I'm trying to kill something?

• 56:29Speaker 00

  No, these are regulatory T-cells, but yes, we activate their receptors so that they're starting to proliferate.

• 56:35Speaker 02

  Okay.

• 56:36Speaker 00

  They become more active, they can do their physiological job. And we only, critically, we only train the model on the resting T-cell. And we told the model, hey, this is how the active T-cell look like, now go and predict what all of the perturbation are going to do in this active T-cell. And the model have not seen how perturbation working in active T-cells. And we set up a couple rigorous tests. One, linear baseline. Took the perturbational delta in the resting case, just transpose that linearly onto the active T-cell, and that's our linear baseline. Essentially, think about this as a combinatorial perturbation prediction problem. One of the perturbation is activation of the cell.

• 57:20Speaker 00

  The other is all of the genome-wide perturbations. Can I just linearly add the two effects together? That would be a linear baseline. And second, we apply other models from the field. And last, but we applied Excel. Critically, Excel has not seen active cell, T-cells, and is able to make accurate prediction, not only on the known biology, the TCR complex, predicting their effect accurately, that these are going to inactivate the T-cells, which is exactly what we would expect to see, but also it predicted the putative T-cell inactivators that we found in the screen correctly as well. So that's very exciting to us, and that suggests the possibility that we might be able to use these virtual cell models completely out of context, in the unseen context, and

• 58:05Speaker 00

  predict new biology. And so we're very excited to follow up on those hits and validate them in the lab. Just very briefly, a couple other cases that we saw exciting generalizing capability of this model. Remember, we did a multi-cell type differentiated I P S C experiment. There we specifically held out one cell type from training. So the model has not seen that cell type, train on the other cell types as well as the rest of the data sets. The model made very good prediction across thousands of genes, thousands of perturbations in that unseen cell type. So again, suggesting the model's ability to generalize out of cell type. And the last experiment, I think we're very excited, is that we trained this on T-cell

• 58:48Speaker 00

  cell line, but there was just very recently a primary T-cell perturb seek published from Alex Marsan's lab. That's an impressive amount of work. It's not easy to do this scaled screening in primary cells. Very few labs have that kind of capabilities. Much easier to do that in T-cell lines. Again, the model is able to generalize out of cell lines into primary cells and make accurate predictions there.

• 59:15Speaker 02

  So they actually perturbed primary cells, not, not cell lines?

• 59:20Speaker 00

  Primary T-cells harvested from donors. And we were from multiple donors, and Excel trained on just one T-cell cell line is able to make predictions in, across multiple donors from primary T-cell experiments.

• 59:34Speaker 02

  This is a validation of the whole theory, right? That you can train on these slightly weird cells and that it will be good, because you know you're covering the domain well enough or whatever it is, that you're able to actually predict in real, real cells that come directly from real people?

• 59:52Speaker 00

  That's right.

• 59:53Speaker 01

  I think building virtual cell is not to replace biological experiments, as you mentioned. What we're trying to do, really the holy grail of virtual cell is to have a model to generalize to unseen contexts that is harder or even impossible to conduct biologic experiments on, right? So far, Excel focusing on cellular cell lines. And eventually, we want to extend to more complicated biological systems such as animals, organoids, and eventually, as we mentioned before, to patients, to human biology, right? And bearing a few numbers, 90% of disease has no cure.

• 1:00:33Speaker 01

  And most of the drug failed at the phase three clinical trials on patients. And the success rate of these three trials is as low as 5% to 10%.

• 1:00:44Speaker 02

  And phase three means?

• 1:00:45Speaker 01

  The final stage on the patient trials.

• 1:00:48Speaker 02

  So that's when you generalize from toxicity in phase two to efficacy in phase three, right?

• 1:00:55Speaker 01

  From a small cohort into a much larger cohort.

• 1:00:58Speaker 02

  Oh, sorry, toxicity is one, right? Two is small cohort. And so a lot of--

• 1:01:03Speaker 01

  To the large cohort.

• 1:01:04Speaker 02

  So the generalization problem, okay, this drug, I very carefully selected my patients, and it works pretty well. And now I get a bunch more patients, and suddenly it doesn't work very well. And that's the big problem that you're--

• 1:01:16Speaker 01

  Certainly, the promise of virtual cell is that can we build such a model that learn all the causality of biology so that can be grounded to predict the response eventually on patients so that we can, for certain drugs, we can select the right patients to conduct the clinical trials on. Right, so this is a long-term vision, but we are already see some early hopes that Excel train on diverse set of causal data sets can already generalize to some unseen cell types. So certainly, there's a lot of experiments to be done to validate this model. So even continuously finding this model, but I think that we certainly see some early hopes.

• 1:01:59Speaker 03

  So you were talking about linear models, and this brings up this famous or infamous arc challenge about, you know, perturbation. And there's been this theme about complicated foundation models oftentimes not beating linear baselines. And I'd like to get your take about that as terms of what is this? You know, first of all, is this different? I mean, I think some of maybe your own models might also have had trouble beating linear baselines in the past. Is there something different about your current data strategy or where you're going and where's the field going? And what is the role of foundation models versus, or in virtual cell models versus, you know, these simple baselines?

• 1:02:42Speaker 01

  Yeah, a few things. First of all, those benchmarks, as you mentioned, are conducted on reproloca data sets, which are very small data sets. And the metrics people report are mostly MAE. Certainly, you can imagine if the, because single cell data set are so sparse, the average profiles of all the cells, certainly, you can imagine it's a great kind of local optimum to minimize the MAEs. This is why sometimes the average profile of cells has lower MAE even than technical replicates, which are considered of ground truth for kind of perturbation experiments.

• 1:03:26Speaker 01

  So that itself shows that that metric is not reliable. However, most of these benchmarks are still comparing foundation models that train on static expression data sets such as cell by genes, SGB is often benchmarked against. Internally, we also find that when it comes to MAE, sometimes SGB kind of fail to outperform linear models, just because of the reasons I just stated. And what sets Excel different from these static expression models such as SGB or Gformers is that we actually, instead of train on gene expression data sets, we train on causal data sets. We train on massive amount of genome-wide perturbation data sets so that it learns

• 1:04:08Speaker 01

  better about the dynamics of the interventions. And in our pre-print, we extensively compare with linear model as well. And as Chu mentioned, linear model totally fail to extend to unseen cell types. You can quickly imagine why. And I believe that foundation model or other more complicated AI models that train on the right data will outperform these linear models in harder tasks, particularly in generalization tasks. And that's why I keep mentioning the right data set with the right AI model will lead to huge improvements.

• 1:04:48Speaker 01

  But I think the field still needs to see more biological validations to be more convincing that the virtual cell direction is the right one.

• 1:04:58Speaker 00

  Yeah, I think the field suffer from a lack of consistent and uniformly accepted benchmarks. What gets measured will get improved. And in our paper, we measured, I think that one of the metrics that we put a lot of thought into and saw the model really shine is metrics around gene expression changes. So, you know, Pearson Delta, the similarity between predicted changes and ground truth changes upon a perturbation. That's very hard to cheat on. You have to really get the changes right. And what really blew my mind away is when I saw the model make prediction, just print out the heat map of the gene expression changes.

• 1:05:39Speaker 00

  Look at the actual raw data and line up the linear baseline prediction, the ground truth, and Excel prediction all together. It's visually very clear to see that Excel prediction is very much more similar to ground truth than the linear baseline.

• 1:05:55Speaker 01

  This is a wow moment I was talking about in the beginning.

• 1:05:58Speaker 00

  That's right. And it's not hard to understand why. When we put, so this is the first time that someone can put together not just one perturb see, but seven genome-wide perturb see campaigns together. Something that jumped out to us biologists right away is that some of the perturbations are context universal, meaning that the genes do the same thing regardless of the cell types you experiment in. Might not be surprising to you that these are your housekeeping genes, right? Of course, they do the same thing every cell. And then there are all of these other clusters of genes that have very context-specific functions. They do different things in different cells. Again, not hard to imagine why. In iPSCs in our stem cells, we saw developmentally relevant genes, genes that are important for neuronal differentiation.

• 1:06:43Speaker 00

  They only light up in iPSC experiments, of course, right? That makes sense. So think about biology, it's so complicated. You have to capture these, you know, context-universal perturbation effects. You also have to somehow learn the context-dependent perturbation effects. It's not hard to then see why a very sophisticated nonlinear model is able to capture and learn all of those biology much better.

• 1:07:07Speaker 03

  Are your perturbations always single gene perturbations? Or do you have more? Because my understanding of regulatory networks is oftentimes, sometimes it can be a single gene does a ton of things. For example, I think males are just differentiated due to one gene being enabled at like day seven of embryo development or something, and that differentiates everything is this one gene. But then sometimes you have large networks of genes which all are very redundant, which allows for more subtle feedback mechanisms and so on. So I could imagine a lot of single gene perturbations as being kind of irrelevant and that you might want to start having a more commentatorial strategy here.

• 1:07:51Speaker 00

  Yeah, that's a great question. And Bo and I have thought about this a lot. Actually, it's interesting that you brought up group protective biology in, I studied a lot of in female cells, the compensation, the dosage compensation mechanisms. There, female cells have two X chromosomes. Male cells have one X chromosome. To match the dosage output from X chromosomes, strategy that the mammalian cells employ is one gene that produce a RNA that does not encode for any protein, just a non-coding RNA. That RNA wraps around one of the female X chromosomes and turns down most of the gene expression from that chromosome, shove it away in a corner of the nucleus, and it's

• 1:08:35Speaker 00

  called a bar body, and it's never heard from again. And so, absolutely agree with you, one gene can do a lot. But in biology, you also have redundancy. You have compensation. You have all kinds of mechanisms where knocking down one gene is not sufficient to always see a phenotype. What if four genes redundantly do the same thing, right? Taking out one is not going to be sufficient. So where we started with one cell type at a time, loss of function, single gene perturbation, and look at only RNA expression as the output, we're expanding the platform along all of those axes. So that's what we do today to build a scaffold of the data for training models like Excel.

• 1:09:16Speaker 00

  We are now beginning to grow in all three axes of the platform, going beyond transcutomics alone to like a multimodal data, going beyond just one gene perturbation alone to look at also pathway activation and inactivations, turning on and off entire cascade of gene chain reactions. And also going beyond just cell lines monocultures into more and more complex, translationally relevant systems into primary cells, into organoids, and doing even direct in vivo perturbation screens. So we believe with all of that expansion, the data will be all the more exciting to train models on.

• 1:09:54Speaker 01

  This is also why we incorporate PPI networks as a prior knowledge into our model. And although the model right now are trained on single gene perturbations, but once the model is trained, you can actually predict combinatorial perturbations just on the model in only silico, right? So in the sense that you can just perturb the tokens of two genes at the same time and see what's the response. Certainly, without training the actual combinatorial perturbation data sets, the accuracy may not be there. But at least with the existence of such models, we can start to generate hypotheses using in silico perturbations.

• 1:10:33Speaker 02

  How do you see the role of the scientists changing in the age of AI? And you may, because you're not using, your focus is not language models themselves and agentic science and things like that, then you may have a different, slightly different take. You're building, you know, these very specific models. But still, one, how are you and your students able to maintain such a high pace? And I suspect it may have something to do with generated by AI partly, but also, and how do you see the role of the scientists, the academic, changing?

• 1:11:09Speaker 01

  Yeah, that's a great question. So my official split of time is 80% on Zara, 20% on my university affiliations. But turns out the reality is 100% on Zara, 100% on-- You invented a time machine. So the way I try to keep up is suddenly we, my lab use lots of AGNT AI trying to monitor all the AI papers every day and which every week we have lab meetings, we're trying to discuss different topics in AI for biology, AI for healthcare, et cetera. And it's, it's to be very frank, even as a professor, I find it's extremely hard to catch up.

• 1:11:54Speaker 01

  The pace of AI is just so incredibly fast and to the point that sometimes I feel anxiety waking up says, oh my God, this, this paper, so many people published and what happened to our existing unpublished work? And suddenly you can imagine students probably face 10x anxiety. So sometimes I try to encourage students to really kind of using different tools, trying to stay focused and finding the niche area that we become experts on, right? But with the era of generative AI, now AGNT AI, I find the way people, at least academics does science are extremely different now.

• 1:12:39Speaker 01

  Overall, most of professors or students in academics start to be very struggling in terms of fundings, in terms of the pace of publications. And that's why you can see lots of major breakthroughs come from industry, right? Like Alpha Fold, for example. So how academics survive or even thrive at such era of AGNT AI is certainly something everybody is thinking about. We see lots of faculties left university and join industry for, simply for the reasons of resources, right? If you are doing research on AI, do you have enough GPUs at your school? It's the first question you should ask.

• 1:13:23Speaker 01

  When a student join a professor's lab, the first question they often ask is how many GPUs do you have, right? So suddenly in that sense, industry has a major advantage over academic labs. But I think what academics labs have advantages on is really the kind of the pace of innovations and also the niche areas as this specific academic lab can be extremely expert on. So also having the, the freedom of thinking sometimes also make you easier to innovate on ideas that maybe industry people didn't even think about.

• 1:14:03Speaker 01

  Overall, I think the whole field needs to be a lot more innovative to catch up and hopefully with, with the help of different tools and, and hopefully the government start to invest more into academia because I still deeply believe that academic is the main source of innovation for the whole field and particularly when it comes to biotech. So hopefully we, we see more investment into academia so that we stay afloat.

• 1:14:33Speaker 02

  I agree with you, but, but why? Why do you think that, why, why shouldn't the money just go to industry? Why, why should the government put any money into academia?

• 1:14:43Speaker 01

  I still believe the power of, you know, academic freedom. This is actually the, the original motivation we have such, the existence of academic professors who not only we teach, but also we do research. And there's also benefits of teaching and research at the same time in the sense that when you teach a subject, you actually have to become the experts on and that forces you to keep updating your knowledge base and then find easy ways to convey your knowledge in to students. And by doing that, you actually start to innovate on different ideas and, and myself,

• 1:15:23Speaker 01

  for example, I teach a, a big class in University of Toronto about deep learning and neural networks. It's a gigantic class with 600 students every year. And by teaching that, it forces me to update, update the slides, lectures every year and myself reading lots of materials, trying to update myself so that I can find ways to convey some of the knowledge into, to students, right? So that's how I keep updating myself whenever there's, I remember vividly we have GPT two, we updated the lectures and then different language models, how the multi-thread GPU communication is, is used in training larger scale neural networks, et cetera.

• 1:16:06Speaker 01

  So I force myself just update the models. And by talking to different students, we really generate lots of novel ideas to apply the cutting edge AI models to very specific niche areas in biology or in healthcare. Maybe it's unique to Canadian academic system. By being a professor in academic, we also have access to lots of healthcare datasets, which are very hard for industry to access due to many legal reasons or regulatory reasons. And that's why you see some of the papers we publish through academic hospitals in Canada, where we developed some of the state of the art foundation model for ultrasound images.

• 1:16:51Speaker 01

  So that's what I mean that there's certain level of freedom of academic thinking that really drives lots of innovations. And I still believe, maybe it's biased, but myself still believe that having certain level of freedom of academic thinking really leads to lots of kind of innovations that is unthinkable in industries.

• 1:17:17Speaker 00

  I a hundred percent agree with Bo there. So, you know, just thinking about the lab workflow that we do, a lot of these are building upon innovations that were first pioneered in academia as well. CRISPR, of course, was discovered in academia. CRISPR applied to mammalian high-throughput screening, also demonstrated in academia first. Singles RNA-seq, this kind of droplet encapsulated singles RNA-seq first demonstrated in academia. Then different companies tried to build it up into commercial offerings. Putting all of these together to do perturb seek at first, also pioneered in academia, right? CRISbox lab, Jonathan Weisman's lab, Evie Riegev's lab, many of the pioneers in academia. And then I think these, especially on the lab side, these innovation takes so long

• 1:18:02Speaker 00

  and the discovery process can be so accidental, right? That it's perhaps not ideal for pure industry to take on. But once they show early promise, scaling them and robustifying them and generating data that's not only massive but high quality, especially for AI scale, I think that's something that can be very well done in industry, both the mindset as well as the kind of the resources that we can support, that sometimes can be hard for academia labs to match.

• 1:18:39Speaker 03

  Zara has been very generous with releasing your data sets and your models. You've been, seem to be very committed to open science. Given some of the things you were just talking about and, you know, the discussion about like what is the best, most important data strategies for virtual cells or understanding human biology? Where do you think academia should go next, now and next? Since, you know, Zara has probably a budget comparable to, you know, probably dozens or hundreds of bio labs right now. What do you think that if you are an academic, a professor, especially in a wet lab, what would you want to be focusing on?

• 1:19:20Speaker 01

  First of all, myself is a deep believer of open science. That's why all the models we talk about here that are, are open source, you can find all the data weights from my lab GitHub.

• 1:19:32Speaker 03

  It's very thorough.

• 1:19:33Speaker 01

  Yeah, thank you. And the reason I believe open science is that, as you mentioned, most of the time academic lab, we started an idea and we prototype it. It's not scalable, it's not even a good product and industry can take it to scale things up. This is also why SGBD quickly become one of the most widely used single cell foundation model in pharma companies. So that's very encouraging to us. And this is why XERIS also start to open source some of the datasets, some of the models. Part of the reason is that we believe virtual cell is such an early field. And it doesn't help to withhold certain datasets or certain models because it's so early.

• 1:20:16Speaker 01

  A better win-win situation is everybody gets to in this field, start to contribute data together, start to contribute models together to exchange ideas so that this field can move forward in a much faster pace. We see successful examples in protein space, right? Because of the availability of open source datasets in PDBs, therefore, we have models such as AlphaFold, RosettaFold. Again, AlphaFold to also open source, therefore, we can quickly iterate different models. That's why you see kind of a booming situation in protein space. We want to do the same thing for virtual cells. Let's put all the data together, let's have the same standard protocols to generate high-quality datasets.

• 1:20:59Speaker 01

  Let's put all the resources together to generate next generation of virtual cell models. When it comes to academic labs, suddenly Chu can comment on wet lab, how wet lab academics can survive. But from dry lab perspective, I do encourage all the, you know, dry lab AI researchers in universities start to collaborate with industries so that they can get more resources to develop their own ideas. And with the era of HNT AI, now everybody can code. So it's more important to have a right taste about your project so that you don't just let, just burn tokens without purpose, right?

• 1:21:40Speaker 01

  So we want more academic students, academic professors to have higher taste of research so that we make the right utility of HNT AIs.

• 1:21:53Speaker 03

  As a professor, your job is to have taste, obviously. But as a student, how do you develop taste as in a world where so much of the thinking scientific process could be essentially outsourced to an LOM? Or, you know, there's not a world where you're forced to bang your head against something and learn taste by the hard way?

• 1:22:15Speaker 01

  This is why we need academic training where you get into a field you know nothing about. And hopefully, after you graduate, you become the expert about this particular topic in the world, right? This is why you have to go through different programs to talk to your peers, talk to your professors, to get an idea about what's a good research taste to begin with. And but more importantly, I always teach students that the best way to learn something is to just program it. By programming, you kind of know what's the details hidden in all the mathematical equations in the paper, which often you omit.

• 1:22:55Speaker 01

  But in the era of HNT AI, seems slightly different in the sense that we used to spend lots of time coding, a little bit of time just debugging. But now we let the agent do most of the coding, but we spend most of the time debugging, which seems to be definitely interesting to me. And we had lots of discussion with the students in the lab, what's the best way to spot bugs by AIs? So how do you find places where AI is particularly good at? Also, how do you find places AI are still limited at? It's kind of, you need lots of China era as well.

• 1:23:38Speaker 01

  And in the end, you still need to kind of validate your model using real-world evidence, right? So that's why collaboration with wet labs, collaboration with clinical teams to validate your model, provide a feedback signals to your taste, as we discussed, is certainly a very important training program.

• 1:23:58Speaker 00

  On the wet lab side, academic has extremely important roles to play. I think we'll enter a field of, an era of symbiotic innovation and cross-pollination of ideas. Just like in AI field, I think we have great ideas coming out of academia still all the time. But industry now increasingly are contributing new ideas on architecture, on all of that as well. In the wet lab side, certainly, industry seems to be able to scale this type of data generation quite well. But biology is so much more than just cell-based perturb seek. Beyond RNA seek, we would like to measure many other analytes, right?

• 1:24:39Speaker 00

  Proteins, metabolites, lipids, protein-protein interactions. How do we do that at scale? Beyond individual cells, we would like to be able to measure cell interaction, spatial, cell in their native context, or even whole animal level in vivo perturbations. Again, how do we do that at scale with lots of great innovation coming out of academia? Actually, just one great paper last week. And so how do we connect all of these together? I think we have many years of work ahead of us to fully crack data generation for all biology. And that, I think, we need to scale, the industrialization, the innovation from industry. We also need that from academia, I think, will together move this field to the next level.

• 1:25:23Speaker 02

  One question that we've been trying to ask everyone is, in your field, which you could say maybe is AI and, you know, sort of high-throughput experimentation, or however you wanted to find it. If you could wave your magic wand and have a bottleneck removed for you, or a key problem solved, what would that be?

• 1:25:45Speaker 00

  Protein.

• 1:25:46Speaker 02

  Okay.

• 1:25:47Speaker 00

  If there is a way to do protein sequencing or high-throughput protein measurement, the same scale that we can do genomics, that would be amazing. Yeah, I'm training genomics field, but if I can do that, I would incorporate that technology in a heartbeat. RNA is amazing, it foreshadows which proteins are going to get made. But protein, by and large, are the functional units in a cell. Not only does their abundance matter, their post-translational modification matter, their localization in a cell matter. If you can measure all of those things, their conformational states, their modifications, their abundances, their localizations at scale, single cell or even spatially,

• 1:26:28Speaker 00

  I think such datasets would be incredibly useful to train the next generation of financial models. I know there's a lot of innovations in that direction. Can't wait to see that become, come of age.

• 1:26:39Speaker 01

  My hope is, I hope to see a breakthrough in sequencing technology, not just the reduced cost, but sequencing technology that can sequence the same cells at a different time points. I think this is much lacking right now because you have, in order to sequence the cell, you have to kill the cell, right? So can we have a technology that can measure the cell states at different time points for the same set of cells? I think that will bring a very different dimension to the dataset so that we can start to measure the temporal dynamics of cells. So far, everything we measure, everything we model is extremely static. So can we have a technology that measure different cell response at different time points

• 1:27:25Speaker 01

  for the same set of cells? We'll unlock massive opportunity to kind of model the dynamics of cells. To me, that is a real virtual cell.

• 1:27:34Speaker 03

  That's a really interesting idea. I never would have thought about that. Wow. Would you be okay with even just partial, like, small snippets of genes? Or maybe three-prime regions of a small number of transcripts?

• 1:27:46Speaker 01

  Yeah, we can start with a small, a small set of gene panels to begin with, right? But eventually, if, since we are talking about the magic wand, eventually, if we can have a system that observe how cell evolve at different time points, and we have enough data to actually model such development, I think that would be real virtual cell model.

• 1:28:11Speaker 00

  There's a small attempts in, oh, sorry, earlier attempts in just, for example, sucking out portions of the cells, taking almost biopsies from the cell to do, you know, a fraction of the cytoplasm measurements. So that might be similar to the idea you talk about. There's also work from Poplinis Lab to have the cells secrete little vesicles. And you harvest that in the cell, in the cell culture media to measure what the cells are, you know, producing longitudinally. But there hasn't been technology that can actually measure the entire cell's transplom while still keeping the cell. You can't have the cake and eat it. Yeah.

• 1:28:49Speaker 03

  Cool. Yeah. Thank you for taking the time to chat with us. It's been great. Learned a lot. It was a lot of really interesting discussions that I think, and I especially really appreciate your commitment to Open Science and all of the cool models and data you've released. Is there any last, like, thoughts you have or anything you'd like the audience to know, follow up with?

• 1:29:08Speaker 01

  Overall, I think virtual cell is such a new and fast-moving field. We hope to have more and more people join us. And our Excel paper is out. And we look forward to receiving your comments and the feedbacks. And also, we're hiring.

• 1:29:23Speaker 00

  Yeah, we are always looking for talented engineers, technologists, biologists, drug hunters, AI scientists, computational biologists. So look on xera.com, look for the open roles. We'll be happy to chat with you.

• 1:29:37Speaker 02

  Thank you.

• 1:29:37Speaker 01

  Thank you very much.

• 1:29:38Speaker 00

  Great. Thank you.