Prof. Randall Balestriero - LLMs without pretraining and SSL

Randall Balestriero joins the show to discuss some counterintuitive findings in AI. He shares research showing that huge language models, even when started from scratch (randomly initialized) without massive pre-training, can learn specific tasks like sentiment analysis surprisingly well, train stably, and avoid severe overfitting, sometimes matching the performance of costly pre-trained models. This raises questions about when giant pre-training efforts are truly worth it. He also talks about how self-supervised learning (where models learn from data structure itself) and traditional supervised learning (using labeled data) are fundamentally similar, allowing researchers to apply decades of supervised learning theory to improve newer self-supervised methods. Finally, Randall touches on fairness in AI models used for Earth data (like climate prediction), revealing that these models can be biased, performing poorly in specific locations like islands or coastlines even if they seem accurate overall, which has important implications for policy decisions based on this data. SPONSOR MESSAGES: *** Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich. Goto https://tufalabs.ai/ *** TRANSCRIPT + SHOWNOTES: https://www.dropbox.com/scl/fi/n7yev71nsjso71jyjz1fy/RANDALLNEURIPS.pdf?rlkey=0dn4injp1sc4ts8njwf3wfmxv&dl=0 TOC: 1. Model Training Efficiency and Scale [00:00:00] 1.1 Training Stability of Large Models on Small Datasets [00:04:09] 1.2 Pre-training vs Random Initialization Performance Comparison [00:07:58] 1.3 Task-Specific Models vs General LLMs Efficiency 2. Learning Paradigms and Data Distribution [00:10:35] 2.1 Fair Language Model Paradox and Token Frequency Issues [00:12:02] 2.2 Pre-training vs Single-task Learning Spectrum [00:16:04] 2.3 Theoretical Equivalence of Supervised and Self-supervised Learning [00:19:40] 2.4 Self-Supervised Learning and Supervised Learning Relationships [00:21:25] 2.5 SSL Objectives and Heavy-tailed Data Distribution Challenges 3. Geographic Representation in ML Systems [00:25:20] 3.1 Geographic Bias in Earth Data Models and Neural Representations [00:28:10] 3.2 Mathematical Limitations and Model Improvements [00:30:24] 3.3 Data Quality and Geographic Bias in ML Datasets REFS: [00:01:40] Research on training large language models from scratch on small datasets, Randall Balestriero et al. https://openreview.net/forum?id=wYGBWOjq1Q [00:10:35] The Fair Language Model Paradox (2024), Andrea Pinto, Tomer Galanti, Randall Balestriero https://arxiv.org/abs/2410.11985 [00:12:20] Muppet: Massive Multi-task Representations with Pre-Finetuning (2021), Armen Aghajanyan et al. https://arxiv.org/abs/2101.11038 [00:14:30] Dissociating language and thought in large language models (2023), Kyle Mahowald et al. https://arxiv.org/abs/2301.06627 [00:16:05] The Birth of Self-Supervised Learning: A Supervised Theory, Randall Balestriero et al. https://openreview.net/forum?id=NhYAjAAdQT [00:21:25] VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning, Adrien Bardes, Jean Ponce, Yann LeCun https://arxiv.org/abs/2105.04906 [00:25:20] No Location Left Behind: Measuring and Improving the Fairness of Implicit Representations for Earth Data (2025), Daniel Cai, Randall Balestriero, et al. https://arxiv.org/abs/2502.06831 [00:33:45] Mark Ibrahim et al.'s work on geographic bias in computer vision datasets, Mark Ibrahim https://arxiv.org/pdf/2304.12210

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  We just launched this experiment and then we are very surprised to see that the hugely over parameterized model not only trained out of the box, you have very nice training curves, but also they don't overfit aggressively at all. And what we found empirically is that we just out of the box use typical supervised training. We don't have to play with hyper parameter optimizer and you have very, very stable training.

• 0:26Speaker 00

  So this brings also the question, is it worth it to spend so much money to gather gigantic pre-training data set, spend like months on many GPUs to produce those models. But at least for some application, it seems to not be much better than random.

• 0:45Tim Scarfe?

  MLST is sponsored by TwoForAI Labs. Now they are the DeepSeek based in Switzerland. They have an amazing team. You've seen many of the folks on the team. They acquired Minds AI, of course. They did a lot of great work on ARC. They're now working on O1 style models and reasoning and thinking and test time computation. The reason you want to work for them is you get loads of autonomy, you get visibility, you can publish your research. And also they are hiring as well as ML engineers, they're hiring a chief scientist. They really, really want to find the best possible person for this role. And they're prepared to pay top dollar as a joining bonus. So if you're interested in working for them as an ML engineer or their chief scientist, get in touch with Benjamin Cruzier, go to twoforlabs.ai and see what happens.

• 1:29Speaker 00

  Originally, the main motivation was to see, okay, how much information you gain by doing a pre-training, right? And is this next token prediction really making your network learn something about language and reasoning? And so then we are saying, okay, one way to compare this, at least empirically, is to just take a randomly initialized model, train it from scratch on a supervised task like sentiment prediction, sentiment analysis. And then in theory, because we have a very, very small training data, let's say like 20,000 samples, and because those models have like 7 billion parameters, the pre-trained one will perform very nicely with a little bit of flora fine tuning. Because it already knows the two reason about the world, right?

• 2:12Speaker 00

  So maybe you just adjust a little bit to the specific task that you want. But since you have so many prior knowledge, you will solve the task very easily. But the random one either will overfit completely, because you have like seven billion parameters and only 20,000 training samples, or maybe it will not learn at all, because, you know, the training dynamics will be completely chaotic. And so we just launched this experiment and then we are very surprised to see that the seven billion or like the hugely over parameterized model not only train out of the box, like you have very nice training curves, almost like you train MNIST, but also they don't overfit aggressively at all. Like they overfit less than if you just train MLP on MNIST, basically. And this is very surprising.

• 2:57Speaker 00

  And so basically from this, we said, okay, actually, maybe there is a deeper question, which could be how much implicit bias you have in those language models? Because already we knew from computer vision, that for example, ImageNet, you can have a 50 million model on a 1 million data set. So you have like this 50 to 1 ratio. And you have the implicit bias that prevents you from overfitting and just solving the task, right? But still, it's 50 to 1. So this may sound a lot for, you know, like statisticians. But now it's like 7 billion to 20,000. So like the ratio is like gigantic, right? And yeah, no, to me, it was very surprising that the scale,

• 3:38Speaker 00

  like the size of this ratio, still allows you to learn something that does not overfit. And this is very surprising, because in vision, for example, transformers are known to overfit Maurice Lee's and ResNet. So they seem, at least in vision, to have actually less implicit bias or implicit regularization. But at least with this type of next token causal architecture, LLM.

• 4:04Speaker 00

  Yeah, you don't seem to work it easily to your data. So this was quite surprising.

• 4:09Tim Scarfe?

  Yeah, and we should bring in the name. So this was your workshop paper at the self-supervised learning workshop here at NeurIPS and it's called For Perception Tasks: Is LLM Pre-Training by Next Token Prediction Worth the Cost? So this is absolutely fascinating, right? So we've been given this belief that we need to have these huge pre-trained models, they're trained on all the data on the internet, and it turns out that certainly for discrimination tasks, so things like classification rather than generation, actually you can just start from scratch with a fairly small model and you get sometimes even better results.

• 4:44Speaker 00

  Yeah, and even small or even large model, like you just start from scratch, you do this very simple supervised classification task, right? Okay, given this prompt, is it good or a bad sentiment, or what type of job is the prompt describing? You know, this type of, I would not call it reasoning, but yeah, more semantic classification. And it turns out that, yeah, you start from random. Even if you have a small training data set, you will have performances that are sometimes as good than a pre-trained model. So this brings also the question, is it worth it to spend so much money to gather gigantic pre-training data set, spend like months on many GPUs to produce those models? And for

• 5:29Speaker 00

  some cases, for generation, all right, there is no question, this is what you need to do. You have your next token prediction, you learn how to generate samples. But at least for some application, it seems to not be much better than random. So it's quite interesting.

• 5:44Tim Scarfe?

  So what are the differences in the learned representations?

• 5:46Speaker 00

  So that's something we did not really look at, like low dimensional representation of what you learn. It's possible. So some work tried to look at the attention, entropy, and the, like, you know, those mechanistic interpretability viewpoint of LLMs. So it will be interesting to see if you have this sort of, you know, neural collapse thing that happens. So even if you're like a seven billion parameter, maybe you end up learning a very, very, very simple Subnetwork that does the task, a bit like lottery ticket hypothesis as well. And that naturally emerge from the training dynamics. Or is it really exploiting all the parameters? I think that's one thing. So to extend the workshop paper to conference, we want to probe into more like. What are the useful parameters?

• 6:31Speaker 00

  What did they learn? Are each layer actually learning something? Or maybe the first layers don't really learn anything, just the last few ones are learning something. So yeah, there is lots of open questions for this.

• 6:42Tim Scarfe?

  What does it tell us about the nature of understanding? And it may be even intelligence, because we think that the reason why these things understand is because they're. They just have all of these representations to all of these different, you know, things in their experience.

• 6:56Speaker 00

  Yeah.

• 6:56Tim Scarfe?

  And and now we can shortcut to, you know, to want a better word. What does that tell us?

• 7:02Speaker 00

  Yeah, I think that's a good question. So in this case, we must look at very specific classification tasks. So for example, you have a description of a job, what job it is. Is it like a good or bad sentiment? And this you are able to solve it good, but you are not able to go out of distribution to solve a new type of question. For example, for this job description, then you cannot answer, okay, is this job paying you more than this job? Because this was not present in the training data, right? So I think you get very good models cheaply, quickly from random initialization, but they will be very specialized. And I think the benefit of having maybe the pre-training may come if you want to do more of like open-ended. Classification or reasoning.

• 7:44Speaker 00

  So I think it really depends on the type of application you want to solve. What's your downstream task and how much you want to generalize to new scenarios? But at least now it shows that, yeah, it's not just pre-training with next to can prediction is better for everything.

• 7:59Tim Scarfe?

  So, I mean, going back five years, data scientists used to build specific classification models for doing everything. And now we're in this regime of we need these really big models. And we do in-context learning and maybe even some fine tuning. And we get them to do fairly specific discriminative tasks. But now you're saying we should almost go back to where we were five years ago and start building specialized models again. Only now, rather than building classification models, or actually we're still using. The transformers and the LLMs, but we're making them do specific tasks.

• 8:28Speaker 00

  Yeah, exactly. I think if you only want to solve a few specific tasks, use this prior knowledge to have a nice architecture, supervised data set for that and just do that from scratch. This is something that's gonna probably work much better. But again, you need to make sure that the downstream application will never go too out of distribution. So that's why it really depends on the application and the type of use cases that you have. But I think at least here it shows that there exists some task where an next token prediction is not the answer. And in fact, it's not just not the answer, but it's not better than random initialization, which is really sort of the worst case scenario.

• 9:11Tim Scarfe?

  Interesting. I mean, from a fairness and bias point of view, a lot of people say that, you know, large language models are bad in a way because there's the dominance of North American cultures and so on. But you could also argue the converse, which is that the good thing about them is that they do have some awareness of value. You know, so we can fine tune them to have guardrails and to sort of say the right thing and so on. Is that harder to do with this approach?

• 9:34Speaker 00

  Yeah, so here, because you are in a fully supervised setting, you don't have as much flexibility to, let's say, change the behavior of your model or it will have to take the form of supervised fine tuning. But because you don't have a generative capability, it certainly restricts the type of interaction you have with the model and how you can improve it, right? Because the output is just, okay, is it a good or bad sentiment? It's not something that gives you a full answer that then you can try to argue against and generate a fine tuning data set from. It's just, okay, good, bad, and that's it.

• 10:09Tim Scarfe?

  Another thing is training strategies. So you know, like the big players building these LLMs, they have lots of internalized knowledge around, you know, even the order in which you train the language models. Everything is important, you know, certainly in the old days of like basic models, you know, you just stick a load of data in there, no one really cares. So, you know, now do people need to be sort of thinking about the specialized knowledge? Maybe thinking about curriculum learning and all of this kind of stuff?

• 10:31Speaker 00

  Yeah, so this is a good point. So we did a paper recently called the Fair Language Model Paradox, where we show that when you do this next token prediction, because you have some tokens that are very low frequency, it's very hard to train on them and it takes a very long training. So it's very wasteful, right? And the problem is that because you do this next token prediction, you need to really capture all your distribution of tokens and so you spend a lot of time. But in this case, if the low-frequency tokens are not useful to solve your task, you actually don't need to capture it at all. So in terms of training dynamics, this is actually a much simpler problem in many cases. And what we found empirically is that we just out of the box, use typical supervised training.

• 11:14Speaker 00

  We don't have to play with hyperparameter optimizer. And you have very, very stable training. So that's one thing that could be also interesting for future work is to see, is this something that is easier to optimize? And maybe that's why those like 7 billion parameter model can learn and not overfit on like 10,000 samples. And then this also bring other things that maybe this on its own could be a better initiation for next token prediction as well. So this is a very open up in the air. But maybe you could think of a simpler supervised objective that will be a better pre-training solution. And then you can use for like next token prediction if you wanted to. But at least this would be a better starting point from random.

• 12:00Speaker 00

  So you almost reverse the train.

• 12:02Tim Scarfe?

  So we've spoken about two extremes. So on the one extreme, we have pre-training and you can like use it for any downstream task. And on the other extreme, we have, you know, you start from scratch just with one task. Is there an intermediate solution? So what if I did this new approach but for multitask, let's say for five tasks?

• 12:17Speaker 00

  Yeah, yeah. So that's a great question. So if you really think about it in the limit, you could formulate a next token prediction as a multitask, where you want to each task expecting is the next token this one or not. So in the extreme case, you could just recover a next token prediction on one end. And on the other end, you have what we have here. So just one task, very coarse, high level. Predict if it's a good or bad sentiment or whatever. So in between, you have a huge spectrum that you can exploit. And if you can find, as you said, maybe five very different representative tasks. This should be enough to or could be enough to learn a representation that is as general as possible.

• 12:58Speaker 00

  And then you can use this for maybe new tasks that come on the go. So I think the research question is how to design the minimum amount of tasks. So that you have as diverse representation as possible. And of course, you don't want to go to the extreme of just doing again next token prediction. But this, this is a very, very nice research question because if you have this spectrum and you can control where you want to be. Then you can really have a pair use case choice. So it's not, okay, you are always here or always here. Tell me what you want to do, how much new task you expect your model to be exposed to. And I tell you where you need to be in this spectrum. So this could be like very interesting as well.

• 13:38Tim Scarfe?

  Very cool, very cool. It does make me think though that these models understand through naive statistical alignment. And is it possible that the benchmarks we use just don't cap, you know, they, the, the gap of understanding that we've lost from moving from the pre-trained models isn't being captured?

• 13:55Speaker 00

  Yeah, I think because especially in the recent years, we focus a lot on generative decoder only methods. All the evaluation and the type of objectives we put on ourselves in really is really about good generation, right? Even if you want to answer a question, you need to generate a good explanation. You need to understand what are the intermediate steps. And I think the fact that we focus on generative models means that we completely bias the evaluation and the way we approach this thing. And maybe you could have still knowledge that is learned without being able to generate anything. So I think this is also something that could be interesting to look at, or at least keep in mind when we explore those models.

• 14:38Tim Scarfe?

  But philosophically though, isn't generation analogous to thinking in some sense? So don't, don't models that generate, aren't they smarter in some deep way?

• 14:46Speaker 00

  Probably what you want to do is maybe imagine. What could be, but I don't think you want to do generation is with very granular details like next token generation. Because if you think about it, even just in terms of like a classification task, you have a lot of different uncertainty depending on the token. If I start the sentence, okay, I saw this movie for minutes, there is no way you can tell what was the next token for after four, right? So this means that you know a priori it would be like a time component, right? Maybe it's like one hour, 10 minutes, two hours. But you really need to be able to generate the, I don't know, 52 minutes or whatever the answer was to actually understand that I was seeing a movie, therefore I was staying

• 15:29Speaker 00

  in a place for at least more than five seconds, right? So I think token is way too granular. And if you had maybe like concept tokens, that's where you could start seeing, okay, this is meaningful because that's closer to maybe what we do. But right now we are very, very, very low level because tokenization is a lossless compression, right? So this is too close to the raw data. And yet, yet we have the life easy compared to computer vision because already you work in language, which is very compressed representation of knowledge, but still token is probably too low level still.

• 16:04Tim Scarfe?

  Well, that was a fascinating paper. Let's move on to your next one. So the birth of self-supervised learning as supervised theory, and that was with Yan Lacun.

• 16:11Speaker 00

  Yes.

• 16:12Tim Scarfe?

  And yeah, basically you said that the observed differences between self-supervised learning and supervised learning are not due to the loss function themselves, but rather the labeling of the data set used in training. Give us the elevator pitch.

• 16:22Speaker 00

  Yeah, so basically what we show in this paper is that you can have a supervised objective, like let's say Liss squares to make it simple. So you have the inputs, you have your network's prediction and you have the labels. And you can turn this objective, which tries to predict sample XN to prediction YN into a self-supervised learning objective, which tries to compare samples with each other. So basically you go from saying, okay, this image is a car or a dog, to saying, are those two images the same or not, which is like the self-supervised type of joint embedding world. And so you can show that if you have labels. Or you have knowledge of this pairwise relationship.

• 17:02Speaker 00

  They are actually learning the same representation up to some symmetry that is irrelevant if you do linear probing. So the loss function in itself, the SSL one or the supervised one, try to do the same thing. They just operate on a different view of the labeling. Whether this image is that or are those two images or two samples representing the same thing. So given that, then the next question is how come self-supervised learning is able to generalize better than supervised? And from this perspective, what you can say is that it's because it's as if they were solving a supervised task where the labels are not about predicting all the cars to cars, but are very, very, very fine-grained label wherein the limit, each image is its own class, basically.

• 17:47Speaker 00

  So if you think about supervised learning in this extreme setting, you also don't overfeed to the task because you don't collapse any image to another one. And so theoretically speaking, you can solve many downstream tasks as you want. So this equivalence of loss at least brings a slight new perspective on the fact that it's not really about the objective. It's more about how you design the SSL pipeline or you say, okay, with this sample is related to this sample. But it's not the objective that makes you learn a better representation.

• 18:16Tim Scarfe?

  Okay. And in the paper, you were talking about how SSL can maximize the worst-case downstream task performance. Can you sketch that?

• 18:22Speaker 00

  Yeah. So basically, if you think about all the possible realization of downstream task, you could have some very coarse scale ones. You have maybe different pictures of cars and buses. And you just want to say, okay, it's a car or a bus, so no details need to be encoded to solve this. But then you can have downstream task where you want to say, okay, which brand of car is it or which color of car is it? So you have a distribution of downstream task, right? And so the point now is that you want to learn the representation. So that if you look at the distribution of downstream task performance, you are able to be as good as possible on most of them, right? So you don't want to be very good on some and then in the tail, you are very bad on the majority of them. And so then from this, you can try to say, okay, what will be the labeling

• 19:04Speaker 00

  that tries to make your worst case as good as possible? And from this, you can say, okay, this is actually the labeling that self-supervised is actually implicitly doing.

• 19:14Tim Scarfe?

  How does the class balance affect the difference in the losses?

• 19:18Speaker 00

  Oh yeah, so this is a very good point, actually. In a follow-up paper we are doing right now, we show that current SSL objectives assume class balancedness. And this is something we already highlighted quickly in this SSL supervised learning as a uniform cluster prior paper we did a couple of years ago. And we show that current SSL objectives assume balanced representation of classes or concepts. And this means that if you train on ImageNet, things work out very well because concepts are sort of equally represented. But then if you go to other data sets like iNaturalist, which are very heavy tail, then you have a huge bias in your representation. So until now, people did not really know how to solve this. And so one way people approach this is through data correction.

• 20:03Speaker 00

  And they say, OK, I'm just going to remove the over-sampled concepts to try to make it more uniform. And then I do self-supervised learning on this. But because now we have this theoretical formulation and this equivalence of losses, we can use the exact same settings that people used in supervised learning to re-weight depending on the frequency of classes. We can use that to come up with a new self-supervised learning loss that takes this imbalance into account. So this type of thing is enabled from this mathematical formulation, and it's principled. So the way we do this weighting, you can prove that it's the right way to do it from this supervised theory. And so this is really nice because suddenly, from this seemingly naive connection, you can now come up with new generation of self-supervised learning models,

• 20:48Speaker 00

  where you can actually match what the real world that data distribution is like. So non-uniform distribution of classes, maybe even if you have some samples that are more noisy than others, you can include that information as part of the SSL objective as well. So suddenly, you have a new world of possibilities that comes. And because there is this connection, you can actually prove, OK, this is the right way to do it, at least from this supervised theory viewpoint.

• 21:12Tim Scarfe?

  You also pointed out a connection to V-Creg.

• 21:14Speaker 00

  Exactly. So basically, what we do in the paper is that we show if you have least-squared supervised type of objective, and you turn it into SSL1, what you obtain is basically V-Creg. So then you have a few variations. It could be V-Creg or WMSC, depending on how you do this from supervised to SSL. But you can show that depending on the type of supervised loss, you recover different type of SSL1s. If you look maybe more at cross-entropy supervised learning, it's gonna be more like a sim-clear type of loss. But you have this one-to-one correspondence. And this is also very nice, because in supervised learning, at least you know when one loss may be preferred compared to another one. And this has been studied for a long time, right? Because supervised learning has been around forever.

• 21:59Speaker 00

  And so now we can reuse those insights for self-supervised learning. So this, to me, is also a very, very strong benefit of this thing, is that suddenly all the theory and like the thousands of papers that have been done in supervised learning, we can just take it and apply it in SSL. Another example is neural collapse, for example, that has been proven in supervised setting. Now it applies like in five lines in a SSL setting as well. So this connection is really beyond just trying to say, okay, it's not the objectives that make SSL better. It's really tying those two huge communities together towards a goal where you have a single unified objective to learn representation.

• 22:41Speaker 00

  And this is nice too, because if you speak to people, they will think, okay, you have supervised learning on one side, SSL on the other side, and basically you are either in one camp or the other. But now what we show is that you actually SSL is pretty much everything in representation learning, and supervised is just one realization of SSL. Then V-Creg without labels is another one. Then this one is another one. So you really have a better understanding of this relationship and what representation learning is trying to do.

• 23:12Tim Scarfe?

  Galaxy Brain question incoming. Could you combine SSL and supervised objectives in some way to improve generalization?

• 23:18Speaker 00

  Yes, yes. So there is one paper, which is supervised contrastive learning. So the way they do it is that they use the labels within a simple framework to try to basically do fully supervised learning, but with a simple objective. So first of all, we can show that indeed this makes sense and that basically we can explain the empirical results that they have. But actually, we can do a little bit more than that. So if you are in a semi-supervised setting, for example, it may not be clear how to combine those two losses anymore. Or maybe you could say, OK, I have the two and I have a coefficient to weight them. But then you need to do cross-validation and so on. But now, from this perspective, you can combine them in a very principled way and you can understand which weighting makes sense, depending on how much sample you have in one or the other.

• 24:02Speaker 00

  And you can use all the literature, again, from supervised learning for this setting as well. So this is something you can do very easily with this formulation as well.

• 24:12Tim Scarfe?

  OK, so if SSL and supervised are two sides of the same coin, I mean, of course, we can use this theoretical framework to design new forms of SSL framework. But does it, you know, is the distinction relevant if they are the same thing?

• 24:24Speaker 00

  I think it's not just two sides of the same coin. SSL is more general than supervised learning. So it's really SSL could be the more general objective to learn representation. The more prior knowledge you have, the more you know about your downstream task, the more you know about your labels. And then SSL like slowly becomes supervised learning through the labels that you use for the SSL objective. But then, because, as you said, you have this hierarchy now, it does not really make sense to say you have either supervised learning or SSL. Rather, what makes sense is to say, OK, what's this relation matrix? What is pairwise matrix? If you build it from labels, it's supervised learning. If you build it from other prior knowledge, for example, two consecutive frames in a

• 25:07Speaker 00

  video basically have the same class, then you are more in an unsupervised SSL setting. But it's all about how do you build this pairwise relation matrix? That's the main question.

• 25:18Tim Scarfe?

  Very cool. Right, let's move on to the next paper. So No Location Left Behind. During and improving the fairness of implicit representations for earth data. So there's loads and loads of modeling frameworks now that do these implicit neural representations of geospatial earth data. So things like climate modeling, resource allocation, environmental modeling. I was actually interviewing Johannes from NXAI yesterday. I don't know if you know him, but he's working on similar stuff. The problem is you've studied this and you've found that there's loads of biases and fairness problems.

• 25:45Speaker 00

  Yeah, exactly. So basically what we show is that when you want to model, for example, let's say, temperature or precipitation, to make it simple. And you want to learn, for example, implicit neural representation, it means that you want a model so that if you give a location and a date, for example, it can predict what was the temperature there. So if you have this type of implicit neural representation, it's very good because if you learn a nice model, then you can actually interpolate those values, so maybe estimate what the temperature was in this part of the globe where you did not have a sensor. But you can also do extrapolation as well. If you assume you really learn the true physical model of the world, you could start saying, okay, what the temperature will be two years from now, right?

• 26:26Speaker 00

  So this is very nice to have this type of model for all sorts of applications. The thing is that when you do this nowadays, depending on the architecture and the different design choices that you do, you will maybe have a very good prediction on average. So when you look at the red performance around the whole globe. But actually, if you look, for example, around islands or coastal area, your prediction is going to be very bad, almost random. So this is something that can be very concerning because if you use this type of model to decide about a policy that will affect a specific island, using this model prediction is as good as using like random guesses. So it can be very detrimental and people need to be aware of those biases.

• 27:08Speaker 00

  So what we found is that, for example, for this type of climate data, islands are often disregarded, coastal area, basically a region where you have a big gradient in the type of data that you try to model.

• 27:21Tim Scarfe?

  How much of a responsibility do modelers have to detect these kinds of biases in the data?

• 27:27Speaker 00

  So I think there is like two components, as you said. So one could be that just the dynamic of the data you are trying to model is harder near Ireland, or maybe it's even unpredictable because you don't have enough observations to do that. So you have some uncertainties that probably you can never recover from good design. But still, what we found here is that a lot of the biases now comes from the architecture and all you want to do to encode those positions, the type of basis you use to do the prediction. So right now it seems that a big chunk of the bias comes from the architecture. But I totally agree that I don't think we can remove the bias entirely because there is maybe just different type of uncertainty, a different part of the planet as well.

• 28:11Tim Scarfe?

  I mean, the world is a very, very complicated place. I mean, realistically, to what extent can we mathematically model it?

• 28:16Speaker 00

  Yeah, so that's a good question. So I think it depends the type of horizon that you have and the type of data that you want to model. If you have a system that is much more chaotic or can vary very quickly without much changes in the past observations, that's something that current models are having a very hard time with. If you want to predict something else, for example, temperature in North America, not near the coastal area, so really inland, maybe that's why you have less gradient dynamics, things are a bit more stationary, especially in short time, so then it can become much better. But I think at this point, we don't have an architecture that is really able to understand that you have different physics, different dynamics, models at different parts of the globe.

• 29:00Speaker 00

  And so because of this, you just see what's the best on average and it means you miss out a lot of details.

• 29:06Tim Scarfe?

  Can you tell us about some of the technical framework?

• 29:08Speaker 00

  So one thing we showed, for example, at least for this type of globe data representation, is that people use Fourier basis to model the prediction. And this is something that is better than not using any basis at all. But what it means that you imply the type of signal you're predicting is very stationary and not localized at all. And this is a very strong prior, right? So this may be. Be true for some things, but for other things like precipitation or temperature, where you have localized very high gradients, then it's a strong bias. And if you come from signal processing community, you know very well that to have better localization, you go from Fourier to wavelets. And so that's one thing we did in this paper.

• 29:50Speaker 00

  And we showed that using wavelet basis to encode those data allows you to have better localization and this removes some of the biases. And here it's more of a proof of concept that different design choices give you different type of bias trade off. Wavelet is not the answer to everything, right? But I think the next step is to really be able to encode less and less a priori which basis to use and let the model learn from the data on its own. And we are not yet at this point, at least for this type of climate data.

• 30:24Tim Scarfe?

  How could it handle noisy or missing data?

• 30:26Speaker 00

  This depends really on the type of model you use. So for example, if you have INR, then you will not use the missing data as part of your training pipeline. And that's one of the benefit of them. So if one of your sensors stopped recording during some years, you just don't use that as part of your training data because you really control where do you have the data? And when you have it, what the prediction should be.

• 30:48Tim Scarfe?

  So these earth models, they are now informing policy around the world. Who should we hold accountable? I mean, is it the technology? Is it the scientists who design the models? Is it the policymakers who interpret the results?

• 31:00Speaker 00

  I think it's very hard for the person who designs the model to know a priori what it's going to be used for. So I think it's more downstream. When you know clearly what you want to do with it, you should first set up a nice evaluation pipeline to make sure that it's something you can actually use to make those decisions. And then you can report any type of failure mode you observe for people to improve on the design. But a priori, it's very hard to imagine what this model will be used for. So in an ideal setting, you wish that there will be no bias at all. But in practice, the world of possibilities being so large, it needs to be more of a feedback loop. And then

• 31:41Speaker 00

  iterate until you have something that you can really trust and then you can act on it.

• 31:46Tim Scarfe?

  Earth modeling data is very anthropocentric, right? So, you know, we we focus on on human populations and so on.

• 31:51Speaker 00

  Yeah.

• 31:51Tim Scarfe?

  Should we also focus on, you know, like just ecosystems and places that got nothing to do with humans?

• 31:56Speaker 00

  Oh, yeah, that's a great question. And in fact, that's one of the big issue with a lot of the data set, which is crowdsourced. Because by definition, the amount of data that you get is proportional to the number of users you have depending on the location. And this means you have a huge bias in what your model is learning and what your model is focusing on, which means you miss out on a lot of things. So I think that's also one thing that, okay, crowdsourcing can give you a lot of data quickly, but it's very biased data. So then the question is, how much of this biased data versus maybe paying a lot more and capturing other part of the globe? How much of the two you should have?

• 32:34Speaker 00

  And maybe you could be able to show that under some specific condition, just having 10% of the data, which is high quality, uniformly sampled, and then 90% which is crowdsourced, you can try to use those 10% to uncover your representation and then use all that data together. But there is a huge amount of research question in that, because that's a very big source of bias.

• 32:56Tim Scarfe?

  And there's a bit of a policy question, but we are using these things, you know, to do resource allocation, right? You know, so giving more resources to to some populations might be taking it away from others. And then there's the fairness over time thing as well, which is that what is fair like now might not be fair in 100 years' time. So how should we think about that?

• 33:14Speaker 00

  Oh, yeah, that's a good question. I think this is also very application specific. So for example, if you want to predict where to build a house to solve some specific problem, maybe you don't really mind having bad prediction where there is no population anyway, because you are not gonna build a house there. So in this case, maybe the crowdsourcing type of data is actually good. But this could really be dependent on the type of application. And just one thing I will say regarding the point you made before, this type of bias actually is something that you have in computer vision. So there is like a very nice paper done by Mark Ibrahim. Basically, they showed that most Of the data we have like from Imagine it is from North America. And so maybe you reach like 90% state of the art performance to predict, for example,

• 33:58Speaker 00

  type of chairs, cars, but only for North American models. And when you start looking at type of cars or chairs in Central Africa or East Asia, suddenly the model performance is extremely bad. So this type of problem is something you have across modalities. And that's something that's a very big, big issue.

• 34:17Tim Scarfe?

  Randall, it's always a pleasure and an honour to have you on the show. Thank you so much.

• 34:21Speaker 00

  Likewise, likewise. Thank you so much.