Why Scaling Prediction Cannot Create Intelligence - Alexander Mattick

Alexander Mattick is a researcher at Fraunhofer IIS and a PhD researcher at the University of Technology Nuremberg (UTN), and a regular on Yannic Kilcher's Discord. He first came on MLST in 2022, after helping research the Yann LeCun and Randall Balestriero episode on interpolation. SPONSOR: --- Cyber Fund built the Monastery to help founders ship products that were impossible a year ago. Applications for Batch 1 are now open. Apply now: https://cyber.fund --- Alexander treats inference as the thread running through modern machine learning: once you have a model, what does it cost to get an answer out of it? He works through Monte Carlo, GFlowNets, energy-based models, diffusion, normalising flows and flow matching, with four short explainers he recorded himself. He is blunt about energy-based models: you can sample from them in principle, but it is rarely worth the compute. JEPA and "world model", he says, are closer to branding than to technical categories. Next: theories of deep learning, none of which he thinks predicts enough yet to guide practice, then reinforcement learning. --- 0:00 Cold open: information is expensive 0:51 Welcome back, Alexander Mattic 2:08 Alexander's research background 2:50 Inference: densities, sampling and Monte Carlo 6:42 GFlowNets, energy functions and MCMC 9:45 Explainer: energy-based models 11:03 Why model a density at all? 17:30 From learned energies to flow matching 25:08 Explainers: diffusion and normalising flows 28:33 Are energy-based models generative? 33:22 JEPA, contrastive learning and collapse 41:13 Why non-language modalities need flows 44:51 Inference as search: branch and bound 49:43 Q-learning and delayed consequences 55:14 Flow matching, optimal transport, Fokker-Planck 1:00:03 Explainer: flow matching 1:01:49 AlphaFold, latents and scale versus architecture 1:07:52 Two families of deep learning theory 1:15:04 What a good theory would predict 1:23:53 The manifold hypothesis and compression 1:28:25 Is reward enough? 1:32:01 Control theory versus reinforcement learning 1:37:22 The Bitter Lesson and expensive information 1:42:08 Constrained RL: the constrained MDP toolbox 1:50:12 Creativity as constrained search 1:55:44 Reality is protean: when abstractions hold 2:00:32 What is a world model? 2:04:38 Prediction is not control 2:08:13 Robot demos, MPC and reliability --- REFERENCES: [6:55] GFlowNets (Bengio et al., 2021) https://arxiv.org/abs/2106.04399 [38:46] Contrastive Self-Supervised Learning (Anand, 2020) https://ankeshanand.com/blog/2020/01/26/contrative-self-supervised-learning.html [38:56] LeJEPA (Balestriero and LeCun, 2025) https://arxiv.org/abs/2511.08544v3 [47:10] RL for Node Selection in Branch-and-Bound (Mattick) https://openreview.net/forum?id=0ez68a5UqI [56:20] Flow Matching for Generative Modeling https://arxiv.org/abs/2210.02747v2 [1:12:41] Disentangling feature and lazy training in deep neural networks https://arxiv.org/abs/1906.08034v4 [1:31:05] Reward is enough (Silver) https://doi.org/10.1016/j.artint.2021.103535 [1:35:12] Learning ReLU networks to high uniform accuracy is intractable (Berner et al.) https://arxiv.org/abs/2205.13531v2 [1:40:20] Dota 2 with Large Scale Deep RL https://arxiv.org/abs/1912.06680v1 [1:45:41] Constrained Update Projection for Safe Policy Optimization (Yang et al., 2022) https://arxiv.org/abs/2209.07089 [1:46:11] SafeMPO (ICLR 2026) https://openreview.net/forum?id=1m0EU6QXj6 [1:50:17] Why Creativity Cannot Be Interpolated https://archive.mlst.ai/paper/why-creativity-cannot-be-interpolated/ [1:51:39] Invalid Action Masking (Huang and Ontañón) https://arxiv.org/abs/2006.14171 [2:00:04] Probability Theory: The Logic of Science (Jaynes, 2003) https://www.cambridge.org/core/books/probability-theory/9CA08E224FF30123304E6D8935CF1A99 [2:01:53] Training Agents Inside of Scalable World Models (Hafner et al., 2025) https://arxiv.org/abs/2509.24527v1 [2:03:43] World Models (Ha and Schmidhuber, 2018) https://arxiv.org/abs/1803.10122v4

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Speaker 00

  This episode is brought to you by Michelob Ultra. Good times mean competitions between you and your friends, like seeing who can throw some pigskin the farthest or rush the field the fastest. Winner gets an ultra because it's not just a superior like beer, it's also giving fans superior access to their favorite sporting events. Superior is worth playing for. Visit michelobultra.com/superioraccess to enter. Enjoy responsibly. Copyright 2026 Anheuser Busch InBev Michelob Ultra Beer, St. Louis, Missouri.

• 0:28Speaker 01

  Flowing ad budget on metrics that look great till the CFO sees them. That's bull spend and marketers are calling it out in dashboard confessions.

• 0:39Speaker 02

  I remember telling my boss, it'll be good for the brand when leads were slow. Yeah, it wasn't.

• 0:46Speaker 01

  Cut the bull spend. LinkedIn lets you target by company, job title and more. Advertise on LinkedIn. Spend $250 on your first campaign and get a $250 credit. Go to LinkedIn.com/campaign. Terms and conditions apply.

• 0:58Speaker 03

  Effectively, information can be very expensive. And in fact, I would say most information is very, very expensive. If you completely say, well, constraints don't matter. I don't need prior information. I don't want to encode anything in there. We still have the problem of now needing to relearn everything from scratch. He believes that you can figure out like a suitable reward function for every task. That is enough. I think that energy based models are not really useful, or at least not useful anymore. I think that world models and so both JEPa and world models are ultimately branding. There were JEPa like models before JEPa and there were world models before world models. Right now, all of these methods are not set up like this.

• 1:42Speaker 03

  I think they are set up to look very nice in demos and to actually do some really impressive stuff. How do you know, like how many times have you failed to do something?

• 1:51Tim Scarfe?

  Amazing. Well, Alexander Matic, it's amazing to have you back on MLST. It's been many years since you were on the show and we were so grateful for you when you last came on the show because you helped us do some research for our interview with Jan LeCun and Randall Balestriero about the spline theory of neural networks. And we did that intro all about interpolation and extrapolation. But just for folks at home, so your nickname is Zixack on Yannick Kilcher's Discord server. And this gentleman is by far the smartest person on Yannick's Discord server. And that is saying something, believe you me. So Alexander, it's wonderful to have you on the show. Can you give us a bit of a background? This episode is supported by Cyberfund.

• 2:34Tim Scarfe?

  If you're building at the frontier of AI, they want to hear from you. Cyberfund believes the future belongs to AI natives who want to achieve the impossible. And that is why they're introducing the monastery for AI native founders. It's an environment of pure focus and rapid execution for founders operating at AI native speed. And they're offering teams $2 million each to participate. Apply now at cyber.fund.

• 3:06Speaker 03

  Yeah, so my name is Alexander Matic. I've been, so I first did my bachelor's at FAU in Erlangen in Germany. Then I also did my masters there. So my bachelor's was in computer science. My masters is in artificial intelligence. And then I went over to Fraunhofer, which is kind of one of the biggest like applied research institutes. And now I'm also dual affiliated with the Technische Universität of Nuremberg, which is kind of an like still under construction, kind of in the very much in the buildup of becoming a real thing. So it's really exciting to, to, to actually be basically one of the first people who actually does like its PhD there.

• 3:50Tim Scarfe?

  Well, folks, today we're going to get into savant levels of technical detail. We're going to talk about inference. So things like energy based models and diffusion and flow matching. We're going to talk about theories of deep learning, constrained reinforcement, uh, learning and world models, and maybe some other stuff as well. So why don't we start with inference? So can, can you just sketch out the, the history of this? So, you know, obviously back in the olden days, we might have parameterized a basic normal function. And then, and then we had things like energy based models and, and flow matching and normalized flows, all this kind of stuff. Sketch out the whole history.

• 4:23Speaker 03

  In the very beginning, so one of the first ideas is of course, if we have these like nice parameterized densities, where we actually know regardless of any parameter we put in, we will end up with a normalized density. So imagine like again, the normal function, but there are plenty of others, like a categorical distribution, it's the same idea. And the core idea was, well, the difficult thing when we try to do inference is that we always have this, like the density has to be normalized at some point. And the first idea was of course to say, and you go and say, well, let's just pick a group of functions, which is always going to be normalized. And this gives us kind of the like very traditional variation inference problem. Effectively, we have a normal distribution with unknown parameters and now we try to

• 5:08Speaker 03

  fit them onto data. The issue with this is, and people notice this very quickly, is that now you're very constrained by the actual density. So let's say your data is not normally distributed. Let's say it has, let's say it's bimodal. Now your actual density no longer fits, right? So basically what would happen is if you, if you imagine you have like a mode here and you have a mode here, then a norm, then just fitting a normal would put all the mass basically in the middle and kind of try to split the difference. So the first idea, which kind of came up more or less at the same time, was doing something like rejection sampling. So rejection sampling is something that you may have already seen like in school, where

• 5:51Speaker 03

  you try, where you did like you, you painted a square, you had a circle on side of it and they did like random points. And then the fraction of random points that is in the circle, you can use that to compute pi. Exactly the same idea, you can do with any kind of density, as long as you have something which is kind of above it, right? Let's say we have like just a box, which is essentially a uniform distribution. Then you scale it up to be basically around your density and then you randomly pepper it with points. That's rejection sampling and it works pretty well, assuming you have a relatively close envelope over your function. So it doesn't end up scaling very well, especially if you have high dimensional data, right?

• 6:33Speaker 03

  Because you, like it's one of the quintessential examples of like this curse of dimensionality. So then people went and said, well, we can do a little bit better by framing this as basically as the limiting process of a Markov process. And this gives us something like MCMC, where basically the idea is that we may not be able to create the density, but we are able to sample from it. And that's one of the very interesting things about kind of everything inference or like density based is that you can actually do a lot of things without actually knowing the density itself, just by sampling and rejecting samples. The issue with this is that if you've ever run like Markov J Monte Carlo, you will notice

• 7:16Speaker 03

  this is insanely slow, especially in high dimensions. Again, we have curse of dimensionality. You are basically, like not only do you have like these high dimensional problems where you have many modes and kind of escaping from one mode to the next is super difficult, but also just by the fact that you have still like an accept and reject step, you end up kind of doing a lot of work for things that don't end up going anywhere. So.

• 7:42Tim Scarfe?

  Alex, just a quick point of order on that. What about GflowNet? So didn't Benjo come up with some kind of amortized way to replace Monte Carlo sampling?

• 7:53Speaker 03

  So Gflow networks are kind of an interesting case, but I don't think they really solve the same problem. So what a Gflow network does is they essentially, let's say you have like a disentangled distribution, right? You have effectively a graph which you walk around and you can construct your sample up slowly over time. And the idea of a Gflow network is if we have some, let's say final density in the end, which is for example, like effectively it's a reward function. And then the idea of Gflow networks is at every node, kind of the inflowing and outflowing that kind of energy has to be the same.

• 8:33Speaker 03

  So essentially what you end up doing is you end up intensifying the paths which lead to higher reward and you reduce the ones which lead to a lower reward. However, you basically are back into a closer to the variational notion where effectively we will make the assumption that you have a specific structure and you assume that structure is correct, which of course, if you have, let's say some like molecular problem where effectively you start building up something, it's pretty, like we can assume that this process is accurate, but this is not a general way of doing inference, right? The general way is still, uh, you need to do either like sampling when you have a very, very general function family to actually be able to do that.

• 9:12Tim Scarfe?

  Very cool. So yeah, keep, keep the story going. So at some point, energy based models popped up and these normalizing flows, explain that to me.

• 9:21Speaker 03

  Yeah, so energy based models actually directly come from MCMC, right? The idea or like the reason it's called energy is because it comes from physics. Like in the case of physics, they literally had like a function of energy. And then you would assume that kind of the like particles kind of conform proportional to the energy, right? If, let's say you have a, a specific box with hot particles in there, you would imagine they follow some form of distribution because of every particle is like fast, some are a lot faster, some are a lot slower. And that gives you kind of this statistical mechanics point of view. And indeed, I think it was, I think it was Teller who did actually the first developments on like MCMC and kind of got into this idea of actually kind of doing simulations to

• 10:05Speaker 03

  actually get the, the actual parameter or like the particle distributions. Indeed, when you look at what MCMC does is effectively, like sometimes you phrase them literally as doing energy based sampling, but in other cases, it's effectively, it's the exponential of the energy, right? It's, it's more or less, it depends on exactly which algorithm we talk about, exactly who you have teaching you. But essentially this is exactly what you would imagine kind of the, the gold standard of an MCMC method, right? The MCMC method has an energy, which is an un-normalized density. And now you want to turn this into a normalized density, which is kind of what the particles are actually doing.

• 10:45Speaker 05

  An energy based model is any model that parameterizes complex densities using an energy function. Energy functions correspond to an un-normalized density. The lower the energy, the more favored the state, just like in physics where low energy states are preferred. Being un-normalized makes EBM's quite easy to train, since no complex intervals need to be computed during training. Instead, an energy based optimization simply pushes up areas with low sample support and down areas with high sample support. This makes energy based modeling quite flexible with many different objectives, all targeting slightly different assumptions when it comes to, for instance, availability of a reference energy to track.

• 11:25Speaker 05

  While easy to train, this comes at a steep cost during sample generation. These energies are unnormalized, we need to perform complex Markov chain Monte Carlo optimization to sample states proportionally to the energy without having access to a normalization constant. This makes energy-based modeling unfavorable for generative usage. The most expensive part of sampling is executed over and over again during inference, instead of being amortized during training. Further, energy-based models do not offer access to the probability of any sample, which is helpful for many applications such as anomaly detection or to correct accumulated errors when unrolling a dynamics model.

• 12:03Tim Scarfe?

  Just another quick point of order. I mean, can you give a little bit of context here? So, many of us are machine learning people, and we are not sort of in the mindset of working with densities and probability distributions and so on. We are using models that give us point estimates. So what is the benefit of having these densities? Is it because they are composable? Because now we have to do more work at inference, right? So now we've got this density, and we actually need to kind of search around on it and do stuff with it. That's what this inferencing thing is. What does it give us?

• 12:32Speaker 03

  So, I think there are two answers to this. One is, I think most of deep learning is actually doing inference. And if you go and you fit a categorical distribution, like you do your large language model, you have your kind of density, which is effectively like this autoregressive structure, there's just a decomposition of your final density. What you actually want is you want to model the probability of generating a certain sentence given, let's say, a prompt. And what you end up doing is, you end up kind of producing step by step the actual answer by going token by token by token. This works really nicely in something like language modeling, because you ultimately

• 13:16Speaker 03

  still have a discrete measure. Right? You have all your tokens, and you choose which one to actually get. And a categorical distribution is more or less, it's the universal thing. I can always just write down the probabilities of everything. Even if you look at something like regression, or let's say you do like image detection, let's say you do depth estimation, something like that, with your mean squared error, you're still doing inference, right? It's, the mean squared error is just a negative log likelihood of your data under a normal distribution with a fixed standard deviation. And so, you're still doing inference at some point. What you lose is, you lose kind of the breadth of the density.

• 14:00Speaker 03

  But that's more an assumption, right? You can still do exactly the same way you're currently doing, but get an entire mean and standard deviation. And that can be super useful. Like, let's say, like, my background is reinforcement learning. In reinforcement learning, one of the best things you can do to actually improve the performance of your algorithm is to say, I don't actually want a point estimate, I want some breadth to do better exploration. And that's the big advantage of, let's say, a policy gradient method over a Q-Learn. Because a Q-Learn only tells you kind of what to do. It doesn't tell you, like, if you do the different thing, I expect you to be worse by this and that amount. And maybe if you had this, you would be able to go and say, well, for tactical exploration, I choose the one which might be slightly worse, but ends up being better in the long

• 14:45Speaker 03

  run. So, if you're interested in something like active learning, then you also want densities. So, I would push back on the fundamental notion that we are not doing inference if we look at like normal deep learning.

• 14:59Tim Scarfe?

  As an aside, I interviewed Chris Watkins last week. He was my professor. He invented Q-Learning in the late 1980s. So that was a great interview. And maybe you should briefly explain what Q-Learning is, because obviously this exploration thing is the biggest problem in reinforcement learning. But, but just help me understand though. So, what is a good density, right? Because, I mean, Bayesian folks, for example, you know, they valorize this idea of a normalized density or a calibrated density. And you were talking about the, you know, this softmax discrete distribution that we have on the, on the backend of a language model. I mean, what does good look like? To what extent is that a good density?

• 15:39Speaker 03

  Oh, that is actually, that is actually a much bigger question, because it really depends on what you end up using your density for. And if you, let's say, you are a completely deep learning pill, you want to have just a large model which kind of describes the density of images, right? You do diffusion, we're going to get to that, but we're doing diffusion for image modeling. Then what you're ultimately interested in is, you want a density which explains images the best, right? So you want to have them as high of a likelihood as possible for like real images, or if you have like a prompt, that given the prompt, the highest likelihood for things that are

• 16:21Speaker 03

  actually of that kind of prompt description. And then ultimately, kind of what you're actually interested in is just kind of a density which has, which is high enough on the good parts and low enough on everything else. And that, of course, then gives you kind of, that then goes into the discussion of like a prior. A prior effectively just tells you, well, we don't actually want to memorize our dataset. We want to have like, let's say, a smooth from peak to peak, to effectively get like images in between the data points, and that is a prior. And a prior is just a thing that tells you, from like your discrete dataset, how can I get from, let's say, the 50 points of my dataset to the 51st point between two different samples?

• 17:05Speaker 03

  Because if we didn't have a prior, then ultimately, the exact solution would be just memorize the data, and it would be rather useless. If you go on the other extreme, what the Bayesian people do, they say, well, I actually, like, usually what you then go is, you say, you want information in your latents, or just basically reveal information which is not in the data by looking at latents. So a classic example is, I flip a coin, I can observe kind of heads or tails, and what I'm actually interested in is, is the coin fair or not? If the coin isn't fair, then kind of you will see this in the latent, because the latent is effectively is a distribution over, let's say, probabilities it could be.

• 17:50Speaker 03

  And if it's the case that the probability is, let's say, it's 99% sure that the coin is fair, I can accept the hypothesis that the coin is fair. If it's 99% that it always lands on tails, then, well, I can make the inference that it's actually an unfair coin. And that is what like the statistics people are usually interested in. And it's a big divide between someone who does like statistics and like the medical domain, who actually wants to figure out, given those tests, what's the likelihood you have a disease compared to someone in deep learning who just cares about, hey, I want to generate pretty images of, I don't know, pink elephants?

• 18:30Tim Scarfe?

  Yes, and I suppose my intuition is that, you know, we're machine learning people, we have a finite sample of data, and our purposes might be, you know, at some point we've got to interpolate between these data points, we've got the curse of dimensionality and so on. So, we have to put some kind of, I mean, we're amortizing this process, and we have to smooth it and make it amenable to the types of prediction that we're doing.

• 18:55Speaker 03

  Exactly. So, and this kind of goes nicely back to kind of the discussion of like variation inference with the sampling. Because these, of course, have this hybrid thing with like energy-based models, where effectively you say, well, I'm still, I still want to do sampling, I still want to do MCMC, but I learn the energy. And this is what the energy-based people did. And this was a very nice idea, at least at first, because it gives you the property that you can learn basically arbitrary densities. Assuming your energy function is kind of rich enough, that can approximate the energy at least, then you can effectively just pawn off the problem of normalizing density to later, to sampling time.

• 19:39Speaker 03

  But if you, if you set up your, like your loss function in some clever way, you end up being able to, to do the training at least without needing to do sampling. And that is a huge advantage, because now you can, if you can directly fit to like an energy, which you might have like literally measured, or you do this for, let's say, samples of some density, and you try to push up the energy for where the samples are rich, and you push it down where the samples are not rich. Exactly the same way as you do with the density. The problem, of course, you run into with this is that now you have kind of the difficulty of both worlds. Like, you have the problem of like variation inference, where, hey, what if my functions are strong enough?

• 20:23Speaker 03

  And then you have the problem of the sampling-based world, where suddenly you have the issue of, my actual sampling process is really, really slow. So, and this is kind of always limited energy-based models. So, the, the next idea which kind of came up more or less in parallel is this idea of what if we, what if we simplify our problem a little bit? And effectively give ourselves like an infinite pool of latence? Because latence in kind of influence our power. Because if you can, if you can split your problem up into kind of diff, smaller subproblems, but just conditioning on some information that can make it conditionally independent, and these subproblems can become a lot simpler.

• 21:06Speaker 03

  So, in many cases, a model becomes cheaper of the more latence you have. Of course, it's not always true, but generally speaking, you want to have kind of flexibility, that you have tiny steps on like your, which is what you, for example, parameterize. And then, the, the actual, let's say, big steps are just by the fact that you have latence which you condition on. And of course, the like absolute limit would be, you say, well, what if I have infinite latence? And that's what gives you diffusion. Now, diffusion is the idea of, hey, I have a Markov process, similar to like MCMC, like you have a Markov process just with infinite steps, like you make it smooth, continuous. And then, you go and you say, well, at every step, I now, I now sample based on the, the

• 21:51Speaker 03

  previous steps' information, like in the normal Markov step. This makes you, this has one big difficulty, namely, now, you actually need to model effectively time. Because you effectively, what you're now saying is, we have latence, like the latent at time step 0.5. So, and of course, if you go turn to the latent at point, at time point 0.6, you end up kind of having effectively a different density. So, now, we have this time dimension as an additional variable. So, this is what you pay for. But in general, this is much, much cheaper still than needing to do like the actual MCMC process. Because now, you just, you follow essentially a density, so with like the randomized, like

• 22:36Speaker 03

  you randomly inject more noise at every time step. But you no longer need to have like this accept reject step, you no longer have the problem of like local minimum and so on. Because essentially, what you did is, you've rephrased this as a stochastic differential equation, which essentially then goes from like the, the initial density to final density. And this is just by basically paying for with one additional variable, which effectively indexes over the latence. And this gives you a huge improvement, because you no longer need to do effectively this, this time, it still exists in MCMC, like it's the, the sampling steps, but effectively you amortize that as well, and that makes it a lot cheaper.

• 23:20Speaker 03

  And then, you can go a little bit further and say, well, what if we go away from this, this Markovian structure in the first place? What if we say, or maybe not from this Markovian structure, but maybe from a specific, let's say like diffusion style, like this Fokker-Planck style? What if instead, we just phrase it as like reparameterizing a density from beginning to end, kind of in tiny, tiny increments? And that then gives you what is effectively these flow metrics, or like, or more generally, like continuous normalizing flows. The continuous normalizing flows are effectively phrasing the problem of finding a density as you start off with kind of initial guess, right? Just as an optimization, you start with initial guess, which is like your normal distribution,

• 24:04Speaker 03

  and then you end up with your, your final density. And what you end up doing is, you need to kind of find effectively an optimization process which gets you from there to there, which is a gradient flow, which is what gives you these, like effectively optimal transport style problems. We say, we have the continuity equation, which says, well, the, the change in, kind of the change over time has proportional to like the divergence of the vector field of the, the current density. And if you actually, if you go and you, you write this out in, kind of without invoking densities, but let's say we invoke this with points, you see that this is just kind of doing gradient descent in continuous domain.

• 24:47Speaker 03

  And of course, it's the cheapest thing at all, because now, you no longer do any sampling at all. Now, you phrase the problem as, I do optimization in infinite dimensional space. So, you, you pay for the infinite dimensions, but you get the, the lack of sampling. So, the, the, effectively, the entire trajectory was, you start off with like the, the very expensive sampling and like the variation models, which were very limited. Then you said, well, let's do kind of both in the middle, which were these energy-based models. And then you said, well, can we make the energy-based models less like rejection sampling? And that then gave you diffusion models. And now, you say, well, when we actually do this trajectory from like an initial distribution

• 25:31Speaker 03

  to a final distribution, we don't need to do kind of this, this Markov chain process at all. We can phrase it as just a pure optimization problem without historicity, by instead kind of updating the density at every point. And the only thing you now need to pay for is invertibility, because you need to be able to go kind of in one direction, generate samples, and the other direction to get the likelihood. You need the likelihood to do your, to actually do like inference. And that's really cheap, because like a, like a vector field, at least in a density sense, is invertible, like you, instead of going downhill, you just go back uphill.

• 26:08Speaker 05

  Diffusion models, density optimization, is an invertible noisy process. During training, first, a sample gets repeatedly corrupted with normal noise until only normal noise remains. The training objective is to simply revert a single corruption step, for instance, going back from step t equals 5 to t equals 4. This corresponds to learning a vector field, V T of X, that tells you how to change a sample to make it look more like a target sample. During inference, for sample generation, you simply draw a sample from a normal distribution and follow the learned denoising process backward. Inversion like this works, since in the background, this corresponds to solving a stochastic differential equation.

• 26:50Speaker 05

  This equation is reversible coupling between a normal distribution and your target distribution.

• 26:56Speaker 05

  This also allows you to compute the probability of any sample from your target distribution. Simply follow the target sample through the noisy process and accumulate the divergence of your vector field. This shows the major upside of diffusion models. Instead of needing to perform MCMC during inference time, we only need to follow our vector field across time, which can be further accelerated using sophisticated ODE solvers. The downside of diffusion models is that we now blow to the representation with a synthetic time dimension.

• 27:26Speaker 05

  This extra cost, the neural network needs to work with, and also slows on training. We effectively solve many tiny problems of reverting individual noisy steps. However, this is a one-time training expense instead of an inference time expense, making this cheaper for real-world generative usage. Normalizing flows are models that treat density optimization as a reparameterization process. Instead of trying to model our density from scratch, a normalizing flow starts off with a known base measure and transforms it into the target measure.

• 27:58Speaker 05

  If a series of transforms applied to the base measure is invertible, we can use this to essentially inherit the normalization of the base measure. We only need to track the local expansion or contraction, which corresponds to the Jacobian of a transformation, trivial to do with any framework supporting back propagation. Sampling simply follows the flow forward. From a base measure sample, apply the model to get the target measure sample. Computing probabilities goes the other direction. Follow the flow backward from the target back to base by re-skating based on a function's Jacobian.

• 28:34Speaker 05

  One core limitation of naive normalizing flows is the constraints that functions have to be invertible for sampling to work. The way around this is surprisingly easy. A continuous normalizing flow frames a normalizing flow as following a path from the base to target measure based on a vector field that gives local directions. The trick is that such a path, a gradient flow, is necessarily invertible, even if the function giving you the vector field is not. Just walk the same path backward. This has the nice property of allowing arbitrary couplings between measures. The base measure could be a normal distribution, a student T distribution, or even another normalizing flow. Anything where you can compute the likelihood of a sample works.

• 29:17Speaker 05

  However, this also comes at a non-zero cost. For training, we need to compute the full ODE from the target sample back to the base measure every single time, since that is needed to compute the likelihood of the sample for maximum likelihood training.

• 29:33Tim Scarfe?

  Okay, okay. I think there's quite a few things we need to unpack here because we've suddenly got very, very technical. So if I understand, so in a normalizing flow, it does need to be invertible, but in this flow matching, it doesn't need to be invertible. And just to help the audience understand here, so what we're talking about here with this flow matching is it's the ability to model a vector field or an optimal transport between noise and the data that we want to model. And I still think we're missing the link here a little bit of energy-based models. You know, Le Cun, when he talks about energy-based models, he talks, I mean, many folks would have seen the H Jeppa architecture, for example, when he talks a lot about Contrastive learning and he talks about, you know, how you can build specific models to do specific types of things.

• 30:15Tim Scarfe?

  I mean, maybe we should start with that. I mean, what kinds of prediction architectures, what kinds of predictive tasks can you do with EBM's and these normalizing flows?

• 30:23Speaker 03

  I think Jeppa is still a little bit further away. But what I would say is the difference between something like an energy-based model and something like a diffusion model is that, so both of them are fundamentally generative models. I don't really get this idea of like energy-based models not being generative. Like I've heard this a bunch of times, but it's just not true. The interesting thing is you can maybe use them in settings where you don't need to be generative. But fundamentally, you are still generative, like you still can generate samples. It's just very expensive, so nobody does it.

• 30:53Tim Scarfe?

  Sorry, can you just explain that? Because that is something Le Cun says quite a lot. He says that a good property of these energy-based models is that because they're not generative in some way, they are better. That, can you explain that?

• 31:05Speaker 03

  Fundamentally, you can still do exactly the same thing. Okay, you take your energies model, you throw them into an MCMC sampler and technically you will get samples out. This is very technically, of course, many of these energy-based models are not actually good enough to generate nice samples. Right, but at least in theory, if you had, let's say, let's say you had the perfect energy-based model. Like someone comes up from the heavens, says, well, this is your, this is the right parameters, you plug them in and now you have kind of the, the actual data generating process. You can use them to generate data. Right, in fact, this is not even that special of a property. Fundamentally, as soon as you have something which is kind of, let's say proportional to your density, you can throw this into MCMC sampler and you can generate samples.

• 31:50Speaker 03

  So the distinction between kind of having a generative model and having a, have a non-generative model, I think is really, really small. Because the, the question is not whether they can do it, it's whether it's economical to do it. And then I would agree with Le Cun, like it's not economical to do like energy-based models to generate data. But I would imagine this is a disadvantage or an advantage. The nice thing about energy-based models is you can do certain things cheaper. Namely, kind of the, the very big one is you can do kind of comparisons between data points. Right, because like there is, let's say there's effectively an unknown scalar inside energy-based models. But what you can do is you can effectively still go and say, well, this point is, let's say,

• 32:35Speaker 03

  three times as likely as this other point, but just effectively computing the ratio. And if you just need this, so if you have a problem, well, effectively you just need pairwise comparisons, then energy-based models can be quite good. Right, you effectively, you don't need to generate data points to produce something, like let's say not a likelihood, but something like an, a score or like, score is also maybe a bad word because it's also inference-coded, but you still get kind of the, the relative difference between two, two points. Of course, the downside and what makes it hard for us to be generative is even if the ratio between two points might be really large, it still might be really, really bad because you don't have an absolute score.

• 33:19Speaker 03

  It's like saying, well, this movie is a five. Well, five out of five, five out of 10, five out of 100, like it doesn't tell you enough to actually figure out whether it's a good sample or bad sample. And actually figuring this out in absolute sense, this is what the expensive part. The, so energy-based modeling is really nice if you never need to sample anything. So this is what makes it really cool for, let's say, pre-training. Right, if you do pre-training, then doing something like an energy-based model, that might make sense. Of course, there are many other things that also make sense and it's, I think it's up to debate of whether it's actually a good idea to do energy-based modeling over anything else.

• 34:02Speaker 03

  But yeah, I think this is a completely different discussion between kind of doing, between kind of the question of whether these energy-based models are good at doing inference or bad at doing inference. Because essentially, you're just saying, well, I train an energy-based model, but I don't end up using it for like real inference. I use it to effectively measure differences in intensity.

• 34:22Tim Scarfe?

  Okay, and just help me understand a little bit more. So I think Le Cun would say almost any model could be an energy-based model. And something that I've struggled to understand is you can design energy-based models, you can design where to place the latents, you can make it quite domain-specific. Or it could be quite general, you could have a energy-based model of a transformer or something like that. So help me understand that.

• 34:44Speaker 03

  More generally, this is also one of my big problems with like the Jeppa-style things. With Jeppa, like joint embedding predictive architecture, that could describe literally anything. Right, it basically just says, well, it's not contrasted. But let's say, like I recently, in one of the like paper discussions at Yannick Kirsch's Discord, I made the distinction that I found generally that like DINO embeddings are much better than like the

• 35:12Speaker 03

  the Jeppa-style embeddings, at least for the stuff that I tried them on. This might be very different for your application. But I think this is kind of like a false comparison. Because if DINO came out today, they would call it a Jeppa architecture. Because it's such an open domain and the same thing is true with,

• 35:30Speaker 03

  let's say variation inference or like energy-based models. You have, you have a lot of flexibility. It's just kind of what you, so energy-based models is just a name for what you end up predicting. And what you end up predicting is kind of this effectively un-normalized intensity map, right? Effectively, it's like a heat map. And anything that is this can be framed as an energy-based model, which is also why like they, they show up everywhere. Because if you want to, you can phrase it as an energy-based model. And that makes it oftentimes a less useful description. What I would kind of, where I kind of place my difference is if it's already normalized. So something like, I predict, like let's say in, like in reinforcement learning, I predict,

• 36:15Speaker 03

  let's say the mean and standard deviation given the input data. Then I say, well, this is not really a good energy-based model. Like technically it is, practically, you wouldn't think of it like that. Then on the other extreme, you have something like a score-based models, which effectively model the gradient of energy and do tricks to basically also remove all the normalization. And that then gives you something like these, these Langman diffusion-style things that gives you diffusion modeling. And well, it doesn't give you a flow matching, but it gives you something like flow matching. But yeah, I would agree that the difference is really, really wishy-washing.

• 36:54Tim Scarfe?

  And can you just explain this non-contrasted thing? So Randall was telling me at NURIPS that apparently, I think it was Hinton was the big contrasted guy and Simclear was one of the original contrastive architectures and Le Cun is big on this non-contrasted thing. He talks about it as shrink wrapping the latents. What does he mean by all of that?

• 37:13Speaker 03

  The big, so there are different ways of kind of framing like these unsupervised learning problems. One of the, so one of the first ideas was, okay, we can phrase this as kind of making similar things similar and dissimilar things dissimilar, right? And this gives you ultimately a contrastive style model. By basically what SimCILR effectively does is if you look at the loss, effectively it's like a cross-entropy but over the batch. And what you then do is you have the positive part, which is like the, the different augmentations which get pulled together. And the diss, like the, the dissimilar parts that get pushed apart. The downside with this is it's ultimately a very, very moving target, like a batch-to-batch moving target.

• 37:53Speaker 03

  Because a kind of contrastive just means, well, it's somehow the right things have to be together and somehow everything has to be apart. This means there are many, many, many solutions which actually satisfy this condition, but are not actually useful. So in practice, you want something which is A, stronger and B, does not have this instability that you fundamentally have like these SimCILR-style methods. And if you look at SimCILR, they only start making sense at huge batch sizes. And then there is like follow-up work, like the, the MOCO, Momentum Contrastive Models, which essentially say, well, we can simulate huge batch sizes by effectively taking, let's say, a buffer over time.

• 38:35Speaker 03

  This of course adds a little bit of noise, but tends to work pretty well. But fundamentally, I agree with Le Cun that we need to go back away from these contrastive models. Because contrast, contrastivity alone is not strong enough of a statement to to actually learn necessarily useful models. So what Le Cun now says is, well, we should actually go, effectively we should revisit old ideas, which is try to map things into like a compact latent. And the first idea of this was like a variational autoencoder. A variational autoencoder says, well, you have to be able to map the data to a normal distribution or let's say a beta distribution, whatever you, you set in the middle. And then you have to be able to map this back.

• 39:17Speaker 03

  And effectively this compresses everything down into this, this tiny gate in the middle. And you, effectively the, the top part is a generative model, the bottom part is effectively it's like an embedding model. A problem with these variation auto-encoders, and it's something that's also been seen very, very early on, is that you effectively need to pay for things that don't end up making a difference. Right? The classic example, I don't know who did it, has the example of like, it's an actual dollar versus like a painting of a dollar. Like, it represents the same idea, but it's very, very different on a pixel-to-pixel basis. And this then gives you something like DINO,

• 39:57Speaker 03

  or something like VicRec, or something like the JEPAR-style architectures. Specifically, like, one of the newer ones, which I really like, is like this LED JEPAR-style thing, which effectively does, like, uses the statistical tests to actually figure out, okay, am I different from a normal distribution? Or am I the same as a normal distribution? And based on that, they can effectively differentiate through that test and move you closer or farther away from the actually now fixed goal of the density. So you replace the contrastive, push things apart or pull things together, with be somehow on this fixed target. Of course, you still have some flexibility, right?

• 40:41Speaker 03

  Because if you, like, it's if every point starts pushing into the mode of the density, at some point you would have a kind of pushing away effect again. So you have a similar thing as a contrastive model, but now it is contrastive compared to a reference density and not compared to itself. And it makes it a lot more stable.

• 41:00Tim Scarfe?

  So what does all of this mean for energy-based models?

• 41:03Speaker 03

  In my opinion, I think that energy-based models are not really useful. At least not useful anymore. In general, of course, you can frame your model as being energy-based. Like, I will not, if I'm a reviewer, I will not ding you for it. But ultimately, I don't think it's a useful way of thinking about the vast majority of problems. Like, there are exceptions to that. Like, let's say you are actually in like a physics setting. Then maybe it makes a lot of sense to actually phrase this as an energy-based model. But I would argue that it's not, like, it's not a useful way. Like, it's so general of a statement, it's not a useful way of thinking about the problem. For things like contrastive or contrastive for unsupervised,

• 41:48Speaker 03

  like these self-supervised learning methods, I think there is, generally speaking, a better way of thinking about the problem than energy-based. And ultimately, I think this is where these methods lie. If you want to do generative diffusion model or flow match as better, I don't think anyone will argue with me about that. And if you want to do unsupervised learning, I think you can phrase it as an energy-based model, but I don't think it's a useful framing.

• 42:13Tim Scarfe?

  Yeah, and we'll get to diffusion in a minute. But I mean, can you help me understand, where is flow matching being used? Because folks in the audience, they might not be very familiar with it. I mean, for example, I was interviewing Mistral a couple of days ago, and they were using it in their Voxtral models. So there are people out there using flow matching, but where are they using it?

• 42:30Speaker 03

  The big difficulty for doing something like, let's say, something language modeling-like, right? Ultimately, everybody right now wants to do language-modely things without language, like big audio models, big image models, big world models. We can get to that as well later. But all of these models, they want to do exactly the same thing that large language models did for language. And the problem that they quickly run into is that most domains are not actually that discrete. Like, you can make them discrete by basically hitting with a hammer long enough, but practically, they're not discrete. The problem is that if you are not discrete, things like this auto-aggressive decomposition become kind of dicey. Because the, like, for language models, the nice part is they can effectively

• 43:15Speaker 03

  go step by step, and you can predict kind of from left to right. If you have a continuous density, like an image, you can still do the auto-aggressive, but it's not longer a natural way of doing things. And it's not a natural way of decomposing a density. Because ultimately, that's what all of these models do. Be that diffusion, be that energy-based, be that auto-aggressive, all of them try to have a huge density, an entire sentence, and decomposes into smaller parts which you can actually handle. And the idea that you want to do, like, most people settle on for continuous processes is instead of separating kind of from left to right, let's say, token by token by token, is they say, basically, we separate from bottom to top. Like, we start off with all tokens, but kind of corrupted and kind of go back up.

• 44:00Speaker 03

  And that immediately allows you to do things like image generation. Because I explained it just now with tokens to make the comparison sensible. But ultimately, you start off with something and you end up with something. In fact, discrete, in this case, is the more difficult part. Because of discrete diffusion, you end up with some kind of dicey things with kind of... I guess language models are probably fine, but you get into dicey things with basically, if you actually sample from the density every single point, it's not possible, but you have to be a little more careful. But the nice part is that, once you have kind of this general recipe of decomposing your structure, then you can apply it to anything. And this is what makes diffusion models or flow matchers,

• 44:43Speaker 03

  which is, like, if you read diffusion model and you look into it, oftentimes it's flow matching because it tends to be a little bit more stable. But all of these methods are just ways of decomposing a density in a really, really general way. And what you do in diffusion is you give yourself these infinite numbers of latents and say, well, given the infinite number, it has to be decomposable in some point, in some way. And in the, or in the case of flow matching, you literally give kind of the reference of, okay, this is what you need to do, just do it now, but general. And in auto-aggressive structures, you do exactly the same thing, but instead of giving yourself the latent codes, you say, well, I have a very natural decomposition of my text already, just use that.

• 45:27Speaker 03

  And, but ultimately, kind of all, both of these solve the same type of problem. It's just whether you do the decomposition in the observable space, whether you do the decomposition in the latent space. And for things like, for high-dimensional things, which themselves have less information per, effectively, per variable, it tends to make sense, or at least empirically tends to make sense, to go into diffusion instead.

• 45:51Tim Scarfe?

  And can we just talk more about inference in general? So I think this is something that people don't think enough about, because we don't do, we don't really do things that resemble inference that much. With language models, we just do greedy sampling. So we've got this density and we just take the, you know, we just take a token, take a token. In something like AlphaGo, for example, we had a structured inference process. So we did this, you know, Monte Carlo tree search, and we were kind of searching around and actually algorithmically updating nodes that we'd visited and feeding that back and retraining the model. With these vision models, and when we're using flow models and stuff like that, we are actually doing like quite sophisticated inferencing processes. And I interviewed Clemon Bode. He had this Searching Latent Spaces paper, and he was kind of like searching a decoder to actually find the best programs to generate for ARC.

• 46:36Tim Scarfe?

  And we've got this whole, you know, test time computation, test time adaptation, and so on. So this whole thing about doing work at inference time is very exciting to me. And I just don't think that that many people are talking about it.

• 46:48Speaker 03

  Yeah, I think we give people not enough credit when we talk about kind of doing inference. So first of all, inference is a really, really overloaded term, right? If you go into logic, like, they say, we do inference, like logical inference. And if you go into like Bayesian, they also say, they also do inference. Now I say, well, we are actually also doing inference. And so I think we have to be a little bit careful not to use the term too liberally. Because, and like, I have to be careful myself, because at some point, I use all meaning. At some point, it becomes the energy-based model, which you can apply to anything. But ultimately, I think in many cases, it is, it is very, very instructive to think about these types of models

• 47:32Speaker 03

  as just composing, just conditional dependencies, conditional dependencies and marginalization. Something like AlphaGo is actually a little bit more difficult, because what they do is effectively they do this like non-parametric, effectively they come, arguably, they come more from like, less the Bayesian side and more the frequentist side, which is an entirely different kind of worms that don't want to open up right now. But at the end of the day, all of these, all these methods really, or at least all these mainstream methods really benefit from this idea of, we have a big problem, we need to decompose this, and every step is one step in a bigger inference process. Let's say, let's give an example from one of my papers. I did a paper about doing, effectively, mixed initial programming, learning heuristic for mixed initial programming.

• 48:17Speaker 03

  And specifically, like, in mixed initial programming, to not go too deep, they have a process called branch and bound, which essentially, you start off with like an approximation, and then based on a solution, you pick two different solutions, just like in like a tree search, or it is a tree search, and you go, at this node, you go further down. But closer to something like AlphaGo, like Alpha-Beta pruning, you can prune away lots and lots of nodes if you know more information. But basically saying, well, this node is worse than the solution that I already found, so I can remove that. So it becomes really important, if you have, like, it's a huge tree, with, let's say, thousands of open nodes at the bottom, to choose which one to actually end up selecting. And what I did is I frame this as an inference problem.

• 48:59Speaker 03

  Where basically, I say, well, if we go down from the root to one of the nodes, you effectively do a conditional probability. Like, you have a probability to choose either left or right, then you have another choosing left or right. And that gives you a much more general problem, a much more general model, which kind of grows with your problem, right? So as your problem grows, your inference process grows and kind of ingests more and more information. And the same thing is true with something like, let's say, a large language model ultimately, which kind of does research, kind of looks up more data, ingests it, and now the next step is now conditioned on that information. Now, how useful this is for your, like, let's say, like the viewer's daily work, it really depends on what you end up doing.

• 49:43Speaker 03

  If your work is, let's say, I scale up a large language model, then you probably don't need to think about this. If your, if your research is like figuring out really, really efficient architectures, maybe, but I would also say, probably it doesn't matter. But if you kind of, if you go and you use them, or you do something like RL, where ultimately, kind of, everything is like you update some density, unless you do like Q-learning. But if you update like densities, then you should be thinking about this as an inference process, because especially Bayesian inference gives you a lot more structure, a lot more information than what we give it credit for. Like, something like just decomposing a problem into conditional probabilities gives you, like, gets you like 99% of the way there,

• 50:26Speaker 03

  but it can already help guide the actual problem. And I think that something like all these agentic things could really benefit from actually thinking about this as like a Bayesian problem of kind of going out, gathering data, and updating your conditional belief based on the information that you just ingested.

• 50:43Tim Scarfe?

  Very cool. And we've mentioned Q learning a couple of times. Can you just give us a quick refresher on that?

• 50:48Speaker 03

  Okay, so the idea of Q learning is actually it's kind of simple. So what you do is the problem that you run into in reinforcement learning is that you don't want to be greedy with like the immediate actions, right? Because what you do immediately could be really detrimental. Let's say you manage a warehouse and the greedy thing to do to maximize profit right now is just sell everything in your in your supply closet and never buy anything because then the profit is maximal. That's not necessarily long term the right thing. So what we would need to do is we don't need to look at the current reward, but we need to look at kind of the future as well. But that gives you a problem. The problem is, well, I now need to model the future. So the way that Q learning does it is it says, well,

• 51:33Speaker 03

  let's decompose it as kind of a head, which is the current reward, plus the tail, kind of the rest.

• 51:40Speaker 03

  And that then gives you, if you, let's say we have such a decomposition, then we could effectively act greedily with respect to this, kind of to this combined metric, right? Because if we have a Q function which tells us, well, if you go down this path, the the future will behave like this. If we go down this path, it will have like this, and this path will behave like this. Now we can be greedy again with respect to the Q function. So effectively, the Q function fundamentally is a closure over the, like, over the path that the policy can take. The way you train this is effectively by just enforcing exactly this. So you say the current reward plus the next state's Q function should be equal to

• 52:23Speaker 03

  the Q function starting from the original state. So you effectively say R plus Q from the next state for a specific action. And then you say this has to be the same as doing a Q function from the original state already. This then gives you two options, right? The first option is we could choose kind of for the next action to choose here, right? Because the Q function has both a state and an action at this node, at this future node. You could choose, well, we pick the action which is actually taken in the real world. This gives you SARSA, right? State, action, reward, next state. The other option is, well, I could go and say, instead of just picking the next

• 53:05Speaker 03

  action which has actually happened, is I pick the optimal action, right? Because let's say we have a discrete number of actions, just look at every single one and say, well, I pick the one which is the best. So effectively, you hallucinate a world in which the, you actually chose the correct path. It has the very nice property of now you you effectively are no longer coupled to the specific policy. And if you choose the correct, if you choose the action that the policy chose, you now effectively update with respect to the policy. If you choose with the, if you choose the optimal state, effectively you're you're updating with respect to the optimal policy already, like the Q star. The problem with this is now that you're no longer, you have some form of, effectively you have a counterfactual inference problem

• 53:49Speaker 03

  where you need to deal with what actually happened versus what happened in our, like, let's say, golden state where we walked along the kind of optimal path.

• 53:57Tim Scarfe?

  Very cool. And just coming back to EBM's for a sec, we've been really negative about EBM's and LeCun actually had, he spun out two companies, AMI and Logical Intelligence doing this stuff. Are there any situations, do you think, where EBM's are better?

• 54:12Speaker 03

  Again, I think the big one is if you just need ratios. Because then it can be a lot cheaper because you don't end up generating, let's say, the entire trajectory. It can also be helpful if you have something energy-like already. Like, again, something like physics where effectively you, based on data, you try to learn the energy function, something like this.

• 54:32Speaker 03

  Other than that, I think it becomes really, really difficult to justify. Of course, it really depends, like, as we said, energy-based models are really overloaded term. Because if you don't, if you use inference as a pre-training task, I don't see a reason why not. I don't see a reason why you should, but I don't see a reason why it wouldn't work. The question really is, is if we end up looking, like, let's say, LeCun, like, let's say, we we jump forward five years from now and LeCun's startup is doing whatever he did, like, they are doing currently research. Whether, what they end up doing, like, they might still call it energy-based models, but whether it's actually an energy-based model as we, as you and I would be thinking about this right now, I highly doubt that.

• 55:13Speaker 03

  Because while it's possible, I don't think it's, I don't think it's the best way of doing this. And the, and the, the bad thing, or the good thing and the bad thing about deep learning is that if you do something wrong, it generally doesn't just not work, it just has worse performance. And that makes it really, really difficult to research, which is why there are thousands of papers which essentially claim more or less contradictory things, because depending on subtlety test, things can just work or not work. And they can be really, really, so even if you do something completely wrong, like I have recently, I have a paper which I tried to get done for ICLR where I had like an implementation bug, it still kind of worked.

• 55:54Speaker 03

  And there are plenty of other things where still, like, it doesn't, on paper, it doesn't work, but in practice, it just works. Because especially reinforcement learning, it just, it hammers on it until it figures out something. And I think that many, many things right now are like this, like, they are not necessarily optimal, but they're also too bad to actually bother fixing it.

• 56:14Tim Scarfe?

  Very good, very good. And how do, I don't think we've done enough on this, but how do, how do we go from flow matching to diffusion models?

• 56:21Speaker 03

  So again, the jump from flow matching to diffusion model is a lot larger. Ultimately, if you actually look at, so flow matching actually comes from, so there are two ways of thinking about flow matching. One is you can think about them as like the continuation of continuous normalizing flows. The idea of a continuous or like any normalizing flow is you have a base density, so a normal distribution. And you frame your problem now as trying to figure out a transformation which goes from that base density to your target density. If you make sure that this density is invertible, technically a little bit less is possible, but say it's invertible,

• 56:59Speaker 03

  then we can actually both do the sampling, so you go from the base density, you go forward to the sample space. So this is the sampling direction, and if you have a sample, you can go backward and figure out the the probability of generating something by looking at what it is on the normal. So this is the first idea which you can kind of branch off from, and the second is from optimal transport. So in optimal transport, there's this very, very famous theorem by Branyam and Benhamu, who basically said, well,

• 57:30Speaker 03

  we can frame an optimal transport problem from one density to another as a kinetic problem, where effectively we follow this transport equation. So the, the density change over time is equal to the negative of the, like, the current density times the velocity field and we would do the divergence around it. The, and this effectively is a continuous, effectively it's like gradient descent, but in continuous time and instead of over having a finite number of parameters, you do it over like the, the actual density. So it's a, it's called a gradient flow. And these have really, really nice properties. So even in like finite

• 58:11Speaker 03

  dimensions, and the, these continuous normalizing flows, like these gradient flows have super nice properties. So what you end up doing, we use this fact that effectively the velocity field and the, the density change is coupled together by saying, instead of learning the density, we learn the change in density, so the vector field. And this is the parameterization. Or at least for, for like, for these flow measures. So what you now say is, instead of picking any velocity field, I do a reference, let's say like the rectified flow, which is like, let's say, a straight shot, and then we fit to this. How we go to, so we can go from this back to diffusion.

• 58:52Speaker 03

  And specifically, if you look at like the Fokker-Planck diffusion, so Fokker-Planck, the Fokker-Planck equation is the effectively the PDE form of stochastic differential equation describing diffusion. So a diffusion process is kind of this idea of we have both the mean change and then we have the standard deviation which adds in the noise. We can frame this as instead of doing like the sampling, effectively sampling our way through, we can view this as changing your density from the start to the end. This is the Fokker-Planck equation. So effectively, we now, we, instead of doing sampling, we now have to solve a PDE. And usually people of course go the other direction because doing sampling is much cheaper and much easier than doing and solving a PDE.

• 59:37Speaker 03

  But let's say we look at this from a PDE point of view. If we actually look, we can rewrite this as basically the velocity field inside of it is so that the Fokker-Planck equation reduces back down to having a very specific velocity field. And if you look at this velocity field, you end up doing, so you end up realizing two things. One is, of course, like, there's a fundamental connection between these two methods. The second thing is you can see why diffusion models are kind of iffy to train or more so than diffusion. Because if you look at the velocity field, the velocity field actually depends on the density. So effectively, what we now have is we have the, like, the self-coupling

• 1:00:19Speaker 03

  which originally in like a flow match, we can get rid of, we can completely get rid of ever needing to touch the density. While in flow matching, well, in diffusion, you end up having to touch the density. Of course, there's this trick with like the, the score transformation, which means you kind of get around this. But ultimately, the, the vector field now depends on the

• 1:00:41Speaker 03

  now depends on the current state of our diffusion.

• 1:00:46Speaker 03

  The nice part about this is that the diffusion style has some really nice properties, specifically that it's kind of minimizing the the, the relative entropy, so the KL divergence between like your, your target density and your, your, your current density in this like optimal transporty sense.

• 1:01:03Speaker 05

  Flow matching is a method striding the space between diffusion processes and continuous normalizing flows. The upside of normalizing flows is the ability to couple arbitrary measures with each other, while diffusion is theoretically restricted to couple with a normal density. The upside of a diffusion is the fast training since we do not need to solve a full ODE for every optimization step.

• 1:01:26Speaker 05

  Flow matching exploits the fact that the actual path taken from base measure to target measure is arbitrary. This means that we can simply impose a reference direction which we fixed beforehand. This is quite similar to what we implicitly did for diffusion models. A diffusion model's reference is given implicitly through the noise process. In fact, a diffusion model is simply a very specific reference flow that aims to minimize the KL divergence.

• 1:01:54Speaker 05

  Flow matching still frames sampling as a reparameterization problem, but now fixes the degree of freedom in a path priori. Pick a path, for instance, a straight line between a real sample and a base sample, and train the model to emulate that.

• 1:02:09Speaker 05

  The model's task is now to simply learn a general parametrization of the example flows, a simple supervised regression problem.

• 1:02:17Speaker 05

  The upside of fixing the paths twofold. First, it allows us to give supervision on every part of the flow without needing to simulate the entire ODE. The intermediary targets are known and can be used as supervision signals. Second, it allows us to shape the path based on properties we want. For instance, straight-line paths tend to be easier to sample from than highly nonlinear ones. As such, flow matching allows all the flexibility of normalizing flows while having the superior sampling properties of the diffusion models.

• 1:02:49Tim Scarfe?

  And can we make this a little bit more concrete? So for example, I interviewed John Jumper, and he was talking about how he used diffusion in the third version of AlphaFold. And he was talking about some really interesting properties because it's very much an iterative process. So he was saying that we were running it through this diffusion model and the big structure kind of converges first, and then it learns the details afterwards. And many folks would have used stable diffusion models, for example. So you can actually kind of see these images materializing over time. But yeah, just be a little bit more concrete. I mean, where are these being used? How can people use them?

• 1:03:18Speaker 03

  I think this is pretty hard to say because ultimately you would expect something like this for a general model. And the reason is that why the latents themselves don't hold any information. Like a latent is completely unobserved. They don't hold fundamental information. If you want them to hold fundamental information, you need to go to the Bayesian people who do like the parametrization of the coin flipping, which I had at the beginning.

• 1:03:45Speaker 03

  The actual latents don't hold any information. But of course, in practice, if you enforce the fact that the change from, let's say, latent number one to latent number two should be really small, then you would expect something like this, where effectively the big structure appear first. Because if you don't have the big structures, the small changes later on cannot reconstruct this. So I don't really have a good argument for why it has to be like this. And I don't even think it has to be like this. I think this is more a notion of what happens to be easy for these models. But I would imagine that most insane models would structure it like this. This doesn't mean that you couldn't build a diffusion model, which doesn't do that.

• 1:04:31Speaker 03

  I could imagine going and enforcing effectively a constraint that says, well, you have to do the small things first. I don't think it would be a good model. You might need, let's say, thousands of times more parameters to actually fit this. You still would be able to train this, at least I think so. Mathematically, there's no reason why it shouldn't work. Practically, there is lots of reasons why it shouldn't work.

• 1:04:54Tim Scarfe?

  That's actually really interesting because we see a similar thing with just standard neural networks, right? You can look at the types of representations they learn during the training process. And vision models, they learn low-frequency features early on, and then they learn higher and higher-frequency features. And you're kind of saying it doesn't have to be a certain way. Maybe it just depends on the architecture and the data and the domain and things like that. But it's quite interesting though, isn't it, with these types of models? That you can basically do an arbitrary amount of computation. So on stable diffusion, you can do 25 diffusion steps, or you could do 150 diffusion steps. And if you overcook it, the image starts to look weird because you get little micro details on the skin, and it doesn't look very good anymore.

• 1:05:39Tim Scarfe?

  So yeah, I mean, how should practitioners reason about this?

• 1:05:43Speaker 03

  I think reasoning about architecture is really, really difficult. Because I... So the more I think about architectures at scale, like let's say we have these billion parameter models, I am not sure it matters as much as people think. I think it matters in the sense that you need to make sure that your input data actually fits into the model.

• 1:06:07Speaker 03

  So let's say you couldn't do like an MLP, which does like the same type of large language model-y things just because it has a finite input space. But I don't think that there is a fundamental limitation on one of these architectures if you look at it at huge scales. Because at huge scales, all of these perform a lot less, like parameterized models of a specific architecture, and more like function spaces. Effectively, the actual change of the individual parameters does not really mean anything anymore, but more kind of the bulk change of the entire field around this. Like this is effectively the mean field theory approach to deep learning analysis,

• 1:06:50Speaker 03

  where effectively it doesn't make sense anymore to look at individual particles as in physics. Like you don't model a stream of, let's say, billions of... billions, kajillions of hydrogen molecules inside your water stream or like a physics simulation. You model the actual bulk properties with like an Avia Stokes. And the same thing, I think, is true with large language or let's say any large model. And that makes it really, really difficult to actually make insightful statements about what architecture leads to what because I don't think it ultimately matters that much. I think it matters way more, the scale. And of course, it might be that, let's say, one architecture just scales more nicely.

• 1:07:35Speaker 03

  One architecture needs a billion parameters. The other one needs 100 billion parameters just because the other architecture is just less nicely scaling. That's probably the reason why we choose deep neural networks and not just a shallow one, which is really, really, really wide. But I don't think that in the limit, and I don't mean like infinite limit, but in the large-scale limit, it makes a difference. And I think it's much more kind of how you would expect the information to proceed through these models, especially in diffusion. Like let's say we have these diffusion models. In diffusion models, we have this structure that they are doing...

• 1:08:16Speaker 03

  How is it called? That they are doing KL divergence minimization in a Wasserstein space. So you would expect also that the higher information things to appear first because they are the free wins. I don't think it has to be like that, but I would expect it to be like that. The same thing with a rectified flow. In a rectified flow, I don't think it should be that surprising that if you basically do a linear interpolation, those big things appear first. But I am not sure if I could advise people or like in good conscience advise people and say, "Well, you should do this and that for your architecture" because I don't think it matters.

• 1:08:52Tim Scarfe?

  All right, cool. So we should move on to theories of deep learning. So much to our chagrin, we've tried many times. I mean, certainly, God knows. We've interviewed many people who have said that they've got a theory of deep learning. But you've said to me that roughly this can be broken down into parametric, a parametric view and a functional view. Explain.

• 1:09:13Speaker 03

  If we look at deep neural networks, there is of course a very, very obvious way, like a first instinct way and say, "Well, this is just a huge nonlinear program." Basically, you look at this as you would in the optimization people. The optimization people, they say, "Well, I have like a huge, let's say, polynomial, in this case, a huge parameterized in a real neural network." And we now look at, okay, what are the properties? What would need to happen for this to converge to something? What do we need to do to figure out a global optimum? What do we need to do to effectively model the change in parameters over time? And this is ultimately a really, really interesting way of thinking about the problem. But it's not necessarily that useful of a problem.

• 1:09:57Speaker 03

  Now, there are of course counterexamples, things where we actually look at kind of the parameterized nature of these, and we can actually make some bigger inferences. Which effectively where the optimization people can look into deep learning, especially if you look at the explain theory of deep learning. Like if you actually look at what the mathematicians are doing, like all the computer people, they all say, "Well, that looks like something I can do." And ultimately, there are really, really interesting things there as well. But I don't think they give you that much insight. They can give you statements like, "Okay, where's the actual complexity of a learned model?" Like basically, "How many of these linear regions exist?", something like this. They can also sometimes get some really nice explanations.

• 1:10:40Speaker 03

  Like I really like the paper by, I think, Balashevsky, who did the explanation of why batch norm works. And he just framed it as, "Well, batch norm actually concentrates the things in the middle, and therefore all the hyperplanes are more concentrated in there." It's effectively more linear, more nonlinear. But...

• 1:10:59Speaker 03

  I don't think it's ultimately that insightful, because at some point, again, bulk properties, it doesn't matter anymore what an individual parameter does. And then we very quickly go into like this domain of saying, well, if parameters don't matter, can we get rid of parameters in our analysis? And that then lets us walk into something like the neural attention kernel theory. It allows you to like, they are some randomized, like randomized sketching type results. And it's also like mean field theory, which I really like from an analysis point of view. I'm not sure whether it's that useful ultimately, but I really like it from like a mental model. That's what they phrase it like is they say, well, if we look at like a, let's say a one hidden layer MLP,

• 1:11:39Speaker 03

  very simple, we can view this as effectively every input neuron is like, is this inner product with an activation around this. So in like, we can view this as, let's say a thousand, let's say test functions. So the activation function is let's say you make a sigma to make it very, very instructive between zero and one. And it figures out, well, is this test, so this weight, this inner product is this test successful? And then the next layer, there's a combination of all of these. What we can do is we can phrase it as what happens if instead of having fifty layer, fifty inputs, uh, systems, we have an infinite number. If you put this into the infinite limit, you effectively now write this down as like a density.

• 1:12:21Speaker 03

  Right? Because effectively every single point exists, it just matters kind of how present it actually is, kind of how relevant it becomes. And now, uh, we have this, the structure of effectively fitting a density over, uh, over weights and kind of the, yeah, fitting a density over these weights. And that is very nicely to analyze, because now we are back in like literally a diffusion style problem. Because now we can model the change in density over time as a change in distribution over time, which is ultimately a diffusion style problem. And that, uh, in neural attention columns are very similar idea. Just instead of looking at this from like a density point of view, they simplify a lot more and say, well, let's actually, actually make it a function space.

• 1:13:06Speaker 03

  And the function space then kind of spans the, the area. Now, there are some limitations with this analysis. For instance, neural attention kernel theory assumes the kernel is stationary. But even that is, it's an assumption that we make for like analysis point of view. There is nothing that fundamentally hinders us in looking at the kernel itself changing. So effectively we have two dynamics where we have both the kernel change dynamics and we have the, uh, kind of kernel itself describing the, the local function. And this is also what mean field theory does. Mean field theory effectively in, in the back of their, their big paper, they have a, a section where you talk about, well, actually the connection to neural attention kernel theory is we just assume the kernel also changes over time.

• 1:13:50Speaker 03

  And that of course makes the analysis way, way better. That you can now effectively also model not only the lazy learning, but also the feature learning domain. But has a huge limitation now that it becomes much, much harder to analyze. Still, I think that from analysis point of view, these end up being more useful. Specifically because the predictions end up lining up a lot better than what we would expect for the parameterized world. Like in parameterized world, we say infinite, or like a huge numbers of parameters, this thing is going to overfit instantly. Or like this thing is, but it's completely over parameterized, why do you do this? From a function point of view, you say, well, no, it's not over parameterized. I'm just going closer and closer and closer to the actual true function space I'm fitting to.

• 1:14:34Speaker 03

  But a, a, a function which is a billion parameters is much, much closer than a function with fifty parameters to being at this like infinite, uh, scaling domain. So effectively, the, the function analysis really, uh, explains this idea of over parameterization. While the parameter, uh, while the parametric kind of says, well, actual, actually this over parameterization might be bad, but we can exactly explain kind of how the, the local architecture is. I would, however, I would imagine that in the long term, the function analysis people, they are going to be more useful just because they explain a specific architecture, but they explain architectures in general.

• 1:15:18Speaker 03

  Like a neural attention kernel, it works for pretty much anything, right? It, no, I've been proved works for literally anything, but the idea of looking at this from a function space point of view works even if you have like, let's say we all go back and do like MLP mixes now. We no longer do transform, we'd all do MLP mixes. It still would work from a theory point of view. Because we, or like we don't know, we no longer use ReLUs. In fact, we don't no longer use ReLUs, we use like these gated activations. Lots of like research in the parametric world no longer really makes sense, because they all assume kind of this, uh, this spline like structure. Instead, like the, the function space theories, people say, well, the function space looks a little bit different. But it's still, um, like ultimately it's, it's like something like a barren space. And then it's the analysis that works.

• 1:16:04Tim Scarfe?

  Yeah, I mean, I, I suppose, I'm just curious to know what would a good theory of deep learning resemble? You know, may, maybe it's something that gives us a good explanation. I mean, you were just saying with this functional analysis, you know, the kind of questions we might ask are, um, is this network over parameterized? Or we might want to ask, is this architecture optimal? And it feels a bit like the blind men and the elephant. So we've got folks like Randall coming at it from a geometric point of view, that there's a lot of people doing that. Um, we've also got, you know, our good mutual friend Keith Duggar. So he's coming at it from a computability point of view. And he's, he's talking about, are they, are they Turing machines? He's talking about the space of algorithms that, that, that they could possibly represent. And then there's, you know, things like, you know, complexity, um, views.

• 1:16:48Tim Scarfe?

  There are folks, uh, you know, looking at statistical physics, talking about spin glasses and heavy tails. You see what I mean? That, you know, even like effective theories folks and complexity theories folks. So these are all different perspectives on the blind elephant. Which one is best and why?

• 1:17:04Speaker 03

  I ultimately think that the most important thing is that it makes kind of the correct explanations. Like it, it comes up with the correct solutions. I don't care that much over the specific structures. Like the, again, uh, like I don't want to sound too negative on like the parametric point of view. If it makes good predictions, then I'm completely fine with this. I just don't think it is as, um, as tuned to being productive. But the, the important thing is, and I think it's also missing for, for many of these theory papers is that you go and you actually check whether the test works, basically you try to falsify your theory or try to figure out where does this theory actually break? In this case, I would actually think that something like, like physicsy is a lot more useful.

• 1:17:50Speaker 03

  Uh, just because kind of this, uh, this loop of I make a prediction and I go, I make an experiment. I, I think this is much more, much more useful. And I think that the, the big thing is that theory will never be able to explain literally anything. Because like it might be able to explain things on like a pure functional point of view. I like to say the mean field people, they can completely explain if I have, let's say, uh, a density composed of, let's say, a Gaussian mixture model, composed of this thing. Then I can completely tell you, hey, you need this and that size, uh, or like add this and that size. The error to the infinite model will be this size, completely fine.

• 1:18:29Speaker 03

  The problem of course, or same thing with like the, the SVM people, like kernel based people, they have like source and capacity conditions, which effectively is how much does your function have, how much does your data have? The problem is, and in practice, we will never have that information for our real data. In practice, we just have, maybe we have a clue, right? I think we can oftentimes kind of estimate, for instance, we can often estimate how many clusters there could be in our data with relatively simple, uh, methods. But we can't really have very deep information into our structures. Because ultimately it's still the real world and we still approximate everything as theoretical chaos. But the, the, like the, the physicsy side would then say, well,

• 1:19:14Speaker 03

  we try to do kind of, we do like a best effort and we try to solve everything that is solvable or is predictable. And based on that, we can still make real, uh, kind of predictions for the real system. And which is effectively then the transition from physics over to engineering. And where the, the distinction lies doesn't, it ultimately it doesn't matter, even if you can actually make a hard cut. And I think the, the important thing is that we need to have something which makes accurate predictions. And for that, the loop between the, the theory people, which do like the, the really nice things like mean field, uh, neural attention kernels. Um, also something like diffusion, like diffusion is also, it's a non-parametric, uh, expla- uh, explanation of what these models do.

• 1:19:58Speaker 03

  You just say, well, they fit some density. Reinforcement learning, what I do is also fit some density, be happy. But the, the actual loop should be to actually then take this and trying to make. Inference for the real world, like if I if I prove something convergent and I test it and it doesn't, then we would then kind of, and I think this is where the like there should be a deep, a theory of deep learning. People will look at, we should now go and say, well, exactly that, we need to explain this. The problem is, right now, it's really, really difficult because we have so little theory that the reason it could not work, or it did not work, could be just an implementation error because we don't have a good enough idea of what we do yet.

• 1:20:42Speaker 03

  I previously mentioned that I basically wasted like, let's say, like two or three months with an implementation bug in one of my, one of my papers. Just the fact that we cannot figure out whether there's an implementation bug in like a a continuation of existing work, that alone should mean that the theory is not yet at the level and this is what we should be, what we should be looking at.

• 1:21:02Tim Scarfe?

  Yeah.

• 1:21:03Speaker 03

  I don't think there's really a path right now which gives us that, that amount of progress. Like not the non-parametric people, not the parametric people, not the like mechanical interpretability people, no one can tell us.

• 1:21:17Tim Scarfe?

  Yeah, I guess we live in really strange times and so, you know, like we don't even know what to believe anymore because, you know, we, we, we had all of these, all of these things that we used to say confidently a few years ago. And yeah, we've got things like the double descent phenomenon, we've got the mystery of overcoming the curse of dimensionality. From a sort of linguistics point of view, people talk about systematicity and productivity and why, why, why are the models syntactically competent? Why can they produce all of these sentences that have never been uttered before? Why are they, why are they learning these abstract models? Are they too incomplete? All, all of these different things. And we need to have a theory that kind of explains this phenomena.

• 1:21:57Speaker 03

  Yeah, I, I, I think there's also a big problem of that a lot of the research is really disjointed. And like, it's not like in physics, we have a standard model of physics and we kind of, we work around this. If you, if you go to school and you learn the standard model, you can then basically start, okay, we have a common starting off point. Like a common example, for instance, that people say, well, why does deep learning kind of beat the curse of dimensionality? Well, I can tell you, like Barron, back in like Barron's theorem, they literally proved that the actual convergence is not dimension dependent, it's a dimension free bound, it depends on kind of the, the frequencies of the target function you're fitting. And kind of intuitively, like once you think about this, it actually also makes sense. And if you have a high dimensional function which is constant almost everywhere, and

• 1:22:41Speaker 03

  then it's much, much easier to fit if you have a two dimensional function which looks like a heartbeat monitor. And in fact, I think the same thing is true for most other estimators as well. Like if you think about like a decision tree, same property, the, the, the high dimension, if it's constant in all dimensions but one, then still it's, it's dimension, it's, it doesn't care about all the other dimensions. So, but you would need to have something like a standard work which actually kind of looks at this and combines all of this. So something that doesn't figure out like the, the next paper, but something that actually goes and writes down, kind of tries to figure out how do all of these things can fit into one coherent narrative. And I'm not sure whether deep learning, or like machine learning, as it's presented

• 1:23:26Speaker 03

  right now, is actually equipped to do that. And because right now, the, the pace is so fast and like you, you have, effectively you have three big conferences every single year, if you don't count, count triple AI, count that, you have four. Effectively, you go from conference to conference to conference to conference and you, and all of these papers, like, I don't know, maybe it's, it's just me, maybe I'm just unlucky. But if I look at the papers I'm reviewing, they're really incremental, all of them. And I, I don't fault all of, all the people there for, for publishing like this because first of all, it gets rewarded. And secondly, it doesn't make sense to go and say, well, I'm setting up a big deep learning theory, I write this book over my PhD, like I don't do cumulative, I actually

• 1:24:06Speaker 03

  do a manuscript, I, I have in the end my, like I say, 5,000 pages big book, where I characterize everything that is, is to know. By the time you're done, it's going to be completely outdated, useless. Like probably lots of things you write and they are completely wrong. So I think that right now, there is no, there, there is no way to have, of course, we have a theory of deep learning. I, I talked about the, this paper, like this, there will be a theory of deep learning. It's kind of a vacuous statement because we arguably already have plenty of them. The, the, the problem is more that to have something like physics have, I don't think the entire field is set up to do that.

• 1:24:47Speaker 03

  And I think that it will not be set up to do that and up until the point where things start slowing down.

• 1:24:53Tim Scarfe?

  And can we just have a quick digression on the manifold hypothesis? So our mutual friend, Randall Balestriero, he's not a fan of the manifold hypothesis. So he, he's got this spline theory, which is basically that in an input sensitive way, the, the, the ambient space gets partitioned into these little honeycomb regions and, and you were helping me understand this at the time. So you get this weird teleporting thing happen where like if you change the input a little bit, you'll teleport into another honeycomb somewhere else in the space and it's kind of anti-manifold hypothesis. But I spoke with Goodfire last week, they're this interpretability company in San Francisco and they've got all of this work about the, the geometry that these models learn.

• 1:25:33Tim Scarfe?

  So for example, days of the week and months of the year, they can take a PCA projection of, of the, the activations and then they, they fit this kind of Bezier curve to it. And, and they see these beautiful kind of geometric structures in the activation space and, and they're really kind of manifold peeled. So what, what do you think about that?

• 1:25:54Speaker 03

  I, I think it's, it's really hard to make such a statement. Like whether, whether it exists or not, it, like something like the manifold hypothesis, like if it's not the manifold, like if it's not like a low dimensional manifold itself, something like this has to exist so we can actually compress things down. And ultimately, I don't, I don't even think that Randall would disagree with me on that, that if it actually is like, if, if we actually have like, let's say, the model outside, like the real world is the smallest thing there is and there's no simplification, then what are we doing?

• 1:26:26Speaker 03

  But it would also push back against all of these explainability techniques. Because generally speaking, they're, they're not very robust. Like they make, firstly, they, they often make extremely simplifying assumptions. Like, let's say, like the, the circuit, people, like the, like the, the transformer circuits, but they usually make the, they usually make the assumption that the actual routing is constant as you change, kind of as you permit your, your, that's not true. Like you effectively, your model transformer, like a, a huge MLP mixer. So I, I think there are many, many like silent assumptions and I think there's even a paper like this where they show that if you actually change some, make some slight tweaks on like a setup of like these experiments, like the results completely flip.

• 1:27:11Speaker 03

  I think like all of these are just, they're not robust enough to actually make such statements of making statements of like the manifold hypothesis exists, but it's also robust enough to say it doesn't exist. I, yeah, I, I, I think it's really, really difficult. I think it's also difficult because I don't know in, I don't know whether the language exists to realistically talk about these problems. Because like the, there are of course people that go very much into like the, these, the abstract functional space. I don't think this is completely true. It's often perhaps very useful, but it's not completely true just because we don't have infinite dimensional functions. It's not, they are finite sized and there's a big difference between finite and infinite.

• 1:27:54Speaker 03

  But if you have infinite dimensions, every problem becomes convex. In fact, every problem becomes linear. So on the other side, with the parametric people, we say, well, we just have these, these spline regions and that, that explains everything. But if, if it was like this, then we would need to be able to make predictions that say, well, if you go away from the splines, maybe it doesn't work anymore. Or like, to what extent does it stop working? And then there are the explainability people that say, well, we can actually extract this and that circuit, or we can extract this and that representation. Like maybe one of the, the best papers I, I saw in like explainability was the paper about Grokking, where they, where they extracted like the modular arithmetic code. Look at this, like curves, it completely makes sense, there are sine and cosine everywhere,

• 1:28:38Speaker 03

  especially because they, they do it in the input. But if this is how these models actually describe modular arithmetic, there's not a chance we will explain how the model, like a duck. So I don't think there is a way of of actually currently having just a language of describing of describing what these models do. So I don't think that the like something like a manifold hypothesis, I don't think it's true, I don't think it's wrong, I don't, I don't think it's a useful way, like even if it's true, I don't think it's a useful way of thinking about the problem. Same thing is even if like the extracted like grocking star representations, even if they're true, I don't think they're a useful way of actually figuring out what these models do and

• 1:29:21Speaker 03

  kind of how they get the solution and whether things work or don't work.

• 1:29:25Tim Scarfe?

  Yeah, I don't know. I mean, it's such a tantalizing idea, you know, like there's this idea of a homunculus, that we've got like a little man in inside our brain, or you know, these grid in place cells, basically that we've got a little world inside our brain that actually captures some kind of geometric invariant, some kind of model of the world outside. And it's a beautiful idea, isn't it, that, you know, when when you visualize these representations, for example, geography, you actually have some kind of geometric invariant of how the different countries in the world are related to each other. And it's fascinating that that this would be convergent, that different model families would learn similar types of geometry that represent abstract structural invariances of the world. It's very, very exciting. But yeah, I agree, maybe we're reading too much

• 1:30:08Tim Scarfe?

  into that. We should move on to Reward is Enough and constrained RL. So Rich Sutton, the wonderful Rich Sutton, he had this paper out called Reward is Enough. What did he mean by that?

• 1:30:21Speaker 03

  The idea of, so the idea of like especially like the reinforcement maximus is that they say, well, technically, I can write down any objective I want as some form of reward. Basically, just in doubt, you literally just go and say, well, you should be doing, I don't know, you should be moving the arm like this, you can write this down as a reward, basically, it's one everywhere it is. Or like if you move to a goal, it's one where the goal is and zero otherwise. And this is a very true statement, right? If I have an optimization problem, let's say I have a really bad one, like a mixed integer nonlinear program, huge, like millions of parameters, huge numbers of constraints, whatever, I can still write down the solution really easily. And that's, I have my X star,

• 1:31:06Speaker 03

  so my actual solution, I just write it down as like minimized error between X and X star. It's a complex problem, it instantly works. Okay, if it's relatively low dimensional, you can solve it by hand, even if it's really, really high dimensional, like the actual problem behind it is really, really high dimensional, really complex. So in that sense, if the the idea of like reward being enough is completely true. But technically, it is, technically, you can define a reward function for pretty much anything, it will somehow figure it out. The question is more whether this is a realistic assumption to make, like whether having like a God given reward function, which just does something, is

• 1:31:47Speaker 03

  whether it's a realistic assumption, even finding it, and once we have it, whether it's realistically optimizable with any technique. And that is where I think the the actual, like the practicality or like the the reasonableness of this breaks down.

• 1:32:03Tim Scarfe?

  And just help me understand, is Sutton advocating for the God reward function, or is he more practical? Is he saying that there are reward functions in different applications or whatever?

• 1:32:14Speaker 03

  So of course, like the the God function was a very, very extreme example. But ultimately, what he believes is that he believes that you can figure out like a suitable reward function for every task. And that is enough, like we don't need to research all of these other complicated things. But instead, if we if we just become really, really good at maximizing reward, then everything else becomes obsolete. I mean, I think it's a very, very optimistic view of the world. I'm not sure whether it's, I'm not sure how far he himself would be going with this. I don't want to put words in his mouth, but I don't think it's a reasonable thing to do, even in like

• 1:32:58Speaker 03

  the the kind of infinite limit.

• 1:33:01Tim Scarfe?

  I mean, give us a bit more context here. So when I was talking with Chris Watkins the other day, he was kind of saying that he met Rich at some conference in the late 1980s or whatever, and there were people in his lab at Cambridge doing all sorts of control theory. So for example, there's this cart pole balancing problem. And and you can just solve that analytically, but this reward thing was different, because it's basically, in this particular case, I think it was a binary signal. So it's either balanced or it's not. And and it's a signal from the future. So you have to do a lot of stuff before that thing says yes. And and at the time, they thought that this was an interesting problem to solve, because this is much more kind of like amenable

• 1:33:44Tim Scarfe?

  to, you know, maybe it's a better representation of animal learning.

• 1:33:48Speaker 03

  Yeah, I think there's in general, there's a there's a big overlap between control theory and what the reinforcement or reinforcement learning people do. And but there's also a huge distinction. And notably, the the main difference between doing actual control and doing reinforcement learning is control assumes that you have, essentially, you have a model of the plant, or at the very least, you have some idea of the behavior, like the actual, the actual response to an input. That is not true in reinforcement learning. In reinforcement, you say, well, I don't have a model at all, I just have interactions with the environment. And that immediately makes the problem more, first of all, it makes it more interesting, but also makes it a lot more

• 1:34:28Speaker 03

  complicated. Because now, you you no longer are trying to solve the actual problem at hand, you also solve the problem of even figuring out what to do in the first place. A controller, let's say we have like a reference, which it tries to follow, we have some model of the like plant dynamics, let's say, make it very extreme, we have an MPC controller, or we have a, or we have whatever else, or like a linear co-regulator, with some idea of kind of what this, what this thing does, or if we have like a PID, we have like a human who has an idea of what it should do and how it should behave. And then effectively, you outsource all the exploration. While in reinforcement learning, you say, well, I I put this into the loop itself. And that gives you,

• 1:35:10Speaker 03

  of course, it gives you a huge amount of power. Because now, you can actually solve much more general problems, problems we don't even have an idea of how to solve them in the first place. And this is also maybe where like the the reward is enough, people then go in and say, well, if we can solve an arbitrary problem with like arbitrary definition, and we can do like the exploration we can do, we can do the learning, we can do the optimal control, like why do we do, why do we need anything else? And again, while I think this is true, again, in the limit, I also think it's a very wasteful way of thinking about the world, especially since things like exploration become really dicey in many domains. Like, I don't want like my,

• 1:35:52Speaker 03

  like my open heart surgery robot to do exploration on, first of all, not me, but not on anyone else either. So I think that the the actual real world complexity of needing to have either of first of all, not allowing too much exploration in in the practical sense, but also to give really, really hard guarantees, becomes difficult to do in like pure reinforcement learning. The classic example where where we can actually prove that doesn't work for machine for like deep learning, is we cannot give like L infinity control, like L infinity guarantees. So because everything is sampling, sampling and kind of absolute maximal errors don't really play well

• 1:36:35Speaker 03

  with one another. You can have a controller, which in expectation is really, really nice, but has one really, really huge failure case. Right? So on average, something which tracks it perfectly and has one huge error where kind of the robot clocks you on the head, is completely the same as one thing which kind of just slightly wanders around. And in fact, there's this nice paper about, what was it called, like deep training, deep re networks to uniform accuracy is intractable, or something like this, put into Google, we'll find it. But they basically show that for real networks, it's not even possible. Or basically, it is possible, but it's exponential number of parameters. Because basically, what they say is, when we have the the relus, if you have two, effectively form like a hat, something like this, and just figuring

• 1:37:20Speaker 03

  out kind of how to place and how to scale these hats is already a complete problem, like, oh, rather, you need exponential amount of data to figure out kind of these these deviations from your reference. And I think having something like this is, it's necessary for many domains, where actually, like a like damage is really, really, really bad. And having just a chance in expectation that damage and good things happen most of the time is often not good enough. Like you want stronger things, where then suddenly you go into things like robust reinforcement learning, or something like this, or what I do for my page, the constrained reinforcement learning, where effectively we do a little bit more, just to get this little bit of an extra

• 1:38:05Speaker 03

  boost in either safety or performance, or just including prior data, like imitation, and we just say, well, I have actually lots of data around, and I don't want to do open heart surgery from scratch. So I think it's very naive to think about this in the right way.

• 1:38:22Tim Scarfe?

  Yeah, because I suppose this is the other vexed issue with with our friend Rich Sutton, is that he has this bitter lesson essay, which is basically that he doesn't think that we should be building knowledge and constraints into the system, he thinks that we should just scale, you know, models of computation. And I'm a big believer that constraints are incredibly valuable. And by constraints, you know, more broadly, I'm just talking about knowledge. I think constraints happen at multiple levels of abstraction. So in the physical world, constraints might just be the laws of physics or whatever. And in our minds, the constraints might be things that we already know. And this guy, Chris, the other day, he was saying to me that one of the amazing things about RLVR is that it kind of, it doesn't solve, but it helps the exploration problem,

• 1:39:04Tim Scarfe?

  because the base LLM knows so much. So it has lots of conceptual understanding, and rather than just blindly searching all over the place, because obviously, back in the days of deep Q networks and whatnot, we actually needed to guide and and put a lot of priors into the exploration process, because otherwise, not very many interesting things will happen. And now, of course, with the RLVR on the language models, that they have a lot of base knowledge. But it feels to me that like how you get there and how you constrain and shape the search space is critical to making it work.

• 1:39:36Speaker 03

  I have the nice property that I'm a very theorem many person, but I also work like Fraunhofer, which is very applied. So I actually did some project, which I actually applied reinforcement learning. And this is a big problem in reality, that effectively information can be very expensive. And in fact, I would say most information is very, very expensive. And reinforcement, if you completely say, well, constraints don't matter, I don't need any prior information, I don't want to encode anything in there, assuming this is even possible, being completely unconstrained, etc., and completely without prior knowledge, let's assume all of this is the case, we still have the problem of now needing to

• 1:40:17Speaker 03

  relearn everything from scratch. And the reality is that we want to be gathering as much information that we already have paid for and gathered with our blood, sweat and tears, we want to put all of that in the model first, and then we go and we continue from there. There's no point in basically starting from nowhere. There's also the reality that if you actually look at a reward function, like an actual industry-ready reward function, you will immediately see that they have constraints in there. Like, they just reframe them as well, it's a reward function, because yeah, of course, you can put them into as penalties. But ultimately, you're just kind of, you're just doing a lot of work to not write down

• 1:41:01Speaker 03

  a constraint. The nice example of this, which I always like to use as a motivation for constraint RL, is if you look into the paper about what's it called, OpenAI5, which is the Dota-playing AI. If you look into the appendix, there's a huge table, where on the left, they have all these different attributes, which they put into your reward function, on the right, they have some explanations and so on. In the middle, they have just a bunch of random numbers. And these are, they have the weights for their reward function. If you tell me that there is a way to actually figure this out, there's not a chance. So especially if you look actually at what these reward functions are, like at the bottom, for instance, there's one, which says, well, the eight, like the, there's a negative reward

• 1:41:46Speaker 03

  for being in the wrong lane for a specific amount of time. That's a constraint, because right is done exactly the same as saying, well, don't be in there more than five seconds or whatever. Or rather, in mouse constraint reinforcement, you say, well, in expectation, so if you take a thousand trajectories, don't be in the wrong lane too often, or something like this. There's another, like, another nice example, I can also run into trouble with reinforcement learning, is above, there is a section where they have like, killing an enemy hero actually gives a negative reward. And the reason is, and they have an explanation next to it, is that killing an enemy hero gives you a lot of gold, and there's already like a bonus for getting gold. So if you also had like a zero positive reward for killing an enemy hero, you're sitting to the middle of the map and just

• 1:42:30Speaker 03

  kill the enemy hero over and over and over again, without making a progress. The problem there is that two of these quantities can be completely correlated. So if you model constraints, you basically get rid of this correlation problem. Because if you, for example, write down the constraint explicitly of, I should not be, like, there should not in expectation, I should not have more, less, let's say, more than, how was it, I should not have, let's say, 50% less enemy hero curse than my opponent in expectation. But it's a small guarantee, but effectively, the stuff that you want to get rid of, you already get rid of with that.

• 1:43:07Tim Scarfe?

  Yeah, I mean, Leonardo da Vinci famously said that constraints give us freedom. And I'm a huge, I'm a huge fan of that. But you specialize in this mode of reinforcement learning called constrained reinforcement learning. Can you explain that?

• 1:43:21Speaker 03

  So constrained reinforcement learning is still a relatively up and coming field, I would say, at the very least. So especially since there are many different formulations. The classical one is one of the constraint market decision process. So effectively, what you have is exactly the same setup, you maximize the cumulative rewards. But then you have constraints, which just say, the expected cumulative costs should be below a threshold. This is, ultimately, you have very, very similar structures between the reward and the constraints. But there are also other ways of framing this. Like, the one big disadvantage of this expected cost is, for instance, you don't get rid of long tails. Right? If you

• 1:44:01Speaker 03

  say it should be less than 25 cost, then it could be that it has 50 trajectories of cost zero, and then 50 of cost 50, so averages out to 25. But it's not necessarily what you always want. On the other hand, for something like the OpenAI5 case, it would be exactly what you want. Right? You want to have, let's say, some games where exactly you are, like everyone is in one lane, and then some games where it's like the normal distributed amount. If you want to kind of go further, then there are also solutions which do like conditional value at risk, which is effectively, you look at kind of the probability of being in a tail, and kind of you weight this by kind of how far out on the tail you are. There

• 1:44:43Speaker 03

  are like percentile risks, like with 99% probability, you should be within like the some, let's say 50% of some target, whatever. Basically, like the pack guarantees. And all of these have like different domains. Like, I recently wrote a paper, which I have to re-write and re-submit, which is about kind of doing this if you actually know constraints completely, like you have a complete model, like in an operations research setting, but you don't know the objective. And let's say you actually have a complete model of, let's say, your local gas network, or like your local heating network, district heating network,

• 1:45:23Speaker 03

  and, but you don't actually know kind of the exact costs of, let's say, solar versus whatever, or like the actual demands. And then you train, or like you try to train based on that. And there, exactly, you never are allowed to have any failure. Like, the pipes should not burst, they should not burst like with 90% probability, it should just not burst. And kind of all of these guarantees, you try to encode into the training process itself. And there are different ways of doing that. So one of the very earliest was, basically, they went and looked at TRPO and said, well, we have this like performance improvement bound, we can do exactly the same thing for costs. And this gave you like CPO, like

• 1:46:05Speaker 03

  where effectively it says, well, if you are within the constraints, and we are with 99% probability after doing that set, we're still in there. Problem with that, of course, is, yeah, but if we are not in the constraint set, then effectively, it just has to do like effectively greedy descent, effectively it replaces the reward of the cost function, it just does descent back down. And you basically see some like oscillations on the boundary, something like that. Then further ideas went and said, well, what if we do projection? Right? We effectively, we have like an approximation of where we expect the cost surface to be, we project back onto it, that has nice, so the nice property of this is, constrained reinforcement learning is actually completely convex. Again, this is because we assume

• 1:46:47Speaker 03

  we have densities, we don't have parameters, we have densities, so in density space, everything becomes linear and convex and whatever, nice. So projection actually makes sense. The downside of this is that you can have some really, really bad runs. And the reason is, what if you effectively constrained system is hallucinated, like you hallucinate a constraint somewhere, with something I talk in like my recent paper in like Safe MPO, you hallucinate a barrier, and suddenly, the, like your constraint model, it never walks in there, because it's very careful to actually project around. In fact, there's a problem which becomes worse the better your solver becomes. It's like this hallucination problem. And then, of course, there are the final people who just say, well, we have zero duality

• 1:47:32Speaker 03

  gap, why not just use Lagrangeian approaches, which is effectively like this min max style? This works, but then you have the problem of also this oscillation, right? Where effectively you go down and you oscillate around, because you basically you're below the cost, because your Lagrange was slightly too large, then you're very safe, so the ground drops, and you go above, and you go below, and so on. You have these big or small oscillations. Them methods are tried to get around this, like there's something which basically has PID style, like it's tuning. That's pretty interesting, but ultimately, you still run into exactly the same issues. Right? You have the oscillation errors in there. And that's fundamentally because like estimating Lagrange is from data, like constraints from

• 1:48:17Speaker 03

  data is really difficult, because you have this variance in there. And then, there's like a final family, which is for example, also a Safe MPO paper I published recently in there, that is like CBPO, like this, or like, what is the other one, like FourCops is another one, which basically tried to do like effectively local approximations, and try to solve the problem locally optimally. And then they train the model to effectively just emulate this locally optimal solution. And that has some nice properties, it can be very, very precise, but now you need to be really, really careful not to maneuver yourself into like a, effectively an unsolvable setting, where like your sub-prom is unsolvable because you had a bad batch. And of course, it's model-based, but that's, model-based is easy to explain, you effectively just do model predictive control, but with constraints.

• 1:49:02Speaker 03

  It's really hard to do that properly, because the bad thing is, you could be in a setting where basically almost every proposal you do is just unconstrained, or outside the constraints. But ultimately, the algorithms are very similar.

• 1:49:15Tim Scarfe?

  Yeah, I mean, I suppose the million-dollar question is, given that you said before that we already have constraints in normal reinforcement learning, what's the cash value of explicitly modeling the constraints?

• 1:49:24Speaker 03

  So the big advantage is, so what you gain is, of course, you first gain the ability to more accurately, especially if you have like experts, like you go outside, you talk to some engineer, he will be much, much more easy to talk about when you talk about constraints, than when you talk about, let's say, tuning a specific weight of a reward function. Especially since constraints oftentimes can be known ahead of time with like high certainty. Like it's say a robot, you know that a certain load just will break the robot, you know that even without trying it out. And if you try to do this with like the effective tuning, effectively, there's a very high likelihood that you have accidentally something which goes above the threshold. Especially since it's oftentimes really, really difficult

• 1:50:08Speaker 03

  to think about these coefficients. The second thing, the second big thing is, you get composability. Because one of the problems, I talked about with the OpenAI example, was that if two elements of your reward function are correlated, and you, and for example, you add something to it, like let's say, you didn't at first, you just had like a bonus for killing enemy, you did not have like a gold bonus at first. And now you add the gold bonus later, because well, during engineering, you found, well, that would be a good idea as an additional guide. Suddenly, your entire system breaks, and you need to retune all the weights, because the weights are tied relative to each other. If your model has a constraint, let's say,

• 1:50:48Speaker 03

  you should have more gold than your opponent in expectation, right? That's a reasonable idea. Then

• 1:50:56Speaker 03

  Then you would not need to retune because effectively like you still have these coefficients in the Lagrangian setting, but they get automatically tuned to enforce the actual target that you want, not the target you implicitly encode with these tiny weights.

• 1:51:12Tim Scarfe?

  Very cool. And so I'm really interested in constraints. So you know Jeremy Bud Bud, right? He's also in Yanex Discord server. And we wrote this article called Why Creativity Cannot Be Interpolated. And basically, the idea is that creativity is all about deeply respecting constraints, and that representations are path dependent in some way. You might have read that Fractured and Tangled Representations paper from Kenneth Stanley. And it's kind of interesting because in neural networks, they seem to have soft constraints. So they learn this statistical distribution and then you sample from them and they kind of respect the constraints, but they don't sort of, they're not hard constraints. And I visualize problem spaces a bit like a maze. So a maze has walls in it and the walls are the constraints. And if you respect the constraints, then you don't walk through the walls.

• 1:51:57Tim Scarfe?

  And you can be creative because you can just take paths, you can take trajectories through the maze, and you can find new undiscovered subspaces, if you like, because you're not constantly wasting time walking through the walls and not respecting the path that got you there. So, you know, I have this deep intuition that, you know, maybe it's neuro-symbolic, maybe it's some kind of way of hardening the constraints that we learn within particular domains. And so it's unclear to me whether neural networks could in principle have these hard constraints, or whether we would always need to somehow enforce them in after the fact.

• 1:52:32Speaker 03

  That's a really interesting question. So generally speaking, if you can encode your constraints into like your architecture, you're probably going to be a lot more well off. A classic example for reinforcement, you just, you do action masking, or kind of you scale your actions to be in the range, which is reasonable. The question is usually that there are things outside the model which actually inform the constraints. So for example, reinforcement learning, actually having action constraints, pretty simple, like effectively just project onto some set. The difficult thing is kind of having effectively state constraints or something in the environment. Because now you can no longer just enforce these by going and modeling them in your architecture.

• 1:53:15Speaker 03

  You have to be able to interact with the state and the architecture together. There's sometimes ways of actually also encoding this into your model. But whether this is generally possible, I would honestly, I would doubt this. Especially since there are plenty of constraints which are NP-hard to enforce. Like let's say we have a CSP style problem, this is NP-hard to enforce. How are you going to do that in a static architecture? Like mathematically, it already, it breaks from a complexity point of view. Of course, you can then do things like effectively do things like a diffusion-esque thing, or like a Monte Carlo-esque thing. But fundamentally, then you still need to externalize some structure. If it's of the constraints, then it's the way you actually generate the output.

• 1:54:01Speaker 03

  So I do think there is something to it with this idea that constraints can help doing like sensible exploration, by basically removing things from the state space which are irrelevant. The complexity here is kind of actually forming how to compare these unconstrained versus constrained settings. Because fundamentally, if you look at literature, there is the paper by Altman, who did like his PhD in like a constrained reinforcement learning about like the actual, like the mixed interprogramming or like LP style. And one of the things that was even known back then is that constrained reinforcement is a lot harder. Specifically because there is not necessarily a stationary deterministic policy anymore.

• 1:54:46Speaker 03

  I have to be careful with this. There are non-stationary deterministic policies, and there are deterministic, and there are stochastic policies which are stationary. But stochastic policy which is stationary. But basically, this means that already this idea of doing something like Q-Learning does not work. It does not generally give you a solution. Because basically, you can get into some really, really, if you think kind of at the boundaries. So technically, if you look at this and if you ask them, they would say, well, mathematically, it's a lot harder. But of course, this is kind of, it's a wrong comparison. Because you're comparing a constrained problem versus an unconstrained problem. A constrained problem, of course, is harder because it needs to do more things.

• 1:55:29Speaker 03

  So it's, so the actual comparison would be something like, I have an unconstrained problem and I have a constrained problem, which is, both of which are really, really difficult potentially, but then compare them on the same task. And that becomes difficult because just designing a reward function, which actually implicitly informs a specific constraint is difficult. So maybe, probably would need to do kind of the other way around. Like you would need to have, okay, the RL agent actually figure out an optimal solution. And then we need to go and based on kind of the violation and so on, you would need to define the constraints. But then, of course, you start off with the constrained RL method on the back foot

• 1:56:12Speaker 03

  because it basically needs to emulate the unconstrained model. So I think it's really hard to compare of whether it makes a difference in, whether it makes an actual difference. It might just be that for the solutions that we see, it makes a difference, but it doesn't make sense in the absolute, it doesn't make difference in the absolute setting. But I can't think of an easy example, which does not, or like an easy experiment, which does not fundamentally disadvantage either one, either method.

• 1:56:44Tim Scarfe?

  Yeah, I mean, I often think about when constraints can be harmful. And I often use this expression, I got this from Marvita Chiramuta. So she said that reality is protean. And what she meant by that is there's this ancient myth in Greek mythology about this guy Proteus, and you could pin him down and he'd have to answer your question. But when you let go, he would shapeshift. So there's this thing that with problems in the real world, when we interact with them, they shapeshift and they're non-stationary. But what we try to do, though, as mathematicians and scientists, is we learn knowledge and representations at an increasingly high level of abstraction. And the purpose of that is that the knowledge is less shapeshifty.

• 1:57:26Tim Scarfe?

  So it kind of feels like we almost need to have a spectrum of representations where we admit fewer constraints because we allow for the fact that lower level representations change as we interact with them. But maybe we privilege higher abstract representations that are more known and invariant.

• 1:57:46Speaker 03

  Yeah, I think there's generally something to it of, basically, if, once you have abstraction, it becomes really, really difficult to, effectively, the fundamental constraint that you want for a hierarchical abstracted system is that the lower systems guarantee some invariant, but the upper systems can actually guarantee. This is also, in my opinion, one of the main reasons why hierarchical reinforcement learning never really took off. Because the lower level does not give you enough guarantees for the higher level to actually exploit. This is why in control, you can do that. Control gives you very strong guarantees. In doubt, you can have like an agent infinity controller, which gives you like effectively worst case, like effectively adversarial robustness. And you can still work with this.

• 1:58:31Speaker 03

  But the same thing cannot be done with reinforcement learning because the lower level, it just does whatever and the upper level just needs to deal with this. So I think there is something to just effectively in an abstraction hierarchy to basically, to basically make a statement that the lower level just has some guarantees. I don't fix what the lower level even does, like I don't care. It just has to uphold a certain thing. And that is something, it's a constraint that I can basically rely upon when I build my abstraction on top. And we see this to some extent already, even in deep learning. Like many people invoke the universal approximation theorem. Very few people have actually read what the universal approximation theorem actually does. But many people invoke this because they basically say, well, the invariant is,

• 1:59:15Speaker 03

  if I have a large enough model, it's just going to do the right thing. If it's not technically what the universal approximation theorem says. But still, you need to have in abstraction, you need to effectively have a stable interface. What the lower level things do, you don't care. But you need to be able to rely on this, and this could be also seen as a constraint. Does the constraint comes kind of from the prior methods, potentially also from the upper level. You generally speaking, have a level below you, which kind of has a constraint in the form of capability. And you have a level above you, which is a constraint in a sense of need. Like a robot needs to not crush you.

• 1:59:52Tim Scarfe?

  And where do our constraints come from? So how do we find these incredibly distilled, evolved abstract representations?

• 2:00:01Speaker 03

  I think that the main difference between a good and a bad abstraction is, again, is usefulness. If you, there are plenty of abstractions which never actually took off, never actually, we only see the good abstraction because the good abstractions are the ones that are actually used. There's a long tail of ones which was started and then kind of forgotten about very quickly. Ultimately, what would be an example to not piss off too many people?

• 2:00:31Speaker 03

  Fuzzy logic, so a piece of people, but fuzzy logic people, they're too little. They're not enough people to actually complain. Fuzzy logic was an abstraction with for basically like decision making and uncertainty, which never really took off because it had some limitations in like actual usage. For instance, like the reason we use like it's a Bayesian inference instead of or like probability theory in general over fuzzy logic. It was fuzzy logic has an un-nice property that it doesn't can't do fuzzy logic to do inference on fuzzy logic. For Bayesian, you can you can effectively do to Bayesian inference to to learn a Bayesian model. I think that the reason we see certain abstracts and not other abstracts is just because ultimately

• 2:01:16Speaker 03

  someone picked them up and they worked really nicely. This also, of course, on the contrary, means there are plenty of abstraction would be really, really nice, which never end up seeing like the the actual time of the day because but they they they just were never discovered by the right person to actually use.

• 2:01:32Tim Scarfe?

  Well, I think this leads us on quite nicely to world models. So what is a world model? Is it a compressed latent dynamics model? Or is it a density over observations? Or is it something else entirely?

• 2:01:45Speaker 03

  World models have exactly the same problem as JEPA because it's such an absolutely overloaded term. It can mean literally anything. And we have seen people talking about world models, which are effectively extensor type models, where effectively we have, let's say, a robot which has, let's say, like a tactility sensor or whatever, and then it tries to predict forward kind of what these things do or like a like a robot and it tries to actually do like the the motor predictions and so on, or like the effect of where the end effect that would be given some input.

• 2:02:17Speaker 03

  And we've seen things like world world models, which are effectively just image or like video diffusion models. Then we have seen more world models, which end up being like the the JEPA style, where effectively you have some some internal state, which then of course gives the problem of you don't know whether the internal state is actually useful. Many of internal states are just not that they they look nice, they are consistent, they're not useful, they're hard to benchmark. Then there are other people which say, well, we actually do like reconstruction. And they say, well, we actually reconstruct things which we don't actually end up using. But all of these are still world models. And I've also seen people like the the new Dreamer paper, which I don't know at what version they are, where effectively they they pre-train a large video diffusion model

• 2:03:00Speaker 03

  on like all the data to play Minecraft. I've seen people call this a world model. I've seen people say, well, it's just model based RL. I I don't think there's a reasonable difference anymore. The have to cut that I personally would make is if the the data collection is part of the loop, I will call it model based RL. If it's out of the loop, then I would call it a world model. And the reason is that, again, it's basically it's the control versus reinforcement learning setup, which I mentioned previously. In control or like model predictive or like nonlinear model predictive control, you assume you have all the data, like you have your plant, but you don't need to do any exploration in there. You just you you hammer on this.

• 2:03:42Speaker 03

  But for a model based reinforcement, you would also discover the world model and therefore you need to do exploration. And generally speaking, I think, but there are also people who think differently, say a world model is a thing which has trained on a lot of data to predict something temporal, which is not language.

• 2:04:02Speaker 03

  And I don't think there's a better description.

• 2:04:03Tim Scarfe?

  Yeah.

• 2:04:05Tim Scarfe?

  I mean, so many people are talking about this. I interviewed Nvidia about their Cosmos three Omni model, which they called a world model. And and they were kind of saying, well, you know, it's not just a world model. You you can use this as the basis to train a specialized world model. You know, you can use it to build build your robot policy for doing whatever. And our friend Yan LeCun, he's he's talking about world models all of the time. Is this just branding? I mean, what what's going on? Is this anything new?

• 2:04:31Speaker 03

  I think so. I think that world models and so both JEPA and world models are ultimately branding. There there were JEPA like models before JEPA, and there were world models before world models. In fact, at the Schmidt Huber paper, the world model Schmidt Huber paper, this is model based RL. So arguably, people that now use it for something else other than model based are just using the model as using the name wrong. I think it's mostly branding of techniques which are very loosely defined. They are just fitting some form of dynamics model, whether the dynamics model is probabilistic, deterministic, whether it's fit on like it's a visual data on audio.

• 2:05:14Speaker 03

  I haven't seen audio data, but probably there also exists one on on whatever, like they're all called. They're all called world models. It's just it's a dynamics model which predicts the state from like the next state from the previous state on an action. And I don't think there's much more to it. And if that's it, then people have been doing this for a very long time without calling it a world model.

• 2:05:39Tim Scarfe?

  And how how does this fit with the whole RL story?

• 2:05:42Speaker 03

  I think that it is a little bit orthogonal because so I always hear like LeCun saying, well, we don't need our we just need world models, right? Let's take the similar like the yeah, but it's just saying, if we have a world model which describes the world perfectly, then we can just use that and we don't need to do RL anymore. We can just do planning in that world model and we are fine. So we don't, we shouldn't spend all our time doing RL. The problem with that is, of course, it's a it's a very, very simplified assumption because I can give you a perfect world model of chess and you still cannot solve this. Like you will still not beat like Magnus Carlsen because there's a difference between

• 2:06:23Speaker 03

  being able to predict the future and being able to actually make inferences for what is the correct action to take. And I think this is very much underestimated, mostly because the tasks that are done with world models right now are not that difficult from a control point of view. And from a control point of view, like if you if you actually go, I was at RIG, which is the like the German robotics conference. And talking to people there is like one of the big challenges is having doing effectively a sensing part in a reasonable way, which also gives you relatively strong guarantees. Because if you firstly don't get strong guarantees, you might crush someone.

• 2:07:08Speaker 03

  And secondly, getting sensor information was always the bottleneck or for for most time the bottleneck to, I give an example, there is still not a good way of really doing tactile, like high tactile sensing, like the the reason you see like these like Chinese also like American, German, whatever, humor robots, where they always do things like back flips and sprinting and so on, and never just do the following tasks, just stand up, go into the next room, open up the fridge, kind of look in the back, pick out a carton of milk and bring it to me. The reason I do that don't do it is because that would be a much, much, much harder task because you have this this this contact rich environment, which is hard to model and it's hard to gather data.

• 2:07:50Speaker 03

  The bet for all of these like world model startups or world model papers is we can gather all the data you will need and just kind of distill everything, all these dynamics, can distill this into a model. And I don't think this is kind of, I I don't think it's a reasonable expectation because the world is just, the world is so strange. I don't think this will be that reasonable. On the other side, of course, I I think you can get very, very far with this. I think especially like world models with like reinforcement inside the world model, you get nice information, you can do the simulation really fast, you get potentially things that create information, depending on how you set it up, that most you can do, like you can do infinite rollouts.

• 2:08:33Speaker 03

  It's a it's a huge, it's a paradigm shift for reinforcement learning, which we can also see in something like large language models, the something like GRPO, which effectively does doesn't learn a value function, does like the the advantage estimation on the fly, but just simulating a bunch of stuff that doesn't make sense in the real world, where like every simulation stuff is hugely expensive. It makes sense if you simulate it, which is really, really cheap and just you roll out everything, you run GRPO and you get amazing results just because like you you can hammer data on it. And you still have the same exploration problems, but you still need to figure out what is the optimal strategy for chess. But you have so much more ability to do experiments and that changes the types of algorithms you want to use.

• 2:09:14Tim Scarfe?

  And just devil's advocate though, did you see last week? And note to my editing agent, please pull up a video of this. It was on Twitter last week. I forget the exact name of the company, but they were talking about physical prompting. So they they basically built this, it was like a foundation Omni model type thing. So it does like actions and video and God knows what. And they they had a robot arm and and they were teaching it to do things. So they they they gave it a demonstration. They said, you know, you pick up this ball, you put it in the the bowl. And and apparently it had these emergent abilities. So for example, sometimes the ball would get stuck to its hand and it would actually use its other hand and it would brush the ball off the hand. And maybe that's a different thing because you're kind of providing constraints or providing demonstrations, it kind of figures out, you know, how to do particular tasks. But I was quite intrigued by that.

• 2:09:57Speaker 03

  Yeah, so I haven't seen the video you're talking about. But again, I would imagine if we imagine that these models generalize, then yes, we could imagine things like that, especially if you do like planning in, if you do planning in sufficiently large world model and assuming we can actually do, assuming assuming compute does not play a role and kind of latency does not play a role. And if we have all the all the data for the specific type of task, I can imagine NNPC is ultimately a solution. Like there is, if if you go to a university and you're looking for like the like the advanced control theory lecture and the the lecture after doing NNPC, they're all completely NNPC pilled because they say, well, I don't need to do basically anything.

• 2:10:41Speaker 03

  I can just, I have my huge nonlinear model, I just throw something like MPPI against it, it just works and I don't need to do anything else. And then they notice, well, actually, this this this takes a while to run, even if my model is really, really small. And I have all of these these local optima. If I add constraints, then it's very, very easy to have a model which kind of were actually like my most of these lectures I proposed are actually not not really feasible. And then, okay, I can do like, I can do optimization over this, but now I need to make my model very small or like have a specific form. Because if it's not hasn't have the specific form, it doesn't fit into like a linear programmer or whatever. And suddenly, basically, the the opinion starts to dim the more you use it.

• 2:11:24Speaker 03

  And I have a feeling that the same thing is going to be true for like the the world models. I think there's going to be a lot of like really, really easy things, like the the low hanging fruit, which are just going to be grasped. It might be that for many tasks, this low hanging fruit actually is already enough. But I have a feeling that we are going to hit some really, really hard limitations, not necessarily by the fundamental idea, but basically by the nature of how the real world works and kind of how diverse tasks can become and how complex tasks can become. Because ultimately, like, things like moving around balls on a table, like, yeah, I can imagine that working. I can also imagine the same thing working with like an NPC controller.

• 2:12:09Speaker 03

  For instance, like the example, like I could say, it gets stuck and you use the other hand to move it around. An NPC controller can come up with this. It would start planning and would figure out, okay, I need to do something like this, assuming that the model is accurate enough and you actually do that complete NPC. That's, I think that's not the argument. The argument is, can we make a useful enough model with strong enough guarantees to actually deploy this in production? Because deploying this in production, in production could mean kind of at home, I don't think this is where we will see these robots, at least not at first, not at least not in the like short to medium term and probably also long term. I don't know. But if we look at a production setting, like, if you, if you

• 2:12:54Speaker 03

  go to, let's say BMW and you say, well, I have a robot, it does the right thing 90% of the time, the other 10% of the time, it does something else. They will, like they was like, no, I will not pay you that money for a robot. I'd rather spend that money to kind of fine tune my CNC G code for a couple of seconds than, because it would give me more effective speed. Because the, the level of guarantee that you need in the real world is much higher than what people actually think. If you look at most of these things which actually do a complex decision making, like large language models, they're all like a software setting. We can roll back whatever I want. And even there, like if you look at Twitter, as routinely people just delete their computer because they did not sandbox

• 2:13:38Speaker 03

  it right. But if we are in the real world, kind of concept of deleting your, your computer would be to crush someone. So the level of safety is a lot higher that you need a lot of reliability, especially if like tasks which you do over and over again and need to work over and over again. And arguably, I would say that the, the, that the peak accuracy matters very little. Like the, the, what you need is you need some form of like, like in every state, there is a minimal probability of being correct. Because then you can start doing things like effectively, like if you can throw a common filter against this and do like multiple measurements and actually get the right results. If you have 51% accuracy

• 2:14:23Speaker 03

  and do whatever many measurements, you could, you drop your accuracy or like you, you drop your probability of error very quickly. And I think that right now, all of these methods are not set up like this. I think they are set up to look very nice in demos and to actually do some really impressive stuff. But how do you know, like how many times did it fail to do something? And what is the, of course, the consequence for like juggling balls is really low, but what are the consequences of doing something wrong? And, and like the physical world, this is, this is a lot bigger than problem.

• 2:14:55Tim Scarfe?

  Wonderful. Well, Alexander Matic, it's been an honor and a pleasure. Thank you so much for joining us today. And folks at home, if you want to catch up and chat with Alexander, so your, your, your natural habitat is Yannick Kiltch's Discord, but you, you do occasionally come into our Discord as well. And we're very grateful when you do. But thank you so much for joining us today.

• 2:15:13Speaker 03

  Thank you. Thank you.

• 2:15:19Speaker 06

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' health care to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, thehousewithjordypag.com. Not authorized by any candidate or candidate's committee.

• 2:15:49Speaker 07

  Your heart can tell you a lot about your health. Apple Watch Series 12 measures your heart rate every five seconds with the most accurate heart rate sensing and wearable. So your vitals app, now with heart rate variability, can tell you when something is off. And your readiness score can let you know when to rest and when to push. Hear the story in every heartbeat with Apple Watch Series 12. The features described are for wellness purposes only and not for medical use. iPhone 11 or later required. Based on Apple conducted study of heart rate accuracy, August 2026. Visit apple.com slash apple watch series 12.