How Deep Learning Finally Cracked Messy Tables - Frank Hutter

Frank Hutter, co-founder of Prior Labs, talks about TabPFN, a tabular foundation model that makes predictions in a single forward pass, and the research behind it. TabPFN is pre-trained on synthetic datasets drawn from a prior over structural causal models, rather than on real data. At prediction time it takes the whole training table as context and outputs an approximation of the Bayesian posterior predictive distribution, without per-dataset training or hyperparameter search. Frank explains how this grew out of his earlier work on AutoML and neural architecture search, how the priors are built and revised, and why tabular data was hard for deep learning for so long. The conversation also covers the TabArena benchmark, how the architecture changed from TabPFN v1 to v3, scaling to larger tables, using the model with coding agents, test-time compute, Google's TabFM, causal inference and interventions, and relational data. At the end, a short update Frank recorded after the interview covers the TabPFN-3.5 release. Prior Labs: TabPFN-3.5: https://priorlabs.ai/tabpfn-3-5 https://priorlabs.ai/careers TOC: 00:00 Introduction 00:44 Welcome and Frank's background 02:05 Why tabular data was hard for deep learning 10:17 Pre-training on synthetic data 12:52 The TabArena benchmark 19:28 From AutoML to neural architecture search 26:34 TabPFN as a learned algorithm 30:50 Bayesian prediction in one forward pass 39:37 Scaling to larger tables 47:48 Using TabPFN with coding agents 57:47 Output heads and architecture from v1 to v3 1:05:29 Test-time compute and adaptation 1:13:32 Google's TabFM 1:16:53 How the priors are designed 1:18:40 Correlation, causation and interventions 1:35:22 Relational and multimodal data 1:38:31 Use in organisations 1:46:38 The open research arm 1:50:21 Update: TabPFN-3.5 REFS: TabPFN v2, Nature (Hollmann et al., 2025) https://www.nature.com/articles/s41586-024-08328-6 Transformers Can Do Bayesian Inference (Müller et al.) https://arxiv.org/abs/2112.10510 TabArena (Erickson et al.) https://arxiv.org/abs/2506.16791 AutoGluon-Tabular (Erickson et al.) https://arxiv.org/abs/2003.06505 Beyond IID: How General Are Tabular Foundation Models, Really? https://arxiv.org/abs/2606.30410 Neural Architecture Search: A Survey (Elsken, Metzen & Hutter) https://arxiv.org/abs/1808.05377 Auto-WEKA (Thornton et al.) https://www.cs.ubc.ca/~hutter/papers/AutoWEKA-KDD2013.pdf TabPFN v1 (Hollmann et al., 2022) https://arxiv.org/abs/2207.01848 TabPFN-3 technical report https://arxiv.org/abs/2605.13986 TabPFN-2.5 report https://arxiv.org/abs/2511.08667 CAAFE (Hollmann et al.) https://arxiv.org/abs/2305.03403 TabICL (Qu et al.) https://arxiv.org/abs/2502.05564 TabICLv2 (Qu et al.) https://arxiv.org/abs/2602.11139 Google TabFM https://research.google/blog/introducing-tabfm-a-zero-shot-foundation-model-for-tabular-data/ TALENT benchmark (Ye et al.) https://arxiv.org/abs/2407.00956 Do-PFN (Robertson et al.) https://arxiv.org/abs/2506.06039 CausalPFN (Balazadeh et al.) https://arxiv.org/abs/2506.07918 Causal Foundation Models with Partial Graphs (Reuter et al.) https://arxiv.org/abs/2602.14972 RelBench (Robinson et al.) https://arxiv.org/abs/2407.20060 RelArena-α, TabPFN-Rel and RPI https://arxiv.org/abs/2608.16319 TabPFN on GitHub https://github.com/PriorLabs/TabPFN TabPFN-3.5 technical report https://arxiv.org/abs/2609.17895 Otto Group Product Classification Challenge (Kaggle, 2015) https://www.kaggle.com/competitions/otto-group-product-classification-challenge ---RESCRIPT:https://app.rescript.info/share/e99676c25ee6189fbf54c9be07eb623e

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Speaker 00

  This episode is brought to you by Michelob Ultra. Good times mean competitions between you and your friends. Like seeing who can throw some pigskin the farthest or rush the field the fastest. Winner gets an ultra because it's not just a superior like beer. It's also giving fans superior access to their favorite sporting events. Superior is worth playing for. Visit michelobultra.com/superioraccess to enter. Enjoy responsibly. Copyright 2026 Anheuser Busch InBev Michelob Ultra Beer, St. Louis, Missouri.

• 0:30Speaker 01

  Flowing ad budget on metrics that look great till the CFO sees them. That's bull spend and marketers are calling it out in dashboard confessions.

• 0:39Speaker 02

  I remember telling my boss, it'll be good for the brand when leads were slow. Yeah, it it wasn't.

• 0:47Speaker 01

  Cut the bull spend. LinkedIn lets you target by company, job title and more. Advertise on LinkedIn. Spend $250 on your first campaign and get a $250 credit. Go to LinkedIn.com/campaign. Terms and conditions apply.

• 1:00Tim Scarfe?

  Deep learning did not work for tabular data and now it works dramatically better than cat boost and XG boost.

• 1:07Speaker 04

  TapBFN is really this natural progression of AutoML where we learn this entire algorithm that is executed in a forward pass, a million rows times 1,000 columns. So if you were to throw this into an LLM, you would have a billion elements and you need to tokenize each of these numbers. So that would maybe be three, four tokens each. So you have three, four billion tokens in your contacts and LLMs wouldn't be very happy there. You're getting some observational data and you want to predict if I change this variable here, what will happen? So if I give this patient this medicine, what will happen?

• 1:44Tim Scarfe?

  Frank, it's amazing to have you here. Welcome to MLST.

• 1:48Speaker 04

  Thank you for having me. It's great to be on the show. Actually, two years ago, people asked me, hey, is there a possibility that you might be on MLST at some point? And I was like, ah, maybe at some point. And yeah, super exciting to be here now. Yeah. Hey, my name is Frank. I have been a machine learning professor for the last 12 years before starting Prolapse as a CEO. Now co-CEO Research, I'm very excited about that. I focused a lot on AutoML, automated machine learning, where I started the workshop series on AutoML, ran that for eight years. We transitioned that to a conference on AutoML, co-wrote the first book on AutoML, did the first MOOC on AutoML.

• 2:29Speaker 04

  So really did a lot in that space, but then also moved over to deep learning. Worked a lot on optimization of deep learning. So Adam W. and Kursan Liling is from Ilya Lashchulov and myself. So that's being used for training any transformer in the world. And then we put the two together and did deep learning for tabular data. And that is something that didn't used to work. And with TapBFN, we actually made it work. And yeah, I'm scaling this up, building foundation models and are really excited to revolutionize this world.

• 3:06Tim Scarfe?

  I think one thing that people might not appreciate is that tabular data is absolutely everywhere. And speaking from bitter experience, it's a nightmare to work with tabular data. So anyone who's built a machine learning model with tabular data, you've, I mean, obviously like pandas and, you know, it's like it learns, got some stuff in there, but you have to deal with missing values. You know, what do you do with the categorical features? How do you do some feature engineering? How do you do transformations that reflect the semantics of the problem? You, you, you need to really, really know what you're doing and it feels kind of kluge. You always, even though you can make an ML model, you always feel kind of dirty afterwards because you feel like you've done all of these shortcuts and kind of corrupted it in some way. Tell me about that.

• 3:47Speaker 04

  Yeah, I fully agree. Tabular data is very dirty and I think that's one of the reasons that deep learning took so long to actually do well for it. There's, there's been countless attempts for deep learning for tabular data. 2019, TabNet by Google was really hyped, thousands of citations. Yeah, the new thing for tabular data and it just doesn't work. It doesn't generalize to new data sets. It just, you know, you have heterogeneous data, you have outliers, you have missing values, you have categorical values, you have binary values, you have ordinal values,

• 4:28Speaker 04

  you have missing data, missing at random, missing not at random, all kinds of these data complexities. And yeah, you, you need to actually take that into account. And just the standard deep learning approach, sort of like, yeah, treated everything as the same. Like any image is the same in terms of, you know, like there, there's pixels and you typically have very, very similar spatial relationships between the pixels and so on. And you can take images, like you can have an ImageNet that is just all basically cover all kinds of different images in the world and learn on that one data set.

• 5:11Speaker 04

  But what you need for tabular data is actually, yeah, a whole lot of different tables. Because if you have one table from medicine and then you have another table from insurance, there's just nothing you can learn in terms of the individual rows. From one can't tell you anything about the other one. But what you can do and where you can do the transfer is on the levels of, of these patterns, detecting the patterns, how the different features interact, et cetera, detecting causality potentially. And on that level, using in context learning, you can actually transfer across different data sets. And that, that's why, yeah, within context learning, we finally saw this breakthrough for tabular data.

• 5:56Tim Scarfe?

  Yeah, I mean, I'm just, I'm just having nightmares thinking back to doing stuff like this. I mean, for example, I suppose traditional machine learning algorithms, they, they kind of wanted the data as a Euclidean space within a certain range. So you might have to normalize it, standardize it. Some of the algorithms that, you know, like if you're using SPM, it might actually not work very well if the ranges of the, you know, different fields were different. And yeah, it was an absolute nightmare. But, but can you just explain this a bit more? So, so before the job of the data scientist, you might do a whole bunch of, you know, cross scatter plots and you would do some like, you know, feature analysis and whatnot. And you, and you would convert it into this beautiful kind of convex Euclidean space.

• 6:39Tim Scarfe?

  And, and now we have this system using a transformer. And, and I guess the, the architecture is vaguely resembling deep, deep set. So it can actually intrinsically understand discrete data. And this is new, this is very, very new. So why are we in a different world now to where we were before?

• 6:58Speaker 04

  It can understand discrete data and also with, with architectures that actually work on, on individual columns or different types of preprocessing that are in the network that actually look at the statistics of a column and then actually make sure that you have the right embedding for that column. That's all embedded now and you don't actually really need to do this yourself anymore. Dealing with missing values, dealing with outliers, et cetera. That there was all like a lot of steps in your preprocessing. Having to deal with that is of course very nice and then just makes you so much faster. I mean, that, that's one of, one of the things we wanted to get to with tabular foundation models is that, you know, you really help data scientists do their work so much faster.

• 7:44Speaker 04

  I think in this day and age, new data scientists really don't need to understand everything about XG boost anymore and that there are certain hyperparameters and what changes if you change the hyperparameters, et cetera. That the job really changes. It's all about like actually thinking about what is the right framing of this problem? Where, where does the data come from? Is there going to be data shift? Is there data drift? Are there problems with fairness? Are there problems with causality? Are there problems with privacy? When I, when I field this model, what's going to happen? How's this gonna affect how the data is being collected, et cetera? That, that's a whole lot of completely different ballgame.

• 8:25Speaker 04

  And, and there is a very exciting questions there that people right now just have no time to look at. And actually being able to spend more time, more, more human quality time on a problem rather than fiddling with your hyperparameters. I, I think that that's going to be, be a much more fulfilling job as a data scientist in the future.

• 8:50Tim Scarfe?

  I suppose another reason why tabular data hasn't been very sexy. I mean, a lot of data scientists, what they love about the deep learning revolution is this notion of scale that you can go out on the internet. I mean, even, you know, I'm working in a bank or something and I might be doing document recognition and my headspace is very much, oh, let's go and gather up all of the documents and let's see if there's other documents on the internet. Let's build a data set. Let's actually, because there's this notion that I can just scale and I can add more data and it's going to work better. Whereas the mindset with tabular data is very much, it's almost like a lack of imagination. People are thinking only in terms of the particular data set I have here, not how it can be like, you know, not how I can enrich it with other data sets or not how I can use a tabular foundation model.

• 9:35Speaker 04

  Yeah, absolutely. This is, humans like to put their text online. They like to put their images online, their videos online. It's very easy to make a, yeah, a viral tweet with a generative video model, et cetera. But people do not put their tabular data sets online. This is just not very glamorous. And also it's kind of abstract sometimes to, to think about tabular data. But when it comes to our health or to our science and yeah, our financial system, et cetera, then it becomes just very concrete. And it's just very clear that this is a really important problem that, that needs to be solved.

• 10:15Speaker 04

  And having these tabular foundation models that are pre-trained on synthetic data, that was, yeah, the lever that actually allowed us to, to capture this. And this was necessary. We just couldn't take the corpus of tabular data that has been prepared by us, by, by, by someone for us. But we just needed to, yeah, generate our synthetic data because there, there is no tabular. Like there, there's some tabular data, but it's mainly like 50,000 data sets that are there. And, you know, and, and yeah, there is, on, on the web, there is millions of data sets

• 10:55Speaker 04

  that are millions of tables, but they're tables like on Wikipedia, this basketball player has this number on their back. It's not a data set that you can actually reasonably learn a statistical learning algorithm from. And those types of data sets just haven't been there and we needed to generate them in order to then learn on them.

• 11:17Tim Scarfe?

  And just to hammer that home, there has been an ImageNet moment for tabular data. Right, so until the day before yesterday, figuratively speaking, deep learning did not work for tabular data. And now it works dramatically better than CatBoost and XGBoost.

• 11:33Speaker 04

  TabP F N is the first algorithm that's actually been learned from data to be better at what it's supposed to do. So we've pre-trained that on hundreds of millions of data sets to actually just be as good as it can possibly be on the test set. And that is not peeking at the test set. That's not, you know, the, what a lot of people do when you're overfitting or something. You don't get to look at the test set at test time, but at meta train time, over hundreds of millions of data sets, you have learned to really solve this problem. And we're feeding the entire data set in context to the transformer. And the transformer can attend to the right parts of the data set in order to figure

• 12:18Speaker 04

  out what are the patterns in this data set? And what are the patterns that I should then also apply to the test data in order to predict well?

• 12:25Tim Scarfe?

  And an important detail is that these guys have synthetically generated a whole bunch of the training data. And this overcomes the, the overfitting problem to some extent, maybe not entirely. And you can explain that, but you know, so you synthesize data and then you build this foundation model.

• 12:43Speaker 04

  Yeah, absolutely. So I mean, there's a lot of strong theoretical foundations. We're approximating the Bayesian posterior over any type of prior that we can sample from. And that prior that we chose is a prior that has a lot of causality in there. And so we're basically building the Bayesian posterior of all kinds of structural causal models that could explain the data. And yeah, in contrast to L L M's and vision models, et cetera, we have trains on synthetic data. We had to train on synthetic data because there isn't a whole lot of data, tabular data on the internet. But rather, we can, like, we're the first foundation model that's actually state of

• 13:27Speaker 04

  the art, yet entirely trained on synthetic data. And this is great because we don't have any leakage, we don't have any memorization issues, et cetera. We can directly control exactly what's in the data. We don't have to worry about biases, et cetera, because we just have a codified pipeline for generating the synthetic data from scratch.

• 13:52Tim Scarfe?

  You guys run this website. Now, many, many folks at home, you've seen L M Arena. So that is using an algorithm related to the Elo algorithm in chess, essentially where, you know, in chess, you play people against each other. And then, you know, depending on the information gain of the result, you get a rank for every single player and it kind of converges over time depending on the parameters. So L M Arena did that with language generation and, and humans could rate whether one was better than another. And you've got this thing called Tab Arena. You actually run the site yourself.

• 14:22Speaker 04

  Yeah, so I mean, Tab Arena started as a collaboration between a whole lot of different people in tabula. The first author is Nick Erickson, who at the time was at AWS. He built AutoGlue on there, which was, yeah, the best Auto ML system for tabular data there was until, until now where, you know, you're combined with tabular foundation models and it gets much better. And yeah, also a bunch of other folks with, I think, five, six different affiliations. And yeah, by now, Nick has joined our team and several other authors have also joined our team. Because yeah, they all want to actually push tabular predictions to the max.

• 15:05Speaker 04

  And Tab Arena is, yeah, very open. We include any type of new model when it's a living benchmark. So when we see issues in any of the data sets or people say, hey, why is this data set here not in there? I think really this, this is an issue of, or hey, this is a new data set. Let's update this in order to get less overfitting of the community to this data set than we updated. And so, yeah, if you would like to get involved, by all means, we're continuously build new benchmarks. We have a benchmark that's actually now beyond arena, that goes beyond Tab Arena.

• 15:50Speaker 04

  We are building a benchmark for relational data, for feature selection. And all of these efforts are entirely open source and we invite collaborations with anyone.

• 16:02Tim Scarfe?

  And is it conceptually similar to L M Arena? So there are a whole bunch of different categories and, you know, you sample a comparison and then a diverse population of humans rate one as being better. Is it a similar kind of thing?

• 16:13Speaker 04

  Yeah, so in tabula, you don't need to have that, that step where the humans rate, but you just have a training set and a test set. And what is similar is that you compute Elo scores. And so you have basically those tournaments and you can say, look, if the Elo score is 100 points more than the probability that this algorithm wins against this algorithm on any of the data set samples is this high. So that, that is one similarity to L M Arena. But it's, yeah, it's basically an entirely open platform that publishes everything in

• 16:55Speaker 04

  terms of all the artifacts, all of the per split predictions of all the algorithms that are being compared and has been reproduced by people in the open source community.

• 17:07Speaker 04

  And yeah, so that makes it much easier for, for anyone to benchmark the algorithms properly. And running your own benchmark costs, you know, like something like $2 on a GPU. So this is something that's totally inclusive to anyone.

• 17:24Tim Scarfe?

  Yeah, that makes a lot of sense because I guess with language models, it's not objective. So, you know, humans need to compare the, you know, because we can't basically design an objective function. And with the humans, that's a problem as well, because humans love it when Star Trek is mentioned and they love, you know, G P T slop and all sorts of weird things are going on there. So in this case, you have an objective criteria and you can run that thing. Just final thing though, so the great thing about Tab P F N is that you can synthesize data. And what we don't want in benchmarks, I mean, it happens grotesquely with normal benchmarks, you know, because sometimes you're just training on the benchmark. But is there, is there an inadvertent form of leakage here where you're continuing to generate more data and you can kind of see what's going on on the benchmark? Is there a tendency to kind of inadvertently leak some data?

• 18:09Speaker 04

  It is not the case that we synthesize data and then that data goes on to the benchmarks, right? It's the benchmarks that those are all real data sets that come from the community and there we curate them from thousands of different possible data sets. And then we drop them for various reasons and every step is exactly documented. For example, I don't know, like for beyond arena, we dropped everything that's less than 100 samples. We could have also included those, but you know, with less than 100 samples, maybe you don't have as much signal anymore, et cetera, et cetera. So we drop duplicates and we drop, so for example, I don't know, you could see MNIST

• 18:50Speaker 04

  as a tabular data set, just taking every, every pixel as a number. But those data sets we drop because, yeah, we only want to have data sets where you would actually use a tabular machine learning algorithm and where that would make sense in order to not just get distracted with all kinds of data sets that look tabular, but really you should just use a vision classifier for MNIST. And because there, there are, yeah, spatial patterns and so on, that, and yeah, spatial similarity between neighboring pixels that you want to exploit. And if you cast this as a tabular data set, then you, you sort of pretend that there is that type of data in tabular data and that you don't want to have these patterns in

• 19:35Speaker 04

  tabular data typically. And so, yeah, we drop data sets for various reasons that are very well described. And again, all of these papers are open source collaborations where we're happy about really anyone who, who wants to put in the effort to curate these data sets. This is terrible work, much easier now with agents, but still it's hard to define correctly what the inclusion characteristics should be. And then, yeah, we just run these experiments and keep the website up to date and run, like, look at all kinds of Pareto curves, like how good is the predictive performance?

• 20:17Speaker 04

  How good, how fast is it in terms of training, in terms of inference latency, et cetera? And yeah, look at all kinds of statistics of the competing methods.

• 20:29Tim Scarfe?

  And apparently you did your PhD under the legendary Kevin Murphy.

• 20:33Speaker 04

  Yeah, indeed. Actually, three supervisors, Holger Huus, Kevin Leighton Brown and Kevin Murphy. Kevin Murphy was maybe the least believer in automated algorithm configuration, but he was still excited about some of the things we could do. And actually, in the end, when, when he went to Google, then, then he became excited about neural architecture search. And so then we actually collaborated more than before.

• 20:58Tim Scarfe?

  You've got a really interesting background because I remember when, when I was a data scientist, you know, about 10 years ago, there was lots of discussions about things like Auto ML and neural architecture search. And in fact, you were one of the guys behind that NAS paper. And I remember reading about that at the time, years and years and years ago. And yeah, just sketch out that, that history.

• 21:18Speaker 04

  Yeah, so Auto ML goes really way back to my PhD. My PhD was on automated algorithm configuration, which was all about, you know, Making all of the nitty gritty decisions in designing algorithms more automated. It came from SAT solvers where you needed to make like hundreds of different decisions at categorical spaces. And this was really annoying. So in my masters, I wrote a local search algorithm for one particular problem and worked on different types of problems and always you needed to make all these decisions. And I started automating that actually with local search to find better parameters of my own local search algorithm.

• 22:01Speaker 04

  And then, yeah, I got exposed much more to machine learning and then I used machine learning in order to find better parameters of my SAT algorithms. And then I moved more towards using machine learning in order to find better hyper parameters of machine learning algorithms.

• 22:16Speaker 04

  And that led then to, you know, model selection, choosing one of different types of algorithms. But then each different algorithm would have a lot of hyper parameters of its own. And then you have these hierarchical, really complex spaces. And you want to do optimization in that in a efficient way, but you also want to generalize to different parts of the distribution. So you don't want to be stuck and over tuned to one particular type of problem distribution, but really generalize to all kinds of problems you're going to see afterwards. And yeah, then there was this other design space, basically of neural architectures that for which the same types of approaches really are directly applied.

• 22:58Speaker 04

  It's a very categorical, very structured, all kinds of hierarchical decisions, et cetera. And so, yeah, we pioneered based optimization for that. And yeah, also definitely saw a lot of issues in neural architecture search in terms of, you know, how to benchmark different algorithms and so on. It's very complex to run these algorithms, very costly, and that makes it very hard to do good empirical science.

• 23:25Tim Scarfe?

  Exactly. I suppose, I mean, I remember when I was doing my PhD all those years ago, I was using something called Weka. And, you know, so you could try, oh, you know, let's use a support vector machine. Let's use a Bayesian network, let's do kernel ridge regression. And it was a wonderful toolbox where, you know, you say, I've got a prediction problem. And you've got, you've got a bunch of data and you've got some signals and labels. And you could just prototype all these different approaches and you could do, you know, cross validation. And I guess in the olden days, machine learning was mostly manual. And, you know, there was some hyper parameter optimization and so on. But I think this auto ML thing was kind of hinting to this structural component.

• 24:06Tim Scarfe?

  So wouldn't it be better if we could search the space of machine learning model structures in respect of the particular problem?

• 24:15Speaker 04

  Yeah, absolutely. I love that you mentioned Weka because there was a first base algorithm or base library of classifiers that we were working with. We did this auto Weka. I remember Weka, the image for this was this Weka bird. And so we had a little robot riding a Weka bird to be auto Weka. Yeah, you described it. It's all kinds of people uses. Often people from, you know, the sciences, life sciences and biology, et cetera, who are not machine learning experts. And there's all this stuff in Mecca, like dozens of different classifiers, dozens of preprocessors, all kinds of different hyper parameters of these. And, you know, what would people use?

• 24:57Speaker 04

  I don't know, like an SVM was default hyper parameters because their friend told them that that would be good. And that's just not how you get the best performance. And so you really want to, yeah, automate this in order to have better approaches for the masses. And that was really the, yeah, the leading principle behind auto ML is to democratize state of the art machine learning to everyone, also those without a PhD in machine learning.

• 25:23Tim Scarfe?

  So the problem with Weka was I was always so excited about neural networks. But in Weka, the neural network model was incredibly slow and it was always worse than using just, you know, simpler models. So I suppose then we had frameworks like Keras, for example. And then we started experimenting with, you know, different types of activation functions. And, you know, maybe we could have a CNN layer here and an MLP there. And it's very composable. You could almost just construct a neural network architecture and compose together frozen ones that you'd done previously or fine tune them. But again, we're back to this incredibly manual process. So how did the sort of the automated ML touch the neural network space?

• 25:57Speaker 04

  Yeah, I mean, so basically, as you describe, it's people were annoyed of having to do this manually and were thinking, hey, how can we actually automate the space? What should the design space look like? What are typically the right elements in the space? And how wide should I make my network? And how deep should I make my network? That was sort of the early design spaces. But then, yeah, convolutional neural networks came around, attention came around. And then you have all kinds of different hybrid architectures. And then on top of that, for different chips, you have different latency and so on. So you have this hardware awareness in there. Then you had, you know, the performance of the algorithms, like accuracy.

• 26:40Speaker 04

  But you also had latency, you had memory consumption, you had all these different objectives. So it became multi-objective and was just really a big playground for very exciting model development or method development in order to search through these spaces.

• 26:57Tim Scarfe?

  Amazing. So I suppose there's that old story. So as you were saying on your PhD, you were specializing in SAT solvers, which is just kind of how can I search the space of possibilities? And something really big happened, I guess it was around the, maybe the GPT model moment, something like that, where we had this concept of a foundation model. So rather than data scientists kind of trying to build low level models and searching the space of possibilities, what they increasingly were doing was using these foundation models that have been trained on loads and loads of data and then working from that as a starting point. So what was that transition like?

• 27:34Speaker 04

  Yeah, there was a really exciting transition. ChatGPT came out in November 2022. And a week before that, we actually published a Tabby FN one paper, which was the first foundation model for tabular data, which completely turned tabular data on its head. And you didn't need to actually do this model selection anymore, but you would have one pre-trained model that would actually use the entire data set in context and learn across millions of different data sets, how to make predictions for the test data in one forward pass. One particularly exciting part of auto ML over 15 years was the story of meta learning,

• 28:16Speaker 04

  learning the parameters of neural networks across different types of data sets in order to actually work well on unseen data sets. And you can actually think this bigger and learn and think about learning entire algorithms that generalize to new types of data sets. And Tabby FN is really this natural progression of auto ML where we learn this entire algorithm that is executed on a forward pass. And depending on the data set that you feed in, it learns a different classifier. So it's not just one network that is one classifier, but it's a network that can learn different classifiers depending on the input. And you could take this network and, you know, write it out as onyx, put it on a sensor.

• 29:01Speaker 04

  And then you have a sensor that runs not just a classifier, but a machine learning algorithm that depending on the inputs will actually give you different classifiers as output. And so this, yeah, you can see this as, for example, traditional algorithms like XGBoost, et cetera. All of these algorithms are hand derived, hand coded, et cetera. And Tabby FN is the first algorithm for tabular data that's actually fully learned end to end over a large set of different data sets that it's supposed to work well on. And it's, we were trying to optimize across entropy loss on the unseen test portion of each of these data sets. When you have a new data set, you of course don't see the test set. You can't peek at that.

• 29:46Speaker 04

  But at training time, that is the loss metric you actually want to optimize is you want to do well for this test portion. And we look at hundreds of millions of data sets and make sure that the algorithm works well on those. And then it will also generalize to new data sets. And that is a beauty of these. AutoML for for algorithm development. It just lets you to be really declarative. You can say for these types of data sets, you should work well. And for TAPIF N one, we had a fairly simple set of data sets. And then over the years, we made this more and more complex.

• 30:26Speaker 04

  For example, with TAPIF N two, we put in missing values and outliers and all kinds of other data complexities like that. Uninformative features was something that was broken before. Categorical features were something that didn't work well before. And we put more and more of these complexities into the creation process for the data sets that we would train on. And then, well, the objective was to do well on data sets with these types of data complexities. And all neural networks and deep learning just needs to do its magic and actually optimize for that objective. And it can do that well by pure supervised learning. So there's no magic there.

• 31:07Speaker 04

  And then out comes the an algorithm that executes in a forward pass. And we don't have this long training loop anymore. We don't have the search over hyperparameters, et cetera, at test time, but we can just do a forward pass. All of the other part of AutoML still is there, but it's there in coming up with this algorithm. So we can still do a search of our hyperparameters of the architecture of our neural network. We can search over our learning rates, et cetera, for the pre-training. But all of that at test time falls away and data scientists get what they always wanted. Namely, a just really fast method that works out of the box.

• 31:50Tim Scarfe?

  Now, I've just got this this model. And the next component is what you were talking about, this adaptive inference. So it's very, very similar to chain of thought adaptation in a normal transformer. So you know, you you you put a prompt in. And what it's actually doing is is your prompt is kind of conditioning the transformer. And it's kind of doing some kind of computation which is specific for the input that you give it. So in your context, without actually training the thing from scratch, you can put some tabular data in there. And and you're saying in a single forward pass, it is essentially creating a model that works specifically for your case.

• 32:27Speaker 04

  Yeah, absolutely. And and this is literally a single forward pass. So with with chain of SAD and L L M, you're you're auto regressively doing this. So you produce one token at a time. But with tabular foundation models, it's literally the one token you need to predict. And and that is it. Because you have sort of your training data, X train and the Y train, the label. And then you have X test, which is the data point you want to predict for. And you only want to predict the Y test. And if you have many X tests, then the X test two shouldn't depend on what you predicted for X test one. So there's this independence and it's really just one token at a time. And you you don't roll out auto regressively.

• 33:10Speaker 04

  And that lets us actually just directly optimize for the objective function that matters in the end.

• 33:18Tim Scarfe?

  So you've spoken about how this is conceptually similar to Gaussian processes. And I suppose we should also just contextualize that in in most machine learning models, even in current transformers. You you can say that they're approximating some kind of Bayesian model, but they're approximating a maximum likelihood point estimation. Whereas your model is actually approximating the pure Gaussian process. It's actually approximating the uncertainty across all of the possible values. Why is it possible for your model to do that and other models don't do that? Like, what's going on there?

• 33:51Speaker 04

  Yes, so what we approximate is directly the Bayesian posterior predictive distribution. What other types of Bayesian inference methods first do typically is to get a posterior distribution over the functions. So maybe to back up a little bit, so you have a prior over functions or equivalently a prior over parameters. And then you observe some data. And then you want to reason about the posterior over function or the posterior over parameters that describe the function. And for computing that posterior, you can use Markov chain Monte Carlo or you can use variational inference. And both of them have their issues. M C M C is just really slow and variational inference is, well, there's complex math and the approximation sometimes don't work out perfectly.

• 34:35Speaker 04

  And it's also sometimes a bit slow. And but then, once you have this posterior over functions or parameters, then typically, in order to get the posterior predictive distribution, the P of Y given X and the data, that is actually typically easy. If you have a sample based approximation of the posterior over functions, then yeah, that that's basically just a sum over that sample based approximation. But the tricky part is is posterior over the latence over the parameters or the functions. And that step, we just entirely skip. We just directly go to the Bayesian posterior predictive distribution, which is a one dimensional distribution. It's just the P of the Y.

• 35:20Speaker 04

  Y is a scalar for regression or it's just for for classification of the probability over the K classes. That we can do in a forward pass. How do we do that in a forward pass? Well, we sample from the prior. We sample functions from this prior over functions. And then we sample data points from each of these functions. So if you think of Gaussian processes, we have a Gaussian process prior, we can sample different functions from that. You you have some kernel, et cetera, that's specifying the Gaussian process so that that's telling you something about, you know, how how bumpy it is, et cetera. And so you sample functions from that.

• 36:02Speaker 04

  And then you sample data points from each of these functions and call some of them your training points and some of them your test points. And you basically learn to predict the test points or the Y value of the test points from the training points. And if you can do that for millions of samples of this function, then you have actually learned to sort of capture this the structure of the prior in in your network. And the network has learned to actually approximate this posterior distribution over what the Y value should be at these missing test data points. And has learned that over these millions of samples from your prior.

• 36:42Speaker 04

  And then, given a new data set, this is the first time you actually have a data set. Before that, it was really just samples from the prior. So then you have a real real data set. And you can compute the Bayesian posterior predictive distribution for that in a forward pass, just like you computed it for each of your millions of training data points. So you basically meta learned this posterior predictive distribution or meta learned to approximate the posterior predictive distribution for arbitrary data set inputs purely by two things. You need to be able to sample from your prior and you need to be able to fit strong neural networks in a supervised learning fashion.

• 37:24Speaker 04

  And both of these are actually quite easy if you have a mechanistic prior that you can sample from. And that's that's sort of been our workhorse to improve our priors. And then sort of the deep learning machinery, we just piggyback on what folks do in L L M. Then architectures get better, optimizers get better, and our methods also directly get better. And of course, we we adapt the architectures to tabular data as well.

• 37:47Tim Scarfe?

  Yeah, so we we can we can compute this posterior distribution. And so traditionally in in Bayesian models, we we kind of integrate over all of the possible hypotheses. And it's possible in this space. Because I guess I want to try and distinguish how this is different from something like transformers. So what you've done here is you've trained these models using synthetic data. So you've come up with a whole bunch of priors about causal relationships in different types of structured data. And you couldn't do this with language, for example. I mean, I mean, yeah, you could have like a context, you know, free grammar and you could generate a bunch of language. But it would just be some weird, inscrutable language that we've just invented. The really cool thing here is you can say, okay, there are these principles that are common to tabular data.

• 38:32Tim Scarfe?

  And I can represent those as priors. I can sample from them. And in your predictive architecture as well, the output space, it might be, well, it's it's going to be discrete. It might be categorical, which means there are relatively few examples. Or if it's a regression problem, you can discretize it to a tractable number of things. So essentially, you can actually do this thing end to end in a way that wouldn't be possible with many other types of models.

• 38:58Speaker 04

  Yeah, that's absolutely true. So if your prior is, for example, just a, you say, it's really simple, there's only linear curves. You can sample from that. What will come out as a Bayesian posterior predictive distribution is actually Bayesian linear regression. If you sample from Gaussian process prior, what comes out is a Bayesian posterior predictive distribution, which is actually just the G P posterior or an approximation thereof. If you have a Bayesian neural network or a neural network that that you sample from, then out comes a Bayesian neural network prediction.

• 39:36Speaker 04

  So again, not a posterior over the neural network parameters itself, but a posterior predictive distribution. Take all the possible neural networks out there from your Bayesian neural network and integrate over them. And if our prior is the space of structural causal models, then our posterior is, well, the integral over all possible structural causal models that could cause a data. Say, for each of them, how likely is this SEM to cause this particular data? And what would the SEM actually predict for the test points? And then you integrate over all of them. And of course, there is a huge number of them. It's probably countably infinite, but.

• 40:25Speaker 04

  We only see a limited number of them, like hundreds of millions, but our network is so small it can't possibly memorize that, so it actually learns to generalize across the space, even just from seeing a finite number.

• 40:38Tim Scarfe?

  So I suppose the interesting thing is that you can scale this up and at some scale it will asymptotically converge on what the Gaussian process would have been, and it's already reasonably close to that based on the graphs that you've shown. And this is just incredibly exciting, because I suppose what we haven't said yet is that you can think of this like the image in that moment for tabular data. So when you look on this tab arena, was it around 25, 26, there was a step change. So, you know, folks on Kaggle, they're using XGBoost and CatBoost and, you know, is it, what was the other one, LightGBM? Is that right? Yeah. And there was this absolutely massive step change. And there's a couple of things going on. So now you can just generate essentially as much training data as you want.

• 41:22Tim Scarfe?

  You can bake it into the model. And I think the important concept is that it's doing a kind of amortized inference. So whereas before, when you're doing, you know, training and inference, you had to do a lot of work, now you have a single forward pass, which means essentially you have a fixed amount of computation.

• 41:38Speaker 04

  Yeah, exactly. So, I mean, we basically learn this network that's trained in order to do as well as possible over the types of data sets that we feed in as input. And what we've done historically is to, you know, generate relatively simple data sets. Was TabPfN1 like super simple, TabPfN2 a bit more complex with outliers, et cetera. But even with TabPfN3, they're still relatively simple IID data sets. And there is, you know, relatively low number of data points compared to, you know, the billions of data points that are out in some data sets.

• 42:22Speaker 04

  So with TabPfN1, like 10, what, 1,000 data points, TabPfN2, 10,000 data points, TabPfN2.5, 100,000 data points, TabPfN3, a million data points. So we're getting there. We're getting there like two orders of magnitude a year. But still, we're not at a billion. And a lot of the data sets on Kaggle, for example, are so large that, you know, the tabular foundation models haven't caught up there when you use them in a forward pass. So you can have a harness around it where we have the scaling mode and thinking mode, et cetera, where you can actually basically have wrappers around this and use test camp compute in order to scale up. And then it could also work for the data complexities of Kaggle.

• 43:07Speaker 04

  But for now, it hasn't been used for that many Kaggle competitions, because there aren't a whole lot of Kaggle competitions that are in this really small data regime of like 10,000 data points that we could deal with TabPfN2, or 100,000, we can tackle since last November, a million we can tackle since like two months ago. There just haven't been a whole lot of Kaggle competitions of that size. But in general, just actually not that many tabular competitions anyways, because, you know, language and vision, et cetera, has gotten a lot more attention. But tabular data is so important in the world that I will, I definitely see a lot

• 43:50Speaker 04

  more, yeah, focus on these types of modality now again. And that will also lead to more Kaggle competitions in the future. And we are focusing on ever more complex data sets in our prior generations. So for example, then non-IID, group data, temporal data, text in our tables, et cetera. All of these are data complexities that you see everywhere on Kaggle and that were just nowhere in our benchmarks, that were nowhere in our data generation processes. And we're changing that. It's in our benchmarks now. So we have this beyond arena that has all these data complexities in there.

• 44:33Speaker 04

  And then that shows that, you know, TabPfN2.6 was not particularly good for some data complexities. TabPfN3 helped on some of them, but it's still not quite there. And then, so that's a, that's a signal we're optimizing for right now.

• 44:47Tim Scarfe?

  Yes, I think let's come back to the multimodality stuff later, because that's really, really interesting. But I suppose a good mental frame for this is, you know, like when you use a normal language model, there's a context. And you have this quadratic complexity and, you know, things just slow down. If you're using Claude, for example, when you, when you go up to a million, it's really struggling and it's slowing down a little bit. And I think it's a similar thing here, which is that the context now is essentially the table that you put in. And so is it around 100,000 rows now?

• 45:16Speaker 04

  Yeah, so it's up to a million rows and up to like, a million rows times 1,000 columns. So if you were to throw this into an LLM, you would have a billion elements and you would need to tokenize each of these numbers. So that would maybe be three, four tokens each. So you have three, four billion tokens in your context and LLMs wouldn't be very happy there. So they just don't work out of the box for this type of data. They're not made for tabular data. They would just need to read the table in some sort of sequence, because they're sequence models. So if they read like one row at a time, then yeah, they, they don't understand that if you switch two rows or you switch two columns, it's the same data set.

• 46:01Speaker 04

  And so that, that's why, yeah, they don't exploit these invariances. And they have a whole lot of parameters that are super important in order to capture the semantics of the world and order to capture world knowledge, but that you actually don't need in order to capture sort of the statistical reasoning that you need in order to serve this, this standard interface of XGBoost, et cetera, where you just have the X train, Y train, X test and want to predict Y test. There you don't even see column names. So you really don't need to know anything about semantics for this. If you do know semantics, then you can do more.

• 46:44Speaker 04

  So if you have like five rows or something like that, and you know, I don't know, this is a turn data set and you know exactly what the column names are, et cetera, then in comes the world knowledge and their LLMs would be great. But if you have a million rows, then you actually want to learn about the statistics of these numbers. And that's where the LLMs are just really fall flat on their nose.

• 47:08Tim Scarfe?

  Yes, I suppose we should contrast what the trade is here. So, you know, I could train an XGBoost model and it would be very expensive. And but I suppose the good thing is I can then do streaming, parallelizable, row-wise prediction, and I can go through the data set. Whereas the good thing about your system is that it's actually taking the entire table, the entire data set as an input, which means it's learning potentially global relationships, first and second order relationships, you know, table-wise invariances. So it's modeling at a level of sophistication which is far away from XGBoost. And the beauty of it is, because it's a foundation model, it just does it automatically.

• 47:48Speaker 04

  Yeah, it has learned to do this in a forward pass. Since you mentioned that XGBoost, you can stream a lot of predictions in parallel. So historically, TabPfN was very slow at inference time, because, yeah, you always had this entire data set that you pass in into the forward pass. And so there wasn't really a difference between the training stage and the prediction stage. And now, just like in LLMs, you actually, we have a KB cache. And so there is a difference between training and test. And at test time, all you need to attend to is to this KB cache and you don't need to attend to each of the training data points. You also don't need to keep your data around, because, you know, also for privacy reasons,

• 48:31Speaker 04

  et cetera, you don't want to feed all your data when you make any prediction. But you just need to keep the weights that are stored in your KB cache. So it's much faster now and on GPU, it's getting close to XGBoost prediction times as well.

• 48:48Tim Scarfe?

  We were contrasting this to something like AutoML before. But you had quite a good slide, actually, I was reading your deck before I came here. And you made the observation correctly, that if you fed, let's say, 1,000 rows into a language model, just a bare language model, it's not going to do anything interesting, is it? Because it doesn't understand the structure in the data, it's not modeling those invariances. But I'm just not sure that's an entirely fair comparison, because now we live in the world of Claude code and codex. And I could say to the agent, you know, here's a load of data and it would be generating Python code, it would load it into a, you know, a data table. And it would understand a lot of the semantics, it would understand what the meaning of this was, or that, you know, this is, this is a risk thing, an insurance thing, an accountancy thing.

• 49:34Tim Scarfe?

  And it would be building all of these structured models. And I could tell it to crystallize a machine learning model. So I could say, okay, well, there's some text data here, then let's use the mini LM encoder and let's have like a XGBoost, you know, head on it or whatever. And the beauty of it is, is that it's adaptive. So next week, when there's a prediction error, Claude will understand why the error happened and it will generate some synthetic data and it might put some symbolic rules in and it might say, well, if the recall is less than this, then I'm going to, I'm going to escalate it to a Haiku model or something like that. So you see now we're in this, it's almost like the dream, you know, like a few years ago, wouldn't it be amazing if we could have this? So how does it compare to some kind of adaptive architecture like that?

• 50:14Speaker 04

  Yeah, so I'm absolutely in agreement that LLMs are amazing for coding. Agents are great for feature engineering, they're great for exploratory data analysis, they're great as a user interface, actually let you double check that all of the data is entered correctly, like chat with you where the data actually comes from. And so for this feature engineering, they're amazing.

• 50:41Speaker 04

  I mentioned that TabPfN would be published sort of a week before ChatGPT. And then when ChatGPT came out, we were like so excited. And the first thing we did is like drop TabPfN and actually do a Gentic Data Science with ChatGPT to actually do feature engineering on top of TabPfN. So you actually do the feature engineering that where you do need the world knowledge. I don't know, this, this, the super simple example there is sort of if you know the height of a patient and the weight of a patient and you want to predict something where it would be helpful to know if they are obese or not. Then of course you can compute body mass index and it's just, you know, a simple computation.

• 51:22Speaker 04

  And you could do this computation or you could have the network learn to do these types of computations early on in the first layer and so on. But if you don't need to, if you can just do this outside with a coding agent, that is so much easier. And so that's what the first thing we did is to write this café paper that did automated data science with LLMs. And so this is the type of relationship I absolutely see is for the data preparation, cleaning, et cetera. LLMs are fantastic and we're using them left and right. But in the end, they need to call a model and yeah, they can call XGBoost,

• 52:02Speaker 04

  but it's just not going to be as good as the tabular foundation models that are getting better by the day.

• 52:08Tim Scarfe?

  I suppose we could have our cake and eat it because what folks can do now at home is they can tell Claude or Codex to go and grab TabPfN V3 because for non-commercial use it's free. And they could actually get the agent to use TabPfN. And here it gets a little bit interesting, right? Because I think one of the great things about TabPfN is that our data sets and tables, they're legible. But what if Claude did weird stuff? Because you know, sometimes to get better representation or friction, you do weird, inscrutable feature engineering and whatnot. Maybe Claude does that. Maybe Claude takes two of our features and it turns it into a hundred features and we end up with a spaghetti mess. Would that be a problem?

• 52:47Speaker 04

  I mean, I don't think it would be too much of a problem because you would prompt Claude to actually tell you exactly what it's doing and to give you the code and give you the features, et cetera. And that is actually quite similar to what a data scientist would do, right? They write some sort of feature engineering pipeline and that might get a little messy over time.

• 53:10Speaker 04

  But you know, it's simple code. In the end of the day, it's simple operations that you apply to your different features.

• 53:18Speaker 04

  What Claude would also do if it's doing a good job is to go access all kinds of feature stores online, et cetera. If you, for example, have a data set and I don't know, one, one of the columns tells you about the country that the data point is from, and maybe it's a shopping data set. And then you can say, well, in this country, actually this particular day is a holiday. So there is going to be more demand because people have more time to actually go shopping online, et cetera. And so that is going to be a super important feature that you can then generate is like, is this a holiday in this country or not? And that, you know, requires external information.

• 53:59Speaker 04

  You can't just see this from the context and probably Claude doesn't really want to, yeah, just remember this, but actually look this up in some knowledge base. And of course it can do that. And you can now use Claude with TabPfN as we have an MCP server that's integrated with Claude, integrated with Gemini, also with all kinds of other, yeah, other frameworks. And it is really easy to get going. And then basically you can just say, hey Claude, build me a data set for this problem and then use TabPfN in order to predict. And actually, yeah, a friend of mine used this for the World Cup.

• 54:42Speaker 04

  So we, we had a little bit of fun and made predictions for every game. But that friend actually used Claude Methos in order to just really prompted, hey Claude, please build me a data set from historical games, et cetera, like any way you want. Tabular data set and then use TabPfN in order to predict every game. And with that, they actually won their kick dip in their company. And so it, you cannot have this cake you need it, absolutely. And this is to me really sort of the convergence point of AutoML because we can give state of the art machine learning to novices who really, they don't even know, need to know how to code.

• 55:26Speaker 04

  They don't need to know really what a tabular prediction problem is, et cetera. And the, yeah, the friction is just so low, but still at the same time you get state of the art performance. And yeah, we actually have a variety of different demos. We have one demo where it's integrated in data bricks. And yeah, there you actually can look through your dashboards, et cetera, and ask the agent questions. And for some of the questions, you actually want to do a tabular prediction and it will just do that. It won't even tell the user that it's using TabPfN, why should it, right? It's making better predictions. And, and so that is definitely the world that, that we will be in, in a couple of years.

• 56:09Speaker 04

  Where you'll see tabular foundation models used left and right, often without you knowing it.

• 56:15Tim Scarfe?

  Yeah, no, I'm actually genuinely excited to try it myself because it's one of those things that just has utility almost everywhere, especially if you can get a coding agent to help you do it. So that's amazing. There are some other advantages that we haven't spoken about. So for example, you can use them to generate data or do things like density estimation or even embeddings. So I guess we want to just get away from it. It's not just something which is only used for prediction. You can actually use it for explainability or as part of building some other predictive architecture.

• 56:45Speaker 04

  Yeah, and I think that's super important, right? To move beyond just classification and regression. It is a foundation model, it can do all kinds of things. We want to push towards relational data. A whole lot of data that is tabular actually comes from merging two different tables in a relational database. And working directly on the relational data would be so much more powerful. There's time series, which is actually very related there. It's causality. So that is something we haven't talked about much, but this is, this is really core because you do want to make distinctions between, yeah, just causation and correlation.

• 57:24Speaker 04

  There, it's so helpful to be able to understand the world in a causal manner. Then you understand it just much more deeply. And this is a big challenge and you know, causal ML is a big field. And I think we can make a very solid dent in there with tabular foundation models.

• 57:44Tim Scarfe?

  Yes, and I think we should come back to that because that's very, very important. But so yeah, right now folks can just put, just in Python, import this library. And it's very similar to scikit-learn or something like that. So you know, you can just say, I've got some data here. I've got my, you know, my signals and my labels fit. And out of the box, it does regression and classification. Now, if I understand correctly, you can't combine classification and regression because you've decided architecturally to specialize in those two models. So if you want to combine those modalities, I think you need to create two models, is that correct?

• 58:17Speaker 04

  It helps to, to have separate models for classification and regression sometimes because yeah, the gradients propagate a bit differently and it's, it's a bit tricky to train both at the same time. But if you can train both at the same time, that's of course so much nicer. And yeah, of course we're also working on that. And we also want to have a causal head and we want to have a time series head, et cetera. And just train all of that jointly, that that would be so much nicer.

• 58:48Tim Scarfe?

  Yeah, can you, can you explain that? So when, when doing regression, you're using an MLP head. And when you're doing classification, you've got this very interesting, I guess it resembles an attention head. So you've got a special kind of decoder head for that. Tell me about the rationale there.

• 59:03Speaker 04

  It's not a problem to have different types of heads for different tasks, right? We like, for example, for causality, we would need a different type of head than for the other methods. And the way we used to do actually regression is a regression as classification, where you just have this binning distribution, where you bin the space into just like, say, 10,000 classes. And where a lot of the data falls, you have very small bins and where not so much of the data falls, you have large bins. So that there's the same number of samples in each of the bins. And then you just train a standard classification model. The rationale for that was, well, actually back then, my PhD student Sam Miller did these experiments.

• 59:49Speaker 04

  And he did try also, yeah, a regression hat that actually just predicted the mean and the variance. And, and it was just better to predict in this classification manner.

• 1:00:02Speaker 04

  One of the reasons for this might have been that actually, well, transformers were just like all the hyperparameters, et cetera. Everything is just made in order to do classification really well. And having 10,000 classes was not an issue at all. And, and so we actually just stuck with that. One of the really nice side effects of that was that we could do these multi-modal predictions. We could say, well, I don't know if the bird flies against like a pole. Like they're going to turn left or they're going to turn right. They're not going to hit against the pole. So you're going to have a distribution here and a distribution here and pretty much zero probability mass in between. And you can do these predictions again in a forward pass.

• 1:00:45Speaker 04

  And so you can have like really well calibrated outputs with this type of distribution.

• 1:00:50Tim Scarfe?

  And can you tell me about how you do the representation? So I mean, there's a bit of a lineage, right? So the version one kind of was, it more resembled a normal transformer. So you're just, just kind of tokenizing things. And if I understand correctly now, in version two and three, it's far more structured. And you have a specific way of kind of mapping the attention because the transformer is permutation equivariant, maybe we should start with that.

• 1:01:16Speaker 04

  Um, yeah, so the, the architecture for TabPfN V1 was very much like a transformer.

• 1:01:23Speaker 04

  Just that you drop the positional embedding because well, attention is already invariant to the order. And the only thing that makes a transformer actually, yeah, pay attention to where the words are in the sequence is this positional embedding. And we didn't want that, we wanted to be invariant, so we just dropped the positional embedding. It was the simplest thing we could do.

• 1:01:45Speaker 04

  Then, but what that required is to actually take a row and encode that into an embedding. And we use a super simple method for that. In TabPfN one, we actually had only continuous features and we just used a linear layer.

• 1:02:02Speaker 04

  And you know, that that is maybe not the best because then we if we applied it for categorical values, we would just encode the categorical categorical values as one, two, three, four, and then put a linear layer on there. And of course, that that is not the best. So in TAPIFN two, what we did is that is actually really have an architecture that knows about rows and columns. And there we did an attention over like, so we had an embedding for each individual element of the matrix. So for each value, then you would have an attention over the rows and an attention over the columns. And you would alternate these. And

• 1:02:44Speaker 04

  then you could, of course, yeah, really understand, aha, in this column, the the numbers are one, two, three, four. Probably this is a categorical, you can treat this differently than than the numerical one, two, three, four. And yeah, we would also in TAPIFN two have much more categorical information in the prior already. So there's just better and better treatment of these categorical parameters. And yeah, also complexity wise, I should talk about that because complexity wise, TAPIFN two was actually much worse than TAPIFN one, because TAPIFN one was only quadratic in the number of rows. But it's

• 1:03:29Speaker 04

  sort of, yeah, bad because of the encoding of of the row into a single embedding. And then that that is really inherently limiting. Whereas at TAPIFN two, you're actually, since you had so many embeddings, for each of the embedding, you would actually have this attention over all the rows. And you have that times all the columns, because there there's that many elements of of this row. So you have N numbers of rows and M columns, and then you have N squared times M plus N times M squared. That is the complexity of of TAPIFN

• 1:04:08Speaker 04

  two. And and that was okay for up to 10,000 data points. But if we wanted to scale higher for 100,000, a million, etc., we needed to go beyond that. Well, actually, for 100,000, TAPIFN 2.5, we still did that. We we got away with just going to beefier GPUs. But then, at the same time, yeah, the Gail Wolker Cross Group actually developed Tabical, which was really nice. There was an architecture that basically, in the end, used the TAPIFN one architecture, only only quadratic over the number of rows. But it had a much more complex mechanism to actually do this embedding through a different column and a row transformer.

• 1:04:51Speaker 04

  And since that architecture was actually sort of good enough for the for the the size range, we wanted to do for TAPIFN three. And also in terms of inference speed, we actually just basically adapted that for TAPIFN three and all innovated in many different ways, for example, for many classes and for the output head, etc. But the base architecture, we didn't actually have to change from that. We have a lot of things cooking, but we didn't actually need them for our desert orator for TAPIFN three. So they're going to be in future versions.

• 1:05:24Tim Scarfe?

  Of course, but just to be clear, what is the complexity of V three?

• 1:05:26Speaker 04

  Of V three, it's actually the same as Tabical. So it's basically, it's N squared for the number of rows. But then it's N times M squared for the preprocessing, for for the embedding that, yeah, but so it is already faster, but it's still quadratic. And there is a gazillion methods to make it quadratic. And of course, we're looking at those. And yeah, so next target is 10 million. And we're pretty confident.

• 1:05:53Tim Scarfe?

  That's super interesting. I mean, even the rows and the columns, that is a wonderful example of the kind of inductive prior that we put into these models, right? Because we we have certain assumptions about the data. And it seems reasonable that there are kind of relationships, you know, row wise and column wise. But what would a fast attention look like? So for example, if you were going to cut down that that span of attention, would it be some kind of locality prior?

• 1:06:17Speaker 04

  Yeah, I mean, so in in language, you have this locality prior. In tabular data, you don't, right? Because you want to have this invariance over features. But you could have some sort of locality in the embedding space.

• 1:06:29Tim Scarfe?

  Very cool. Now, I am very excited about kind of test time adaptation. So, you know, O one came out, and that was remarkable, because the models became intelligent, because they were kind of through this, you know, RL training, they were learning to prompt themselves, they were actually deliberating thinking. And what a lot of folks have been doing with with these language models is is test time adaptation, where they actually deliberately kind of, you know, do some kind of structured inference at test time. And it just feels like this is the perfect opportunity to do that. So at the moment, you've made a deliberate decision to have an inductive model. And that has some interesting properties, certainly in terms of like, you know, caching, calibration, performance, that kind of thing. I mean, you know, if it were transductive, what that would mean is that the the test

• 1:07:13Tim Scarfe?

  samples could actually bind to to the training samples. And that could be bad, because let's say, I'm doing a hospital prediction thing, and I might have some weird patient who's an outlier. And now this other patient is affecting my prediction of the first patient. So it's not necessarily a good thing, but it can be an incredibly powerful thing. Is that interesting to you?

• 1:07:35Speaker 04

  So I don't think I would go too much into the transductive direction. Can go into the semi supervised version, where you have some low numbers of, yeah, labeled data, and then you have a whole lot of unlabeled data. And you know, this is a type of data you actually want to work well for. But in the end, I I think I would stick with an interface where what I'm making predictions for doesn't affect the predictions for the other data points, because data scientists just wouldn't have that. I think that would be so weird if you predict the same data point in a different batch, you get different

• 1:08:19Speaker 04

  results, it would be super strange.

• 1:08:22Tim Scarfe?

  Yeah, it's interesting. I mean, in the the ARK prize, for example, the reason it works so well is you only have a few examples of something. And those other two examples, they they give you so much information, and we're desperately trying to adapt the the model. But I suppose another thing is just some kind of adaptive thinking in general. So I could imagine, and I think you've got something like this, but I don't think it's publicly known what you do. But, you know, just to think out loud here, the way I might approach something like that is you might have some kind of a latent variable, and you can actually have some latent which could evolve with more computation. So I do more forward pass passes, and I'm actually evolving some kind of representation. Is that something you you think about that right now, we have a fixed amount of computation, but there

• 1:09:05Tim Scarfe?

  are many tasks that we're uncertain about that we might need to think, you know, search around a little bit to get a better prediction?

• 1:09:11Speaker 04

  Yeah, there's a whole lot of analogies here. So for example, I mean, you could have, for some cases, you might want to make a prediction really quickly, because it's a really simple data point. And so you could have early exit of a big network. For harder data points, you want to go all the way through. For even harder data points, you might want to think a lot harder. And there is a whole lot of ways of thinking harder. Like, you could go and do fine tuning on, like, you could look at, hey, this is my data set. Let's generate other data sets like this and do fine tuning on that. And actually, one of my master's students had a paper about that. And that for very small data sets actually

• 1:09:55Speaker 04

  can help improve. Then what you can do is you can do prompt tuning. So if you have, for example, a very large data set, then you could say, well, what of this large data set should I actually feed into my context? You could select, say, you have like a billion data points, you can select 100,000 data points to put into your context. And you could do that in a gradient based manner. But you could also actually hallucinate 100,000 data points that that approximate these billion data points better. We also had a paper at New York about that. One one fun fact there is you can actually also use this for interpretability.

• 1:10:37Speaker 04

  So one of my previous PhD students, Robin Schirmeister, actually took a medical data set of 1,000 data points, and condensed that into two data points that you would feed as a prompt. And that was actually the same predictive performance as feeding 1,000 data points. So that sort of gave you the prototypes of the data. And there's a whole lot of things like that you can do. You could also have different networks that were trained with different priors and, yeah, do standard cross validation on them. There is work that shows that transformers can actually do model selection, so in a forward pass. And so you

• 1:11:17Speaker 04

  could have basically train a transformer with like 10 different priors, and have it choose the right one at test time in a forward pass. There are so many different things you can do. And I think the sky's the limit.

• 1:11:30Tim Scarfe?

  Yeah, exactly. I mean, yeah, I mean, there's also like sampling and ensembles, and you could have some kind of a scratch pad. I'm just really excited about this in general, though, because I wonder whether there's a tension there, because we're kind of couching this as a foundation model, you know, without adaptation. But it feels that adaptation is always strictly better. I mean, this is why I'm quite excited about using the agents, because, you know, new information comes in, and we crystallize and optimize and optimize, and we end up getting something better. And it feels like this could become the base of an architecture to do that. The data set distillation, that's really interesting. I think I interviewed a guy called Andrew Ilyas from MIT. He had a great paper on on data set distillation, and this machine teaching and so on. And, you know, the basic idea is that if you just kind of prune and distill the data set, you can get dramatically better

• 1:12:15Tim Scarfe?

  performance. And this can be an online adaptive process, because when folks are implementing this technology, they they they live in where they live, you know, it's that they have very specialized data, things are changing, and this adaptation process could be part of the predictive architecture.

• 1:12:30Speaker 04

  Yeah, absolutely. And there's so many cool things, as I said, that you can do that you couldn't do with with methods like XGBoost. For example, you could like do feature importance in a forward pass, or you can do a forward backward pass and say, well, if I change this feature, how much would my output change? Like, you get a gradient signal for that. And so forward backward trivial. But you could also do that, for example, for a data point, like, how important is this data point here for training my model? Was that you can actually detect adversarial attacks, etc. And you could say, ah, in this is really strange. This is one data point that completely dominates my model. How about

• 1:13:11Speaker 04

  I train again without that data point? And then is that going to be much better? And if so, then let's iterate this process, etc. Let's figure out what are the important features, train with only the important features, and then drop the other ones. So many different things you can do there.

• 1:13:26Tim Scarfe?

  Let's talk about interpolation versus extrapolation. Now, my good friend Randall Bellistriero, he had this paper on the spline theory of neural networks. And he was actually talking a lot at the time about how, you know, MLPs are like decision trees, but they extrapolate. And so this is genuinely a new phenomenon. And you've studied this because I've seen some of the graphs. But these models actually extrapolate in ways that normal models don't. What's the behaviour there?

• 1:13:51Speaker 04

  Extrapolation is super important also for me coming from AutoML in Bayesian optimization. If you want to, if you only see parts of the space and you want to actually extrapolate to other parts of the space, for example, scaling laws, you only see points for like small neural nets. But you actually want to predict how good your large neural network is going to do. You need to actually be able to generalize. And yeah, I mean, you can just put it in your prior that you want to generalize. Just say, well, I want to train on these data points from here and I want to be able to predict over here. If that's on your prior, that the train and test is not I A D, then you can also learn auto distribution predictions.

• 1:14:32Tim Scarfe?

  So I think about three weeks ago, was it? Google released something called Tab FM. And I took some notes about it. So the first thing is imitation is the sincerest form of flattering. So, but you know, they basically were inspired very much by the work you've done. And they scaled it up around 30 times. And they got significantly better performance on this TabArena benchmark. So what's the story with that?

• 1:14:58Speaker 04

  Yeah, I mean, it's, as you said, it's sort of flattering that they're also jumping on this. And basically, I'm taking the same architecture, the same types of priors, et cetera. And, you know, I'm solving it as Google compute to just get bigger and better models. It's not surprising that this is better. I think it's maybe a little premature to go that big because it also comes with some issues. The model is, yeah, very large and sort of quite slow as a corollary. It's sort of like 15 times slower than, like, it's just a forward pass, which is super cool.

• 1:15:39Speaker 04

  I mean, the fact that it can do this just in a forward pass, get this performance that is really nice to see. But it's a very slow forward pass. It's like 15 times slower than our forward pass. So we have this test time compute Tab FM thinking that's actually 10 times slower than our standard model. So still, our test time compute is actually faster than a forward pass of Tab FM. And yeah, and it's stronger in terms of Elo scores, et cetera. But it is a forward pass. And so it shows that if we actually do, yeah, look at the scaling loss and of course have scaling loss already. And if we do, yeah, invest that compute, that we will get better and better.

• 1:16:26Speaker 04

  They did focus quite a bit on small data. So in particular, for the smallest data sets in TabArena, it's much better than previous methods. For the next, so for tiny, for small, it's still better. For medium, it's actually sort of breaking even and medium is up to 100,000 data points. And then that was sort of our focus with the previous models. And Tab FM 3, we actually already went to other benchmarks. We went to the talent benchmark, which is up to a million data points. And to this beyond arena, and we haven't actually focused that much on TabArena, which is up to 100,000. And this is a good reminder that we should also focus a bit more on that one.

• 1:17:09Speaker 04

  And so of course, our current benchmarks also will make sure that the next model generation that that's cooking is also, yeah, again, much stronger on TabArena itself. But we don't want to overindex on it because, yeah, we do care a lot to like to scale to a million, to 10 million data points, to non I A D group data, et cetera. And Tab FM being such a big model, we actually just can't even run it on beyond arena. Like it just breaks and out of memory, et cetera, for sort of data sets that go beyond sort of roughly 100,000. But it's very cool that they're part of this now. And so very excited for the field to grow.

• 1:17:53Tim Scarfe?

  And there is the question of the moat thing. So you guys actually did the research, you understand this. So presumably you can evolve this thing in interesting directions. But one thing we haven't spoken about that is part of your moat is a lot of principled knowledge went into designing these priors, right? And we haven't really discussed how you did that yet. How did you make those priors?

• 1:18:14Speaker 04

  I mean, the priors really, prior development is a fairly iterative process. The way we started was basically, we had for Tab FM 2, we had this set of holdout data sets that we wouldn't train on, but that we would use in order to say, if we make a change on the prior, train a network, see how well does it do on these data sets, where are some failure modes? And like actually see, ah, there's a failure mode in this data set. What are some of the, yeah, structures in this data set that it might be failing because of? For example, you know, outliers or missing values or uninformed features, et cetera.

• 1:18:58Speaker 04

  And then we would also put those into our prior. And then sort of over time, basically, we just made it ever more complex to capture more and more possible ways that the data might have come about. Because, you know, if your prior is limited, for example, again, just linear lines, then the posterior is going to be limited based on linear regression. It's a perfectly fine model, but it's just very bad. And if that, if your data is actually not linear. And so we would actually have very complex priors that can capture as much of the world as possible.

• 1:19:40Tim Scarfe?

  Let's talk about causality. So this is one of the things that's spoken about a lot in this context. I mean, first of all, just can you explain what is the difference between causation and correlation? I mean, to what extent can we expect these models to be causal and what would that even mean?

• 1:19:54Speaker 04

  Yeah, let me, let me explain that with an example. So if you, if you think of a medical example where you have a disease that some patients have, and these patients become a particular type of medicine, like they got a particular type of medicine that helps them against that disease. And the stronger the disease is, the more of that medicine they get. If you have a data set where you just see how much or which medicines a patient gets, and you want to predict whether they have that disease or not, then we can basically build a perfect model that says, well, if they have, if they get this medicine, then they have this disease.

• 1:20:34Speaker 04

  And you might be tempted to say, haha, let's stop giving them that medicine and they won't have that disease anymore. But that would be foolish, right? Because the causal relationship is the other way around. Because they have the disease, they get the medicine, not the other way around. And so that's where causality and causation is really important to distinguish. When you start with these interventions. So I just said, let's stop giving them that medicine. So that means you change a variable in causal ML that says do operator. So you do this, do give the patient this medicine rather than observe the patient is taking that medicine.

• 1:21:19Speaker 04

  And those are two entirely different things. Because, yeah, if I observe the patient is getting this medicine, then I actually, then there is some relationship that the reason that they're getting it is that they have this medicine. If they have this disease. But if I just decide that they're going to get this, then I just know nothing about the disease. And so that, getting a hold of that is very tricky, but that's where causal ML comes in.

• 1:21:49Tim Scarfe?

  And this do calculus came from Pearl. He had this ladder of causation, just frame that up.

• 1:21:55Speaker 04

  Yeah, so, yeah, the do calculus does come from Pearl. And there is this, you know, in the Pearl school of causality, you know the graph. And if you know the graph, then you can compute all kinds of things. But in practice, often you just don't know the graph. And, you know, we like, if you're a company and you want to actually do pricing on some of your products and you observe hundreds of different features of your customers and so on. And you want to decide prices for some of your products. You don't know exactly which of these features cause which and so on. And you still nevertheless want to make decisions in this causal space

• 1:22:39Speaker 04

  and, you know, be aware that there might be causal relationships here. And so there's, there's two things you can do. One is, or two, two different problems to solve. One is to actually just make predictions, knowing about these causalities. And so that's basically where, that's basically where our SEMs come in, where we actually make predictions under, like sort of integrating over all the possible causal relationships. I don't know the right causal relationship, but.

• 1:23:18Speaker 04

  The data is sort of suggesting this one more, so I will wait this one more, et cetera. And then I basically integrate over the space of all the SEMs. There's also this question of, hey, actually, tell me about the causal relationships. That is much trickier. And actually, there's also a third question, which is, well, tell me about which interventions to do. And so that is, the first one was just predictions under observations. The third one is predictions of, yeah, interventional, like, or the effects of interventions. And that one, you can actually also do with these foundation models.

• 1:24:00Speaker 04

  And the nice thing is that, again, all we need to change is sort of the training process and the meta-training. So during, during meta-training, we can control, like, usually we sample a causal graph, yeah, observe the variables in there, and now we would, again, sample this causal graph and observe the variables, but then we would take the same sample and actually make an intervention in there and observe the effects of that intervention. And then we have pairs of, this is observational data and this is an interventional prediction. And we can learn from hundreds of millions of, yeah, graphs from our prior. If I see these types of observations and I make this intervention, this will be the effect.

• 1:24:43Speaker 04

  And so you can actually meta-learn to make predictions about interventional effects from purely observational data, within some limits. There is some limits of identifiability, et cetera, but you would just integrate out over all the possible worlds that are, yeah, all the possible structural causal models that are, yeah, in line with the data.

• 1:25:08Tim Scarfe?

  This is also a problem with language models, and it's related to this concept of abstraction, which I'm very interested in. So linguists, they have distilled a very parsimonious, possibly broken, kind of, let's say, generative grammar, or the minimalist program or something like that. And we have, we have distilled that down, so it's legible to us. Because if you think about it, causality could happen at multiple levels of resolution. It could be all the way down to the light cone. And what we do is, is we come up with these causal variables. And part of how we distill those causal variables is because we are actors in the world. So we can, we can actually try things, we can explore counterfactuals, and then I guess over time, we just distill it down. And we can imbue these into our AI models.

• 1:25:53Tim Scarfe?

  This is kind of what you're saying, that you can have like a load of core structured causal knowledge. And then you can do this prediction with causal trick if you can kind of latch on the causal knowledge into the ML model. But I think, does it go without saying at the moment that the ML model cannot acquire de novo causal knowledge without us giving it to it?

• 1:26:14Speaker 04

  I mean, in order to know something about the effect of interventions, the model needs to observe the effect of interventions somehow. So what we usually do is randomized control trials, A/B tests, et cetera, where we actually do interventions. And then, based on this interventional data, you can fit models. But often you just have a whole lot of observational data and you don't have the interventional data. And what this new line of work on causal foundation models actually allows you, to some degree, is to actually reason about interventions without observing them at test time.

• 1:26:54Speaker 04

  But you observe them during meta-train time, where you say, well, this is my causal model that I sampled. If in this causal model I make this intervention, then this is going to happen. And you just learn the effects of that over hundreds of millions of data sets. And then you have observed a whole lot of interventional effects and you can actually make predictions about these. Sometimes you have non-identifiability. There might be that A causes B or B causes A, and you don't quite know which one it is. But we don't need to say which one it is. We just need to solve the basic integral of, well, what would this one say? And how likely is it? And what would this one say? And how likely is it?

• 1:27:33Speaker 04

  And when you're actually not sure which one it is, and say they make completely different predictions, then in the extreme, you want to predict, I don't know, with 0.5 probability, this is right, and this is my result, and 0.5 probability, this one is right, and this is my prediction. And again, you have this multi-model distribution where you actually have zero probability that it's in the middle. It's either that or that, and I just don't know the right causal graph. But in many other cases, you can actually figure out with a higher probability it's this causal model. And, and then actually, yeah, the probability mass shifts over to the predictions that that causal model would make.

• 1:28:17Tim Scarfe?

  So if I've understood you correctly, you're saying that we can have high-fidelity models that could imagine counterfactuals. So they can kind of say, I can, I do this intervention, I could imagine this possibility and this possibility. And the model is so good that the imagined space of possibilities is quite good. So the model could then in principle select and create a causal model from that. But I guess I'm trying to understand, what is the bar for a causal model? Because even if we do RCTs in the real world, it's still somewhat statistical arbitrary. So, you know, we might establish a causal relationship, but is it, is it a binary thing? Are we in the domain where these statistical causal models will be nearly as good as having causal status, but not quite, or something like that?

• 1:28:59Speaker 04

  I mean, so you can run RCTs, and typically when you run RCTs, then you get much better predictions. But actually now with these causal foundation models, you can, in some cases, get very close to the type of performance you get if you have RCTs. So there are already papers that then reduce the number of data points you need to have in your RCT in order, and still get the same type of performance. And that, of course, you know, I think will completely revolutionize medicine and A/B tests and so on. If in the future, you know, we can get away with much shorter studies

• 1:29:39Speaker 04

  and can get our medicine to the market much earlier and still with the same confidence or even higher confidence. And so I think there's a whole lot of potential to do a lot of great things there, and I'm very excited about this. But of course, you know, we need to do this properly and theoretically grounded and so on, and this is high stakes. But yeah, the potential gains are huge, and I'm super excited about that.

• 1:30:06Tim Scarfe?

  And what would that look like? So if you hypothetically created, you know, like, your next product was tab causal, not tab, you know, but something causal, what would that look like? So would it be couched as just purely observational data? Or would there, would there be some kind of data scientist who would be structuring it in a certain way? So that the purpose of the model would be, you know, just as TapiFen does, to essentially grab that causal structure very, very efficiently? And then the benefit of this causal model is it would be interpretable, it would be reliable, explainable, that kind of thing?

• 1:30:38Speaker 04

  We already have some papers on this. So we have this DuPfN paper, together with Bernard Chilcuff, who is an advisor of ours, and sort of, yeah, one of the leaders of causal ML worldwide, most cited scientists in causal ML in the world. So he definitely knows what he's doing. And we also, there are some other teams working on this. CausalFM comes from Layer Six and the University of Toronto. We actually have the lead author of that just joined us last week as well. So we have a lot of great people on the team that work on causality. And basically, you can imagine this as you're getting some observational data

• 1:31:19Speaker 04

  and you want to predict, if I change this variable here, what will happen? So if I give this patient this medicine, what will happen? Well, yeah, what will happen to one other variable? For example, yeah, the disease status, is that going to get better or worse? And in some cases, you can actually make predictions about this. In some cases, you will always make a prediction, but the prediction might be just complete uncertainty, I just don't know. But if the data actually gives, like, basically has some traces of causality that points in some direction that, hey, it's more likely that A causes B,

• 1:32:01Speaker 04

  then we can give more weight implicitly, so that, of course, this is learned by the model. But then we can give more weight to the model where A actually causes B and then make better predictions. And the cool thing is, now we actually have a follow-up work of that as well, published at the last SEML, where domain scientists, if they know that A causes B, they can actually specify that. And then we build the structured causal model over all, sorry, the posterior of all structural causal models, where A actually causes B and not B causes A, and not where they're independent. And so that allows us to really use this domain expertise of people.

• 1:32:44Speaker 04

  And then there's also works that can probabilistically get out some causal structure. So that can say, given this data, how likely is it that A causes B? And that, that allows you to do all kinds of things, like root cause analysis, et cetera. And I think that's really exciting. Our current models do not do that yet, but that is definitely something that we are working on for the future. And we just actually started our open source research arm, where we collaborate with anyone in the community on tough problems like this. Because I think these problems are too exciting and too important for humanity

• 1:33:28Speaker 04

  that we just work on them by ourselves. And we should really work together with other, the greatest researchers in the world in order to tackle these problems. And then, of course, we'll build products around them. But it's, it's really important to solve these problems at the core. And there, we don't want to, yeah, have sort of any red line in the way, but really be able to work with anyone, open source.

• 1:33:53Tim Scarfe?

  Yeah, that's really exciting. And I mean, just final question on that bit. What is the potential for causal structure learning? So at the moment, we decide what the variables are and we give them names and they're legible, they're parsimonious and whatnot. But I can imagine a future where an AI would be able to come up with something which was very high resolution, but much better. So I guess, is there a trade between kind of the fidelity of the causal structure versus its intelligibility and explainability to us?

• 1:34:21Speaker 04

  What LLMs will come up with is, of course, you know, the world knowledge and causal relationships in the world. I do not think that they will come up with proper causal relationships between, yeah, like, features that...

• 1:34:42Speaker 04

  We see from the data, if you have like tens of thousands, hundreds of thousands, millions of data points, again, it's in the numbers and the statistics and that's what the L L M's are not strong at. And I don't see them getting stronger at that anytime soon.

• 1:34:59Tim Scarfe?

  Well, I mean, just on that, that's a tangential question, which is that L L M's are incredible. You know, you saw, you did you read that the Erdos unit, unit distance conjecture? But it was 125 pages of chain of thought. What they don't seem to do, and the reason why we are part of the process is, is we can kind of look at the spaghetti and we could say, ah, that's basically this, you know, and we can, we can abstract it and we're, we're doing more with less, we're compressing. Why, why don't the L L M's do that?

• 1:35:30Speaker 04

  So honestly, I think that is something that they will also learn to do. Because, you know, L L M's do compress their knowledge. If you, you know, write a billion tokens, you can't keep that in your memory, you need to compress it somehow, and they will. And they will learn to do this better and better. And I don't know, form lemmas and whatnot that they, they will then remember. But that is still sort of in the space of, yeah, in the space of language and not in the space of the data and the numbers. And I think

• 1:36:08Speaker 04

  the challenge will be to bring the two together and really, yeah, use the world knowledge, use the amazing chain of thought capabilities, et cetera, but also, yeah, bring in the causal information that we can see from the data.

• 1:36:22Tim Scarfe?

  So we promised we'd come back to multimodality later. So some of the things that you're looking at now is text data, relational data, graph data, that, that kind of thing. How, how does that play out in the Tapia F N?

• 1:36:37Speaker 04

  Yeah, so relational data right now, that's an interesting one. There, we haven't actually worked on this really much at all. But we got this for free by the open source community, who just actually developed a, an embedding of a relational database. So queries in a relational database where you want to predict some entities in one table, and you want to actually take as a context all the connected tables that, you know, where you have other tables that tell you something about this particular entity. And so you can actually have an embedding of taking that relational database query and

• 1:37:22Speaker 04

  putting that into a single table. And then they actually just ran Tapia F N. In this case, I think they ran Tapia F N two or 2.5. And we took that method and all did put a little bit of on, on top and ran Tapia F N three with that. And what we did is actually built better benchmarks and so on to, to make sure that we're optimizing the right objective. And yeah, Tapia F N three out of the box was actually just, yeah, the best foundation model for these relational databases. We evaluated this on the original rail bench from Kumo. It was better than Kumo R F M, which is their, their own relational foundation

• 1:38:04Speaker 04

  model. That also has an individual Tabular foundation model at the end. And yeah, then we also worked a bit more on making a better version of rail bench that actually we're releasing next week. I think when this interview airs, it's going to be three weeks ago.

• 1:38:25Tim Scarfe?

  Amazing. I mean, one interesting thing is, we were talking earlier about, there's a bunch of priors in, in the model, right? So for example, it, it might have a locality prior, it's got, you know, columns and rows and whatnot. What would it mean to put in other types of geometrical priors? So for example, could you in principle just put a relational prior in there? And it would just understand, you know, like, so you could actually put in multiple data tables and it would just understand the relationships between them. And it could somehow implicitly do joins and all of that kind of stuff, or maybe graph data or tree data. Is that possible in principle?

• 1:39:01Speaker 04

  Yes. So basically, this is clearly the name of the game of building priors. You just look at the types of data you want to be good at, put that in your prior, generate your data, and then the model is going to be good at that. Of course, you need to solve some architectural challenges. If you have different relational tables, how exactly do you put this into your network? How do you make sure that this is efficient and so on? But yeah, the sky's the limit there. I think this is totally going to happen.

• 1:39:32Tim Scarfe?

  And how do you see this playing out in enterprise? Because I think there is a bit of an issue in enterprise that there are security boundaries, sometimes tabular data and operational data in general is the hardest to get. So, I mean, speaking personally, I can't wait to go into QuickBooks and, you know, download all of my data and stick it into Tapia F N. But in, in the enterprise, how do you see this playing out?

• 1:39:52Speaker 04

  Yeah, tabular data is the most common modality in the enterprise. And nevertheless, there, there aren't, historically, there aren't these tabular foundation models out there, but, but people use traditional models like XGBoost, et cetera, and now are being exposed to coding agents and L L M's, et cetera. Many newbies sort of try the L L M's on the tabular data and see them fall flat on their nose, because they haven't, haven't seen this type of data, because this data hasn't been on the internet, so it hasn't been trained to work well on this data. But then, yeah, there, there's this big opportunity

• 1:40:32Speaker 04

  to actually, yeah, that, that is where tabular foundation models are required. And the more the tabular foundation models will actually learn about the data in the org, the better they will get. And then so you can also take a tabular foundation model and our models are open weight, and you can actually fine tune it then on the data of the org, and it will get much better performance for this particular data, because then, then you can actually, yeah, just extract the, the type of intricate patterns in that type of data and work much better for that.

• 1:41:14Tim Scarfe?

  And how do you think an organization should design structures around this? So for example, there might be some kind of data platform. Do you think that individual kind of folks in the organization should be building out models like this? Or should there be some kind of a platform team? How, you know, how would you do the data governance, et cetera? Because if you think about it, there's, there's a wealth of tabular data in the organization. Some of it is in Office 3, 365, some of it is just from operational data, you know, might be a retail system over here, financial system over here. So essentially, what you want to do is kind of make this data available, but there also needs to be controls. How, how would you do that?

• 1:41:52Speaker 04

  It's going to look pretty similar to L L M's, because, yeah, you, you don't want to make your data available to everyone. You also don't want individuals training your own L L M's. That doesn't make any sense. Rather, you want sort of one part of the org really understanding these models, knowing how to fine tune them, knowing where, where do they still fall short, where, where should I, you know, bring in some other data, where, where do I need to check whether maybe XGBoost is still better, if you have a billion data points or whatnot. And so, so I see this as a pretty centralized and, yeah, services

• 1:42:32Speaker 04

  that are being offered. And of course, as prilabs, we want to support in that. But yeah, the need to work with the orgs of how, how to actually play this out.

• 1:42:46Tim Scarfe?

  And what is your kind of revenue model? Because as I understand at the moment, you know, you can just use your Claude agent, it can use Tapia F N, you can just do a bunch of stuff. But what, what are you building as, as a kind of commercial offering?

• 1:43:00Speaker 04

  Yeah. So you can use Tapia F N in many different ways. You can use it from the open source with a non-commercial license. You can actually try it out, test it. It's entirely free to test it in a non-production setting. If you actually then want to use it in production and make money with it, then, well, you need to pay a license fee. So that's one business model is this license fee. And then we have all kinds of things around it. We have this thinking mode, scaling mode, working better for large data. We also have models that work particularly well for very small data. We, of course, also have some services to get people off the ground, to get them better performance with tabular

• 1:43:45Speaker 04

  foundation models. We have fine tuning as a service. So all kinds of layers on top, basically, that you can stitch together. But ultimately, you can think of it as a platform that just makes a data scientist dramatically more efficient.

• 1:44:02Tim Scarfe?

  Okay. And then I'm just sort of like imagining a couple of years ahead. So you might have a forward engineering team. You might eventually have models that are too big for customers to serve themselves. So presumably, you, you might offer some kind of a serving stack for that. The fine tuning thing is interesting. We didn't talk about that. But do I understand that you can fine tune the models, but the customers can't yet fine tune their own models? Or?

• 1:44:23Speaker 04

  The models are not too large for the customers to serve yet, but we might be able to serve them faster. We might have different tricks. So we do already offer an API where, yeah, we, we just control the, yeah, the, the infrastructure and we can make sure that, you know, we can batch different, different predictions. We can make sure that the GPU is keep busy. And if a client was to reserve a GPU in order to do all the predictions, that GPU might idle all the time, and it might actually be much more cost efficient for us to do the serving than for the clients to do the serving. So this, this API is also something that, that is part of the commercialization. And then we,

• 1:45:07Speaker 04

  we have private V P Cs where, you know, for data that cannot leave the org, that, but, but if you already do have a cloud partner, then it can run there. So for example, Azure and A W S, you can already use it there. And then of course, in S I P and the Gen A I hub, we have Tapia F N. And yeah, so there is, there's a variety of different ways to commercialize. Fine tuning, we're particularly good at fine tune. There's also some fine tuning wrappers. People can fine tune the models. But again, if they fine tune the models to the data in their org, then

• 1:45:50Speaker 04

  Yeah, there's already the license for the model that already that also applies when when they find you in the model. So the fine tuning is we can do it or the customer can do it. Both would work.

• 1:46:03Tim Scarfe?

  What would happen if OpenAI or Anthropic just embedded it in their harness? Would you would you just charge them straight away? Or would it be based on how it was used by the customer?

• 1:46:14Speaker 04

  Yeah, so that's going to be interesting to see how the token economy will will work out. I think the tabular prediction tokens will be much cheaper than, you know, the LLM trying to do it themselves. You can have an agentic agent that goes and writes some code and gets something that's maybe close to that, but that takes a lot of time and tokens. And rather, you could actually just call it tabular foundation model. And it should do it faster and better and for less tokens, or for maybe an amount of tokens that costs you less. And so in this agentic economy of the future, that's what we're developing

• 1:46:58Speaker 04

  the methods for. They should be just better than any any other method. And, you know, LLMs call calculators because they're better at doing maths. And just like that, they should call tabular foundation models because they're better and cheaper at doing this than they can do that themselves.

• 1:47:16Tim Scarfe?

  I mean, I'm just imagining that if someone from Anthropic was listening to this, their first thought would be, I'm going to update the system prompt and just to use tab PFN if anyone's using tabular data. Or, you know, most normal people at home, if they listen to this, they're going to be thinking, I'm going to update my Claude global skill to use tab PFN anytime I'm using with tabular data.

• 1:47:35Speaker 04

  Absolutely do that. We have skills files and everything.

• 1:47:38Tim Scarfe?

  Yeah, I mean, it just seems like it seems like a no brainer. Yeah, the other thing is, so you're launching a research arm. Tell us about that. What are you guys doing?

• 1:47:46Speaker 04

  Yeah, so we're launching an open research arm. We're already doing a lot of research in Prilabs, and we want to do a lot more fundamental research also. Because, well, we just did get acquired by SAP, and we have a lot of, yeah, funds available to do a lot of cool stuff. The leading motive for the acquisition has been DeepMind. And so we want to, yeah, be the best place in the world to work. We want to publish up a storm. We want to be the place where everybody goes in order to do the greatest work, and good things will happen from there. And in the open research arm, it's going to be fully

• 1:48:30Speaker 04

  open research. We can collaborate with anyone in the world, with universities, with the Ellis Institute, with, you know, other partners, like with, I also still have a university group at the University of Freiburg. There, we already collaborate with Layer Six, for example, in Toronto, University of Toronto, with Singapore, with the Ellis Institute in tubing, et cetera, et cetera. And all of those should just be possible super freely and easily. And so basically, we're going to, yeah, take what we do at the university and scale that up, and also do that in Prilabs. And it's going to be, of course, super energetic

• 1:49:12Speaker 04

  for people if they want to have impact and then also see some of their work being moved into a product. And we also want to tackle moonshots. So like, really take high risk, high gain problems that if we solve them, could really move the needle for humanity. If you're listening and you have data for that, if you could do tabular predictions better for this data, maybe super complex data, and you could, I don't know, with this, then help cure cancer, or help the global food production, or help any type of problems that are good for humanity, please get in touch. This is definitely something we want to actually really focus on and do some cool stuff.

• 1:49:56Tim Scarfe?

  Amazing. So yeah, to be clear, you are hiring. So researchers, interns. I'm excited about this because we hear too much about America and China dominating the AI space. We're here in the south of Germany, in Freiburg, and doing frontier AI research. There are loads of talented folks in Europe. So give Frank. And do they give you an email, Frank?

• 1:50:19Speaker 04

  I just, please go to our website and apply. We have over 11,000 applications and have hired 45 of them. But if you're excited to work in that type of team, where we had the luxury of a whole lot of people already applying, and us being able to pick the world's best folks in the space, if you're excited to work in this, yes, by all means, please do apply, priorlabs.ai/careers. We hire for pretty much every role you can think of.

• 1:50:55Speaker 04

  In particular, now also the research scientists that do model development and do sort of forward looking model development, you can also think for like, six, nine months out and build better architectures, et cetera. But also the completely open source arm, where you can collaborate with anyone in the world.

• 1:51:16Tim Scarfe?

  Amazing. Well, Frank, thank you so much for joining us today. It's been great.

• 1:51:20Speaker 04

  Thank you so much for having me.

• 1:51:22Tim Scarfe?

  Now, as many of you may have seen, since we recorded this interview, Prilabs have now released a new version of Tab PFN, which is version 3.5. And it's looking really, really good. So I got in touch with Frank, and he has recorded a very short update for us, which I'm going to tag on the video now.

• 1:51:41Speaker 04

  We just released our newest model, Tab PFN 3.5. And with that, we evaluate on much more than Tab Arena. And I'm super happy to say that we took first place on seven different tabular related benchmarks, including Table Swiss text, multimodal data, relational data, et cetera, covering a lot of the problems that data scientists actually face. We didn't focus on Tab Arena that much, but also improved there. So Tab PFN 3.5, Pareto dominates other tabular foundation models, for example, being 20 times faster in a forward pass than Tab FM for the same quality, or also over 100 Elo points higher while still being faster. But more importantly, we improved dramatically on the Beyond Arena benchmark that I mentioned,

• 1:52:25Speaker 04

  which is much more similar to the types of data complexities that data scientists actually face every day, with group data, temporal data, data scales from tiny to a million data points, text in the tables, high cardinality features, and so on. And as a result, Tab PFN now is also much more relevant to data science competitions. For example, our research scientist, Nick Erickson, who previously built AutoGlue on AWS, tried Tab PFN 3.5 on the historical Auto challenge from Kaggle from 2015, which had 3,500 competitors and a prize pool of $10,000, and was won by this multi-layered stack ensemble of 36 different

• 1:53:08Speaker 04

  models with handcrafted feature engineering. And Nick had used that competition to improve AutoGlue on over the years. And with the AutoGlue on paper, had actually gotten a top 1% solution in 24 hours on 96 CPUs. And now, he used Tab PFN 3.5 and got the number one ranked solution in one line of code and one minute of compute on one RTX Pro 6000 GPU. So I'm super excited about Tab PFN 3.5 as a practical tool for data scientists now.

• 1:53:39Tim Scarfe?

  So there you go. That was the update from Frank on Tab PFN 3.5. I think it's a genuinely exciting time to be alive, because you can just take this foundation model and you can apply it to so many real world problems. I've even applied it to the spammer filter on our Discord server now, which is absolutely amazing. Admittedly, you need a fairly beefy machine to use it, and you need quite a lot of memory on the machine. But it really, really is cool. So I hope you folks have a play around with it. I think it's free for non-commercial use as well. Anyway, I hope you enjoyed the show. See you on the next one.

• 1:54:12Speaker 05

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' health care to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the House Majority PAC dot com. Not authorized by any candidate or candidate's committee.

• 1:54:42Speaker 06

  Your heart can tell you a lot about your health. Apple Watch Series 12 measures your heart rate every five seconds with the most accurate heart rate sensing in wearable. So your vitals app, now with heart rate variability, can tell you when something is off. And your readiness score can let you know when to rest and when to push. Hear the story in every heartbeat with Apple Watch Series 12. The features described are for wellness purposes only and not for medical use. iPhone 11 or later required. Based on Apple conducted study of heart rate accuracy, August 2026. Visit apple.com slash Apple Watch Series 12.