Foundation Models for Structured Data

Predictive modeling is a core element in modern systems, and powers capabilities such as fraud detection, loan approvals, and recommendation systems. These systems typically operate on structured, relational data stored in enterprise databases, with rows, columns, and interlinked tables. While computer vision and natural language processing have undergone a neural network revolution, the tabular data layer underpinning predictive modeling still largely relies on manual feature engineering and task-specific models. Relational deep learning proposes a new approach. It treats databases as graphs and applies transformer-style attention mechanisms directly over structured relational data. Researchers are now building foundation models for tabular data that aim to generalize across predictive tasks without painstaking feature engineering. Jure Leskovec https://x.com/jure is a Professor of Computer Science at Stanford University and he previously served as Chief Scientist at Pinterest and was an investigator at the Chan Zuckerberg Biohub. Most recently, he co-founded the machine learning startup, Kumo. AI https://kumo.ai/. In this episode, Jure joins Sean Falconer to discuss the limitations of traditional predictive modeling, why structured enterprise data requires its own modality-specific neural architectures, how graph transformers generalize attention to relational databases, and more. The post Foundation Models for Structured Data https://softwareengineeringdaily.com/podcasts/foundation-models-for-structured-data/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.

• 0:00Speaker 00

  Predictive modeling is a core element in modern systems and powers capabilities such as fraud detection, loan approvals, and recommendation systems. These systems typically operate on structured relational data stored in enterprise databases with rows, columns, and interlinked tables. While computer vision and natural language processing have undergone a neural network revolution, the tabular data layer underpinning predictive modeling still largely relies on manual feature engineering and task-specific models. Relational deep learning proposes a new approach. It treats databases as graphs and applies transformer-style attention mechanisms directly over structured relational data.

• 0:42Speaker 00

  Researchers are now building foundation models for tabular data that aim to generalize across predictive tasks without painstaking feature engineering. Yuralek Kevitz is a professor of computer science at Stanford University, and he previously served as chief scientist at Pinterest and was an investigator at the Chan Zuckerberg Biohub. Most recently, he co-founded the machine learning startup Kumo AI. In this episode, Yur will join Sean Falconer to discuss the limitations of traditional predictive modeling, why structured enterprise data requires its own modality-specific neural architectures, how graph transformers generalize attention to relational databases, and more.

• 1:26Speaker 00

  This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.

• 1:45Speaker 01

  Yuri, welcome to the show.

• 1:47Speaker 02

  Thanks for having me. Great to be here.

• 1:49Speaker 01

  I'm really excited to get into, I think there's a variety of topics we can dive into today, but maybe before we get there, just to kind of ground the audience a little bit, like, who are you? What do you do? What's sort of your background?

• 2:00Speaker 02

  Yeah, I am primarily a professor at the Computer Science Department AI Lab at Stanford. Been there 15 years. My research focus on AI, especially AI over structured, tabular, relational type data. I work a lot with graph data. And in my career, the models we've developed are today used at Facebook, at Meta, at YouTube, and at a number of different places. In my career, I was also a chief scientist at Pinterest for six years, group interest from 150 employees post-IPO, and was basically building large-scale AI machine learning platforms there.

• 2:38Speaker 01

  What drew you back to academics after having experience at a place like Pinterest?

• 2:43Speaker 02

  I think Stanford is the most amazing place. It's where the future happens. And what is also amazing about Stanford is that it has this allow us to kind of flow between industry and academia and really understand what are the biggest problems out there that are worth solving. We go to the industry, we come back with the ideas, the students at Stanford are amazing. And I would really say kind of the future happens at Stanford, and that's the most exciting part to me.

• 3:07Speaker 01

  Yeah, I spent some time at Stanford myself as a student and then was drawn out to industry and never found my way back. But I guess like now that there's so much going on in the space of artificial intelligence, especially with companies, you have your OpenAI, your Anthropics of the World, and every large cloud provider doing amazing work in the space. How do you think in terms of where research is going to make significant contributions versus where maybe like the private sector of companies are going to make significant contributions?

• 3:39Speaker 02

  That's a great question. I would say research academia is different, right? We cannot compete on scale, we cannot compete on pushing products to customers and things like that. That's why we have startups, that's why we have industry. And whenever we make some new research breakthrough and we want the world to see that, we spin off a company, we spin off a startup and scale it up there. But then at the same time, there is huge value in academia, in research, in education, because the risk profile for us is very different, right? We can truly Explore, we can truly fail. We don't have performance reviews. We don't have all that that is in the industry. So we can always kind of ask about what are the paths not walked yet? What are the interesting new directions that maybe the industry is too conservative to take?

• 4:24Speaker 02

  Can we show the path there? And there's been examples of this throughout history where basically academia found a new path or researchers found a new path where the industry was just like plowing forward full scale. And right now it's similar in, I would say, in the field of AI. Of course, the frontier labs are making humongous progress, scaling up these models and so on. But there is so much unexplored, there is so much more to do, and that's what we are focusing in on our research.

• 4:50Speaker 01

  Yeah, I mean, I think a big part of that would be there's no sort of commercial obligation in academics. You could chase a problem that may or may not ever have some sort of commercial application just because it's an important problem or at least to the individual to explore and maybe mean something in the long run to how we think about ourselves, our own intelligence, some other type of scientific endeavor.

• 5:12Speaker 02

  Exactly. And even with that, right, I would say that we are very careful what kind of questions we ask. And we always ask ourselves if we solve this problem, who's going to care about it? Who can benefit from the solution? So being connected to the real world is a very important part of the way we think of the research we are doing.

• 5:28Speaker 01

  I want to get into this a little bit and first talk a little bit about predictive modeling, which I think is something that has a long, rich history in machine learning, artificial intelligence. There's fairly simplistic ways of doing some form of predictive modeling. And then there's very sophisticated approaches as well. Can you give a little bit of background in terms of sort of the history? What are we talking about when we say predictive modeling? Why does that matter? And then what is kind of the history of the discipline?

• 5:56Speaker 02

  Yeah, that's a very interesting question, and I think as we go through it, it will become clear why this is interesting, right? But predictive modeling has been around forever and it's all about forecasting, risk estimation. It's about filling in some information that doesn't exist yet. And where does this matter? This truly matters for, let's call it quantitative decision making, right? If I go ask for a loan. There is a predictive model that estimates the probability that I'm going to pay back that loan. If I'm in a hospital, the hospital wants to estimate what's the risk that if I get discharged, that I get readmitted. Right? If I am, let's say, dealing with customers, I wanna estimate what is the lifetime value of the customer. I wanna estimate how likely this customer is going to churn.

• 6:41Speaker 02

  I wanna estimate what next product or what next show item to recommend to the customer. If I'm a, let's say, financial institution, I wanna estimate what's the likelihood that this particular transaction is fraudulent. When, let's say, a user logs in, I need to estimate how likely is this a stolen identity? Somebody else is logging in and things like that. These are all predictive type problems where, based on the historical patterns, based on the data, we wanna estimate something that we don't know yet, we wanna forecast something and so on. And this has been around for a very long time and people have been building, machine learning, statistics, data science, have been building these predictive models for the last twenty, thirty years.

• 7:23Speaker 02

  And the point is that every percentage improvement in accuracy of these models means humongous business impact. Right? Even one, two percent improvement can have humongous business impact.

• 7:34Speaker 01

  How would I think about this in terms of with forecasting? Say it's like financial forecasting, I'm taking a bunch of history and I figure out what is the function that describes that history? And then I'm projecting that out to see like where could this, I don't know, trend in the stock market go or something like that? How is it, when you think of something like classification? So I wanna go and take a bunch of history as my training set and I'm gonna use that to train some sort of classifier to figure out whether an email is spam or not. Is that predictive modeling? Or is that a different type of classification of how we would think about that AI model?

• 8:09Speaker 02

  This is all what I would say falls under predictive modeling. Both examples, time series forecasting, any kind of classification, churn modeling, as you gave the example, where the idea is that based on some historic patterns, you are trying to forecast, is this person going to cancel subscription in the next month? And because we don't have the information what is happening next month, we have to forecast.

• 8:33Speaker 01

  If we take a specific example of like a recommendation systems, I think Amazon's been pretty famous for using recommendation systems for products for a long time. Like, how do those systems typically work?

• 8:44Speaker 02

  Yeah, so the reason we started opening this is because this field has remained practically unchanged for the last 20, 30 years. Right? It's all based on this idea that you bring in, let's say, a data point, a unit of something that is described with a set of characteristics or a set of features, and then you are making that prediction, right? So for example, for a recommender system, the idea would be that you have a description of the user, which would be maybe, how long ago did the user register? When was the last time the user logged in? What were the last seven products the user visited? And what categories are these products from and so on? And you have this kind of profile of the user. And then you would say, okay, I also have to now build a profile of the product.

• 9:27Speaker 02

  And now I need to learn some function that takes the profile of the user, profile of the product, and tells me how likely is, let's say, the user to purchase that product. And then if I find top 10 products that the user is most likely to purchase, I show those to the user and my sales go up 20, 30, 50%, right? That's how this is generally done. And now, majority of the work goes in building that scoring function that takes the user profile and the item profile and gives me the prediction. And if you say, how accurate is this prediction going to be? You have two aspects to it. One is, how accurately do I build the user and the product profile? Let's call it this way. And then, how powerful that predictive model on top is?

• 10:10Speaker 02

  And the point is that this is like super painful, super slow and super manual to do, right? Like, you need to hire a team of data scientists, they need to build these profiles. This is called feature engineering. They come up with some historical summaries of the user activity, put those as into the user profile. They do something similar with the product. And then they create these training data sets to build the models on top. And the models on top can be this kind of decision tree, XGBoost, CATBoost type things that work kind of well in practice and are completely respectable, as well as to more sophisticated neural network approaches and so on. But the bottom line is that you need about two full-time people to build a single model, right?

• 10:52Speaker 02

  Like, if you say, how expensive this is, it's like, I need two employees to support one model. So if I now want to have 10 models in production that are making these decisions on the fly, I need that number times two number of people to support that.

• 11:06Speaker 01

  And if I build like my e-commerce recommendation system, I also need fraud detection. I can't just pick up my recommendation system model and apply it to fraud detection. I gotta go train like essentially a net new model. I gotta do feature engineering just for the fraud detection. Probably use maybe even a different type of model to train and test against. And then probably operationalize that model with a different set of people.

• 11:27Speaker 02

  Exactly, exactly. And I would say now, like, what is the exciting thing, right? Why are we talking about the past? I think the exciting thing here is that this entire area hasn't seen real progress in the last 20, 30 years. And if you think about what has happened in the broader AI ecosystem is that we went fully neural network. Right? What I mean by that is on the, let's say in computer vision, we used to do some edge detection, some feature detection, and then build a classifier on top to say what's in the image. We don't do that anymore, right? Today, a neural network just learns directly from the pixels of the image. The same thing I would say happened on the, let's call it natural language processing area, right?

• 12:06Speaker 02

  Where we used to do all kinds of parsing and feature extraction from sentences to try to say something about what the text is saying. And today, the attention mechanism just attends over the tokens of the text and kind of the AI, the reasoning is born, right? So I think the exciting thing here is that machine learning hasn't gone through this neural network revolution. And that's the exciting new thing here is that there is the neural network revolution for machine learning ready to happen. That completely changes how we are building these models, how accurate they are, how much feature engineering it takes and things like that.

• 12:42Speaker 01

  So why is this kind of hard in practice? And why couldn't we just take? We've had this revolution around things like large language models that understand text very well. And now they understand images and audio files and even video. Can we not take those and just apply them to this problem?

• 13:00Speaker 02

  I think the argument is the following. I would say if you ask what kind of data are we using when we are making this predictive modeling, predictive problems, it's structured relational data, right? So this is data that is stored in tables that are interlinked with primary foreign key relation. This is usually stored in a database in some structured form, right? And this is the most useful data that enterprises have because this is kind of the ground truth of the enterprise. All the events, all the activity, it's all stored there, right? So now what I'm saying is if you say for images, to process images, we have special neural network to process images. To process text, we have special neural networks. And to process this tabular data, we need special neural networks. It's a different data modality.

• 13:44Speaker 02

  Image has its own set of networks that are different architectures trained in certain ways. Text has a set of networks trained in specific ways and so on. And the tabular data also needs its own set of neural network architectures and its own way to train this that can directly kind of attend over this structured tabular data. So I think it's a different data modality. That's why it needs a different approach.

• 14:07Speaker 01

  Right, I mean, I think with something like text where there's probably billions, if not trillions of examples of how sentences come together. And I can take a document from one place and a document from another place. And it can, learning one of those documents can probably help me infer something about the other document. I think with the way I think about this problem around tables, rows, columns in a databases, is there that much I can, from a pattern standpoint, like learn from one table versus another table? Like those patterns seem like they could be fundamentally different in terms of how the person has modeled the data. So it makes a lot of sense that this is sort of a different class of problem. But how do you go about, I guess, like attacking that problem?

• 14:48Speaker 01

  And does this make sense in terms of the patterns from one table not necessarily allowing you to infer something about the patterns from another table?

• 14:56Speaker 02

  That's a great question. I think the way to think of this is, as you have this private data, right, you have patterns, properties in it that are unique to you. I think another point to make is also like you cannot just textify a table and give it to a large language model, right? Large language models are amazing at what I would say qualitative, human-like reasoning. But they are not really good with numbers, if you wanna to say it very simply. And if you think about what are we storing in tables? We store quantitative data, so we need to do quantitative reasoning, not qualitative reasoning over huge amounts of data. Another point I think that is important here is to say that no enterprise has data in a single table, right? You have data spread across multiple tables.

• 15:40Speaker 02

  Usually you would have your customer catalog, you would have your product catalog, you would have a set of transaction records, you would have your website browsing click data, you would have your supplier data, you would have your returns data. And all these tables are interconnected. And the only way to learn over this is to learn over this collection of tables as they are interconnected with each other. And maybe to say more, right? In the past, the way we deal with this, we would say, oh, let's take the user table, let's take the transaction table, let's join them, and then somehow summarize the number of purchases, the number of transactions you had in the last time period. But there is kind of an infinite way of creating these summaries.

• 16:22Speaker 02

  I can count, I can sum over some time period, over shorter time period, in the mornings, in the evenings. I can add the prices, I can look at product categories, right? So the number of ways you can summarize the data is kind of blows up. And the problem with machine learning is that we kind of predetermine how to summarize the data before we start building the model. So the way to make these things better is to generalize the attention mechanism to attend over the raw events in the database and learn how to summarize them to give you that prediction. And that's the key differentiator. You don't need to be joining tables anymore, but let the attention mechanism, very similarly as in a language model, it attends over the previous words to say, okay.

• 17:06Speaker 02

  What's truly the meaning of this word? Here we are saying, if we are making a prediction, if we are filling in, let's say, some cell in a table, let's attend over the other rows in the same table, other columns, other tables far out, and figure out how to bring all that information so that we can make the accurate prediction. So it's really about generalizing the notion of attention mechanism to this structured, multi-tabular data.

• 17:30Speaker 01

  If we have that, was it unlocked from an application standpoint? Today, I feel like a lot of people are trying to apply large language models to databases for the purposes of being able to do intelligent natural language to SQL conversion. Is this yield a better version of that?

• 17:48Speaker 02

  That's a great point, right? So text to SQL is amazingly useful. But if you think about SQL, SQL is summarizing what has happened last month, what has happened last week. So you are kind of aggregating some past and maybe creating a dashboard to understand historical trends. And then maybe you can use those historical trends to do some kind of qualitative decision-making about what to do tomorrow. But if you think about predicting transaction fraud, deciding which customers to send an offer to, and so on, for that you need predictive modeling, right? So text to SQL won't get you anywhere with that. You may use SQL to generate historical patterns and then build a model on top.

• 18:29Speaker 02

  But as we talked about that, that's super brittle, manual, and takes a lot of time. So the approach we invented in my research group at Stanford, and then we founded a startup around it called Kuma.ai, is this notion of relational deep learning, where we basically take the transformer architecture and generalize it so that it can attend over this structured relational enterprise data. And the key to this approach is to think of the data as a graph, to think of your enterprise's data as a set of connections between the entities in your database, right? To think of the tables, how they are interconnected as a graph, and then generalize the attention mechanism to be able to attend over this relational structured information.

• 19:15Speaker 01

  So I mean, thinking of a database as a graph, I think people in the data modeling world have been using those concepts for a long time. Like, how is this different? Like, if you think in sort of designing schemas and things, people used to build like entity relationship diagrams, where essentially you have your tables as a node, and then you have relationships defined in terms of edges across like foreign keys and so forth. Like, is this something fundamentally different, or is it using like a similar concept as essentially the basis for doing this deep learning?

• 19:46Speaker 02

  Yeah, it's a great point. Any database is a graph, right? As you've said, and these concepts have been around for a very long time. But I feel like nobody has put one plus one together, in a sense, right? That we've been working on this graph-based machine learning graph transformers and things like that for a long time, but it was mostly applied to social networks. And people kind of, that community didn't realize that actually the database is, any database is a graph. And I think the database community was so stuck in feature engineering and running historical SQL queries, that they did not think about, oh, how can we take these AI tools and apply them to the database? Right? So what I'm saying in some sense is supernatural, right? We knew for a very long time the database is

• 20:28Speaker 02

  a graph, nothing changes. But what in some sense changes is that now the field of graph machine learning is not this obscure, it only applies to social networks type thing, but it applies to any database. And the benefit that comes with it is that is the same neural network revolution that we have in computer vision, that we have in text, natural language understanding, videos, and so on, now to another data modality, which is the structured tabular data modality. With the same set of benefits and the same set of amazing outcomes that we have already seen play out in other data modalities.

• 21:04Speaker 01

  And does this only sort of work against like a traditional database? Or could you potentially generalize this to other tabular forms of data, like spreadsheets?

• 21:14Speaker 02

  Yeah, I mean, where the data sits, it kind of doesn't matter that much. It can be in a database, it can be in Salesforce, it can be in Databricks, can be in Snowflake, can be in Spreadsheets, can be in JSON, as long as it's structured, semi-structured, and it may include images, and it may include text, and include columns, and categorical values, and all kinds of geographic information, all kinds works nicely in this framework.

• 21:40Speaker 01

  How does training work and how is it different than training with traditional transformer models?

• 21:45Speaker 02

  Yeah, that's a great point. So so far we said two things, right? The first thing we said was any set of enterprise data is structured, semi-structured, is usually stored in these relational tables. These relational tables are a graph. So now what we can do is we can take, in the old days, we would take a graph neural network and apply it to a graph, but in today's age, we take transformers, in particular graph transformers, that have the generalized notion of attention and positional encodings that can attend over this structure to give you the prediction. Now that, okay, we have these two steps. Now the third step is how do we train? And we have, I would say, two options here. One option is to actually build a pre-trained foundation model that is database schema and predictive task agnostic.

• 22:30Speaker 02

  So what this means is that now you have a single pre-trained model that can connect to any structured data, because it just represents and thinks of it as a graph. And then you get basically the specification of the predictive task on the fly, and the model is able to make you that prediction. So this means that you don't even need to be building task-specific models on the fly, but you basically have the same type of chat GPT-type experience, but now for predictive-type of questions, right? So for predictive modeling-type questions, for churn, fraud, readmission prediction, all kinds of advertising use cases, customer 360 use cases, marketing, sales, and so on. And the way this is trained, as you as you asked, is it's trained basically to teach

• 23:15Speaker 02

  the model how to do in-context learning, right? So basically, how to look at subsets of your database, use those examples to then generalize and be able to make the prediction. And when we say what are the subsets of the database, are these small local subgraphs around the entity or around the entity that you're making the prediction about.

• 23:35Speaker 01

  So if you're doing in-context learning, is there learning though that goes on in order to first force sort of this attention mechanism? Or, you know, when I think of training traditional large language models against text, sort of the input is all these, you know, massive amounts of text that, you know, then become tokens. Those are essentially the inputs of the model that help adjust the weights. And so is something similar going on? Like, you know, I think I'm not quite following before the context learning.

• 23:59Speaker 02

  So what you have to do here is you also have to amass humongous amount of pre-training data. This pre-training data now needs to be structured. So you need to amass a number of different tabular databases, things like that. And then you also need to define a number of different predictive tasks on top of this data. What is interesting is that we have just shown, we just published a paper about a method called Plural, where we can basically synthetically generate a lot of data and then train these models on top of that. But you are exactly right. The same way as in, let's say, training large language models, you need large amounts of text. Here, you need the large amounts of structured tabular relational data.

• 24:39Speaker 01

  In terms of, say, like a user, you know, interacting with this, they want to be able to do predictions against their own, you know, databases or tabular data, what is the input there? You know, because, again, with going back to the large language model, my input is essentially, you know, text that then becomes tokens as the input, which is the same thing that the model started as a training corpus. Here, it's a bunch of, you know, database data, you know, graph structures of these databases. So how do I actually ask questions about churn prediction?

• 25:09Speaker 02

  Yeah, great question. So at the setup time, you need to connect the model to your database and say, here's my database, these are my tables. And then to prompt the model, you need to specify the task. You can specify the task in natural language, or you can specify it in a domain-specific structured language that we called predictive query. And, you know, to say, I want to predict churn, you need to specify what does that truly mean? So you could say, I want to predict whether the count of transactions or count of purchases of this particular user is going to be zero over the next 30 days. Right? Proper definition of churn. And if you define it this way, then what the model is going to do, it's going to go into the database, take a set of sub-samples from the database,

• 25:55Speaker 02

  and then send them through the neural network to give you the prediction for that specific, let's say, user or customer that you want to say, what's the likelihood they will have zero transactions in the next thirty days? Right, so the way this operates is that the model basically goes, fetches data from the database, and then this data from the database is sent through a frozen model to get an output in the end. So the point is, there is no task-specific training needed here. There is no need to train the model for your specific database. It's very similar to large language models where the model is pre-trained. You give it text, it understands text, you ask it question, you get the answer. So here is similar, the model goes to your data, understands the data,

• 26:37Speaker 02

  and gives you this forward-looking predictive answer. Of course, just to say, right, now, if you write this kind of ad hoc querying, we see is very useful for, let's say, humans using this type of models or agents using this type of models where you don't know the question ahead of time. If you are a large bank and you need to predict one million times per second whether a given transaction is fraudulent or not, you would of course go and maybe fine-tune a smaller version of the model to make this faster and more cost-effective. So you can use the big, pre-trained model, or you can fine-tune it for a specific task to get more speed. Again, similar to what we see in large language models.

• 27:17Speaker 01

  Is the mathematics behind inference for these relational deep learning neural networks similar to that with sort of large language models?

• 27:26Speaker 02

  You mean in terms of the model sizes and things like that?

• 27:29Speaker 01

  Yeah, the model size and I guess the, you know, major computation that you're going to be doing behind the scenes.

• 27:34Speaker 02

  Yeah.

• 27:34Speaker 01

  The bottom line is, what is the cost of inference? And how does that sort of compare to the other models that now have become sort of familiar to people at large?

• 27:43Speaker 02

  That's great, right? So these models are Transformer attention based models, so you need GPUs to run them effectively. They are smaller than large language models. So the amount of compute that is needed is way smaller. So what this also means then that predictions and output is faster and cheaper. So that's the way the way I would describe this. What's the difference? The difference is that you need to have a very strong data backend where the data is represented as a graph. So you need a special, let's say, graph engine that is tuned for this type of AI workloads that allows you to make these computations scalably and quickly. And on the other side, you need a GPU for the results to come out.

• 28:28Speaker 02

  But since the models are are smaller than large language models, you know, there's, let's say, some billion parameter models or or something like that, you can run them quite efficiently and quite quickly.

• 28:38Speaker 01

  What's involved with getting that graph structure for your own database?

• 28:42Speaker 02

  Yeah, so what we have built at our startup Kumodot AI is exactly the infrastructure that allows you to basically take in any database. The engine will internally optimize that representation into this graphical form so that then these graph transformers can efficiently be run on top. And that's a hard part, right? Like the linear structures are kind of easier because you can just kind of feed them through. Graphs are hard because they are just this interconnected sets of objects. You cannot chop them, you cannot linearize them. So you really need to be able to kind of do this quick breadth for searches or sub sampling of sub graphs over this database. And that's the hard part. It's basically building the optimized infrastructure that allows you to do this at scale of

• 29:26Speaker 02

  tens, hundreds of billions of nodes and edges.

• 29:29Speaker 01

  Is that equivalent to, I guess, like with text generating like the embedding?

• 29:33Speaker 02

  That's very interesting. Yeah, exactly. What the model does internally, it generates the embedding, right? So now the embedding in the text kind of captures the, let's say, semantic meaning of the word. If you think of it this way, here in this database graph view of the world, now you have the embeddings of your entities that capture the semantic meaning of those entities that is predictive of the, you know, downstream activity, whatever is the task that you are trying to solve.

• 30:01Speaker 01

  You know, the concept of like a graph neural network, how long has that concept been around for?

• 30:06Speaker 02

  So I think the graph neural networks were invented maybe 10 years ago, something like that, even a bit less. That's when the field started. And I would say a graph neural network is this idea that when, if you have the data organized in a graph, when a node wants to compute something about itself or make a prediction, maybe this is a user is a node, then not only we are using the information about that node, but you also use the information from neighbors and neighbors of neighbors and so on. And the idea is that basically now neighbors are passing information messages to their neighbors and this way kind of to the node in the center that is of interest. That has worked really well for small scale task specific models.

• 30:47Speaker 02

  But now the field has moved forward to basically this generalized graph transformer type architectures, where we are not passing information across the edges, but it's actually the attention mechanism that attends over the center node, its neighbors, neighbors of neighbors, and so on. And you can see how this nicely applies to the database world where if we're making a prediction about an entity, maybe a user, maybe a product, we are then attending over the nearby tables and one hop tables, two hop tables, and bring that information all in the same place.

• 31:19Speaker 01

  So is the value there of using the attention mechanism to sort of apply, distribute these weights over the graph versus using the traditional message passing that's happening in a graph neural network? Is that the essentially it's more dynamic? So I don't have to have more of a purpose built model versus a generalized model that I can apply to, you know, any essentially, you know, database problem in this, in this particular context?

• 31:42Speaker 02

  That's exactly the right, the right intuition, right? The attention mechanism is so, is so flexible and so contextualized and context dependent that it really allows for much better generalization and for very effective pre-training. This then means that these types of models can really be applied to a very large diverse set of use cases and can generalize to them very effectively.

• 32:07Speaker 01

  How does the performance of these more general models compare to if I was to go and take more of a traditional approach where I take my two data scientists and I go and I build more of a purpose built model using my domain specific data, build that model. You know, obviously it's more rigid, but do I get, is there a quality difference?

• 32:26Speaker 02

  There is a quality difference and we shouldn't be surprised that there is a quality difference. And the way it works is the same as it worked in computer vision, where we went to basically, and in L L M's as well, where we are basically going to this superhuman level performance. Right? You could say, hey, you know, I'm so good at recognizing any, like, let's say, in computer vision, you could say, I want to build detect and build a model that detects whether there's an elephant on the image or not. And you could be saying, hey, I, I studied zoology, I know elephants inside out, I'll build a perfect elephant detector. And if somebody would go and say, I know how to detect elephants, I'll build these amazing features to detect elephants, everyone would be, would be laughing at them.

• 33:08Speaker 02

  And now, you know, when we go to tabular data, I think the outcome is similar, right? Like, it's very hard to engineer perfect features, you know, that, that give you that accurate prediction, the same way as it's very hard to engineer perfect features that detect whether something is an elephant or not, right? So if you train in a purely data-driven way, a neural network that starts attending from raw pixels, in our case, this would be a raw rows and columns and cells of the database, and learns how to aggregate all that information into this, you know, representation, this embedding that then tells me, is this an elephant or not, or will this user churn or not, it's kind of the same effect, right? So the point is that with these types of models, we can get to the superhuman level

• 33:51Speaker 02

  of performance. And what I'm trying to say, we shouldn't be surprised by it, because we've seen it before, just as I was, you know, now giving this example with computer vision, that today, nobody is surprised about. And for the machine learning database, you know, structured data world, the same revolution is out there. And it's not surprising, it's amazingly natural. That's what I'm trying to say.

• 34:13Speaker 01

  Kind of just, you know, to get a step back and looking at sort of the larger trend that's happened around the field of A I, especially in like, probably the last decade or so, where we've gone, you know, full neural networks, it's a lot more sort of this like bottoms-up learning, where we're kind of learning the patterns, you know, versus something where, if you look at like the early days of like rule-based systems, where we're kind of trying to encode all the rules that we could imagine into some sort of tree structure that we then navigate, like, why do you think that this approach has ultimately been so much more successful than trying to kind of take like our knowledge of that system and then encode it into a set of rules that a computer can execute?

• 34:54Speaker 02

  That's a great question, and exactly what you just said also applies, let's say, to this predictive modeling world, right, where we cannot anticipate all the rules, all the hardcoded signals that give us that prediction. And I think the reason, the reason is the following, right, the world is so unique, so diverse, there is so many inputs that it's very hard to preconceive all of them and, you know, write them down as rules or signals or whatever we are talking. And, you know, data that kind of captures the world is, is the king. And let the neural network learn directly from the data, directly from those raw signals, as noisy as they are, how to combine them into the final signal.

• 35:37Speaker 02

  So I think the reason it works so well is because when we are designing these rules as humans, we don't understand, let's say, the noise, we don't kind of really understand the richness. And also, right, our visual and our perception neural networks are all already kind of learning how to take this raw, noisy inputs that our bodies are sensing into some higher level of representations, right? So it's very hard for us to reconstruct, right? It's all like, the way we think it's through the neural network. So rather than having one neural network build deterministic rules, we should just train our artificial neural network on the same set of inputs, and it will be very good.

• 36:15Speaker 01

  Yeah, I mean, in a lot of ways, it seems to, I guess, mimic the behavior of humans or maybe even animals to some degree in terms of how they learn. Like, it's not like, you know, if you have a baby and it's trying to learn how to navigate the world, you know, the parent is giving them like, here's your set of rules that you need to like execute in order to crawl across the floor. Like, they're kind of doing that more through, I would say, like, plain experimentation and like the, it's almost like a reinforcement warning or something like that.

• 36:43Speaker 02

  Exactly. And then, of course, right, I think when you talk about rules, I would say, those are also in some sense important, but the way I think of those is more like explanations, right? When, when the model, when the neural network makes a decision, makes a recommendation, you can always ask why. And at that time, it's important for it to give you some general rules, some general reasoning, what led it believe to make this type of prediction, recommendation. Also, what is important, I think, in this machine learning or this predictive modeling side of things is having very strong accuracy estimates. You cannot have hallucinations. It really needs to be kind of data driven and rooted in the patterns that are in the data. And with these models, you can achieve that as well.

• 37:27Speaker 01

  Now, this idea of like the relational deep learning, you're applying this to databases, helping understand tabular structure. But there's lots and lots of other domains that can be described in terms of graphs. So can this approach be applied to other sort of domains? Even if I think about like object oriented programming, I can describe the class structure and the relationship between certain objects as a graph. I'm sure there's things in, you take this to the world of biology and biological structures. I'm sure a lot of those things can be described as graphs. Is there a way to apply this same approach to other types of domains and problems?

• 38:06Speaker 02

  Oh, definitely. I think a lot is very interesting, right? If you think about biological structures, molecules, proteins and things like that, those are essentially graph structures, right? And even like models like AlphaFold and so on, they are really, in the end, learning to reason over this graph of amino acids and this graph of proximities as the proteins starts to fold. The same happens at the scale of, let's say, small molecules for drug development and so on. So I would say this, the graph view of the world becomes very important whenever you basically have a set of entities, objects. This can be atoms, this can be amino acids, can be users, products, whatever those entities are that interact with each other, right? Like, and you know, now what are you predicting?

• 38:49Speaker 02

  You can be predicting or estimating the toxicity of the molecule. You can be outputting the 3D structure, the 3D coordinates of the protein, or you can be doing fraud detection over this interconnected graph of entities to understand whether a transaction is fraudulent or whether the account has been taken over or something like that. In the end, it's kind of mathematically, it's all the same. And that's what makes this so beautiful.

• 39:15Speaker 01

  With foundation models today, more and more of these are multimodality, where they can handle images, audio files, text, of course, video even. Are we eventually going to move to a world where also essentially tabular data or semi-structured data is part of one of those modalities?

• 39:37Speaker 02

  I think so. I think that's where this is going, right? You cannot textify an image. You cannot textify a table. It just makes it kind of super hard to learn over that, right? But in principle, you could. You could just write the image as a sequence of RGB values in ASCII and say, "Tell me, is this a cat or not?" But the point is, we figured out that we need special domain-specific, modality-specific encoders. And I think where this is going is now that, basically, we have these modality-specific encoders on top of the reasoning-based large language models for images, for videos, and of course now also for this structured tabular data.

• 40:17Speaker 01

  Yeah, I mean, even in audio, we saw a huge performance increase when we stopped textifying audio and we actually trained models directly against the audio, because there's so much nuance in audio versus what might be just available in the text. So it ends up, I think, leading to, when you textify it, essentially you run into problems where the model doesn't really understand certain pauses and stuff like that.

• 40:40Speaker 02

  I think it's a beautiful example and also just shows that the two of us right now are just talking. So everything we say should be just captured in words, but it's actually not, right? Like you see, when you work on the raw signal, on the raw speech signal, the performance, there is more information there, the performance goes up.

• 40:57Speaker 01

  Yeah, absolutely. Well, Yuri, is there anything else you'd like to share? What is some of the, from your perspective, like the big problems that need to be solved in this space?

• 41:07Speaker 02

  I think the exciting part is that I feel like the structured data world has been kind of a bit left behind. People know how to run SQL and then for everything else, they feel like they need to build a manual, you know, machine learning models super painfully and super slowly. And I think what we discussed today, the exciting thing is that we have now foundation model for structured relational data. For example, Kuma RFM is one such example. We have this approach of relational deep learning. We have the transformer architectures that can now learn directly over this structured tabular data. So actually, the structured data world that has humongous business impact is ready for the AI revolution to actually take place there as well.

• 41:50Speaker 02

  And you know, today we shed some light on it and that's what I'm very excited about.

• 41:55Speaker 01

  Yeah, absolutely. Well, thank you so much for being here. This was really, really interesting.

• 41:58Speaker 02

  Yeah, amazing. Thank you for having me.

• 42:00Speaker 01

  Yeah, cheers.