Evaluating models without test data

WeightWatcher, created by Charles Martin, is an open source diagnostic tool for analyzing Neural Networks without training or even test data! Charles joins us in this episode to discuss the tool and how it fills certain gaps in current model evaluation workflows. Along the way, we discuss statistical methods from physics and a variety of practical ways to modify your training runs. Featuring: Charles Martin – GitHub https://github.com/charlesmartin14, LinkedIn https://www.linkedin.com/in/charlesmartin14, X https://x.com/charlesmartin14Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: WeightWatcher https://github.com/CalculatedContent/WeightWatcherTalk from the Silicon Valley ACM meetup https://www.youtube.com/watch?v=Tnafo6JVoJsA deep dive into the theory behind WeightWatcher (a talk from ENS) https://youtu.be/xEuBwBj_Ov4Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  All natural data has a power law structure, a fractal structure to it. The way neural networks learn is they learn the multifractal nature of the data. And that's why they work so well on things like text and images and why they don't work great on tabular data sets. So there are correlations in the data. Data is correlated. You're trying to learn the correlations. And frequently, you're trying to learn very subtle correlations you couldn't find in some other way, you know, using some simple clustering algorithm or an SVM or something like that. So what we're doing is we're measuring the fractal nature of the data. And every layer of a neural network gives you some measure of the fractal properties in that level of granularity. And so alpha is like a measure of the fractal dimension.

• 0:41Speaker 00

  And what we know is that it measures the amount of correlation in that layer.

• 0:57Speaker 01

  Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive, and accessible to everyone. Subscribe now if you haven't already. Head to practicalai.fm for all the ways. Special thanks to our partners at Fastly for delivering our shows super fast to wherever you listen. Check them out at fastly.com. And to our friends at fly.io. We deploy our app servers close to our users and you can too. Learn more at fly.io.

• 1:32Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International. And I'm joined as always by my co-host, Chris Benson, who is a tech strategist with Lockheed Martin. How are you doing, Chris? I'm doing very well, Daniel. How is it going today? It's going well. You know, I've been training quite a few models recently, NLP models for question answering and other things. And one thing that always comes up in that is, you know, how long do I train this thing? Am I overtraining it? Am I undertraining it? How do I test it appropriately? Am I testing it right? What else should I be doing? I'm all of these thoughts are running through my mind.

• 2:16Daniel Whitenack?

  And I'm pretty excited because today we have joining us Charles Martin, who is an AI and data science consultant with calculation consulting. And this is basically one of the things that that he thinks about a lot and builds tooling for. So, welcome, Charles.

• 2:31Speaker 00

  Hey, great. Thanks for having me, guys.

• 2:33Daniel Whitenack?

  Yeah, well, the main, I think the thing that I saw of your work that really interested me was this Weight Watcher tool, which is an open source diagnostic tool for analyzing neural networks without the need for access to training or even test data, which is super interesting, and I want to get into all the details about that. But maybe just describe for us kind of pre Weight Watcher. What led up to Weight Watcher? What were the sort of motivations that were going through your mind? And maybe the things that you were encountering in your own work that led you to think about this this problem?

• 3:10Speaker 00

  Sure, sure. So I do consulting in AI, and I had some clients working with me to do text generation. So this is years before GPT and all these amazing diffusion models that existed. And we were training LSTMs to generate text. There are things like, you know, weight loss articles and reviews on Amazon, stuff like that. And I realized that, you know, as I use these models, I can't really evaluate them. Because if you're training like a classifier, like an old SVM or XGBoost, you know, you can look at the training accuracy. But if you're trying to design a model to generate text or some other natural language processing problem, like, say, designing embedding vectors for search relevance, it's really hard to evaluate whether your model's converging or not.

• 3:54Speaker 00

  And now I had studied statistical physics of neural networks when I was a postdoc in theoretical physics. So I knew that there are techniques from physics that make it possible to analyze the performance of these models and to estimate how well they're performing. And what I realized is that nobody in the machine learning or AI community really knows about this stuff. Because it's like, you know, from the early 90s, early to late 90s, where a lot of this research was done. And, you know, the people doing AI and machine learning are not theoretical physicists. You know, they're computer scientists. They don't know about the works. I said, you know, I-

• 4:25Daniel Whitenack?

  Except for you and Daniel there.

• 4:27Speaker 00

  Oh, yes. You know, it's a very broad field. And there's so many people doing AI now that it's really fun because there's so many different backgrounds. And I was at a conference, maybe 10 years ago, maybe nine years ago, and I met an old friend of mine, Michael Mahoney, who's a professor at UC Berkeley. I was at MLConf, it was run by the guys at that time who were doing, oh, what was the name of that company? They had a recommender system, a recommender product. They were Turi AI, they were eventually acquired by Apple. And I was talking to Michael, I said, you know, there's a lot of theory around deep learning that is very similar to what we see in protein folding. And my advisor was actually the, him and his student, John Jumper, developed the first version of AlphaFold.

• 5:09Speaker 00

  So what happened was Google acquired AlphaFold, they hired John Jumper, who was the student from Chicago, and basically souped up his thesis. And that's where AlphaFold comes from, this amazing technology from DeepMind that can predict protein folding. So there was a lot of theoretical work I had done as a postdoc, and I was talking to my advisor about some of the stuff they were doing in protein folding way back before AlphaFold was released. I thought, you know, I think I'd like to try my shot at doing research again and see if I can develop some theory that would allow me to understand why deep learning works. And that project, you know, it's been about seven years now of research, and that's led to the Weight Watcher tool.

• 5:45Daniel Whitenack?

  Cool. So like, it's probably very typical for people to think about, you know, oh, I'm going to evaluate my model. I have a test set. But could you describe a little bit about two things? One is like, why from your perspective, at least in certain situations, like a test set doesn't give you the indication of behavior that you're, or behavior or performance of a model that you're wanting? And then how that connected to these things from the physics world?

• 6:17Speaker 00

  Right. So let's say we're training a model to generate text. There's no test set, right? You have to read the text and ask, okay, does it look human or not? And that's sort of where the first problem came, is that there are many problems in generating, when you're generating things. Another would be, let's say you're doing search relevance. I'm trying to predict what somebody wants to click on. I have clients like Walmart, for example, we built these systems for them. It's very expensive to run an A/B test. So you can test things in the lab and you can like make a model like an SVM model to predict what people will click on, but you don't really know how it's going to perform until you put in production. And there are all sorts of biases that exist in the data because there's like presentation bias. People tend to click on things that are in the first element and that

• 7:00Speaker 00

  screws the model up. So there are many cases, another good example is in quantitative finance, when you're trying to predict the stock market and you have models where you would like to train some neural network to learn something about how the news predicts the market, but if you train it directly on the market, you'll overfit it always. And so you have to have some way of evaluating whether your models are converging properly or not without just looking at the out of sample, you know, test sample. You don't, a lot of data is out of sample, or you don't, you can't really evaluate it without human judgments, or it's very expensive.

• 7:33Chris Benson?

  Would you, we kind of would infer that we're, that we're probably seeing a lot of practitioners running into these kinds of issues over time. And, you know, in a lot of cases, as, you know, if you look over the last few years, as everyone's kind of ramped up in the space and have been learning how to do different types of deep learning training, do you think that in terms of those accuracy issues that a lot of practitioners are kind of missing it altogether? Or do you think they know that it's there, they just don't know how to solve it? Or can you lay the land with it?

• 8:03Speaker 00

  Well, let me give you an example. Let me give you an example. There's a recent paper that came out of Google DeepMind on the scaling properties of very large language models. And it showed that what we thought we knew about large language models from two years ago from OpenAI, a paper that they wrote, was totally wrong. They misunderstood how the scaling properties work. And the question is things like, when you have a model and you're trying to train it, should you be trying to optimize the hyperparameters or should you be adding more data? You can think of it like, and that's sort of very crude sense, you know, you're trying to train these models and essentially what was happening at OpenAI is you're training these large language models and they didn't realize that they should be adapting the learning rate to the data set size. And when you change that, when you adapt the learning rate to the

• 8:46Speaker 00

  data set size, you get very, very different results than if you don't. And it looks like, and we know that a lot of these large language models, like BERT, for example, are just not properly converged. There are a large number of layers that are Simply undertrained. And I, I think that basically there's the theory that people are using. There's no way to look at a model and ask, how close are you to conversions? If you think about something like an SVM, let's go back, you know, I'm an old guy. Let's go back 10, 15 years ago, we run SVMs. There's something called the duality gap. You can look at the duality gap in an SVM and you can ask, how close are you to the bottom of the, of the, you know, you have, it's a convex optimization problem and you can tell how close is your solver to actually being at the optimal solution. You can tell that, that's theoretically known.

• 9:31Speaker 00

  So it's somewhat puzzling that, you know, now you have sort of deep learning. People understand that deep learning is sort of like a convex optimization or rugged convex optimization because they know you don't have, you don't have local minima and there's an issue that there are lots of saddle points, but no local minima. But yet there's no theory which tells you whether you're converged or not. And, and so it's like, what, what's going on? So people are trying to solve this and I think this is where, you know, you, you, you start training a model and you don't know, have you, have you trained it enough? Do you need to train it more? Let me give you a really practical example, which we have with, we have a user who's using Weight Watcher to train semi-supervised models to determine whether the land you own is, qualifies for carbon credits, right?

• 10:16Speaker 00

  So they're trying to, can we use AI to help with climate change? And one of the biggest problems they have is how much data should we add to the model? We have a model, we have data. Acquiring data, acquiring good, high-quality labeled data is very, very expensive. You could easily spend millions of dollars on a data set, maybe 10 million. I know guys, self-driving car companies will spend easily $10 million on a data set. So it would be nice to know, given the model that you have, do you need to, if you add more data to it, will it help? So we can answer that question with Weight Watcher.

• 10:48Chris Benson?

  If you can kind of talk a little bit about some of the underlying, because you're pointing out that there's a lot of opportunity for people to not be optimal in their approaches and kind of miss some of that. So it almost raises, almost raises a kind of a bigger issue that we may have as a community. If, if, if that's the case in terms of like, how do we solve some of those problems in the large? Aside from the specific tools, you know, what, what, what are your, what are you thinking in terms of like, how should people approach these problems different?

• 11:15Speaker 00

  Well, look, I think the first thing you have to ask is, I'm beginning to train a model. Is my model big enough? Is it small enough? Do I really want to spend millions of dollars doing brute force hyper parameter tuning? You know, should I be tuning the like, here's a basic question comes up with every client. I have a model. Forget about deep learning, SVM. Should I add more data or should I have more features? Let's say I have XGBoost. Should I add more data, add more features, or do more hyper parameter tuning? It's all expensive. What direction do you go? And, you know, there is a, it's a difficult, these are difficult problems. And if you add more data, is the data the right quality? Is the data mislabeled? Are there duplicates in the data? Is the data too similar to the data you've already added? Is it too different from the data you already added?

• 12:00Speaker 00

  Basic questions, we just don't have, I mean, very, very basic, broad level questions that we have almost no answers to. Everything is brute force. You know, if you want to train a neural network, you, you go out, you get weights and biases. Or you go to Google Cloud and you just spend a fortune on hyper parameter tuning. Do you really have to do that? Or isn't there something better you could do? Here's another example. When we started this project, there were maybe 50 open source pre-trained models. Open source models, right? VGG, the VGG series, ResNet, things like that. You go to Hugging Face now, there are over 50,000. Which one do you pick? Should you pick BERT? Or something else? Everyone uses BERT. BERT is highly under, is highly under optimized. If you compare BERT to XLNet. XLNet is much, much better.

• 12:45Speaker 00

  Not only do the, do the academic papers show that XLNet performs better on at least 20 different metrics. You can use Weight Watcher, I have a blog post, you can see that it's just night and day between XLNet and BERT. But is it worth the money to spend to try to optimize XLNet? Why does everybody focus on BERT? Because it has a cute name and it's made by Google. I mean, you know, it's really hard to know which model to pick. And it's hard, these models are very hard to improve. If you're trying, so there are a lot of just broad open questions like this. Which model do I pick? How much data should I add? How do I evaluate the quality of my data? Do I really need to do brute force searching on everything? If I put something into production, how do I know if the model doesn't, it breaks? I don't know if you guys worked in production environments.

• 13:29Speaker 00

  I work in environments where things break every six weeks. It's Thanksgiving comes, model's broken. Christmas morning, model's broken. How do you monitor these things? So I think, I think we're in our, I think machine learning and certainly AI is in the intimacy of engineering, certainly compared to where we are in software engineering. We're 20 years behind where software engineering.

• 14:11Daniel Whitenack?

  So Charles, I definitely, it's interesting kind of these, I guess, scenarios that you bring up because it's definitely something that happens. I mean, sometimes in a, in an actual real world setting, like with my team, it's like, we have what data we have. What model is appropriate that fits that level of data, right? Or maybe you have a whole bunch of data and the question is, do I need all of it for this, you know, model that I've already kind of decided on? Or all of these sorts of things. And then you get to the, to the training questions that you've, that you've brought up. I'm wondering if you could just give us a sort of high level overview of, because I think the main thing that, that if I'm understanding right, the main kind of tool that's come out of this train of research

• 14:56Daniel Whitenack?

  that you've been working on is the Weight Watcher tool. Could you just give us a kind of broad overview of what the tool actually, like functionally does and where it fits into, into a researcher or a developer or a data scientist workflow?

• 15:13Speaker 00

  Sure. So the tool can be used both when you're trying to train models, AI models, or you're trying to monitor them in production. From a training perspective, the tool gives you insights into whether your model has converged.

• 15:29Speaker 00

  And it does so at a layer by layer basis. So I'm not aware of any other technology that allows you to look at the layers of a neural network and ask, has one layer converged and has another layer not converged? So there are cues you can look at. You can look at something called the alpha metric, which is the amount of correlation in the model. And if the alpha, usually if you have a computer vision model, your alpha should be down around two. In natural language processing transformer models, the alpha should be between three and four. If your alphas are larger than that, chances are the layer is not properly trained. You can then visualize each layer and you can look at the layer, its correlation structure. And that correlation structure should be fairly smooth. It should be linear and smooth on a log log plot.

• 16:14Speaker 00

  If it's choppy or has sort of a strange shape to it, something's wrong. If your layers have lots of rank collapse, so lots of zero eigenvalues, something's wrong. We've identified something called a correlation trap, which is in deep learning language would be, you didn't clip your weight matrices, you didn't regularize that layer correctly. So you can use the tool during the training of a neural network to monitor the training. You can find layers that are basically broken, they're not trained correctly. Think of it like you're building a house and there are cracks in the bricks. You put a brick in, it's cracked, you need to replace it. You can adjust regularization up and down on the layer, you can adjust learning rate up and down on the layer.

• 16:58Speaker 00

  You might find that when you're training a model, some layers are beginning to, they're well-trained and they begin to overfit, so you might want to freeze them. So you can freeze, so as people talk about early stopping, I talk about early freezing. So you might freeze some of the early layers and let the later layers converge. So Weight Watcher allows you to do all of this by, as a, you know, it's very much a, you have to do it by hand, you have to go in and visualize it and see what's going on. But it allows you to inspect your models to determine whether they're trained correctly. It also allows you to look at models in production. So if you're deploying AI models in production and you know, maybe you're retraining your models regularly, it would allow you to give you like a warning flag, like a model alert system that would tell you, hey, you broke this layer. We have an example in our paper, we have a paper in Nature.

• 17:44Speaker 00

  Where we show that in one of the Intel systems, they applied a data compression algorithm to compress the model to go on the hardware and they screwed up one of the layers and you can see this with Weight Watcher, it will flag it for you. So while you're, so as you're deploying models in production, it can monitor them for you. And remember, it doesn't require any data. So it's a very light, very light touch, you know, very simple integration to integrate into your AL Ops monitoring pipelines. I think of it sort of like an AI uptime tool. It allows, it gives you like an early warning and I, you know, so this is how you use the tool. You can use it during training to make sure your models are converging well, or they haven't converged properly, you go back and fix them. Or you can use them after training in production to monitor for problems.

• 18:26Daniel Whitenack?

  So I was kind of trying to think of analogies in my head while, while you were talking and you, and you gave a good one in terms of the, the house and, and the cracks. One of the things I was thinking about, like you mentioned, you mentioned BERT earlier, which no doubt in, in the sort of time when BERT came out, it was quite an advancement and like many people have built amazing things on, on BERT. But I was thinking about like that and where we've come from there and, and also thinking about, uh, my wife owns a manufacturing business and they've got this principle in manufacturing about find the current biggest bottleneck in your process, right? Address that. As soon as you address that, there's going to be a next biggest bottleneck that you address next, right?

• 19:09Daniel Whitenack?

  And you kind of just keep working your way through. So I'm wondering like BERT obviously is a, a good advance, but then like you can analyze that model and see maybe where the next biggest sort of offending area is and kind of address that. And I was also thinking about the tool that you were mentioning, all the things you could do with it, you could probably analyze your model in development for years, you know, fixing all sorts of things and doing all sorts of things, right? But at some point you have to ship your model, right? So maybe there's this process of, and I'm wondering your thoughts on this of like you, using the tool to find sort of these like worst offending parts of your model.

• 19:48Speaker 00

  Yeah, yes.

• 19:49Daniel Whitenack?

  And maybe like at a certain point, you get to a point of diminishing returns or something like that, right?

• 19:55Speaker 00

  Yeah, this is a coarse grain tool. It's not meant to go in and study epoch by epoch and try to fine tune exactly what's going on. That's exact. I'm really glad you brought this up, Daniel. Because yeah, sometimes you work with the academics and they always want to use it as a regularizer. You want to optimize the loss. No, no, no, that's not the point. It's an engineering tool. It's an engineering tool. It's designed to go in and find out where the cracks are. So if you're, I don't know if you guys in San Francisco, you know about the Millennium Tower?

• 20:21Chris Benson?

  Yeah.

• 20:21Speaker 00

  So my little nephew, he's all into construction and he's always talking about they got to tear the Millennium Tower down. Tear it down. Junk it. Cause it hasn't, like they built this tower and it's like the leaning tower of Pisa. It's tilting. And if you go into the basement of the Millennium Tower, and this is like, you know, like condos, like multimillion dollar condos. You know, I think probably like, you know, the Marissa Meyer may own a condo there. I mean, it's ridiculous. And they built this thing and downstairs you look and there are cracks in the steel. It's like, guys, the thing's gonna, it's gonna fall down. It's cracked. And it's like, this is what Weight Watcher does. You go into your models and ask, are there gross problems that should not be there? Right? This layer is overtrained. This layer suggests that the data is mislabeled.

• 21:04Speaker 00

  This layer has a correlation trap. This is what you're trying to do. And you know, frequently in engineering, you're under time constraint. So you know, you gotta get this thing out and into production. You want to make sure it's not crazy. And it allows you to, Weight Watcher allows you to detect problems that you could not detect in any other way. And that's the key. It allows you to find a major problem.

• 21:24Chris Benson?

  So one of the things I was wanting to ask you, uh, cause you said something a moment ago and then kind of circling back to that, that I'm very curious about. To bring me and other people in our audience along that may not be as familiar with that. I often rely on Daniel's expertise on this and I want to rely on yours on this. You mentioned when we're talking about, you know, kind of testing those layers as you did. Going back to the alpha and you specified, you know, for, you know, ranges of two, uh, for, for the visual and the three to four for like natural language models and stuff. I'm assuming that that's one of the mechanisms that you're using in the software. Can you talk a little bit about what are the other mechanisms that are there along with that? And maybe how alpha is used, like what, if somebody is not familiar with that concept,

• 22:07Chris Benson?

  what is it about alpha that's identifying that so that they understand that a particular layer might be brittle in the sense of it's not fully converged, you know? How are you approaching that? Kind of bring us along to try to catch us up with you on, on how you're thinking about that on a.

• 22:22Speaker 00

  Like, why does it work?

• 22:23Chris Benson?

  Yeah, why does it work? What is it about Alpha and other things that you're using in the software that yield that level of insight that you're describing?

• 22:31Speaker 00

  So what we know from, where does deep learning work? Deep learning works on natural things, natural images, voice, text, things that are really part of the natural world. And the natural world exhibits a multi-fractal structure. If you look at a tree, I don't know if you remember like L systems or computer science or some of Mandelbrot's work. Most natural systems have, or just think about text, zip laws, power law structure in text and documents. All natural data has a power law structure, a fractal structure to it. And when you, the way neural networks learn is they learn the multi-fractal nature of the data. And that's why they work so well on things like text and images and why they don't work great on tabular data sets.

• 23:17Speaker 00

  So what you're doing is there are correlations in the data. Data is correlated. You're trying to learn the correlations. And frequently, you're trying to learn very subtle correlations you couldn't find in some other way, using some simple clustering algorithm or an SVM or something like that. So what we're doing is we're measuring the fractal nature of the data. And every layer of a neural network gives you some measure of the fractal properties in that level of granularity. And so Alpha is like a measure of the fractal dimension. And what we know is that it measures the amount of correlation in that layer. In other words, you're learning the data is obviously not random. It can't be random. You're learning, you're trying to learn patterns. So what we've discovered empirically, and there's some deep theoretical reasons for

• 24:01Speaker 00

  this, but qualitatively, what's happening is you're learning the natural patterns in the data and those patterns, they have to be there. So if you're looking at text data and you start seeing Alphas around six or seven or eight, the layer hasn't learned the correlations. It just didn't learn anything. And it's just sort of there or it learned it, the correlations are so weak that it's not really contributing to anything. So you just, and we know that many of these models are just have these extra layers, they're way over parameterized and they're, you know, so that's what's happening. And if the correlations, if there are strange or spurious correlations. There are things that that cause alpha to be small for spurious reasons. Like, you know, you didn't regularize your layer correctly. And so there's a giant weight matrix. Like you have these, you didn't clip the weight matrix all in.

• 24:45Speaker 00

  So the regulars are failed. So it can detect the difference between when there are problems with the optimizer and when there's actual natural structure in the data. And it allows you to distinguish between these two. And that's what it's doing.

• 24:58Daniel Whitenack?

  Am I correct? Just, just for, uh, for clarity's sake, in terms of when, when we say like, it's doing this without the test data or the, or the training data, really you're doing these calculations and you're detecting these, these parameters, these metrics based on the weight matrices, right? Is that correct?

• 25:17Speaker 00

  Yes, only on the weight matrices. We don't, you don't need to look at the data.

• 25:22Daniel Whitenack?

  So in that case, is it, uh, like the tool itself, in terms of how people would run it? Because it's doing these matrix calculations, is, is it necessary? Like, could you speak to like the computational bits?

• 25:36Speaker 00

  Okay.

• 25:36Daniel Whitenack?

  Other than like, am I gonna spend, am I gonna spend five hours waiting for Weight Watcher to analyze my model? Or is it gonna happen in five seconds?

• 25:44Speaker 00

  The current model right now, it depends on the size. It runs a singular value decomposition on each layer. So that's a G, that's a high memory GPU, CPU level. It's a high memory CPU intensive task. It doesn't, it's not optimized for GPU. So you run a normal CPU. It does require some memory. Most layers aren't too large, so it could take anywhere from a couple of minutes to an hour to be. I mean, if you're trying to run it on GPT and you have a thousand layers, it's gonna take some time, right? If you just have a few layers in your model and you're training like a small model, it's, it's, it's very, very fast. You know, generally you would hope that it is faster than an epoch in training. But it's not GPU optimized. So one of the things we're working on is I'd like to, if I commercialize the product, is to make a version that's very, very fast.

• 26:28Speaker 00

  It's like you would like distribute all the calculation on the nodes and come back to you. So that's the kind of like it from. So this is an open source tool, but it, it runs a. Simple SVD calculation. So it's a little compute intensive, but again, it's sort of my, my theory on this is that if you're training small models, it's pretty fast. If you're training really, really big models, well, you're going to have the compute, you chances are you have the compute resources anyway. And you're not renting a GPU for it. You don't need the GPU. You can run it. So that's, that's sort of the takeaway.

• 27:34Daniel Whitenack?

  Well, Charles, I, I mean, when I first saw the, the tool, I was very interested in it. And I did take time to go ahead and just pull, pull it in one of my notebooks and, and look at one of, look at one of my own models. Cause I wanted, I did want to get hands on with it. It was a question answering model based on XLM Berta. And I analyzed it with Weight Watcher. I did not do every single thing that you described on your, on your repo. Cause I, I'm still, you know, dipping, dipping my toes, right?

• 28:05Speaker 00

  I, I guess great. I, it ran, it actually.

• 28:07Daniel Whitenack?

  It ran. Yeah. And so it's a PyTorch based model. It ran. I, I didn't time it. So I don't know exactly how long, but I did find out, at least I found out, according to Weight Watcher, 10, 10 of my layers are under trained. So.

• 28:21Speaker 00

  That could be.

• 28:22Daniel Whitenack?

  Yeah. I, I, I at least found, found that out. So could you speak a little bit about like the tool itself? So you mentioned like how people can integrate it in their workflows. Could you mention a little bit more about the open source project and like how people, like if I'm, like I did, and I want to do this on one of my models, how, how would I go about doing it? And how easy is it to, to get it running on a model?

• 28:47Speaker 00

  Well, you know, this is just, it's a tool I've been writing in my spare time based on my research. There's no funding for any of this. I published with UC Berkeley, but they're not funding any of this. They're just sort of like. I'm just there to just kind of help me out a bit. I've written it all myself. It's all open, so I was having one of my staff guys help me out early on. Pip install Weight Watcher. The way it's written now, you probably need to have both TensorFlow and PyTorch installed in your environment. If you want, I can make a version that doesn't require both of those. No one's asked yet. One of the challenges I have with the tool is that I have 60,000 downloads. I have no idea who's using it. So if you're using the tool, let me know. So I can help you. I don't know what you're doing with it and I'm not going to, you know, I don't want to end up in feature creep. Where I design features in the wild, you know, I need to know what you're doing.

• 29:32Speaker 00

  So if you tell me, I'll help you. We have a Slack channel. You can go on Slack and you can ask me and I'll help you. But basically it's pip install Weight Watcher. And you just give it a model. You say Weight Watcher equals Weight Watcher, model equals my model, and you say watcher analyze. That's it. And it will return a data frame with quality metrics. If you say watcher analyze plot equals true, it will generate a bunch of plots. It will generate the plots. It's meant to be, I've been running it in a Jupiter notebook. That's how I run it. In principle, you could run it in a production environment. Again, it's really a very, it's not even an alpha one. Tool yet. It's still like 0.56, 0.57. So, you know, if you do that, reach out to me, you know, we can make a version that's more stable.

• 30:15Speaker 00

  If you need to run it in a production environment, but I've mostly been using it and it runs in the Jupiter notebook. You get a data frame. You analyze the data frame. You run a Google Colab notebook. You say plot equals true. It gives you a bunch of plots. If you add some other options, it'll give you more plots and then you analyze the plots.

• 30:30Chris Benson?

  So let me ask you a question. It's kind of a follow up to what you and Daniel were just talking about. If you're looking at the workflow, like in, so, you know, Daniel said there were like, what, 10 layers that had not converged, you know, sufficiently? How does that change the workflow? For someone who hasn't done what Daniel's done and gotten his hands on, someone just listening. Talk a little bit about what they were doing before versus the workflow they're doing now. Now that they have the insights that Weight Watcher is bringing to it, what does that look like for the practitioner?

• 30:58Speaker 00

  Well, here's the first thing, and this is exactly what happened with one of Michael's postdocs and students. Go back and look at the regularization. Did you add enough dropout on your layer? Are the learning rates too large? Do you not have enough data? Is your model just too big? Are the earlier layers converging if the later layers are not? Maybe you should freeze some of the earlier layers and give the later layers time to converge. Maybe you need to run it longer. You need to run SGD longer. Maybe you need to adjust some of your hyperparameters because you're not getting tuned. Try to adjust your hyperparameters so alpha goes down, not that it goes up. Those are the kind of things you need to do during training.

• 31:33Daniel Whitenack?

  Yeah, so if you were, maybe you could also mention the workflow. I find it very interesting what you were saying about the workflow of potentially using this within the training loops as well, like as you're training the model.

• 31:47Speaker 00

  Right.

• 31:47Daniel Whitenack?

  So one thing you could do is definitely run your model, right? Like I did, and then look at it afterwards and see, oh, jeez, I need to do something about this or that. And then, of course, then probably is the harder part of the problem is connecting with like, okay, does that mean I do one of those things you just mentioned? Or another one of those things you just mentioned? But what about that workflow in the training loop? How might that work? I know that maybe some people have heard of certain things related to like optimizing, either not doing brute force hyperparameter tuning, but doing some sort of some AutoML type of stuff or something.

• 32:31Daniel Whitenack?

  Like, people have thought about these things. So like, when you're pulling Weight Watcher into the training run, how would you think about that being used?

• 32:40Speaker 00

  If you want to give Google Cloud a million dollars to do AutoML and then have them own your models for you and feed them back to you, knock yourself out. I don't want to do that. I don't want to be trapped. You know, that's what the AutoML offering is. It's an offering to blow millions of dollars. Or if you want to get some tool like H2O and auto-tune a model and then find out it doesn't scale, and then you have to redo it. We've had clients with that problem.

• 33:03Daniel Whitenack?

  Right. I think it's, there's this wider field, though, of sort of, I guess, meta-learning and kind of learning on that. And I don't know if this would fit, like, the Weight Watcher stuff would fit into that larger space of research, I guess.

• 33:19Speaker 00

  Look, what are you trying to do? Like, what does it mean to be optimal? If being optimal means that your alphas are close to two or three? Then you should adjust your hyperparameters such that the alphas go down. That's what you do. And now, doing that analytically, typically in doing what are called analytic derivatives, meaning you try to compute the gradient from that, that's somewhat difficult. It could be done because you have to compute the eigenvalue spectrum and then you have to fit it and then you have to figure out the derivative. And that's a very complex nonlinear calculation. It's very iterative. It could be done numerically or it could be done analytically with some work. It's a lot of work. I would love to have VC funding like Hugging Face to do that, but I don't. It's just me, me and you. So you just try to tune your parameters. Alpha goes up, go the other way. You know, if you turn your learning rate up and you find your alphas are going up,

• 34:04Speaker 00

  tune the learning rate the other way and hopefully they'll go down. Obviously, it's a complex optimization problem because you have, you know, you have 100 layers, you have 100 alphas. And so you're trying to tune different layers and you're trying to tune your your layer learning rates and your amount of dropout and the amount of momentum. So in principle, you could try to do that algorithmically in a way using like a Bayesian type approach where you try to get your alphas to go down on every layer. I mean, it is in principle, you could do that. But it's a complex, you know, complex optimization problem, but that's what I would recommend. And and I think it's theoretically well grounded. I mean, the point is that you want to learn more correlations. Typically, what I found is that it's a good tool for newbies because you get into a model, you start doing something, things are totally wrong and you can go in and fix some problems.

• 34:49Speaker 00

  Okay, now we fixed it. We found like, what did we not do? Like, I didn't put the proper regularization on these layers. Let me add regularization and try again. And you can see that, okay, that's much better. So from a newbie perspective, it's a very good tool because it helps you get started. Now, it does work. Keep in mind, the tool works at the end of training, not in the early stages of training. You've got to let the thing bake for a while. You know, you can't, it doesn't, once it's about halfway through training, then you can start looking at things. It's got to have some correlations. But this is what it's for. Typically, this is sort of, you know, trying to do large scale meta learning, which just meaning you'd have to integrate the tool into some sort of Process that allows you to look at the alphas or look at more details in the layer,

• 35:32Speaker 00

  the shape of the spectral density, the number of spikes, the alphas, the the volume of the spectral density and figure out how to tune from that. I mean, this could even be used in a reinforcement learning situation where the reward, instead of the reward being something that, you know, the the agent takes, the reward is all I got smaller alpha. So I have rewards on every layer and I sum the rewards in some average way to try to get the optimizer to work, even in situations where I don't know what the reward is for a reinforcement learning situation. Obviously, that's would be nice in areas like you're trying to trade in the markets because you can't take actions that trade you. You can't trade on historical data and and expect to learn from that. So this gives you a way of sort of doing things in a supervised or semi-supervised way

• 36:15Speaker 00

  that doesn't require peeking at the test data to optimize. And then that's, I hope that answers the question, but that's sort of the idea. And there are lots of things people I think want to try. I I think it'd great if you tried them.

• 36:25Daniel Whitenack?

  Yeah, I mean, I definitely appreciate you being transparent about where where where the tool is and all that and and really the the possibilities that might happen with the tool and kind of the, there's a lot of opportunities to explore usage and and further development.

• 36:41Speaker 00

  Part of the, why I want to do with the tool is build an open source community. I can't do everything myself and there's lots of things to do. And if people want to get involved in a community, join the Slack channel, we can build things, right? That's what open source is. And I think there are a lot of people may have ideas and will be able to contribute in ways that, you know, would just expand it. And I think that it's, again, right now, to me, the way you train neural networks now, it's like you build a bridge, you drive a car over the bridge, you see if the bridge falls down and you do it again and again and again. How many cars are you going to crash into the ocean until you get the bridge right? No, people don't build bridges like that. You know, you build bridges by having engineering principles. You understand here are the engineering principles that go in and this is the load it can take and this is the wind shear and, you

• 37:23Speaker 00

  know, you you try to build bridges that actually stay up. And right now, I think deep learning is so brute force. It's like you just spend as much money as you can, do as much brute force as you can. And if it doesn't work, you try it again. And there's no principles behind. What you're doing and we're trying to add some sense, you know, and principles that are based in deep theory, like there are empirical rules of thumb, but there's also deep theoretical reasons why they work, just, just like in any other field of optimization.

• 37:50Chris Benson?

  I'm curious, I'm kind of going back to the, the engineering and the kind of, you know, talking about, you know, as this matures much, you know, entrailing the software engineering world, but one of the, one of the decisions that we all make as we're, as engineers that we're doing is kind of like, as we're creating open source community and we're trying to provide the value for that community that you're talking about, do you see the future as being community specifically built around Weight Watcher or, or is there an opportunity potentially to add the value that Weight Watcher is bringing and the new, those insights that you described and roll them into some of the other existing communities? Do you have, do you have any opinions or, or, you know, thoughts about how you integrate this in for the value of the larger community?

• 38:35Speaker 00

  Well, I think what I'd like to have is a community of people who are training models and getting them to interact with each other. A lot of the people, like I said, it's hard to get feedback. People are doing things in industry and because they are constrained by NDAs, they can't really talk about what they're doing. And I think it gives people an opportunity to really get into the space and learn how training of neural networks works without being constrained by your employer or your contract. So you can really do that's a lot of what this is. I think there are other communities doing things like people building hyper parameter optimization tools or people building reinforcement learning tools. We'd be happy to integrate the tool in. The challenge is always, you know, you want to make a tool that is self-contained. You know, if people forked the tool and begin

• 39:19Speaker 00

  changing it, it ends up, I don't know if you guys know the story of Emacs. I was at Champagne Urbana when this happened, you know, they wanted to port Emacs to basically X Windows and Stallman didn't want to do it and they forked it. You have X Emacs, you have Emacs. It killed it, you know, forking Emacs killed it because you have the X Emacs crowd and you, and you know, these guys went off and started Netscape and you know, probably all retired now or they're sitting at the top of, you know, hang out at the roof of Google vesting or Netscape. But this is the problem. You want to make sure you have an open source community. You don't want, I mean, I want people to Contributing field they can do things. If we fork it and it goes into other communities, it kills it because now those contributions don't come back. You end up in these sort of weird battles and there's no value in that. I mean, what we want to do is help people, you know, to help people and if it's necessary at this point, you know,

• 40:03Speaker 00

  commercialize the tool I would and turn into something which we can support. You like Hugging Face. I mean, Hugging Face is a lot of open source, but you know, any sophisticated technology needs maintenance. If you buy a copier machine, it's not open source because it needs maintenance, you know. So even a tool like Weight Watcher needs maintenance. So I would love to be able to work with people who would like to put it in production as an open and develop it. And then if at some point we realize, look, you know, we really need to put a service contract around this so that we can maintain it and solve some of the harder problems for you, be happy to do that. And I think that that's really the what we're trying to do. And because this is, you know, there's also a lot of opportunity for scientific research. You know, Weight Watcher has been a lot of it's come from doing research, statistical mechanics and learning theory.

• 40:46Speaker 00

  You know, we have papers in JMLR, Nature, ICML, KDD. There's a lot of opportunity for students to, you know, we have one student who is at a bank who just did his master's thesis on Weight Watcher. And so there's a lot of that kind of opportunity as well, and I think there's a lot of room for improvement.

• 40:59Daniel Whitenack?

  As we kind of get to the end here, I was wondering just quickly as we as we close out, I know you've spent a lot of really valuable time and investing in the areas maybe that people aren't focusing on in the AI community in terms of the training side of things and ways to help them in those gaps. As you look forward to, you know, the the future of where the AI community is going, what what encourages you about sort of the direction of things? Or what excites you about what's exciting for you in the community right now?

• 41:34Speaker 00

  You know, for me, I'm a physicist at heart. I did theoretical chemistry. I did theoretical physics, you know. And sometimes I'm kind of the runt of the litter, like my, one of my classmates, you know, colleagues went off and started AlphaFold, which solved the 50-year grand challenge. I have another who has started a company who's going to label all the world's translational medical data. So I'm used to this. For me, this is an opportunity to really show that we can use theoretical physics in a way that can have a broad impact. You can use theory to build sophisticated engineering tools. And a connection between a lot of the deep sort of Cold War education I have to build tools for engineers. There's a very famous statement by Carver Mead, who's a very famous electrical engineer from Caltech, who said, "Every useful experiment eventually

• 42:17Speaker 00

  becomes a tool. Everything you can measure eventually becomes a tool, you know, that you give to an engineer." And so I would just like people to realize, look, there's, you can do deep theory. There's a lot of interest, a lot of fun and interesting stuff to do, and we can turn theory into tools that people can use and build a community and, you know, just have a broader impact. I think that AI, I mean, I did AI in the 90s. People thought we were crazy, like, this stuff doesn't work, nobody believed it, right? Why are you doing neural networks? People think neural networks were invented by computer scientists, but there's a whole group of theoretical physicists doing this stuff for years and, you know, understanding sort of who we are, how the brain works, how we think, what's actually going on up here. And I think it's a very exciting time. And that's why I'm doing this. I think there's a lot we can offer from the scientific community.

• 43:01Speaker 00

  There's a broad, I think there are really deep, broad connections between general science and what's going on in AI. And that could connect back to the engineering world. And I think that there are big problems, like, like, one of the things I'm really proudest of with Weight Watcher is that there are companies using it to help climate change. When it is a huge problem, if you can use it to find some way to solve this massive problem we have, I think that would be fantastic.

• 43:24Daniel Whitenack?

  That's awesome. Well, I think that's a wonderful way to close out. Really, really appreciate your perspective there. And yeah, thank you so much for taking time to join us, Charles. It's been a pleasure.

• 43:34Speaker 00

  Hey, I really appreciate it, too. I'm glad we were able to set this up, and I look forward to the podcast, and I really look forward to anyone who tries to use the tool, wants to use it. Please reach out to me, let me know how it's working, complain to me if you don't like it. I'm not going to fix it if you don't tell me. I don't know what's wrong with it. I'm not gonna, I can't fix what I don't know is broken. And I would love that people join the community and build something great together.

• 43:56Daniel Whitenack?

  Awesome. Thanks so much.

• 43:57Speaker 00

  All right. Thanks, guys.

• 43:58Daniel Whitenack?

  Thank you.

• 44:08Speaker 01

  All right, that is our show for this week. If you dig it, don't forget to subscribe. Head to practicalai.fm for all the ways. And if Practical AI has benefited your life, pay it forward by sharing the show with a friend or a colleague. Word of mouth is the number one way people find shows like ours. Thanks again to Fastly for fronting our static assets, to fly.io for backing our dynamic requests, to Breakmaster Cylinder for the beats, and to you for listening. We appreciate you. That's all for now. We'll talk to you again on the next one.