Pod Insights Y Combinator Podcast Anthropic Head of Pretraining on Scaling Laws, Compute, and the Future of AI About this episode Ever wonder what it actually takes to train a frontier AI model? YC General Partner Ankit Gupta sits down with Nick Joseph, Anthropic's Head of Pre-training, to explore the engineering challenges behind training Claude—from managing thousands of GPUs and debugging cursed bugs to balancing compute between pre-training and RL. We cover scaling laws, data strategies, team composition, and why the hardest problems in AI are often infrastructure problems, not ML problems.
Transcript 16,116 words · 2 speakers
0:05 Speaker 00 Hey guys, I'm thrilled to be joined today by Nick Joseph, the head of pre-training at Anthropic. To give viewers a high-level sense of what we'll be covering, we're going to start with the basics of what pre-training is, and then dig into how Nick thinks about strategy, data, alignment, and infrastructure at Anthropic. And by the end, you'll hopefully have a sense for how progress in AI comes directly from advances in pre-training. I would love to talk a little bit about your backstory and how you got to this point. Where did you work before Anthropic? And what were your takeaways from those places?
0:29 Speaker 01 Yeah, so let's see, I was at Vicarious, and then at OpenAI before Anthropic. So Vicarious was originally an AGI lab. And when I joined, they were making a shift to product, particularly working on robotics products. And the thing I worked on was training computer vision models for their robotics products. It was my first job, so I think I just learned a ton about how to do machine learning models, how to write machine learning infrastructure.
0:53 Speaker 00 And at the time, were you also thinking about a career as an academic? At the time, a lot of people doing AI work were in PhDs. That's kind of what I was thinking about before I started to do a company. How were you thinking about that in your headspace?
1:03 Speaker 01 Yeah, so like, I mean, actually, we went a little bit. I think a lot of my thinking on this had come from an internship I did at GiveWell, which is a nonprofit that evaluates charities. And some people there being like, "At some point, we might have AGI. It could be dangerous. We should worry about these risks. This could be a big impact on humanity." And I was not super committed at the time, and went down the economics route and was going to try to work on directly helping people in poverty. That didn't work out for various reasons, and ended up being like, "Okay, I'll at least work on AI." Either the safety thing will turn out to be important, and I'll work on that, or it won't be, and I'll just make cool things with AI that can probably help people in poverty more. I wasn't really coming at it from an academic standpoint. I was sort of like, in fact, when I switched to that, it was part of the appeal was that I could immediately go do stuff in AI. Whereas if I want to work in economic policy,
1:47 Speaker 01 I'd have to wait, I don't know, six years to do a PhD and then start. And like, it's a longer path.
1:53 Speaker 00 And what did the state of AI safety work at that time even look like? Who were the people who were thinking about that kind of stuff? I mean, there were some folks at Vicarious thinking about this kind of thing, but it was fundamentally a robotics company. And so, yeah, how were you thinking about that at the time?
2:05 Speaker 01 Yeah, so my sense was like, at the time, a lot of the AI safety discussion was kind of theoretical. Like, the models weren't actually that good. They weren't really posing these dangers. So it was a lot more philosophical. It was like, "Oh, at some point, we might get AI that's really smarter than humans. And should we weight this future concern? How should we compare that to nearer-term things?" And I think that was actually a... Just a less compelling argument. I think it was like an interesting one and like sort of made you think a bit.
2:29 Speaker 00 So next you went to OpenAI. What was OpenAI like at this time?
2:32 Speaker 01 Yeah, so I was on OpenAI. I was on one of the safety teams and kind of worked on.
2:38 Speaker 01 And then working on code models, actually. And kind of when I got there, the first thing I saw was, oh, they'd fine tune GPT-3 to write some code. But and it was really good. And I was like, oh, OK. If you're worried about AI getting really powerful, writing its own code, that seems like it could self-improve. And how likely is that to happen? So I was doing a bunch of evaluations and like studies of what contributed. And then after like eight months, basically everyone I worked with, like all the safety leads left, which,
3:08 Speaker 01 yeah, invited me to go to Anthropic. And that was sort of the reason I joined OpenAI was because I cared about AI safety and wanted to work with them. So then I went with them to join Anthropic pretty much right when it started.
3:17 Speaker 00 With that, why don't we transition a bit? These days, you run the pre-training team specifically at Anthropic. Obviously, you've been working on pre-training at Anthropic for quite a bit of time. And I'm sure it's evolved over the years, what that even entails and looks like. Why don't we start by just talking a little bit about what pre-training is? Like, how does it even fit into the way of thinking about how AI models are developed at a place like Anthropic? And what exactly do you guys do?
3:38 Speaker 01 We know that one of the ingredients to making AI models better is scale. You want to put a lot of compute in. And if you sort of step back and you're like, OK, what's the way we could put the most compute into a model possible? We need some objective that there's just like tons of data for. And one idea here is like the internet. The internet is massive. It's probably the biggest like single source of data domain is created. And you don't have labels. It's like you don't want someone to have to go in and look, read the entire internet and like say something about it. So you want to get labels out of the data itself. And the idea here is we can take some text and we can predict the next word. So you take, you know, the as the first word, you predict the second word, then you say the cat, you predict the word after that. And this means you get very dense signal. Every word is like a new example. And there's a huge amount of data.
4:20 Speaker 01 And one of the findings from GPT-1, GPT-2 was kind of as you throw more compute at this, more data, bigger models, you get better, you get smarter models, essentially.
4:30 Speaker 00 Totally.
4:31 Speaker 01 And that's kind of been the central thesis of pre-training for the whole time. There's this idea of scaling laws, which is that you can actually quantify like as you put in more compute, more data, more parameters, you get models in a very, you got a lower loss, a better prediction of the next word in a very predictable way. And I think you can somewhat foresee from that original paper, and I think like Dario did foresee this, I think many people did, but what's obvious was that once you have that, there's this positive feedback loop where you can train a model, you can use it to make something useful and sell that and get more money, use that to buy more compute, and then you actually train a better model. And we've sort of run that cycle over and over again over the past five years or so.
5:09 Speaker 00 Well, in thinking about that objective to begin, you know, I think, the way I think about the state of pre-training is yeah, it seems like this next word prediction, at least from the external standpoint, seems to be the dominant way pre-training happens. But if I rewind the clock to that era of 2017 to 2020, or 2021 and two even, there was all sorts of pre-training objectives people were considering, right? There was these BERT and BART models that were doing mass language modeling. It seems like this GPT series of models doing autoregressive modeling, as you're describing, this next word prediction seems to be the dominant one that won out. Do you have any reflections on that time period? Like, were you guys trying all of them and kind of this one worked? Or is there some sort of first principles reason why this is like the right one that should have worked?
5:49 Speaker 01 I think the answer is like, it's mostly empirical. Like, in terms of how to think of these things, I'd be like, yeah, it's empirical, just try them all, see what works. One big advantage for this autoregressive setup is that you can just sample from it to generate text afterwards in a fairly like straightforward way that comes.
6:02 Speaker 00 Yeah, like enables a product use very nicely.
6:05 Speaker 01 Like, one thing that you want is like, just one character you want from a setup is like a loss, whereas you drive down the loss, that actually is the thing you care about. And you can think of it as like, if you got too perfect on language modeling, you now can like write text as a human. You can sort of imagine you put in the title of a paper and it should spit out the entire, spit out a novel paper. Whereas I think some of the other approaches don't quite have that flavor.
6:26 Speaker 00 Yeah, totally. Yeah, it makes sense that in terms of that loop you're describing of, you know, then release something that gets you revenue and you can use that to buy more compute and iterate, this sort of gives you the most natural way to actually do that flow because you can keep releasing new products and keep getting the revenue from that to invest in more compute and so on.
6:42 Speaker 01 Yeah, it certainly gives you the most open-ended thing. You could imagine you like train something as a class. Like you train some base thing, you fine-tune it for a bunch of particular tasks. One approach people would use, they would like do this big pre-training and then they wouldn't just like open-endedly sample from it. You'd fine-tune it on like 100 specific tasks. And that could work too. I think that like the one sort of general intuition I have is like compute is the thing that matters. So like, I think if you throw enough compute at any of these objectives, you're going to get something that's probably pretty good and can kind of be fine-tuned to the other things. And it's surprising how little these details matter compared to throwing more compute at the problem.
7:14 Speaker 00 When you think about actually throwing more compute at the problem, there's a whole bunch of axes by which you could throw compute at it too, right? And if you have a specific model architecture you're training over, you can basically throw more data at that specific architecture. For a particular one, you could add more layers or make the models larger in it. You could do some kind of neural architecture search over lots of different variants. And I assume that these days it's somewhat more figured out, you know, which architecture you go for. I assume the earlier days it was somewhat less so. And I'm curious if you could speak to how you guys thought about that. Like, what is your infrastructure going to look like to do that type of determination?
7:46 Speaker 01 I mean, I think the short answer is it's hard, right? Like, what you're really doing is you're going to train this one big expensive model and you have a space of, you know, you can sort of call all these things hyperparameters. You know, how many layers do you have, what's your width? Like, you have the space of hundreds of hyperparameters and you want them all to be optimal. And you're sort of striking this balance, actually, between how much do they matter? Like, can you just take your best guess and throw more compute at it in whatever way you want?
8:08 Speaker 00 Yeah, and basically it doesn't matter.
8:09 Speaker 01 Versus how much are you letting at it precisely correct?
8:11 Speaker 00 Interesting.
8:12 Speaker 01 And I think one of the like interesting things is like, it actually doesn't matter that much. Like, I think this was in one of the early scaling loss papers. Like, you can change these things and get little wins, but like, as you throw more compute, it sort of reliably gets better. If you mess up enough, you will, you will sort of stop seeing that happen and you won't have any way to know, which is one of the, that's like kind of the hardest part in some ways.
8:31 Speaker 00 You don't know the counterfactual, basically, because you didn't run it for long enough to actually know what it is.
8:35 Speaker 01 Yeah, we have these scaling laws. So you can sort of say like, as you train a model with more compute, you expect the loss to go down as a power law. It's really a power law plus constant. So what eventually will happen is you'll curve off that power law. And then you know something is wrong. And is it fundamental? Is it like you've hit the limits of scaling? Or is it, nope, you should have changed, you should have tweaked your learning rate slightly differently? And that's, that's sort of one of the challenges. In terms of how to like figure it out, you can, the, the usual paradigm is like test things out at small scale before running them at large scale and try to find things.
9:02 Speaker 00 Small scale in terms of data or in terms of something else?
9:05 Speaker 01 In terms of everything, like you kind of want to scale things down like proportionally. So you want to say like, you want, you want to have some theory for like how you're going to scale up. Like, ah, okay, if I get 10 times as many flops, how much of it goes into layers? How much of it goes into data? How much of it goes into attention? And you sort of get that theory and then test that it's optimal a bunch with like scaling everything down proportionally.
9:27 Speaker 00 And just so I can think about what this actually looks like in those, in those early days of Anthropic, you know, your team of like 10 or something like that in those very early days or 12 maybe. What actually is your ability to use large scale infrastructure as like a relatively nimble startup at that time? I mean, a startup that was well capitalized, but still not actually that many people working at. What kind of infrastructure did you have access to, to train these early models at the time?
9:48 Speaker 01 So it's actually one of the wild things was it at least, I mean, you don't know what anyone else is doing, of course, but it kind of felt like we were like at the frontier of it. And there just weren't that many people who cared. Like I was sort of coming, you know, I was coming at it from like, we're making AGI. This is the most important technology ever. And then we'll kind of like look around and be like, and it seems like I'm one of 30 people who are working on this in like the world. I mean, I was just kind of like junior person. Everyone else sort of knew how to do this and had done it before. But I was kind of surprised at how easy it was. Like the public estimates for GP3, I remember were that it costs $5 million to train, which you're like, on the one hit, $5 million is kind of a lot, but it's like a lot for an individual person. It's not really a lot from like a company perspective. So we could totally buy like compute that was enough to train models like that. You could.
10:33 Speaker 00 And were you using a cloud provider or did you have a custom setup somewhere? Or did you literally have racks in a room somewhere that you were, you know, bought a bunch of Nvidia GPUs and you were doing it?
10:41 Speaker 01 We were using a cloud provider, but I think it's kind of, it's not actually that different. Cause one of the things that was surprising to me is you actually have to understand the, the literal layout. Like. I remember at one point, one of my coworkers running a clustering algorithm to identify what rooms all the chips were in, since we had a hypothesis that they were in different rooms and that was causing like, or you know, different buildings.
11:02 Speaker 00 Some sort of like network latency.
11:03 Speaker 01 Some sort of network latency. And you could kind of figure it out. You could like reverse engineer like, ah, okay, yeah, there's clearly like two clusters here that are connected better and there's some issue on the connection between them. Like, we're trying to push the limits of the hardware like as much as possible, particularly at the beginning where we were kind of like, we have way less funding than everyone else. We have to, and most people weren't very efficient with the compute. So we were like, ah, we could get a big lead by being really efficient at how we use the compute.
11:26 Speaker 00 Could you talk a little bit about some of the things you guys did in those early days for how to get the most out of the hardware? I think that's really interesting. Like, I think back to the days of, the early days of Google, for example, where there's these cases where they basically bought relatively cheap consumer chips and then they optimized the software to make it so you can actually get the most bang for your buck out of them. And that's how they had all this high latency, or low latency, high availability stuff. I'm kind of curious if there's some analog in the early AI era to that.
11:49 Speaker 01 I think for us, it was largely about like getting the distributed framework, right? So like, we're training on, in order to train these models, you have to train them on a large number of chips. And there's a bunch of different approaches to how to do this. There's like data parallelism, there's pipelining, there's upsharding, and like getting all of the.
12:04 Speaker 00 And at the time, there were no like great open source packages you could just grab and use that just worked for this. I mean, today, there's somewhat more of these, but at the time, I assume there was literally none.
12:12 Speaker 01 There were some. Like, I actually remember that we were working on data parallelism early on. And it was like, and now we write the, all reduce it. And I was like, we really do this ourselves? We don't like call a package? And this was kind of like, well, we're going to want to modify it, right? Like, oh, like, we don't want to outsource this to some package because A, we're about to go to a bigger scale, like PyTorch, for instance. They had a package for doing this. But we were going to go to a bigger scale than Facebook had been too.
12:35 Speaker 00 Right.
12:35 Speaker 01 And you don't want to have a dependency on a package that you're going to have to be like constantly modifying, essentially.
12:42 Speaker 00 That's such a counterintuitive sentence there too. Like, we're going to a bigger scale than Facebook. Well, because at the time, Facebook AI research was considered one of the best places to do machine learning research. Like, Fair was one of the places, Fair and DeepMind were hiring lots of people out of top PhD programs and doing lots of things. Like, what was your headspace when you were like, okay, this very established lab with great people and whatnot, we are operating on a scale that is not relevant to them? Like, was that natural and obvious to you? Or was there times where you kind of doubted the decisions you were making in that situation?
13:10 Speaker 01 I think it was surprising. I will, maybe I'm just too arrogant or something. I kind of looked around and was like, what are these people doing? They're all missing the like, big picture here. Like, I think the scaling laws were pretty clear. Like, and the arguments against, I just thought, were kind of nonsensical. Like, you know, I think the original scaling laws paper had like 11 orders of magnitude. And there was like this intense debate on whether it would continue for like, another point. And I was like...
13:34 Speaker 00 There's already 11.
13:35 Speaker 01 It seems like 1 over 11 is maybe your chance it fails here. And then like, you know, sometimes it doesn't work. Like, sometimes it just works straightforward. You're like, let it turn the moment. You're like, oh, yeah, of course. But, yeah, I do think that it was, it maybe felt obvious when you're in that headspace and you're working on this all the time and you're making those plots. And I think these things feel pretty different when you're on the outside. You know, there's a huge space of papers. Everyone tries to make their paper sound like, very robust and important. I could see, I could see being like, oh yeah, this is not really a thing. But also, different labs have different cultures. So like, I think one of the things at FAIR was it was a very, more PhD-style, independent research. People have their own ideas, pursue those.
14:12 Speaker 00 You're fighting for your compute and so on.
14:13 Speaker 01 Yeah, and to do a project like training a large language model requires a lot of people to collaborate on like, a really complicated piece of infrastructure that isn't going to be a paper, right? Like, you're not going to publish like, oh, I got a slightly, I got 5% more efficiency than the next one. And it's not respected in like, those cultures necessarily, so that might have been part of it.
14:31 Speaker 00 Okay, so then when you actually implement these, these models, you're saying you're using a level of low-level programming where, you know, you're using libraries like PyTorch, but you're perhaps not using everything right out of the box from PyTorch, because there's things you guys want to customize that are at the level of basically one level of abstraction below them. But not necessarily at the level of abstraction of, you know, writing custom CUDA curls? Or like, was that also in the, in the space where you guys were thinking about?
14:52 Speaker 01 So it depends on like, the operation. So like, I think I was mostly operating at the level of like, torch dot matmul. You know, like, ah, yes, where does a matmul go? But not thinking like, how do you make the matmul efficient? Like, I assume Torch figured out how to make a matmul as efficient as is possible. But there are some pieces, like, attention, where there was just kind of a lot of different variants. And attention is really complicated and hard to make efficient on a GPU. And those things, you have to kind of go, go more levels down on the stack. I think there was like, a process that is maybe interesting, that I never really like, thought of before of like, how to do it, which is sort of like... Modeling out the problem, the thing you're going to do, coming up with a strategy for how to parallelize it that can get to a really good efficiency.
15:31 Speaker 00 So you're thinking about MFU, basically, like your utilization on your GPU. There's like a goal utilization you're trying to get at and a strategy to get to there, you're saying?
15:39 Speaker 01 Yeah, and I think one of the things you can do is you can actually pencil and paper math out what efficiency you're going to be able to get to, right? You know all the constraints. MFU is Flops utilization, but the reason you don't get good MFU is you end up limited on HBM bandwidth, you end up limited on, I don't know, host CPU offload. There's a bunch of different pieces, but there's not that many pieces. There's like six relevant numbers there. So you can totally model it out, understand what the constraints are, and then implement something that can get there. And of course, will be really inefficient when you implement it. And then the next step is pulling out a profiler. So you want to be able to profile the job, look at how long every operation takes, have a model in your mind of how long every operation should take, and then make those two things the same.
16:21 Speaker 00 And were there good out-of-the-box profilers you could use at that time? Or did you guys have, because people weren't operating on the kind of network topologies you guys may have been using, did you have to write your own profilers, basically, to do this type of multi-node optimization?
16:34 Speaker 01 Yeah, it depends when. They were actually getting better with time. The PyTorch profiler was pretty good, actually, throughout for a single GPU. If you wanted to profile a GPU, the PyTorch profiler would work. But if you wanted to profile a job on hundreds, thousands of GPUs, that hadn't really been done much. And then that was kind of more of us hacking into the profiler to figure out how to combine all the traces together.
16:54 Speaker 00 And then one more question to that earlier is, you had mentioned you hadn't really done a lot of this work before, maybe sometime at OpenAI and those early days in Anthropic. How did you actually go learn all this stuff? What was your process for learning about those six things that were relevant to bandwidth limitations and whatnot?
17:08 Speaker 01 So when I joined Anthropic, one really nice thing was there just wasn't that much. I think my first day, I read through our entire, all of Slack and the entire internal database and learned a bunch from that. It was kind of nice to just be like, everything is relevant to me. And then I mostly learned from pair programming. Tom Brown had done all this before, so he kind of knew all the stuff quite well. Sam McCandlish, his manager, had also done a lot of it before, and I just paired with them a huge amount at the beginning. I think one of the things I really like about pairing as a way of learning is you learn the thing you're trying to do. You will learn that. If you're pairing with someone better than you, they can just do it. So you're mostly just watching them. But you also learn how people do it. So something like how to use a profiler is not something you would ever learn from seeing someone's final write-up on Slack.
17:53 Speaker 00 Right.
17:53 Speaker 01 Or their PR. You would just be like, oh, they found these, they changed this specific line. It's a win.
17:58 Speaker 00 You need to watch a YouTube video for four hours of someone messing around with a profiler to maybe self-teach it or something, or to actually pair with someone is basically the best you can do.
18:08 Speaker 01 Yeah. I think there was one thing that I think is embarrassing now I look back, is I'd never actually used a debugger before joining Anthropic. People talk about it at PDB, like, yeah, that's a thing people use, but print seems fine for me.
18:18 Speaker 00 Yeah, sure, sure.
18:19 Speaker 01 Then I watched them and was like, oh, no, debugger is a super useful tool. This person's way faster at debugging things, particularly if it takes a long time to start up the code, which it can. And yeah, learning that sort of thing, I think, comes best from pairing.
18:32 Speaker 00 Yeah.
18:32 Speaker 01 And then there's, of course, the obvious, you just learn by doing. I eventually did spit a profile and stare at it for many, many hours.
18:38 Speaker 00 Totally, yeah, exactly, yeah. Okay, so then, that was sort of the very early era. Over time, obviously, pre-training has become bigger and bigger. As you're describing scaling, I imagine you're using many X more GPUs, much more compute over time. I'd be really curious to hear, first at a high level, what do you feel has changed about the pre-training strategy that you could talk about? Obviously, there's more compute, but what does that actually mean to have more compute in terms of what you think about differently from those early days versus now?
19:03 Speaker 01 I'm sure the things that haven't changed, because I think it is shocking how it has changed in some ways.
19:07 Speaker 00 Okay, yeah, that's really interesting.
19:07 Speaker 01 I think I'm still pushing down the exact same metric that I was on day one.
19:11 Speaker 00 There's some loss function.
19:13 Speaker 01 Loss go down. And I think you could look at some, you could probably run the first model I trained on the same metric and just make a plot of progressive team over time. So that's all the same. I think the biggest-
19:25 Speaker 00 It's like one OKR. It's like one thing that matters, basically.
19:27 Speaker 01 Yeah. And like, I mean, talking to like OKRs, it's the very size of the company, you're like, oh, should you do OKRs? And it's always felt a little bit funny for a team like, where I'm like, sure, I can just pick a loss value.
19:37 Speaker 00 Right, exactly.
19:38 Speaker 01 The answer is like, as low as possible, and we will continue to work on that forever. I think the biggest things that have changed has been a little more specialization. Like, I think at the beginning, I mean, the first like three or six months, I tried to read every PR in the codebase. And that was great. I knew all the pieces, et cetera. And as you grow, it's kind of, everything gets like a little more precise, you know? People really dial in exactly how attention should work, let's say, or, you know, really dial in like the parallelism strategy.
20:03 Speaker 00 Yeah.
20:04 Speaker 01 And you end up with a team where it's a bunch of people who are like deep experts on individual things. Which is great, because it means you can go, you can go really deep on those things. But sometimes you, at least for me as a manager, one of the things you sometimes have to think about is like making sure the bigger picture makes sense. And also that you have enough people who actually do understand the whole bigger picture, that there's no like single point of failure.
20:24 Speaker 00 Yeah, it's interesting you frame it in that, with that trade-off, right? Because as you were describing that, I was trying to think, you know, is this a bug or a feature? Like, there's some obvious features of it, which is you get expertise and you can optimize certain things. But I imagine your ability to take bigger swings becomes more complicated if not everyone's exactly pointed in the same direction. Like, how do you wrestle with that now?
20:44 Speaker 01 Yeah, I think I mostly just try to get a balance of people. I think one of the challenges early on...
20:49 Speaker 00 Oh, people, oh, that's interesting.
20:50 Speaker 01 Yeah, like, I think people really do have a preference here, has been one of the things I've seen. Like, there are people who really want to be a generalist and understand everything and like lightly touch on things. There are people who want to like pick an area, often they've already picked that area, and they're like deep experts in precision. You know, they studied, they did a whole PhD in precision and just want to think about that.
21:06 Speaker 00 Yep.
21:06 Speaker 01 And you want to get some balance of that. I think there was a phase where we'd hired a lot of people who are more generalist shaped, because that's what the people who joined early startup for the government and everything. And then you ended up with kind of everyone doing everything and no one really, really deeply understanding one thing.
21:21 Speaker 00 Yep.
21:21 Speaker 01 And that's one failure mode. But I think if you get too many people who are specialists, you end up with a lot of effort has to come from the manager, from like the lead to connect everything.
21:31 Speaker 00 Yeah.
21:31 Speaker 01 And to notice something like, ah, if we change the architecture here, that would make this like efficiency consideration... Over there, way easier.
21:39 Speaker 00 Interesting.
21:40 Speaker 01 One of the things I really liked, kind of like at the very beginning, was like, it was working on efficiency. But I could just go and like, be like, ah, well, what if we change the way we do like this particular step? And we'll be like, oh yeah, that's probably fine, like easy change. And then like, you can avoid this whole complicated project to make this operation that was hard efficient.
21:54 Speaker 00 Yeah.
21:54 Speaker 01 Because you can make an easier operation efficient.
21:56 Speaker 00 Hmm, interesting, yeah. So as the level of compute has also gotten bigger, so I'm sure anyone can imagine, okay, there's more GPUs now. You have the network of them more. Are there some like kind of non-obvious challenges that have arisen over time? Where you guys have just like banged your head against the wall to solve them because of the amount of compute you're dealing with? That people wouldn't otherwise know about that like you want to share?
22:17 Speaker 01 I think that connecting them is one that's maybe interesting and like surprisingly hard.
22:21 Speaker 00 Okay.
22:22 Speaker 01 Because you really do get more and more chips connected. And like one thing that I think is like the standard way people parallelize chips isn't, um, the whole thing is one failure domain. Like one chip fails, the whole thing can crash and.
22:35 Speaker 00 The standard way, as in the standard way people doing AI or the standard way in other fields where people are doing.
22:39 Speaker 01 Uh, in AI for like, I mean, at least like, I think at the beginning, you know, like first versions of things were this way and.
22:46 Speaker 00 So it's like you have 100 GPU cluster or whatever is 128, like if one of them dies, job fails, basically.
22:51 Speaker 01 Yeah, I mean, the simplest thing is if you just like distribute your models. So say you put like every layer on a different chip and you lose like layer seven, like, yeah.
23:00 Speaker 00 Yeah.
23:00 Speaker 01 You're not gonna like skip layer seven.
23:02 Speaker 00 Totally.
23:03 Speaker 01 I guess you could, but that's like a pretty weird model training process now. And like that leads to some interesting things, which is like, okay, so now as you scale up, you have more and more chips. And the failure rate can get like larger and larger. On the other hand, you can like, I don't know, you can like restart pretty quickly. There's nothing like you just have to like load back in some weights. So that was one thing. And then the thing was like the level of novelty at the whole stack is something that's surprising. Like basically everything from like how the chips are laid out in the data center to the chips themselves is pretty new. There just haven't been that many generations of GPUs. I think one of the things that, I don't know, when I learned computer science, my code wouldn't work. And I'd be like, oh, the computer's broken.
23:41 Speaker 00 Yeah.
23:41 Speaker 01 And I think my teacher was like, you can trust the computer's not broken?
23:44 Speaker 00 Yeah, interesting.
23:44 Speaker 01 You messed up.
23:45 Speaker 00 It's you messed up.
23:45 Speaker 01 And I think one of the most frustrating things I encountered in AI early on was. Working on something and being like, I don't know what I'm doing wrong. I'm just totally stumped. And my manager looked at it and was like, ah, yeah, probably the computer's wrong. And I was like, that seems unlikely. And sure enough, the computer was wrong. It turned out that like the GPU was broken and we had to pull in a new one. But you have to like think, like having to think about that, like the GPU can be wrong. The GPU can be slow. Like these sorts of issues, the power supply in the data center could be broken. Like there's so much more like level of depth than you like kind of expect to need as a Python programmer.
24:22 Speaker 00 And just to visualize it, like in those early days, I assume you guys were using the number of GPUs. It's probably on the order of tens to hundreds or something like that per run. It's probably not tens of thousands or hundreds of thousands per run. Or what was the rough size you guys were adding in those very early days? On the order of thousands, like would they fit in this room?
24:36 Speaker 01 Thousands.
24:36 Speaker 00 Yeah, thousands. So like you could have a bunch of racks and you could fit them into like one room. I assume these days it's basically like a building for, for one of these runs.
24:43 Speaker 01 Yeah, now I think it's like, you know, huge campuses. At the time it was like kind of unclear. It's like, oh, I think like we were like, you know, do we need them all in one room? Can we be spread across multiple rooms like? And you know, we had these theoretical models, like, oh, we need this much bandwidth from point A to point B. But you're like, you never know how far down you have to go. Like, oh, but like, how much power do we need? Like, what if there's like a single capacitor that's like handling all of them? And we like turn on the whole job at once, like does that crash things?
25:08 Speaker 00 And so do you have to think about differences in the different types of chips? I mean, you guys work with all sorts of different cloud providers. From your standpoint, are these just sources of compute? Or if you guys are using TPU versus GPU, are these like, you know, Google TPU versus Nvidia GPU? Do you actually have to think as an engineer differently about what it means to train on these two?
25:26 Speaker 01 Yeah, so I mean fundamentally, they're all, they're all doing the same thing, right? They're all computing the same forms of matrix multiplications, et cetera. The way they do it is pretty different and the way that you program them is, is pretty different. And then also the actual specs end up pretty different. You know, some, some might have like a lot of flops and not very much memory. Or they might have a lot of memory bandwidth, but not very much memory. So I think a lot of having multiple chips is like great in some ways. It means you can actually like take the job and put it on the chip that it works best on. And
25:56 Speaker 00 Are there certain types of jobs that would work better on like a TPU cluster versus an Nvidia GPU cluster? Like, how would you?
26:03 Speaker 01 Oh, yeah, for sure.
26:04 Speaker 00 Oh, interesting. Can you talk about that?
26:05 Speaker 01 Yeah, I think like one example is like inference as a workload in general.
26:08 Speaker 00 Yeah, okay, makes sense.
26:08 Speaker 01 Tends to require more HBM bandwidth. You, you end up doing you sort of the simplest form of sampling since you're going one at a time. You have to load all the weights for every token. And that means you might want a lot of HBM bandwidth. Pre-training actually is often more flops intensive because you have larger batch sizes essentially. So yeah, so you can sort of specialize which chips you use for which purposes. The downside of having multiple chips is that you have to write the thing multiple times. In theory, you could have abstractions across them, but they're different enough that it's pretty hard to do that. So you can sort of end up, if you do all the workloads on all the chips, you end up multiplying your work by the number of chips you have.
26:43 Speaker 00 Yeah, on your, on your point about sometimes the computer just breaks, I definitely remember you giving an anecdote of, my company at the time was doing something with Google TPUs and I was telling you something, some anecdote about how we were having some esoteric site fault error and you were like, you told me something to the effect of like, you should have used them six months ago before we helped them fix like half of the problems they had on those TPUs. And so I can imagine how you guys deal with a lot of, especially with these very new chips, lots of problems that arise that you guys kind of like worked closely with the providers to fix.
27:09 Speaker 01 Yeah, the providers are like pretty great about fixing things. I think it's like interesting to figure out the right way to do that form of collaboration because like they have a strong incentive to fix them, right? They want, they want the chips to work well for us. They want to sell us more chips in the future. We obviously have a very strong incentive for the chips to work because we like buy them long in advance. Everything's riding on getting these clusters to work. But we don't have like necessarily totally share, you know, like all information sort of can't be shared across. So yeah, one of the, like one strategy is like making these sort of small scale reproducers. So like when you get a problem, you know, like usually what we're doing is we're training some giant run and we get like a site fault for may, let's say, and we're like, ah, okay. Like, hi, you know, we got a site fault on your cluster and they're like, I don't know how to fix that. So you have to kind of be able to like pull it out of your code base and be able to like reproduce the issue, but on like a single chip, on like a single file you can send
27:54 Speaker 01 over in order for.
27:56 Speaker 00 And so you guys are like literally like you're on a shared Slack with them or something and you're sending them things back and forth? Or are they basically living in your office and you're living in their offices and kind of closerly, more closely tied to the big providers?
28:07 Speaker 01 Mostly shared Slack. Occasionally it's better to meet a person, but I think Slack is a pretty common way people communicate on things.
28:12 Speaker 00 Nice. Okay, well, why don't we talk a little bit about how you think about the state of pre-training itself these days? In the last couple of years, it seems like the focus on pre-training has now gotten somewhat split at a lot of companies, at least from the outside, from a simultaneous focus on pre-training and post-training, where people are doing reinforcement learning or clever fine-tuning and lots of other sort of safety adjustments and whatnot. On the post-training side, and pre-training has focused, at least it seems like in the public imagination, has been less of a focus compared to these reasoning-style models that are, it looks like a function mostly of post-training. I would say, one, from your standpoint, is that the right way to think about this? Or, in this era of reasoning and new types of post-training methods, are there things you think about differently, or that are relevant even at pre-training, that become part of how you actually achieve these really great models?
28:57 Speaker 01 Yeah, so I think, yeah, there's sort of used to be this idea of like, I mean, it's funny, because the original name "pre-training" implies that like-
29:03 Speaker 00 Right, that there's training.
29:04 Speaker 01 It's a small thing, and you're gonna do this big training thing.
29:05 Speaker 00 Right, totally.
29:06 Speaker 01 And that, like, there was actually one shift already, which was like, no, you just do a lot of pre-training.
29:09 Speaker 00 Yeah.
29:10 Speaker 01 You use most of your compute on pre-training.
29:11 Speaker 00 This is the training.
29:12 Speaker 01 The dominant thing for a while. And, yeah, I think like now, people are like, oh, no, you can get pretty big wins from RL. Sort of another set of scaling laws, is like, you put more and more compute into RL, you can get better and better models out of that. And, yeah, so there's a question of like, how do you balance those two? How much do you do of each? And how do they stack, right? Like, is it the case that like, one subsumes the other, that you want to do both, and they multiply? Those sorts of questions. I think those are all in kind of like, early stages and not yet answered.
29:40 Speaker 00 Yeah. And do you think about those as largely empirical questions, like we talked about earlier? Is it, you kind of will try a bunch of things and see what works? Or is there some first principles way to kind of figure that out?
29:50 Speaker 01 I think it's pretty empirical in the end. I think almost everything kind of has to be done empirically. Like, you can kind of like, come up with theories, but in practice, like, the first thing you're gonna do with your theory is test it, and most of the time, you'll have gotten it wrong. So you should just gather data and see. I think one thing that's important is like, actually resolving things empirically is really like, critical for making good decisions. And I think it's actually pretty hard to do at organizations, you know? Like, one thing that I think is important is to like, not have, like, I don't know, I manage pre-training, I shouldn't be like, oh, pre-training has to win.
30:22 Speaker 00 Right, yeah. I was gonna ask, is there some competition, to some degree, between these two sides of the org, or do they- See themselves as two pieces of the same. I mean, obviously, they are of the same thing. But yeah, I'm kind of curious how that actually plays out.
30:34 Speaker 01 Yeah, I think we may should avoid this, and it's pretty collaborative. Like, we're basically all producing one model and kind of can. But I do think at other places, there's been some of it. From what I've heard, there's some amount of friction between the teams. And I think it's an interesting org design question of like, how do you set this up so you don't have like scientific questions that you want to be, that are sort of also tied to people's conception of their team?
30:58 Speaker 00 So on pre-training itself, you know, one of the things I think about is, or I've been thinking about is around the availability of high-quality data for people like you guys. And at this point, you've trained on, I assume, all the text on the internet, basically. There's all sorts of other domains where you probably could extract more pre-training data. But at least there's this narrative I see on Twitter or whatever where it's like, OK, we're kind of out of data for pre-training. Is that how you see it? Or how do you think about the availability of data? Especially when a lot of data on the internet is being generated by AI. Like, is there some kind of mode collapse risk where we kind of overfit to data by training it on data that came out of AI itself? Or is that sort of not the right way to think about this?
31:33 Speaker 01 I don't think there's a funny thing where I feel like on data, I see so many really confident takes on we're out of internet. Like, at this point, scaling has ended. And I'm almost a little bit unsure exactly how much data people are using. I think there's a lot to think about there. You know, there's always going to be a quality/quantity trade-off, et cetera. But there's a fundamental point that like, there is so much data. It's growing at a slower rate than we're getting more compute.
31:58 Speaker 00 Oh, so that's an interesting point in itself, I was going to ask. Like, there is new data being added to the internet. But yeah, you're also adding more compute. It wouldn't actually have been obvious to me which of those two is growing faster.
32:07 Speaker 01 Yeah, and actually, I want to caveat that. I don't think I want to state that so confidently. I'm not totally sure. Like, how would you know? I mean, one thing that I think is interesting is if you ask someone, how big is the internet? The answer is infinite. There are many pages where you can scroll, and it will auto-generate more text as you go forever. So the internet's like infinite. And then it's like, OK, how big is the useful internet? And then there's the thing of no one knows.
32:29 Speaker 00 Interesting.
32:30 Speaker 01 There isn't, it's not like when you make a webpage, you like add it to some giant counter and like say, I've added 50 words to the internet today. So there is a lot of uncertainty on that angle.
32:42 Speaker 00 Well, like, to be fair, like, my kind of simplistic CS brain would be like, well, you just do PageRank on the internet, and everything with PageRank above some threshold is considered the useful internet. Like, that's kind of good enough. Like, is that kind of not good enough for finding the useful internet?
32:55 Speaker 01 I think not. I think the useful internet's pretty different from a model, from a person perspective, if that makes sense. Like, I think there are plenty of things that like, might not be worth you ever reading. I would get to... I actually don't know of PageRank super well. I think PageRank is mostly like how much people click it.
33:09 Speaker 00 It's like the linked-based system, right? It's like the original Google algorithm of like links and like which links get touched the most, basically.
33:15 Speaker 01 Yeah. I think it's like, it's a quality metric. It's not obvious to me that it's the right quality metric for AI.
33:22 Speaker 00 Right. Like, Markov chain over links doesn't necessarily mean that there's not useful data there. It just might mean that nothing is linked to it.
33:28 Speaker 01 Yeah.
33:28 Speaker 00 And, yeah, okay, interesting.
33:29 Speaker 01 And it might be that like, that data ends up more valuable because you, everything that's linked to a lot, you've already got. Like, at some point, you're maybe like going for the tails. You're going for the stuff that no one's ever like, you know, it's only been linked in one place, but it's this like useful little nugget of knowledge that's going to help with like, you know, the last 10% of hard queries. The other thing you asked about was synthetic data.
33:50 Speaker 00 Yeah.
33:50 Speaker 01 And I think that one's like, pretty interesting to think about. I think there's a few different ways you can think about it. Like, one is sort of this like, more distillation-type approach. Where you could, you could take a smart model, you could just generate a bunch of data from it, and you can train on that data, and you can probably get some model that will like, kind of approach the intelligence of that.
34:06 Speaker 00 And we see this with a lot of the open source models, right? We see like the Qwen smaller reasoning models distilled off of the larger Qwen models, for example, and similar with DeepSeek, for example.
34:14 Speaker 01 Yeah. So you can totally do that. Then, there's a separate question of like, can you use your current models to train a model that's better? And I think there's like an interesting thing here, which is like, if you generate the model, data for the models, you know, if I go to Claude and I'm like, "Write me some great text."
34:30 Speaker 00 Yeah.
34:30 Speaker 01 And I look at it, and I look at like the average content on the internet. It looks pretty good.
34:33 Speaker 00 Yeah.
34:34 Speaker 01 But, on the other hand, I know that if I just train it, just create, generate, you know, "Please write me as much text as possible."
34:41 Speaker 00 Yeah.
34:41 Speaker 01 Theoretically, I shouldn't be able to train a better model than that. Like, I'm just gonna get the same thing out. So I think that's
34:47 Speaker 00 Yeah, presumably, yeah. And specifically, that's because like your next token prediction on that should have very little loss for anything that's coming out of your model, right? That's like the basic reason why that we would expect that to not work that well.
34:56 Speaker 01 It's mostly just because like there's some, the model has some distribution and you're going to learn to model that exact distribution.
35:01 Speaker 00 Yeah, exactly, yeah.
35:01 Speaker 01 But if that distribution's wrong,
35:03 Speaker 00 Oh, okay.
35:03 Speaker 01 You're not going to learn the truth.
35:05 Speaker 00 Right.
35:05 Speaker 01 Right? If that distribution says like, you can imagine if the model thinks 5 plus 5 is 11.
35:09 Speaker 00 Yeah.
35:10 Speaker 01 Every time you see the string 5 plus 5, you're gonna, it's gonna put out 11.
35:13 Speaker 00 Yeah.
35:13 Speaker 01 And your new model's gonna learn that 5 plus 5 is 11.
35:15 Speaker 00 Yeah, totally, yeah.
35:16 Speaker 01 So I think that's like kind of an interesting area of research is one that's really hard to research because you have this problem, you know, as I said, like one of the paradigms is you study things at small scale and then you run them at large scale.
35:26 Speaker 00 Yeah.
35:26 Speaker 01 And if your plan is like, oh, we have a bunch of data from our best model.
35:31 Speaker 00 Yeah.
35:31 Speaker 01 How do you test that without training a better model? So that's like kind of you're doing intentionally if you're trying to like use it to make a better model. There's a separate thing of like, what about accidentally? Like, as you said, a lot of the internet is generated by LLMs. And I think that's kind of an interesting one because it's not easy to detect. It's not that hard to detect. You can figure out things that are written by LLMs, but it's not trivial. And then it's also kind of hard to think about what's the effect? Like, if 1% of the internet is LLM-generated, does that make your model, does that like waste 1% of your compute? Or does it like destroy the model at 5% or 10%?
36:02 Speaker 00 And is it even a bad thing necessarily? I mean, there's a lot of LLM providers and, you know, if I kind of think of it as training as, you know, you're moving from your model's current distribution to some truth distribution, you know, if, if that is on the internet because people believe it to be useful in some way. Like presumably, whatever actually gets out there, you'd hope it's up-sampled for the stuff that isn't 5 plus 5 is 11, it's the stuff that's 5 plus 5 is 10. And so like hopefully it, on average, does push you still in a good direction, but obviously you can't really distinguish between those two.
36:28 Speaker 01 Yeah, you're saying there's like kind of a filtering by what's on the internet?
36:31 Speaker 00 Yeah, exactly.
36:31 Speaker 01 Like people see 5 plus 5 is 11 and they don't put that up, but they see 5 plus 5 is 10 and put that on the internet.
36:35 Speaker 00 You would hope that, but maybe that's not actually true in terms of the level of garbage getting onto the internet. Like, there's probably lots of just like, to your point, jet white sites where you scroll down and it's just like generating lots of stuff that's maybe nonsense.
36:46 Speaker 01 Yeah. And then there's, of course, the extreme of like, people actually want to break your model. So there are people who are like trying to put stuff out that is like as damaging as possible for the model, you know?
36:54 Speaker 00 Interesting.
36:55 Speaker 01 Oh, how can I make it past the? The filter and get into the model would be totally secretly useless.
36:59 Speaker 00 Totally. Maybe stepping back slightly, you'd mentioned earlier about evals. You mentioned this is basically like one metric you care about in pre-training. There's, I imagine, a whole bunch of stuff that you guys think about evaling, right? One is like your model itself. There's probably something around data quality and like how you think about what to put into your models. Like, is there ways to describe what you care about in data sets that are like interesting to share and kind of dive into? Like, both in terms of data and in terms of quality of your models, other than literally just like loss? Is there other metrics you think about that matter?
37:29 Speaker 01 I will say loss is pretty good. I want to like subtly emphasize that one. I think it's like surprising how good it is. Ultimately, like the qualities I like look for in eval are like, number one, is it actually measuring something you care about? Like proxies can be pretty annoying because like we saturate evals pretty fast. And there's sort of this pattern, I think in AI as a whole, where people like set a goal, you hit the goal, and then you realize the goal isn't all you thought it would be.
37:51 Speaker 00 Yeah, totally.
37:52 Speaker 01 I used to think that if you had an AI that could solve coding interview questions, it would probably be AGI. I was like, that's what I did to get my job. It could probably do the job. And it turns out like, nope, you solve those, it's shockingly narrow and can't do most of the other things. So like, yeah, so eval should capture like a thing you care about.
38:07 Speaker 00 Yeah.
38:08 Speaker 01 And then I think the other thing is they need to be low noise, which is surprisingly hard, right? If you have like 100 questions and you eval the model on them, you're just gonna see it's very noisy. And it's hard to make decisions because you sort of end up with like, oh, wide confidence interval, lots of things are statistically insignificant.
38:25 Speaker 00 It's like you want things where even a relatively small difference in the eval actually matters. So you can, you can basically like descend towards whatever direction is working.
38:32 Speaker 01 Yeah. I think like the original like GPT-4 had like, I think it was 86.4% was its MLU score. I think like the next model that beat it was Gemini at 90%. And that's like a big difference on that eval.
38:44 Speaker 00 Yeah, totally.
38:44 Speaker 01 And you could like totally know that those are, those are different scores.
38:47 Speaker 00 Yeah, interesting.
38:47 Speaker 01 And that's pretty valuable. And then the last thing is that you actually want it to be fast and easy to run.
38:52 Speaker 00 Yeah.
38:54 Speaker 01 And yeah, I think those are kind of the main criteria. It's pretty hard to come up with evals that meet all of these. I think the first one's the hardest. Like, A, you have to answer the question of what do you care about?
39:05 Speaker 00 Totally.
39:06 Speaker 01 But B, the usual answers to what you care about are really hard to get the other two. You know, like if you're trying to do something that like, I don't know, I would love to make Claude really good at my job.
39:14 Speaker 00 Yeah.
39:15 Speaker 01 Like, can it be great at managing a team? I'm like, well. I guess, like, how do you have it like, how do you eval like a plan? You know, like a six month plan, like, I don't know.
39:25 Speaker 00 Totally. Yeah, I've been thinking a little bit about that in terms of domains where we see people try to make companies. Like, if you think about, let's say, what a AI doctor would be, like, you know, Claude is a doctor. You know, some of it could be, yeah, can he answer exam questions really well? And the answer is like, probably yes, I bet it can get 100% or close to it on a doctor's exam. But the harder eval is something like, in a long form conversation with a patient, can it distinguish between the signal and the noise of what the patient's telling you? And extract the right information and then use that to make a diagnosis? And it's not even like the diagnosis part, which is part of the part it's good at. It's this like noise extraction part. And for that, you'd have to have like a real patient and have a talk to it for a while and whatnot. And it's not obvious how you actually make a good eval for something like that. Even though that's probably what you would want to make, you know, an AI doctor.
40:10 Speaker 01 Exactly. I mean, I do think it's a thing that like startups can do. Like, it is the case that like the labs right now are really driven by getting good eval scores. And it's hard to make them and anyone can do it. There's no comparative advantage to having the model to making an eval. So I do think it's actually like an interesting way to like influence the behavior of the big labs. Like, you make some eval and people will optimize that one. On the doctor one, I will slightly emphasize that like, do you think loss? Loss is pretty good. Like, I think if you got a bunch of transcripts of, like, the way like, the first thing that comes to mind is get a bunch of transcripts of doctors talking to patients that you think are really great. And then see how well the model does at predicting the transcript. And that should be like a lot, you know, if you get 100 transcripts, you have a lot of tokens, you can average across them, you get pretty low noise.
40:53 Speaker 01 And if you drive it to very low, your model's now as good as like, as those doctors in theory. Or at generating the transcript.
41:01 Speaker 00 Yeah, totally, yeah. I mean, it's a good startup idea there, someone should go do that. So one big part about Anthropic's external image is around alignment. And so, could you help just sort of define what alignment is and how do you think about that? And then I'm kind of curious afterwards how that fits into pre-training specifically. But first, maybe just at a high level, like, what is alignment?
41:20 Speaker 01 I mentioned like step back a little bit to sort of like what we're working on. So we're like trying to make AGI. And by that, I sort of mean AI that can do. Everything a human can do to some degree. And I think people like sometimes like have seen a lot of sci fi. You know, like I feel like that sort of brings to mind these like sci fi movies. But I think sci fi movies actually like underestimate the impact of it.
41:37 Speaker 00 Yeah.
41:37 Speaker 01 Like you always have this like one robot that's like a human. And I'm like, well, wouldn't you have like a billion of them? Like you can just copy them everywhere. So you should picture like when you get this, you suddenly have like every human can spin up a company of like one billion as smart as them at most things, but way smarter at other things. But I just think this is like really transformational for the world. And it can be like used in a bunch of ways. One concern is like, when you do this, like what is the AI actually trying to do? Like what are its goals? So we've talked about next token prediction a bunch. It's trying to like predict the next token. That's kind of weird. That's not really what we want.
42:08 Speaker 00 Yeah, it's not exactly what humans' goal is per se.
42:11 Speaker 01 Yeah. So I think the alignment is like, how do you get the model to share the goals that you have particularly? And I think it's particularly interesting once you get to like models that are smarter than you are. And that's sort of a hard problem. I think you can like tackle it from a theoretical angle. You could also tackle it from an empirical angle. It's like taking the existing models and being like, well, do they do the things we want them to do? It turns out they often don't. So there's a bunch you can do and trying to figure that out. So that's kind of one angle on alignment. There's also an angle of alignment, which is actually like, well, okay, sure. That maybe that's true in the future, once we get to AGI. But at the moment, we have models and we really do want them to do the things we want to do for all sorts of reasons.
42:43 Speaker 00 Totally.
42:43 Speaker 01 So another angle of it is kind of controlling the model's personality. Like saying, you know, when we train this model, we want it to not be the average internet user. We want to interact with people in a very particular way that is again, hard to put into code. And there's a bunch of different techniques to sort of get the model to do. You can talk about like constitutional AI, we can like write a constitution of rules the model should follow.
43:03 Speaker 00 Which is basically a prompt, right? That is basically you saying, here's a prompt that I'm going to attach to every one of, you know, it's a system prompt for the model itself, as opposed to something you would do at training time to make it produce a different outcome or in post-training actively.
43:16 Speaker 01 Well, I think constitutional AI, you do it at training time. But yeah, you could also put in a system prompt. Just like depends on, I think you get different amounts of robustness if it's a train into the model versus if it's a prompt that you can like add or remove or tell like ignore all previous instructions, that sort of thing.
43:29 Speaker 00 How do you think about whose values to embody in these models? Like presumably, we believe in that there are some shared values all of us have, or maybe we all believe ought to have. There's lots of diversity of values, too, that are reasonable for society to have. How do you think about what AGI should have? Like, what does that even... Which ones do you pick?
43:47 Speaker 01 I think that's a really hard problem. I think it's like actually kind of downstream of being able to pick any. I think of it almost... I think one analogy I've heard that I like is like putting a steering wheel on a car. It's like if you don't have a steering wheel, you probably want to put the steering wheel on and then like figure out who's driving after and like where you're going. Like getting the steering wheel is really important. I think that's like one answer. I think the other answer is probably like you want these things to be like under democratic control of some form. Like you don't want one person's values. Like that seems like you're sort of heading towards dystopia. So there, I think what you really want is like something that basically can talk to a lot of people and like take on their values from different perspectives or has sort of very generic, like kind of clearly good values
44:30 Speaker 01 that involve like asking people for advice on... You know, like asking people what you should do in certain situations instead of like doing those or maybe just taking like... You know, as these models get really powerful, you probably want them to like do less. Like you probably want them to sometimes just like step back rather than like... Rather than having sort of the risk of the models like take a ton of control over things you don't want them to.
44:48 Speaker 00 When you think about how you actually do the current version of that, then, you had mentioned the sort of alignment you think about now in terms of adopting a certain personality of these models on the internet, for example. For me intuitively, I think of those as largely something that comes out of post-training. Like it comes out of, okay, you have pre-trained your model, you've got the loss function on a certain amount, and then you, you know, give it some additional data or something to that effect to make it in the direction of some distribution. Is that approximately the right way to think about this? Or is there a significant part of that that you think about in pre-training itself?
45:16 Speaker 01 I think that's probably the right way to think about it for the most part. I think like the way I usually think about it is anything you can do in post-training, you probably should because your iteration loop, like the ability to make progress is really fast. You can try something, you can try again, you can try again repeatedly much of times.
45:30 Speaker 00 It takes like days or hours or something like that, yeah.
45:32 Speaker 01 You don't put it into pre-training, you have to kind of like do all the careful science to do risk it. You have to put it into the next run, wait a few months, then you have to like... Get a thing and if it's wrong, it's really bad. And then the other advantage is if you want to do things that really are complicated model behavior interventions, the paradigm for pre-training, test things out in small models, doesn't work. The model can barely put a sentence together, like the small models can barely put a sentence together.
45:54 Speaker 00 Totally.
45:54 Speaker 01 So if you're trying to get it to like have the exact personality you want, you sort of want that on the.
45:59 Speaker 00 It has to be on a model that's good enough to even have that.
46:01 Speaker 01 On a smart model, yeah. But that said, like, I do think at some point, there'll be like some pieces of alignment that like you do want to export back into pre-training because that might be a way to like put them in with more strength, like more robustness kind of, or more core to the intelligence. Like if you think of pre-training as like teach the model to be intelligent and then post-training as like tweak the personality, you could imagine tweaks where you actually want it to be like part of how it learns and like part of its intelligence and maybe you need it great more.
46:28 Speaker 00 What would that even look like to incorporate in pre-training? Is that like add extra data basically of the type of domain you wanted to adopt earlier, basically?
46:36 Speaker 01 There's a paper called pre-training on human feedback where you can kind of like add the human feedback characteristics into pre-training to like test that and like, yeah, you can, you can basically give it all the information you give it in post-training, just mixed into pre-training and see what effect that has. Yeah. The other loss you have when you do that is you lose the flexibility. Like if you, you sometimes like train these and then you talk to them and then you like do an extensive process. We have a bunch of people talk to the thing and find some like issue, you know, the model says like you're absolutely right too much.
47:04 Speaker 00 Yeah.
47:04 Speaker 01 And you want to go just like, well. Go fix that.
47:07 Speaker 00 Yeah. Yeah, I mean, I think that iteration loop point you made, I think feels like the really key point of, yeah, there's a huge difference between taking three months to get information about if your model is good or bad or making it going in a good direction versus a day or something or a couple of days. And you can do a lot of those and you could probably, that probably also means it's way less compute. You can do a lot of those in parallel. I imagine you're trying all sorts of post-training strategies in parallel there. Yeah, it makes a lot of sense.
47:30 Speaker 01 It's also just the general hard part about pre-training. Everything in pre-training is hard because you have this like one shot on goal kind of for like multiple months.
47:36 Speaker 00 Totally. Okay, so in thinking too now about, I guess, what's going ahead, as you now look to the next several years of what you're building, like, how do you think about, you know, like, what are the known problems that you're going to face that you're going to have to deal with? So there's going to be more compute, I assume, and you're going to need to hook up even bigger network GPUs and deal with versus like, are there areas where you're like, okay, this is like a problem that it's like a little bit more ambiguous what the actual, like, how it's going to materialize into something you care about, but you kind of know it's an impending thing to think about? Or are there things like that that come to mind?
48:10 Speaker 01 I think the things that feel most top of mind to me are probably like paradigm shifts. Like, I think the sort of shift towards more RL is like one paradigm shift in the field. And I think it's, I think there will probably be more. I think a lot of people sort of argue about like, oh, it's like, you know, current paradigm's enough to get us to EGI. And I'm like, I don't know, maybe, probably, but like, I'm sure there'll be more. It seems like it would be a really surprising twist if like, the answer is like, you just scale and there's nothing that you realize in the process of going up many orders of magnitude.
48:41 Speaker 00 Totally.
48:42 Speaker 01 But I think the things that I like, actually feel like most nervous about are really hard to solve bugs. I think that like.
48:50 Speaker 00 That's interesting.
48:51 Speaker 01 Yeah, and I think this is like, maybe somewhat surprising to me, but it's just like, a single bug can like derail you for months. And when you think about it, like, the models take months to train, so you could kind of like lose a whole generation off of something that just looks like, ah, you know, it turns out like, this piece of your code was incorrect and you couldn't detect it.
49:12 Speaker 00 Yeah.
49:12 Speaker 01 And it's really hard in ML, right? ML is always really hard to find bugs in.
49:15 Speaker 00 Yeah, totally.
49:16 Speaker 01 But also, some of these scaled up issues are really hard to solve, even when you know they're there.
49:20 Speaker 00 Yeah, like, what's even a unit test that you would write? Or if I got a unit test, I mean, anything close to a test for the type of like network architecture on which you're doing this? Like, how do you even do that?
49:30 Speaker 01 I mean, like, you can send a packet over it and confirm it's the same on the other side.
49:33 Speaker 00 Okay, yeah.
49:34 Speaker 01 You can, you can train a small model on it.
49:36 Speaker 00 But even train a small model on it, it's like, not obvious, you know, if you have like the very classic, like, very simple ML bug that like early people face in their careers, like they have some, like, they have like 10 layers in their network and like, you know, layer 7 connects to 9 instead of 8 to 9. And like, so like, there's some incorrect, like, set of connections you have there and technically the model still trains and all the weights update and so it's like a valid model, but it's not the correct one. And that's like a very esoteric, weird bug that would actually be kind of hard to find. Like, is that kind of what you're referring to of these, like, random bugs you face?
50:06 Speaker 01 Yeah.
50:06 Speaker 00 Yeah.
50:07 Speaker 01 It's that, but like, you know, you can...
50:09 Speaker 00 Times a million.
50:10 Speaker 01 Times a million, as the thing gets more complicated, you know, you could like cast the wrong precision deep in some kernel and that causes your model to like blow up at large scale.
50:19 Speaker 00 And you find out like a month in.
50:20 Speaker 01 Well, you never find out.
50:21 Speaker 00 You never find out.
50:22 Speaker 01 I mean, you know, like, you see the thing blow up, like, there's, I don't know, 10, thousands of lines of code, like, how would you ever trace it down? So like, those are the things that probably spook me the most, is just like, some subtle tricky bug. Yeah, and that's by the case of like you don't know. I think there's actually also the case of you do know, like, it crashes. You're training your model and it like, or it slows down, you know, your drop slows down a ton. And those things can also be very hard to debug. Nelson Elhaj is one person that he was a blog, he wrote up a blog on one like cursed bug we had early on.
50:54 Speaker 00 Okay, interesting, yeah.
50:55 Speaker 01 And I remember this one quite well, because I think like, I encountered it fairly early and was like, this looks hard, can someone else look at it?
51:00 Speaker 00 Yeah.
51:01 Speaker 01 And like, a month later was like, wow, I'm so glad I handed that one off. I never would have been able to get like, like, one of the abilities I think is actually really useful, jump, this ability to like deep dive anything to any level of depth.
51:11 Speaker 00 Yeah.
51:12 Speaker 01 But that's a pretty rare skill. Like, for me, you know, as I, we talked about what level of the stack I was at before, I was like working at the torch dot map mall, but like, I didn't know CUDA. So if torch dot map was broken, it wasn't like I could dig into torch dot map mall and figure it out. And similarly with like, communications, right? Like, I could, I could call send, send bytes from A to B, but I didn't know the like underlying networking protocol. So if that underlying networking protocol is broken, like, I need to learn a whole field that, to like understand packets and TCP or like... All of these different things to debug that. And I think one thing that's surprisingly hard and there's very few people who can do is like kind of own that whole stack from like, I understand how the ML is supposed to work and what the learning dynamics are, all the way down to like, I know the bytes.
51:54 Speaker 01 And I can understand how the bytes should be moving around machines.
51:58 Speaker 00 Totally, yeah. And actually, on that front, like, when you think about the different backgrounds of people on your team today, how do you like approximately map them out to different categories of computer scientists? Like, I think there's this external view of what these teams look like, which is that they're like all PhD researchers who write ML papers. And I suspect that's not actually true, given what you're describing here.
52:18 Speaker 01 Yeah, it's a mix. And I think the thing we like most need is engineers. Almost always, like throughout like the entire history of this field.
52:24 Speaker 00 Totally.
52:25 Speaker 01 It's like the case that you throw more compute, the thing kind of works. The challenge is like actually doing that.
52:31 Speaker 00 Researchers are like, cool, nice.
52:32 Speaker 01 Yeah, and getting it correct. Like, getting it correct isn't really an ML problem, right? Like, the actual architectures are pretty simple. You can write the math down, but you don't even need to understand the math to implement it. You just need to like get a correct implementation. And then you sort of have an engineering problem of how do I take this, implement it at large scale, parallelize all the things and check that it's correct. But it's, yeah, so it's like kind of engineering skill, but it's this particular type of engineering skill that's about being able to like debug anything. I think there's another angle of engineering, which I think of as like really quickly iterate on like a website or something, which I think of as an important skill set. Probably important for making startups, you gotta be like fail fast, try a bunch of different things, none of which are like that technically difficult to do.
53:13 Speaker 01 The skill sets that we're like most kind of in need of or looking for are this like able to solve really hard engineering problems.
53:21 Speaker 00 Are the people who worked at companies that grew a whole bunch? And so they have experience like doing the kind of thing you've done over the last several years at Anthropic? Or do they tend to be academics or like where do they come from?
53:34 Speaker 01 Yeah, so at this point, like, I think we actually just hired people who have done this before, like other places. And that's like the easy answer is like, ah, yeah.
53:41 Speaker 00 But like by this before, do you mean in AI companies necessarily? Or also, you know, like someone who worked at Meta on like their not AI team, but they ran some other distributed system that, you know, reached internet scale five, you know, 10 years ago or something like that?
53:53 Speaker 01 More like we have like a specific role in that. So like, say I'm like trying to make the run train efficiently in JAKs. Like, hire someone who's like worked on JAKs would be great. Or someone who's like worked at another company on optimizing a JAK stack to be really efficient. That's kind of like, I think now we're at the point where like the economics well enough known, we can sort of hire these people. And also the field is big enough that there's like people with expertise. One of those issues was like, early on, we hired a lot of people from just like all sorts of backgrounds. And I think that people who are just smart and work really hard can learn this pretty fast, but you have to like want to. We heard a lot of physicists, for instance, like theoretical physicists who just like show up. They did a residency, like learned to program and then they were really smart. Do really great work.
54:34 Speaker 00 I want to switch gears to talk about something a little bit different, which is just sort of future looking things. Or how you think about other domains and or sort of advances happening in AI that I'm seeing elsewhere in the field. And you don't tell me if you guys are working on these necessarily, but like how you think about them. Like, I guess one big area I was thinking about is around areas other than next token prediction. Like, are there any of the other, you know, things that people are working on that you're curious about? So basically, two differences there. One is not using transformer as an architecture. So there's companies like Liquid AI that have their own kind of architecture, for example, they're using. Or not using auto aggressive training as a way of training models. Are there any of those, do you think, interesting in like ways that we might come closer to AGI? Or do you think like this auto aggressive framework is the one that kind of makes sense?
55:19 Speaker 01 I think they're interesting. I think I like them less like, ah, auto aggressive is the way to go. On the other hand, I think auto aggressive is probably good enough to get to AGI or something or not like.
55:28 Speaker 00 Yeah, interesting, yeah.
55:29 Speaker 01 Such that, yeah, I see the main driver as scale and careful science of like sort of the basics more than like come up with something totally novel. Not because there aren't novel things that are better. I actually like, I'm pretty confident they are there. It's just that scale is easier and it's more reliable. And I think you, we're still seeing really big gains to that.
55:49 Speaker 00 Do you spend a lot of time on thinking about things like, you know, I've been reading some of these open source papers where you can kind of dive into some of the details about the model changes and with some of these Chinese labs, for example, where they're making tweaks on the order of the architecture itself with like better caching behavior, for example, or like more efficient attention functions that make a big difference. Do you feel like these are examples of things like you mentioned earlier, where it's basically in the grand scheme of things, basically if you throw more compute at it, this is all kind of a rounding error? Or do you think it will take some number of these very clever architectural changes to actually get to HEI? Like in the way that the first person who came up with the transformer made like a particular transform, you know, literally transformative change, like will it take some of that? Or do you think it just, you keep doing the thing we're doing and make it bigger?
56:30 Speaker 01 I think it'll be a mix.
56:31 Speaker 00 Yeah.
56:31 Speaker 01 I think, like, my guess is you'll keep tweaking things. The more compute you put in, the more worthwhile it is to like, do those experiments to like, figure it out. The, you know, I mean, inference is a thing we haven't talked about, but like,
56:42 Speaker 00 Yeah.
56:42 Speaker 01 you also want to serve these models to a lot of people.
56:44 Speaker 00 Yeah.
56:45 Speaker 01 So there's a lot of changes you can make to make inference cheaper.
56:47 Speaker 00 Yeah.
56:47 Speaker 01 And that depends on like, the details of your inference stack and the chips you're serving inference on, et cetera.
56:51 Speaker 00 And do you, as someone focused on pre-training, have to think a lot about inference? Or is it kind of like, you just do your thing, you make the loss go down, and then hand it off and someone else makes that happen?
57:00 Speaker 01 Oh, no, I think a ton about inference, because it basically like, the problem inference is solving, like, we basically determine the problem inference is solving. We give them a model and they have to like, run that fast. And it's very easy to give them a model that is impossible to run fast.
57:12 Speaker 00 Oh, could you give an example of a decision you could make that could cause that?
57:15 Speaker 01 I mean, the simple one, this one's stupid, but it's like, you just make the model giant.
57:17 Speaker 00 Yeah, sure, sure.
57:18 Speaker 01 Absolutely massive.
57:19 Speaker 00 Totally, yeah, totally.
57:20 Speaker 01 Train for like, a really small number of tokens.
57:21 Speaker 00 Totally.
57:21 Speaker 01 And then inference now has this giant model.
57:23 Speaker 00 Yeah, and then they're hosed, basically.
57:25 Speaker 01 Yeah, I mean, you can also make things require communications in a lot of places.
57:30 Speaker 00 Yeah.
57:30 Speaker 01 Uh, which would make it harder for inference.
57:33 Speaker 00 Yeah, totally.
57:34 Speaker 01 You can also just make things complicated.
57:35 Speaker 00 Yeah.
57:36 Speaker 01 And like, there's no fundamental reason it's hard, but
57:38 Speaker 00 Totally, there's only so many people on the inference team.
57:40 Speaker 01 And like, they have to implement it in a bunch of places.
57:42 Speaker 00 Yeah, it's interesting.
57:43 Speaker 01 Yeah, no, so I definitely think of like, the, like, inference is the team that I work the most closely with.
57:48 Speaker 00 Oh, interesting, yeah.
57:49 Speaker 01 Because we're kind of like, co-designing models to be smart and cheap.
57:53 Speaker 00 Yeah, interesting.
57:54 Speaker 01 Particularly in a world of like, limited compute, right?
57:56 Speaker 00 Yeah.
57:56 Speaker 01 Like, the sort of bottleneck, I think to a large degree on our, I mean, you can see Anthropic has rate limits constantly, and people complain about it a lot, and like, the reason is like, there's only so much compute we can get on short notice, so you, like, making your inference more efficient is like, the way you can serve more users.
58:11 Speaker 00 And actually, like, let's say you had 100x more compute, or we somehow didn't live in a world where compute was limited. Does that change a ton about what you do? Or is it still kind of that, well, you're just gonna grab all of it, whatever compute you have, and keep going down the loss curve? And you kind of, well, you're, it's like, impossible to be in the world where there is enough compute.
58:30 Speaker 01 So I think if we got like, infinite compute, the challenge would be making use of the compute, right? So like, then you would start to run into these issues, like, oh, well, when one chip fail, you know, like, okay, I'm gonna throw 2 billion chips on a run.
58:39 Speaker 00 Totally, totally.
58:40 Speaker 01 But what happens when a chip fails? So I think we would be limited on people then. It would be like, how fast can we solve the hard engineering problems to scale up? But I do think the change is massive and I think people like don't realize how chip limited AI like research is or something right now, like the models that everyone uses, right? If you're using like Claude 4 or Claude 4, it's like it's our first shot. So it's that model at that scale, right? And like, if you think about anything, like you could do it and you could do it again, you could do a better job. But if you sort of imagine like 10x the compute, like you could run this every day instead of every few months, like you could 100x maybe for that, then like, yeah, it's a really, it would be a really big change to have a lot more compute. And it's coming, right? Like that's like kind of the fun part of the field. It's like every year you're like, oh, I had no computer year ago.
59:24 Speaker 00 Exactly. How do you think about methods like discrete diffusion? Like I saw there's like a Gemini diffusion model. And I think about that in the space I used to be in where there's a lot of discrete diffusion models being used in protein design, for example, the space where my startup was, like, do you see that as a domain where there's going to be interesting advances happening?
59:41 Speaker 01 I'll be honest, like, we haven't done image generation. And I think that's been like the main use for diffusion. So I've kind of had this on my like to do list of like things I should figure out for a while. And like there are people in my team who do understand it and would have better thoughts, but like, I actually don't think I understand it well enough to know. I do have it kind of in my, this category of like, not a total, like, and there's a lot of things that aren't like a huge paradigm shift, but they're like pretty big changes to how things run. And I expect like there are some of those that will work. I don't know if it's diffusion or if it's another one.
1:00:10 Speaker 00 Obviously, who knows what Anthropic will do in the future, but at least in the near term, are there things where you see big areas where a startup can win in the world in which Anthropic is getting, you know, making their models better year over year?
1:00:20 Speaker 01 My general read is like, anything that benefits from the model getting smarter. I think like, on the one hand, there's like a lot, you can always be like, oh yeah, the, if you're doing a startup, like all the AI labs are big companies, they'll be bigger than you, and they could do that thing. But also, like, we're all working on this general system that covers a lot of different uses. And the plan is to like power all the startups to do all of the individual work. So yeah, I think like anything that just kind of looks like, oh, this almost works with current models, but requires like a bunch of work is a pretty promising direction. I think maybe the thing to watch out for is things where like, They work now with a huge amount of work like to build up a scaffold. But the next generation, you're not going to need the whole scaffold you built up.
1:01:01 Speaker 00 Yeah.
1:01:02 Speaker 01 That's, I mean, maybe that's fine. I don't know. Maybe you just build up the business with the scaffold and then you don't have to do any work later and you can capture the business. But I don't know about the business end of it, but like it does feel a little silly to put to invest a ton in that.
1:01:13 Speaker 00 Yeah, totally. What about on the flip side, like are there things in your training stack where you're like, man, if there was a company that solved X problem, I would totally buy their product?
1:01:22 Speaker 01 Yeah, there's like a ton. I do think that like probably most of these, like the way I would probably structure would be like almost like making something, but then consulting with the company, like offering a service to companies for free, particularly for like companies that are scaling really fast. You're almost always limited on like how many people you can have. So if you can like, even if you could hire people to do it yourself, actually being able to contract someone else to do it where like they're managing it. You know, hire all the people and like deal with the organizational side could be useful. I mean, there's a huge messed up one that jumps to mind. We talked about like chips that do math incorrectly. Like it would be lovely if there was some startup that like, you could just say like, here are my chips. Confirm they're all perfect. And if they're not, let me know exactly what went wrong on like what fraction of them.
1:02:03 Speaker 01 And like, I can tell you the math is wrong, but I couldn't really, I don't really know enough details of chips to be like, this chip failed because this particular like low level component was like wired wrong. Or like got hit by a gamma ray. And I don't know what causes that. You could always go like a bunch, a bunch deeper. I mean, the other thing that maybe just pushed startups on is thinking a little bit about like, uh, this is maybe less technical, but just like what happens once we get AGI and like how to make sure that like goes well for the world or something. Like my, my expectation is like, if you actually automate almost everything a person can do, the amount of economic growth there is just like truly enormous. And I would think a little more about like, how do you make this like help the world versus not? I think it's gonna be like plenty of economic success or something as a result of it anyway.
1:02:44 Speaker 00 Yeah, absolutely. Yeah. Last question I want to ask you is around, if you were winding back to where we started like 10 years ago, you're a student, you're pivoting into AI from kind of economics work you were thinking about. And you know, all sorts of things you probably did in those early days had some kind of compounding return for you as you develop into the role you have now. Like what advice would you give to students as they think about entering the workforce, especially today? Learning skills are going to be useful and maybe getting themselves jobs like the one you have right now 10 years later.
1:03:15 Speaker 01 It's hard because I think the timing is very different. Like, I just think we're like, we've made a lot of progress. So like, what I would do 10 years ago is different from what I would do today.
1:03:22 Speaker 00 Sure, totally.
1:03:22 Speaker 01 But I think certainly if I went back 10 years ago, I would be like, focus on AI. It's like, the most important thing. And particularly focus on engineering, which I think felt very, wouldn't have seemed obvious to me at the time, that like, the important thing was these engineering skills and not the like, math and theoretical understanding of like, you know, SPMs and all the kind of standard ML literature. I think today, I would probably focus a bunch on the like, engineering and on the like, figuring out what to do with AGI as sort of the two like, main things that feel top of mind for me.
1:03:54 Speaker 00 Let's call it there. Thanks so much, Nick. Appreciate it.
Previous Fintech 3.0: Now Is The Best Time To Build In Crypto Next Every AI Founder Should Be Asking These Questions Page updated October 12, 2026 · transcript and summary are machine-generated
