Ep 20: Anthropic CEO Dario Amodei on the Future of AGI, Leading Anthropic, and AI Doom Chances

Special Crossover Episode: We're excited to share this conversation from "The Logan Bartlett Show," another Redpoint podcast that focuses on untold stories from tech's inner circle. In the episode, Logan Bartlett interviews Dario Amodei (CEO, Anthropic) on the future of AI. In the episode, Dario gives detailed predictions on the AI industry for 2024, 2025, and beyond. He discusses his days at OpenAI, leaving to start Anthropic, why he doesn’t like the term AGI, what AI developments he’s most excited about right now, and much more. Sharing here because we believe our AI enthusiasts will find this episode particularly enlightening. Enjoy!

• 0:00Speaker 00

  Welcome to Unsupervised Learning, Redpoint's AI podcast. I'm Patrick Chase and today we have a really special episode with CEO and co-founder of Anthropic, Dario Amodei. This is a special episode in that it features an interview hosted by Logan Bartlett, my partner at Redpoint. But given the relevance, we wanted to share it with our audience here on Unsupervised Learning. During the interview, Dario shares his predictions about the future of AI, the chances of AI utopia versus catastrophe, how Anthropic is reducing AI bias, why he hates the term AGI, advice for builders, and much more. A truly insightful conversation with one of the people that is leading the AI movement, and I think you'll enjoy it. Thanks so much for joining us. Dario, thanks for doing this.

• 0:41Speaker 01

  Thanks for having me.

• 0:42Speaker 00

  So I normally don't tell people's backgrounds in a linear fashion, but I actually haven't heard yours. I don't know if you've ever really told it in earnest, like childhood, growing up, sort of what led you to starting Anthropic. So maybe can you share a little bit about like your background, childhood, growing up?

• 0:58Speaker 01

  Yeah, yeah. So I don't know that my childhood was that interesting or that that different from, you know, from people who are in tech or found companies. I mean, I was always, I was always really interested in math. It felt like it had, you know, a sense of, sense of objectivity, right? You know, one kid could say, oh, this show is great. And the other kid could say, oh, it's terrible. But you know, if, when you're doing math, you're like, oh, man, there's an objective answer to this. So that was always very interesting to me. And, you know, I grew up with a younger sister who's one of my co-founders, and we always wanted to save the world together. So it's actually, actually, actually kind of amusing that, you know, we're, we're, we're working, we're working on something together that, you know, at least, at least potentially

• 1:36Speaker 01

  could have, could have very, very wide scope. So yeah, I mean, in terms of how got from there to Anthropic, my interest in math led me to be, you know, physics major undergrad. But near the end of undergrad, I started reading, it was initially the work of Ray Kurzweil, who, you know, I think is a bit crazy about a lot of things, but just the basic idea that there's this, this acceleration, there's this exponential acceleration of compute, and that that's going to provide us enough Compute somehow, we had no idea how then, we had no idea it was going to be neural nets. You know, what will somehow get us to like very powerful AI.

• 2:19Speaker 01

  And I I found that idea really convincing, so I was about to start grad school for theoretical physics. And, you know, decided as soon as I got there that I wanted to do biophysics and computational neuroscience. Because, you know, if there was going to be AI, you know, it didn't feel like AI was working yet. And so I wanted to study the closest thing to that, that, that there was, which was, you know, our, which was our brains. You know, the, the, the closest, it's a natural intelligence. So therefore the closest thing to, to an artificial intelligence that, that, that exists. Um, so I studied that for a few years and kind of worked on networks of real neurons. And then, uh, you know, shortly after I graduated, I was at Stanford for a bit.

• 3:00Speaker 01

  And then I saw a lot of the work coming out of, of Google, of Andrew Ng's group at Stanford. And so I said, okay, uh, you know, I should, I should get involved in this area. You know, my, my reaction at the time was, oh, my God, I'm so late to this area. The revolution has already happened. What year is this? Yeah, twenty fourteen, right? So, so, you know, I was just like, oh, my God, like, you know, this tiny community of fifty people, like, they're the giants of this field. It's too late to get in. If I rush in, maybe I can get some of the scraps. Um, that was, that was my mentality when I, when I kind of entered, entered the field. And now, of course, it's, you know, it's, it's nine years, nine years later than that. And, you know, every, you know, I, I, I interview someone every day who's like, I, you know, I really want to get into this field. Um, and so I ended up working with Andrew Ng at Baidu for a year.

• 3:44Speaker 01

  I ended up working at Google Brain for a year. Uh, then I was one of the first people to join OpenAI in twenty sixteen. Uh, I was there for about, uh, five years. Uh, and by the end of it, I was VP of Research, was driving a lot of the research agenda. Um, we built GPT two, GPT three, reinforcement learning from human feedback, which is, you know, of course, the method that's used in chat GPT. And used along with other methods in our, in our model Claude. Uh, and, you know, one of the big themes of. Those five years was this idea of scaling that you can put more data, more compute into the AI models and they just get better and better.

• 4:29Speaker 01

  And I think that, you know, that thesis was really, was really central. And the second thesis that was central is you don't get everything that way. You know, you can scale the models up, but there, there are questions that are unanswered. It's, uh, you know, ultimately sort of the fact value distinction. You scale the model up, it learns more and more about the world, but you're not telling it how to act, how to behave, what goals to pursue. And so that, that dangling thread, that three variable was the, was the second thing. And so those were, those were really kind of the two lessons that I, that, that, that I learned. And of course those ended up being the two things that, that Anthropic is really about.

• 5:03Speaker 02

  What year was it that you joined OpenAI?

• 5:05Speaker 01

  Uh, it was 2016.

• 5:08Speaker 02

  And what was the original connection? Was it just that this seemed to be where the smart people were going?

• 5:13Speaker 01

  Yeah, so I was actually, uh, initially invited, uh, to join the organization before it existed, uh, in, uh, late 2015 as it was, uh, as it was forming and, uh, decided, decided not to. Um, but then a few months after it started, I kind of, you know, a bunch of smart people ended up joining. And so I said, oh, maybe I'll, maybe I'll do this after all.

• 5:37Speaker 02

  You were there for a number of years and at some point you made the decision that Anthropic was going to be, I guess it wasn't initially, you've had an unusual path. It wasn't initially a company, right? Originally it was started as a quasi research lab. Is that fair?

• 5:53Speaker 01

  I mean, our, our, our, our strategy has certainly evolved, but actually it was a for-profit, public benefit corporation, uh, since the beginning. Uh, and we had, even since the beginning, we had something on our website saying, you know, we're doing research for now, but you know, we see the potential for commercial activity down the road. So I think all of these things we kind of kept open as potentialities, but you know, you're right, for the first, for the first year and a half, it was mostly building technology and we were agnostic on, you know, what exactly we were going to do with that technology or when. Um, we kind of, we kind of wanted to keep our options open. We felt it was better. Better than saying, you know, we're about this or we're about that.

• 6:33Speaker 02

  And what was the thought at the time of, hey, we should go do something on our own? Was that your thought? Was it a group of people?

• 6:42Speaker 01

  Yeah, it was. It was definitely the thoughts of a group of people. So there were seven co-founders who left. And then I think in total, we got fourteen or fifteen people from OpenAI, which was about ten percent of the size of the organization at the time. You know, it's funny to look back on those days because, you know, in those days, we were the language model part of OpenAI. Like, you know, we, along with a couple of people who stayed, you know, were those who had developed and scaled the the language models. There were many different other things going on at OpenAI, right? There was, you know, there was a robotics project, a theorem proving project, projects to play video games. Some of those still exist. But, you know, we felt that we were this kind of coherent group of people.

• 7:27Speaker 01

  We had this view about, you know, language models and scaling, which, to be fair, I think the organization supported. But then we also had this view about, you know, we need to, we need to make these models safe in a certain way. And, you know, we need to do them within an organization where we can really believe that these principles are incorporated top to bottom.

• 7:49Speaker 02

  OpenAI had a whole bunch of different things and still does experimenting around. Like, was it evident? At what point along the way was it evident that large language models were something that there was a lot of wood to chop and a lot of opportunity around?

• 8:05Speaker 01

  Yeah, I mean, I think, I don't know, it was obvious at different times to different people. So, you know, for for me, I think there were, there were a couple of things. You know, the general scaling hypothesis. I wrote this document called The Big Blob of Compute in 2017. Which I'll, you know, probably, uh, probably publish at some point, although it's primarily of, you know, like historical interest now. Um, and so very much in my mind, and I think the minds of, you know, like a small number of other, of other, uh, of, of, of, of, of other people on both the team that left and the team that didn't leave and, you know, some, some in other places in the world as well, it was clear that there was really something to scaling. And then as soon as I saw GPT one, which was done by Alec Radford,

• 8:50Speaker 01

  um, who's still, uh, still at OpenAI, our team actually had nothing to do with GPT one, but we recognized it immediately and saw that the right thing to do with it was, was to scale it. And so for me, everything was clear at that moment. And even more clear as we kind of scaled up to GPT two, when, you know, we saw that, you know, the model was capable of my, my favorite thing was like, you know, we were able to get the model to perform a regression analysis. So, you know, you give it like, you know, the price, the price of a house and ask it to predict number of square feet or something like that. You gave it a bunch of examples, then you gave it one more price and you're like, how many square feet? It didn't do great, but it did better than random. And in those days, I'm like, oh my God, this is like some kind of general reasoning and prediction

• 9:33Speaker 01

  engine. Like, oh my God, what is this thing I have in my hands? Right? It's, it's completely crazy. So I, I, you know, it has been my view ever since then that, you know, this, this would be not just language models, but you know, language models as exemplar of the kinds of things you can scale up. You know, this, this, this would be, uh, you know, really, really central to, to the future of technology.

• 9:56Speaker 02

  Did you consider yourself like a founder or a CEO prior to actually doing it?

• 10:01Speaker 01

  Um, no. So I, I, I really kind of never thought of myself that way. Like if you went back to, you know, me in childhood, it would have been very unsurprising, you know, for me to be a scientist. But, you know, I never kind of thought of myself as like a founder or a CEO or a business person, right? You know, I always thought of myself as a scientist, someone who discovers things. But I think just having been at Several different organizations convinced me that in fact I did have my own vision of how to run a company or how to run an organization because I'd seen so many and I thought, well, I don't know, I'd actually do it this way, right? And so sort of the contrast, you know, not that, not that I disagreed with every decision that made, but just watching all

• 10:43Speaker 01

  these decisions go by took me to the point where I'm like, actually, I do, I do have opinions on these questions, right? I do have an idea of how you would grow an organization, how you would run a research effort, how you would bring the products of that research organization out into the world in a way that's, you know, makes business sense, but is also responsible. I don't think, I don't think I really had those thoughts naturally, but as I was brought into contact with organizations that, that, that did that, then, then I became excited about those things. I kind of almost, almost reluctantly learned that I actually had strong opinions on these things.

• 11:17Speaker 02

  Not to draw it in contrast to any specific names, but maybe just in others in the field that I would consider large language foundation model as a product. What's, what's something foundational, I guess not to use a cute term, but something foundational that you believe that anthropic, that you would draw in distinction to others in the space?

• 11:42Speaker 01

  Yeah, I would, I would say a couple of things. So, you know, one is just this idea that we should be building in safety from the beginning. Now, I'm aware we're kind of not the only ones who've, who've, who've, who've said that, but I feel like we've built that, we've really built that in from the beginning. We've thought about it from the beginning. We've started from a place of caution and kind of, you know, commercialized things, brought things out into the world, starting from, hey, you know, can we, can we open these switches one by one and see what actually makes sense? I think in particular, you know, a way, a way I would think about it is that what we're aiming to do is not

• 12:22Speaker 01

  just to be successful as a company on our own, although we are trying to do that, but that we're also trying to kind of set a standard for the field, set the pace for the field. So this is, this is a concept. We've called race to the top. So, you know, race to the bottom is a, is a popular term where, you know, everyone is, you know, competing to, you know, lower cost or delivers things as fast as possible. And as a result, they cut corners and things get worse and worse. So that, that dynamic is real and we always think about how not to contribute to it too much. But there's also a concept of race to the top, which is that if, if you do something that looks better, it naturally has the effect that other players end up doing the same thing. And this, this happened for interpretability.

• 13:05Speaker 01

  For a couple of years, we were the only org that worked on interpretability seen inside neural nets. There are various corporate structures that we've, we've, that we've, that we've, you know, that we've, we've implemented that, you know, we hope others, others may emulate. And, you know, recently we released this responsible scaling plan that I could talk more about later, but, but generally we're trying to set the pace. We're trying to do something good, inspiring, also viable, and encourage others to do the same thing. And, you know, at the end, again, maybe we win in a business sense and of course, that's great, but maybe someone else, you know, maybe someone else wins in a business sense. Or, you know, we all win, we all split it.

• 13:48Speaker 01

  But the thing that matters is that the standards are increasing.

• 13:51Speaker 02

  I want to talk about all that stuff in a second, but why do you think philosophically that like AI development or the scaling of models and safety? Why are they intertwined? I've heard you maybe coiled together in, in, in different ways.

• 14:05Speaker 01

  Yeah, yeah. So I think this actually isn't such an unusual thing. Like I think this is true. It's true in most fields, right? Like, you know, the, the, the common analogy is like, is like bridges. You know, make, building a bridge and making it safe, they aren't exactly the same thing, but they both involve all the same principles of like civil engineering. And it's kind of hard to work on bridge safety in the abstract aside from, you know, You know, kind of outside the context of like a concrete bridge. Like, you know, what you need to do is you need to look at the bridge. You're like, OK, well, these are the forces on it. You know, these are the, this is the stress tensor. This is, you know, the strength of the material or whatever. That's the same thing you come up with in building the bridge.

• 14:48Speaker 01

  If safety differs in any way, maybe it differs in thinking about the edge cases. Like in safety, you have to worry what goes wrong 0.1% of the time. Whereas in building the bridge, you have to think about the, you have to think about the median case. But, you know, it's, it's all the same civil engineering. It's all the same, you know, forces, forces of mechanical physics. And I think it's, I think it's the same in AI, in AI and large language models. And, and in particular, safety is itself a task. Like, you know, is this thing the model's doing right or wrong? Where, where, you know, right or wrong could be something as prosaic as is the model telling you how to hotwire a car?

• 15:28Speaker 01

  Or as, you know, scary and sophisticated as is the model going to help me build, build a bioweapon? Or is it gonna, you know, take over the world and make swarms of nanobots? Or, you know, whatever futuristic thing, figuring out whether the model's going to do that and the behavior of the model is itself an intellectual task of the kind that models do. And so the problem and the solution to the problem are mixed together in this way. Where, uh, you know, every time you get a more powerful model, you also gain, gain the capability to understand and potentially rein in the models. So we have this problem where these two things are just, are just mixed together in, in a way that's, I think, hard, I think, hard to untangle. And I think that's the usual thing.

• 16:09Speaker 01

  I think the only reason that's surprising is that the community of people who thought about AI safety was historically very separate from the community of people who developed AI. Like, they had a different, they had a different attitude. They came to it from a, you know, kind of a, you know, more philosophical perspective, more of a moral philosophy perspective. Whereas those who built the technology were engineers. But just because the communities were different, doesn't, doesn't, you know, that, that doesn't imply that the actual, the actual content turned out to be different. So I don't know, that's. That's my view on it.

• 16:43Speaker 02

  It was becoming a business and commercializing your effort. If you could scale and figure all the safety stuff out without being a business, would you pick that path? Or is the business side of it inherently intertwined as well as something that interests you?

• 17:01Speaker 01

  Yeah, yeah, so I'd say a few things on that. One is, I think it's actually going to be very difficult to build, or would have been very difficult, to build models of the scale that we want without being a commercial enterprise. I mean, you know, people make jokes about VCs being willing to pour huge amounts of money into anything, but like, you know, I think that's only true up to a point, right? Like, you know, there's business logic behind it. You guys have LPs, like, things need to, you know, it's not just all hype terrain, right? Things need to make sense eventually. And so, you know, we're now getting to the point where you need, you know, certainly multiple billions of dollars, and I think soon tens of billions of dollars,

• 17:45Speaker 01

  to build models at the frontier. And to study safety with those models at the frontier requires you to have intimate access to those models, particularly for tasks like interpretability. So first of all, yeah, I just think it's very hard. On the other hand, you know, or in support of that point, I also think that there are some things that you learn from the business side of things. Some of it is just learning the muscle and the operation of things like trust and safety. So, you know, today, we deal with trust and safety issues like, oh, you know, people are trying to use a model for inappropriate purposes, right?

• 18:29Speaker 01

  You know, not things that are going to end the world, but things that, you know, we'd rather that people not do. I think the ultimate significance of being able to develop methods to address those things and, you know, kind of enforcing those things in practice when they're used at scale by users is it allows us to practice for the cases that are really, really high stakes. And I think without that organizational, institutional practice, it might be difficult to kind of just be thrown into the shark tank. You know, congratulations, you've built this amazing model. You know, it can cure cancer, but also, you know, someone could make a bioplague that would kill, that would kill a million people. You've never built a trust and safety org. You have to deploy this model in the world and make sure we do one and not.

• 19:14Speaker 01

  That would, that would just be a very, a very difficult thing to do. And I, I don't think we would get it right. Now, all that said, I mean, I, I will freely admit, you know, my, my passion is this, my passion is the science and the safety, right? You know, that's, that's kind of my first, my first passion. Um, the, you know, the, the, the, the business stuff is, the business stuff is quite a lot of fun. Um, you know, I think, you know, we've just, just watched all the different customers, you know, just learning about the whole business ecosystem has been great. Um, but, you know, definitely my, my first passion is, you know, is, is, is, is the science of it and making sure it goes well.

• 19:51Speaker 02

  Was there a serious debate about like being a business versus not, uh, in the early days? Was that like a real conversation among the

• 19:58Speaker 01

  Yeah, so I think, I think certainly everyone was aware from the beginning that, you know, we, you know, that there was a good chance that we would commercialize the models at some point, you know, we, we had this thing on our website. I'm not sure if it's up there anymore, but you can see it on the Wayback Machine that said, you know, for now, we're doing research, but we see commercial potential down the road. So everyone who joined saw that and everyone who joined knew that. Um, but there was a question of, you know, when exactly should we do it? So there was a, there was a period around, I think it was April, May, June of 2022, when we had kind of the first version of Claude, um, which was actually like a smaller model than Claude 1.

• 20:38Speaker 01

  But, you know, we were, we were training the model that, that would become Claude 1 at that, at that time. And we realized that with RL from human feedback, we didn't have our constitutional AI method yet. That this thing was actually great to interact with and, you know, all of our employees were having fun interacting with it on Slack. We showed it to some, to a small number of external people and they had lots of fun. They had lots of fun interacting with it. So it definitely occurred to me and others that, hey, there, there could have been a lot of commercial potential to this. I don't think we anticipated the explosion that happened at the end of the year. Like, we definitely saw potential. I don't think we saw that much potential.

• 21:16Speaker 01

  But yeah, we, you know, we definitely had a debate about it and I wasn't sure quite, quite what to do. I think our concern was that with the rate at which the technology was progressing, a kind of big, loud, public release might accelerate things so fast that the ecosystem might not, might not know how to handle it. And, you know, I didn't want our kind of first act on the public stage, you know, after we'd said it, you know, after we put so much effort into being, being, being responsible to, to accelerate things, to accelerate things so greatly. I, I generally feel like we made the right call there. I think it's actually pretty debatable. Um, you know, there's many, many pros, many cons, but I think overall, overall, we made the right call.

• 22:00Speaker 01

  Uh, and then, you know, certainly as, you know, as soon, as soon as the other models were out and kind of the gun had been fired, then we started putting these things out. We're like, okay, all right, you know, now, now there's definitely a market in this. People, people, people, people know about it. And so, you know, we should, we should, we should, we should, we should get out ahead. And, you know, indeed, we've, we've managed to, you know, put ourselves among, you know, among the top two or three players in this space.

• 22:22Speaker 02

  Was that gun being fired and ChatGPT sort of taking off? Was that similar to the maybe fear that you had in, in of like, hey, this might start a race?

• 22:32Speaker 01

  Yeah, yeah, similar and in fact more so. Um, so, you know, I think we saw it with, you know, Google's, you know, Google's reaction to it. You know, that there was definitely, you know, just judging from the public, public statements, you know, a sense of fear and existential threat. And I, you know, I think they responded in a very economically rational way. I don't, I don't blame them for it at all. Um, but you put the two things together and, you know, it, it, it really created an environment where things were, you know, racing forward very quickly. And look, I, I love technology as much as the next person. There was something like, you know, super exciting about the whole, you know, make, make them dance. Oh, we're responding with something, you know? I mean, I can, I can get just as excited about this as everyone, but given the rate

• 23:13Speaker 01

  at which the technology is progressing, um, you know, there was a worrying aspect about this as well. Um, and so in this case, I'm at least, at least on balance, glad that, you know, we, we weren't the ones who fired that starting gun.

• 23:24Speaker 02

  Yeah, got it. Well, you recently- Only announced an investment from Amazon.

• 23:27Speaker 01

  Yes.

• 23:28Speaker 02

  Before that, you did a round with Spark and a little bit more traditional venture capitalists.

• 23:32Speaker 01

  The round with Amazon, it's complicated and I can't go into the details, but it's not a full closed

• 23:37Speaker 02

  Sure.

• 23:37Speaker 01

  price.

• 23:38Speaker 02

  Corporate round and all that. Before that, you had an unusual round as well with FTX, right?

• 23:45Speaker 01

  Yes.

• 23:45Speaker 02

  Or was it, how did that come to be?

• 23:47Speaker 01

  Yeah, so, you know, honestly, there was actually very little to that. So, you know, there's kind of a community of people who cared a lot about AI safety. You know, back when he was doing FTX before he committed all the fraud or was caught committing all the fraud that he committed. Sam Bankman-Fried was, you know, presented himself as someone who cared a lot about issues like pandemics, AI safety. So, you know, he was known to people in my community and honestly, there's not much to tell. I, you know, I only talked to him a handful of times.

• 24:20Speaker 02

  The entity is still related to FTX, right? So there's potential that the Anthropic investment could one day make FTX people whole, depending on how it all plays out.

• 24:31Speaker 01

  That's one of the ironies that, you know, there's a bankruptcy estate or bankruptcy trust that owns these non-voting shares. And, you know, so far they've declined to sell them off, but they're interested in doing so in a general sense. And so.

• 24:47Speaker 02

  I'm told there's people that are very interested in buying those shares from them.

• 24:50Speaker 01

  Yeah, and, you know, I can't comment on the market dynamics there and we don't, we don't really control them, right? It's a sale between different parties, but hey, if those, if those shares lead to, you know, the people who had their money stolen, getting some or all of their money back, then, you know, that's kind of a random chance, but certainly a good thing.

• 25:10Speaker 02

  Be a good outcome. So what does the business of Anthropic look today? You guys are focusing mostly on enterprise customers?

• 25:16Speaker 01

  Yeah, we are focusing mostly on enterprise customers. I would say we have both an enterprise product and a consumer product. It makes a lot of sense if you're building one of these models to at least try to offer them in every direction that you can, right? Because the thing that's expensive in terms of both money and people is building the base model. Once you have it wrapping you into the consumer product versus wrapping it in an API, while both of those things do take substantial work, you know, are not as expensive as the, you know, kind of the base work and the model. And so we have a consumer product that honestly is doing pretty well, but. At the same time, you know, our real focus definitely is, definitely is enterprise. Um, we've found that some of the properties of the model in a, in a practical sense, right?

• 26:01Speaker 01

  The safety properties in the model in a very practical sense, as opposed to kind of, you know, like a philosophical or future sense, um, are actually useful for the enterprise use cases. Um, you know, we try to make our models helpful, honest and harmless. Um, honesty is a very good thing to have, you know, in knowledge work settings, you know, number of our customers are in like the finance industry, the legal industry, starting to get stuff on the health side, different, you know, productivity, productivity apps. Um, those are all cases where a mistake is bad, right? You know, you're doing some financial analysis, you're doing some legal analysis, like you, you really, you really have a premium on like, make sure the thing knows what it doesn't know.

• 26:44Speaker 01

  Uh, so, you know, giving you something misleading is much worse than, than not giving you, not giving you anything at all. I mean, that, that's true across the board, but I think it's especially true and true and true, true, it's especially true in those industries. Um, for, for enterprises, often kind of like inappropriate or embarrassing speeches, you know, something, something that they're, they're very concerned about, even if it happens very rarely. Uh, and so the ability to kind of steer and control the models better, I think is, is very appealing to, to, is very appealing to a number of enterprise customers. Um, another thing that's been helpful is this, you know, we have this longer context window. So context window is like how much information the model can take in and process. So our context window is a hundred thousand tokens.

• 27:29Speaker 01

  Tokens are this weird unit. It really corresponds to 70,000 words. Uh, but you know, the next, the model with the next biggest context window is GPT four, where there's a version of it that has 32K tokens, 32,000, um, which is three times less, but the main GPT four has 8,000, which is about 12 times less. Um, and so just the ability to, for example, something you can do with Claude that you can't do with any other model is, you know, read a mid-sized book or novel or textbook or something, just stick it into the context window. Upload it and then start to ask questions about it. And so that's something you can't do or can't do nearly as easily with any other model. And then another thing that's actually been appealing is raw cost.

• 28:14Speaker 01

  So the sticker price of Claude two is about four X less than the sticker price of GPT four. And the way we've been able to do that, I can't, I can't go into the details. But we've worked a lot on algorithmic efficiency for both training and inference. So we're able to produce something that's, you know, in the same ballpark as GPT four and better for some things. And we're able to produce it at a substantially lower cost. And we're in fact excited to extend that cost advantage because, you know, we're working with custom chips with various different, different companies and we think that could give an enduring advantage in terms of, in terms of, in terms of inference cost.

• 28:56Speaker 01

  So, so all of those are particularly helpful for the, for the, for the enterprise case and, and, you know, we've, we, we, we found pretty strong, pretty strong enterprise adoption, you know, even, even in the face of, you know, competition from multiple companies.

• 29:10Speaker 02

  Hi, I'm Logan Bartlett, the host of this podcast. This is not an ad. As you may know, we do not advertise or monetize this podcast in any way. I just wanted to take a quick second to tell you that we have a bunch of killer guests coming on over the course of the next few weeks. And so if you're enjoying these conversations behind the scenes with both entrepreneurs and investors, please do subscribe to our channel so you don't miss out. Back to the episode. With something that's perhaps unintuitive to someone that isn't living and breathing this every day about enterprise interest in artificial intelligence as someone sitting at that nexus.

• 29:44Speaker 01

  First of all, I see a huge advantage to folks who think in terms of the long term. So there's, there's a fact that's kind of, you know, the bread and butter for those of us who are building the technology. But, you know, getting, getting it across to the customers is, I think, one of the most important things, which is the pace at which the models are getting better. Some of them get it, others are starting to get it. But the reason this is important is, you know, put yourself in the shoes of a customer, right? They've got, they've got our model clawed. They wanna, you know, they wanna build something and you know typically they wanna start with something small. And of course, naturally, they think in terms of what can the model do today? And what I always say is, do that. We gotta start. We gotta iterate from somewhere.

• 30:27Speaker 01

  But also, think in terms of where the models are gonna be in one or two years. It's gonna be a one or two year arc to, you know, to go from proof of concept to small scale deployment, to a large scale deployment, to true product market fit for whatever it is that, that you're launching. Um, so you should basically skate where the puck is going to go. You should think, okay, the models can't do this today, but look, they can do it forty percent of the time. That probably means they can do it eighty or ninety percent of the time in one or two years. So let's have the faith, the belief of faith, to build for that. Instead of building for, uh, you know, what the models are, what the models are able to do today. And if you think that way, the possibilities of what you can do in one to two years are much more expansive.

• 31:12Speaker 01

  And we can talk about having a kind of longer term partnership where we, where we build this thing together. And I think, you know, the customers that have, that have thought that way are, you know, ones that, that, that, you know, we, we've been able to work together with on a path towards creating a lot of value. Um, we also do lots of things that are just targeted as, as what you can do today. But often the things I'm most excited about are those that see the potential of the technology. And by starting to build now, they'll have the thing tomorrow as soon as the model of tomorrow comes out, instead of it being another year to build, to build after that.

• 31:48Speaker 02

  This is something that I think is, uh, particularly true of, of anyone and particularly any leader, but, uh, you had said recently, attaching your incentives to the approval or cheering of a crowd in some ways destroys your mind and in some ways can destroy your soul. You haven't been as public as other folks in the, in the space have been. I assume that's very purposeful stylistically and ties into that. Can you talk a little bit about your thoughts around it?

• 32:15Speaker 01

  Yeah, I mean, part of it, it's just, it's just kind of, it's just kind of my, my, my style for one thing. I mean, you know, I think- I think as a scientist, you know, I prefer to speak when there's kind of something clear and substantive to say. Um, you know, I mean, I'm not totally low profile. I mean, I'm on this podcast right now and I I've been on a few. So, you know, given the general volume of the field, there's some need to to get the message out there and that need will probably increase over time. Um, but I think I have noticed and, you know, it's not just, you know, Twitter or social media, but some phenomenon that's a little bit connected to them that, you know, you can really, if you, if you think too much in terms of kind of pleasing people in the short, short

• 32:57Speaker 01

  term, or, you know, making sure that you say something, making sure that you say something popular, it can really kind of lead you down a bad path. And I've, you know, I've seen that with a lot of, you know, with a lot of very smart people. I mean, I'm sure you could name some as well. I'm not going to give any names. Who, who have gotten caught up in this and, and, you know, years later you look at them and you're like, wow, this is a really smart person who's, who's acting much dumber than they are. Um, and I, you know, I think the way it happens, I don't know, I could give an example, right? So, you know, take, you know, like a debate that's important to me, which is like, you know, should we build these things fast or should we make these systems safe? So there is an online community, mostly on Twitter of, you know, people who, who are, you know, think, think we should slow down and then kind of an online community of builders who are really excited about, you know, we should make this stuff fast.

• 33:40Speaker 01

  And if you go to certain corners of Twitter, like you get these really extreme versions of each one, right? On one hand, you get people who say like, we should stop building AI, we should have a global worldwide pause. Right? I think, I think that doesn't work for a number of reasons. I mean, we have, our responsible scaling plan is sort of, is incorporate some aspects of that. So I think it's not an unreasonable, uh, you know, discussion or debate to have, but you know, there's this kind of really extreme position. And then that's, that's kind of created this polarization where there's this, this other extreme position. It's like, we have to build as fast as possible. Any, any regulation is just regulatory capture, you know, we just need to maximize the speed of progress.

• 34:22Speaker 01

  The most extreme people say things like, it doesn't matter if humanity's wiped out, AIs are the future. I mean, that's a really extreme position. And, and so, you know, think of. Kind of, you know, the, the, the position of someone who's kind of trying to be thoughtful about this, trying to build, but build carefully. If you, if you kind of enter that fray too much, if you, uh, you know, if you feel like you have to make those people happy, what can end up is either you get polarized on one side or another. And then, then you kind of repeat all the slogans of that, of that side and you become a lot dumber than you would otherwise. Um, the, you know, if, if you're, if you're really good at dealing with Twitter, you know, you can, you can try and make both people happy, but you know, that, that involves a lot of playing to both sides.

• 35:06Speaker 01

  And I cer- I cer- I certainly don't want to do that. That's what I talk about in kind of, kind of losing your soul. Um, you know, the truth is the actual, the, you know, the position that I think is actually responsible might be something that would make all of those people boo instead of, instead of all of them cheer, right? And, and, and so you just, you just have to be very careful if you're, if, if you're taking that as your barometer, you know, who's yelling at me at, on, on Twitter, who thinks I'm great on Twitter, you're not going to arrive at the position that makes everyone boo, that, that might just be the correct position.

• 35:37Speaker 02

  What timeline do you think about then when you're saw, it's not the instantaneous dopamine head of a tweet. You mentioned talking to enterprises about one to two years and what can be, but like, what timeline are you solving for?

• 35:48Speaker 01

  Yeah, I mean, I, I guess, I guess I kind of think like five to 10 years from now, everything will be like a little bit more clear. Um, and you know, it, it'll, I think it'll be more clear which decisions were good decisions, which decisions were bad decisions. You know, I think, you know, certainly less than that time scale is a time scale on which, you know, if dangerous things with these models are indeed possible, as I believe they are, but I could be wrong, I think they may play out on that time scale and you know, we'll, we'll, we'll, you know, we'll be able to see like which companies addressed these dangers well. Um, or were the dangers not real and, and people like me warned about them and we were just totally wrong.

• 36:31Speaker 01

  Or, uh, you know, will it turn out that some, that some tragedy happened and you know, people, people like me should have been more extreme and worrying about it? Or, or will it? Turn out that, you know, that that companies like Anthropic, you know, picked picked the right path and, you know, navigated a a a dangerous situation well. I I don't know which how it's going to turn out. I mean, I hope it turns out that, you know, we navigated a dangerous situation well, and we averted catastrophe, and there were there were hard trade offs, and, you know, we we addressed them uh, skillfully and thoughtfully. That's my hope for how it's going to turn out. But I I don't know that it's going to turn out that way. But but, you know, I feel like looking at that, you know, in five years, in ten years, that's that's just going to be a fair judgment of all the things that that I'm saying and doing.

• 37:15Speaker 02

  What what would you want the average person listening who is aware of AI, knows what Anthropic is, knows what uh, OpenAI is, knows what Google and others are doing in the space about safety and about risk? What would you want them to know from your perspective?

• 37:32Speaker 01

  Yeah, so I think, you know, if I were if I were to kind of just put it in a few sentences. I think what I would say is, look, I have I have two concerns here. One is the concern that people will misuse powerful AI systems. People misusing technology is nothing new. Um, but but one thing that I think is new about AI is that it its ability to put all the pieces together is much greater than any previous technology. I think in general, we've always been protected by the fact that, you know, if you take a Venn diagram of people who want to do really bad things and and, you know, people who have strong technical and operational skills.

• 38:13Speaker 01

  Generally, the overlap has been pretty small. If you're a person who has a PhD or is capable of running a large organizations, you have better things to do than come up with evil plans to, you know, murder people or destroy society, right? That it it's just, you know, not not very many people are motivated in that direction. And then, you know, the people, the people who are, you know, often they're they're just. I mean, not not all of them, but in many cases, not that bright or not that skilled. The problem is, you know, now could we take unskilled person plus skilled AI plus bad motives? And so, you know, I testified in Congress about this, about the risk of bioweapons. I think cyber is another area.

• 38:55Speaker 01

  Bunch of stuff around national security and, you know, the relationships between between nations and stability. So that's one corner of it. And then I think the other corner of it is what the AI systems themselves may do. And, you know, there's lots you can, you know, kind of find in the Internet and various communities on this. But but I often put it in a simple way, which is, one, the systems are getting much more powerful. Two, we, there's obviously not much of a barrier to getting the systems to act autonomously in the world, right? People have taken GPT-4, for instance, and turned it into AutoGPT. There was even a WormGPT, which was, you know, supposed to act as a computer worm. So, you know, powerful smart systems that can take action.

• 39:40Speaker 01

  So, you know, kind of very long leash of of human supervision. And because of the way they're trained, they're not easy to control. I mean, we all saw Bing in Sydney. So you put those three things together and, you know, there's at least, you know, I think some chance that as the systems get more and more powerful, you know, they're they're going to do things that we don't want them to do. And it may be it may be difficult to fully control them, to fully rein them in. I think that's further out than the misuse, but it's, you know, it's something we should think about.

• 40:12Speaker 02

  We've touched on a few of the different things you guys have done from a safety standpoint. So I want to talk through the, I guess, the three ones I took down. So you you Long Term Benefit Trust and Public Benefit Corporation. Can you explain what that is and how you decided to do that?

• 40:27Speaker 01

  Yeah, yeah, so we were incorporated as a public benefit corporation from the from the beginning.

• 40:32Speaker 02

  Which means?

• 40:33Speaker 01

  Basically, public benefit corporation, it's actually very much like a C corp, except the the investors can't sue the company for failing for failing to maximize or for failing to maximize.

• 40:46Speaker 01

  minimize profits. Um, you know, I think, I think in practice, in the, in the vast, in the vast majority of cases, it operates like a normal company. I mean, I think that's one theme I want to get through here, like 99% of what we do. Um, you know, we would, we would make the same, you know, the same decision that, that a normal company would. Um, you know, most of the time, the, you know, the, the, the, the, the, the logic of business, which is basically the logic of providing mutual, mutual value. Um, also makes sense from a public benefit corporation, but there's maybe this one, one percent of key decisions. You know, I might, I might think about the, you know, the, the, the, the delay of release of Claude.

• 41:27Speaker 01

  Um, decisions that might relate to, hey, we have a very powerful model, but we need to make really sure that it, you know, this thing can't create a bioplag that'll kill millions of people before we, before, before we release it. So I think there are going to be a few key moments in the company where this, where this makes a difference. Um, and then LTBT, you know, as I said, a public benefit corporation is, it's not that different from a C corp. Um, the idea of the LTBT is to have a set of, uh, um, uh, so LTBT is long-term benefit trust. So right now, the governance of Anthropic is pretty much like that of a normal corporation. But we have a plan that was written into our original series A documents and has been iterated on since then.

• 42:10Speaker 01

  That will, will gradually hand over the ability to appoint a majority of Anthropic's board seats to a, a kind of trust of people. And on that trust of people, we've, we've, we selected the original ones, but then it becomes, it becomes self-sustaining. We selected for a kind of three types of experience. Uh, one type is experience in AI safety. One type is experience in national security topics, as I think this is going to become relevant. And, and another type is, you know, thinking about things like philanthropy and the macroeconomic distribution of income. Um, so I, I think of those as my best guess as to kind of the, you know, the, the, the three topics where

• 42:51Speaker 01

  something that kind of trend, you know, kind of transcends the, the kind of ordinary activity of companies is, is going to come up.

• 42:59Speaker 02

  And is that who you ultimately report into when this structure is finalized? That'll be the the board that Anthropic answers to?

• 43:07Speaker 01

  So this set of five people appoints a majority, but not all of the corporate board of the company. So there's basically two, there's, there's two of these bodies. Um, and, and the LTBT appoints the corporate board. Um, now, look, that said, I mean, we all, we all kind of, you know, in, in, in practice, the company is almost always run, you know, day to day by the CEO, right? Like, you know, it's, it's very, even, even speaking of the, you know, even speaking of the corporate board, not just for Anthropic, but, but any other company. I mean, you know, you think of, you know, think of as a CEO, how many decisions you directly make yourself versus how many it's like, oh, I have to get the board on board with that. I mean, you know, there are some when you, when you, when you, you know, when you, when you, when you, when you, when you, when you raise money,

• 43:52Speaker 01

  when you, you know, issue new employee, when you issue new employee shares, when you make a major strategic decision. Um, the LTBT is kind of an even more rarefied body. Um, and, and, you know, I, I, you know, I've set the expectation with them that, you know, their role is to, you know, get involved in the things that, you know, that really involve critical decisions for humanity. You know, there might only be three or four such decisions in the entire history of Anthropic.

• 44:18Speaker 02

  Now, Constitutional AI, can you talk about what that is and what the inputs into it were?

• 44:22Speaker 01

  Yes, so Constitutional AI is a method that we developed around the end of, of last year. Um, so easiest to explain it by contrasting it with this previous method called reinforcement learning from human feedback, which I and some other people were the co-inventor of at, at, at OpenAI, uh, in 2017. So reinforcement learning from human feedback, the way it works is, okay, I've trained the giant language model. I've paid my tens of, maybe hundreds of millions of dollars to, to train it. Uh, and, and now I, I want to give it some sense of how to act. Uh, so, you know, there, there are questions you can ask the model that don't have any clear, factually correct answer. So, you know, I could say, what do you think of this politician? Or what do you think of this policy?

• 45:06Speaker 01

  Or what should I as a human do in this situation? And you know, the the model doesn't doesn't have any definite definite answer to that. So the way RL for human feedback works is you hire a bunch of contractors. You give them examples of how the model is behaving. And the humans kind of, you know, they they kind of give feedback to the model. They say this answer is better than that answer. And then and then and then over time, the model updates itself to learn to do whatever is in line with what the human contractors say. Um, one of the problems with this is, you know, one, it's expensive. It requires a lot of human labor. But but in addition to that, it's it's very opaque, right? You know, if I if I serve the model in public and then, uh, you know, someone says, hey, why is this model biased against conservatives?

• 45:50Speaker 01

  Or why is this model biased against liberals? Or why is this model just give me weird sounding advice? Or why does it, you know, why does it give things in a weird style? Uh, I can't really give any answer. I can just say, well, I hired 10,000 contractors I don't know. And, uh, you know, this this was the statistical average of what of what the contractors generally generally proposed. Or the, you know, the mathematical generalization of it. It's not a very satisfying answer. So the method we developed is called Constitutional AI. And the way that works is you have a set of explicit principles that you give to the model. Um, you know, so the, you know, the principles could be something like on on a political question. You know, present both sides of the issue and don't take a position yourself. You know, say, here are some arguments for, here are some typical arguments against. Opinions differ.

• 46:35Speaker 01

  Uh, so, uh, with that, you basically, just as with RL for human feedback. You you have the model give and you have you have the model give answers. But then you have the model critique its own responses, uh, for whether they're in in line with the model constitution. And so you can run this in a loop. Basically, the model is both the generator and the evaluator with the the constitution is kind of the pin source of truth. Uh, and so this allows you to eliminate human contractors and instead go from this set of principles. Now, in practice, we find it useful to augment that method with human contractors so that you can get the best of both worlds. But you use less human contractors than you were before. You have more of a guiding principle. And, you know, then if

• 47:20Speaker 01

  Then if someone, you know, then if someone calls me up in Congress and says, hey, why is your model woke? Or why is your model anti-woke? Or why is your model doing this crazy thing? You know, I can point to the Constitution and I can say, hey, these are our principles. You could have one of two objections. Maybe you don't agree with our principles. You know, fine, we can have a debate about that. Or, or it's a technical issue. These are our principles. Somehow the train of our model, you know, wasn't perfect and wasn't in line with those principles. And I think separating those two things out is, is, I think, I think very useful. And even like enterprise customers have found this to be a useful thing. The kind of customizability and the ability to separate the two out.

• 47:57Speaker 02

  And, and the inputs into this were, so you use the UN Declaration of Human Rights, Apple's Terms of Service.

• 48:04Speaker 01

  Yes.

• 48:04Speaker 02

  What else went into coming up with the principles?

• 48:06Speaker 01

  Yeah, there were, there were some, uh, there were some principles that were developed for use by an early DeepMind chatbot. Um, but yeah, Apple's Terms of Service, UN Declaration of Human Rights. We added some kind of other things, like we asked the model to respect copyright. Um, so this is one way to, you know, to greatly reduce the probability that the model outputs copyrighted text verbatim. You know, we can, we can all debate, uh, you know, what the, what the status is of, you know, you know, models that we train on corpuses of data. But we can all agree that on the output side, we don't want the model to output vast reams of copyrighted text. That's a, that's a bad thing to do and we, we, we aim not to do that. And so we just put in the Constitution not to do that.

• 48:49Speaker 02

  One of the other things that you introduced around safety, broadly speaking, is responsible scaling policy. Can you talk about what that is?

• 48:55Speaker 01

  Yes. Um, so our responsible scaling policy is this set of commitments that we recently released. That is a framework for how to safely make more and more powerful AI systems and confront the greater and greater dangers that we're going to face with those systems. Um, so maybe the easiest way to understand it is, you know, to think of kind of the two sides of the spectrum. Um, so one extreme side of the spectrum is like, build things as fast as possible, like, you know, release things as much as possible.

• 49:26Speaker 01

  You know, maximize technological progress and I, you know, I understand that position and have sympathy for it in many other, you know, in many other contexts. I just think, you know, AIs are particularly tricky technology.

• 49:38Speaker 02

  You have to put E slash ACC in your Twitter bio if you believe that, I think.

• 49:43Speaker 01

  Maybe I should put both.

• 49:44Speaker 02

  Yeah, yeah.

• 49:45Speaker 01

  And so the other extreme position, which, you know, I also have some sympathy for, despite it being absolutely the opposite position, is, you know, oh my God, this stuff is really scary. And the most extreme version of it was, you know, we should just pause. We should just, we should just stop, you know, we should just stop building the technology, you know, indefinitely or for some specified period of time. And I think my problem with that has always been, well, okay, let's say we pause for six months. What do you actually gain from that? What do you, what do you do in those six months? Particularly with the more powerful models being needed for safety of more powerful models. It's kind of like you've frozen time, you've stopped the engine. What do you, what do you get at the end of it?

• 50:27Speaker 01

  And if you were to pause for an indefinite length of time, then you raise these questions like, well, how do you really get everyone to stop? There's an international system here. There's dictators who want to use this stuff to take over the world. I mean, you know, people use that as an excuse, but it's also true. And, and so, you know, that extreme position doesn't, doesn't make too much sense to me, to me either. But what does make sense to me is, hey, let's think about caution in a way that's actually matched to the danger. Right now, you know, whatever we're worried about in the future, right now, today's systems, they have a number of problems. But I think they're the problems that come with any new technology, not these kind of special problems of, you know, bioweapons that would kill millions of people or, you know, the models,

• 51:09Speaker 01

  you know, kind of, you know, kind of taking over the world in some form. So, so, you know, let's have, let's have relatively normal precautions now. But then let's define a point at which, you know, when the model has certain capabilities, we should be more careful. So the way we've set this up is we've defined something called AI safety levels. So there's something called biosafety levels in the US government, which is like, you know, For a given virus, you define how dangerous it is. It gets categorized as like BSL one, or BSL three, or BSL four. And that determines the kind of containment measures and procedures you have to take to, you know, to control that virus. So we, we think of AI models. We think of AI models in the same way.

• 51:54Speaker 01

  There's value in working with these very powerful models, but they have dangers to them. And so we have these various thresholds between ASL one and ASL two, between ASL two and ASL three, between ASL three, three and ASL four. And at each level, there's, there's certain criteria that, that we have to meet. So right now, we're at ASL two, as we've defined it. Before we get to ASL three, we have to, we have to develop security that we think is sufficient to prevent any kind of, anyone who's not a super sophisticated state actor from stealing the model. That's one thing. Another thing is we have to make sure that when the models reach a certain level of capability,

• 52:35Speaker 01

  we're really, really certain that they're not going to provide a certain class of dangerous information. And to figure out what that is, you know, we're going to work with some of the best biosecurity experts in the world, some of the best cybersecurity experts in the world, to understand what really would be dangerous compared to what can be done, you know, today, today with a Google search. We've defined these tests and these thresholds very carefully. And, and so how does that relate to the two sides of the spectrum? Compared to a pause, the ASL thresholds could actually lead us to pause. Because if we get to a certain capability of model, and we don't have the relevant safety and security procedures in place, then we have to stop developing more powerful models.

• 53:18Speaker 01

  So the idea is, there could be a pause, but it's a pause that you can get out of by solving the problem. It's a pause you can get out of by developing the right safety measures. And so it incentivizes you to develop the right safety measures, and in fact, incentivizes you to avoid ever having to pause in the first place by proactively developing the, the, the, the right set of safety measures. And, and as we go up the scale, we may actually get to the point where you have to very affirmatively show the safety. The model where you have to say, you know, yes, like, you know, I'm able to look inside this this model, you know, with with with an X ray with with interpretability techniques and say, yep, I'm sure that this model is not going to engage in this dangerous behavior because, you know,

• 54:01Speaker 01

  there there isn't any circuitry for doing this, or there's this reliable suppression circuitry. So so it's really a way to shoehorn in a lot of the safety requirements, put them in the critical path of making the model. And and and hey, if you can be the first one to solve all of these problems, and therefore safely safely scale up the model, not only will you have solved the safety problems, but that kind of aligns the business incentives with with with the safety incentives. Our hope, of course, is that others will adopt the responsible scaling plan, and that eventually it can also be an inspiration for policy so that, you know, every so that so that so that everyone is held to some version of the responsible scaling plan. And, you know, how does it relate to to this this other thing of like, build as fast as we can?

• 54:45Speaker 01

  Well, look, I mean, one way to think about it is like, the responsible scaling plan doesn't slow you down, except where it's absolutely necessary. It only slows you down where it's like, there's a there's a critical danger in this specific place with this specific type of model, therefore you need to slow down. It says nothing about, you know, stopping at some certain amount of compute, or stopping for no reason, or stopping for a specific amount of time. It says, keep keep building until you get to certain thresholds. If you can solve the problems with those thresholds, then keep building after that. It's just that as the models get more and more powerful, you know, safety has to build along with the capabilities of the model. And our our hope is that

• 55:26Speaker 01

  if we do that and others do that, it creates the right culture internally at Anthropic, and it creates the right incentives for for, you know, the ecosystem and companies other than Anthropic. You know, I'm aware that since we published our responsible scaling plan. Several other organizations are internally working on responsible scaling plans. You know, for all I know, one of them, one or more of them might be out by, by the time this, this podcast is out. Hopefully they put out something. Hopefully they try and make it better than ours. Uh, that would, that would be a win for us.

• 55:56Speaker 02

  You have some aspects of your job, and you alluded to this earlier. 99% of your job probably looks mostly like a, a normal company would, but your time, I would guess is probably not 99%. How much of your time is spent on stuff that is, uh, weird for a normal startup CEO testifying in front of Congress or whatever that bucket is versus just day-to-day operations of running a business? Or is it hard to disentangle?

• 56:23Speaker 01

  Yeah. I mean, it's so, I, I don't know. I, I would say it's maybe 75, 25 or something like that.

• 56:28Speaker 02

  75 normal?

• 56:29Speaker 01

  75 normal, 25, 25% weird. I mean, it certainly takes a lot of time to, you know, to talk to large number of customers, to, you know, to hire for various roles, to, you know, look at financial metrics, to inspect the building, the models. I mean, that eats up a lot of time, but, you know, I also spend a decent amount of time, say, talking to government officials about what the future is going to look like, you know, thinking about the national security implications, trying to, trying to advise folks on, you know, what can go wrong. We did this whole project of, you know, working with some of the world expert biosecurity experts on, you know, what would it really take for the model to, to help someone to do something very dangerous?

• 57:12Speaker 01

  There are certain missing steps in bioweapon synthesis, um, for obvious reasons. I'm not going to go into what those steps are. I don't even, I don't even know all of them, but you know, we spent, spent a good number of months and, you know, decent amount of my, of my personal time, along with the incredibly hard work of the team, the team that worked on it, um, you know, thinking about this and, you know, presenting it to officials within our government, within other allied governments. And, you know, that's, that's just a pretty, pretty unusual thing to do, you know? I mean, that, that, that, that, you know, that, that feels something like something more out of a military thriller or something like that. Um, so, you know, that's, that's unusual. Speaking to Congress is unusual. Um, you know, thinking about where Where you know, like where we're gonna be in like, you know, three or four years like, you know,

• 57:56Speaker 01

  you know, are the models gonna, you know, like run rampant on the internet or something like that? Like, spend a good, good deal of time thinking about, you know, how do we, how do, how do we, how do we, how do we prepare for that scenario? You know, and another thing is like, you know, thinking about, you know, could, could at some point, you know, could, could the models at some point be morally significant entities, right? That's really wacky, really strange, like, I, I still don't know how, you know, how, how to be sure of that, or how, or, or, or, or, or how you'd measure that. It might be, you know, might be an important thing, it might not be an important thing. But, you know, we take it, we take it, we take it seriously. And so there is definitely this weird juxtaposition of like, you know, I'm, like, you know, looking, looking for a chief product officer one day and like, you know, thinking about

• 58:39Speaker 01

  bioweapons and, you know, you know, model moral status the next day.

• 58:45Speaker 02

  There was an Vox article that said something to the effect of, uh, an employee predicted there was a 20% chance that a rogue AI would destroy humanity within the next decade, uh, to the reporter, I guess that was around. Is, I mean, does all this stuff weigh heavily on the organization on a daily basis? Or is it mostly consistent with a normal startup for the average employee?

• 59:06Speaker 01

  Yeah, so I don't know, I'll give my own experience and it's kind of the same thing that I, you know, that I recommend, that I recommend to others. Uh, so, you know, I really freaked out about this stuff in 2018 or 2019 or so, when, you know, when I first believed that, you know, turned out to be, you know, at least in some ways correct, that the models would, would scale, scale very rapidly and, you know, they would have this, this, this importance to the world.

• 59:31Speaker 02

  Was there a specific thing that made you realize that or that you saw? Or was it a bunch of things coming together?

• 59:36Speaker 01

  Playing with GPT-2 is what, is what did it for me, it's what made it real. I mean, GPT-3 was more so and, you know, Claude and GPT-4 are, are, you know, of course even more impressive. But like, the moment where I, I really kind of believed the scaling trends that, that, that we had been seeing, that, you know, was really real and would lead to real things was like, the first time I looked at GPT-2, I was like, oh my, this is like, this is crazy. This is, you know, there's, there's nothing like, there's nothing like this in the world. Like, it's crazy that this is possible.

• 1:00:02Speaker 02

  In particular, the jump between the prior version to that and seeing that.

• 1:00:06Speaker 01

  Yeah, the the it was the delta and it was just the things that it was capable of like it felt like a general. It felt like a general induction or general reasoning engine. For years after that, people said models couldn't do reasoning. I looked at G P T two and I'm like, yeah, you scale this thing up. It it it's really going to be able to see see any pattern and reason about reason about anything. I mean, again, we still don't know that for sure. This is still unsure. There are still people who say it can't and for all I know, they're right. But uh, that was kind of the moment that that I that that I saw it. And so, you know, I had a very, very difficult year or so where, you know, I tried to, you know, come. Where I tried to come to terms with what I believe to be the significance of it.

• 1:00:47Speaker 01

  Now, five years later, I'm much more in a position where it's like this is this is the job we got to do, right? Where, you know, you sign up to do this, you have to be a professional and you have to address risk in a sensible way. And, you know, I found it useful to, uh, you know, to think about the strategies used by people who professionally address risk or deal with dangerous situations, right? People who are on, you know, military strike teams, people in the intelligence community, people who kind of, you know, deal with, you know, high, you know, like, um, high stakes, uh, you know, critical decisions for, you know, for national security or disaster relief or something like that. I mean, you know, doctors, surgeons, you talk to all these people and like, you know, they have, they have, you know,

• 1:01:31Speaker 01

  they have techniques for thinking of these decisions rationally and, you know, making sure that they don't get caught up in them too much. Um, and so I, you know, I I I I kind of try to adopt those techniques and I've told other people in the org to think to think in that way as well.

• 1:01:45Speaker 02

  You had made a comment that you don't like the concept of personalizing, uh, companies in this whole, hey, the memification of a C E O in some regard. Is that is that just a personal tact to who you are? Do you think it's actually like a societal issue that?

• 1:02:00Speaker 01

  Yeah, I mean, it's definitely my personal style, but, you know, I think this is closely connected to, you know, the thing about thing about Twitter, like, you know, I think people should think about companies and the incentives they have. Actual substance of the decisions they make. Nine times out of 10, you know, if someone seems kind of, you know, I don't know, charming or relatable or, you know, you know, you talk to them on Twitter and it seems like there's someone who, you know, you could sit down with them and really like, you know, that's, that could just be very misleading, right? It's, you know, it's not, it's not necessarily a bad sign, but I think it's pretty uncorrelated to like, what, what is the actual effect that the company's having in the world? Um, and I think people should focus on that and kind of, we've tried to focus on that in terms of the structural elements, right?

• 1:02:43Speaker 01

  The, you know, I'm not the only one who is ultimately responsible for these decisions. The L T B T is kind of designed as this, as this check, as this supervisory body. So, you know, everyone doesn't have to look, what's Dario going to do in this situation? And then Anthropic is only one company within a space of many other companies. There are also government actors there, and, and this is, this is the way it should be. No one person, no one entity should, you know, should have too much say over any of this. Um, I think, I think that's always unhealthy.

• 1:03:13Speaker 02

  We've talked about a lot of the negative sides or implications that come around with running an Anthropic. I'm sure there's some positive ones, other than being in the eye of the nucleus or storm or all the stuff that's going on today. Do you have any, uh, weird data points on like number of applicants or like the inbounds you get or all that?

• 1:03:33Speaker 01

  Yeah, I mean, we can talk about, you know, amazing positive stuff in the short term, and I'm also excited about positive stuff in the long term. So maybe, maybe let's take those one by one and, you know, I think, you know, if, you know, I think we should talk more about the positive stuff. I mean, I often see it as my duty to make sure people are aware of the concerns in a responsible and sober way, but that doesn't mean I'm not excited about, about all the potential I am. So speaking about, you know, Anthropic in particular, I mean, you know, millions of people have signed, have signed up to use Claude. Um, uh, thousands of enterprises are, are thousands of enterprises are using it and, you know, smaller, smaller number of very large, uh, very, smaller number of, of, of, of very large players have started to adopt it.

• 1:04:17Speaker 01

  So, you know, I've just, I've just been excited by some of the use cases, like, you know, when we look at, you know, Particularly legal, financial, um, things like counting. Uh, when you know, when you see suddenly people are able to talk to documents, right? You can just, uh, you know, you can just upload a company's financial documents. You can just upload a legal contract and ask questions that, you know, you would have needed a human to spend many hours on. Uh, and so, you know, this, this is just in a very practical way. This is just like saving people's time and providing them with services that, that they just, it would be very difficult for them to have otherwise. So I don't know, it's, it's hard not to be excited by that. And of course, excited by, you know, all the, all the amazing things the technology can do.

• 1:04:59Speaker 01

  I mean, you know, I know of someone who was like, you know, used it to like translate math papers from Russian. And, you know, they were good enough that, you know, it all, it all made sense. And they were, they were able to, to kind of, uh, you know, they were able to understand something that would have been very difficult for them to understand it before. Um, in, in the long run, I mean, I'm even more excited. I mean, I've talked about this a little before, but you know, having been in biology and neuroscience. I'm very convinced that the limiting factor there was that the basic systems were getting too complicated for humans to make sense of. Um, if you look at the history of science, things like physics, there's very simple principles in the world.

• 1:05:42Speaker 01

  You know, we, we managed to, to, to solve those because, I mean, physics is not fully solved, but you know, many, many parts of, of the basic operation of our world, we understand. And then within biology, things like, you know, viral disease or bacterial disease. It's very simple, you know, there's, there's something invading your body. Um, you, you know, you need to find some way to like kill the invader without hurting yourself. And because you and the invader are pretty different biologically, it's not that hard. So we've solved the problem. What's left is things like cancer, Alzheimer's disease, the aging process itself. Um, you know, to some extent, things like heart disease. And, you know, I, I worked on, you know, I worked on some of those things in, in, you know, in my, in my, my career as a biological scientist.

• 1:06:28Speaker 01

  And just, just the complexity of it, right? It's like, you know, you're trying to understand. You know how how proteins, you know, build cells and how the cells get dysregulated. It's like, you know, there's like thirty thousand different proteins and each one of them has like, you know, twenty different post-translational modifications and each of those interacts with the other proteins in this like really complicated web, you know, that that, you know, makes one cell run and that's just one type of cell. And there's like hundreds of other type of cells. Uh, and and so one of the things we've already seen with the language models is that they know more than than you or I do, right? A language model, you know, they know about the history of samurai in Japan. At the same time as they know about the history of cricket in India, you know, at the same time as, you know,

• 1:07:12Speaker 01

  they can, they can tell you something about, you know, uh, you know, like, you know, the biology of the liver or something like that, like you list enough of these topics and there's no one on earth who knows, who has, who has that breadth, even to the level that even to the level that a language model does, even with all the things that it says wrong right now. And and so my hope is that in terms of biology, right now, we have this network of like, you know, thousands of experts who all have to work together. If you can have one language model that can connect all the pieces and, you know, not just kind of like big data will help biology. That's not my thesis here. My thesis is that they'll be able to do and work along with the humans.

• 1:07:55Speaker 01

  A lot of things that human biologists, human medicinal chemists do and and really track the complexity and and be a match for the complexity of, you know, these these disease processes that are happening that are happening in our body. And so I'm hopeful that we'll have, you know, another kind of renaissance of medicine, you know, like we had in the late nineteenth century or early twentieth century, when, you know, all these diseases, we didn't know how to cure. We're like, oh, we discovered penicillin, penicillin, we discovered vaccines. So, you know, I I'll I'll take cancer as one, like, like, you know, any, any biologist or medicinal chemist, you know, who I said, could we cure cancer in five years? They'd be like, that's, that's fucking insane. There's so many different types of cancers, these breakthrough, you know,

• 1:08:38Speaker 01

  we have all these breakthroughs that handle one really narrow type of cancer. I think if we get this AI stuff right, then we maybe we could really do that. So I I I know it's hard to be hard to be more inspiring than that.

• 1:08:49Speaker 02

  Yeah, totally. You said you've been right about a lot of things related to AI, but you've also been wrong and surprised by a bunch. But what have you been most wrong about or surprised by?

• 1:08:59Speaker 01

  Yeah, I don't know. So I I mean, I've been wrong about I've been wrong about a bunch of stuff like I think how this prediction stuff works is like. If if you're thinking about the right things, if you're predicting the right things, you know, you only have to be right about like twenty twenty percent of stuff for. You know, for it to have these huge, huge consequences, right? If you if you if you predict five things that like. No one in the world thinks is going to happen and would have enormous consequences and and you're right about one of them. I mean, it's a it's a little bit like VC, right? You know, it's like, you know, if you if you, you know, if in in in nineteen ninety nine, you invested in Google. And for for companies that no one heard of, um, that's a pretty good portfolio, end up being wrong about about lots of stuff. So like one example of. That, uh, I don't know, I could come up with a few examples, but but one is.

• 1:09:43Speaker 01

  Uh, I thought certainly going back in like twenty nineteen or so, when you know, I first kind of saw the the scaling situation. I thought that we were going to scale for a while with these pure language models. And then what what we needed to do was immediately start working on agents acting in the world, not necessarily robotics, but like. You know, there had been all this stuff on Go.

• 1:10:08Speaker 01

  Uh, Starcraft, Dota, these other video games, all of which used reinforcement learning. Before the era of the large language models. So I thought we were going to put the two together almost immediately. And then almost all the training by by now, by twenty twenty three, twenty twenty four was was going to be. Uh, you know, the these these these large language models that were already as big, you know, already as big as they could usefully be made. Would would uh would kind of act in the world. Uh, but we found instead is we've just kept scaling the language models. And I, you know, I still think all the RL stuff is going to be promising. It's just we haven't gotten to it because it's in the lowest hanging fruit. Because it's simpler to just spend more money to make to make these models bigger than to design design something new. It's. It's completely economically rational.

• 1:10:52Speaker 01

  And and they the models just keep just keep getting better and better. Um, which, you know, I didn't didn't doubt that they would get uh didn't doubt that they would get better. But I I I guess I imagine that things would happen in a little bit of a different order.

• 1:11:04Speaker 02

  Do you think data will be a scaling issue in the near term?

• 1:11:07Speaker 01

  Yeah, so um I think there's actually some chance. Um, I would say there's a 10% chance that we get blocked by data. Um, the reason I mostly don't think it is, you know, the the deeper you look on, you know, the Internet's a big place. And the deeper you look, the more high quality data you find. And this is without even getting into kind of licensing of private data. This is just, you know, kind of, you know, this is this is just publicly available data. Uh, and then there are a bunch of promising approaches, which I won't get into detail about for how to make synthetic data. Uh, and, uh, you know, then, again, I can't get into detail, but we thought a lot about this. And I I bet the other L L M companies have thought a lot about it as well. And I would guess that one of those two, at least one of those two paths, very likely to pan out.

• 1:11:52Speaker 01

  But it's not it's not a slam dunk. Um, I don't think we've proven yet that this will work at the scale we need it to work. That this will work for a, you know, a $10 billion model that, you know, that that needs, God knows how many trillion. Uh, God knows how many trillion words fed into it, real or synthetic.

• 1:12:10Speaker 02

  Those numbers are so big for people and the amount of money that uh is being spent to train these models. Um, where for the average person listening, like, where does all that money go into? Uh, and how how should they think about like the need over time to continue to iterate on this?

• 1:12:29Speaker 01

  Yeah, so, you know, what I'll say is, at least to my knowledge, no one has trained a model that costs billions of dollars today. Um, people have trained models that cost, I think, of order $100 million. Um, but I think billion dollar models will be trained in 2024. And my guess is in 2025, 2026, several billion dollar, maybe even $10 billion models will be will be trained. There's, you know, there's enough compute in the industry and enough ability to do data centers that that's possible. And, you know, I think I think it will happen, right? If you look at what Anthropic has has raised. You know, has raised so has has raised so far, at least it's been publicly disclosed. Um, you know, we're at we're at roughly five point five billion dollars or so. We're not going to spend that all on one model.

• 1:13:14Speaker 01

  Um, but you know, we certainly, we certainly are going to spend, you know, multiple billion dollars on training a model sometime in the next sometime in the next in the next two or three years. Where does that go? It's almost all compute. It's almost all GPUs or custom chips. Um, uh, and, uh, you know, and and the data center and data center that surrounds them. Eighty to ninety percent of our cost is capital and almost all our capital cost is compute. Um, you know, the the number of people necessary to train these models, the number of engineers and researchers is growing. Uh, but it's the cost is absolutely dwarfed by, uh, by by, you know, is dwarfed by the cost of compute.

• 1:13:53Speaker 01

  You know, of course, we also have to pay for like the buildings people work in, but you know, that that again is some some tiny fraction of of of what the cost of compute is.

• 1:14:01Speaker 02

  Maybe ending on a, uh, on an optimistic note here, and we touched on a bunch of like the potential medical breakthroughs and things like that, but why should people be optimistic about what Anthropic's doing about the future AI and everything that's going on?

• 1:14:16Speaker 01

  Yeah, I don't know. So I'd answer the question in two ways. I mean, one, I'm optimistic about solving the problems. I mean, I am getting super excited about the interpretability work. Like, people didn't necessarily think this was possible. I still don't know whether it's possible to, you know, to really do a good job interpreting the models, but I'm very excited and very pleased by the progress we've made. I'm also excited about just, you know, the wide range of ways we've been able to deploy the model safely, like the wide range of of of happy customers who who who who just say, you know, this this model has been able to solve a problem that we had, it solved it reliably. We haven't had, you know, all we haven't had all these safety problems where we've managed to solve them. We've deployed something safely in the world. It's being used by lots of people.

• 1:15:01Speaker 01

  That's that's great. That's one level of great. And I think the second level of great is this this this this this this thing you you alluded to with like, you know, medical breakthroughs, mental health breakthroughs, like. You know, energy breakthroughs are already doing pretty well, but you know, I imagine AI can speed up material science very, very, very, very substantially. Um, so you know, I think, I think a, a, if, if we solve all these problems, I think a world of abundance really is a reality. I don't think it's utopian, given what I've seen that the technology is capable of. And you know, of course, there are people who will look at the flaws of where the technology is right now and say it's not capable of those things. And they're right, it's not capable of those things today.

• 1:15:43Speaker 01

  But if the, if the scaling laws that I'm talking about really continue to hold, then I think, I think we're going to see some, some really, some really radical things. Um, you know, one of, one of the things, you know, it's not a, not a complete trend, but you know, I think as, as we, as we gain more, you know, mastery over ourselves, our own, our own biology. Um, you know, the ability to manipulate the technological world around us. Uh, you know, I have some hope that that will also lead to a, you know, to a, a, a, a kinder and more moral society. Um, uh, you know, I think, I think in many ways it, it has in the past, although not uniformly.

• 1:16:23Speaker 02

  Why don't you like the term AGI?

• 1:16:25Speaker 01

  So I like the term AGI 10 years ago, um, because you know, no one was talking about the ability to do general intelligence 10, 10, 10 years ago. And so it felt like kind of a useful concept. Um, but, but now I actually think ironically, because we're much closer to the kinds of things AGI is pointing at. It's sort of no longer a useful term, you know, it's, it's, it's, you know, it's a little bit like if you see some object off in the distance on the horizon, you can point at it and give it a name. But you get close to it and you know, it turns out it's like a big sphere or something. And, and you're standing, you're standing right under it. And so it's no longer that useful to say this sphere, right? It's, you know, it's, it's basically, it's kind of all around you and it's very close.

• 1:17:07Speaker 01

  And, and it actually turns out to, to denote things that are quite different from one another. Um, so, so one thing I'll say, I mean, I, you know, I said this on a previous podcast, I said, I think in two to three years, the

• 1:17:22Speaker 01

  modalities and tools that we add are going to be at the point where they're as good at human professionals at kind of a wide range of knowledge work tasks, including science and engineering. Um, I, I definitely, that, that would be my prediction. I'm not, I'm not sure, but I, I think that's going to be the case. And, you know, when people, people kind of like commented on that or put that on Twitter, they said, oh, Dario thinks AGI is going to be two to three years away. Um, and, and so that, that then conjures up these image of, you know, there's going to be swarms of nanobots building Dyson spheres around the sun in, in two to three years. And like, of course this is absurd. I, I, I don't necessarily think that at all.

• 1:18:00Speaker 01

  Um, again, the, the specific thing I said was, you know, there are going to be these models that are, that are able to, able to on average match the ability of, of human experts in a wide range of things that they can do. There's so much between that and, you know, the super intelligent god, if, if that latter thing is even, is even possible or even a coherent concept, which it may be or it may not be. You know, one thing I've learned on the business side of things is that there's a huge difference between a demo of a model can do something versus this is actually working at scale and can actually economically substitute. There's so many little interstitial things that's like, oh, the model can do 95% of the task.

• 1:18:42Speaker 01

  It can't do the other 5%, but it's not useful for us unless, you know, unless we're able to substitute in AI end to end for the process. Or it can do a lot of the task, but you know, there are still some parts that need to be done by humans and it doesn't integrate with the humans well. It's not complementary. It's not clear what the right interface is. And so there's, there's so much space between in theory can do all the things humans can and in practice is actually out there, out there in the economy as full coworkers for humans. And there's a further thing of like, can it get past humans? Can it, can it outperform the sum total of human, say, you know, scientific or engineering output? That's, that's like a, you know, that's, that's another point.

• 1:19:27Speaker 01

  That point could be, you know, could be like a year away because the model gets, is better at making itself smarter and smarter. Or it could be many years away. And then there's this further point of like, okay, you know, can the model like, you know, like explore the universe and set out a bunch of like, you know, von Neumann probes and, you know, build Dyson spheres around the sun and, you know, Calculate the meaning of life is forty-two or whatever. Um, you know, that's, that's like a, that's, that's like a further point that also raises questions about, you know, what's practical in an engineering sense and in all of these kind of weird, weird things. So that's, that's like another further point. It's possible all of these points are pretty compressed together because there's like a feedback loop, but it's possible they're very far away from each other.

• 1:20:09Speaker 01

  Um, and, and so there's this whole unexplored space of like, you say the word AGI, you're like referring, you're smushing together all of those things.

• 1:20:18Speaker 01

  I think some of them are very practical and near term. And then I have a, a hugely hard time thinking about like, does that, does that lead very quickly to, to all the other things? Or, you know, does it lead after a few years? Or are, are those other things like not as coherent or meaningful as we think they are? I think all, I think all of those are possible. So it's, it's just, it's just kind of a mess. We're just, we're kind of flying very fast into this, this, this glob of concepts and possibilities. And, and we don't have the language yet to separate them out. We just say AGI and I don't know, it's, it's just kind of a, it's, it's like a, it's like a buzzword for a certain, certain community or certain set of science fiction concepts. When, when really we kind of, it, it's pointing at something real,

• 1:21:03Speaker 01

  but it's pointing at like twenty things that are very different from, from one another. And we, we badly need language to actually talk about them.

• 1:21:10Speaker 02

  What do you think happens on the next major training run for LLMs?

• 1:21:14Speaker 01

  Um, so my guess would be, you know, nothing truly insane happens, say in any training run that, that, you know, happens in twenty twenty-four. I think all the, you know, all the, you know, the stuff, the good and bad stuff I've talked about, you know, to really invent new science, the ability to, to cure diseases.

• 1:21:34Speaker 02

  Dyson nanowatts.

• 1:21:35Speaker 01

  To make bio, yeah, the ability to make bioweapons. Yeah, and maybe someday that the Dyson spheres. The, the least impressive of those things, I think, you know, will happen, you know. You know, I would say no sooner than 2025, maybe 2026. Um, I think we're just gonna see in 2024 crisper, more commercially applicable versions of the models that exist today. Like, you know, we've seen a few of these generations of jumps. I think in 2024, people are certainly gonna be surprised. Like, they're gonna be surprised at how much better these things have gotten. But it's, it's not gonna quite bend reality yet, if you, if you know what I mean by that. Uh, I, I, I think we're just gonna see things that are crisper, more reliable, can do longer tasks.

• 1:22:19Speaker 01

  Of course, multimodality, which we've seen in the last, uh, you know, we've seen the last few weeks from multiple companies, is gonna play a big part. Ability to use tools is gonna play a big part. Uh, so, you know, generally, these things are gonna become a lot more capable. They're definitely gonna wow people. Uh, but this reality bending stuff I've, I'm, I'm talking about, I don't expect that to happen in 2024.

• 1:22:42Speaker 02

  How do you think the analogy of, uh, versus a brain breaks down for large language models?

• 1:22:47Speaker 01

  Yeah, so it's, it's actually interesting. This is, this is one of the, you know, being a former neuroscientist, this is one of the, the mysteries I still wonder about. So the general impression I have is that the way that the models run and the way they operate, um, I don't think it's all that different. Um, you know, of course, the physiology, all the details are different. But I don't know, the basic combination of linearities and nonlinearities, the way they think about language, to the extent that we've looked inside these models, which we have with interpretability. I mean, we see things that would be very familiar in, you know, the brain or a computer architecture. You know, we have these, you know, we have these, we have these registries, we have variable abstraction, we have neurons that fire on different, different concepts.

• 1:23:32Speaker 01

  Um, again, the alternating linearities and, and, and nonlinearities. And just, just interacting with the models, they're not that, you know, they're not that different. Now, what is incredibly different is how the models are trained, right? The, if you compare the size of the model to the size of the human brain in synapses, which of course is an imperfect analogy, uh, but there's some Something like still maybe a thousand times smaller. And yet they see maybe a thousand or 10,000 times more data than the human brain does. If you think of, you know, the number of words that a human hears over their lifetime, it's a few hundred million. Uh, if you think of the number of words that a language model sees, you know, the, the, the, the, the latest ones are in the trillions

• 1:24:16Speaker 01

  or maybe even tens of trillions. Uh, and, and that's just, you know, that's like a factor of like 10,000 difference. Uh, so it's, it's as if we've kind of, you know, that, that neural architectures have some, you know, there's lots of variants to them, but they have some universality to them. Um, but that somehow we've climbed the same mountain with the brain and with neural nets in some very, very different way, according to some very, very different path. And so, you know, we get systems that when you, when you interact with them are, you know, I mean, there's still a hell of a lot they can't do. But I don't see any reason to believe that they're, you know, fundamentally different or fundamentally alien.

• 1:24:57Speaker 01

  But what is fundamentally different and what is fundamentally alien is the completely different way in which they're trained.

• 1:25:03Speaker 02

  You said that alignment and values are not things that will just work at scale. And we've talked about constitutional AI and some of the different, um, viewpoints there. But can you extrapolate on that view?

• 1:25:14Speaker 01

  Yeah, I mean, this is a bit related to the point that I said earlier about, uh, you know, that, that there's kind of this fact value distinction, right? You cram a bunch of facts into the model, you train it on, you know, everything that's present, that's present on the internet. And, and it kind of leaves this, this blank space or this, this, this undetermined variable. Um, I, you know, I, I, I basically just think that like, you know, it's, it's up to us to, you know, to determine the values, the personality, especially the controllability of these systems. There's another sense in which I would say this, which is just that, you know, that, that naturally these are statistical systems and they're trained in this very indirect way, right?

• 1:25:59Speaker 01

  Even, even the constitution, it's like the constitution is pretty solid. But then the actual training process uses a bunch of examples. Samples, it's kind of opaque. And of course, the the part where you put in place tens of trillions of words like no human ever sees that. So it's it's still, I think, very opaque and and and hard to track down. And and so, you know, I think it's I think it's very prone to to failures. And, you know, this is why we focus on interpretability, steerability and and reliability. We we really want to kind of tame these models. Make sure that you're able to control them and that they do what humans want them to do.

• 1:26:39Speaker 01

  Uh, I I don't think that comes on its own, you know, any more than like that comes on its own for for airplanes, right? The early airplanes like, you know, probably they wouldn't crash every time you fly them. But like, you know, I wouldn't wouldn't want to, you know, I wouldn't want to get in the Wright Brothers plane every day. Um, and just bet that every day it would not crash and would not kill me. Um, it's it's just not, it's not safe to that standard. And I think today's models are basically like that.

• 1:27:03Speaker 02

  Why is mechanistic interpretability so so hard to do?

• 1:27:09Speaker 01

  Yeah. So, you know, mechanistic interpretability is this, uh, you know, it's it's an area that we work on, which is basically trying to look inside the models and, you know, and and kind of analyze them like like like an X-ray. Um, and I think the reason it's so hard, it's actually the same reason why it's hard to look inside the brain, right? The the brain wasn't designed to have humans look inside it. It was, you know, it was designed to serve a function, right? The the the the interface that's accessible to to other humans is, you know, your your speech, not not, you know, the actual neurons in your brain, right? They're they're not designed to be read in that way. Um, of course, the advantage of reading them in that way is, you know, it's it's it's you you get something that's

• 1:27:52Speaker 01

  much closer to a ground truth, not a perfect ground truth, but, you know, if if I really understood how to look in your brain and understood uh, uh, and understood what you were thinking, you know, it would be much harder for someone to to deceive someone else about about their intentions, um, or for behaviors that might emerge in some new situation to to not to not to not be evident. Um, so there's a lot of value in it. But but yeah, there's you know, there's there's nothing in both the case of the brain and in the case of the large language models. You know, they're they're not designed or trained in a way that makes them easy to look at. So you know, it's a little, it's a little bit like, you know, we're inspecting this alien city. That wasn't built to be understood by humans. It was built to function as an alien city. Uh, and so we might get lucky, we might get clues, we might be able to figure it out.

• 1:28:37Speaker 01

  But there's kind of no guarantee of success. We're we're on our own. Uh, and and so that's what we do. We kind of do our best. That said, I am becoming increasingly optimistic that interpretability. Uh, can be, I don't know about fully solved, but that it can be. An important guide to showing that models are safe. And and even that it will have commercial value. In, you know, kind of in the areas of like trust and safety or classification filters or moderation. Fraud detection. I think there's even legal compliance aspects to interpretability. So my co-founder, Chris Ola. Has been working on interpretability. He's run a team at Anthropic for the last two and a half years.

• 1:29:19Speaker 01

  Before that, when we were at OpenAI, he ran a team that worked on interpretability of vision models for three years before that. And for that entire period. Uh, it's been, it's been just basic research, right? There's been no, you know, commercial or business application. You know, Chris and I have just kept it, kept it going. Because we we believe that this is something that will. Pay off from a safety perspective and maybe even from a business perspective. And now actually, for the first time. Um, you you know, you you will, you will, you you will see by the time this podcast comes out. Uh, we're, we're, we're releasing something. That shows that we've really been able to solve something or. Make good progress towards solving something called the superposition problem.

• 1:30:01Speaker 01

  Um, which is, which is that if you look inside a neuron, it corresponds to many different concepts. We found a way to disambiguate those concepts. So that we can see all the individual concepts that are lighting up inside, inside a. Inside one of these large LLMs. It's not a solution to anything, to everything. But it's just a really big step forward. And for the first time, I'm optimistic that, you know, give us two or three years. I don't know for sure, but we might actually be able to get somewhere with this.

• 1:30:30Speaker 02

  And depending on your level of understanding of all this stuff, why would that be important for safety?

• 1:30:35Speaker 01

  Yeah, so I would go back to the X-ray analogy there, right? You know, if I can really look inside your brain, you know, if I, if, if, if, if, if I can say this is what's happening. I mean, you know, I can ask you questions and like, you know, you can say things that sound great. And, you know, I've, I've no idea if you're telling the truth or if it's all just bullshit, right? But if I, if I, if I look inside your brain and I have the ability to understand what I'm seeing, then it, then it becomes much, it becomes much harder to be misled. Similarly, with language models, you know, I can test them in all kinds of situations and it'll seem like they're fine. The fear is always, oh, but you know, if I talk to the language model in this way, I could get it to do something really bad.

• 1:31:20Speaker 01

  Or if I put it in this situation, it could do something really bad on its own. That's always the fear, right? That's the fear we have every time we deploy a model. We've had a hundred people test it. We've had a thousand people red team it. But when it goes out into the world, a million people will play with it. And, and one of them will find something that's, that's truly awful. Uh, and you know, you know, we'll, we'll, we'll find, oh, well, if I use this trick for talking to the model, you know, it's, it'll, it'll, it'll, you know, it'll finally be able to produce that, that, that bioweapon. Or, oh, if I put it in this place where it has access to infinite resources on the cloud, it'll, you know, just self replicate itself infinitely. Uh, and, and so interpretability is at least one attempt at a method to address that problem.

• 1:32:04Speaker 01

  To say, okay, instead of thinking, instead of trying to test the model in every situation that it could be in, which is impossible, we can look inside it and try and decompile it and say, what would the model do in this situation? Well, we understand the, the algorithms that it's following. We understand what goes on in different parts of its brain, at least to some extent. So we can, we can pose the hypothetical and say, hey, you know, what would, what would happen in this, this whole? Part of the space, what would happen in this whole class of areas? And you know, if we can do that, we have some, we have kind of some ability to exclude certain behaviors to say, okay, we know the model won't do that, which, which you never have behaviorally, you know?

• 1:32:47Speaker 01

  You know, it's just, it's just like, it's just like humans, you know, you know, I'm like, you know, what, what would you do in a life threatening situation? I don't know what I would do, you know, in a, I don't know what I would do in a life threatening situation. I don't know what you would do in a life threatening situation. It's hard to know until you're actually in the situation, but if I knew enough about your brain, I might be able to say.

• 1:33:04Speaker 02

  What's your view on open source models?

• 1:33:06Speaker 01

  Yeah, I mean, that's a, you know, that's a, um, obviously a complex topic. I mean, I think as with many things in, as with many things in, you know, in, in, in AI versus the rest of technology, you know? From a normal technological perspective, I'm extremely pro open source. Like I think, you know, it's accelerated science, it's accelerated innovation, it allows, you know, errors, errors to be fixed faster and development to happen faster. And I certainly think, you know, for the smaller models, for the smaller open source models, uh, this is, this is, this is true for AI as well, and I don't see much danger to smaller models. Therefore, I think open source, as it's being practiced by every open source model that's been released up to this point, seems perfectly fine to me. Um, my worry is more around the large models.

• 1:33:52Speaker 01

  Um, and my worry in particular is that, you know, these models that are offered via API, and I mean, I'm talking about models of the future that really are dangerous, not, not models in two or three years, maybe one year, not, not today's, uh, not, not, not, not today's models. Uh, if they're offered by API, or even if you have fine tuning access to them, there's a lot of levers that you have to control the behavior of the model, right? You can put in your constitution, don't produce bioweapons, right? And then if the model does it anyway, you can, you can basically make changes to the model. You can say, okay, we're just retracting that version and serving a new version of our model that patches a particular hole. You can monitor users. So if a million people are using the model and within that,

• 1:34:35Speaker 01

  There's these five bad actors in this terrorist cell. You can use your trust and safety team to identify the terrorist cell, cut them off and even call law enforcement if you want to do it. Um, so it, it, it really provides an ability. You don't have to get things right in the first time. And if something dangerous happens, you can, you really have the ability to fix it. With, with models where their weights are released, uh, you, you don't, you don't have any of that control, right? The minute you release the, the, the, the model, all, basically all of this control is lost. And so that's our concern. That, that doesn't mean, by the way, that large open source models shouldn't exist. But the way we put it in our responsible scaling plan is we say, okay, when models get to the, to the level where they're smart enough to create these dangerous capabilities.

• 1:35:19Speaker 01

  And the next one for us is ASL three. We're at ASL two right now. Uh, then models have to be tested for dangerous behavior according to the complete attack surface, according to which they're going to be released in reality. So if you're just releasing an API, then you have to test that, you know, you can't build a bioweapon with, with the API. If you're releasing the model with an API and fine tuning, then the people who are testing the model have to, have to mock up the test with the fine tuning. If the model's being released in practice, then the right test to run would be, I'm a mock terrorist cell. I'm, I, you know, uh, I get the weights released to me. I can do anything I want with those weights.

• 1:36:00Speaker 01

  Um, is there some way to release, release the weights of the model so that they can't, they can't be abused? I, I think there might, might very well be, but I think people who want to release model weights have to confront that problem and have to find a solution to that problem. I'll say, by the way, because there's a person on my team who, uh, who this is, this is one of their pet peeves. Um, the word open source, I don't think is necessarily appropriate in the case of all of these models. I think it is appropriate in the case of like, you know, small developers and companies where kind of their whole, their whole business model is about, is, you know, i- i- their whole business model is about open source. But when much larger companies have released the weights of these models, they generally have not released them under open source.

• 1:36:45Speaker 01

  Under open source licenses, they've generally asked people to pay them when they use them in commercial ways. So I would think of this as, you know, less open source and more that model weight releases a particular business strategy for these for these for these large companies. And again, I'm not saying that that model, you know, model weights can't be released. Um, you know, I'm saying that the, you know, the tests for them need to be commensurate with the issues and that we shouldn't automatically say, oh, open source, open source is good. Some of these are not open source. They're they're the business strategies of large companies that that involve releasing model weights.

• 1:37:21Speaker 02

  Paul Cristiano recently on a podcast said he thinks there's a 50% chance. I think the way he phrased it was that his, the way he he ends up passing away is something to do with AI. Do you think about percentage chance doom?

• 1:37:39Speaker 01

  Yeah, I I think it's popular to give these percentage numbers and and, you know, I mean, the truth is that I'm not, I'm not sure it's easy to put to put a number to it. And if you forced me to, it would it would fluctuate all the time. Um, you know, I think I've, I think I've often said that, you know, my my chance that something goes, you know, really quite catastrophically wrong on the scale of of, you know, human civilization, you know, it might be somewhere between 10 and 25%. When you put together the risk of something going wrong with the model itself, with, you know, something going wrong with human, you know, people or organizations or nation states misusing the model or or

• 1:38:20Speaker 01

  it kind of inducing conflict among them, or or just some way in which kind of society can't can't handle it. That that said, I mean, you know, what that means is that there's a 75 to 90% chance that this technology is developed and and and and everything goes fine. In fact, I think if everything goes fine, it'll go not just fine, it'll go really, really great. Um, again, this stuff about curing cancer, I think if if we can avoid the downsides, then this stuff about, you know, about curing cancer, extending the human lifespan, um, you know, solving problems like like mental illness, I mean, this all. This all sounds utopian, but I don't think it's outside the scope of what the technology can do.

• 1:39:00Speaker 01

  So, you know, I, I, I often try to focus on the seventy-five to ninety percent chance where things will go right. And I think one of the big motivators for reducing that ten to twenty-five percent chance is, you know, how, how great it'll, you know, i- is trying to increase, is trying to increase the good part of the pie. Um, and I think the only reason why I spend so much time thinking about the ten, that, that ten to twenty-five percent chance is, hey, it's not going to solve itself. You know, I think the good stuff, you know, companies like, like ours and like the other companies have to build things. But there's a robust economic process that's leading to the good things happening. It's great to be part of it. It's, you know, it's, it's great to be one of the ones building it and causing it to happen. But there, there's a certain robustness to it.

• 1:39:44Speaker 01

  And, you know, I find, I find more meaning. I find more, you know, when, when this is all over, I think, you know, I personally will feel I've done more to contribute to, you know, whatever utopia results. Um, if we focus, you know, if, if I'm able to focus on kind of, you know, reducing that, that risk that it goes badly or it doesn't happen. Because I think that's not the thing that's gonna, that's gonna, that, you know, that's not the thing that's gonna happen on its own. The market isn't going to provide that.

• 1:40:11Speaker 02

  Do you worry more about the misuse, uh, by people misusing it or the AI themselves, or is it just different timelines?

• 1:40:18Speaker 01

  Yeah, uh, partially different timelines. I mean, you know, if I had to tell you, I would, I would say the, the misuse to me seems more concrete. Um, and I think, you know, will, will happen sooner, you know, hopefully we'll stop it and it won't happen at all. Um, I, you know, I think the, the AI itself doing something bad is, is also a quite significant risk. Um, it's a little off in the future and it's always been a bit more shadowy and vague. But that doesn't mean it isn't real. Um, I mean, you know, you, you just look at the rate the model is getting better and you look at something like Bing or Sydney. It really gives you a taste of like, hey, these things can really be out of control and, you know, psychopathic.

• 1:41:00Speaker 01

  And the only reason Bing and Sydney didn't cause any harm is that, you know, they were out of control and psychopathic in a very limited way, limited both in that, you know, it was confined to text and limited in that, you know, it just wasn't that smart, you know, tried to manipulate the reporter, tried to get him to leave his wife, but like, it wasn't really, you know, it wasn't really compelling enough to, you know, to get a human to fall in love with it, but someday maybe a model will be. Uh, and you know, maybe it'll be able to act in the world and then you put all those things together and you know, you know, I think there is, there is some risk there. I think it's harder to pin down, but I'm, personally, I'm worried about both things and, you know, I think, I think our job, because we see such a positive potential here is, you know, we have to,

• 1:41:45Speaker 01

  you know, we, we, we have to, we have to find all the, all the possible bad outcomes and like shoot them down. We have to get, we have to get all of them. And then if we get all of them, then, then, then, you know, then, then we can, we can live in a really great world, hopefully.

• 1:41:58Speaker 02

  Is, if you could wave your hands and have everyone follow a single policy, would it be a responsible scaling policy? Would everyone have one of those?

• 1:42:05Speaker 01

  Yeah, I mean, I think, you know, I think, I think if we make a constraint of like realism, right? Where, you know, it's like, you know, I, I in fact can't wave my hand and get, you know, everyone in, you know, China or Russia or somewhere else to, you know, stop building these powerful models. Um, you know, like there, there are just some levels of like world or international coordination that are just, are just, are just not going to happen because of, because of realism. So if you stipulate that like, so, you know, you can't, you can't, you can't just make everyone stop, or you can't just make everyone build in a certain way. The idea that, hey, you know, for most things, people should just be able to build what they want to build, but, but, you know, we're, we're coordinating off

• 1:42:47Speaker 01

  these, you know, these particular levels of capability, these particular points in the, in the, in the development curve, where something concerning is happening and say, hey, mostly do what you want, but you've got to take this, there's a small fraction of stuff you've got to take really seriously. And, you know, if you, if you don't take it really seriously, you're the bad guy. Um, you know, that's something that I think I can reasonably recommend that everyone's, that everyone sign on to that. There's some sacrifice to it. There's some loss to winning the race, but it's, it's, you know, it's only as much sacrifice as is, as is, as is needed. And because it's so targeted, I think you can make a strong moral case that, hey, if you don't do this, you're an asshole.

• 1:43:25Speaker 02

  How do you think about the trade offs between building in public and having people aware with what you're doing with on the flip side maintaining that secrets or that the appropriate things stay within the org?

• 1:43:37Speaker 01

  Yeah, yeah, I mean, you know, this is this is one of these, this is one of these kind of difficult trade offs, right? So, you know, definitely an org benefits from, you know, kind of everyone knowing about everything. But on the other hand, as we've seen with multiple AI companies, you know, secrets, secrets leak out. And, you know, even just from a commercial perspective, forget safety, like with models built in, you know, the next year or two, let's say a model costs $3 billion. And you have an algorithmic advance that, you know, means you can build the same model for 1.1.5 billion dollars, right? These kind of 2X, 2X advances along the scaling curve have occurred in the past and, you know, may occur in the future.

• 1:44:21Speaker 01

  And, you know, companies, including ours, may, may, may, may be aware of such advances. So, so basically that's like three lines of code that's worth 1.5 billion dollars. You know, you, you don't want a wide set of people and you may not even want everyone within your company to know about them. And at Anthropic, at least, people have been very understanding about that. People don't, people don't want to know these secrets. People are, people, people are on board with the idea, hey, you know, it's not, it's not a marker of status that you know these secrets. These secrets, you know, should be known to the tiny number of people who are actually working on the relevant thing. Plus, you know, the CEO and a couple, a couple other folks who, you know, need, need to be able to put, put the entire picture together, right?

• 1:45:04Speaker 01

  This is, this is, this is kind of compartmentalization and, and, and, and need to know basis. And of course it has some costs because information doesn't propagate as freely. But, but again, you know, just as with the RSP, you know, let's, let's take the 8020. Let's take the little, the few pieces of information that are really, you know, that are really essential to protect and, and be as free as we can with everything else.

• 1:45:27Speaker 02

  If you hadn't gone down this AI path, would you be doing academic work right now?

• 1:45:33Speaker 01

  I I think that was my assumption, like that that was what I kind of always imagined doing. I imagined being a scientist and, you know, scientists work at universities. But the really interesting thing about the the, you know, this this AI boom is that to really be at the forefront of it. Uh, you know, you have to have these huge resources. Um, and, you know, I think I think the huge resources are are basically, you know, they're they're basically only available. Uh, at companies, uh, you know, first it was the large companies like like like Google, but, you know, more recently. You know, startups like ours had been able to raise raise large amounts of money. And so I I I was kind of drawn to that direction because it had the ingredients necessary to, uh.

• 1:46:16Speaker 01

  You know, to to build the things we wanted to build and study the things that we wanted to study as scientists. And, you know, I would say many of my many of my co-founders feel the same. Um, one thing, one of my co-founders who is a physicist, what he often, you know, what he often brings up is, and, you know, it's it's. It's kind of more a more a um, more an academic question because I don't think things are going to go in this direction. But, you know, he said, you know, in in in in in my field, we build these, you know, ten billion dollar telescopes in space. And, you know, we build these ten billion dollar particle accelerators. Why did the field go the way, why why why did the field, you know, kind of, kind of, kind of go in this direction instead? Right? Why didn't all the AI academics get together and, you know, build a, you know, build a ten billion dollar cluster?

• 1:46:58Speaker 01

  Why did it happen in, you know, in in startups and in large companies? Um, I don't really know the answer to that. Um, things could have gone the other way, but it doesn't doesn't seem like that's the way things have gone. And I, you know, I don't know if it's for the better or the worse. I mean, we've learned a huge number of things, uh, you know, by by by working with customers and seeing how these things impact the economy. So maybe the path things went is the best best path they could have gone.

• 1:47:25Speaker 02

  How did I actually don't know, how did they get access to capital in the in the prior or in the alternative way of doing that? I was it through government grants and stuff?

• 1:47:33Speaker 01

  Yeah, these large telescopes, they're often kind of like government consortia or private. You know, kind of, kind of, kind of large scale private philanthropy. It's, it's, it's honestly this huge patchwork mix. I, I'm surprised it even happens because if, if I think about it happening in this field, I, I just, I just, I just can't imagine how it would happen. But in these other fields, somehow they've made it work. Um, I, I don't actually know how, but so that I don't know, that's, that's just like, oh, that could have been like just a weird alternate history of our, of our, uh, of our industry. That didn't happen and you know, I, I, I very much doubt it's going to happen, although who knows? Um, you know, we, we went on, we went on this path instead and I don't know, there's a lot that's exciting and interesting about, about this path. And you know, one way or another, this is, this is the situation we're in.

• 1:48:19Speaker 02

  Has working with your sister been as fun and saving the world as you had hoped it when you were, when you were kids?

• 1:48:25Speaker 01

  Yeah, it's surprisingly similar to what, to what I imagined. I mean, you know, if, if you were to, if you were to look back on the things we were saying to, to, to, you know, to, to one another as like an adult observing it, you would have been like, this is crazy, this is crazy and you know, kids dream of course, but um, no, I mean, uh, you know, it's, it's, it's just, it's just amazing that we're able to work on this together.

• 1:48:45Speaker 02

  Very cool. Daria, thanks for doing this.

• 1:48:47Speaker 01

  Thanks for having me.