A conversation with OpenAI's CPO Kevin Weil, Anthropic's CPO Mike Krieger, and Sarah Guo

A conversation with Mike Kreiger (CPO of Anthropic) and Kevin Weil (CPO of OpenAI), moderated by Sarah Guo (Conviction). Recorded live at Lenny and Friends Summit on October 24th in San Francisco

• 0:22Speaker 0

  Sarah, you're the queen of AI investing.

• 0:24Speaker 2

  This is a phrase never ever to be used again, but it's great to be here with both of you. So I had two different ideas for our final discussion. The first was a product off, because these two men have the merge to di to prod button, both of them, and I was like, oh, please just release everything we know is coming over the next six or twelve months. Ignore all internal guidelines. The second was we just redesign Instagram together since they both actually run Instagram before.

• 1:01Speaker 2

  Lame, I know. But uh but really excited to hear from you both anyway. Uh so this is actually a relatively new role for both of you. Kevin, let's start with you. Like you've done a bunch of really different interesting things. Like, what was the reaction you got when you took the job from friends and the team?

• 1:18Speaker 0

  Uh generally excitement. I mean, uh it's uh I think it's one of the most interesting and impactful roles. There's so much to figure out. Um I've never had such a challenging, interesting uh sleepless product role in my life. Uh it's it's got all the challenges of a normal product role where you're trying to figure out who you're building for and what problems you can solve and things like that. But normally when you're building product, you're building off of kind of a fixed technology base, right? You know what you have to work with, and uh and you're trying to build the best product you can. Here it's like every two months computers can do something computers have never been able to do before in the history of the world, and you're trying to figure out how that changes your product, and the answer should probably be a fair amount.

• 2:10Speaker 0

  Um so it just it's so interesting uh and fascinating to see on the inside as AI gets developed. But uh I've been having a blast.

• 2:19Speaker 2

  Mike, what about you? I I remember hearing the news, I was like, ah, I didn't know you could convince the founder of Instagram to go work on something that existed already.

• 2:27Speaker 1

  Yeah, my favorite three reactions, like people who know me were like, oh, that makes sense, like you're gonna like have fun there. Uh the middle people were like, why? Like, you don't have to work, like why are you doing this? And like then if you knew me, you'd know me like I can't not, and I think that like I couldn't stop myself. And the third was like, oh, you could hire the founder of Instagram, which was also fun. And it's like, I mean, not many people could, but like there's like probably a list of three companies that would have been interesting. Um so yeah, there's like a range of reactions depending on how well you knew me and how like you've seen me in my like uh semi-retired state, which lasted like six weeks, and I was like, all right, what are we doing next?

• 3:03Speaker 2

  Um so we had dinner together with a bunch of friends recently, and you I was like impressed by the childish delight that you had around like, yeah, I'm learning about all this enterprise stuff. Like tell me if it's about serving customers that are not all of us with Instagram, or uh just you know, working in a uh an organization that's research driven, like what's the biggest surprise so far.

• 3:27Speaker 1

  I mean those are two, I think both very worthwhile pieces of this role that are like very new to me as well. Like I um when I was 18, I made this like very 18-year-old vow, which was like every year of my life I wanted to be different. Like I don't have the same year twice. And I was like, it's why like, you know, I didn't there's been times where I was like, oh, like another social product. I'm like doing that again. First of all, like your bar is like really distorted, and second of all, like it just would feel like too much of the same thing. So yeah, enterprise has been wild. I'm really curious about your experience with that as well. Like you're like, you know, uh your feedback loop, I actually imagine it's a lot more like investing, is far longer, right? You're like you have that initial convo, and you're like, I think they like me, and then you're like, oh no, it's now in like some requisition state, and it's gonna take like six months before they like even get to deployment before you know whether it's right. And so like getting used to that pace, we're like, what?

• 4:14Speaker 1

  Why isn't this shipped yet? They're like, Mike, you've been here two months, like this is like it's making its way through the VPs, like it's gonna get there eventually. So like getting used to different timelines for sure. But like the part that is fun is actually getting the feedback in the kind of engagement where you're like, once it gets deployed, you still have somebody that can call you and you can call them and be like, how's it working for you? Like, is this good? Like, whereas users, like you're doing like data science and aggregate, and like sure you can bring in like one or two people, but it's like they won't they don't have enough financial incentive writing on telling you where you suck and where you're doing well, and like that's been a different but also like rewarding side of that for sure.

• 4:50Speaker 2

  Kevin, you've worked on such a wide range of products before. How much do your instincts like apply?

• 4:56Speaker 0

  Yeah, I was gonna add on to the enterprise point too, uh, and then I'll get to that. Like the other interesting thing about enterprise is it's not necessarily about the product, right? There's a buyer, yeah, and they have goals, and you could build the best product in the world that all the people at the company might be happy to use, and it still doesn't necessarily matter. Exactly. Um I was in a meeting with one of our big enterprise customers, and they were like, um uh this is great, we're really happy, da-da-da. You know, the one thing we need is we really need you to tell us 60 days before you launch anything. And I was like, I also would like to know 60 days. So very, very different. Actually, and it's interesting, right?

• 5:41Speaker 0

  Because at OpenAI, we have a consumer product, and we have an enterprise product, and we have a developer product. So we're kinda doing all at once. Umise. In I'd say in like half the job, it works. You know, when you have when you have a sense of the product you're trying to build, you know, y we're getting towards uh you know the end of shipping uh advanced speech mode or something, or you're getting towards shipping Canvas and you're making final touches, trying to understand who you're building for and like what you know exactly what problems you're trying to solve. It it works then, because it's that's a little bit more like the tail end of it is shipping a normal product. But the beginning of these things is nothing like that.

• 6:27Speaker 0

  Um like there will just be these capabilities that we don't know uh you you have some sense as you're training some new model that it might have capability X. You don't really know. Nor does the research team, nor does anybody, right? You're like, I think this might be possible. And it's kind of like coming through the mist at you, but it's this emergent property of a model, and you know, you so you don't know whether it's gonna really work, and you don't know whether it's gonna be like 60% good or 90% good or 99% good. And the product that you would build that would make sense with something that works 60% of the time is super different than 90 or 99% of the time, right? So you're kind of just waiting, and you're you know, at least like I don't know if you feel this, checking in with the research team from time to time, like, hey guys, how's it going?

• 7:20Speaker 0

  How's that model training? Uh any any insight on this? And they're like, it's research, we're working on it, you know. It's uh we don't know either. We're we're we're we're working through this at the same time. And it's I mean it makes it super fun because you're kind of like discovering things together, but very sort of stochastic too.

• 7:41Speaker 1

  It's the thing it most reminds me of, like from the Instagram days where like Apple, like WWC announcements, you're like, this could either be awesome for us or could like absolutely like cause chaos for it. It's like that, but your own company is the one kind of disrupting you from within, which is like a very like it's very cool, but also like, oh, this might totally up in my product roadmap now.

• 8:01Speaker 0

  Yeah.

• 8:02Speaker 2

  Uh what does that cycle look like for both of you? You described it as um, you know, like peering through the mist, trying to look at the next set of capabilities. I mean, can you can you plan if you don't know exactly what is coming? And what is the iteration cycle to discover new things that should belong in your product?

• 8:19Speaker 1

  I think like on the intelligence side, you can sort of squint and see like, all right, it's advancing this way, and so the kinds of things that you'll want to do with the model and start building the product around that. So I there's three ways, right? Intelligence feels not predictable, but at least on like a slope that you can kind of watch. There's the capabilities you decide to invest in from the product side and then do fine-tuning with the actual research teams. Um artifacts, we spent a lot of time between research. I think the same was true with campus, right? Like you're doing a like co-design, co-research, co-fine-tune, and that's like I think a real privilege of getting to work at this company and getting to do design there. And then there's a capability front, so maybe speech mode for open AI for us is the computer use uh work that we released this week. You're like, all right, 60%, all right. Yes, all right. Uh and like so what we try to do is embed designers early in the process, but knowing that like you're not placing a bet like the the experimentation talk was saying, like your um your output for experiments should be learning, not necessarily like perfect products you're gonna ship every time.

• 9:16Speaker 1

  I think the same is true when you're partnering with research. Like your outcome is hopefully demos or informative things that like could spark product ideas, not like a predictable product process where you're like, well, it's this de-risked by now, which means it's gonna look that way when research comes along.

• 9:31Speaker 0

  I've also uh one thing that I've really enjoyed because uh research is at least parts parts of research are very product-oriented, especially on the post-training side, like Mike was saying. And then parts of it are really like academic research at some level. And so you will just also occasionally hear about some capability, and we'll be in a meeting and you'll be like, oh, I really wish we could do this thing. And a researcher on the team will be like, oh no, we can do that. We've had that for three months. And we're like, really? What does that mean? Like, okay, where do I learn more? And they're like, oh, well, we didn't think we didn't, we didn't know it was important. So, you know, I'm working on this other thing now. Um, but you do just get like magic happening sometimes too.

• 10:14Speaker 2

  Uh one thing like we think a lot about when we're investing is actually like can you do anything with a model if it is 60% successful at a task instead of 99%? And on like lots of tasks, it's closer to 60, right? But the task is really important and valuable. Like, how do you how do you think about that internally in terms of evaluating progression on a task and then what what types of things you like put in sort of the burden of product to make it graceful failure or to like sort of cross the miles of the user versus you know, we just need to wait for the models to get better.

• 10:49Speaker 0

  I'd argue there are a lot of things that you can actually do when something is 60% right, you just you just need to really design for it. Um you have to expect that there's a human in the loop a lot more than there would be otherwise. Like if you look at uh take like GitHub Copilot, right? That was kind of the first AI product that really opened people's eyes to like this thing can be useful not just as you know QA, but for really economically valuable work. And that launched, I don't know exactly which model that was built off of, but I mean it was multiple generations ago. So I guarantee you that model wasn't perfect at anything related to coding. I

• 11:24Speaker 2

  think it was GPT 2, which is like pretty small. So

• 11:27Speaker 0

  yeah, I mean, and so but the fact that it was still valuable for you, because if it got the code, you know, some significant fraction of the way there, that was still stuff you didn't have to type yourself and you could edit it. And so there are experiences like that that I think totally work. I think we'll see the same kinds of things happening with um with sort of the the shift towards uh agents and longer form tasks, where you know it may not be perfect, but if it can save you five or ten minutes, that's still valuable, and even more if the model can understand where it doesn't have confidence and can come back to you and say, I'm not sure about this, can you actually help me with this? Then you know the c the combination of human and model together can be much higher than 60%.

• 12:09Speaker 1

  I also find that 60% is magic 60% number, like it's kind of lumpy. I made it up five minutes ago. That is our new that's the Mendoza line of uh of AI. Like I think it's often very lumpy, where like it'll do very well on some tasks and not well on others. And I think that also helps like uh when we ever we run like pilot programs with customers, it's really interesting when we'll get like the same day feedback from two different companies when we'd be like, it's solved our whole problem, like we've been trying to do this for three months, thank you. And other than being like, it was way off. It's like worse than the other model, and so like uh it's also humbling to know that you have your own internal evals, but like the rubber hitting the road and actually seeing the model out in the world is where it's kind of the equivalent of like you do all this design and then like you put it in front of one user and they're like, oh wow, I was wrong. Uh the model has that feeling as well, where you're like, we try as hard as we can to like have a good sense, but then people have their own custom data sets, they have their own internal use.

• 13:01Speaker 1

  They've prompted it a certain way, and like uh so that the lies that sort of almost like bimodal nature of when you actually put it out in the world.

• 13:08Speaker 0

  I'm curious if you feel this. I think there's a very real sense in which models today are not intelligence limited, they're eval limited. Yeah, they can actually do much more and be much more correct on a wider range of things than they are today. And it's really about sort of teaching them. They have the intelligence, you need to teach them certain specific topics that you know maybe weren't in their original training set, but they can do it if you do it right.

• 13:32Speaker 1

  Yeah, we've seen that all the time where like um uh there was a lot of like exciting AI deployments that happened in like you know, maybe three years ago, and now they're like, we think the new models are better, but we never did evals because all we were doing was just shipping cool AI features three years ago. And like the hardest hump to get people over is like, let's step back and like what is success actually look like for you? Like what problem are you solving? And like often the PM is rotated, so it's like somebody's inherited it, and then be like, all right, what does that look like? All right, let's write some evaluations. What we've learned is like Claude is actually good at writing evaluations and also grading them. So like we can automate a lot of this for you, but you have to tell us what success looks like. And then let's go and actually iteratively improve our way over there. And like that is often like the difference between like 60% of the task and like 85% of the tasks. If you come interview at Anthropic, which maybe you should uh at some point, maybe you're happy in your role, maybe not.

• 14:21Speaker 1

  Um you'll see one of the things we do in our interview process actually like make you get uh a prompt from like crappy eval to good, and like just we want to see what you think, but like not enough of that talent exists elsewhere. So we're trying to get that like if there's one thing we can teach people, that's probably the most important thing. Yeah,

• 14:37Speaker 0

  writing evals. I mean, it's I I actually think it's going to become a core skill for PMs.

• 14:42Speaker 1

  We actually had this, and maybe this is like a little inside baseball, but I thought this was interesting. Like internally, we had our research PMs who like work a lot on model capabilities and model development, and then we had our like more like product surface PMs or API PMs. And we ended up realizing that like the job of a PM in 2024, 2025, building AI powered features looks is looking more and more like the former than the latter in a lot of cases. Like uh we launched uh like our uh code analysis and like basically cloud can analyze CSVs and write code for you now. And the PM there was like getting it 80% of the way there and then having to hand it over to the PM that could write the e-valves and then go to like fine-tune and like prompt and I was like, that's actually the same role. Like the quality of your feature is now gated on how well you have done the evals and the prompts. And so like that PM like definition is definitely just merging now.

• 15:29Speaker 0

  Yeah, absolutely. I mean we we set up a boot camp and like took every PM through uh writing evals and like what it was like difference between good and bad evals, and you know, we're we're definitely not done there. We've got to keep iterating and getting better on it, but it is such a critical part of making a good product with AI. Yeah.

• 15:48Speaker 2

  As part of this recruiting call for any of the people who want to be good at building AI product or research product in the future. Um we can't come to your boot camp, Kevin. So how do we develop some intuition for getting good at this eval and iteration, Lou?

• 16:05Speaker 0

  I actually think it's something you can you can use the models themselves for, like you were talking about. You can ask the models at this point, what makes a good eval? Give me, you know, I want to do this, can you write me a sample eval? And it will it will be pretty good.

• 16:17Speaker 1

  Yeah. I think that's like that goes a log way. I think there's also this question of like it's and I if you listen to like everybody from like Andre Karpathi to others who have like spent a lot of time in the field, like nothing beats looking at data. And so like people often get caught up um being like, well, we already have these evaluations and the new model is like 80% there rather than 78%. We can't shit, we're like, you know, it's worse. And I was like, if we looked at the cases where it fails, and you're like, oh, actually, this was better, it's just our grader is not as good, you know, or um it's funny, like again, well, inside baseball, you know, like every model release has the model card. And some of these model uh these evals we've seen, like even the golden answer, I'm like, I'm not sure human would say it. Or like I think that math is actually a little wrong. Like getting 100% is gonna be really hard because even just grading them is very challenging.

• 17:02Speaker 1

  So like I'd encourage you to like the way you build the intuition is go look at the actual answers, even the sample them and be like, all right, yeah, maybe we should evolve the evals, or maybe like the vibes are good, even if the eval is like uh tied. So like getting real and getting like deep on the data, I think matters.

• 17:17Speaker 0

  I also think it'll be really interesting to see how this evolves as we go towards longer form or agentic tasks because it's one thing when your evals are like, I gave you this math thing and you were able to like add four digit numbers and get to the right answer. You know, it's easy to know what what good looks like there. As the models start to do more uh long form, more ambiguous things. Go get me a hotel in New York City, you know, what's what's right there? A lot of it will be about personalization. Uh you know, if you ask any two humans who are perfectly competent, they're gonna do two different things. So your grading becomes much softer, and it you know, it'll just be interesting. We I think we'll have to evolve yet again. Like speaking of having to reinvent stuff over and over again.

• 18:03Speaker 0

  I

• 18:03Speaker 1

  think a lot like when you think about I think both labs have some concept of like this is what capabilities look like as things evolve. Like it looks a little bit like a career ladder, like what bigger and longer horizon tasks are you taking? And maybe like evals start looking more like performance review. I'm in performance review season, so this is the metaphor that's in my head. Sorry. But it's like, you know, like did the model like meet your expectation of like what a competent human have done? Did it exceed it because it did it twice as fast, or like discovered some restaurant you wouldn't have known that it greatly exceed meets most? Like it starts being like more nuanced than just like right or wrong.

• 18:34Speaker 0

  Let alone you have humans writing these evals and the models are getting to the point where they can often beat humans at certain tasks. Like people prefer the model's answers to a human's answers, and so if you're humans writing your evals, like yeah, you know, so what does that mean?

• 18:49Speaker 2

  Uh what, okay. Evals are clearly the key. Um we're gonna go spend a bunch of time with these models teaching ourselves to write evals. What other skills should product people be learning now? You you're both on that learning path.

• 19:03Speaker 1

  I think uh prototyping with these models is a thing that is underused. Like our best PMs do this, where we'll get in some long conversation about like should the UI be this or that, and before our designers have even like picked up their Figma, like our desig are often our PMs, or sometimes our engineers will be like, great, I prompted Claude, I did like an A-B comparison of what these two UIs could look like, let's try them. And I'm like, oh, this is really cool. I'm like, play that out, and like we'll be able to prototype much like a far greater variety and evaluate um like on a much faster scale than before. So like that skill of like using these tools to actually be in prototyping mode, I think is a really really useful one.

• 19:42Speaker 0

  That's a good one. I would also, I you you sort of said this, but I think it's also gonna push PMs to go deeper into the tech stack. Yeah. Um because it's and and maybe that changes over the years. Like that if you were doing like database tech in, I don't know, 2005. Maybe it required you to be able to go really deep in a different way than it would if you were doing database tech now, like layers of abstraction get built, and you maybe don't need to know all the fundamentals. But it's not like every PM needs to be a researcher by any means, but I think having an appreciation for it, spending time and learning the language and gaining intuition for how this stuff works a little bit, I think will go a long way.

• 20:22Speaker 1

  I think the other piece, like you're dealing with this like stochastic, non-deterministic system, which like evals are our best attempts to do it, but like product design in a world where like you're not in control of what the model is gonna say. You can try. And so, like, what are the feedback mechanisms that you need to close that loop? Like, how do you decide when like the model's gone astray? How do you collect that feedback in a in a rapid way? You know, like what are the guardrails you want to put in? Like, how do you even know what it's doing in aggregate? Like, it's a much more like you're understanding like the output of this intelligence across a lot of outputs over a lot of people every single day. It just requires a very different set than like, oh, the bug report is you clicked on the button and didn't follow the user. It's like that's a pretty knowable kind of problem.

• 21:04Speaker 0

  Right. And maybe this will change, you know, five years from now when people are used to it, but I think we're all still in the mode of adapting to this sort of non-deterministic user interface ourselves. Uh and certainly people who are not, you know, tech people here in this room working on tech products who are using AI are are definitely not used to it. Like it goes against all of the intuition that we've built up for the last like 25 years of using computers. Uh and so like the the idea that you're gonna put in the exact same things. Normally, if you put in the exact same inputs, computers give you the exact same outputs, and that is no longer true. Uh and it it's not just that we have to adapt to it building products. We have to also put ourselves in the shoes of the people who are using our products and think about what this means for them.

• 21:49Speaker 0

  And there's like, I mean, there are downsides to it. They're also really cool upsides. And so it's fun to kind of think about how you can use that to your advantage in different ways. I remember

• 22:00Speaker 1

  like

• 22:00Speaker 0

  we we

• 22:00Speaker 1

  did like a lot of like rolling user research at Instagram. So we have like the same, like, or the researchers would bring in different people every single week, whatever was like prototype ready would get put through it. And we do the same thing at Anthropic, but what's interesting is like for those sessions, what would often surprise me is like how users were using Instagram. There's something interesting about like their use case or like their reaction to a new feature, and like now it's like half that and half what the model did in that situation. You're like, oh, it did the right thing. This is great. So there's like this like very like you can almost like a sense of pride, maybe of like when it reacts well and you're like in a user research environment, and then like also the like frustration, you're like, oh no, you misunderstood the intent, and now you're like 10 pages down into this answer, and so like it it's also like maybe as well of getting Zen about like letting go of control and you know what's gonna happen in those environments.

• 22:46Speaker 1

  Yeah.

• 22:47Speaker 2

  You have both worked on these consumer experiences that taught new behaviors to you know many hundreds of millions of people uh quickly. Uh these AI products are happening actually faster

• 22:58Speaker 1

  than

• 22:58Speaker 2

  that, right? And you know, if if PMs and technical people don't have that much intuition naturally for how to use them, how do you think about educating end users at the scale you're both working with on something that is so unintuitive?

• 23:13Speaker 0

  I mean, it it is kind of amazing how fast we all adapt. Uh I was talking to somebody the other day and they were telling me about their first Waymo ride. Who's ridden in a Waymo?

• 23:25Speaker 1

  Who wrote one here?

• 23:26Speaker 0

  Yeah. If you haven't ridden in a Waymo, you're in San Francisco, ride a Waymo to wherever you're going when you leave here, it's a magical experience. But they were like my first 30 seconds, I was like, oh my god, watch out for that bicyclist. And then five minutes in, it was like, oh my god, I'm living in the future. And then 10 minutes later, it was like bored scrolling on your phone. Like

• 23:52Speaker 2

  you

• 23:52Speaker 0

  know, how quickly we become used to something that is just absolute magic.

• 23:56Speaker 2

  Yeah.

• 23:57Speaker 0

  Um I think there, I mean, uh Chat GPT is less than two years old. And it was absolutely mind-blowing when it exists, or when it when it first came, and now I think if we had to go back and use the original, whatever it was, GPT 3.5, I think.

• 24:12Speaker 2

  The horror, yeah.

• 24:13Speaker 0

  Yeah, like no everybody would be like, ugh.

• 24:16Speaker 2

  So dumb.

• 24:17Speaker 0

  How could I possibly, you know? And and you know, the stuff that's that's happening today that we're working on that you guys are working on, it all feels like magic. Twelve months from now, we're gonna be like, can you can you believe we use that garbage? Because it's gonna, I mean, that's how fast this thing is moving. But it's also amazing to me how quickly people adapt. Because I mean, as much as we try and bring people along, like there are also um there's just there's a lot of excitement. People understand that this is like the world is moving in this direction, and um we've got to try and make it the best possible move that we can, but uh it's it's happening and it's happening fast. Well,

• 24:53Speaker 1

  I think we're trying to get better at, and that's is also letting the product be like educational in a very literal way, which is like a thing we did not do early, and now we're changing is just tell Claude more about itself, which was like you know, it's in its training set that it's you know uh artificial intelligence created by anthropic, whatever, but now we're literally like, and here is how you use this feature we should because people would ask, and again, this came from user research, because we'd be like, they would be like, How do I use this thing? And then Cloud would be like, I don't know, have you tried like looking at it on the internet? You're like, no, that's not helpful. So like uh like we're really trying to ground it in an at launch time. We're like, you know, it's a process we're improving, but like it's it's cool to now see like this is the exact length of the documentation, like here's how you do it. Like I can help you stuff by stuff, oh, you're stuck, I can help you here. So uh these things are actually very good at solving uh UI problems and like user confusion, and like we should use them more for that.

• 25:44Speaker 1

  Yeah,

• 25:44Speaker 2

  it's gotta be different when you are um you know trying to do like change management in an enterprise, though, right? Because there's a there's a status quo for how you're doing things, there's organizational process, like how do you think about educating entire organizations about productivity improvements or whatever else can come?

• 26:00Speaker 1

  I think the enterprise one is really interesting because like even like these products have like millions of millions of users, but like the power users are very much, I think still like early adopters, people who like technology, and then there's like a you know long tail. Whereas when you go into enterprise, you're deploying to like an organization that is like often there's folks who are like not very non-technical, and like I think that's really cool actually seeing fairly non-technical users get exposed to like a chat powered LLM for the first time and then getting to see it. Then you have the luxury of like getting to run like a session where you teach them about it and like have educational materials. Um I think we need to learn from what happens in those, and then say like that's what we need to do to teach the next hundred million people how to use these UIs.

• 26:43Speaker 0

  And they're usually power users internally, and they're uh they're excited to teach the rest of people, and you know, like with open AI, we have these custom GPTs that you can make, and organizations make thousands of them often, and it's a way for the power users to make something that makes AI easier and like immediately valuable for the people that might not know how to use it otherwise. Um so like that's one cool thing. You find the pockets of power users, and they actually will sort of be evangelists.

• 27:12Speaker 2

  I I have to ask you then, because you're you know, your organizations are both like all power users, right? So you know, you're living in your little pocket of the future. Uh I'll ask about one thing, but feel free to redirect. My how am I supposed to use computer use? This is amazing. Like, what are you guys doing?

• 27:28Speaker 1

  Yeah, well, internally, like we're I mean, this is to Kevin's earlier comment around like when is it gonna be ready? All right, like go. This like it was pretty late breaking, like we like had conviction that it was like this is like good, and like we don't want to put this down, like it's early still, and it's like it's still gonna make mistakes, but like how do we do this as well? The funniest use case, like while we were beta testing it was like somebody's like, I wonder if I can get it to order us a pizza, and like it did, and they're like, Great, there's dominant like the the moment where Domino shows up at your office and it was ordered entirely by AI is like a very it was a very cool like seminal moment, and then we're like, oh, but it's dominoes, but like you know, it's

• 28:01Speaker 0

  like but like it

• 28:02Speaker 1

  was definitely this like a mess. Yeah, uh but it was AI, you know, so it was all it was it was it was good. And also like ordered quite a bit of pizza, so it was like maybe hungrier than intended. Uh some early things that we're seeing that we think are really interesting. One is UI testing, which is like I was like at Instagram we had basically no UI tests because they're hard to write, they're like they're brittle, um, and they're like often like a little bit like, oh, like we moved this button around and like it should still pass. That was the point of the PR, but like now it's gonna fail. We're gonna have to like do this whole other snapshot. Um, and like early signs are like computer use works really well for like, hey, does it work as intended? Does it do the thing that you want it to do? And I think that's like been been very, very interesting. And then what we're starting to get into too is like what are the agentic things that just like involve a lot of like data manipulation. So we're looking at it with our support teams and our finance teams around like those PR forms are gonna fill themselves, but like it's very repetitive.

• 28:52Speaker 1

  You often have data in one silo, you want to put it in a different silo, and it just requires like human time. Like I keep using the word drudgery when I talk about computer use like, can we automate the drudgery so you can focus on the creative stuff and not like the you know 30 clicks to do one single thing.

• 29:06Speaker 2

  Uh Kevin, I I think we have a lot of teams that are um experimenting with O1. You can obviously do much more sophisticated things. You also can't use it as a one-for-one replacement if you're already using one of the you know GPT-4.0 models or whatever in uh in your application. Like, can you give us some guidance and what are you guys doing with it internally?

• 29:27Speaker 0

  So I think one thing that people maybe don't realize that actually a lot of the most sophisticated customers of ours are doing, and that we're certainly doing internally, is it's not really about one model for any particular thing. You end up putting together sort of workflows and orchestration between models. And so you use them for what they're good at. O one's really good at reasoning, but it also takes a little bit of time to think, and it's not multimodal and you know has other limitations.

• 29:54Speaker 2

  I realize it's a basic question, but

• 29:56Speaker 0

  yeah, so uh we yeah, people are, I think, pretty Used to the concept, the like scaling pre-training concept. You go GPT 2, 3, 4, 5, whatever. And you're doing bigger and bigger runs on pre-training. These models are getting you know smarter and smarter. Um or rather maybe they know more and more. But they're kind of like system one thinking, right? It's it's you ask it a question, you immediately get an answer. It's like text completion. You know, it's sort of if I ask you. Me

• 30:25Speaker 2

  asking you questions right now, and you just have to stream one token at a time, keep going, don't think.

• 30:30Speaker 0

  It's amazing actually how much human, like your intuition about how other humans work will often like help you in intuiting about how these models work. Um, you know, you ask me a question, I got off onto the wrong like sentence, it's hard to recover. The models totally do the same thing. Um but uh so you've got that that sort of larger and larger pre-training. Oh one is actually a different way of scaling uh intelligence by doing it at uh at query time, basically. So instead of system one thinking, I ask you a question and immediately tries to give you an answer, it'll pause. Same thing I would you you know, you would do if I asked you a question, I said solve this sudoku, do this New York Times connections puzzle. Yeah, you would start going, okay, these words, how do they group together?

• 31:18Speaker 0

  Okay, these might be these four. Well, no, I'm not sure. Could be yeah, you know, you're you're like forming hypotheses, using what you know to refute these hypotheses or affirm them, and then from that continuing to reason on it's how it's how scientific breakthroughs are made, it's how we answer hard questions. Um, this is about teaching the models to do it. And right now, you know, they'll think for 30 or 60 seconds before they answer. Imagine what happens if they can think for five hours or five days. Um so it's basically a new way to scale intelligence, and we feel like we're just at the very beginning. Uh, you know, we're at the like GPT one phase of um of this new form of reasoning. Um but in the same way, it's not you don't use it for everything, right?

• 32:04Speaker 0

  There are some times when you ask me a question, you don't want me to wait 60 seconds, you I should just give you an answer. Um so we end up using our models in a bunch of different ways together. So, for example, like cybersecurity, you would think not really a use case for models. They can hallucinate. That seems like a bad place to hallucinate. But you can fine-tune a model to be good at certain tasks, and then you can fine-tune models to be very precise about the kinds of inputs and outputs that they expect, and have these models start working in in concert together, and you know, models that are checking the outputs of other models, realizing when something doesn't make sense, asking it to try again. Um that ends up being how we get a ton of value out of our own models internally.

• 32:52Speaker 0

  It's like specific use cases uh and orchestrations of models together designed, sort of working in concert to do specific tasks, which again, going back to like reasoning about how we work as humans, how do we do complex things as humans? You have different people who often have different skill sets and they work together to accomplish a hard task.

• 33:14Speaker 2

  I can't let you guys get away without telling us something about the future and what's coming. And so you don't have to give us release dates. I understand you don't know. But uh if you if you look out, I think the furthest anyone can look out in AI right now is like, well, tell me if you can see the future, but like let's say like six months, twelve months, like what's an experience that you imagine is gonna be possible or prevalent.

• 33:39Speaker 1

  I think a lot about um like to breaking the well, I think a lot about this all the time, but like the um two maybe two words to be like plant seeds in in everybody's mind. Like one is proactivity, like how do the models become more proactive? Like once they know about you and they're monitoring, like they're reading your email in a good creepy way, and they're like, uh, you know, because you authorized them too, and then they like you know spot an interesting trend, or you start your day with something that's a like um like a proactive like uh recap of what's going on, some conversations you're gonna have. I I pre-did some research for you. Hey, your next meeting is coming up. Like, here's what you might want to talk about. I saw you have this like presentation coming up. Here's the first draft that I put together. Like that kind of proactivity, I think is gonna be really, really powerful. And then the other part is being more asynchronous. So like uh I think 01 is like uh early UI in this exploration, which is like it's gonna do a lot and it's gonna tell you kind of what it's gonna do along the way.

• 34:29Speaker 1

  And like you can sit there and wait for it, but you could also like be like, oh, it's gonna think for a while, I'm gonna go like do something else, maybe tab back, maybe it like can tell me when it's done. Like, yeah, I expanding the time horizon, both in terms of like you didn't ask a question, it just told you something. I think that's gonna be interesting. And then you did ask a question, and you're gonna be like, great, like I'm gonna go reason about it, I'm gonna go research it. I might have to ask another human about it, like, and then I'm gonna like maybe come up with my first answer, I'm gonna vet that answer. You'll hear back from me in like an hour. Like breaking free of those like uh constraints of like expecting an answer immediately. I think will let you do things like, hey, I have this like whole like mini project plan, like go flesh it out, or like not just like I want you to like change this one thing on the screen, but like fix this bug for me, like take my PRD and like adapt it for these new market conditions, like adapt it for these three different marketing conditions that emerge, like being able to push those dimensions, I think is what I'm personally most excited about on the product side.

• 35:25Speaker 0

  Yeah, I completely agree with all of that. Um and it's the models are gonna get smarter at an accelerating rate, I think, which is also part of how all of that uh comes to pass. Another thing that will be really exciting is seeing the models able to interact in all the same ways that we as humans interact. You know, right now you mostly type to these things, and you know, I mostly type to a lot of my friends on WhatsApp and other things, but I also speak. I also can see, and uh we just we launched this advanced voice mode relatively recently. I was in uh I was in Korea and Japan having conversations, and I I would just I would often be with somebody with whom I had no common language whatsoever. Before this, we could not have said a word to each other. And instead, I was like, hey chat GPT, I want you to act as a translator.

• 36:14Speaker 0

  When I say something in English, I want you to say it in Korean. And when you hear something in Korean, say it back to me in English. And all of a sudden I had this universal translator, and I was having business conversations with another person. Uh and it was magical. And you think what that can do, like not just in a business context, but think about people's willingness to travel to new places if you don't ever have to be worried about not speaking the language, and you've got this like Star Trek universal translator in your pocket, you know, and so uh experiences like that. I think it's gonna become commonplace fast, but it's magical, and I'm excited about that in combination with all the stuff Mike was just saying.

• 36:56Speaker 2

  Oh, one of my favorite pastimes now, just you know, since uh voice mode release is actually watching there's a genre of TikTok of well, this just speaks to how old I am. Like there's a genre of TikTok where you just like uh it's just young people talking to voice mode, like pouring their heart out, using it all these ways where I'm like, oh my God, like there's this old term being like a digitally native or mobile native, and I'm like, I like pretty strongly believe in this AI thing, and I would not think to interact in this way. But people who are 14 years old are like, well, I expect the AI to be able to do that, and I love that.

• 37:34Speaker 0

  Have you ever given it to your kids?

• 37:36Speaker 2

  Uh I haven't yet. My kids are like five and seven, knows them, so we'll but we'll get there. I mean,

• 37:40Speaker 0

  minor eight and ten, but like on a car ride, they'll be like, Can I talk to chat GPT?

• 37:45Speaker 2

  Yes.

• 37:45Speaker 0

  And they will ask it the most bizarre things. They will just have weirdo conversations with it, but they're perfectly happy talking to an AI.

• 37:54Speaker 2

  Yeah, actually, one of my favorite experiences, and maybe we'll close and ask you for like the most surprising behavior, kids or not, is uh um like when my parents read to me, like I got lucky, I was lucky if I got to choose the book, and it wasn't my dad being like, we're gonna read this physics study I'm interested in, right? My kids, I don't know if it's just like parenting in the Bay Area, but my kids are like, okay, mom, make the images, right? I want to tell a story about the dragon unicorn in this setting. I'm gonna tell you exactly how it's gonna happen, create it in real time. And I'm like, that's a big ask. I'm glad you believe and like know that's possible, but it's it's a wild way to like create your own entertainment. So what is the um most surprising behavior you've seen in your own products recently?

• 38:42Speaker 1

  I think it's a behavior and a relationship. Like people really start understanding the nuance of like what Claude is, or just have like a uh new revenge of the model, and it's like they get the nuance. Like it's like I guess the behavior is like almost befriending or like really like developing a lot of like two-way empathy around what's happening, and then like the oh, you know, the new model like felt like it was smarter, but maybe a little more distant, but maybe you know, and it's like it's like that kind of like nuance, which like you like I it's it's given me as a product person a lot more empathy around like you're not just shipping a product, you're shipping like intelligence, and intelligence and empathy are like what makes like interpersonal relationships important. And if somebody showed up and they're like, I was upgraded, like I say, you know, I scored 2% higher on this math score, but like I'm different in this way.

• 39:29Speaker 1

  You'd be like, oh, I gotta adapt now, and maybe you know be a little worried about it. So like that that's been an interesting journey for me, like understanding the mentality for people that when they're using our products.

• 39:41Speaker 0

  Yeah, model behavior is absolutely a product role. Like the personality of the model is is key, and there are interesting questions around how much should it customize versus how much should you know OpenAI have one personality and Claude has some distinct personality? And are people gonna use one versus the other? Because they happen to like it. I mean, that's a that's a very human thing, right? We're friends with different people because we happen to like different people better than others. And it's um that's an interesting thing to think about. We did something recently um and uh it sort of went viral on Twitter. People started asking the model based on everything you know about me, based on all of our past interactions, you know, what what would you say about me? And the model will will respond and it will like give you a give it a description of what it you know kind of thinks based on all of your past interactions.

• 40:34Speaker 0

  And it it is this sort of you're you're starting to interact with it almost like some sort of person or entity in interesting ways. And um, anyways, it was fascinating to see people's reaction to that.

• 40:47Speaker 2

  Kevin, Mike, thank you so much for doing this and giving us a glimpse into the future.

• 40:51Speaker 1

  Thank you so much.