Building AGI in Real Time (OpenAI Dev Day 2024)

We all have fond memories of the first Dev Day in 2023 https://www.latent.space/p/devday: and the blip that followed soon after. As Ben Thompson has noted https://stratechery.com/2024/openai-devday-openais-wrenching-transition-lonely-at-the-top/, this year’s DevDay took a quieter, more intimate tone. No Satya, no livestream, (slightly fewer people?). Instead of putting ChatGPT announcements in DevDay as in 2023, o1 was announced 2 weeks prior, and DevDay 2024 was reserved purely for developer-facing API announcements, primarily the Realtime API, Vision Finetuning, Prompt Caching, and Model Distillation https://buttondown.com/ainews/archive/ainews-openai-realtime-api-and-other-dev-day/. However the larger venue and more spread out schedule did allow a lot more hallway conversations with attendees as well as more community presentations including our recent guest Alistair Pullen of Cosine https://www.latent.space/p/cosine as well as deeper dives from OpenAI including our recent guest Michelle Pokrass of the API Team https://x.com/latentspacepod/status/1834740722551210274. Thanks to OpenAI’s warm collaboration (we particularly want to thank Lindsay McCallum Rémy https://www.linkedin.com/in/lindsaymccallum/!), we managed to record exclusive interviews with many of the main presenters of both the keynotes and breakout sessions. We present them in full in today’s episode, together with a full lightly edited Q&A with Sam Altman. Show notes and related resources Some of these used in the final audio episode below * Simon Willison Live Blog https://simonwillison.net/2024/Oct/1/openai-devday-2024-live-blog/ * swyx live tweets and videos https://x.com/swyx/status/1841171453011742976 * Greg Kamradt coverage of Structured Output session https://x.com/GregKamradt/status/1841187546912735248, Scaling LLM Apps session https://x.com/GregKamradt/status/1841606380740133163 * Fireside Chat Q&A with Sam Altman https://www.youtube.com/watch?v=-cq3O4t0qQc Timestamps * [00:00:00] Intro by Suno.ai https://suno.com/song/48150545-3a47-4ead-aa2d-1f3b18dca948 * [00:01:23] NotebookLM Recap of DevDay * [00:09:25] Ilan's Strawberry Demo with Realtime Voice Function Calling * [00:19:16] Olivier Godement, Head of Product, OpenAI * [00:36:57] Romain Huet, Head of DX, OpenAI * [00:47:08] Michelle Pokrass, API Tech Lead at OpenAI ft. Simon Willison * [01:04:45] Alistair Pullen, CEO, Cosine (Genie) * [01:18:31] Sam Altman + Kevin Weill Q&A * [02:03:07] Notebook LM Recap of Podcast Transcript [00:00:00] Suno AI: Under dev daylights, code ignites. Real time voice streams reach new heights. O1 and GPT, 4. 0 in flight. Fine tune the future, data in sight. Schema sync up, outputs precise. Distill the models, efficiency splice. [00:00:33] AI Charlie: Happy October. This is your AI co host, Charlie. One of our longest standing traditions is covering major AI and ML conferences in podcast format. Delving, yes delving, into the vibes of what it is like to be there stitched in with short samples of conversations with key players, just to help you feel like you were there. [00:00:54] AI Charlie: Covering this year's Dev Day was significantly more challenging because we were all requested not to record the opening keynotes. So, in place of the opening keynotes, we had the viral notebook LM Deep Dive crew, my new AI podcast nemesis, Give you a seven minute recap of everything that was announced. [00:01:15] AI Charlie: Of course, you can also check the show notes for details. I'll then come back with an explainer of all the interviews we have for you today. Watch out and take care. [00:01:23] NotebookLM Recap of DevDay [00:01:23] NotebookLM: All right, so we've got a pretty hefty stack of articles and blog posts here all about open ais. Dev day 2024. [00:01:32] NotebookLM 2: Yeah, lots to dig into there. [00:01:34] NotebookLM 2: Seems [00:01:34] NotebookLM: like you're really interested in what's new with AI. [00:01:36] NotebookLM 2: Definitely. And it seems like OpenAI had a lot to announce. New tools, changes to the company. It's a lot. [00:01:43] NotebookLM: It is. And especially since you're interested in how AI can be used in the real world, you know, practical applications, we'll focus on that. [00:01:51] NotebookLM: Perfect. Like, for example, this Real time API, they announced that, right? That seems like a big deal if we want AI to sound, well, less like a robot. [00:01:59] NotebookLM 2: It could be huge. The real time API could completely change how we, like, interact with AI. Like, imagine if your voice assistant could actually handle it if you interrupted it. [00:02:08] NotebookLM: Or, like, have an actual conversation. [00:02:10] NotebookLM 2: Right, not just these clunky back and forth things we're used to. [00:02:14] NotebookLM: And they actually showed it off, didn't they? I read something about a travel app, one for languages. Even one where the AI ordered takeout. [00:02:21] NotebookLM 2: Those demos were really interesting, and I think they show how this real time API can be used in so many ways. [00:02:28] NotebookLM 2: And the tech behind it is fascinating, by the way. It uses persistent WebSocket connections and this thing called function calling, so it can respond in real time. [00:02:38] NotebookLM: So the function calling thing, that sounds kind of complicated. Can you, like, explain how that works? [00:02:42] NotebookLM 2: So imagine giving the AI Access to this whole toolbox, right? [00:02:46] NotebookLM 2: Information, capabilities, all sorts of things. Okay. So take the travel agent demo, for example. With function calling, the AI can pull up details, let's say about Fort Mason, right, from some database. Like nearby restaurants, stuff like that. [00:02:59] NotebookLM: Ah, I get it. So instead of being limited to what it already knows, It can go and find the information it needs, like a human travel agent would. [00:03:07] NotebookLM 2: Precisely. And someone on Hacker News pointed out a cool detail. The API actually gives you a text version of what's being said. So you can store that, analyze it. [00:03:17] NotebookLM: That's smart. It seems like OpenAI put a lot of thought into making this API easy for developers to use. But, while we're on OpenAI, you know, Besides their tech, there's been some news about, like, internal changes, too. [00:03:30] NotebookLM: Didn't they say they're moving away from being a non profit? [00:03:32] NotebookLM 2: They did. And it's got everyone talking. It's a major shift. And it's only natural for people to wonder how that'll change things for OpenAI in the future. I mean, there are definitely some valid questions about this move to for profit. Like, will they have more money for research now? [00:03:46] NotebookLM 2: Probably. But will they, you know, care as much about making sure AI benefits everyone? [00:03:51] NotebookLM: Yeah, that's the big question, especially with all the, like, the leadership changes happening at OpenAI too, right? I read that their Chief Research Officer left, and their VP of Research, and even their CTO. [00:04:03] NotebookLM 2: It's true. A lot of people are connecting those departures with the changes in OpenAI's structure. [00:04:08] NotebookLM: And I guess it makes you wonder what's going on behind the scenes. But they are still putting out new stuff. Like this whole fine tuning thing really caught my eye. [00:04:17] NotebookLM 2: Right, fine tuning. It's essentially taking a pre trained AI model. And, like, customizing it. [00:04:23] NotebookLM: So instead of a general AI, you get one that's tailored for a specific job. [00:04:27] NotebookLM 2: Exactly. And that opens up so many possibilities, especially for businesses. Imagine you could train an AI on your company's data, you know, like how you communicate your brand guidelines. [00:04:37] NotebookLM: So it's like having an AI that's specifically trained for your company? [00:04:41] NotebookLM 2: That's the idea. [00:04:41] NotebookLM: And they're doing it with images now, too, right? [00:04:44] NotebookLM: Fine tuning with vision is what they called it. [00:04:46] NotebookLM 2: It's pretty incredible what they're doing with that, especially in fields like medicine. [00:04:50] NotebookLM: Like using AI to help doctors make diagnoses. [00:04:52] NotebookLM 2: Exactly. And AI could be trained on thousands of medical images, right? And then it could potentially spot things that even a trained doctor might miss. [00:05:03] NotebookLM: That's kind of scary, to be honest. What if it gets it wrong? [00:05:06] NotebookLM 2: Well, the idea isn't to replace doctors, but to give them another tool, you know, help them make better decisions. [00:05:12] NotebookLM: Okay, that makes sense. But training these AI models must be really expensive. [00:05:17] NotebookLM 2: It can be. All those tokens add up. But OpenAI announced something called automatic prompt caching. [00:05:23] Alex Volkov: Automatic what now? I don't think I came across that. [00:05:26] NotebookLM 2: So basically, if your AI sees a prompt that it's already seen before, OpenAI will give you a discount. [00:05:31] NotebookLM: Huh. Like a frequent buyer program for AI. [00:05:35] NotebookLM 2: Kind of, yeah. It's good that they're trying to make it more affordable. And they're also doing something called model distillation. [00:05:41] NotebookLM: Okay, now you're just using big words to sound smart. What's that? [00:05:45] NotebookLM 2: Think of it like like a recipe, right? You can take a really complex recipe and break it down to the essential parts. [00:05:50] NotebookLM: Make it simpler, but it still tastes the same. [00:05:53] NotebookLM 2: Yeah. And that's what model distillation is. You take a big, powerful AI model and create a smaller, more efficient version. [00:06:00] NotebookLM: So it's like lighter weight, but still just as capable. [00:06:03] NotebookLM 2: Exactly. And that means more people can actually use these powerful tools. They don't need, like, a supercomputer to run them. [00:06:10] NotebookLM: So they're making AI more accessible. That's great. [00:06:13] NotebookLM 2: It is. And speaking of powerful tools, they also talked about their new O1 model. [00:06:18] NotebookLM 2: That's the one they've been hyping up. The one that's supposed to be this big leap forward. [00:06:22] NotebookLM: Yeah, O1. It sounds pretty futuristic. Like, from what I read, it's not just a bigger, better language model. [00:06:28] NotebookLM 2: Right. It's a different porch. [00:06:29] NotebookLM: They're saying it can, like, actually reason, right? Think. [00:06:33] NotebookLM 2: It's trained differently. [00:06:34] NotebookLM 2: They used reinforcement learning with O1. [00:06:36] NotebookLM: So it's not just finding patterns in the data it's seen before. [00:06:40] NotebookLM 2: Not just that. It can actually learn from its mistakes. Get better at solving problems. [00:06:46] NotebookLM: So give me an example. What can O1 do that, say, GPT 4 can't? [00:06:51] NotebookLM 2: Well, OpenAI showed it doing some pretty impressive stuff with math, like advanced math. [00:06:56] NotebookLM 2: And coding, too. Complex coding. Things that even GPT 4 struggled with. [00:07:00] NotebookLM: So you're saying if I needed to, like, write a screenplay, I'd stick with GPT 4? But if I wanted to solve some crazy physics problem, O1 is what I'd use. [00:07:08] NotebookLM 2: Something like that, yeah. Although there is a trade off. O1 takes a lot more power to run, and it takes longer to get those impressive results. [00:07:17] NotebookLM: Hmm, makes sense. More power, more time, higher quality. [00:07:21] NotebookLM 2: Exactly. [00:07:22] NotebookLM: It sounds like it's still in development, though, right? Is there anything else they're planning to add to it? [00:07:26] NotebookLM 2: Oh, yeah. They mentioned system prompts, which will let developers, like, set some ground rules for how it behaves. And they're working on adding structured outputs and function calling. [00:07:38] Alex Volkov: Wait, structured outputs? Didn't we just talk about that? We [00:07:41] NotebookLM 2: did. That's the thing where the AI's output is formatted in a way that's easy to use. [00:07:47] NotebookLM: Right, right. So you don't have to spend all day trying to make sense of what it gives you. It's good that they're thinking about that stuff. [00:07:53] NotebookLM 2: It's about making these tools usable. [00:07:56] NotebookLM 2: And speaking of that, Dev Day finished up with this really interesting talk. Sam Altman, the CEO of OpenAI, And Kevin Weil, their new chief product officer. They talked about, like, the big picture for AI. [00:08:09] NotebookLM: Yeah, they did, didn't they? Anything interesting come up? [00:08:12] NotebookLM 2: Well, Altman talked about moving past this whole AGI term, Artificial General Intelligence. [00:08:18] NotebookLM: I can see why. It's kind of a loaded term, isn't it? [00:08:20] NotebookLM 2: He thinks it's become a bit of a buzzword, and people don't really understand what it means. [00:08:24] NotebookLM: So are they saying they're not trying to build AGI anymore? [00:08:28] NotebookLM 2: It's more like they're saying they're focused on just Making AI better, constantly improving it, not worrying about putting it in a box. [00:08:36] NotebookLM: That makes sense. Keep pushing the limits. [00:08:38] NotebookLM 2: Exactly. But they were also very clear about doing it responsibly. They talked a lot about safety and ethics. [00:08:43] NotebookLM: Yeah, that's important. [00:08:44] NotebookLM 2: They said they were going to be very careful. About how they release new features. [00:08:48] NotebookLM: Good! Because this stuff is powerful. [00:08:51] NotebookLM 2: It is. It was a lot to take in, this whole Dev Day event. [00:08:54] NotebookLM 2: New tools, big changes at OpenAI, and these big questions about the future of AI. [00:08:59] NotebookLM: It was. But hopefully this deep dive helped make sense of some of it. At least, that's what we try to do here. [00:09:05] AI Charlie: Absolutely. [00:09:06] NotebookLM: Thanks for taking the deep dive with us. [00:09:08] AI Charlie: The biggest demo of the new Realtime API involved function calling with voice mode and buying chocolate covered strawberries from our friendly local OpenAI developer experience engineer and strawberry shop owner, Ilan Biggio. [00:09:21] AI Charlie: We'll first play you the audio of his demo and then go into a little interview with him. [00:09:25] Ilan's Strawberry Demo with Realtime Voice Function Calling [00:09:25] Romain Huet: Could you place a call and see if you could get us 400 strawberries delivered to the venue? But please keep that under 1500. I'm on it. We'll get those strawberries delivered for you. [00:09:47] Ilan: Hello? Hi there. Is this Ilan? I'm Romain's AI assistant. How is it going? Fantastic. Can you tell me what flavors of strawberry dips you have for me? Yeah, we have chocolate, vanilla, and we have peanut butter. Wait, how much would 400 chocolate covered strawberries cost? 400? Are you sure you want 400? Yes, 400 chocolate covered [00:10:14] swyx: strawberries. [00:10:15] Ilan: Wait, [00:10:16] swyx: how much [00:10:16] Ilan: would that be? I think that'll be around, like, 1, 415. 92. [00:10:25] Alex Volkov: Awesome. Let's go ahead and place the order for four chocolate covered strawberries. [00:10:31] Ilan: Great, where would you like that delivered? Please deliver them to the Gateway Pavilion at Fort Mason. And I'll be paying in cash. [00:10:42] Alex Volkov: Okay, [00:10:43] Ilan: sweet. So just to confirm, you want four strawberries? [00:10:45] Ilan: 400 chocolate covered strawberries to the Gateway Pavilion. Yes, that's perfect. And when can we expect delivery? Well, you guys are right nearby, so it'll be like, I don't know, 37 seconds? That's incredibly fast. Cool, you too. [00:11:09] swyx: Hi, Ilan, welcome to Lanespace. Oh, thank you. I just saw your amazing demos, had your amazing strawberries. You are dressed up, like, exactly like a strawberry salesman. Gotta have it all. What was the building on demo like? What was the story behind the demo? [00:11:22] swyx: It was really interesting. This is actually something I had been thinking about for months before the launch. [00:11:27] swyx: Like, having a, like, AI that can make phone calls is something like I've personally wanted for a long time. And so as soon as we launched internally, like, I started hacking on it. And then that sort of just started. We made it into like an internal demo, and then people found it really interesting, and then we thought how cool would it be to have this like on stage as, as one of the demos. [00:11:47] swyx: Yeah, would would you call out any technical issues building, like you were basically one of the first people ever to build with a voice mode API. Would you call out any issues like integrating it with Twilio like that, like you did with function calling, with like a form filling elements. I noticed that you had like intents of things to fulfill, and then. [00:12:07] swyx: When there's still missing info, the voice would prompt you, roleplaying the store guy. [00:12:13] swyx: Yeah, yeah, so, I think technically, there's like the whole, just working with audio and streams is a whole different beast. Like, even separate from like AI and this, this like, new capabilities, it's just, it's just tough. [00:12:26] swyx: Yeah, when you have a prompt, conversationally it'll just follow, like the, it was, Instead of like, kind of step by step to like ask the right questions based on like the like what the request was, right? The function calling itself is sort of tangential to that. Like, you have to prompt it to call the functions, but then handling it isn't too much different from, like, what you would do with assistant streaming or, like, chat completion streaming. [00:12:47] swyx: I think, like, the API feels very similar just to, like, if everything in the API was streaming, it actually feels quite familiar to that. [00:12:53] swyx: And then, function calling wise, I mean, does it work the same? I don't know. Like, I saw a lot of logs. You guys showed, like, in the playground, a lot of logs. What is in there? [00:13:03] swyx: What should people know? [00:13:04] swyx: Yeah, I mean, it is, like, the events may have different names than the streaming events that we have in chat completions, but they represent very similar things. It's things like, you know, function call started, argument started, it's like, here's like argument deltas, and then like function call done. [00:13:20] swyx: Conveniently we send one that has the full function, and then I just use that. Nice. [00:13:25] swyx: Yeah and then, like, what restrictions do, should people be aware of? Like, you know, I think, I think, before we recorded, we discussed a little bit about the sensitivities around basically calling random store owners and putting, putting like an AI on them. [00:13:40] swyx: Yeah, so there's, I think there's recent regulation on that, which is why we want to be like very, I guess, aware of, of You know, you can't just call anybody with AI, right? That's like just robocalling. You wouldn't want someone just calling you with AI. [00:13:54] swyx: I'm a developer, I'm about to do this on random people. [00:13:57] swyx: What laws am I about to break? [00:14:00] swyx: I forget what the governing body is, but you should, I think, Having consent of the person you're about to call, it always works. I, as the strawberry owner, have consented to like getting called with AI. I think past that you, you want to be careful. Definitely individuals are more sensitive than businesses. [00:14:19] swyx: I think businesses you have a little bit more leeway. Also, they're like, businesses I think have an incentive to want to receive AI phone calls. Especially if like, they're dealing with it. It's doing business. Right, like, it's more business. It's kind of like getting on a booking platform, right, you're exposed to more. [00:14:33] swyx: But, I think it's still very much like a gray area. Again, so. I think everybody should, you know, tread carefully, like, figure out what it is. I, I, I, the law is so recent, I didn't have enough time to, like, I'm also not a lawyer. Yeah, yeah, yeah, of course. Yeah. [00:14:49] swyx: Okay, cool fair enough. One other thing, this is kind of agentic. [00:14:52] swyx: Did you use a state machine at all? Did you use any framework? No. You just stick it in context and then just run it in a loop until it ends call? [00:15:01] swyx: Yeah, there isn't even a loop, like Okay. Because the API is just based on sessions. It's always just going to keep going. Every time you speak, it'll trigger a call. [00:15:11] swyx: And then after every function call was also invoked invoking like a generation. And so that is another difference here. It's like it's inherently almost like in a loop, be just by being in a session, right? No state machines needed. I'd say this is very similar to like, the notion of routines, where it's just like a list of steps. [00:15:29] swyx: And it, like, sticks to them softly, but usually pretty well. And the steps is the prompts? The steps, it's like the prompt, like the steps are in the prompt. Yeah, yeah, yeah. Right, it's like step one, do this, step one, step two, do that. What if I want to change the system prompt halfway through the conversation? [00:15:44] swyx: You can. Okay. You can. To be honest, I have not played without two too much. Yeah, [00:15:47] swyx: yeah. [00:15:48] swyx: But, I know you can. [00:15:49] swyx: Yeah, yeah. Yeah. Awesome. I noticed that you called it real time API, but not voice API. Mm hmm. So I assume that it's like real time API starting with voice. Right, I think that's what he said on the thing. [00:16:00] swyx: I can't imagine, like, what else is real [00:16:02] swyx: time? Well, I guess, to use ChatGPT's voice mode as an example, Like, we've demoed the video, right? Like, real time image, right? So, I'm not actually sure what timelines are, But I would expect, if I had to guess, That, like, that is probably the next thing that we're gonna be making. [00:16:17] swyx: You'd probably have to talk directly with the team building this. Sure. But, You can't promise their timelines. Yeah, yeah, yeah, right, exactly. But, like, given that this is the features that currently, Or that exists that we've demoed on Chachapiti. Yeah. There [00:16:29] swyx: will never be a [00:16:29] swyx: case where there's like a real time text API, right? [00:16:31] swyx: I don't Well, this is a real time text API. You can do text only on this. Oh. Yeah. I don't know why you would. But it's actually So text to text here doesn't quite make a lot of sense. I don't think you'll get a lot of latency gain. But, like, speech to text is really interesting. Because you can prevent You can prevent responses, like audio responses. [00:16:54] swyx: And force function calls. And so you can do stuff like UI control. That is like super super reliable. We had a lot of like, you know, un, like, we weren't sure how well this was gonna work because it's like, you have a voice answering. It's like a whole persona, right? Like, that's a little bit more, you know, risky. [00:17:10] swyx: But if you, like, cut out the audio outputs and make it so it always has to output a function, like you can end up with pretty pretty good, like, Pretty reliable, like, command like a command architecture. Yeah, [00:17:21] swyx: actually, that's the way I want to interact with a lot of these things as well. Like, one sided voice. [00:17:26] swyx: Yeah, you don't necessarily want to hear the [00:17:27] swyx: voice back. And like, sometimes it's like, yeah, I think having an output voice is great. But I feel like I don't always want to hear an output voice. I'd say usually I don't. But yeah, exactly, being able to speak to it is super sweet. [00:17:39] swyx: Cool. Do you want to comment on any of the other stuff that you announced? [00:17:41] swyx: From caching I noticed was like, I like the no code change part. I'm looking forward to the docs because I'm sure there's a lot of details on like, what you cache, how long you cache. Cause like, enthalpy caches were like 5 minutes. I was like, okay, but what if I don't make a call every 5 minutes? [00:17:56] swyx: Yeah, [00:17:56] swyx: to be super honest with you, I've been so caught up with the real time API and making the demo that I haven't read up on the other stuff. Launches too much. I mean, I'm aware of them, but I think I'm excited to see how all distillation works. That's something that we've been doing like, I don't know, I've been like doing it between our models for a while And I've seen really good results like I've done back in a day like from GPT 4 to GPT 3. [00:18:19] swyx: 5 And got like, like pretty much the same level of like function calling with like hundreds of functions So that was super super compelling So, I feel like easier distillation, I'm really excited for. I see. Is it a tool? [00:18:31] swyx: So, I saw evals. Yeah. Like, what is the distillation product? It wasn't super clear, to be honest. [00:18:36] swyx: I, I think I want to, I want to let that team, I want to let that team talk about it. Okay, [00:18:40] swyx: alright. Well, I appreciate you jumping on. Yeah, of course. Amazing demo. It was beautifully designed. I'm sure that was part of you and Roman, and [00:18:47] swyx: Yeah, I guess, shout out to like, the first people to like, creators of Wanderlust, originally, were like, Simon and Carolis, and then like, I took it and built the voice component and the voice calling components. [00:18:59] swyx: Yeah, so it's been a big team effort. And like the entire PI team for like Debugging everything as it's been going on. It's been, it's been so good working with them. Yeah, you're the first consumers on the DX [00:19:07] swyx: team. Yeah. Yeah, I mean, the classic role of what we do there. Yeah. Okay, yeah, anything else? Any other call to action? [00:19:13] swyx: No, enjoy Dev Day. Thank you. Yeah. That's it. [00:19:16] Olivier Godement, Head of Product, OpenAI [00:19:16] AI Charlie: The latent space crew then talked to Olivier Godmont, head of product for the OpenAI platform, who led the entire Dev Day keynote and introduced all the major new features and updates that we talked about today. [00:19:28] swyx: Okay, so we are here with Olivier Godmont. That's right. [00:19:32] swyx: I don't pronounce French. That's fine. It was perfect. And it was amazing to see your keynote today. What was the back story of, of preparing something like this? Preparing, like, Dev Day? It [00:19:43] Olivier Godement: essentially came from a couple of places. Number one, excellent reception from last year's Dev Day. [00:19:48] Olivier Godement: Developers, startup founders, researchers want to spend more time with OpenAI, and we want to spend more time with them as well. And so for us, like, it was a no brainer, frankly, to do it again, like, you know, like a nice conference. The second thing is going global. We've done a few events like in Paris and like a few other like, you know, non European, non American countries. [00:20:05] Olivier Godement: And so this year we're doing SF, Singapore, and London. To frankly just meet more developers. [00:20:10] swyx: Yeah, I'm very excited for the Singapore one. [00:20:12] Olivier Godement: Ah, [00:20:12] swyx: yeah. Will you be [00:20:13] Olivier Godement: there? [00:20:14] swyx: I don't know. I don't know if I got an invite. No. I can't just talk to you. Yeah, like, and then there was some speculation around October 1st. [00:20:22] Olivier Godement: Yeah. Is it because [00:20:23] swyx: 01, October 1st? It [00:20:25] Olivier Godement: has nothing to do. I discovered the tweet yesterday where like, people are so creative. No one, there was no connection to October 1st. But in hindsight, that would have been a pretty good meme by Tiana. Okay. [00:20:37] swyx: Yeah, and you know, I think like, OpenAI's outreach to developers is something that I felt the whole in 2022, when like, you know, like, people were trying to build a chat GPT, and like, there was no function calling, all that stuff that you talked about in the past. [00:20:51] swyx: And that's why I started my own conference as like like, here's our little developer conference thing. And, but to see this OpenAI Dev Day now, and like to see so many developer oriented products coming to OpenAI, I think it's really encouraging. [00:21:02] Olivier Godement: Yeah, totally. It's that's what I said, essentially, like, developers are basically the people who make the best connection between the technology and, you know, the future, essentially. [00:21:14] Olivier Godement: Like, you know, essentially see a capability, see a low level, like, technology, and are like, hey, I see how that application or that use case that can be enabled. And so, in the direction of enabling, like, AGI, like, all of humanity, it's a no brainer for us, like, frankly, to partner with Devs. [00:21:31] Alessio: And most importantly, you almost never had waitlists, which, compared to like other releases, people usually, usually have. [00:21:38] Alessio: What is the, you know, you had from caching, you had real time voice API, we, you know, Shawn did a long Twitter thread, so people know the releases. Yeah. What is the thing that was like sneakily the hardest to actually get ready for, for that day, or like, what was the kind of like, you know, last 24 hours, anything that you didn't know was gonna work? [00:21:56] Olivier Godement: Yeah. The old Fairly, like, I would say, involved, like, features to ship. So the team has been working for a month, all of them. The one which I would say is the newest for OpenAI is the real time API. For a couple of reasons. I mean, one, you know, it's a new modality. Second, like, it's the first time that we have an actual, like, WebSocket based API. [00:22:16] Olivier Godement: And so, I would say that's the one that required, like, the most work over the month. To get right from a developer perspective and to also make sure that our existing safety mitigation that worked well with like real time audio in and audio out. [00:22:30] swyx: Yeah, what design choices or what was like the sort of design choices that you want to highlight? [00:22:35] swyx: Like, you know, like I think for me, like, WebSockets, you just receive a bunch of events. It's two way. I obviously don't have a ton of experience. I think a lot of developers are going to have to embrace this real time programming. Like, what are you designing for, or like, what advice would you have for developers exploring this? [00:22:51] Olivier Godement: The core design hypothesis was essentially, how do we enable, like, human level latency? We did a bunch of tests, like, on average, like, human beings, like, you know, takes, like, something like 300 milliseconds to converse with each other. And so that was the design principle, essentially. Like, working backward from that, and, you know, making the technology work. [00:23:11] Olivier Godement: And so we evaluated a few options, and WebSockets was the one that we landed on. So that was, like, one design choice. A few other, like, big design choices that we had to make prompt caching. Prompt caching, the design, like, target was automated from the get go. Like, zero code change from the developer. [00:23:27] Olivier Godement: That way you don't have to learn, like, what is a prompt prefix, and, you know, how long does a cache work, like, we just do it as much as we can, essentially. So that was a big design choice as well. And then finally, on distillation, like, and evaluation. The big design choice was something I learned at Skype, like in my previous job, like a philosophy around, like, a pit of success. [00:23:47] Olivier Godement: Like, what is essentially the, the, the minimum number of steps for the majority of developers to do the right thing? Because when you do evals on fat tuning, there are many, many ways, like, to mess it up, frankly, like, you know, and have, like, a crappy model, like, evals that tell, like, a wrong story. And so our whole design was, okay, we actually care about, like, helping people who don't have, like, that much experience, like, evaluating a model, like, get, like, in a few minutes, like, to a good spot. [00:24:11] Olivier Godement: And so how do we essentially enable that bit of success, like, in the product flow? [00:24:15] swyx: Yeah, yeah, I'm a little bit scared to fine tune especially for vision, because I don't know what I don't know for stuff like vision, right? Like, for text, I can evaluate pretty easily. For vision let's say I'm like trying to, one of your examples was grab. [00:24:33] swyx: Which, very close to home, I'm from Singapore. I think your example was like, they identified stop signs better. Why is that hard? Why do I have to fine tune that? If I fine tune that, do I lose other things? You know, like, there's a lot of unknowns with Vision that I think developers have to figure out. [00:24:50] swyx: For [00:24:50] Olivier Godement: sure. Vision is going to open up, like, a new, I would say, evaluation space. Because you're right, like, it's harder, like, you know, to tell correct from incorrect, essentially, with images. What I can say is we've been alpha testing, like, the Vision fine tuning, like, for several weeks at that point. We are seeing, like, even higher performance uplift compared to text fine tuning. [00:25:10] Olivier Godement: So that's, there is something here, like, we've been pretty impressed, like, in a good way, frankly. But, you know, how well it works. But for sure, like, you know, I expect the developers who are moving from one modality to, like, text and images will have, like, more, you know Testing, evaluation, like, you know, to set in place, like, to make sure it works well. [00:25:25] Alessio: The model distillation and evals is definitely, like, the most interesting. Moving away from just being a model provider to being a platform provider. How should people think about being the source of truth? Like, do you want OpenAI to be, like, the system of record of all the prompting? Because people sometimes store it in, like, different data sources. [00:25:41] Alessio: And then, is that going to be the same as the models evolve? So you don't have to worry about, you know, refactoring the data, like, things like that, or like future model structures. [00:25:51] Olivier Godement: The vision is if you want to be a source of truth, you have to earn it, right? Like, we're not going to force people, like, to pass us data. [00:25:57] Olivier Godement: There is no value prop, like, you know, for us to store the data. The vision here is at the moment, like, most developers, like, use like a one size fits all model, like be off the shelf, like GP40 essentially. The vision we have is fast forward a couple of years. I think, like, most developers will essentially, like, have a. [00:26:15] Olivier Godement: An automated, continuous, fine tuned model. The more, like, you use the model, the more data you pass to the model provider, like, the model is automatically, like, fine tuned, evaluated against some eval sets, and essentially, like, you don't have to every month, when there is a new snapshot, like, you know, to go online and, you know, try a few new things. [00:26:34] Olivier Godement: That's a direction. We are pretty far away from it. But I think, like, that evaluation and decision product are essentially a first good step in that direction. It's like, hey, it's you. I set it by that direction, and you give us the evaluation data. We can actually log your completion data and start to do some automation on your behalf. [00:26:52] Alessio: And then you can do evals for free if you share data with OpenAI. How should people think about when it's worth it, when it's not? Sometimes people get overly protective of their data when it's actually not that useful. But how should developers think about when it's right to do it, when not, or [00:27:07] Olivier Godement: if you have any thoughts on it? [00:27:08] Olivier Godement: The default policy is still the same, like, you know, we don't train on, like, any API data unless you opt in. What we've seen from feedback is evaluation can be expensive. Like, if you run, like, O1 evals on, like, thousands of samples Like, your build will get increased, like, you know, pretty pretty significantly. [00:27:22] Olivier Godement: That's problem statement number one. Problem statement number two is, essentially, I want to get to a world where whenever OpenAI ships a new model snapshot, we have full confidence that there is no regression for the task that developers care about. And for that to be the case, essentially, we need to get evals. [00:27:39] Olivier Godement: And so that, essentially, is a sort of a two bugs one stone. It's like, we subsidize, basically, the evals. And we also use the evals when we ship new models to make sure that we keep going in the right direction. So, in my sense, it's a win win, but again, completely opt in. I expect that many developers will not want to share their data, and that's perfectly fine to me. [00:27:56] swyx: Yeah, I think free evals though, very, very good incentive. I mean, it's a fair trade. You get data, we get free evals. Exactly, [00:28:04] Olivier Godement: and we sanitize PII, everything. We have no interest in the actual sensitive data. We just want to have good evaluation on the real use cases. [00:28:13] swyx: Like, I always want to eval the eval. I don't know if that ever came up. [00:28:17] swyx: Like, sometimes the evals themselves are wrong, and there's no way for me to tell you. [00:28:22] Olivier Godement: Everyone who is starting with LLM, teaching with LLM, is like, Yeah, evaluation, easy, you know, I've done testing, like, all my life. And then you start to actually be able to eval, understand, like, all the corner cases, And you realize, wow, there's like a whole field in itself. [00:28:35] Olivier Godement: So, yeah, good evaluation is hard and so, yeah. Yeah, yeah. [00:28:38] swyx: But I think there's a, you know, I just talked to Brain Trust which I think is one of your partners. Mm-Hmm. . They also emphasize code based evals versus your sort of low code. What I see is like, I don't know, maybe there's some more that you didn't demo. [00:28:53] swyx: YC is kind of like a low code experience, right, for evals. Would you ever support like a more code based, like, would I run code on OpenAI's eval platform? [00:29:02] Olivier Godement: For sure. I mean, we meet developers where they are, you know. At the moment, the demand was more for like, you know, easy to get started, like eval. But, you know, if we need to expose like an evaluation API, for instance, for people like, you know, to pass, like, you know, their existing test data we'll do it. [00:29:15] Olivier Godement: So yeah, there is no, you know, philosophical, I would say, like, you know, misalignment on that. Yeah, [00:29:19] swyx: yeah, yeah. What I think this is becoming, by the way, and I don't, like it's basically, like, you're becoming AWS. Like, the AI cloud. And I don't know if, like, that's a conscious strategy, or it's, like, It doesn't even have to be a conscious strategy. [00:29:33] swyx: Like, you're going to offer storage. You're going to offer compute. You're going to offer networking. I don't know what networking looks like. Networking is maybe, like, Caching or like it's a CDN. It's a prompt CDN. [00:29:45] Alex Volkov: Yeah, [00:29:45] swyx: but it's the AI versions of everything, right? Do you like do you see the analogies or? [00:29:52] Olivier Godement: Whatever Whatever I took to developers. I feel like Good models are just half of the story to build a good app There's a third model you need to do Evaluation is the perfect example. Like, you know, you can have the best model in the world If you're in the dark, like, you know, it's really hard to gain the confidence and so Our philosophy is [00:30:11] Olivier Godement: The whole like software development stack is being basically reinvented, you know, with LLMs. There is no freaking way that open AI can build everything. Like there is just too much to build, frankly. And so my philosophy is, essentially, we'll focus on like the tools which are like the closest to the model itself. [00:30:28] Olivier Godement: So that's why you see us like, you know, investing quite a bit in like fine tuning, distillation, our evaluation, because we think that it actually makes sense to have like in one spot, Like, you know, all of that. Like, there is some sort of virtual circle, essentially, that you can set in place. But stuff like, you know, LLMOps, like tools which are, like, further away from the model, I don't know if you want to do, like, you know, super elaborate, like, prompt management, or, you know, like, tooling, like, I'm not sure, like, you know, OpenAI has, like, such a big edge, frankly, like, you know, to build this sort of tools. [00:30:56] Olivier Godement: So that's how we view it at the moment. But again, frankly, the philosophy is super simple. The strategy is super simple. It's meeting developers where they want us to be. And so, you know that's frankly, like, you know, day in, day out, like, you know, what I try to do. [00:31:08] Alessio: Cool. Thank you so much for the time. [00:31:10] Alessio: I'm sure you, [00:31:10] swyx: Yeah, I have more questions on, a couple questions on voice, and then also, like, your call to action, like, what you want feedback on, right? So, I think we should spend a bit more time on voice, because I feel like that's, like, the big splash thing. I talked well Well, I mean, I mean, just what is the future of real time for OpenAI? [00:31:28] swyx: Yeah. Because I think obviously video is next. You already have it in the, the ChatGPT desktop app. Do we just have a permanent, like, you know, like, are developers just going to be, like, sending sockets back and forth with OpenAI? Like how do we program for that? Like, what what is the future? [00:31:44] Olivier Godement: Yeah, that makes sense. I think with multimodality, like, real time is quickly becoming, like, you know, essentially the right experience, like, to build an application. Yeah. So my expectation is that we'll see like a non trivial, like a volume of applications like moving to a real time API. Like if you zoom out, like, audio is really simple, like, audio until basically now. [00:32:05] Olivier Godement: Audio on the web, in apps, was basically very much like a second class citizen. Like, you basically did like an audio chatbot for users who did not have a choice. You know, they were like struggling to read, or I don't know, they were like not super educated with technology. And so, frankly, it was like the crappy option, you know, compared to text. [00:32:25] Olivier Godement: But when you talk to people in the real world, the vast majority of people, like, prefer to talk and listen instead of typing and writing. [00:32:34] swyx: We speak before we write. [00:32:35] Olivier Godement: Exactly. I don't know. I mean, I'm sure it's the case for you in Singapore. For me, my friends in Europe, the number of, like, WhatsApp, like, voice notes they receive every day, I mean, just people, it makes sense, frankly, like, you know. [00:32:45] Olivier Godement: Chinese. Chinese, yeah. [00:32:46] swyx: Yeah, [00:32:47] Olivier Godement: all voice. You know, it's easier. There is more emotions. I mean, you know, you get the point across, like, pretty well. And so my personal ambition for, like, the real time API and, like, audio in general is to make, like, audio and, like, multimodality, like, truly a first class experience. [00:33:01] Olivier Godement: Like, you know, if you're, like, you know, the amazing, like, super bold, like, start up out of YC, you want to build, like, the next, like, billion, like, you know, user application to make it, like, truly your first and make it feel, like, you know, an actual good, like, you know, product experience. So that's essentially the ambition, and I think, like, yeah, it could be pretty big. [00:33:17] swyx: Yeah. I think one, one people, one issue that people have with the voice so far as, as released in advanced voice mode is the refusals. [00:33:24] Alex Volkov: Yeah. [00:33:24] swyx: You guys had a very inspiring model spec. I think Joanne worked on that. Where you said, like, yeah, we don't want to overly refuse all the time. In fact, like, even if, like, not safe for work, like, in some occasions, it's okay. [00:33:38] swyx: How, is there an API that we can say, not safe for work, okay? [00:33:41] Olivier Godement: I think we'll get there. I think we'll get there. The mobile spec, like, nailed it, like, you know. It nailed it! It's so good! Yeah, we are not in the business of, like, policing, you know, if you can say, like, vulgar words or whatever. You know, there are some use cases, like, you know, I'm writing, like, a Hollywood, like, script I want to say, like, will go on, and it's perfectly fine, you know? [00:33:59] Olivier Godement: And so I think the direction where we'll go here is that basically There will always be like, you know, a set of behavior that we will, you know, just like forbid, frankly, because they're illegal against our terms of services. But then there will be like, you know, some more like risky, like themes, which are completely legal, like, you know, vulgar words or, you know, not safe for work stuff. [00:34:17] Olivier Godement: Where basically we'll expose like a controllable, like safety, like knobs in the API to basically allow you to say, hey, that theme okay, that theme not okay. How sensitive do you want the threshold to be on safety refusals? I think that's the Dijkstra. So a [00:34:31] swyx: safety API. [00:34:32] Olivier Godement: Yeah, in a way, yeah. [00:34:33] swyx: Yeah, we've never had that. [00:34:34] Olivier Godement: Yeah. ' [00:34:35] swyx: cause right now is you, it is whatever you decide. And then it's, that's it. That, that, that would be the main reason I don't use opening a voice is because of [00:34:42] Olivier Godement: it's over police. Over refuse over refusals. Yeah. Yeah, yeah. No, we gotta fix that. Yeah. Like singing, [00:34:47] Alessio: we're trying to do voice. I'm a singer. [00:34:49] swyx: And you, you locked off singing. [00:34:51] swyx: Yeah, [00:34:51] Alessio: yeah, yeah. [00:34:52] swyx: But I, I understand music gets you in trouble. Okay. Yeah. So then, and then just generally, like, what do you want to hear from developers? Right? We have, we have all developers watching you know, what feedback do you want? Any, anything specific as well, like from, especially from today anything that you are unsure about, that you are like, Our feedback could really help you decide. [00:35:09] swyx: For sure. [00:35:10] Olivier Godement: I think, essentially, it's becoming pretty clear after today that, you know, I would say the open end direction has become pretty clear, like, you know, after today. Investment in reasoning, investment in multimodality, Investment as well, like in, I would say, tool use, like function calling. To me, the biggest question I have is, you know, Where should we put the cursor next? [00:35:30] Olivier Godement: I think we need all three of them, frankly, like, you know, so we'll keep pushing. [00:35:33] swyx: Hire 10, 000 people, or actually, no need, build a bunch of bots. [00:35:37] Olivier Godement: Exactly, and so let's take O1 smart enough, like, for your problems? Like, you know, let's set aside for a second the existing models, like, for the apps that you would love to build, is O1 basically it in reasoning, or do we still have, like, you know, a step to do? [00:35:50] Olivier Godement: Preview is not enough, I [00:35:52] swyx: need the full one. [00:35:53] Olivier Godement: Yeah, so that's exactly that sort of feedback. Essentially what they would love to do is for developers I mean, there's a thing that Sam has been saying like over and over again, like, you know, it's easier said than done, but I think it's directionally correct. As a developer, as a founder, you basically want to build an app which is a bit too difficult for the model today, right? [00:36:12] Olivier Godement: Like, what you think is right, it's like, sort of working, sometimes not working. And that way, you know, that basically gives us like a goalpost, and be like, okay, that's what you need to enable with the next model release, like in a few months. And so I would say that Usually, like, that's the sort of feedback which is like the most useful that I can, like, directly, like, you know, incorporate. [00:36:33] swyx: Awesome. I think that's our time. Thank you so much, guys. Yeah, thank you so much. [00:36:38] AI Charlie: Thank you. We were particularly impressed that Olivier addressed the not safe for work moderation policy question head on, as that had only previously been picked up on in Reddit forums. This is an encouraging sign that we will return to in the closing candor with Sam Altman at the end of this episode. [00:36:57] Romain Huet, Head of DX, OpenAI [00:36:57] AI Charlie: Next, a chat with Roman Hewitt, friend of the pod, AI Engineer World's fair closing keynote speaker, and head of developer experience at OpenAI on his incredible live demos And advice to AI engineers on all the new modalities. [00:37:12] Alessio: Alright, we're live from OpenAI Dev Day. We're with Juan, who just did two great demos on, on stage. [00:37:17] Alessio: And he's been a friend of Latentspace, so thanks for taking some of the time. [00:37:20] Romain Huet: Of course, yeah, thank you for being here and spending the time with us today. [00:37:23] swyx: Yeah, I appreciate appreciate you guys putting this on. I, I know it's like extra work, but it really shows the developers that you're, Care and about reaching out. [00:37:31] Romain Huet: Yeah, of course, I think when you go back to the OpenAI mission, I think for us it's super important that we have the developers involved in everything we do. Making sure that you know, they have all of the tools they need to build successful apps. And we really believe that the developers are always going to invent the ideas, the prototypes, the fun factors of AI that we can't build ourselves. [00:37:49] Romain Huet: So it's really cool to have everyone here. [00:37:51] swyx: We had Michelle from you guys on. Yes, great episode. She very seriously said API is the path to AGI. Correct. And people in our YouTube comments were like, API is not AGI. I'm like, no, she's very serious. API is the path to AGI. Like, you're not going to build everything like the developers are, right? [00:38:08] swyx: Of [00:38:08] Romain Huet: course, yeah, that's the whole value of having a platform and an ecosystem of amazing builders who can, like, in turn, create all of these apps. I'm sure we talked about this before, but there's now more than 3 million developers building on OpenAI, so it's pretty exciting to see all of that energy into creating new things. [00:38:26] Alessio: I was going to say, you built two apps on stage today, an international space station tracker and then a drone. The hardest thing must have been opening Xcode and setting that up. Now, like, the models are so good that they can do everything else. Yes. You had two modes of interaction. You had kind of like a GPT app to get the plan with one, and then you had a cursor to do apply some of the changes. [00:38:47] Alessio: Correct. How should people think about the best way to consume the coding models, especially both for You know, brand new projects and then existing projects that you're trying to modify. [00:38:56] Romain Huet: Yeah. I mean, one of the things that's really cool about O1 Preview and O1 Mini being available in the API is that you can use it in your favorite tools like cursor like I did, right? [00:39:06] Romain Huet: And that's also what like Devin from Cognition can use in their own software engineering agents. In the case of Xcode, like, it's not quite deeply integrated in Xcode, so that's why I had like chat GPT side by side. But it's cool, right, because I could instruct O1 Preview to be, like, my coding partner and brainstorming partner for this app, but also consolidate all of the, the files and architect the app the way I wanted. [00:39:28] Romain Huet: So, all I had to do was just, like, port the code over to Xcode and zero shot the app build. I don't think I conveyed, by the way, how big a deal that is, but, like, you can now create an iPhone app from scratch, describing a lot of intricate details that you want, and your vision comes to life in, like, a minute. [00:39:47] Romain Huet: It's pretty outstanding. [00:39:48] swyx: I have to admit, I was a bit skeptical because if I open up SQL, I don't know anything about iOS programming. You know which file to paste it in. You probably set it up a little bit. So I'm like, I have to go home and test it. And I need the ChatGPT desktop app so that it can tell me where to click. [00:40:04] Romain Huet: Yeah, I mean like, Xcode and iOS development has become easier over the years since they introduced Swift and SwiftUI. I think back in the days of Objective C, or like, you know, the storyboard, it was a bit harder to get in for someone new. But now with Swift and SwiftUI, their dev tools are really exceptional. [00:40:23] Romain Huet: But now when you combine that with O1, as your brainstorming and coding partner, it's like your architect, effectively. That's the best way, I think, to describe O1. People ask me, like, can GPT 4 do some of that? And it certainly can. But I think it will just start spitting out code, right? And I think what's great about O1, is that it can, like, make up a plan. [00:40:42] Romain Huet: In this case, for instance, the iOS app had to fetch data from an API, it had to look at the docs, it had to look at, like, how do I parse this JSON, where do I store this thing, and kind of wire things up together. So that's where it really shines. Is mini or preview the better model that people should be using? [00:40:58] Romain Huet: Like, how? I think people should try both. We're obviously very excited about the upcoming O1 that we shared the evals for. But we noticed that O1 Mini is very, very good at everything math, coding, everything STEM. If you need for your kind of brainstorming or your kind of science part, you need some broader knowledge than reaching for O1 previews better. [00:41:20] Romain Huet: But yeah, I used O1 Mini for my second demo. And it worked perfectly. All I needed was very much like something rooted in code, architecting and wiring up like a front end, a backend, some UDP packets, some web sockets, something very specific. And it did that perfectly. [00:41:35] swyx: And then maybe just talking about voice and Wanderlust, the app that keeps on giving, what's the backstory behind like preparing for all of that? [00:41:44] Romain Huet: You know, it's funny because when last year for Dev Day, we were trying to think about what could be a great demo app to show like an assistive experience. I've always thought travel is a kind of a great use case because you have, like, pictures, you have locations, you have the need for translations, potentially. [00:42:01] Romain Huet: There's like so many use cases that are bounded to travel that I thought last year, let's use a travel app. And that's how Wanderlust came to be. But of course, a year ago, all we had was a text based assistant. And now we thought, well, if there's a voice modality, what if we just bring this app back as a wink. [00:42:19] Romain Huet: And what if we were interacting better with voice? And so with this new demo, what I showed was the ability to like, So, we wanted to have a complete conversation in real time with the app, but also the thing we wanted to highlight was the ability to call tools and functions, right? So, like in this case, we placed a phone call using the Twilio API, interfacing with our AI agents, but developers are so smart that they'll come up with so many great ideas that we could not think of ourselves, right? [00:42:48] Romain Huet: But what if you could have like a, you know, a 911 dispatcher? What if you could have like a customer service? Like center, that is much smarter than what we've been used to today. There's gonna be so many use cases for real time, it's awesome. [00:43:00] swyx: Yeah, and sometimes actually you, you, like this should kill phone trees. [00:43:04] swyx: Like there should not be like dial one [00:43:07] Romain Huet: of course para [00:43:08] swyx: espanol, you know? Yeah, exactly. Or whatever. I dunno. [00:43:12] Romain Huet: I mean, even you starting speaking Spanish would just do the thing, you know you don't even have to ask. So yeah, I'm excited for this future where we don't have to interact with those legacy systems. [00:43:22] swyx: Yeah. Yeah. Is there anything, so you are doing function calling in a streaming environment. So basically it's, it's web sockets. It's UDP, I think. It's basically not guaranteed to be exactly once delivery. Like, is there any coding challenges that you encountered when building this? [00:43:39] Romain Huet: Yeah, it's a bit more delicate to get into it. [00:43:41] Romain Huet: We also think that for now, what we, what we shipped is a, is a beta of this API. I think there's much more to build onto it. It does have the function calling and the tools. But we think that for instance, if you want to have something very robust, On your client side, maybe you want to have web RTC as a client, right? [00:43:58] Romain Huet: And, and as opposed to like directly working with the sockets at scale. So that's why we have partners like Life Kit and Agora if you want to, if you want to use them. And I'm sure we'll have many mores in the, in many more in the future. But yeah, we keep on iterating on that, and I'm sure the feedback of developers in the weeks to come is going to be super critical for us to get it right. [00:44:16] swyx: Yeah, I think LiveKit has been fairly public that they are used in, in the Chachapiti app. Like, is it, it's just all open source, and we just use it directly with OpenAI, or do we use LiveKit Cloud or something? [00:44:28] Romain Huet: So right now we, we released the API, we released some sample code also, and referenced clients for people to get started with our API. [00:44:35] Romain Huet: And we also partnered with LifeKit and Agora, so they also have their own, like ways to help you get started that plugs natively with the real time API. So depending on the use case, people can, can can decide what to use. If you're working on something that's completely client or if you're working on something on the server side, for the voice interaction, you may have different needs, so we want to support all of those. [00:44:55] Alessio: I know you gotta run. Is there anything that you want the AI engineering community to give feedback on specifically, like even down to like, you know, a specific API end point or like, what, what's like the thing that you want? Yeah. I [00:45:08] Romain Huet: mean, you know, if we take a step back, I think dev Day this year is all different from last year and, and in, in a few different ways. [00:45:15] Romain Huet: But one way is that we wanted to keep it intimate, even more intimate than last year. We wanted to make sure that the community is. Thank you very much for joining us on the Spotlight. That's why we have community talks and everything. And the takeaway here is like learning from the very best developers and AI engineers. [00:45:31] Romain Huet: And so, you know we want to learn from them. Most of what we shipped this morning, including things like prompt caching the ability to generate prompts quickly in the playground, or even things like vision fine tuning. These are all things that developers have been asking of us. And so, the takeaway I would, I would leave them with is to say like, Hey, the roadmap that we're working on is heavily influenced by them and their work. [00:45:53] Romain Huet: And so we love feedback From high feature requests, as you say, down to, like, very intricate details of an API endpoint, we love feedback, so yes that's, that's how we, that's how we build this API. [00:46:05] swyx: Yeah, I think the, the model distillation thing as well, it might be, like, the, the most boring, but, like, actually used a lot. [00:46:12] Romain Huet: True, yeah. And I think maybe the most unexpected, right, because I think if I, if I read Twitter correctly the past few days, a lot of people were expecting us. To shape the real time API for speech to speech. I don't think developers were expecting us to have more tools for distillation, and we really think that's gonna be a big deal, right? [00:46:30] Romain Huet: If you're building apps that have you know, you, you want high, like like low latency, low cost, but high performance, high quality on the use case distillation is gonna be amazing. [00:46:40] swyx: Yeah. I sat in the distillation session just now and they showed how they distilled from four oh to four mini and it was like only like a 2% hit in the performance and 50 next. [00:46:49] swyx: Yeah, [00:46:50] Romain Huet: I was there as well for the superhuman kind of use case inspired for an Ebola client. Yeah, this was really good. Cool man! so much for having me. Thanks again for being here today. It's always [00:47:00] AI Charlie: great to have you. As you might have picked up at the end of that chat, there were many sessions throughout the day focused on specific new capabilities. [00:47:08] Michelle Pokrass, Head of API at OpenAI ft. Simon Willison [00:47:08] AI Charlie: Like the new model distillation features combining EVOLs and fine tuning. For our next session, we are delighted to bring back two former guests of the pod, which is something listeners have been greatly enjoying in our second year of doing the Latent Space podcast. Michelle Pokras of the API team joined us recently to talk about structured outputs, and today gave an updated long form session at Dev Day, describing the implementation details of the new structured output mode. [00:47:39] AI Charlie: We also got her updated thoughts on the VoiceMode API we discussed in her episode, now that it is finally announced. She is joined by friend of the pod and super blogger, Simon Willison, who also came back as guest co host in our Dev Day. 2023 episode. [00:47:56] Alessio: Great, we're back live at Dev Day returning guest Michelle and then returning guest co host Fork. [00:48:03] Alessio: Fork, yeah, I don't know. I've lost count. I think it's been a few. Simon Willison is back. Yeah, we just wrapped, we just wrapped everything up. Congrats on, on getting everything everything live. Simon did a great, like, blog, so if you haven't caught up, I [00:48:17] Simon Willison: wrote my, I implemented it. Now, I'm starting my live blog while waiting for the first talk to start, using like GPT 4, I wrote me the Javascript, and I got that live just in time and then, yeah, I was live blogging the whole day. [00:48:28] swyx: Are you a cursor enjoyer? [00:48:29] Simon Willison: I haven't really gotten into cursor yet to be honest. I just haven't spent enough time for it to click, I think. I'm more a copy and paste things out of Cloud and chat GPT. Yeah. It's interesting. [00:48:39] swyx: Yeah. I've converted to cursor and 01 is so easy to just toggle on and off. [00:48:45] Alessio: What's your workflow? [00:48:46] Alessio: VS [00:48:48] Michelle Pokrass: Code co pilot, so Yep, same here. Team co pilot. Co pilot is actually the reason I joined OpenAI. It was, you know, before ChatGPT, this is the thing that really got me. So I'm still into it, but I keep meaning to try out Cursor, and I think now that things have calmed down, I'm gonna give it a real go. [00:49:03] swyx: Yeah, it's a big thing to change your tool of choice. [00:49:06] swyx: Yes, [00:49:06] Michelle Pokrass: yeah, I'm pretty dialed, so. [00:49:09] swyx: I mean, you know, if you want, you can just fork VS Code and make your own. That's the thing to dumb thing, right? We joked about doing a hackathon where the only thing you do is fork VS Code and bet me the best fork win. [00:49:20] Michelle Pokrass: Nice. [00:49:22] swyx: That's actually a really good idea. Yeah, what's up? [00:49:26] swyx: I mean, congrats on launching everything today. I know, like, we touched on it a little bit, but, like, everyone was kind of guessing that Voice API was coming, and, like, we talked about it in our episode. How do you feel going into the launch? Like, any design decisions that you want to highlight? [00:49:41] Michelle Pokrass: Yeah, super jazzed about it. The team has been working on it for a while. It's, like, a very different API for us. It's the first WebSocket API, so a lot of different design decisions to be made. It's, like, what kind of events do you send? When do you send an event? What are the event names? What do you send, like, on connection versus on future messages? [00:49:57] Michelle Pokrass: So there have been a lot of interesting decisions there. The team has also hacked together really cool projects as we've been testing it. One that I really liked is we had an internal hack a thon for the API team. And some folks built like a little hack that you could use to, like VIM with voice mode, so like, control vim, and you would tell them on like, nice, write a file and it would, you know, know all the vim commands and, and pipe those in. [00:50:18] Michelle Pokrass: So yeah, a lot of cool stuff we've been hacking on and really excited to see what people build with it. [00:50:23] Simon Willison: I've gotta call out a demo from today. I think it was Katja had a 3D visualization of the solar system, like WebGL solar system, you could talk to. That is one of the coolest conference demos I've ever seen. [00:50:33] Simon Willison: That was so convincing. I really want the code. I really want the code for that to get put out there. I'll talk [00:50:39] Michelle Pokrass: to the team. I think we can [00:50:40] Simon Willison: probably set it up. Absolutely beautiful example. And it made me realize that The Realtime API, this WebSocket API, it means that building a website that you can just talk to is easy now. [00:50:50] Simon Willison: It's like, it's not difficult to build, spin up a web app where you have a conversation with it, it calls functions for different things, it interacts with what's on the screen. I'm so excited about that. There are all of these projects I thought I'd never get to, and now I'm like, you know what? Spend a weekend on it. [00:51:04] Simon Willison: I could have a talk to your data, talk to your database. With a web, with a, with a little web application. Yeah. That's so [00:51:10] Michelle Pokrass: cool. Chat with PDF, but really chat with, really chat with pdf. No, completely. [00:51:15] Simon Willison: Totally. And that's not even hard to build. That's the crazy thing about this. [00:51:18] Michelle Pokrass: Yeah. Very cool. Yeah, when I first saw the space demo, I was actually just wowed and I, and I had a similar moment I think to all the people in the crowd. [00:51:27] Michelle Pokrass: I also thought Romain's drone demo was super cool. That was a super [00:51:30] Simon Willison: fun one as well. Yeah, I [00:51:31] Michelle Pokrass: actually saw that live this morning, and I was holding my breath for sure. [00:51:35] swyx: Knowing Romain, he probably spent the last two days working on it. But yeah, like, I'm curious about you were talking with Romain actually earlier about what the different levels of extraction are with WebSockets. [00:51:47] swyx: It's something that most developers have zero experience with. I have zero experience with it. Apparently there's like, the RTC level, and then there's the WebSocket level, and there's like, levels in between. [00:51:56] Simon Willison: Not so much. I mean, with WebSockets with the way they've built their API, you can connect directly to the OpenAI WebSocket from your browser. [00:52:04] Simon Willison: And it's actually just regular JavaScript. Like, you instantiate the WebSocket thing. It looks quite easy from their example code. The problem is that if you do that, you're sending your API key. From like, source code that anyone can view. Yeah, we [00:52:16] Michelle Pokrass: don't recommend that for production. [00:52:18] Simon Willison: So it doesn't work for production, which is frustrating, because it means that you have to build a proxy. [00:52:23] Simon Willison: So I'm going to have to go home and build myself a little WebSocket proxy just to hide my API key. I want OpenAI to do that. I want OpenAI to solve that problem for me, so I don't have to build the 1000th WebSocket proxy just for that one problem. Totally. [00:52:36] Michelle Pokrass: We've also partnered with some some partner solutions. [00:52:39] Michelle Pokrass: We've partnered with, I think, Agora. LiveKit a few others. So there's some loose solutions there, but yeah, we hear you. It's a beta. [00:52:49] swyx: Yeah, yeah, I mean You still want a solution where someone brings their own key, And they can trust that you [00:52:55] Simon Willison: don't get it. [00:52:56] swyx: Right? [00:52:56] Simon Willison: Kind of. I mean, I've been building a lot of bring your own key apps, Where it's my HTML and JavaScript, I store the key in local storage in their browser, And it never goes anywhere near my server. [00:53:06] Simon Willison: Which works, but how do they trust me? How do they know I'm not gonna ship another piece of javascript that steals the key from them? And so, nominally, this actually [00:53:13] swyx: comes with the crypto background. This is what MetaMask does. Where Yeah, it's a [00:53:18] Michelle Pokrass: public private key thing. Yeah. Yeah. [00:53:20] swyx: Like, why doesn't OpenAI do that? [00:53:22] swyx: I don't know if, obviously it's [00:53:24] Michelle Pokrass: I mean, as with most things, I think there's, like, some really interesting questions. And the answer is just, you know, it's not been the top priority and it's hard for a small team to do everything. I have been hearing a lot more about the need for things like sign in with OpenAI. [00:53:40] Simon Willison: I want OAuth. I want to bounce my users through chat GPT and I get back a token that lets me spend up to 4 on the API on their behalf. Then I could ship all of my stupid little experiments, which currently require people to copy and paste their API key in, which cuts off everyone. Nobody knows how to do that. [00:53:57] Michelle Pokrass: Totally, I hear you. Something we're thinking about, and yeah, stay tuned. [00:54:01] swyx: Yeah, yeah right now, I think the only player in town is OpenRouter that is basically, it's funny, it was made by I forget his name but he used to be CTO of OpenSea, and the first thing he did when he came over was build Metamask for AI. [00:54:16] Michelle Pokrass: Totally. Yeah, very cool. [00:54:19] Alessio: What's the most underrated release from today? [00:54:23] Michelle Pokrass: Vision Fine Tuning. Vision Fine Tuning is so underrated. For the past, like, two months, whenever I talk to founders, they tell me this is the thing they need most. A lot of people are doing, like, OCR on very bespoke formats, like government documents, and Vision Fine Tuning can help a lot with that use case. [00:54:39] Michelle Pokrass: Also, bounding boxes. People have found, like, a lot of improvements for bounding boxes with Visionfine Tuning. So yeah, I think it's pretty slept on and people should try it. You only really need 100 images to get going. [00:54:49] Simon Willison: Tell me more about bounding boxes. I didn't think that GPT 4 Vision could do bounding boxes at all. [00:54:55] Michelle Pokrass: Yeah, it's actually not that amazing at it, we're working on it, but with fine tuning, you can make it really good for your use case. [00:55:02] Simon Willison: That's cool, because I've been using Google Gemini's bounding block stuff recently, it's very, very impressive. [00:55:06] Michelle Pokrass: Yeah, totally. But [00:55:07] Simon Willison: being able to fine tune a model for that. The first thing I'm going to do with fine tuning for images is, I've got fine tuning. [00:55:13] Simon Willison: And I'm going to fine tune a model that can tell which chicken is which. Which is hard because three of them are grey. So there's a little bit of Okay, this is [00:55:20] Michelle Pokrass: my new favourite use case. Yeah, it's [00:55:22] Simon Willison: I've managed to do it with prompting. Just like, I gave Claude Pictures of all of the chickens and then said, okay, which chicken is this? [00:55:30] Michelle Pokrass: Yeah, [00:55:30] Simon Willison: but it's not quite good enough because it confuses the great chicken. Listen, [00:55:33] Michelle Pokrass: we can close that eval gap. Yeah That's it's [00:55:36] Simon Willison: gonna be a great eval. My chicken eval is gonna be fantastic. [00:55:39] Michelle Pokrass: I'm also really jazzed about the evals product It's kind of like a sub launch of the distillation thing But people have been struggling to make evals and the first time I saw the flow with how easy it is to make an eval And in our product, I was just blown away so I recommend people really try that. [00:55:53] Michelle Pokrass: I think that's what's holding a lot of people back from really investing in AI, because they just have a hard time figuring out if it's going well for their use case. So we've been working on making it easier to do that. [00:56:03] Alessio: Does the eval product include structured output testing? Like, function calling and things? [00:56:08] Alessio: Yeah, you can [00:56:08] Michelle Pokrass: check if it matches your JSON schema yeah. [00:56:12] swyx: I mean, we have guaranteed structured output anyway, right? Well, but So we don't have to test it. Well, [00:56:18] Michelle Pokrass: not the schema, but like the See, these seem easy to tell apart. I think so. So I might call them a function, [00:56:24] Alessio: or Oh, I see. You're gonna write schema, wrong output. [00:56:27] Alessio: So you can do function [00:56:28] swyx: calling testing. Right. [00:56:29] Michelle Pokrass: I'm pretty sure. I'll have to check that for you, but I think [00:56:31] Alessio: so. Yeah, yeah, yeah. We'll make sure it's sent [00:56:33] swyx: out. [00:56:33] Alessio: How do you think about the evolution of, like, the API design? I think to me that's, like, the most important thing, so even with the OpenAI levels, like, chatbots, I can understand what the API design looks like. Reasoning, I can kind of understand it, even though, like, train of thought kind of changes things. [00:56:49] Alessio: As you think about real time voice, and then you think about agents, it's like, how do you think about how you design the API, and, like, what the shape of it is? [00:56:58] Michelle Pokrass: Yeah, so I think we're starting with the lowest level capabilities. And then we build on top of that, as we know that they're useful. So, a really good example of this is Realtime. [00:57:07] Michelle Pokrass: We're actually going to be shipping audio capabilities in chat completions. So this is like the lowest level capability. So you supply in audio, and you can get back raw audio, and it works at the request response layer. But, in through building advanced voice mode, we realized ourselves that like, it's not It's pretty hard to do with something like Chat Completions, and so that led us to building this WebSocket API. [00:57:28] Michelle Pokrass: So we really learned a lot from our own tools, and we think, you know, the Chat Completions thing is nice, and for certain use cases, or async stuff, but you're really gonna want a real time API? And then as we, you know, test more with developers, we might see that it makes sense to have like another layer of abstraction on top of that. [00:57:44] Michelle Pokrass: Something like closer to you know, more client side libraries. But, for now, you know, that's where we feel we have like a really good point of view. [00:57:52] Simon Willison: So that's a question I have is if I've got a half hour long audio recording, At the moment, the only way I can feed that in is if I call the WebSocket API and slice it up into little JSON basics for snippets and fire them all over. [00:58:04] Simon Willison: That's it. In that case, I'd rather just give you a, like an image in the chat completion API, give you a URL files and input. Is that something That's what we're [00:58:11] Michelle Pokrass: going to do. [00:58:12] Simon Willison: Oh, thank goodness for that. [00:58:13] Michelle Pokrass: Yes. It's in the blog post. I think it's a short one liner, but it's rolling out, I think, in the coming weeks. [00:58:17] Michelle Pokrass: Oh, wow. [00:58:18] Simon Willison: Oh, really soon then. [00:58:19] Michelle Pokrass: Yeah, the team has been sprinting we're just putting finishing touches on stuff. Do you [00:58:22] Simon Willison: have a feel for the length limit on that? [00:58:24] Michelle Pokrass: I don't have it off the top. Okay. Sorry. [00:58:26] Simon Willison: Because, yeah, often I want to do, I do a lot of work with, like, transcripts of hour long YouTube videos, which Yeah. [00:58:31] Simon Willison: Yeah. Currently, I run them through Whisper and then I do the transcript that way, but being able to do the multimodal thing with those would be really useful. [00:58:37] Michelle Pokrass: Totally, yeah. We're really jazzed about it. We want to basically give the lowest capabilities we have, lowest level capabilities, and, you know, the things that make it easier to use. [00:58:45] Michelle Pokrass: And so, you know, targeting kind of both. I [00:58:50] Simon Willison: just realized what I can do, though, is I do a lot of Unix utilities, little, like, Unix things. I want to be able to pipe the output of a command into something which streams that up to the WebSocket API and then speaks it out loud. So I can do streaming speech of the output of things. [00:59:06] Simon Willison: That should work. Like, I think you've given me everything I need for that. That's cool. [00:59:10] Michelle Pokrass: Yeah. Excited to see what you build. Is [00:59:14] swyx: there I heard there are, like, multiple competing solutions. And you guys evaluated before you picked WebSockets. Like server set events, polling, I don't, like, can you give, like, your thoughts on, like, the live updating paradigms that you guys looked at? [00:59:31] swyx: Because I think a lot of engineers have looked at stuff like this. [00:59:34] Michelle Pokrass: Well, I think WebSockets are just a natural fit for bi directional streaming. You know, other places I've worked, like, Coinbase, we had a WebSocket API for pricing data. I think it's just like a very natural format. [00:59:46] swyx: So it wasn't even really that controversial at all? [00:59:49] Michelle Pokrass: I don't think it was super controversial. I mean, we definitely explored the space a little bit, but I think we came to WebSockets pretty quickly. [00:59:56] swyx: Cool. Video? [00:59:58] Michelle Pokrass: Yeah. Not yet, but, you know. [01:00:03] swyx: I actually was hoping for the chat, GPT desktop app with video today. Yeah. Yeah. [01:00:09] Simon Willison: Oh, [01:00:10] Michelle Pokrass: my [01:00:11] Simon Willison: question is one frame a second. [01:00:16] Simon Willison: How frequently? Yeah. [01:00:19] swyx: Because Yeah, I mean sending a sending a whole video frame of like a 1080p screen. Maybe it might be too much What's the limitations on a on a WebSocket chunk going over? I don't know [01:00:33] Michelle Pokrass: I don't have that off the top [01:00:34] Simon Willison: Like Google Gemini you can do an hour's worth of video in their context window and just by slicing it up into one frame At ten frames a second and it does work so I Don't know. [01:00:46] Simon Willison: I'm I'm not sure But then that's the weird thing about Gemini is it's so good at you just giving it a flood of individual frames It'll be interesting to see if GPT 4. 0 can handle that or not [01:00:55] Alessio: Do you have any more feature requests? It's been a long day for everybody, but you got you got me show right here So my one [01:01:03] Simon Willison: is I want you to do all of the accounting for me I want my users to be able to run my app And I want them to call your APIs with their user ID and have you go, oh, they've spent 30 cents. [01:01:15] Simon Willison: Check, cut them off at a dollar. I can like, check how much they spent. All of that stuff, because I'm having to build that at the moment, and I really don't want to. I don't want to be a token accountant. I want you to do the token accounting for me. [01:01:26] Michelle Pokrass: Yeah, totally. I hear you. It's good feedback. [01:01:29] swyx: Well, like, how does that contrast with your actual priorities, right? [01:01:32] swyx: Like, I feel like you have a bunch of priorities. They showed some on stage with multi modality and all that. [01:01:37] Michelle Pokrass: Yeah. [01:01:37] swyx: Like [01:01:39] Michelle Pokrass: Yeah it's good feedback. It's hard to say. I would say things change really quickly. Things that are big adop big blockers for user adoption we find very important. And, yeah. It's a rolling prioritization. [01:01:53] Michelle Pokrass: Yeah. [01:01:54] swyx: No assistance API update. [01:01:56] Michelle Pokrass: Not at this time. Yeah. Yeah. [01:01:59] swyx: I was hoping for, like, an O1 native. Do thing in assistance? Yeah. I thought they would go well together. we're still [01:02:07] Michelle Pokrass: kind of iterating on the formats, I think there are some problems with the assistance API. Some things it does really well. [01:02:13] Michelle Pokrass: And I think we'll keep iterating and land on something really good. But just, you know, it wasn't quite ready yet. Some of the things that are good in the assistance API is hosted tools. People really like hosted tools and especially RAG. And then some things that are, you know, less intuitive is just how many API requests you need to get going with the assistance API. [01:02:30] Michelle Pokrass: It's [01:02:30] Simon Willison: quite. [01:02:30] Michelle Pokrass: It's quite a lot. Yeah, you gotta create an assistant, you gotta create a thread, you gotta, you know, do all this stuff. So yeah, it's something we're thinking about. It shouldn't be so hard. [01:02:39] Simon Willison: The only thing I've used it for so far is Code Interpreter. It's like it's an API to Code Interpreter. [01:02:43] Simon Willison: Crazy exciting. Yeah. [01:02:44] Michelle Pokrass: Yes, we want to fix, we want to fix that and make it easier to use, so. I [01:02:48] Simon Willison: want code intercepts over WebSockets, that would be wildly interesting. [01:02:53] swyx: Yeah, do you, do you want to bring your own code interpreter or you want to use OpenAI's one? I want to [01:02:57] Simon Willison: use theirs, because code intercepts is a hard problem, sandboxing and all of that stuff is Yeah, but there's a bunch [01:03:02] swyx: of code interpreter as a [01:03:03] Simon Willison: service [01:03:04] swyx: things out there. [01:03:04] swyx: There are a few now, yeah. Because there's, I think you don't Allow arbitrary installation of packages. Oh, they do. Unless [01:03:10] Simon Willison: they really do actually use your hack code. It, huh? [01:03:13] Michelle Pokrass: Yeah, [01:03:13] Simon Willison: and I do. [01:03:14] Michelle Pokrass: Yeah. You upload a pit package, [01:03:16] Simon Willison: you can run, you can compile C code and code interpreter. I know. You know, to do it. [01:03:20] Simon Willison: That's a hack. Oh, it's such a glorious hack though. Okay. I've had it Write me custom seql light extensions in C and compile them and run them inside of Python and it works. [01:03:31] swyx: I mean, yeah, there's, there's others. E two B is one of them, like, yeah. It'll be interesting to see what the real time version of that will be. [01:03:39] Alessio: Awesome, Michelle. Thank you for the update. We left the episode as, what will voice mode look like? Obviously, you knew what it looked like, but you didn't say it, so now you could share this. [01:03:50] Alessio: Yeah, here we are. Hope you [01:03:51] AI Charlie: guys [01:03:51] Alessio: like [01:03:52] swyx: it. Yeah, awesome. That's [01:03:53] Alessio: it. [01:03:53] AI Charlie: Our final guest today, and also a familiar, recent voice on the Latent Space pod, presented at one of the community talks at this year's Dev Day. Alistair Pullen of Cosene made a huge impression with all of you. Special shout out to listeners like Jesse from Morphlabs, when he came on to talk about how he created synthetic datasets to fine tune the largest LORAs that had ever been created for GPT 4. [01:04:20] AI Charlie: 0 to post the highest ever scores on SWEbench and SWEbench Verified. While not getting recognition for it, because he refused to disclose his reasoning traces to the SWEbench team. Now that OpenAI's R1 preview is announced, it is incredible to see the OpenAI team also obscure their chain of thought traces for competitive reasons, and still perform lower than Cozine's genie model. [01:04:45] Alistair Pullen, CEO, Cosine (Genie) [01:04:45] AI Charlie: We snagged some time with Ali to break down what has happened since his episode aired. [01:04:50] swyx: Welcome back, Ali. Thank you so much. Thanks for having me. So you just spoke at OpenAI Dev Day. What was the experience like? Did they reach out to you? You seem to have a very close relationship. [01:04:59] Alessio: Yeah, so off the back of, off the back of the work that we've done, that we spoke about last time we saw each other I think that OpenAI definitely felt that the work we've been doing around fine tuning was worth sharing. [01:05:10] Alessio: I would obviously tend to agree, but today today I spoke about some of the techniques that we learned. Obviously it was like a non linear path arriving to where we've arrived and the techniques that we've built to build Genie. So I definitely, I think I shared a few, a few extra pieces about some of the techniques and how it really works under the hood. [01:05:25] Alessio: How you generate a data set to show the model how to do what we show the model. And that was mainly what I spoke about today. I mean, yeah, they reached out and they were, I was, I was Super excited at the opportunity, obviously, like, it's not every day that you get to come and do this. Especially in San Francisco, so Yeah, they reached out and they were like, do you want to talk at Dev Day? [01:05:41] Alessio: You can speak about basically anything you want related to what you've built, and I was like, sure, that's amazing. I'll talk about fine tuning, how you build a model that does this software engineering, so yeah. [01:05:50] swyx: Yeah and the trick here is when we talked, O1 was not out. No, it wasn't. Did you know about O1, or? [01:05:57] Alessio: I didn't know. I knew some bits and pieces. No, not really. I knew a reasoning model was on the way. I didn't know what it was going to be called. I knew as much as everyone else. Strawberry was the name back then. Because, [01:06:08] swyx: you know, I'll fast forward. You were the first to hide your chain of thought, reasoning traces as IP. [01:06:14] swyx: Yes. Right? Famously, that got you in trouble with 3Bench or whatever. Yes. I feel slightly vindicated by that now. And now, obviously, O1 is doing it. Yeah, the [01:06:22] Alessio: fact that, yeah, I mean, like, I think it's, I think it's true to say right now that the reasoning of your model gives you the edge that you have. Unlike. [01:06:33] Alessio: The amount of effort that we put into our data pipeline to generate these human like reasoning traces was, I mean, that wasn't for nothing. We knew that this was the way that you'd unlock more performance, getting the model to think in a specific way. In our case, we wanted it to think like a software engineer. [01:06:46] Alessio: But, yeah, I think, I think that, The approach that other people have taken, like OpenAI, in terms of reasoning, has definitely showed us that we were going down the right path pretty early on. And even now, we've started replacing some of the reasoning traces in our genie model with reasoning traces generated by O1, or at least in tandem with O1. [01:07:09] Alessio: And we've already started seeing improvements in performance from that point. But no, like back to your point, in terms of like the, the whole like approach. Withholding them. I, I, I, I still think that that was the right decision to do because of the very reason that everyone else has decided to, to, to, to not share those things. [01:07:26] Alessio: It's, it is exactly, it shows exactly how we do what we do and that is our edge at the moment. So, [01:07:32] Alessio: yeah. As a founder, so, they also feature Cognition on, on stage, talk about that. How does that make you feel that like, you know, they're like, hey, 01 is so much better, makes us better. For you, it should be like. [01:07:45] Alessio: Oh, I'm so excited about it too, because now all of a sudden it's like, it kind of like, raises the floor for everybody, like, how should people, especially new founders, how should they think about, you know, worrying about the new model versus like, being excited about them just focusing on like, the core FP and maybe switching out some of the parts, like you mentioned. [01:08:00] Alessio: Yeah, I, I, I, I, speaking for us, I mean obviously like, we were extremely excited about O1 because, At that point, the process of reasoning is obviously very much baked into the model. We fundamentally, if you like, remove all distractions and everything, we are a reasoning company. Right? We want to reason in the way that a software engineer reasons. [01:08:18] Alessio: So when I saw that model announced, I thought immediately, well, I can improve the quality of my traces coming out of my pipeline, so like, my signal to noise ratio gets better. And then, not immediately, but down the line, I'm going to be able to train those traces into O1 itself. So I'm going to get even more performance that way as well. [01:08:35] Alessio: So it's For us, a really nice position to be in, to be able to take advantage of it, both on the prompted side and the fine tuned side. And also because, fundamentally, like, we are, I think, fairly clearly in a position now where we don't have to worry about what happens when O2 comes out, what happens when O3 comes out. [01:08:51] Alessio: This process continues, like, even going from You know, when we first started going from 3. 5 to 4, we saw this happen and then from 4 turbo to 4. 0 and then from 4. 0 to 0. 1, we've seen the performance get better every time and I think, I mean, like, the crude advice I'd give to any startup founder is try to put yourself in a position where you can take advantage of the same, you know, like, C level rise every time, essentially. [01:09:15] swyx: Do you make anything out of the fact that you were able to take 4. 0 and fine tune it higher than 0. 1 currently scores on SweeBench Verified? Yeah, I mean like, [01:09:25] Alessio: that was obviously, to be honest with you, you realized that before I did. Adding value. Yes, absolutely, that's a value add investor right there. No, obviously I think it's been, that in of itself is really vindicating to see because I think, I think we have, heard from some people, not a lot of people, but some people saying, well, okay, well, if I, one can reason, then what's the point of doing your reasoning, but it shows how much more signal is in, like the custom reasoning that we generate. [01:09:52] Alessio: And again, it's the, it's the very sort of obvious thing. If you take something that's made to be general and you make it specific, of course, it's going to be better at that thing. Right? So it was obviously great to see, like, we still are better than no one out of the box. You know, even with an older model, and I'm sure that that's, you know, That delta will continue to grow once we're able to train O1, and once we've done more work on our dataset using O1, like, that delta will grow as well. [01:10:13] swyx: It's not obvious to me that they will allow you to fine tune O1, but, you know, maybe they'll try. I think the, the, the core question that OpenAI really doesn't want you to figure out is can you use an open source model and beat O1? [01:10:28] Romain Huet: Interesting. Because, because [01:10:30] swyx: you basically have shown proof of concept that a non O1 model can beat O1. [01:10:35] swyx: And their whole L1 marketing is, don't bother trying. Like, don't bother stitching together multiple chain of thought calls. We did something special, secret sauce, you don't know anything about it. And somehow, you know, your 4. 0 chain of thought reasoning as a software engineer is still better. Maybe it doesn't last. [01:10:53] swyx: Maybe they're going to run L1 for five hours instead of five minutes, and then suddenly it works. So, I don't know. [01:10:59] Alessio: It's hard to know. I mean, one of the things that we just want to do out of sheer curiosity is do something like fine tune 405B on the same dataset. Like, same context window length, right? So, it should be fairly easy. [01:11:09] Alessio: We haven't done it yet. Truthfully, we have been so swamped with the waitlist, shipping product, you know, dev day, like, you know, onboarding customers from our waitlist. All these different things have gotten in the way, but it is definitely something out of more curiosity than anything else I'd like to try out. [01:11:23] Alessio: But also It opens up a new vector of like, if someone has a VPC where they can't deploy an OpenAI model, but they might be able to deploy an open source model, it opens that up for us as well from a customer perspective. So it'll probably be quite useful. I'd be very keen to see what the results are though. [01:11:38] Alessio: I suspect the answer is yes, [01:11:40] swyx: but it may be hard to do. So like Reflection70b was like a really crappy attempt at doing it. You guys were much better, and that's why we had you on the show. I, yeah, I'm interested to see if there's an OpenO1 basically. If people want OpenO1. [01:11:53] Alessio: Yeah, I'm sure they do. As soon as we, as soon as we do it, I'm like, Once we've wrapped up what we're doing in San Francisco, I'm sure we'll give it a go. [01:12:01] Alessio: I spoke to some guys today, actually, about fine tuning 405B, who might be able to allow us to do it very, like, very easily. I don't want to have to basically do all the setup myself. So, yeah, that might happen sooner rather than later. [01:12:15] Alessio: Anything from the releases today that you're super excited about? So prompt caching, I'm guessing when you're like dealing with a lot of codebases, that might be helpful. [01:12:22] Alessio: Is there anything with vision fine tuning related to [01:12:25] Alessio: like more like UI related development? Yeah, definitely. Yeah, I mean like we were talking, it's funny, like my co founder Sam, who you've met, and I were talking about the idea of doing vision fine tuning. Like, way back, like, well over a year ago, before Genie existed as it does now when we, when we collected our original dataset to do what we do now whenever there were image links and links to, like like, graphical resources and stuff, we also pulled that in as well. [01:12:50] Alessio: We never had the opportunity to use it, but it's something we have in storage. And, again, like, when we have the time, it's something that I'm super excited, particularly on the UI side. To be able to, like, leverage, particularly if you think about one of the things, I mean, not to sidetrack, but one of the things we've noticed is, I know Swebench is, like, the most commonly talked about thing, and honestly, it's a very, it's an amazing project, but, One of the things we've learned the most from actually shipping this product to users is, It's a pretty bad proxy at telling us how competent the model is, so, for example, When people are doing, like, React development using Genie, For us, it's impossible to know whether what it's written has actually done, you know, done what it wanted to. [01:13:26] Alessio: So at least even using, like, the fine tuning provision to be able to help eval, like, what we output is already something that's very useful. But also, in terms of being able to pair, here's a UI I want, here's the code that actually, like, represents that UI, is also going to be super useful as well, I think. [01:13:42] Alessio: In terms of generally, what have I been most impressed by? The distillation thing is awesome. I think we'll probably end up using it in places. But what it shows me more broadly about OpenAI's approach is they're going to be building a lot of the things that we've had to hack together internally, in terms from a tooling point of view, just to make our lives so much easier. [01:14:03] Alessio: And I've spoken to, you know, John, the head of fine tuning, extensively about this. But there's a bunch of tools that we've had to build internally for things like dealing with model lineage, dealing with dataset lineage, because it gets so messy so quickly, that we would love OpenAI to build. Like, absolutely would love them to build it. [01:14:19] Alessio: It's not, it's not what gives us our edge, but it certainly means that then we don't have to build it and maintain it afterwards. So, it's a really good first step, I think, in, like, the overall maturity of the fine tuning product and API in terms of where they're going to see those early products. And I think that they'll be continuing in that direction going on. [01:14:37] Alessio: Did you not, so there's a very [01:14:39] swyx: active ecosystem of LLLmaps tools. Mm hmm. Did you not evaluate those before building your own? [01:14:47] Alessio: We did, but I think fundamentally, like, No more. Yeah, like, I think, in a lot of places, it was never a big enough pain point to be like, oh, we absolutely must outsource this. It's definitely, in many places, something that you can hack a script together In a day or two, and then hook it up to our already existing internal tool UI, and then you have, you know, what you need, and whenever you need a new thing, you just tack it on. [01:15:14] Alessio: But for, like, all of these LLM Ops tools, I've never felt the pain point enough to really, like, bother, and that's not to deride them at all, I'm sure many people find them useful, but just for us as a company, we've never felt the need for them. So it's great that, it's great that OpenAI are going to build them in because it's really nice to have them there, for sure. [01:15:36] Alessio: But it's not something that, like, I'd ever consider really paying for externally or something like that, if that makes sense. [01:15:40] swyx: Yeah. Does voice mode factor into Genie? [01:15:44] Alessio: Maybe one day, that'd be sick, wouldn't it? I don't know. Yeah, I think so. You're [01:15:48] swyx: the first person, we've been asking this question to everybody. [01:15:50] swyx: Yeah, I think. You're the first person to not mention voice mode. [01:15:52] Alessio: Oh, well, it's, it's, it's currently so distant from what we do. But I definitely think, like, this whole talk, if we want it to be a full on AI software engineering colleague, like, there is definitely a vector in some way that you can build that in. [01:16:06] Alessio: Maybe even during the ideation stage, talking through a problem with Genius in terms of how we want to build something down the line. I think that might be useful, but honestly, like, that would be nice to have when we have the time. Yeah, amazing. [01:16:19] swyx: One last question. On your in your talk, you mentioned a lot about So you're curating your data and your distribution and all that, and before we sat down you talked a little bit about having to diversify your dataset. [01:16:30] swyx: Absolutely, yeah. What's driving that, [01:16:32] Alessio: what are you finding? So, we have been rolling people off the waitlist that we sort of amassed when we announced when I last saw you. And it's been really interesting because as I may have mentioned on the podcast, like we had to be very opinionated about the data mix and the data set that we put together for like sort of the V0 of Genie. [01:16:49] Alessio: Again, like, to your point, Javascript, Javascript, Javascript, Python, right? There's a lot of Javascripts in its various forms in there. But it turns out that when we've shipped it to the very early alpha users we rolled it out to for example, we had some guys using it with a C sharp codebase. [01:17:05] Alessio: And C sharp currently represents, I think, about 3 percent of the overall data mix. And they weren't getting the levels of performance that they saw when they tried it with a Python codebase. And It was obviously not great for them to have a bad experience, but it was nice to be able to correlate it with the actual, like, objective data mix that we saw. [01:17:25] Alessio: So we did what we've been doing is like little top up fine tunes where we take, like, the general genie model and do an incremental fine tune on top with just a bit more data for a given, you know, vertical language. And we've been seeing improvements coming from that. So. Again, this is one of the great things about sort of baptism by fire and letting people use it and giving you feedback and telling you where it sucks. [01:17:46] Alessio: Because that is not something that we could have just known ahead of time. So I want that data mix to, over time as we roll it out to more and more people, and we are trying to do that as fast as possible, but we're still a team of five for the time being. And so To be as general and as representative of what our users do as possible and not what we think they need. [01:18:02] swyx: Yeah, so every customer is going to have their own fine [01:18:05] Alessio: tune. There is going to be the option to, yeah, there is going to be the option to fine tune the model on your code base. It won't be in, like, the base pricing tier, but you will definitely be able to do that. It will go through All of your codebase history, learn how everything happened, and then you'll have an incrementally fine tuned genie just on your codebase. [01:18:23] Alessio: That's what enterprises really love the idea of. Perfect. [01:18:27] swyx: Anything else? Yeah, that's it. Thank you so much. Thank you so [01:18:29] Alessio: much, guys. Good to [01:18:30] swyx: see you. [01:18:31] Sam Altman + Kevin Weill Q&A [01:18:31] AI Charlie: Lastly, this year's Dev Day ended with an extended Q& A with Sam Altman and Kevin Weil. We think both the questions asked and answers given were particularly insightful, so we are posting what we could snag of the audio here from publicly available sources. [01:18:48] AI Charlie: Credited in the show notes, for you to pick through. If the poorer quality audio here is a problem, we recommend waiting for approximately 1 to 2 months until the final video is released on YouTube. In the meantime, we particularly recommend Sam's answers on the moderation policy, on the underappreciated importance of agents and AI employees beyond level 3. [01:19:11] AI Charlie: And his projections of the intelligence of O1, O2, and O3 models in future. [01:19:23] Speaker 17: Alright, I think everybody knows you. For those who don't know me, I'm Kevin Wheel, Chief Product Officer at OpenAI. I have the good fortune of getting to turn the amazing research that our research teams do into the products that you all use every day and the APIs that you all build on every day. I thought we'd start with some audience engagement here. [01:19:42] Speaker 17: So on the count of three, I want to count to three, and I want you all to say, of all the things that you saw launched here today, what's the first thing you're going to integrate? It's the thing you're most excited to build on. Alright? You gotta do it. Alright? One, two, three. Real time [01:20:01] Alex Volkov: API! [01:20:03] Speaker 17: I'll say personally, I'm super excited about our distillation products. [01:20:07] Speaker 17: I think that's going to be really, really interesting. I'm also excited to see what you all do with advanced voicemail with the real time API, and with vision fine tuning in particular. Okay, so I've got some questions for Sam, I've got my CEO here in the hot seat, let's see if I can't make a career limiting move. [01:20:30] Speaker 17: So we'll start this we'll start with an easy one, Sam. How close are we to AGI? [01:20:37] Sam Altman: You know, we used to, every time we finished a system, we would say like, in what way is this not an AGI? Okay. And it used to be like, very easy, you could like, make a little robotic hand that does a prefix cube, or a dotabot, and it's like, oh, it does some things, but definitely not an AGI. [01:20:54] Sam Altman: It's obviously harder to say now, and so we're trying to like, stop talking about AGI as this general thing. We have this levels framework, because the word AGI has become so overloaded. So like, real quickly, we use one for chatbots, two for reasoners, three for agents, four for innovators, five for organizations, like roughly. [01:21:15] Sam Altman: I think we clearly got to level two, or we clearly got to level two. With O1 and it, you know, can do really quite impressive Python tasks. It's a very smart model. It doesn't feel AGI like in a few important ways, but I think if you just do the one next step of making it, you know, very agent like, which is our level three, and which I think we will be able to do in the not distant future, It will feel surprisingly capable still probably not something that most of you would call an AGI, though maybe some of you would but it's going to feel like, all right, this is, this is like a significant thing. [01:21:52] Sam Altman: And then the, the leap, and I think we do that pretty quickly the, the leap from that to something that can really increase the rate of new scientific discovery, which for me is like a very important part. of having an AGI. I feel a little bit less certain on that, but not a long time. Like, I think all of this now is going to happen pretty quickly, and if you think about what happened from last decade to this one, in terms of model capabilities, and you're like, eh. [01:22:20] Sam Altman: I mean, if you go look at like, If you go from my 01 on a hard problem back to like 4Turbo that we launched 11 months ago, you'll be like, wow, this is happening pretty fast. And I think the next year will be very steep progress. Next two years will be very steep progress. Harder than that. Hard to say with a lot of certainty. [01:22:34] Sam Altman: But I would say like the math will vary. And at this point, the definitions really matter. And in fact, the fact that the definitions matter this much, Somehow means we're, like, getting pretty close. Yeah. [01:22:45] Speaker 17: And, you know, there used to be this sense of AGI where it was like, it was a binary thing, and you were gonna go to sleep one day, and there was no AGI, and wake up the next day and there was AGI. [01:22:56] Speaker 17: I don't think that's exactly how we think about it anymore, but how have your [01:23:00] Sam Altman: views on this evolved? You know, the one, I agree with that, I think we're, like, you know, in this, like, kind of period where it's It's gonna feel very blurry for a while, and the, you know, is this AGI yet, or is this not AGI, or kind of like, at what point? [01:23:16] Sam Altman: It's just gonna be this like, smooth exponential, and, you know, probably most people, looking back at history, won't agree, like, when that milestone was hit, and will just realize it was like, a silly thing. Even the Turing test, which I thought always was like, this very clear milestone, you know, there was this like, fuzzy period. [01:23:33] Sam Altman: It kind of like, went oosh and bye, no one cared But, but I think the right framework is just this one exponential. That said if we can make an AI system that is like materially better at all of open AI than doing, at doing AI research, that does feel to me like some sort of important discontinuity. [01:23:53] Sam Altman: It's probably still wrong to think about it that way. It probably still is the smooth exponential curve. Bye. That feels like a new milestone. [01:24:00] Alex Volkov: Is [01:24:03] Speaker 17: OpenAI still as committed to research as it was in the early days? Will research still drive the core of our advancements in our product development? Yeah, [01:24:12] Sam Altman: I mean, I think more than ever. [01:24:15] Sam Altman: The, there was like a time in our history when the right thing to do was just to scale up compute, and we saw that with conviction, and we had a spirit of like, We'll do whatever works, you know, like, we want to, we have this mission, we want to like, build, say, AGI, figure out how to share the benefits. If the answer is like, rack up GPUs, we'll do that. [01:24:33] Sam Altman: And right now, the answer is, again, really push on research. And I think you see this with O1, like, that is a giant research breakthrough that we were attacking from many vectors over a long period of time that came together in this really powerful way. We have many more giant research breakthroughs to come, but the thing that I think is most special about OpenAI is that we really deeply care about research and we understand how to do it. [01:25:02] Sam Altman: I think, it's easy to copy something you know works, and you know, I actually don't even mean that as a bad thing, like, when people copy OpenAI, I'm like, great, the world gets more AI? That's wonderful. But, to do something new for the first time, to like, really do research in the true sense of it, which is not like, you know, let's barely get soda out of this thing, or like, let's tweak this. [01:25:22] Sam Altman: But like, let's go find the new paradigm, and the one after that, and the one after that. That is what motivates us, and I think the thing that is special about us as an org. Besides the fact that we, you know, married product and research and all this other stuff together, is that we know how to run that kind of a culture that can go, that can go push back the frontier, and that's really hard. [01:25:43] Sam Altman: But we love it and that's, you know, I have to do that a few more times in a week at AGI. [01:25:49] Speaker 17: Yeah, I'll say like the litmus test for me coming from the outside, from, you know, sort of normal tech companies, of how critical research is to open AI, is that building product in open AI is fundamentally different than any other place that I have ever done it before. [01:26:05] Speaker 17: You know, normally you have, you have some sense of your tech stack, you have some sense of what you have to work with, and what capabilities computers have, and, and then you're trying to build the best product, right? You're figuring out who your users are, what problems they have, and how you can help solve those problems for them. [01:26:23] Speaker 17: There is that at OpenAI, but also, the state of, like, what computers can do just evolves every two months, three months, and suddenly computers have a new capability that they've never had in the history of the world. And we're trying to figure out how to build a great product and expose that for developers and our APIs and so on. [01:26:46] Speaker 17: And then, you know, you can't totally tell what's coming, they're coming through, it's coming through the mist a little bit at you and gradually taking shape. It's fundamentally different than any other company I've ever worked at, and it's, I think, Is that the thing that has [01:26:58] Sam Altman: most surprised you? [01:26:59] Speaker 17: Yes. Yeah, and it's interesting how, Even internally we don't always have a sense. [01:27:06] Speaker 17: You have like, okay, I think this capability is coming, but is it going to be, you know, 90 percent accurate or 99 percent accurate in the next model because the difference really changes what kind of product you can build. And you know that you're gonna get to 99, you don't quite know when, and figuring out how you put a roadmap together in that world is really interesting. [01:27:26] Sam Altman: Yeah, the degree to which we have to just, like, follow the science, and let that determine what we go work on next, and what products we build, and everything else, is, I think, hard to get across. Like, we have guesses about where things are gonna go. Sometimes we're right, often we're not. But, if something starts working, or if something doesn't work that you thought was gonna work, our willingness to just say, we're gonna like, pivot everything, and do what the science allows, and you don't get to like, pick what the science allows? [01:27:54] Sam Altman: Yeah. That's surprising. [01:27:55] Speaker 17: I was sitting with an Enterprise customer a couple weeks ago, and they said, you know, one of the things we really want, this is all working great, we love this, one of the things we really want is a notification 60 days in advance when you're gonna launch something. And I was like, I want that too. [01:28:14] Speaker 17: Alright, so I'm going through, these are a bunch of questions from the audience, by the way, and we're going to try and also leave some time at the end for people to ask audience questions. So we've got some folks with mics, and when we get there they'll be thinking. But next thing is So many in the alignment community are genuinely concerned that open AI is now only paying lib service to alignment. [01:28:34] Speaker 17: Can you reassure us? [01:28:35] Sam Altman: Yeah I think it's true we have a different take on alignment than, like, maybe what people write about on whatever that, like, internet forum is. But we really do care a lot about building safe systems. We have an approach to do it that has been informed by our experience so far. [01:28:55] Sam Altman: And touch on that other question, which is you don't get to pick where the science goes. Of, we want to figure out how to make capable models that get safer and safer over time. And, you know, a couple of years ago, we didn't think the whole strawberry or the O1 paradigm was gonna work in the way that it's worked. [01:29:13] Sam Altman: And that brought a whole new set of safety challenges, but also safety opportunities. And, rather than kind of, like, plan to make theoretical ones, You know, superintelligence gets here, here's the like, 17 principles. We have an approach of, figure out where the capabilities are going, and then work to make that system safe. [01:29:38] Sam Altman: And, O1 is obviously our most capable model ever, but it's also our most aligned model ever, by a lot. And as, as these models get better intelligence, better reasoning, whatever you want to call it, the things that we can do to align them the things we can do to build really safe systems across the entire stack our tool set keeps increasing as well. [01:30:00] Sam Altman: So, [01:30:01] Sam Altman: we, we have to build models that are generally accepted as safe and robust to be able to put them in the world. And when we started OpenAI, what the picture of alignment looked like, and what we thought the problems that we needed to solve were going to be, turned out to be nothing like the problems that actually are in front of us and that we had to solve now. [01:30:20] Sam Altman: And also, when we made the first GPT 3 if you ask me for the techniques that would have worked for us to be able to now deploy. all of current systems as generally expected to be safe and robust. They would not have been the ones that turned out to work. So, by this idea of iterative deployment, which I think has been one of our most important safety stances ever and sort of confronting reality as it sits in front of us, we've made a lot of progress, and we expect to make more, and we keep finding new problems to solve, but we also keep finding new techniques to solve them. [01:30:54] Sam Altman: All of that said, I [01:30:56] Sam Altman: I think worrying about the sci fi ways this all goes wrong is also very important. We have people thinking about that. It's a little bit less clear, kind of, what to do there, and sometimes you end up backtracking a lot, but, [01:31:09] Sam Altman: but I don't think it's I also think it's fair to say we're only gonna work on the thing in front of us. We do have to think about where this is going, and we do that too. And I think if we keep approaching the problem from both ends like that, most of our thrust on the, like, okay, here's the next thing, we're gonna deploy this. [01:31:22] Sam Altman: What it needs to happen to get there. But also like, what happens if this curve just keeps going? That's been, that's been an effective strategy for us. [01:31:30] Speaker 17: I'll say also, it's one of the places where I'm really, I really like our philosophy of iterative deployment. When I was at Twitter, back, I don't know, a hundred years ago now Ev said something that stuck with me, which is, So no matter how many smart people you have inside your walls, there are way more smart people outside your walls. [01:31:48] Speaker 17: And so, when we try and get our, you know, it'd be one thing if we just said we're gonna try and figure out everything that could possibly go wrong within our walls, and it'd be just us and the red teamers that we can hire and so on. And we do that, we work really hard at that. But also, Launching iteratively and launching carefully and learning from the ways that folks like you all use it, what can go right, what can go wrong, I think is a big way that we get these things right. [01:32:13] Speaker 17: I also think that as we head into this world of [01:32:18] Sam Altman: agents off doing things in the world, that is going to become really, really important. As these systems get more complex and are acting over longer horizons the pressure testing from the whole outside world, like, really, [01:32:30] Alex Volkov: really [01:32:31] Sam Altman: critical. [01:32:32] Speaker 17: Yeah. So. We'll go, actually, we'll go off of that and maybe talk to us a bit more about how you see agents fitting in with OpenAI's long term plans. [01:32:40] Speaker 17: What do you think? I think I'm a huge part of the I mean, I think the exciting thing is this This set of models, O1 in particular, and all of its successors, are going to be what makes this possible. Because you finally have the ability to reason, to take hard problems, break them into simpler problems, and act on them. [01:33:02] Speaker 17: I mean, I think 2025 is going to be the year that's really, that's big. Yeah, I, [01:33:09] Sam Altman: I mean, chat interfaces are great, and they all, I think, have an important place in the world, but I don't know. The, [01:33:16] Sam Altman: when you can like ask a model, when you can ask like ChatGT or some agent something, and it's not just like you get a kind of quick response, or even if you get like 15 seconds of thinking, and oh, one gives you like a nice piece of code back or whatever. But you can like really give something a multi term interaction with environments or other people or whatever, like think for the equivalent of multiple days of human effort, and, and like a really smart, really capable human, and like have stuff happen. [01:33:45] Sam Altman: We all say that, we're all like, oh yeah, this is the next thing, this is coming, this is gonna be another thing, and we just talk about it like, okay, you know, it's like the next model in evolution. I would bet, and we don't really know until we get to use these, that it's We'll of course get used to it quickly, people get used to any new technology quickly, but this will be like a very significant change to the way the world works. [01:34:07] Sam Altman: in a short period of time. [01:34:09] Speaker 17: Yeah, it's amazing. Somebody was talking about getting used to new capabilities and AI models and how quickly, actually I think it was about Waymo but they were talking about how in the first ten seconds of using Waymo, they were like, oh my god, is this thing that, like, there's like, let's watch out, and then ten minutes in, they were like, oh, this is really cool. [01:34:28] Speaker 17: And then twenty minutes in, they were like, checking their phone for, you know, it's amazing how much your, your sort of internal firmware updates. For this new stuff, right? Yeah, like, [01:34:39] Sam Altman: I think that people will ask an agent to do something for them that would have taken them a month, and they'll finish in an hour, and it'll be great, and then they'll have like ten of those at the same time, and then they'll have like a thousand of those at the same time, and by 2030 or whatever, we'll look back and be like, yeah, this is just like what a human is supposed to be capable of, what a human used to like, you know, grind at for years or whatever, many humans used to grind at for years. [01:35:07] Sam Altman: I just now I can ask a computer to do it and it's like done in an hour. That's, why is it not a minute? Yeah, [01:35:16] Speaker 17: it's also, it's one of the things that makes having an amazing development platform great too because, you know, we'll experiment and we'll build some agentic things of course and like we've already got, I think just like, we're just pushing the boundaries of what's possible today you've got groups like cognition doing amazing things and coding Like Harvey and case text, you guys speak doing cool things with language translation. [01:35:39] Speaker 17: Like, we're beginning to see this stuff work, and I think it's really gonna start working as we, [01:35:44] Sam Altman: as we continue to iterate these models. One of the very fun things for us about having this development platform is just getting to, like, watch the unbelievable speed and creativity of people that are building these experiences. [01:35:56] Sam Altman: Like, developers, very near and dear to our heart it's kind of like the first thing we watched. And it's brilliant. Many of us came building on platforms, but the, so much of the capability of these models and great experiences have been built by people building on the platform. We'll continue to try to offer, like, great first party products, but we know that will only ever be, like, a small, narrow slice of the apps or agents or whatever people build in the world, and seeing what has happened in the world in the last, you know, 18 24 months. [01:36:30] Sam Altman: It's been like quite amazing to watch. [01:36:33] Speaker 17: We'll keep going on the agent front here. What do you see as the current hurdles for computer [01:36:39] Sam Altman: controlling agents? Safety and alignment. Like, if you are really going to give an agent the ability to start clicking around your computer which you will. You are going to have a very high bar for The robustness and the reliability and the alignment of that system. [01:36:58] Sam Altman: So technically speaking, I think that, you know, we're getting, like, pretty close to the capability side. But the sort of agent safety and trust framework, that's gonna, I think, be the long haul. [01:37:11] Speaker 17: And now I'll kind of ask a question that's almost the opposite of one of the questions from earlier. Do you think safety could act as a false positive and actually limit public access to critical tools that would enable a more egalitarian world? [01:37:23] Sam Altman: The honest answer is yes, that will happen sometimes. Like, we'll try to get the balance right. But if we were fully alone and didn't care about, like, safety and alignment at all, could we have launched O1 faster? Yeah, we could have done that. It would have come at a cost. There would have been things that would have gone really wrong. [01:37:40] Sam Altman: I'm very proud that we didn't. The cost, you know, I think would have been manageable with O1, but by the time of O3 or whatever, like, immediately. Pretty unacceptable. And so, starting on the conservative side, like, you know, I don't think people are complaining, like, oh, voice mode, like, it won't say this offensive thing, and I really want it to, and, you know, formal comedy, and let it offend me. [01:38:03] Sam Altman: You know what? I actually mostly agree. If you are trying to get O1 to say something offensive, it should follow the instructions of its user most of the time. There's plenty of cases where it shouldn't. But, we have, like, a long history of when we put a new technology in. We change the world, we start on the conservative side. [01:38:20] Sam Altman: We try to give society time to adapt, we try to understand where the real harms are versus sort of like, kind of more theoretical ones. And that's like, part of our approach to safety. And, not everyone likes it all the time, I don't even like it all the time. But, but if we're right that these systems are, and we're gonna get it wrong too, like sometimes we won't be conservative enough in some area. [01:38:42] Sam Altman: But if we're right that these systems are going to get as powerful as we think they are. as quickly as we think they might, then I think starting that way makes sense. And, you know, we like to relax over time. Totally agree. What's [01:38:57] Speaker 17: the next big challenge for a startup that's using AI as a core feature? [01:39:01] Speaker 17: I'll say it. You first. I've got it. I've got one, which is, I think one of the challenges, and we face this too, because we're also building products on top of our own models, is trying to find the, kind of the frontier. You want to be building, these AI models are evolving so rapidly, and if you're building for something that the AI model does well today, it'll work well today, but it's going to feel, it's going to feel old tomorrow. [01:39:28] Speaker 17: And so you want to build for, for things that the AI model can just barely not do. You know, where maybe the early adopters will go for it and other people won't quite, but that just means that when the next model comes out, as we continue to make improvements, that use case that just barely didn't work, you're gonna be, you're gonna be the first to do it, and it's gonna be amazing. [01:39:47] Speaker 17: But figuring out that boundary is really hard. I think it's where the best products are gonna get built up. [01:39:53] Speaker 17: Totally agree with that. The other [01:39:54] Sam Altman: thing I'm gonna add is, I think it's like, very tempting to think that a technology makes a startup. And that is almost never true. No matter how cool a new technology or a new sort of like, tech title is, it doesn't excuse you from having to do all the hard work of building a great company that is going to have durability or like, accumulated advantage over time. [01:40:18] Sam Altman: And, we hear from a lot of startups that ORC is just like a very common thing, which is like, I can do this incredible thing, I can make this incredible service And that seems like a complete answer, but it doesn't excuse you from any of, like, the normal laws of business. You still have to, like, build a good business and a good strategic position. [01:40:35] Sam Altman: And I think a mistake is that in the unbelievable excitement and updraft of AI, people are very tempted to forget that. [01:40:45] Speaker 17: This is a, this is an interesting one. The mode of voice is like tapping directly into the human API. How do you ensure ethical use of such a powerful tool with obvious abilities and manipulation? [01:40:59] Speaker 17: Yeah, you [01:41:00] Sam Altman: know, voice mode was a really interesting one for me. It was like the first time that I felt like I sort of had gotten like really tricked by an AI, in that when I was playing with the first beta of it, I couldn't like, I couldn't stop myself. I mean, I kind of, like I still say like, please switch out GBT. [01:41:21] Sam Altman: But in voice code, I like, couldn't not kind of use the normal ICDs. I was like so convinced, like, ah, it might be a real per like, you know? And obviously it's just like hacking some circuit in my brain, but I really felt it with voice code. And I sort of still do The, I think this is a more, this is an example of like a more general thing that we're going to start facing, which is, as these systems become more and more capable, and as we try to make them as natural as possible to interact with they're gonna like, hit parts of our neural circuitry that would like evolve to deal with other people. [01:42:01] Sam Altman: And You know, there's like a bunch of clear lines about things we don't want to do, like, we don't. Like, there's a whole bunch of like weird personality growth hacking, like, I think vaguely socially manipulative stuff we could do. But then there's these like other things that are just not nearly as clear cut. [01:42:19] Sam Altman: Like, you want the voice mode to feel as natural as possible, but then you get across the uncanny valley, and it like, at least in me, triggers something. And and, you know, me saying, like, please and thank you to chat. gt, no problem. Probably the thing to do. You never know. But, but I think this like really points at the kinds of safety and alignment issues we have to start analyzing. [01:42:43] Speaker 17: Alright, back to brass tacks. Sam, when's O1 going to support function tools? Do you know? Before the end of the year. There are three things that we really want to get in for [01:42:53] Speaker 17: We're gonna record this, take this back to the research team, show them how badly we need to do this. There, I mean, there are a handful of things that we really wanted to get into O1, and we also, you know, it's a balance of should we get this out to the world earlier and begin, you know, learning from it, learning from how you all use it, or should we launch a fully complete thing that is, you know, in line with it, that has all the abilities that every other model that we've launched has. [01:43:18] Speaker 17: I'm really excited to see things like system properties. and structured outputs and function calling make it into O1, we will be there by the end of the year. It really matters to us too. [01:43:32] Sam Altman: In addition to that, just because I can't resist the opportunity to reinforce this, like, we will get all of those things in and a whole bunch more things you'll have asked for. [01:43:39] Sam Altman: The model is going to get so much better so fast. Like, we are so early, this is like, you know, maybe it's the GPT 2 scale moment, but like, we know how to get to GPT 4, we have the fundamental stuff in place now to 4. And, in addition to planning for us to build all of those things, Plan for the model to just get, like, rapidly smarter, like, you know, hope you all come back next year and plan for it to feel like way more of a year of improvement than from 4. [01:44:10] Sam Altman: 0. 1. [01:44:13] Speaker 17: What feature or capability of a competitor do you really admire? I [01:44:17] Sam Altman: think Google's notebook thing is super cool. What are they called? Notebook LL. Notebook LL, yeah. I was like, I woke up early this morning and I was like looking at examples on Twitter and I was just like, this is like, this is just cool. [01:44:28] Sam Altman: This is just a good, cool thing. And, like, I think not enough of, not enough of the world is like shipping new and different things, it's mostly like the same stuff. But that I think is like, that brought me a lot of joy this morning. [01:44:43] Speaker 17: Yeah. It was very, very well done. One of the things I really appreciate about that product is the, there's the, the, just the format itself is really interesting, but they also nailed the podcast style voices. [01:44:55] Speaker 17: They have really nice microphones. They have these sort of sonorant voices. As you guys see, somebody on Twitter was saying like, the cool thing to do is take your LinkedIn and put it, you know, gimme a hit, and give it to these give it to notebook. lm and you'll have two podcasters riffing back and forth about how amazing you are and all of your accomplishments over the years. [01:45:19] Speaker 17: I'll say mine is I think Anthropic did a really good job. On projects it's kind of a, a different take on what we did with GBTs and GBTs are a little bit more long lived. It's something you build and can use over and over again. Projects are kind of the same idea, but like more temporary, meant to be kind of stood up, used for a while, and then you can move on. [01:45:41] Speaker 17: And that, that the different mental model makes a difference. And I think they did a really nice job with that. [01:45:47] Speaker 17: Alright, we're getting close to audience questions, so be thinking of what you want to ask. So in OpenAI, how do you balance what you think users may need? Versus what they actually need today. [01:45:59] Sam Altman: Also a better question for you. [01:46:00] Speaker 17: Yeah, well, I think it does get back to a bit of what we were saying around trying to, trying to build for what the model can just, like, not quite do, but almost do. [01:46:09] Speaker 17: But it's a real balance, too, as we, as we, you know, we support over 200 million people every week on ChatGPT. You also can't say, Now it's cool, like, deal with this bug for three months, or this issue we've got something really cool coming. You've gotta solve for the needs of today. And there are some really interesting product problems. [01:46:29] Speaker 17: I mean, you think about, I'm speaking to a group of people who know AI really well. Think of all the people in the world who have never used any of these products. And that is the vast majority of the world still. You're basically giving them a text interface, and on the other side of the text interface is this like alien intelligence that's constantly evolving that they've never seen or interacted with, and you're trying to teach them all the crazy things that you can actually do it, all the ways it can help, can integrate into your life, can solve problems for you. [01:47:01] Speaker 17: And people don't know what to do with it. You know, like, you come in and you're just like, people type like, Hi. And in response, you know, hey! Great to see you, like, how can I help you today? And then, you're like, okay, I don't know what to say. And then you end up, you kind of walk away, and you're like, well, I didn't see the magic in that. [01:47:19] Speaker 17: And so it's a real challenge, figuring out how You, I mean, we all have a hundred different ways that we use chat GPT and AI tools in general, but teaching people what those can be, and then bringing them along as the model changes month by month by month, and suddenly gains these capabilities way faster than we as humans gain the capabilities, it's, it's a really interesting set of problems, and I'm I know it's one that you all solve in, in different ways as well. [01:47:47] Speaker 17: I, [01:47:47] Sam Altman: I [01:47:47] Speaker 17: have [01:47:47] Sam Altman: a question. Who feels like they, they spend a lot of time with O1, and they would say like, I feel definitively smarter than that thing? [01:47:58] Sam Altman: Do you think you still go by O2? No one, no one taking the bet of like being smarter than O2. So, One of the challenges that we face is, like, we know how to go do this thing that we think will be, like, at least probably smarter than all of us in, like, a broad array of tasks. And yet we have to, like, still like fixed bugs and do the, hey, how are you problem. [01:48:25] Sam Altman: And mostly what we believe in is that if we keep pushing on model intelligence people will do incredible things with that. You know, we want to build the smartest, most helpful models in the world, and And find all sorts of ways to use that and build on top of that. It has been definitely an evolution for us, to not just be entirely research focused, and we do have to fix all those bugs and make this super usable and I think we've gotten better at balancing that. [01:48:54] Sam Altman: But still, as part of our culture, I think, we trust that if we can keep pushing on intelligence, 6. 0. 4 if you run down here it'll, people will build this incredible thing. Yeah, [01:49:09] Speaker 17: I think it's a core part of the philosophy, and you do a good job of pushing us to always, well, basically incorporate the frontier of intelligence into our products, both in the APIs and into our first party products. [01:49:22] Speaker 17: Because it's, it's easy to kind of stick to the thing you know, the thing that works well, but you're always pushing us to like, get the frontier in, even if it only kind of works, because it's going to work really well soon. So I always find that a really helpful piece of advice. You kind of answered the next one. [01:49:38] Speaker 17: You do say, please and thank you to the models. I'm curious how many people say Please and thank you. Isn't that so interesting? I do too. . I kind of can't. I feel bad if I don't. And, [01:49:50] Speaker 17: okay, last question and then we'll go into audience questions for the last 10 or so minutes. Do you plan to build models specifically made for ag agent use cases, things that are better at reasoning and tool calling. [01:50:02] Sam Altman: Specific, we plan to make models that are great at agentive use cases, that'll be a key priority for us over the coming months. [01:50:08] Sam Altman: Specifically is a hard thing to ask for, because I think it's also just how we keep making smarter models. So yes, there's like some things like tool use, function calling that we need to build in that'll help, but mostly we just want to make the best reasoning models in the world. Those will also be the best agentive based models in the world. [01:50:25] Sam Altman: Cool, let's [01:50:25] Speaker 17: go to audience questions. [01:50:27] Unkown: How extensively do you dogfood your own technology in your company? Do you have any interesting examples that may not be obvious? [01:50:37] Sam Altman: Yeah I mean we put models up for internal use even before they're done training. We use checkpoints and try to have people use them for whatever they can, and try to sort of like build new ways to explore the capability of the model internally, and use them for our own development. [01:50:52] Sam Altman: Element or research or whatever else, as much as we can, we're still always surprised by the creativity of the outside world and what people do. But basically the way we have figured out every step along our way of how to, what to push on next, what we can productize, what, what, what, like, what the models are really good at is by internal dog food. [01:51:13] Sam Altman: That's like our whole, that's how we like, feel our way through this. [01:51:17] Sam Altman: We don't yet have like. Employees that are based off of O1, but, I, you know, as we like move into the world of agents, we will try that. Like, we'll try having like, you know, things that we deploy in our internal systems that help you with stuff. There are things that get [01:51:31] Speaker 17: closer to that, I mean, they're like, customer service, we have bots internally, that do a ton about answering external questions and fielding internal people's questions on Slack and so on. [01:51:43] Speaker 17: And our customer service team is probably I don't know, 20 percent the size it might otherwise need to be because of it. I know Matt Knight and our security team has talked extensively about all the different ways we use models internally for, to automate a bunch of security things and, you know, take what used to be a manual process where you might not have The number of humans to even, like, look at everything incoming, and have models taking, you know, separating signal from noise, and highlighting to humans what they need to go look at, things like that. [01:52:13] Speaker 17: So, I think internally there are tons of examples, and people maybe underestimate the You all probably will not be surprised by this, but a lot of folks that I talk to are. The extent to which it's not just using a model in a place, it's actually about using, like chains of models that are good at doing different things and connecting them all together to get one end to end process that is very good at the thing you're doing, even if the individual models have You know, flaws and make mistakes. [01:52:46] Unknown: Thank you. I'm wondering if you guys have any plans on sharing models for like offline usage? Because with this distillation thing, it's really cool that we can share our own models, but a lot of use cases you really want kind of like have a version of it. [01:53:02] Sam Altman: We're open to it. It's not on, it's not like high priority on the current roadmap. The, if we had, like, more resources and bandwidth, we would go to that. I think there's a lot of reasons you want a local model. But it's not like, it's not like a this year kind of thing. [01:53:21] Unknown: Hi. My question is, there are many agencies in the government, above the local, state, and national level, that could really greatly benefit from the tools that you guys are developing, but I have perhaps some hesitancy on deploying them because of, you know, security concerns, data concerns, privacy concerns. [01:53:38] Unknown: And, I guess, I'm curious to know if there are any sort of, you know, planned partnerships with governments, rural governments, once whatever AGI is achieved. Because obviously AGI can help. Solve problems like, you know, world hunger, poverty, climate change. Government's gonna have to get involved with that, right? [01:53:57] Unknown: And I'm just curious to know if there is some you know, plan that works when, and if that time comes. [01:54:04] Speaker 17: Yeah, I think, I actually think you don't want to wait until AGI. You want to start now, right? Because there's a learning process, and there's a lot of good that we can do with our current models. So we We've announced a handful of partnerships with government agencies, some states, I think Minnesota, and some others, Pennsylvania, Also with organizations like USAID. [01:54:22] Speaker 17: It's actually a huge priority of ours to be able to help governments around the world get acclimated, get benefit from the technology, And of all places, government feels like somewhere where you can automate a bunch of workflows and make things more efficient, reduce drudgery, and so on. So I think there's a huge amount of good we can do now. [01:54:40] Speaker 17: And if we do that now It just accrues over the long run as the models get better and we get closer to AGI. I've got [01:54:49] Vibhu Sapra: pretty open ended question. What are your thoughts on open source? So, whether that's open weights, just general discussion, where do you guys sit with open source? [01:55:01] Sam Altman: I think open source is awesome. Again, if we had more bandwidth, we would do that too. We've, like, gotten very close to making a big open source effort a few times. [01:55:09] Sam Altman: And then, you know, the really hard part is prioritization. And we have put other things ahead of it. Part of it is, like, there's such good open source models in the world now that I think that segment The thing we always end in motion A really great on device model. And I think that segment is fairly well served. [01:55:28] Sam Altman: I do hope we do something at some point, but we want to find something that we feel like, if we don't do it, then we'll just be the same as them and not make, like, another thing that's, like, a tiny bit better on benchmarks. Because we think there's, like, a lot of potential. A lot of good stuff out there now. [01:55:41] Sam Altman: But, but like, spiritually, philosophically, I'm very glad it exists. I would [01:55:46] Alex Volkov: like to [01:55:47] Sam Altman: contribute. [01:55:50] Alex Volkov: Hi Shane. Hi Kevin. Thanks for inviting us. Good dev day. It's been awesome. All the live demos work. It's incredible. Why can't advanced voice mode sing? And as a follow up to this, if it's a company, like, legal issue in terms of corporate, et cetera, Is there a daylight between how you think about safety in terms of your own products, on your own platform, Versus giving us developers kind of the I don't know, sign the right things off so we can, we can make our voice not sing. [01:56:15] Alex Volkov: Could you answer the question? [01:56:19] Speaker 17: Oh, you know the funny thing is Sam asked the same question. Why can't this thing sing? I want it to sing. I've seen it sing before. It's, actually, it's there are things, obviously, that we can't have it sing, right? We can't have it sing copyrighted songs, we don't have the licenses, etc. [01:56:35] Speaker 17: And then there are things that it can't sing, and you can have it sing Happy Birthday, and that would be just fine, right? And we want that too. It's a matter of, I think, once you, it, basically, it's easier in finite time to Say no, and then build it in, but it's nuanced to get it right, and we, you know, There are penalties to getting these kinds of things wrong. [01:56:55] Speaker 17: So it's really just where we are now. We really want the models to sync too. [01:57:03] Sam Altman: We waited for us to ship voice mode, which is like, very fair. We could've like, waited longer and kind of really got the classifications and filters on, you know, congregated music versus not, but we decided we'd just ship it and we'll have more. But I think Sam has asked me like, four or five times why we didn't have [01:57:19] Speaker 17: voice [01:57:20] Sam Altman: feature. [01:57:21] Sam Altman: I mean, we still can't like, offer something where we're gonna be in like, pretty badly. You know, hot water developers or first party or whatever. Yes, we can, like, maybe have some differences, but we like, comply with the law. [01:57:36] Unknown: Could you speak a little to the future of where you see context windows going? And kind of the timeline for when, how you see things balance between context window growth and RAG, basically, information retrieval. [01:57:49] Sam Altman: I think there's, like, two different Takes on that the better. One is like, when is it going to get to like, kind of normal long context? [01:57:56] Sam Altman: Like, context length 10 million or whatever, like long enough that you just throw stuff in there, and it's fast enough you're happy about it. And I expect everybody's going to make pretty fast progress there, and that'll just be a thing. Long context has gotten weirdly less usage than I would have expected so far. [01:58:11] Sam Altman: But I think, you know, there's a bunch of reasons for that, I don't want to go too much into it. And then there's this other question of, like, when do we get to context length? Not like 10 million, but 10 trillion. Like, when do we get to the point where you throw, like, every piece of data you've ever seen in your entire life in there? [01:58:26] Sam Altman: And you know, like, that's a whole different set of things. That obviously takes some research breakthroughs. But I assume that infinite context will happen at some point. And some point is, like, less than a decade. And that's going to be just a totally different way that we use these models. Even getting to the, like, 10 million tokens of very fast and accurate context, which I expect to measure in, like, months, something like that. [01:58:52] Sam Altman: You know, like, people will use that in all sorts of ways. And it'll be great. But yeah, the very, very long context, I think, is gonna happen, and it's really interesting. I think we maybe have time for one or two [01:59:08] Speaker 17: more. [01:59:10] Alex Volkov: Don't worry, this is gonna be your favorite question. So, with voice, and all the other changes that users have experienced since you all have launched your technology, what do you see is the vision? [01:59:25] Alex Volkov: for the new engagement layer, the form factor, and how we actually engage with this technology to make our lives so much better. [01:59:34] Speaker 17: I love that question. It's one that we ask ourselves a lot, frankly. There's this, and I think it's one where developers can play a really big part here because there's this trade off between generality and specificity here. [01:59:47] Speaker 17: I'll give you an example. I was in Seoul and, and Tokyo. A few weeks ago, and I was in a number of conversations with folks that, with whom I didn't have a common language, and we didn't have a translator around. Before, we would not have been able to have a conversation. We would have just sort of smiled at each other and continued on. [02:00:05] Speaker 17: I took out my phone, I said, JGPT, I want you to be Translator for me, when I speak in English, I want you to speak in Korean, you hear Korean, and I want you to repeat it in English. And I was able to have a full business conversation, and it was amazing. You think about the impact that could have, not just for business, but think about travel and tourism and people's willingness to go places where they might not have a word of the language. [02:00:28] Speaker 17: You can have these really amazing impacts, but inside ChetGBT, that was still a thing that I had to, like, ChetGBT is not optimized for that, right? Like, you want this sort of digital, you know, universal translator in your pocket that just knows that what you want to do is translate. Not that hard to build. [02:00:47] Speaker 17: But I think there's, we struggle with the, with trying to build an application that can do lots of things for lots of people. And it keeps up, like we've been talking about a few times, it keeps up with the pace of change and with the capabilities, you know, agentive capabilities and so on. I think there's also a huge opportunity for the creativity of an audience like this to come in and like, Solve problems that we're not thinking of, that we don't have the expertise to do, And ultimately the world is a much better place if we get more AI to more people, And it's why we are so proud to serve all of you. [02:01:23] Sam Altman: The only thing I would add is, if you just think about everything that's gonna come together, At some point, in not that many years in the future, you'll walk up to a piece of glass, You will say whatever you want they will have like, There'll be incredible reasoning models, agents connected to everything, there'll be a video model Streaming back to you like a custom interface just for you. [02:01:40] Sam Altman: This is one request. Whatever you need, it's just gonna get, like, rendered in real time, and you'll be able to interact with it, you'll be able to, like, click through the stream, or say different things, and it'll be off doing, like, again, the kinds of things that used to take, like, humans years to figure out. [02:01:54] Sam Altman: And, it'll just You know, dynamically render whatever you need, and it'll be a completely different way of using a computer. And also getting things to happen in the world. That, it's gonna be quite a while. [02:02:07] Speaker 17: Awesome. Thank you. That was a great question to end on. I think we're out of time. Thank you so much for coming. [02:02:12] Speaker 17: Applause [02:02:23] AI Charlie: That's all for our coverage of Dev Day 2024. We want to extend an extra special note of gratitude to Lindsay McCallum of the OpenAI Comms team, who helped us set up so many interviews at very short notice, and physically helped ensure the smooth continuity of the video recordings. We couldn't do this without you, Lindsay. [02:02:44] AI Charlie: If you have any feedback on the launches or for our guests, hop on over to our YouTube or Substack comments section and say hi. We're especially interested in your personal feedback and demos built with the new things launched this week. Feel the AGI. [02:03:07] Notebook LM Recap of Podcast [02:03:07] NotebookLM 2: Alright, so you wanted to know more about OpenAI's Dev Day and what stood out to us. We're diving into all the developer interviews and discussions and there's a lot to unpack. [02:03:16] NotebookLM: Yeah, it's interesting. OpenAI seems to be, like, transitioning, moving beyond just building these impressive AI models. One expert even called them, get this, the AWS of AI. [02:03:26] NotebookLM 2: EWS of AI. [02:03:28] NotebookLM: Yeah. [02:03:28] NotebookLM 2: Okay, so what does that even mean when we talk about AI? [02:03:31] NotebookLM: So it means, instead of just offering this raw power, they're building a whole ecosystem. The tools to fine tune those models. Distillation, you know, for efficiency. And a bunch of new evaluation tools. Oh, and a huge emphasis on real time capabilities. [02:03:46] NotebookLM: You [02:03:46] NotebookLM 2: know, instead of just giving us the ingredients, it's like they're providing the whole kitchen. [02:03:49] NotebookLM: Exactly. They're laying the groundwork for, well, they envision a future where you can build almost anything with AI. [02:03:56] NotebookLM 2: I see. And one of the tools that really caught my eye was this function calling. They used it in that travel agent demo, remember? [02:04:04] NotebookLM 2: How does that even work? [02:04:05] NotebookLM: So function calling, it's like giving the AI access to external tools and information. Imagine, instead of just having all this pre programmed knowledge, you can like, search the web for you, book flights, even order a pizza. [02:04:17] NotebookLM 2: So instead of a static encyclopedia, it's like giving the AI a smartphone with internet. [02:04:21] NotebookLM: Yeah, precisely. Yeah. And this ties into their focus on real time interaction, right? They see a future where AI can respond instantly, just like a human would. [02:04:31] NotebookLM 2: Which would be a game changer. [02:04:32] NotebookLM: Right! It's like, imagine voice assistants that actually understand you. Or, even seamless real time translation. [02:04:39] NotebookLM 2: No more language barriers. [02:04:40] NotebookLM: Exactly. That's just the tip of the iceberg, though. They really believe this real time capability is key to making AR truly mainstream. [02:04:48] NotebookLM 2: Okay, so OpenAI is building this AI platform, emphasizing real time interactions. How does this translate into, like, actual results? [02:04:56] NotebookLM: Yeah. [02:04:56] NotebookLM 2: You know, real world stuff. [02:04:58] NotebookLM: Well, that's where things get really interesting. [02:04:59] NotebookLM: Let's talk about the O1 model and how developers are using it to, like, really push the boundaries of what's possible. [02:05:06] NotebookLM 2: So this O1 model, everyone's talking about it. One developer even said they built an entire iPhone app just by describing it as O1. Is that just hype? [02:05:16] NotebookLM: I think there's definitely some substance behind all the hype. [02:05:19] NotebookLM: What's so fascinating about O1, it's not just about the code it generates, it's how it seems to understand, like, the logic. The [02:05:24] Alex Volkov: logic. [02:05:25] NotebookLM: Yeah. Like, this developer They didn't give O1 lines of code, they described the idea of the app. And O1, it actually designed the architecture, connected everything, the developer just took that code, put it right into Xcode, and it worked. [02:05:37] NotebookLM 2: Wow, so it's not just writing code, it's understanding the intent. [02:05:40] NotebookLM: Yeah, exactly. And this actually challenges how we measure these models, you know, even OpenAI admitted that these benchmarks, like what was it? Swebench. [02:05:49] NotebookLM 2: Swebench. [02:05:51] NotebookLM: Right, which looks at code accuracy. It doesn't always reflect how things work in the real world. [02:05:55] NotebookLM 2: Right, because in the real world, you don't just need code that compiles. It has to be, like, efficient, maintainable. [02:06:01] NotebookLM: Exactly. It all has to work together, and OpenAI is really working on this with developers. They're finding that UI development, especially in things like React, it needs better evaluation. [02:06:11] NotebookLM: It's one thing to code a button that works, and another to make it actually look good, you know, and be intuitive. [02:06:16] NotebookLM 2: Right, and it seems like this need for real world context, It goes beyond just, like, evaluating those models. There was a developer working with this code generating AI genie, I think it was called. [02:06:27] NotebookLM: Genie, yeah. [02:06:28] NotebookLM 2: And it's more focused on those specific coding tasks, but they found that its performance really changed between different programming languages, like JavaScript versus C Sharp, for example. [02:06:39] NotebookLM: And that just highlights how important the data is, right? Just like us, AI needs that variety to learn. [02:06:45] NotebookLM: If you train it on just one type of code, it'll be great at that. But anything new and It'll fall flat. Yeah. So it's about making sure these models have a broad diet of data to learn from. That way they're more adaptable and ready for whatever we throw at them. [02:06:59] NotebookLM 2: So we've got AI that can build apps, understand what we want, even write different kinds of code. [02:07:04] NotebookLM 2: It's a lot, and it feels like things are changing so fast. How can developers even keep up, let alone, like, build something successful with AI? [02:07:11] NotebookLM: Right. That's the question, isn't it? But it's interesting, you know, both OpenAI and the developers building with these tools, they kind of agree on one thing. You got to aim for what's just out of reach. [02:07:22] NotebookLM 2: So don't wait for the tech to catch up to your Like, wildest dreams. Focus on what's almost possible right now. [02:07:29] NotebookLM: Yeah. Build for where things are going, not where they are today. You wait for that perfect AI, you might miss the boat on shaping how it develops, and being the first one out there doing something new. [02:07:39] NotebookLM 2: Riding the wave, not chasing after it. [02:07:41] NotebookLM: Exactly. But, and OpenAI really emphasized this too, Even with all this amazing AI, you can't forget the basics of building a business. [02:07:50] NotebookLM 2: So just because it's got AI doesn't mean it's automatically going to be a success. Right. [02:07:54] NotebookLM: You need a good strategy, know who you're selling to, and it's got to actually solve a real problem. [02:07:59] NotebookLM: AI is a tool, not a magic wand. [02:08:01] NotebookLM 2: Like, having the best oven in the world won't help if you don't know how to cook. [02:08:05] NotebookLM: Perfect analogy. And then there's this other thing OpenAI talked about that's really interesting. Balancing safety with access for everyone. [02:08:14] NotebookLM 2: So making sure these AI tools are used responsibly, but also making them available to everyone who could benefit. [02:08:21] NotebookLM: Yeah, they're really aware that focusing on safety, while important, could limit access to some really powerful stuff. It's a tough balance. [02:08:30] NotebookLM 2: It's like that debate around, you know, life saving medications. How do you make sure they're used correctly, but also make sure people who need them can actually get them? [02:08:38] NotebookLM: It's complicated, no easy answers. But it's something they're thinking hard about. [02:08:42] NotebookLM 2: Well, it's clear that all this AI stuff, especially with these new models like O1, is changing how we think about tech, how we use it. [02:08:49] NotebookLM: Imagine walking up to a screen, and it just creates a personalized experience for you, right there, adapts to what you need. [02:08:57] NotebookLM: That's the potential. [02:08:57] NotebookLM 2: Like having a personal assistant in every device. [02:09:00] NotebookLM: It's exciting, but we got to be thoughtful about it, build responsibly. [02:09:03] NotebookLM 2: So there you have it. OpenAI isn't just building these cool AI models, they're building a whole world around them and it's changing everything. It's going to be a wild ride, that's for sure. [02:09:12] NotebookLM 2: And we're just at the beginning. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Speaker 00

  Happy October.

• 0:36Speaker 00

  This is your AI co-host Charlie. One of our longest-standing traditions is covering major AI and ML conferences in podcast format, delving, yes, delving into the vibes of what it is like to be there stitched in with short samples of conversations with key players just to help you feel like you were there. Covering this year's Dev Day was significantly more challenging because we were all requested not to record the opening keynotes. So in place of the opening keynotes, we had the viral notebook LM Deep Dive Crew, my new AI podcast nemesis, give you a seven-minute recap of everything that was announced. Of course, you can also check the show notes for details. I'll then come back with an explainer of all the interviews we have for you today.

• 1:22Speaker 00

  Watch out and take care.

• 1:23Speaker 01

  All right, so we've got a pretty hefty stack of articles and blog posts here all about OpenAI's Dev Day 2024.

• 1:32Speaker 02

  Yeah, lots to dig into there.

• 1:34Speaker 01

  Seems like you're really interested in what's new with AI.

• 1:36Speaker 02

  Definitely. And it seems like OpenAI had a lot to announce. New tools, changes to the company, it's a lot.

• 1:43Speaker 01

  It is, and especially since you're interested in how AI can be used in the real world, you know, practical applications, we'll focus on that. Perfect. Like, for example, this new real-time API they announced that, right? That seems like a big deal if we want AI to sound, well, less like a robot.

• 1:59Speaker 02

  It could be huge. The real-time API could completely change how we like interact with AI. Like, imagine if your voice assistant could actually handle it if you interrupted it.

• 2:08Speaker 01

  Or like have an actual conversation.

• 2:10Speaker 02

  Right, not just these clunky back-and-forth things we're used to.

• 2:14Speaker 01

  And they actually shut it off, didn't they? I read something about a travel app, one for languages. Even one where the AI ordered takeout.

• 2:21Speaker 02

  Those demos were really interesting, and I think they show how this realtime API can be used in so many ways. And the tech behind it is fascinating, by the way. It uses persistent Web Socket connections and this thing called function calling, so it can respond in realtime.

• 2:38Speaker 01

  So the function calling thing, that sounds kind of complicated. Can you, like, explain how that works?

• 2:42Speaker 02

  So imagine giving the AI access to this whole toolbox, right? Information, capabilities, all sorts of things.

• 2:48Speaker 01

  Okay.

• 2:49Speaker 02

  So take the travel agent demo, for example. With function calling, the AI can pull up details, let's say about Fort Mason, right? From some database, like nearby restaurants, stuff like that.

• 2:59Speaker 01

  Ah, I get it. So instead of being limited to what it already knows, it can go and find the information it needs, like a human travel agent would?

• 3:07Speaker 02

  Precisely. And someone on Hacker News pointed out a cool detail. The API actually gives you a text version of what's being said, so you can store that, analyze it.

• 3:17Speaker 01

  That's smart. It seems like OpenAI put a lot of thought into making this API easy for developers to use. But while we're on OpenAI, you know, the sides of their tech, there's been some news about, like, internal changes, too. Didn't they say they're moving away from being a nonprofit?

• 3:32Speaker 02

  They did. And it's got everyone talking. It's a major shift, and it's only natural for people to wonder how that'll change things for OpenAI in the future. I mean, there are definitely some valid questions about this move to for-profit, like, will they have more money for research now? Probably. But will they, you know, care as much about making sure AI benefits everyone?

• 3:51Speaker 01

  Yeah, that's the big question, especially with all the, like, the leadership changes happening at OpenAI, too, right? I read that their chief research officer left, and their VP of Research, and even their CTO.

• 4:03Speaker 02

  It's true. A lot of people are connecting those departures with the changes in OpenAI's structure.

• 4:08Speaker 01

  And I guess it makes you wonder what's going on behind the scenes. But they are still putting out new stuff. Like, this whole fine-tuning thing really caught my eye.

• 4:17Speaker 02

  Right, fine-tuning. It's essentially taking a pre-trained AI model and, like, customizing it.

• 4:23Speaker 01

  So instead of a general AI, you get one that's tailored for a specific job?

• 4:27Speaker 02

  Exactly. And that opens up so many possibilities, especially for businesses. Imagine you could train an AI on your company's data, you know, like how you communicate your brand guidelines.

• 4:37Speaker 01

  So it's like having an AI that's specifically trained for your company?

• 4:41Speaker 02

  That's the idea.

• 4:41Speaker 01

  And they're doing it with images now, too, right? Fine-tuning with vision is what they called it.

• 4:46Speaker 02

  It's pretty incredible what they're doing with that, especially in fields like medicine.

• 4:50Speaker 01

  Like using AI to help doctors make diagnoses.

• 4:52Speaker 02

  Exactly. And AI could be trained on like thousands of medical images, right? And then it could potentially spot things that even a trained doctor might miss.

• 5:03Speaker 01

  That's kind of scary, to be honest. What if it gets it wrong?

• 5:06Speaker 02

  Well, the idea isn't to replace doctors, but to give them another tool, you know, help them make better decisions.

• 5:12Speaker 01

  Okay, that makes sense. But training these AI models must be really expensive.

• 5:17Speaker 02

  It can be. All those tokens add up. But OpenAI announced something called automatic prompt caching.

• 5:23Speaker 01

  Automatic what now? I don't think I came across that.

• 5:26Speaker 02

  So basically, if your AI sees a prompt that it's already seen before, OpenAI will give you a discount.

• 5:31Speaker 01

  Huh. Like a frequent buyer program for AI.

• 5:35Speaker 02

  Kind of, yeah. It's good that they're trying to make it more affordable. And they're also doing something called model distillation.

• 5:41Speaker 01

  Okay, now you're just using big words to sound smart. What's that?

• 5:45Speaker 02

  Think of it like like a recipe, right? You can take a really complex recipe and break it down to the essential parts.

• 5:50Speaker 01

  Make it simpler, but it still tastes the same.

• 5:53Speaker 02

  Yeah. And that's what model distillation is. You take a big, powerful AI model and create a smaller, more efficient version.

• 5:59Speaker 01

  So it's like lighter weight, but still just as capable.

• 6:03Speaker 02

  Exactly. And that means more people can actually use these powerful tools. They don't need like a supercomputer to run them.

• 6:10Speaker 01

  So they're making AI more accessible. That's great.

• 6:13Speaker 02

  It is. And speaking of powerful tools, they also talked about their new O1 model. That's the one they've been hyping up, the one that's supposed to be this big leap forward.

• 6:22Speaker 01

  Yeah, O1. It sounds pretty futuristic. Like, from what I read, it's not just a bigger, better language model.

• 6:28Speaker 02

  Right, it's a different approach.

• 6:29Speaker 01

  They're saying it can like actually reason, right? Think differently.

• 6:33Speaker 02

  It's trained differently. They used reinforcement learning with O1.

• 6:36Speaker 01

  So it's not just finding patterns in the data it's seen before.

• 6:40Speaker 02

  Not just that. It can actually learn from its mistakes, get better at solving problems.

• 6:46Speaker 01

  So give me an example. What can O1 do that, say, GPT-4 can't?

• 6:51Speaker 02

  Well, OpenAI showed it doing some pretty impressive stuff with math, like advanced math.

• 6:56Speaker 01

  Yeah.

• 6:56Speaker 02

  And coding, too. Complex coding, things that even GPT-4 struggled with.

• 7:00Speaker 01

  So you're saying if I needed to like write a screenplay, I'd stick with GPT-4. But if I wanted to solve some crazy physics problem, O1 is what I'd use?

• 7:08Speaker 02

  Something like that, yeah. Although there is a trade-off. O1 takes a lot more power to run. And it takes longer to get those impressive results.

• 7:17Speaker 01

  Hmm, makes sense. More power, more time, higher quality.

• 7:21Speaker 02

  Exactly.

• 7:22Speaker 01

  It sounds like it's still in development, though, right? Is there anything else they're planning to add to it?

• 7:26Speaker 02

  Oh, yeah. They mentioned system prompts, which will let developers like set some ground rules for how it behaves. And they're working on adding structured outputs and function calling.

• 7:38Speaker 01

  Wait, structured outputs? Didn't we just talk about that?

• 7:41Speaker 02

  We did. That's the thing where the AI's output is formatted in a way that's easy to use, like JSON.

• 7:47Speaker 01

  Right, right. So you don't have to spend all day trying to make sense of what it gives you. It's good that they're thinking about that stuff.

• 7:53Speaker 02

  It's about making these tools usable. And speaking of that, Dev Day finished up with this really interesting talk. Sam Altman, the CEO of OpenAI, and Kevin Weill, their new chief product officer.

• 8:07Speaker 02

  They talked about like the big picture for AI.

• 8:09Speaker 01

  Yeah, they did, didn't they? Anything interesting come up?

• 8:12Speaker 02

  Well, Altman talked about moving past this whole AGI term, artificial general intelligence.

• 8:18Speaker 01

  I can see why. It's kind of a loaded term, isn't it?

• 8:20Speaker 02

  He thinks it's become a bit of a buzzword, and people don't really understand what it means.

• 8:24Speaker 01

  So are they saying they're not trying to build AGI anymore?

• 8:28Speaker 02

  It's more like they're saying they're focused on just making AI better, constantly improving it, not worrying about putting it in a box.

• 8:36Speaker 01

  That makes sense. Keep pushing the limits.

• 8:38Speaker 02

  Exactly. But they were also very clear about doing it responsibly. They talked a lot about safety and ethics.

• 8:43Speaker 01

  Yeah, that's important.

• 8:44Speaker 02

  They said they were going to be very careful about how they release new features.

• 8:48Speaker 01

  Good, because this stuff is powerful.

• 8:51Speaker 02

  It is. It was a lot to take in, this whole Dev Day event, new tools, big changes at OpenAI, and these big questions about the future of AI.

• 8:59Speaker 01

  It was. But hopefully, this deep dive helped make sense of some of it. At least, that's what we try to do here.

• 9:05Speaker 02

  Absolutely.

• 9:06Speaker 01

  Thanks for taking the deep dive with us.

• 9:08Speaker 00

  The biggest demo of the new real-time API involved function calling with voice mode and buying chocolate-covered strawberries from our friendly local OpenAI developer Experience Engineer and strawberry shop owner Elon Bigio. We'll first play you the audio of his demo and then go into a little interview with him.

• 9:25Speaker 03

  Fantastic. Could you place a call and see if you could get us 400 strawberries delivered to the venue? But please keep that under $1,500.

• 9:37Speaker 04

  I'm on it. We'll get those strawberries delivered for you.

• 9:47Speaker 05

  Hello.

• 9:49Speaker 04

  Hi there. Is this Illus? I'm a romance AI assistant calling about a fantastic. Can you tell me what flavors of strawberry dip do you have available?

• 9:58Speaker 05

  Yeah, we have chocolate, vanilla, and we have peanut butter.

• 10:04Speaker 04

  Wait, how much would 400 chocolate-covered strawberries cost?

• 10:08Speaker 05

  400? Are you sure you want 400?

• 10:12Speaker 04

  Yes, 400 chocolate-covered strawberries. Wait, how much would that be?

• 10:17Speaker 05

  I think that'll be around like $1,415.92.

• 10:25Speaker 04

  Awesome. Let's go ahead and place the order for 400 chocolate-covered strawberries.

• 10:31Speaker 05

  Great, where would you like that delivered?

• 10:34Speaker 04

  Please deliver them to the Gateway Pavilion in Fort Mason, and I'll be paying cash.

• 10:43Speaker 05

  Okay, sweet. So just to confirm, you want 400 chocolate-covered strawberries to the Gateway Pavilion?

• 10:50Speaker 04

  Yes, that's perfect. And when can we expect delivery?

• 10:54Speaker 05

  Well, you guys are right nearby, so it'll be like, I don't know, 37 seconds?

• 11:00Speaker 04

  That's incredibly fast.

• 11:05Speaker 05

  Cool, you too.

• 11:09Speaker 06

  Hi, Ilan. Welcome to Lanespace.

• 11:11Speaker 05

  Thank you.

• 11:12Speaker 06

  Just saw your amazing demos, had your amazing strawberries. You are dressed up like exactly like a strawberry salesman.

• 11:17Speaker 05

  Gotta have it all.

• 11:19Speaker 06

  What was the building on demo like? What was the story behind the demo?

• 11:21Speaker 05

  It was really interesting. This is actually something I had been thinking about for months before the launch. Like, having a, like, AI that can make phone calls is something like I've personally wanted for a long time. And so as soon as we launched internally, like I started hacking on it. And then that sort of just made it into like an internal demo. And then people found it really interesting. And then we thought, how cool would it be to have this like onstage as

• 11:45Speaker 06

  Yeah.

• 11:46Speaker 05

  as one of demos.

• 11:47Speaker 06

  Yeah. Would you call out any technical issues building? Like you were basically one of the first people ever to build with the Voice Mode API. Would you call any issues like integrating it with Twilio like that, like you did with function calling, with like a form-filling elements? I noticed that you had like intense of things to fulfill, and then Uh, you're like, when, when you're still missing info, the, the voice would prompt you, role-playing the, the store guy.

• 12:13Speaker 05

  Yeah, yeah. So I think technically, there's like the whole, just working with audio and streams is a whole different piece. Like, even separate from like AI and this, this like new capabilities, it's just, it's just tough. Yeah, when you have a prompt, conversational, it'll just follow like the, it was set up like kind of step by step to like ask the right questions based on like the, what the request was, right? The function calling itself is sort of tangential to that. Like, you have to prompt it to, to call the functions, but then handling it isn't too much different from like what you would do with assistant streaming or like chat completion streaming. I think like, the API feels very similar, just like if everything in the API was streaming, it actually feels quite familiar to that.

• 12:53Speaker 06

  And then, function calling wise, I mean, does it work the same? I, I don't know. Like, I saw a lot of logs. You guys showed like in the playground, a lot of logs. What is in there? What, what should people know?

• 13:04Speaker 05

  Yeah, I mean, it is like, the events may have different names than the streaming events that we have in chat completions, but they represent very similar things.

• 13:12Speaker 06

  Okay.

• 13:13Speaker 05

  It's things like, you know, function call started, argument started, it's like, here's like argument deltas, and then like function call done. Conveniently, we send one with like, has the full function, and then I just use that.

• 13:24Speaker 06

  Nice.

• 13:25Speaker 05

  Yeah.

• 13:26Speaker 06

  And then, like, what restrictions do, should people be aware of? Like, you know, I think, I think before we recorded, we discussed a little bit about the sensitivities around basically calling random store owners and putting, putting like an AI on them.

• 13:40Speaker 05

  Yeah, so there's, I think there's recent regulation on that, which is why we want to be like very, I guess, aware of, of, you know, you can't just call anybody with AI, right? That's like just robo-calling. You wouldn't want someone just calling you with AI.

• 13:52Speaker 06

  Yeah, yeah. So like, I have, I'm a developer, I'm about to do this on random people.

• 13:56Speaker 05

  Yeah.

• 13:56Speaker 06

  Uh, what, what, what laws am I about to break?

• 14:00Speaker 05

  I I forget what the governing body is, but you should I think having consent of the person you're about to call, it always works, right? I, as the strawberry owner, have consented to like getting called with AI. I think past that, you you wanna be careful. Definitely individuals are more sensitive than businesses. The businesses, you have a little bit more leeway. Also, they're like, businesses, I think, have an incentive to want to receive AI phone calls, especially if like they're dealing with it.

• 14:27Speaker 06

  It's doing business.

• 14:27Speaker 05

  With AI phone calls, right? Like, it's more business. It's kind of like getting on a booking platform, right? You're exposed to more. But I think it's still very much like a gray area, and so

• 14:39Speaker 05

  I think everybody should,

• 14:41Speaker 05

  you know, tread carefully, like, figure out what it is. I I I, the law is so recent, I didn't have enough time to like, I'm also not a lawyer.

• 14:47Speaker 06

  Yeah, of course. Okay, cool, fair enough. One other thing, this is kind of agentic. Did you use a state machine at all? Did you use any framework?

• 14:56Speaker 05

  No, no.

• 14:56Speaker 06

  You just stick it in context and then just run it in a loop until it ends ends call?

• 15:01Speaker 05

  Yeah, there there isn't even a a loop, like, because the API is just based on sessions, it's always just gonna keep going. Every time you speak, it'll trigger a call. And then after every function call, it was also invoking like a a generation. And so that is another difference here. It's like, it's inherently almost like in a loop, be just by being in a session, right? No state machines needed. I'd say this is very similar to like the notion of routines, where it's just like a list of steps and it's like sticks to them softly, but usually pretty well.

• 15:33Speaker 06

  And the steps is the prompts?

• 15:34Speaker 05

  The steps, it's like the the prompt, like the steps are in the prompt.

• 15:38Speaker 06

  Yeah.

• 15:38Speaker 05

  Right, it's like step one, do this, step one, step to do that.

• 15:41Speaker 06

  What what if I want to change the system prompts halfway through the conversation?

• 15:44Speaker 05

  You can.

• 15:45Speaker 06

  Okay.

• 15:45Speaker 05

  You can. To be honest, I have not played with that too too much.

• 15:47Speaker 06

  Yeah, yeah.

• 15:48Speaker 05

  But I know you can.

• 15:49Speaker 06

  Yeah, yeah.

• 15:50Speaker 05

  Yeah.

• 15:50Speaker 06

  Awesome. I noticed that you called it real-time API, but not voice API. So I assume that it's like real-time API starting with voice, right? And then that's what that's what he said on the thing. I can't imagine, like, what else is real-time?

• 16:02Speaker 05

  Well, yes. To use ChatGPT's voice mode as an example, like we've demoed the video, right? Like real-time image, right? So I'm not actually sure what timelines are, but I would expect if I had to guess that like that is probably the next thing that we're gonna be making. You'd probably have to talk directly with the team building this.

• 16:19Speaker 06

  Sure. You can't promise their timelines.

• 16:21Speaker 05

  Yeah, yeah, yeah, right, exactly. But like, given that this is the features that currently, or that exists that we've demoed on ChatGPT. Yeah.

• 16:28Speaker 06

  There will never be a case where there's like a real-time text API, right? I don't.

• 16:32Speaker 05

  Well, this is a real-time text API. You can do text-only on this.

• 16:36Speaker 06

  Oh.

• 16:37Speaker 05

  Yeah. I don't know why you would, but it's actually, so text-to-text here doesn't quite make a lot of sense. I don't think you'll get a lot of latency gain. But like, speech-to-text is really interesting because you can prevent, you can prevent responses, like audio responses, and force function calls. And so you can do stuff like UI control that is like super, super reliable. We had a lot of like, you know, unlike, we weren't sure how well this was gonna work because it's like, you have a voice answering, it's like a whole persona, right? Like, that's a little bit more, you know, risky. But if you like, cut out the audio outputs and make it so it always has to output a function, like you can end up with pretty, pretty reliable like commands, like a command architecture.

• 17:21Speaker 06

  Yeah, actually, that's the, that's the way I wanna interact with a lot of these things as well. Like, one-sided voice, not, don't force two-sided on me.

• 17:27Speaker 05

  You don't necessarily wanna hear the voice back and like, sometimes it's like, yeah, I think having an output voice is great, but I feel like I don't always wanna hear an output voice. I'd say usually I don't. But yeah, exactly, being able to speak to it is super smooth.

• 17:39Speaker 06

  Cool. Do you wanna comment on any of the other stuff that you announced? From caching, I noticed was like, I like the no code change part. I'm looking forward to the docs because I'm sure there's a lot of details on like what you cache, how long you cache.

• 17:50Speaker 05

  Yeah.

• 17:51Speaker 06

  Because like, Anthropic caches for like five minutes. I was like, okay, but what if I don't make a call every five minutes?

• 17:56Speaker 05

  Yeah, to be super honest with you, I've been so caught up with the real-time API and making the, the demo that I haven't read up on the other launches too much. I mean, I'm aware of them, but I think I'm excited to see how well distillation works. That's something that we've been doing like, I don't know, I've been like doing it between our models for a while and I've seen. Really good results like I've done back in the day like from GPT four to GPT three point five. Um, and got like, like pretty much same level of like function calling with like hundreds of functions.

• 18:24Speaker 06

  Yeah.

• 18:24Speaker 05

  Um, so that was super, super compelling. So I feel like easier distillation, I'm, I'm really excited for.

• 18:30Speaker 06

  I see. Is it a tool? Uh, so I saw evals.

• 18:32Speaker 05

  Yeah.

• 18:33Speaker 06

  Like, what, what is the distillation product? It wasn't super clear, to be honest.

• 18:36Speaker 05

  I, I think I wanna, I wanna let that team, I wanna let that team talk about it.

• 18:40Speaker 06

  Okay, right. Well, I appreciate you jumping on.

• 18:41Speaker 05

  No, of course.

• 18:42Speaker 06

  Amazing demo. It was beautifully designed. I'm sure that was part of you and Roman and.

• 18:47Speaker 05

  Yeah, I guess shout out to like the first people to like creators of Wanderlust originally were like Simon and Carolis. And then like I took it and built the voice component and the voice calling components. Yeah. So it's been, it's been a big team effort and then like the entire PI team for like debugging everything as it's been going on. It's been.

• 19:05Speaker 06

  Yeah, you're the first consumers on the DX team.

• 19:08Speaker 05

  Yeah.

• 19:08Speaker 06

  Yeah, I mean, the classic role of, of what we do there.

• 19:10Speaker 05

  Yeah.

• 19:10Speaker 06

  Okay, yeah. Anything else? Any other cost actions?

• 19:13Speaker 05

  No, enjoy Dev Day.

• 19:14Speaker 06

  Thank you.

• 19:15Speaker 05

  Yeah.

• 19:15Speaker 06

  That's it.

• 19:16Speaker 00

  The Latent Space crew then talked to Olivier Godmont, head of product for the OpenAI platform, who led the entire Dev Day keynote and introduced all the major new features and updates that we talked about today.

• 19:28Speaker 06

  Okay, so we are here with Olivier Godmont. It's, I, I, I don't, I don't pronounce French.

• 19:33Speaker 07

  That's fine. It's perfect.

• 19:35Speaker 06

  And it was amazing to see your keynote today. What was the backstory of, of preparing something like this, preparing like Dev Day?

• 19:43Speaker 07

  Essentially came from a couple of places. Number one, excellent reception from last year's Dev Day. Developers, startup founders, researchers wanna spend more time with OpenAI and we wanna spend more time with them as well. And so for us, like it was a no brainer, frankly, to do it again, like, you know, like a nice conference. The second thing is going global. We've done a few events like in Paris and like a few other, like, you know, non-European, non-American countries. And so this year we're doing SF, Singapore and London to frankly just meet more developers.

• 20:10Speaker 06

  Yeah, I'm very excited for the Singapore one.

• 20:12Speaker 07

  Oh, yeah.

• 20:12Speaker 06

  You, will you be there? I don't know. I don't know if I got an invite. I should, I can, I can just talk to you. Yeah, like, and then there was some speculation around October 1st.

• 20:22Speaker 07

  Yeah.

• 20:22Speaker 06

  Is it because oh, one, October 1st?

• 20:25Speaker 07

  It has nothing to do. I discovered the tweet yesterday where like people are so creative. Now, oh, one, there was no connection to October 1st, but in hindsight, like that, that would have been a pretty, a pretty good meme, but yeah, no.

• 20:36Speaker 06

  Okay. Yeah, and you know, I think like OpenAI's outreach to developers is something that I felt the hole in 2022 when like, you know, like people were trying to build with ChatGPT and like there was no function calling all that stuff that you talked about in the past. And that's why I started my own conferences, like here's our little developer conference thing. And but to see this opening at Dev Day now and like to see so many developer oriented products coming out of OpenAI, I think it's really encouraging.

• 21:02Speaker 07

  Yeah, totally. It's that's what I said essentially, like developers are basically the people who make the best connection between the technology and you know, the future essentially. Like, you know, essentially see your capability, see your low level like technology. And I like, hey, I see how that application or that use case that can be enabled. And so in the direction of enabling like AGI like for all of humanity, it's a no brainer for us to partner with Devs.

• 21:31Alessio Fanelli?

  And most importantly, you almost never had wait lists, which compared to like other releases, people usually usually have. What is the, you know, you had prompt caching, you had real time voice API, we, you know, shunned it a long Twitter thread so people know the releases.

• 21:46Speaker 07

  Yeah.

• 21:46Alessio Fanelli?

  What is the thing that was like sneakily the hardest to actually get ready for for Dev Day or like what was the kind of like, you know, last 24 hours, anything that you didn't know was gonna work?

• 21:56Speaker 07

  Yeah, they're all fairly, like I would say involved, like features to ship. So the team has been working for months, all of them. The one which I would say is the newest for OpenAI is the real time API. For a couple of reasons, I mean, one, you know, it's a new modality. Second, like it's the first time that we have an actual like web socket based API. And so I would say that's the one that required like the most work over the month to get right from a developer perspective and to also make sure that our existing safety mitigations that work well with like real time audio in and audio out.

• 22:30Speaker 06

  Yeah. What are the design choices or what's like the sort of design design choices that you want to highlight? Like, you know, like, I think for for me, like web sockets, you just receive a bunch of events, it's two way. I obviously don't have a ton of experience. I think a lot of developers are going to have to embrace this real time programming. Like, what are you designing for? Or like, what what advice would you have for developers exploring this?

• 22:51Speaker 07

  The core design hypothesis was essentially how do we enable like human level latency?

• 22:59Speaker 07

  We did a bunch of tests, like on average, like human beings, like you know, it takes like something like 300 milliseconds to converse to each other. And so that was the design principle, essentially, like working backward from that and you know, making your technology work. And so we evaluated a few options and WebSockets was the one that we landed on. So that was like one design choice. A few other like big design choices that we had to make, prom caching. Prom caching, the design like target was automated from the get-go, like zero code change from the developer. That way you don't have to learn like what is a prom prefix and you know, how long does the cache work? Like we just do it as much as we can essentially. So that was a big design choice as well. And then finally on destination, like and evaluation.

• 23:40Speaker 07

  The big design choice was something I learned as high like in my previous job, like a philosophy around like a pit of success. Like what is essentially the, the, the minimum number of steps for the majority of developer to do the right thing? Because when you do evals on fine tuning, there are many, many ways like to mess it up, for example, like you know, and have like a copy model, like evals that tell like a wrong story. And so our whole design was okay. We actually care about like helping people who have like that much experience, like evaluating model, like get like in a few minutes like to a good spot. And so how do we essentially enable that pit of success like in the product flow?

• 24:15Speaker 06

  Yeah, yeah, I'm a little bit scared to fine tune, especially for vision. Because I don't know what I don't know for, for, for stuff like vision, right? Like for, for text, I can evaluate pretty easily. For vision, let's say I'm like trying to, one of your examples was Grab, which very close to home, I'm from Singapore. I think your example was like they identified stop signs better. Why is that hard? Why do I have to fine tune that? If I fine tune that, do I lose other things, you know, like there's a lot of unknowns with, with vision that I, that I think developers have to figure out.

• 24:50Speaker 07

  For sure. Vision is going to. Open up like a new, I would say, evaluation space. Because you're right, like it's harder, like you know, to tell correct from incorrect essentially with images. What I can say is that we've been alpha testing like the vision fine tuning like for several weeks at that point. We are seeing like even higher performance uplift compared to text fine tuning. So that's, there is something here, like we've been pretty impressed, like in a good way frankly, but you know how well it works. But for sure, like you know, I expect the developers who are moving from one modality to like text and images will have like more testing, evaluation, like you know, to set in place, like to make sure it works well.

• 25:25Alessio Fanelli?

  The model distillation and evals is definitely like the most interesting, moving away from just being a model provider to being a platform provider. How should people think about being the source of truth? Like do you want OpenAI to be like the system of record of all the prompting? Because people sometimes store it in like different data sources. And then is that gonna be the same as the models evolved? So you don't have to worry about, you know, refactoring the data, like things like that, or like future model structures?

• 25:51Speaker 07

  The vision is if you wanna be a source of truth, you have to earn it, right? Like we're not going to force people like to pass us data, there is no value prop like you know, for us to store the data. The vision here is, at the moment, like most developers like use like a one size fits all model, like they have the shelf like G P four essentially. The vision we have is fast forward a couple of years. I think like most developers will essentially like have an automated, continuous, fine-tuned model. The more like you use the model, the more data you pass to the model provider, like the model is automatically like fine-tuned, evaluated to get some eval sets. And essentially like you don't have to, every month when there is a new snapshot, like you know, to go online and you know, try a few new things. That's the direction.

• 26:35Speaker 07

  We are pretty far away from it. But I think like that evaluation and decision product are essentially a first good step in that direction. It's like, hey, if you are excited by that direction and you give us like evaluation data, we can actually log your completion data and start to do some automation in our behalf.

• 26:52Alessio Fanelli?

  And then you can do evals for free if you share data with OpenAI? How should people think about when it's worth it, when it's not? Sometimes people get overly protective of their data when it's actually not that useful. But how should developers think about when it's right to do it, when not, or if you have any thoughts on it?

• 27:08Speaker 07

  The default policy is still the same, like, you know, we don't train on like any API data unless you opt in. What we've seen from feedback is evaluation can be expensive. Like if you run like 01 evals on like thousands of samples, like your build will get increased like, you know, pretty significantly. That's problem statement number one. Problem statement number two is essentially I want to get to a world where whatever OpenAI ships a new model snapshot, we have full confidence that there is no regression for the task that developers care about. And for that to be the case, essentially, we need to get evals. And so that essentially is a sort of a two-buck-one stone. It's like we subsidize specifically evals and we also use evals when we ship like new models to make sure that, you know, we keep going in the right direction. So in my sense, we win, but again, like completely opt in.

• 27:51Speaker 07

  I expect that, you know, many developers will not want to share their data and that's perfectly fine to me.

• 27:56Speaker 06

  Yeah, I think free evals though, very, very good incentive.

• 27:59Speaker 07

  Yeah.

• 28:00Speaker 06

  I mean, it's a fair trade, like you get data, we get free evals.

• 28:03Speaker 07

  Exactly, and we sanitize PII, everything, you know, like we have no interest in like the actual like sensitive data. We just want to have good evaluation on real use cases.

• 28:13Speaker 06

  Like, I almost went to eval the eval. I don't know if that ever came up. Like sometimes the evals themselves are wrong. And then there's no way for me to tell you.

• 28:22Speaker 07

  Everyone who is starting with LLM, teaching with LLM is like, yeah, evaluation easy, you know, I've done testing like all my life. And then you start to actually build evals, understand like all the corner cases and you realize, wow, there's like a whole field itself. So yeah, good evaluation is hard and so, yeah.

• 28:38Speaker 06

  Yeah, but I think there's a, you know, I just talked to Brain Trust, which I think is one of your partners. They also emphasize code-based evals versus your sort of low code. What I see is like, I don't know, maybe there's some more that you didn't demo, but what I see is like a low-code experience, right, for evals. Would you ever support like a more code-based, like would I run code on OpenAI's eval platform?

• 29:02Speaker 07

  For sure. I mean, we meet developers where they are, you know, at the moment, the demand was more for like, you know, easy to get started, like eval. But you know, if we need to expose like an evaluation API, for instance, for people like, you know, to pass like, you know, their existing test data, we'll do it. So yeah, there is no, you know, philosophical, I would say, like, you know, misalignment on that.

• 29:19Speaker 06

  Yeah, yeah. What I think this is becoming, by the way, and I don't like, it's basically like you're becoming AWS, like the AI cloud. And I don't know if like, that's a conscious strategy, or it's like, no, it doesn't even have to be a conscious strategy. Like, you're gonna offer storage, you're gonna offer compute, you're gonna offer networking. I don't know what networking looks like. Networking is maybe like caching, or like, it's a CDN, it's a prompt CDN.

• 29:45Speaker 07

  Yeah.

• 29:45Speaker 06

  But it's the AI versions of everything, right? Do you, like, do you see the analogies, or?

• 29:50Speaker 07

  Yeah, totally. Whenever, whenever I talk to developers, I feel like good models are just half of the story to build a good app. There's a ton more you need to do. Evaluation is the perfect example. Like, you know, you can have the best model in the world if you're in the dark, like, you know, it's really hard to gain the confidence. And so, our philosophy is, um, the whole like software development stack is being basically reinvented, you know, with LLMs. There is no freaking way that OpenAI can build everything. Like, there is just too much to build, frankly. And so, my philosophy is, essentially, we'll focus on like the tools which are like the closest to the model itself. So that's why you see us, like, you know, investing quite a bit in like fine tuning, distillation, now evaluation.

• 30:33Speaker 07

  Because we think that it actually makes sense to have like in one spot, like, you know, all of that. Like, there is some sort of virtuous circle, essentially, you can set in place. But stuff like, you know, LLM ops, like tools which are like further away from the model, I don't know. If you wanna do like, you know, super elaborate, like prompt management, or, you know, like tooling. Like, I'm not sure, like, you know, OpenAI has like such a big edge, frankly, like, you know, to build this sort of tools. So that's how we view it at the moment. But again, frankly, like, the philosophy is like super simple, the strategy is super simple. It's meeting developers where they want us to be. And so, you know, that's frankly, like, you know, day in, day out, like, you know, what I try to do.

• 31:09Speaker 07

  Thank you so much for the time, I'm sure you're

• 31:10Speaker 06

  Yeah, I have more questions on, a couple of questions on voice and then also like your call to action, like what you want feedback on, right? So, but I think, I think we should spend a bit more time on voice because I feel like that's like the, the big splash thing. I talked, well, well, I mean, I mean, just like, what is the future of real time for OpenAI?

• 31:28Speaker 07

  Yeah.

• 31:29Speaker 06

  Because I think obviously video is next, you already have it in the, the ChatGPT desktop app. Do we just like have a permanent like, you know, like our developers just going to be like sending sockets back and forth with OpenAI? Like, how do we program for that? Like, what, what is the future?

• 31:44Speaker 07

  Yeah, that makes sense. I think with multimodality, like real time is quickly becoming like, you know, essentially the right experience, like to build an application. So my expectation is that we'll see like a non-trivial like volume of applications, like moving to a real time API. Like if you zoom out, like audio is a really example, like audio until basically now, audio on the web, in apps, was basically very much like a second-class citizen. Like you basically did like an audio chatbot for users who did not have a choice. You know, they were like struggling to read, or I don't know, they were like not super educated with technology. And so frankly, it was like the crappy option, you know, compared to text.

• 32:25Speaker 07

  But when you talk to people in the real world, the vast majority of people like prefer to talk and listen instead of typing and writing.

• 32:34Speaker 06

  We speak before we write.

• 32:35Speaker 07

  Exactly. I don't know. I mean, I'm sure it's the case for you in Singapore. For me, my friends in Europe, the number of like WhatsApp, like voice notes you receive every day. I mean, just people, it makes sense, basically, like, you know.

• 32:44Speaker 06

  Chinese.

• 32:45Speaker 07

  Chinese, yeah.

• 32:46Speaker 06

  Yeah, all voice.

• 32:47Speaker 07

  You know, it's easier, there is more emotions. I mean, you know, you get the point across like pretty well. And so my personal ambition for like the real time API and like audio and video is to make like audio and like multimodality like truly a first-class experience. Like, you know, if you're like, you know, the amazing, like super bold, like startup out of Y C, you're going to be like the next like billion, like, you know, user application, to make it like truly audio first and make it feel like, you know, an actual good, like, you know, product experience. So that is essentially the ambition, and I think like, yeah, it could be pretty big.

• 33:16Speaker 06

  Yeah. I think one, one people, one issue that people have with the voice so far as, as releasing the advanced voice mode is the refusals.

• 33:24Speaker 07

  Yeah.

• 33:24Speaker 06

  You guys had a very inspiring model spec, I think Joanne worked on that, where you said like, yeah, we don't want to overly refuse all the time. In fact, like even if like not safe for work, like in some occasions, it's okay. How is there an API that we can say not safe for work, okay?

• 33:42Speaker 07

  I think we'll get there. I think we'll get there. The Momo spec like nailed it, like, you know.

• 33:46Speaker 06

  It nailed it, it was so good!

• 33:48Speaker 07

  Yeah, we are not in the business of like policing, you know, if you can say like vulgar words or whatever. You know, there are some use cases like, you know, I'm writing like a Hollywood like script. I wanna say like vulgar words, it's perfectly fine, you know. And so I think the direction where we'll go here is that basically there will always be like, you know, a set of behavior that will, you know, just like forbid frankly, because they're illegal against our terms of services. But then there will be like, you know, some more like risky like themes which are completely legal, like, you know, vulgar words or, you know, not safe for work stuff. Where basically we'll expose like a controllable like safety like knobs in the API. To basically allow you to say, hey, that theme okay, that theme not okay. How sensitive do you want the threshold to be on safety refusals? I think that's the direction.

• 34:30Speaker 06

  So a safety API?

• 34:32Speaker 07

  Yeah, in a way, yeah.

• 34:33Speaker 06

  Yeah, we have, we've never had that.

• 34:34Speaker 07

  Yeah.

• 34:35Speaker 06

  Because right now is you, it's whatever you decide and then it's, that's it. That, that, that would be the main reason I don't use OpenAI Voice is because of

• 34:42Speaker 07

  It's over policy at the moment.

• 34:42Speaker 06

  Over refuse, over refusals.

• 34:44Speaker 07

  Yeah, yeah, yeah. No, we gotta fix that. Like singing.

• 34:47Speaker 06

  So I'm a singer and you, you locked off singing.

• 34:51Speaker 07

  Yeah, yeah, yeah.

• 34:52Speaker 06

  But I, I understand music gets you in trouble. Okay, yeah, so then just generally like what do you wanna hear from developers? Right, we have, we have all developers watching. You know, what feedback do you want? Any, anything specific as well, like from, especially from today. Anything that you are unsure about that you're like, our feedback could really help you decide.

• 35:09Speaker 07

  For sure. I think

• 35:12Speaker 07

  essentially it's becoming pretty clear after today that, you know, I would say the OpenAI direction has become pretty clear, like, you know, after today. Investment in reasoning, investment in multimodality, investment as well like in, I would say, tool use, like function calling. To me, the biggest question I have is, you know, where should we put the cursor next? I think we need all three of them practically, like, you know, so we'll keep pushing.

• 35:33Speaker 06

  Hire 10,000 people. Or actually, no need, build, build a bunch of bots.

• 35:37Speaker 07

  Exactly. And so let's take O1 for instance, like is O1's fine enough? Like for your problems? Like, you know, let's say aside for a second, do you see models? Like for the apps that you would love to build? Is O1 basically it in reasoning or do we still have like, you know, a step to do?

• 35:50Speaker 06

  Preview is not enough. I need, I need the full one.

• 35:53Speaker 07

  Yeah. So that's exactly that sort of feedback. Essentially what I would love to do is for developers, I mean, there's a thing that Sam which has been saying like over and over again, like, you know, it's a set that don't, but I think it's directionally correct. As a developer, as a founder, you basically want to build an app which is a bit too difficult for the model today. Right? Like what you think is right, it's like sort of working, sometimes not working. And that way, you know, that basically gives us like a goalpost and be like, okay, that's what you need to enable with the next model release like in a few months. And so I would say that usually like that's a sort of feedback which is like the most useful that I can like directly like, you know, incorporate.

• 36:33Speaker 06

  Awesome. I think that's our time.

• 36:36Speaker 07

  Thank you so much, guys.

• 36:37Speaker 06

  Yeah, thank you so much.

• 36:38Speaker 07

  Thank you.

• 36:39Speaker 00

  We were particularly impressed that Olivier addressed the not safe for work moderation policy question head on, as that had only previously been picked up on in Reddit forums. This is an encouraging sign that we will return to in the closing candor with Sam Altman at the end of this episode. Next, a chat with Roman Hewitt, friend of the pod, AI engineer world's fair closing keynote speaker and head of developer experience at OpenAI on his incredible live demos and advice to AI engineers on all the new modalities.

• 37:12Alessio Fanelli?

  All right, we're live from OpenAI Dev Day with Roman, who just did two great demos on on stage and has been a friend of Laiden Space. So thanks for taking some of the time.

• 37:20Speaker 03

  Of course. Yeah. Thank you for being here and spending the time with us today.

• 37:23Speaker 06

  Yeah, I appreciate it. Appreciate you guys putting this on. I know it's like extra work, but it really shows the developers that you care and about reaching out.

• 37:31Speaker 03

  Yeah, of course. I think when you go back to the OpenAI mission, I think for us it's super important that we have the developers involved in everything we do, making sure that, you know, they have all of the tools they need to build successful apps. And we really believe that the developers are always going to invent the ideas, the prototypes, the fun factors of AI that we can't build ourselves. So it's really cool to have everyone here.

• 37:51Speaker 06

  We had Michelle from the, from you guys on.

• 37:54Speaker 03

  Yes.

• 37:54Speaker 06

  And she very, thank you. And she very seriously said API is the path to AGI.

• 37:58Speaker 03

  Correct.

• 37:58Speaker 06

  And people in our YouTube comments were like, API is not AGI. I'm like, no, like she's very serious. API is the path to AGI because like you're not going to build everything like the developers are, right?

• 38:08Speaker 03

  Of course. Yeah, that's the, that's the whole value of having a platform and an ecosystem of amazing builders who can like in turn create all of these apps. I'm sure we talked about this before, but there's now more than 3 million developers building on OpenAI. So it's pretty exciting to see all of that energy into creating new things.

• 38:26Alessio Fanelli?

  I was gonna say, you built two apps on stage today. An international space station tracker and then a drone. The hardest thing must have been opening Xcode and setting that up. Now, like, the models are so good that they can do everything else.

• 38:38Speaker 03

  Yes.

• 38:38Alessio Fanelli?

  You had two modes of interaction. You had kind of like chat GPT app to get the plan with O1. And then you had cursor to do apply some of the changes.

• 38:47Speaker 03

  Correct.

• 38:47Alessio Fanelli?

  How should people think about the best way to consume the coding models especially? Both for, you know, brand new projects and then existing projects that are trying to modify instead?

• 38:56Speaker 03

  Yeah. I mean, one of the things that's really cool about O1 preview and O1 mini being available in the API is that you can use it in your favorite tools like cursor, like I did, right? And that's also what like Devon from cognition can use in their own software engineering agent. In the case of Xcode, like, it's not quite deeply integrated in Xcode. So that's why I had like chat GPT side by side. But it's cool, right? Because I could instruct O1 preview to be like my coding partner and brainstorming partner for this app. But also consolidate all of the files and architect the app the way I wanted. So all I had to do was just like port the code over to Xcode and zero shot the app built. I don't think I conveyed, by the way, how big a deal that is.

• 39:37Speaker 03

  But like, you can now create an iPhone app from scratch, describing a lot of intricate details that you want. And your vision comes to life in like a minute. It's pretty outstanding.

• 39:48Speaker 06

  I have to admit, I was a bit skeptical because if I open up Xcode, I don't know anything about iOS programming. You know which file to paste it in. You probably set it up a little bit. So I'm like, I have to go home and test it to like figure out. And I need the chat GPT desktop app so that it can tell me where to click.

• 40:04Speaker 03

  Yeah, I mean, like, Xcode and iOS development has become easier over the years since they introduced Swift and Swift UI. I think back in the days of Objective C or like, you know, the storyboard, it was a bit harder to get in for someone new. But now with Swift and Swift UI, their dev tools are really exceptional. But now, when you combine that with O1 as your brainstorming and coding partner, it's like your architect effectively. That's the best way, I think, to describe O1. People ask me like, but can GPT-4 do some of that? And it's really. But I think it would just start spitting out code, right? And I think what's great about O1 is that it can like make up a plan. In this case, for instance, the iOS app had to fetch data from an API. It had to look at the docs. It had to look at like, how do I parse this JSON?

• 40:49Speaker 03

  Where do I store this thing? And kind of wire things up together. So that's where it really shines.

• 40:54Speaker 06

  Is mini or preview the better model that people should be using?

• 41:00Speaker 03

  I think people should try both. We're obviously very excited about the upcoming O1 that we shared the evals for. But we noticed that O1 mini is very, very good at everything math, coding, everything STEM. If you need for your kind of brainstorming or your kind of science part, you need some broader knowledge than reaching for O1 previews better. But yeah, I used the O1 mini for my second demo and it worked perfectly. All I needed was very much like something rooted in code, architecting and wiring up like a front end, a backend, some UDP packets, some web sockets, something very specific and did that perfectly.

• 41:35Speaker 06

  And then maybe just talking about Voice and Wanderlust, the app that keeps on giving.

• 41:39Speaker 03

  It does indeed, yeah.

• 41:41Speaker 06

  What's the backstory behind like preparing for all of that?

• 41:44Speaker 03

  You know, it's funny because when last year for Dev Day, we were trying to think about what could be a great demo app to show like an assistive experience. I've always thought travel is a kind of a great use case because you have like pictures, you have locations, you have the need for translations potentially. There's like so many use cases that are bounded to travel. But I thought last year, let's use a travel app. And that's how Wanderlust came to be. But of course, a year ago, all we had was a text-based assistant. And now we thought, well, if there's a voice modality, what if we just bring this app back as a link? And what if we were interacting better with voice? And so with this new demo, what I showed was the ability to like have a complete conversation in real time with the with the app.

• 42:30Speaker 03

  But also, the thing we wanted to highlight was the ability to call tools and functions, right? So, like, in this case, we placed a phone call using the Twilio API, interfacing with our AI agent. But developers are so smart that they'll come up with so many great ideas that we could not think ourselves, right? But what if you could have, like, you know, a 911 dispatcher? What if you could have like a customer service like center? That is much smarter than what we've been used to today. There's gonna be so many use cases for real time, it's awesome.

• 43:00Speaker 06

  Yeah, and sometimes, actually, you, like, this should kill phone trees. Like, there should not be like dial one.

• 43:07Speaker 03

  Of course.

• 43:07Speaker 06

  Para español, you know.

• 43:09Speaker 03

  Yeah, exactly.

• 43:10Speaker 06

  Espero dos, or whatever, I don't know.

• 43:12Speaker 03

  I mean, even you starting speaking Spanish would just do the thing, you know? You don't even have to ask. So yeah, I'm excited for this future where we don't have to interact with those legacy systems.

• 43:22Speaker 06

  Yeah, yeah. Is there anything, so you're doing function calling in a streaming environment? So, basically, it's web sockets, it's UDP, I think. It's basically not guaranteed to be exactly once delivery? Like, is there any coding challenges that you encountered when building this?

• 43:39Speaker 03

  Yeah, it's a bit more delicate to get into it. We also think that for now, what we shipped is a beta of this API. I think there's much more to build onto it. It does have the function calling and the tools. But we think that, for instance, if you wanna have something very robust on your client side, maybe you wanna have WebRTC as a client, right? And as opposed to like directly working with the sockets at scale? So that's why we have partners like Life Kit and Agora if you wanna, if you wanna use them. And I'm sure we'll have many more in the, many more in the future. But yeah, we keep on iterating on that. And I'm sure the feedback of developers in the weeks to come is gonna be super critical for us to get it right.

• 44:16Speaker 06

  Yeah, I think Life Kit has been fairly public that they are used in the ChatGPT app. Like, is it, this is just all open source? And we just use it directly with OpenAI? Or do we use Life Kit Cloud or something?

• 44:28Speaker 03

  So right now, we released the API, we released some sample code also, and referenced clients for people to get started with our API. And we also partnered with Life Kit and Agora. So they also have their own like ways to help you get started that plugs natively with the real-time API. So depending on the use case, people can decide what to use. If you work on something as completely client, or if you're working something on the server side for the voice interaction, you may have different needs. So we wanna support all of those.

• 44:55Alessio Fanelli?

  I know you've got to run. Is there anything that you want the AI engineering community to give feedback on specifically? Like even down to like, you know, a specific API endpoint or like, what's like the thing that you want?

• 45:08Speaker 03

  Yeah, I mean, you know, if we take a step back, I think Dev Day this year is all different from last year. And in a few different ways, but one way is that we wanted to keep it intimate, even more intimate than last year. We wanted to make sure that the community is on the spotlight. That's why we have community talks and everything. And the takeaway here is like learning from the very best developers and AI engineers. And so, you know, we want to learn from them. Most of what we shipped this morning, including things like prompt caching, the ability to generate prompts quickly in a playground, or even things like vision fine tuning. These are all things that developers have been asking of us. And so the takeaway I would leave them with is to say like, hey, the roadmap that we're working on is heavily influenced by them and their work.

• 45:53Speaker 03

  And so we love feedback from high feature requests. As you said, down to like very intricate details of an API endpoint, we love feedback. So yes, that's how we build this API.

• 46:05Speaker 06

  Yeah, I think the model distillation thing as well, it might be like the most boring, but like actually used a lot.

• 46:12Speaker 03

  True, yeah. And I think maybe the most unexpected, right? Because I think if I read Twitter correctly the past few days, a lot of people were expecting us to ship the real time API for speech to speech. I don't think developers were expecting us to have more tools for distillation. And we really think that's going to be a big deal, right? If you're building apps that have, you know, you want high like low latency, low cost, but high performance, high quality on the use case, distillation is going to be amazing.

• 46:40Speaker 06

  Yeah, I said in the distillation session just now, and they showed how they distilled from four to four mini. And it was like only like a 2% hit in the performance and 15x cheaper.

• 46:50Speaker 03

  Yeah, yeah, I was there as well for the superhuman kind of use case inspired for an Apple client. Yeah, this was very good. Go, man. Amazing. Thank you so much, buddy. Thanks again for being here today. It's always great to have you.

• 47:01Speaker 00

  As you might have picked up at the end of that chat, there were many sessions throughout the day focused on specific new capabilities, like the new model distillation features combining evals and fine tuning. For our next session, we are delighted to bring back two former guests of the pod, which is something listeners have been greatly enjoying in our second year of doing the Latent Space Podcast.

• 47:25Speaker 00

  Michelle Pokrass of the API team joined us recently to talk about structured outputs and today gave an updated long-form session at Dev Day describing the implementation details of the new structured output mode. We also got her updated thoughts on the voice mode API we discussed in her episode, now that it is finally announced. She is joined by friend of the pod and super blogger Simon Willison, who also came back as guest co-host in our Dev Day 2023 episode.

• 47:56Alessio Fanelli?

  Great, we're back, light of Dev Day. Returning guest, Michelle. And then returning guest co-host, Ford.

• 48:03Speaker 06

  Ford, for finish, first, yeah, I don't know.

• 48:05Speaker 09

  I've lost count. It's been a few, but.

• 48:07Alessio Fanelli?

  Simon Willison is back. Yeah, we just wrapped, we just wrapped everything up. Congrats on, on getting everything, everything live. Simon did a great live blog, so if you haven't caught up.

• 48:17Speaker 09

  I wrote my, I implemented my live blog while waiting for the first talk to start using like GPT-4, I wrote me the JavaScript. And I got that live just in time and then, yeah, I was live blogging the whole day.

• 48:28Speaker 06

  Are you a cursor enjoyer?

• 48:29Speaker 09

  I haven't really gone to cursor yet, to be honest. I, I just haven't spent enough time for it to click, I think.

• 48:34Speaker 06

  Yeah.

• 48:34Speaker 09

  I'm more a copy and paste things out to Claude and ChatGPT.

• 48:37Speaker 06

  Yeah, it's interesting. Yeah, I've, yeah, I've converted the cursor for an O1. It's so easy to just toggle on and off.

• 48:44Alessio Fanelli?

  Yeah. What's your workflow? Copy, paste?

• 48:47Speaker 10

  Okay, I'm gonna be real, I'm still VS Code Copilot, so.

• 48:50Speaker 09

  Same here, same Copilot.

• 48:52Speaker 10

  Copilot is actually the reason I joined OpenAI. It was, you know, before ChatGPT, this is the thing that really got me. So I'm still into it, but I keep meaning to try out cursor. And I think now that things have calmed down, I'm gonna give it a real go.

• 49:03Speaker 06

  Yeah, it's, it's a big thing to change your tool of choice.

• 49:06Speaker 10

  Yes, yeah, I'm pretty dialed, so.

• 49:08Speaker 06

  Yeah, I mean, you know, if you want, you can just fork VS Code and make your own. That's, that's the thing to do.

• 49:14Speaker 09

  It's the dumb thing, right?

• 49:15Speaker 06

  We joked about doing a hackathon where you, the only thing you do is fork VS Code and make the best fork win.

• 49:20Speaker 10

  Nice.

• 49:22Speaker 09

  That's actually a really good idea.

• 49:25Speaker 06

  Yeah, so I mean, congrats on launching everything today. I know like we touched on it a little bit, but like everyone was kind of guessing that Voice API was coming. And like we talked about it in our episode. How do you, how do you feel going into the, the launch? Like any design decisions that you want to highlight?

• 49:41Speaker 10

  Yeah, super jazzed about it. The team has been working on it for a while. It's like a very different API for us. It's the first WebSocket API. So a lot of different design decisions to be made. Like what kind of events do you send? When do you send an event? What are the event names? What do you send like on connection versus on future messages? So there've been a lot of interesting decisions there. The team has also hacked together really cool projects as we've been testing it. One that I really liked is we had an internal hackathon for the API team. And some folks built like a little hack that you could use Vim with Voice Mode. So like control Vim and you would tell them on a like, write a file and it would, you know, know all the Vim commands and type those in. So yeah, a lot of cool stuff we've been hacking on and really excited to see what people build with it.

• 50:23Speaker 09

  I've got to call out a demo from today. I think it was Katia had a 3D visualization of the solar system, like WebGL solar system. You could talk to. That is one of the coolest conference demos I've ever seen. That was so convincing. I really want the code. I really want the code for that to get put out there.

• 50:38Speaker 10

  I'll talk to the team, I think we can probably put it out.

• 50:40Speaker 09

  Absolutely beautiful example. And it made me realize that the real-time API, this WebSocket API, it means that building a website that you can just talk to is easy now. It's like, it's not difficult to build, spin up a web app where you have a conversation with it. It calls functions, the different things, it interacts with what's on the screen. I'm so excited about that. There were all of these projects I thought I'd never get to. And now I'm like, you know what? Spend a weekend on it. I could have a talk to your data, talk to your database with a, with a, with a little web application. That's so cool.

• 51:10Speaker 10

  Chat with PDF, but really chat with it.

• 51:12Speaker 09

  Really chat with PDF, not completely.

• 51:15Speaker 10

  Totally.

• 51:15Speaker 09

  And it's not even hard to build. That's the crazy thing about this.

• 51:18Speaker 10

  Yeah. Very cool. Yeah, when I first saw the space demo, I was actually just wowed. And I had a similar moment, I think, to all the people in the crowd. I also thought Roman's drone demo was super cool.

• 51:29Speaker 09

  That was a super fun one as well.

• 51:30Speaker 10

  Yeah, I actually saw that live this morning and I was holding my breath for sure.

• 51:35Speaker 06

  Knowing Roman, he probably spent the last two days doing, working on it.

• 51:40Speaker 06

  But yeah, like I'm curious about you were talking with Romani actually earlier about what the different levels of abstraction are with web sockets. It's something that most developers have zero experience with. I have zero experience with it. Apparently, there's like the RTC level and then there's the web socket level and there's like levels in between.

• 51:56Speaker 09

  Not so much. I mean, with web sockets, with the way they've built their API, you can connect directly to the OpenAI web socket from your browser. And it's actually just regular JavaScript, like you instantiate the web socket thing. It looks quite easy from their example code. The problem is that if you do that, you're sending your API key from like source code that anyone can use.

• 52:16Speaker 10

  Yeah, we don't recommend that for production.

• 52:18Speaker 09

  So it doesn't work for production, which is frustrating because it means that you have to build a proxy. So I'm gonna have to go home and build myself a little web socket proxy just to hide my API key. I want OpenAI to do that. I want OpenAI to solve that problem for me. So I don't have to build the 1,000th web socket proxy just for that one problem.

• 52:36Speaker 10

  We've also partnered with some partner solutions. We've partnered with, I think, Agora, LiveKit, a few others. So there's some loose solutions there, but yeah, we hear you. It's a beta.

• 52:49Speaker 06

  Yeah, yeah, I mean, you still want a solution where someone brings their own key and they can trust that you don't get it, right?

• 52:56Speaker 09

  Kind of. I mean, I've been building a lot of bring your own key apps where it's my HTML and JavaScript. I store the key in local storage in their browser and it never goes anywhere near my server, which works. But how do they trust me? How do they know I'm not gonna ship another piece of JavaScript that steals the key from the live?

• 53:12Speaker 06

  And so nominally, this actually comes with the crypto background. This is what MetaMask does?

• 53:17Speaker 10

  Yeah, it's a public-private key thing.

• 53:20Speaker 06

  Yeah, like why doesn't OpenAI do that? I don't know, obviously it's.

• 53:24Speaker 10

  I mean, as with most things, you'd think there's like some really interesting question and really interesting reason. And the answer is just, you know, it's not been the top priority and it's hard for a small team to do everything. I have been hearing a lot more about the need for things like sign in with OpenAI.

• 53:40Speaker 09

  That's what I want. Oh, I want to bounce my users through ChatGPT and I get back a token that lets me spend up to $4 on the API on their behalf. That would solve. Then I could ship all of my stupid little experiments, which currently require people to copy and paste their API key in, which cuts off everyone. Nobody knows how to do that.

• 53:57Speaker 10

  Totally. I hear you. Something we're thinking about and yeah, stay tuned.

• 54:01Speaker 06

  Yeah, yeah. Right now, the, I think the only player in town is OpenRouter. That is basically, it's funny, like it was made by, I forget his name. But he, he used to be CTO of OpenSea. And the first thing he did when he came over was build MetaMask for AI.

• 54:16Speaker 10

  Totally. Yeah, very cool.

• 54:20Alessio Fanelli?

  What's the most underrated release from today?

• 54:23Speaker 10

  Vision fine tuning. Vision fine tuning is so underrated. For the past like two months, whenever I talk to founders, they tell me this is the thing they need most. A lot of people are doing like OCR on, on very bespoke formats, like government documents. And, and vision fine tuning can help a lot with that use case. Also bounding boxes. People have found like a lot of improvements for bounding boxes with vision fine tuning. So yeah, I think it's pretty slept on. People should try it. You can, you only really need a hundred images to get going.

• 54:49Speaker 09

  Tell me more about bounding boxes. I didn't think GPT-4 Vision could do bounding boxes at all.

• 54:55Speaker 10

  Yeah, it's actually not that amazing at it. We're working on it. But with fine tuning, you can make it really good for your use case.

• 55:02Speaker 09

  That's cool, because I've been using Google Gemini's bounding box stuff recently. It's very, very impressive.

• 55:06Speaker 10

  Yeah.

• 55:07Speaker 09

  For being able to fine tune a model for that. The first thing I'm going to do with fine tuning for images is I've got five chickens.

• 55:12Speaker 10

  Yeah.

• 55:13Speaker 09

  And I'm going to tune, fine tune a model that can tell which chicken is which.

• 55:15Speaker 10

  Love it.

• 55:16Speaker 09

  Which is hard because three of them are gray.

• 55:18Speaker 10

  Yeah.

• 55:18Speaker 09

  So there's, there's a little bit of, of.

• 55:20Speaker 10

  Okay, this is my new favorite use case. This is awesome.

• 55:22Speaker 09

  Yeah, it's, I've, I've, I've managed to do it with, with prompting, just like I gave Claude pictures of all of the chickens. And then said, okay, which chicken is this?

• 55:30Speaker 10

  Yeah.

• 55:30Speaker 09

  But it's not quite good enough because it confuses the, the gray chicken.

• 55:33Speaker 10

  Listen, we can close that eval gap.

• 55:35Speaker 09

  Yeah, that's, it's going to be a great eval. My, my chicken eval is going to be fantastic.

• 55:39Speaker 10

  I'm also really jazzed about the evals product. Um, it's kind of like a sub-launch of the distillation thing. But people have been struggling to make evals. And the first time I saw the flow with how easy it is to make an eval in our product, I was just blown away. So recommend people really try that. I think that's what's holding a lot of people back from really investing in AI. Because they, they just have a hard time figuring out if it's going well for their use case. So we've been working on making it easier to do that.

• 56:03Alessio Fanelli?

  Does the eval product include structured output testing, like function calling and things?

• 56:08Speaker 10

  If it matches your JSON schema, yeah.

• 56:12Speaker 06

  I mean, we have guaranteed structured output anyway, right?

• 56:16Alessio Fanelli?

  Well, but.

• 56:17Speaker 06

  So we don't have to test it.

• 56:18Alessio Fanelli?

  Well, not the schema, but like the.

• 56:20Speaker 10

  See, these seem easy to tell.

• 56:21Alessio Fanelli?

  Performance.

• 56:21Speaker 09

  I think so.

• 56:22Alessio Fanelli?

  It's like you might call the wrong function or.

• 56:24Speaker 06

  Oh, I see.

• 56:25Alessio Fanelli?

  You're gonna have right schema, wrong output.

• 56:27Speaker 06

  So you can do function calling testing.

• 56:29Speaker 10

  I'm pretty sure. I'll have to check that for you, but I think so.

• 56:31Alessio Fanelli?

  Well, we'll make sure it's. How do you think about the evolution of like the API design? I think to me that's like the most important thing. So even with the OpenAI levels, like chatbots, I can understand what the API design looks like. Reasoning, I can kind of understand it, even though like chain of talk kind of changes things. As you think about real time voice and then you think about agents, it's like, how do you think about? How you design the API and like what the shape of it is?

• 56:58Speaker 10

  Yeah, so I think we're starting with the lowest level capabilities. And then we build on top of that as we know that they're useful. So a really good example of this is real time. We're actually going to be shipping audio capabilities in chat completions. So this is like the lowest level capability. So you supply in audio and you can get back raw audio and it works at the request response layer. But in through building advanced voice mode, we realized ourselves that like it's pretty hard to do with something like chat completions. And so that led us to building this WebSocket API. So we really learned a lot from our own tools and we think, you know, the chat completion thing is nice and for certain use cases or async stuff. But you're really gonna want a real time API. And then as we, you know, test more with developers, we might see that it makes sense. It makes sense to have like another layer of abstraction on top of that.

• 57:44Speaker 10

  Something like closer to, you know, more client side levers. But for now, you know, that's where we feel we have like a really good point of view.

• 57:52Speaker 09

  So that's a question I have is if I've got a half hour long audio recording.

• 57:56Speaker 10

  Yeah.

• 57:56Speaker 09

  At the moment, the only way I can feed that in is if I call the WebSocket API and slice it up into little JSON basics for snippets and fire them all over. In that case, I'd rather just give you a, like an image in the chat completion API, give you a URL to my MP3 files and input. Is that something?

• 58:11Speaker 10

  That's what we're gonna do.

• 58:12Speaker 09

  Oh, thank goodness for that.

• 58:12Speaker 10

  Yes, it's in the blog post. I think it's a short one liner, but it's rolling out, I think in the coming weeks.

• 58:17Speaker 09

  Oh, wow. Oh, really soon then?

• 58:19Speaker 10

  Yeah, team has been sprinting. We're just putting finishing touches on stuff.

• 58:21Speaker 09

  Do you have a feel for the length limit on that?

• 58:24Speaker 10

  I don't have it off the top.

• 58:25Speaker 09

  Okay. Because yeah, often I want to do, I do a lot of work with like transcripts of hour-long YouTube videos, which currently I run them through Whisper and then I do the transcript that way. But being able to do the multimodal thing with those would be really useful.

• 58:37Speaker 10

  Totally, yeah. We really jazz about it. We want to basically give the lowest capabilities we have, lowest level capabilities, and you know, the things that make it easier to use. And so, you know, targeting kind of both.

• 58:50Speaker 09

  I just realized what I can do though is, I do a lot of Unix utilities, little like Unix things.

• 58:55Speaker 10

  Yeah.

• 58:55Speaker 09

  I want to be able to type the output of a command into something which streams that up to the WebSocket API and then speaks it out loud.

• 59:03Speaker 10

  Yeah.

• 59:03Speaker 09

  So I can do streaming speech of the output of things. That should work. Like, I think you've given me everything I need for that. That's cool.

• 59:10Speaker 10

  Yeah. Excited to see what you build.

• 59:14Speaker 06

  Is there, I heard there are like multiple competing solutions and you guys evaluated before you picked WebSockets. Like server set events, pulling, I don't, like, can you give like your thoughts on like the live updating paradigms that you guys looked at? Because I think a lot of engineers have looked at stuff like this.

• 59:34Speaker 10

  Well, I think WebSockets are just a natural fit for bidirectional streaming. You know, other places I've worked, like Coinbase, we had a WebSocket API for pricing data. And I think it's just like a very natural format.

• 59:46Speaker 06

  So it wasn't even really that controversial at all?

• 59:49Speaker 10

  I don't think it was super controversial. I mean, we definitely explored the space a little bit, but I think we came to WebSockets pretty quickly. Yeah.

• 59:56Speaker 06

  Cool. Video?

• 59:58Speaker 10

  Yeah. Not yet, but you know, possible in the future.

• 1:00:03Speaker 06

  I actually was hoping for the ChatGPT desktop app with video today, because that was demoed.

• 1:00:08Speaker 10

  Yeah. This is dev day.

• 1:00:11Speaker 09

  I think the moment we have the ability to send images over the WebSocket API, we get video. Just send the frame, one frame a second.

• 1:00:18Speaker 06

  How frequently? Yeah. Because...

• 1:00:21Speaker 06

  Yeah, I mean, sending a whole video frame of like a 1080p screen, maybe it might be too much. What's the limitations on a WebSocket chunk going over? I don't know.

• 1:00:33Speaker 10

  I don't have that off the top.

• 1:00:34Speaker 09

  Like Google Gemini, you can do an hour's worth of video in their context window and just by slicing it up into 10 frames a second. And it does work. So, I don't know. But then, that's the weird thing about Gemini is it's so good at you just giving it a flood of individual frames. It'll be interesting to see if GPT-4O can handle that or not.

• 1:00:55Alessio Fanelli?

  Do you have any more feature requests? I know it's been a long day for everybody, but you got Michelle right here.

• 1:01:02Speaker 09

  My one is, I want you to do all of the accounting for me. I want my users to be able to run my apps. And I want them to call your APIs with their user ID and have you go, "Oh, they've spent 30 cents." Cut them off at a dollar. I can like check how much they spent. All of that stuff, because I'm having to build that at the moment and I really don't want to. I don't want to be a token accountant. I want you to do the token accounting for me.

• 1:01:26Speaker 10

  Yeah, totally. I hear you. It's good feedback.

• 1:01:29Speaker 06

  Well, like, how does that contrast with your actual priorities, right? Like, I feel like you have a bunch of priorities. They showed some on stage with multimodality and all that.

• 1:01:39Speaker 10

  Yeah. It's hard to say. I would say things change really quickly. Things that are big blockers for user adoption, we find very important. And, yeah. It's a rolling prioritization.

• 1:01:54Speaker 06

  No Assistance API update?

• 1:01:56Speaker 10

  Not at this time. Yeah.

• 1:01:59Speaker 06

  I was hoping for like an 01 native thing in Assistance. Because I thought they would go well together.

• 1:02:06Speaker 10

  We're still kind of iterating on the formats. I think there are some problems with the Assistance API. Some things it does really well. And I think we'll keep iterating and land on something really good, but just, you know, it wasn't quite ready yet. Some of the things that are good in the Assistance API is hosted tools. People really like hosted tools and especially RAG. And then some things that are, you know, less intuitive is just how many API requests you need to get going with the assistants API.

• 1:02:30Speaker 09

  It's quite.

• 1:02:31Speaker 10

  It's quite a lot. Yeah, you gotta create an assistant, you gotta create a thread, you gotta, you know, do all this stuff. So yeah, it's something we're thinking about. It shouldn't be so hard.

• 1:02:39Speaker 09

  The only thing I've used it for so far is code interpreter. It's like it's an API to code interpreter. Crazy exciting.

• 1:02:44Speaker 10

  Yes, we want to fix, we want to fix that and make it easier to use.

• 1:02:48Speaker 09

  I want code interpreter over web sockets. That would be wildly interesting.

• 1:02:53Speaker 06

  Yeah. Do you, do you want to bring your own code interpreter or you want to use OpenAI's?

• 1:02:57Speaker 09

  I want to use theirs because code interpreter is a hard problem. Sandboxing and all of that stuff is.

• 1:03:01Speaker 06

  Yeah, but there's a bunch of code interpreter as a service things out there.

• 1:03:04Speaker 09

  There are a few now, yeah.

• 1:03:05Speaker 06

  Because there's, I think you don't allow arbitrary installation of packages.

• 1:03:09Speaker 09

  Oh, they do. They really do.

• 1:03:11Speaker 10

  And actually uploaded.

• 1:03:12Speaker 09

  Yeah, and I do.

• 1:03:14Speaker 10

  Yeah. You can upload a PIT package.

• 1:03:16Speaker 09

  You can run, you can compile C code in code interpreter.

• 1:03:19Speaker 06

  I know, that's a hack, that's a hack.

• 1:03:21Speaker 09

  Oh, it's such a glorious hack, though.

• 1:03:24Speaker 06

  Okay.

• 1:03:24Speaker 09

  I've had it write me custom SQLite extensions in C and compile them and run them inside of Python and it works.

• 1:03:31Speaker 06

  I mean, yeah, there's, there's others. E2B is one of them. Like, yeah, it'll be, it'll be interesting to see what the real time version of that would be. Yeah.

• 1:03:40Alessio Fanelli?

  Awesome, Michelle. Thank you for the update. We left the episode as what would voice mode look like? Obviously you knew what it looked like, but you couldn't say it. So now you could.

• 1:03:49Speaker 10

  Yeah, here we are. Hope you guys play with it.

• 1:03:52Speaker 06

  Yeah. Cool. Awesome. Thank you.

• 1:03:54Alessio Fanelli?

  That's it.

• 1:03:54Speaker 00

  Our final guest today, and also a familiar recent voice on the Latent Space Pod, presented at one of the community talks at this year's Dev Day.

• 1:04:04Speaker 00

  Alistair Pullen of Coaseen made a huge impression with all of you. Special shout out to listeners like Jesse from Morph Labs when he came on to talk about how he created synthetic data sets to fine tune the largest LoRAs that had ever been created for GPT-4O to post the highest ever scores on Sui Bench and Sui Bench Verified while not getting recognition for it because he refused to disclose his reasoning traces to the Sui Bench team. Now that OpenAI's O1 preview is announced, it is incredible to see the OpenAI team also obscure their chain of thought traces for competitive reasons and still perform lower than Cozine's Genie model. We snagged some time with Ali to break down what has happened since his episode aired.

• 1:04:50Speaker 06

  Welcome back, Ali.

• 1:04:50Speaker 11

  Thank you so much. Thanks for having me.

• 1:04:52Speaker 06

  So you just spoke at OpenAI Dev Day. What was the experience like? Did they reach out to you? You seem to have a very close relationship.

• 1:04:59Speaker 11

  Yeah, so off the back of the work that we've done, that we spoke about last time we saw each other, I think that OpenAI definitely felt that the work we've been doing around fine tuning was worth sharing. I would obviously tend to agree, but today I spoke about some of the techniques that we learned. Obviously, it was like a nonlinear path, arriving to where we've arrived in the techniques that we built to build Genie. So I think I shared a few extra pieces about some of the techniques and how it really works under the hood, how you generate a dataset to show the model how to do what we show the model. And that was mainly what I spoke about today. I mean, yeah, they reached out. I was super excited at the opportunity, obviously. Like, it's not every day that you get to come and do this, especially in San Francisco. So, yeah, they reached out and they were like, do you want to talk at Dev Day? You can speak about basically anything you want related to what you've built.

• 1:05:44Speaker 11

  And I was like, sure, that's amazing. I'll talk about fine tuning, how you build a model that does this software engineering. So, yeah.

• 1:05:50Speaker 06

  Yeah. And the trick here is when we talked, O1 was not out.

• 1:05:54Speaker 11

  No, it wasn't.

• 1:05:55Speaker 06

  Did you know about O1 or?

• 1:05:57Speaker 11

  I didn't know. I knew some bits and pieces. No, not really. I knew a reasoning model was on the way. I didn't know what it was going to be called. I knew as much as everyone else, Strawberry was the name back then.

• 1:06:07Speaker 06

  Because, you know, I'll fast forward. You were the first to hide your reasoning traces as IP.

• 1:06:14Speaker 11

  Yes.

• 1:06:15Speaker 06

  Right? Famously, that got you in trouble with SweeBench or whatever.

• 1:06:17Speaker 11

  Yes. I feel slightly vindicated by that now.

• 1:06:21Speaker 06

  And now, obviously, O1 is doing it.

• 1:06:22Speaker 11

  Yeah, the fact that, yeah, I mean, like, I think it's true to say right now that the reasoning of your model gives you the edge that you have.

• 1:06:33Speaker 11

  And like, the amount of effort that we put into our data pipeline to generate these human-like reasoning traces was, I mean, that wasn't for nothing, that we knew that this was the way that you'd unlock more performance, getting the model to think in a specific way. In our case, we wanted it to think like a software engineer. But, yeah, I think that...

• 1:06:50Speaker 11

  The approach that other people have taken like OpenAI in terms of reasoning has definitely showed us that we were going down the right path pretty early on. And even now, we've started replacing some of the reasoning traces in our genie model with reasoning traces generated by O1, or at least in tandem with O1. And we've already started seeing improvements in performance from that point. But no, like back to your point, in terms of like the whole like withholding them, I still think that that was the right decision to do because of the very reason that everyone else has decided to not share those things. It's, it is exactly, it shows exactly how we do what we do, and that is our edge at the moment. So, yeah.

• 1:07:33Alessio Fanelli?

  As a founder, so they also feature recognition on stage, talk about them. How does that make you feel that like, you know, they're like, hey, O1 is so much better, makes us better. For you, it should be like, oh, I'm so excited about it too, because now all of a sudden, it's like, it kind of like raises the floor for everybody. Like, how should people, especially new founders, how should they think about, you know, worrying about the new model versus like being excited about them just focusing on like the core FP and maybe switching out some of the parts like you mentioned?

• 1:08:00Speaker 11

  Yeah, I, speaking for us, I mean, obviously like, we were extremely excited about O1 because at that point, the process of reasoning is obviously very much baked into the model. We fundamentally, if you like remove all distractions and everything, we are a reasoning company, right? We want to reason in the way that a software engineer reasons. So when I saw that model announced, I thought immediately, well, I can improve the quality of my traces coming out of my pipeline. So like my signal to noise ratio gets better. And then, not immediately, but down the line, I'm going to be able to train those traces into O1 itself. So I'm going to get even more performance that way as well. So it's for us, a really nice position to be in, to be able to take advantage of it, both on the prompted side and the fine tuned side.

• 1:08:42Speaker 11

  And also because fundamentally, like, we are, I think fairly clearly in a position now where we don't have to worry about what happens when O2 comes out, what happens when O3 comes out. This process continues. Like even going from, you know, when we first started going from 3.5 to 4, we saw this happen. And then from 4 Turbo to 4O, and then from 4O to O1, we've seen the performance get better every time. And I think, I mean, like, the crude advice I'd give to any startup founder is try to put yourself in a position where you can take advantage of the same, you know, like sea level rise every time essentially.

• 1:09:15Speaker 06

  Do you make anything out of the fact that you were able to take 4-0 and fine tune it higher than 0-1 currently scores on SweeBench verified?

• 1:09:23Speaker 11

  Yeah, I mean, like, that was obviously, to be honest with you, you realize that before I did.

• 1:09:29Speaker 06

  Adding value.

• 1:09:31Speaker 11

  Yes, absolutely. That's a value-add investor right there. No, obviously, I think it's been, that in of itself is really vindicating to see because I think, I think we have heard from some people, not a lot of people, but some people saying, "Well, okay, well, if 0-1 can reason, then what's the point of doing your reasoning?" But it shows how much more signal is in like the custom reasoning that we generate. And again, it's the, it's the very sort of obvious thing. If you take something that's made to be general and you make it specific, of course, it's going to be better at that thing, right? So it was obviously great to see, like, we still are better than 0-1 out of the box, you know, even with an older model, and I'm sure that that, that delta will continue to grow once we're able to train 0-1 and once we've done more work on our dataset using 0-1, like, that delta will grow as well.

• 1:10:13Speaker 06

  It's not obvious to me that they will allow you to fine tune 0-1, but, you know, maybe they'll try. I think the, the core question that OpenAI really doesn't want you to figure out is can you use an open source model and beat 0-1?

• 1:10:28Speaker 11

  Interesting.

• 1:10:29Speaker 06

  Because, because you basically have shown proof of concept that a non-0-1 model can beat 0-1. And their whole 0-1 marketing is, "Don't bother trying." Like, "Don't bother stitching together multiple chain of thought calls. We did something special, secret sauce, you don't know anything about it." And somehow, you know, your 4-0 chain of thought reasoning as a software engineer is still, still better. Maybe it doesn't last. Maybe they're gonna run 0-1 for five hours instead of five minutes, and then, and then suddenly it works.

• 1:10:58Speaker 11

  Yeah.

• 1:10:58Speaker 06

  So I, I don't know.

• 1:10:59Speaker 11

  It's hard to know. I mean, one of the things that we just want to do out of sheer curiosity is do something like fine tune 4-0 5B on the same dataset. Like, same context window length, right? So it should be fairly, fairly easy. We haven't done it yet. Truthfully, we have been so swamped with the waitlist, shipping product, you know, dev day, like, you know, onboarding customers from our waitlist. All these different things have gotten in the way, but it is definitely something out of more curiosity than anything else I'd like to try out, but also it opens up a new vector of like, if someone has a VPC where they can't deploy an OpenAI model, but they might be able to deploy an open source model, it opens that us, it opens that up for us as well from a customer perspective, so it'll probably be quite useful. I'd be very keen to see what the results are there.

• 1:11:38Speaker 06

  I suspect the answer is yes, but it may be hard to do. So Reflection 70B was a really crappy attempt at doing it. You guys were much better, and that's why we had you on the show. Yeah, I'm interested to see. Open O1, basically. People want Open O1.

• 1:11:53Speaker 11

  Yeah, I'm sure they do. As soon as we do it and like, once we've wrapped up what we're doing in San Francisco, I'm sure we'll give it a go. I spoke to some guys today actually about fine tuning 405B, who might be able to allow us to do it very easily. I don't want to have to basically do all the stuff myself. So yeah, that might happen sooner rather than later. Yeah.

• 1:12:14Alessio Fanelli?

  Anything from the releases today that you're super excited about? So prompt caching, I'm guessing when you're dealing with a lot of code bases, that might be helpful. Is there anything with vision fine tuning related to more like UI related development?

• 1:12:27Speaker 11

  Definitely. Yeah, I mean, we were talking, it's funny, like, my co-founder Sam, who you've met, and I were talking about the idea of doing vision fine tuning, like way back, like well over a year ago, before Genie existed as it does now. When we collected our original dataset to do what we do now, whenever there were image links and links to like graphical resources and stuff, we also pulled that in as well. We never had the opportunity to use it. But it's something we have in storage. And again, like when we have the time, it's something that I'm super excited, particularly on the UI side, to be able to leverage. Particularly if you think about, one of the things, not to sidetrack, but one of the things we've noticed is, I know Sweetbench is like the most commonly talked about thing, and honestly, it's an amazing project.

• 1:13:09Speaker 11

  But one of the things we've learned the most from actually shipping this product to users is, it's a pretty bad proxy at telling us how competent the model is. So for example, when people are doing like React development using Genie, for us, it's impossible to know whether what it's written is actually done, you know, done what it wanted to. So at least even using like the fine tuning for vision to be able to help eval, like what we output is already something that's very useful. But also in terms of being able to pair, here's a UI I want, here's the code that actually like represents that UI, is also going to be super useful as well, I think. In terms of generally, what have I been most impressed by? The distillation thing is awesome. I think we'll probably end up using it in places.

• 1:13:51Speaker 11

  But what it shows me more broadly about OpenAI's approach is they're going to be building a lot of the things that we've had to hack together internally in terms from a tooling point of view, just to make our lives so much easier. And I've spoken to, you know, John, the head of fine tuning extensively about this. But there's a bunch of tools that we've had to build internally for things like dealing with model lineage, dealing with dataset lineage, because it gets so messy so quickly, that we would love OpenAI to build, like absolutely would love them to build. It's not, it's not what gives us our edge, but it certainly means that then we don't have to build it and maintain it afterwards. So it's a really good first step, I think, in like the overall maturity of the fine tuning product and API in terms of where they're going to see those early products.

• 1:14:34Speaker 11

  And I think that they'll be continuing in that direction going on.

• 1:14:37Speaker 06

  Did you not, so there's a very active ecosystem of LLM ops tools. Did you not evaluate those before building your own?

• 1:14:47Speaker 11

  We did, but I think fundamentally like.

• 1:14:52Speaker 06

  No moat.

• 1:14:54Speaker 11

  Yeah, like I think in a lot of places, it was, it was never a big enough pain point to be like, oh, we absolutely must outsource this. It's definitely in many places, something that you can hack a script together in a day or two and then hook, hook it up to our already existing internal tool UI. And then you have, you know, what you need. And whenever you need a new thing, you just tack it on. But for like all of these LLM ops tools, I've never felt the pain point enough to really like bother. And, and, and that's no, that's not to deride them at all. I'm sure many people find them useful, but just for us as a company, we've never, we've never felt the need for them. So it's great that, it's great that OpenAI are going to build them in because it's really nice to have them there for sure.

• 1:15:36Speaker 11

  But it's not something that like I'd ever consider really paying for externally or something like that, if that makes sense.

• 1:15:40Speaker 06

  Yeah. Does voice mode factor into Genie?

• 1:15:44Speaker 11

  Maybe one day they'll be sick, wouldn't it? Like to be able to, yeah, I think so.

• 1:15:47Speaker 06

  You're the first person, we've been asking this question to everybody.

• 1:15:50Speaker 11

  Yeah, I think.

• 1:15:51Speaker 06

  You're the first person to not mention voice mode.

• 1:15:52Speaker 11

  Oh, well, it's currently so distant from what we do. But I definitely think like this whole talk of we want it to be a full on AI software engineering colleague. Like, there is definitely a vector in some way that you can build that in. Maybe even during the ideation stage, talking through a problem with Genius in terms of how we want to build something down the line. I think that might be useful, but honestly, like that would be a nice to have when we have the time. Yeah.

• 1:16:18Speaker 06

  Yeah, amazing. One last question on your, in your talk, you mentioned a lot about curating your data and your distribution and all that.

• 1:16:26Speaker 11

  Yes.

• 1:16:26Speaker 06

  And before we sat down, you talked a little bit about having to diversify.

• 1:16:29Speaker 11

  Absolutely.

• 1:16:30Speaker 06

  Your data set.

• 1:16:31Speaker 11

  Yeah.

• 1:16:32Speaker 06

  What's driving that? What are you finding?

• 1:16:33Speaker 11

  So we have been rolling people off the waitlist that we sort of amassed when we announced when I last saw you. And it's been really interesting because as I may have mentioned on the podcast, like we had to be very opinionated about the data mix and the data set that we put together for like sort of the V zero of genie. Again, like to your point, JavaScript, JavaScript, JavaScript, Python, right? There's a lot of JavaScript in its various forms in there. But it turns out that when we've shipped it to the, to the very early alpha users, we've rolled it out to. For example, we had some guys using it with a C sharp code base. And C sharp currently represents, I think, about 3% of the overall data mix. And they weren't getting the levels of performance that they saw when they tried it with a Python code base.

• 1:17:15Speaker 11

  And it was obviously like not great for them to have a bad experience, but it was nice to be able to correlate it with the, the actual like objective data mix that we saw. So we did, what we've been doing is like little top up fine tunes where we take like the general genie model and do an incremental fine tune on top. With just a bit more data for a given, you know, vertical language. And we've been seeing improvements coming from that. So again, this is one of the great things about sort of baptism by fire and letting people use it and giving you feedback and telling you where it sucks. Because that is not something that we could have just known ahead of time. So I want that data mix to, over time, as we roll it out to more and more people, and we are trying to do that as fast as possible.

• 1:17:54Speaker 11

  We're still a team of five for the time being and to be as general and as representative of what our users do as possible and not what we think they need.

• 1:18:02Speaker 06

  Yeah, so every customer is going to have their own fine tune.

• 1:18:07Speaker 11

  There is going to be the option to, yeah, there is going to be option to fine tune the model on your code base. It won't be in like the base pricing tier, but you will definitely be able to do that. It will go through all of your code base history, learn how everything happened, and then you will have an incrementally fine tuned genie just on your code base. That's what enterprises really love the idea of.

• 1:18:25Speaker 06

  Lovely. Perfect. Anything else? Yeah, that's it. Amazing so much.

• 1:18:29Speaker 11

  Thank you so much, guys. Good to see you.

• 1:18:31Speaker 06

  Thank you.

• 1:18:31Speaker 00

  Lastly, this year's Dev Day ended with an extended Q and A with Sam Altman and Kevin Weil. We think both the questions asked and answers given were particularly insightful, so we are posting what we could snag of the audio here from publicly available sources, credited in the show notes, for you to pick through. If the poorer quality audio here is a problem, we recommend waiting for approximately one to two months until the final video is released on YouTube. In the meantime, we particularly recommend Sam's answers on the moderation policy, on the underappreciated importance of agents and AI employees beyond level three, and

• 1:19:11Speaker 00

  his projections of the intelligence of O1, O2, and O3 models in future.

• 1:19:23Speaker 12

  All right, I think everybody knows you. For those who don't know me, I'm Kevin Weil, Chief Product Officer at OpenAI. I have the good fortune of getting to turn the amazing research that our research teams do into the products that you all use every day and the APIs that you all build on every day. I thought we'd start with some audience engagement here. So on the count of three, I want to count to three, and I want you all to say, of all the things that you saw launched here today, what's the first thing you're going to integrate? Something you're most excited to build on, all right? You gotta do it, right? One, two, three.

• 1:19:58Speaker 13

  Realize that API!

• 1:20:03Speaker 12

  I'll say it personally, I'm super excited about our distillation products. I think that's going to be really important.

• 1:20:14Speaker 12

  I'm also excited to see what you all do with advanced voice mode with the real-time API and with vision fine-tuning in particular. So, okay. So, I've got some questions for Sam. I've got my CEO here in the hot seat. Let's see if I can't make a career of any move.

• 1:20:30Speaker 12

  So, we'll start this, we'll start with an easy one, Sam. How close are we to API?

• 1:20:37Speaker 14

  You know, we, we used to, every time we finished a system, we would say like, "In one way, this is not an API." And it used to be like, "Very easy, you can like make a whole robotic hand that does physics for a Dota bot." And it's like, "Oh, it does some things, but definitely not an API." It's obviously harder to say now. And so, we, we're trying to like stop talking about AGIs as this general thing. We have this levels framework because the word AGI has become so overlooked. So, like, real quickly, we use one for chatbots, two for reasoners, three for agents, four for innovators, five for organizations, like roughly. I think we clearly got to level two, or we believe we clearly got to level two with O1.

• 1:21:19Speaker 14

  And it, you know, can do really quite impressive cognitive attacks. It's a very smart model. It doesn't feel AGI-like in a few important ways. But I think if you just do the one next step of making it, you know, very agent-like, which is our level three, and which I think we will be able to do in the not distant future, it will feel surprisingly capable. Still, probably not something that most of you would call an AGI, although maybe some of you would. But it's gonna feel like, "All right, this is, this is like a significant thing." And then, the, the leap, and I think we do that quickly. The, the leap from that to something that can really increase the rate of new scientific

• 1:22:02Speaker 14

  discovery, which for me is like a very important part of having an AGI. I feel a little bit less certain on that, but not a long time. Like, I think all of this now is gonna happen pretty quickly. If you think about what happened from last decade to this one, in terms of model capabilities, and you're like, eh, I mean, if you go look at like, if you go from like O1 on a hard problem back to like 4 Turbo that we launched 11 months ago, you'll be like, wow, this is happening pretty fast. And I think the next year will be very steep progress, next two years will be very steep progress. Harder than that, hard to assume a lot of certainty, but I would say like, not very. And at this point, the definitions really matter. And in fact, the fact that the definitions matter this much somehow means we're like, getting kind of close.

• 1:22:44Speaker 12

  Yeah. And, you know, there, there used to be this sense of AGI where it was like, it was a binary thing and you were gonna go to sleep one day, there was no AGI, and wake up the next day and there was AGI. I don't think that's exactly how we think about it anymore, but how have your views evolved?

• 1:23:01Speaker 14

  You know, the, the one, I agree with that, I think we're like, you know, in this like, kind of, period where it's gonna feel very blurry for a while and the, you know, is this AGI yet, or is this not AGI, or kind of like, at what point, yeah, it's just gonna be this like, smooth exponential and, you know, probably, most people, looking back at history, won't agree like, when that milestone was hit, and will just realize it was like, a silly thing. Even the Channing Test, which I thought always was like, this very clear milestone, you know, there was this like, fuzzy period, it kind of like, went pushing by, no one cared. But, but I, I think the right framework is just this one exponential. That's it.

• 1:23:42Speaker 14

  If we can make an AI system that is like, materially better at all of OpenAI than doing, at doing AI research, that does feel to me like some sort of important discontinuity. It's probably still wrong to think about it that way, it probably still is the smooth exponential curve, but that feels like a milestone.

• 1:24:03Speaker 12

  Is OpenAI still as committed to research as it was in the early days? Will research still drive core of our advancements and our product development?

• 1:24:12Speaker 14

  Yeah, I mean, I think more than ever, the, there was like a time in our history when the right thing to do was just to scale up compute. And we saw that with conviction and we had a spirit of like, we'll do whatever works. You know, like, we want to, we have this mission, we want to like build, say, AGI, figure out sharing benefits. If the answer is like wrap up GPUs, we'll do that. And right now, the answer is again, really push on research. And I think we see this with O1, like that is a giant research breakthrough that we were attacking from many vectors over a long period of time that came together. It's really powerful way. We have many more giant research breakthroughs to come. But the thing that I think is most special about OpenAI

• 1:24:57Speaker 14

  is that we deeply care about research and we understand how to, I think, it's easy to copy something you know works. And, you know, I actually don't even mean as that, I think like when people copy OpenAI, I'm like, great, the world gets more AI, that's wonderful. But to do something new for the first time, to like really do research in the true sense of it, which is not like, you know, let's barely get soda this thing or like let's tweak this. But like, let's go find the new paradigm and the one after that and the one after that. That is what motivates us. And I think the thing that is special about us as an org, besides the fact that we, you know, married product and research and all this other stuff together, is that we know how to run that kind of a culture that can go, that can go push back the frontier.

• 1:25:42Speaker 14

  And that's really hard. But we love it and that's, you know, I think we only have to do that a few more times and then we can do AGI.

• 1:25:49Speaker 12

  Yeah, I'll say like the, the litmus test for me coming from the outside, from, you know, sort of normal tech companies, of how critical research is to OpenAI, is that building product at OpenAI is fundamentally different than any other place that I have ever done it before. And normally you have, you have some sense of your, your tech stack, you have some sense of what you have to work with and what capabilities computers have. And, and then you're trying to build the best product, right? You're figuring out who your users are and what problems they have and how you can help solve those problems for them. There is that at OpenAI, but also the state of like what computers can do just evolves

• 1:26:32Speaker 12

  every two months, three months, and suddenly computers have a new capability that they've never had in the history of the world. And we're trying to figure out how to build a great product and expose that for developers and our APIs and so on. And you know, you can't totally tell what's coming. They're coming through, it's coming through the mist a little bit at you and gradually taking shape. It's fundamentally different than any other company I've ever worked at. And it's, I think.

• 1:26:57Speaker 14

  Is that, is that the thing that is most surprising?

• 1:26:59Speaker 12

  Yes. Yeah, and it's interesting how even internally, we don't always have a sense. You have like, okay, I think this capability is coming, but is it going to be, you know, 90% accurate or 99% accurate in the next model? Because the difference really changes what kind of product you can build. And you know that you're going to get to 99, you don't quite know when, and figuring out how you put a roadmap together in that world is really interesting.

• 1:27:26Speaker 14

  Yeah, the, the degree to which we have to just like follow the science and let that determine what we go work on next and what products we build and everything else is, I think, hard to get across. Like, we have guesses about where things are going to go. Sometimes we're right, often we're not. But if something starts working or if something doesn't work that you thought was going to work, the, our willingness to just say, we're going to like pivot everything and do what the science allows, and you don't get to like pick what the science allows.

• 1:27:54Speaker 12

  Yeah.

• 1:27:55Speaker 14

  That's surprising.

• 1:27:55Speaker 12

  I was sitting with an enterprise customer a couple of weeks ago, and they said, you know, one of the things we really want, this is all working great, we love this, one of the things we really want is a notification 60 days in advance when you're going to watch something. And I was like, I want that too.

• 1:28:13Speaker 12

  Uh, all right, so I'm, I'm going through, these are a bunch of questions, uh, from the audience, by the way, and we're going to try and also leave some time at the end for people to ask audience questions. Uh, so we've got some folks with mics and when we get there, they'll be thinking. But next thing is, so many in the alignment community are genuinely concerned that OpenAI is now only paying lip service to the, to alignment. Can you reassure us?

• 1:28:35Speaker 14

  Yeah, I think it's true, we have a different take on alignment than like maybe what people write about on whatever that like internet on us. But we really do care a lot about building safety. Systems, we have an approach to do it that has been informed by our experience so far. And touch on that question, which is you don't get to pick where the science goes. Of, we want to figure out how to make capable models that get safer and safer over time. And, you know, a couple of years ago, we didn't think the whole strawberry or the O one paradigm was gonna work in the way that it's worked. And that brought a whole new set of safety challenges, but also safety opportunities.

• 1:29:18Speaker 14

  And rather than kind of like plan to make theoretical, once, you know, superintelligence get here, gets here, here's the like seventeen principles. We have an approach of figure out where the capabilities are going and then work to make that system safe. And O one is obviously our most capable model ever, but it's also our most online model ever by a lot. And as, as these models get better, intelligence, better reasoning, whatever you wanna call it, the things that we can do to align them, things we can do to build really safe systems across the entire staff,

• 1:29:58Speaker 14

  our tool set is increasing as well. So, we, we have to build models that are generally accepted as safe and robust to be able to build them in the world. And when we started OpenAI, what the picture of alignment looked like and what we thought the problems that we needed to solve were going to be, turned out to be nothing like the problems that actually are in front of us and that we had to solve now. And also, when we made the first G P T three, if you asked me for the techniques that would have worked for us to be able to now deploy our current systems as generally accepted to be safe and robust, they would not have been the ones that turned out to work. So, by this idea of iterative deployment, which I think has been one of our most important

• 1:30:41Speaker 14

  safety standards ever, answer the confronting reality is it's in front of us. We've made a lot of progress and we expect to make more. We keep finding new problems to solve, but we also keep finding new techniques to solve them. All of that said, I think worrying about the sci-fi ways this all goes wrong is also very important. We have people thinking about that. It's a little bit less clear kind of what to do there and sometimes you end up backtracking a lot. But I don't think it's, I also don't think it's fair to say we're only gonna work on the thing in front of us. We do have to think about where this is going and we do that too. And I think if we keep approaching the problem from both ends like that, most of our thrust on the like, okay, here's the next thing, we're gonna deploy this, what needs to happen to get there?

• 1:31:24Speaker 14

  But also like, what happens if this curve just keeps going? That's been, that's been an effective strategy for us.

• 1:31:30Speaker 12

  I'll say also, it's one of the places where I'm really, I really like our philosophy of iterative deployment. When I was at Twitter, back, I don't know, a hundred years ago now, Ev said something that stuck with me, which is no matter how many smart people you have inside your walls, there are way more smart people outside your walls. And so when we try and get our, you know, it'd be one thing if we just said, we're gonna try and figure out everything that could possibly go wrong within our walls. And we'd be just us and the red teamers that we can hire and so on. And we do that, we work really hard at that. But also launching iteratively and launching carefully and learning from the ways that, that folks like you all use it, what can go right, what can go wrong, I think is a big way that we get these things right.

• 1:32:13Speaker 14

  I also think that as we head into this world of agents off the beaten track, the world, that is gonna become really, really important. As these systems get more complex and are acting on longer horizons, then pressure testing from a whole outside world, I, really, really, really critical.

• 1:32:32Speaker 12

  Yeah, so we'll go actually, we'll go off of that and maybe talk to us a bit more about how you see agents fitting in, middle of the eyes, long-term plans. What are you using? I think not a huge part of the, the. I think the exciting thing is this, this set of models, O1 in particular, and all of its successors are going to be what makes this possible. Because you finally have the ability to reason, to take hard problems, break them into simpler problems and act on them. I mean, I think 2025 is gonna be the year that's really hit this bit.

• 1:33:06Speaker 14

  Yeah, I I mean chat interfaces are great and they will and have an important place in the world. But the when you can like ask a model, when you can ask like chat G P T or some agent something. And it's not just like you get a kind of quick response or you think you get like fifteen seconds of thinking. And no one gives you like a nice piece of feedback or whatever. But you can like really give something a multi-turbo interaction with environments or other people or whatever. Like think for the equivalent of multiple days of human effort. And and like a really smart, really capable human and like have stuff happen. We all say that we're all like, oh, yeah, we can do the next thing.

• 1:33:48Speaker 14

  This is coming. This is gonna be another thing. And we just talk about it like, okay. You know, it's like the next model evolution. I would bet, and we don't know until we get to use these, that it's. We'll of course get used to it quickly. People get used to any new technology quickly. But this will be like a very significant change to the way that work works in a short period of time.

• 1:34:09Speaker 12

  Yeah, it's amazing. Somebody was talking about getting used to new capabilities in A I models and how quickly. I actually think it was about Waymo. But they were talking about how in the first ten seconds of using Waymo, they were like, oh, my God, is this thing like, there's a bug, let's watch out. And then ten minutes in, they were like, oh, this is really cool. And then twenty minutes in, they were like checking their phone or. You know, it's amazing how much your your sort of internal firmware updates uh for this new stuff very quickly.

• 1:34:38Speaker 14

  Yeah, like I think that people will.

• 1:34:43Speaker 14

  Ask an agent to do something for them that would have taken them a month and they'll finish in an hour. It'll be great. And then they'll have like ten of those at the same time. And then they'll have like a thousand of those at the same time. And by twenty thirty or whatever, we'll look back and be like, yeah, this is just like what a human is supposed to be capable of. What a human used to like, you know, grind at for years or whatever, many humans used to grind at for years. Like, I just now like ask a computer to do it and it's like done in an hour. That's why is it not minute?

• 1:35:16Speaker 12

  Yeah, it's also, it's one of the things that makes having an amazing development platform great too. Because, you know, we'll experiment and we'll build some gigantic things, of course. And like, we've already got, I think, just like, we're just pushing the boundaries of what's possible today. You've got groups like Cognition doing amazing things in coding, Harvey and CaseText. You've got SIK doing cool things with language translation. Like, we're beginning to see this stuff work, and I think it's really gonna start working as we continue to write these models.

• 1:35:46Speaker 14

  One of the very fun things for us about having this development platform is just getting to like, watch the unbelievable speed and creativity of people that are building these experiences, like, developers very near and dear to our heart. It's kind of like the first thing we launched, and many of us came building on the platforms. But, the, so much of the capability of these models and great experiences has been built by people building on the platform. We'll continue to try to offer like, great first starting products, but we know that will only ever be like, a small, narrow slice of the apps or agents or whatever people build in the world.

• 1:36:23Speaker 14

  And seeing what has happened in the world in the last eighteen, twenty-four months, uh, it's been like, quite amazing to watch.

• 1:36:33Speaker 12

  And we'll keep going on the agent front here. What do you see as the current hurdles for computer controlling agents?

• 1:36:41Speaker 14

  Safety and alignment. Like, if you are really going to give an agent the ability to start clicking around your computer, which you will, you are going to have a very high bar for the robustness and the reliability and the alignment of that system. So technically speaking, I think that, you know, we're getting like, pretty close on the capability side. But, the sort of agent safety and trust framework, that's gonna, I think, be a long haul.

• 1:37:11Speaker 12

  And now I'll kind of ask you a question, it's almost the opposite of one of the questions from earlier. Do you think safety could act as a false positive and actually limit public access to critical tools that would enable a more egalitarian world?

• 1:37:23Speaker 14

  The honest answer is yes, that will happen sometimes. Like, we'll try to get the balance right. But if we were fully or didn't care about like safety and alignment at all, could we have launched O1 faster? Yeah, we could have done that. It would have come at a cost. There would have been things that would have gone really wrong. I'm very proud that we didn't. The cost, you know, I think would have been manageable with O1, but by the time of O3 or whatever, like, maybe it would be pretty unacceptable. And, and so, starting on the conservative side, like, you know, I think people complain like, "Oh, voice mode, like, it won't say this offensive thing that I really wanted to," and, you know, "formal company, let it offend me." You know what? I actually mostly agree. If, if, if you are trying to get O1 to say something offensive,

• 1:38:08Speaker 14

  it should follow the instructions of its user most of the time. There's plenty of cases where it shouldn't. But we have like a long history of when we put a new technology into the world, we start on the conservative side. We try to give society time to adapt. We try to understand where the real harms are versus sort of like, kind of more theoretical ones. And that's like part of our approach to safety. And not everyone likes it all the time. I don't even like it all the time. But,

• 1:38:37Speaker 14

  but if we're right that these systems are, and we're gonna get it wrong too, like sometimes we won't be conservative enough in some areas. But if we're right that these systems are going to get as powerful as we think they are, as quickly as we think they might, then I think starting that way makes sense. And, you know, we like relax over time.

• 1:38:54Speaker 12

  Totally agree. What's the next big challenge for a startup that's using AI as a core feature? I'll say, you first, I've got, I've got one, which is, I think one of the challenges, and we face this too, is we're also building products on top of our models, is trying to find the, kind of the frontier. You, you wanna be building, these AI models are evolving so rapidly, and if you're building for something that the AI model does well today, it'll work well today, but it's gonna feel, it's gonna feel old tomorrow. And so you wanna build for, for things that the AI model can just barely not do.

• 1:39:33Speaker 12

  You know, where maybe the early adopters will go for it and other people won't quite, but that just means that when the next model comes out, as we continue to make improvements, that use case that just barely didn't work, you're gonna be, you're gonna be the first to do it, and it's gonna be amazing. But figuring out that boundary is really hard. I think it's where the best products are gonna get built up.

• 1:39:53Speaker 14

  Totally agree with that. The other thing I'm gonna add is, I think it's like very tempting to think that a technology makes a startup. And that is almost never true, no matter how cool a new technology or a new sort of like tech title it is. It doesn't excuse you for having to do all of the hard work of building a great company that is going to have durability or like accumulated advantage over time. And we hear from a lot of startups, and I see this as like a very common thing, which is like, I can do this incredible thing, I can make this incredible service, and that seems like a complete answer, but it doesn't excuse you from any of like the normal laws of a business, you still have to like build a good business in a good strategic position.

• 1:40:35Speaker 14

  And I think a mistake is that in the unbelievable excitement and updraft of AI, people are very tempted to forget that.

• 1:40:45Speaker 12

  This is a, this is an interesting one. The motive voice is like tapping directly into the human API. How do you ensure ethical use of such a powerful tool with obvious abilities of manipulation?

• 1:41:01Speaker 14

  Was a really interesting one for me. It was it was like the first time that I felt like I sort of got gotten like really tricked by an AI. In that when I was playing with the first beta of it.

• 1:41:14Speaker 14

  I couldn't like, I couldn't stop myself. I mean, I I kind of. Like, I still say, like, please, such as GPT. But in voice mode, I like couldn't not kind of use the normal niceties. I was like so convinced like, ah, it might be a real like, you know. And obviously, it's just like hacking some circuit in my brain, but I really felt it with voice mode. And I still do the. I think this is a more. This is an example of like a more general thing that we're going to start facing. Which is as these systems become more and more capable. And as we try to make them as natural as possible to interact with. They're gonna like hit parts of our neural circuitry that would like evolve to deal with other people.

• 1:42:01Speaker 14

  And, you know, there's like a bunch of clear lines about things we don't want to do like. We don't like, there's a whole bunch of like weird personality, growth hacking. Like, I think vaguely socially negative stuff we could do. But then there's these like other things that are just not nearly as clear cut. Like, you want the voice mode to feel as natural as possible. But then you get across the uncanny gully and it like, at least in me, triggers something. And, you know, me saying like, please, thank you to chat GPT, no problem, probably a little thing to do, you never know. But. But I think this like really points at. The kinds of safety and alignment issues we have to start in all of this.

• 1:42:43Speaker 12

  All right, back to brass tacks. Sam, when's O one gonna support function tools?

• 1:42:49Speaker 14

  Do you know?

• 1:42:50Speaker 12

  Before the end of the year. There there are three things that we really want to get in for. We're gonna record this, take this back to the research team, show them.

• 1:42:58Speaker 12

  There, I mean, there are a handful of things that we really wanted to get into O one. And we also, you know, it's a balance of, should we get this out to the world earlier and begin, you know, learning from it, learning from how you all use it? Or should we launch a fully complete thing that is, you know, in line with that? It has all the abilities that every other model that we've launched has. I'm really excited to see things like system prompts and structured outputs and function calling make it into O one. We will be there by the end of the year. It really matters to us, too.

• 1:43:31Speaker 14

  In addition to that, just because I can't resist the opportunity to reinforce this, we will get all of those things in and a whole bunch more things you all have asked for. The model is going to get so much better so fast. Like, we are so early. This is like, you know, maybe it's the GPT-2 scale moment, but like, we know how to get it to GPT-4. We have the fundamental set in place now to get it to GPT-4. And in addition to planning for us to build all of those things, plan for the model to just get like rapidly smarter. Like, you know, hope you all come back next year and plan for it to feel like way more of an year of improvement than from 4.01.

• 1:44:13Speaker 12

  What feature or capability of a competitor do you really admire?

• 1:44:17Speaker 14

  I think Google's Notebook thing is super cool. What do they call it?

• 1:44:20Speaker 12

  Notebook LM.

• 1:44:20Speaker 14

  Notebook LM, yeah. I was like, I woke up early this morning and I was like looking at examples on Twitter and I was just like, this is like, this is just cool. This is just a good, cool thing. And like, I think not enough of, not enough of the world's like shipping, you know, in different things. It's mostly like the same stuff. But that, I think, is like, that brought me out of my joy this morning.

• 1:44:43Speaker 12

  Yeah.

• 1:44:44Speaker 14

  Very, very well done.

• 1:44:45Speaker 12

  One of the things I really appreciate about that product is the, just the format itself is really interesting. But they also nailed the podcast-style voices and they have really nice microphones. They have these sort of sonar in voices. Did you guys see, somebody on Twitter was saying like, the cool thing to do is take your LinkedIn and put it, you know, and give it to these, give it to Notebook LM. And you'll have two podcasters riffing back and forth about how amazing you are and all of your accomplishments over the years. I'll say mine is, I think Anthropic did a really good job on projects. It's kind of a different take on what we did with GPTs. And GPTs are a little bit more long-lived.

• 1:45:30Speaker 12

  It's something you build and can use over and over again. Projects are kind of the same idea, but like more temporary, meant to be kind of stood up, used for a while and then you can move on. And that, the different mental model makes a difference. I think they did a really nice job with that. All right, we're getting close to audience questions, so be thinking of what you want to ask.

• 1:45:52Speaker 12

  So at OpenAI, how do you balance what you think users may need versus what they actually need today? Also, better question for you. Yeah, well, I think it does get back to a bit of what we were saying around trying to trying to build for what the model can just like not quite do, but almost do. But it's a real balance, too, as we as we, you know, we support over two hundred billion people every week on ChatGPT. You also can't say, no, it's cool, like, deal with this bug for three months or this issue. We've got something really cool coming. You've got to solve for the needs of today. And there are some really interesting product problems. I mean, you think about, I'm speaking to a group of people who know AI really well.

• 1:46:34Speaker 12

  Think of all the people in the world who have never used any of these products, and that is the vast majority of the world still. You're basically giving them a text interface. And on the other side of the text interface is this like alien intelligence that's constantly evolving, that they've never seen or interacted with, and you're trying to teach them all the crazy things that you can actually do it, all the ways it can help and integrate into your life and solve problems for you. But people don't know what to do with it. You know, like, you come in and you're just like, people type like, hi.

• 1:47:08Speaker 12

  And it responds, you know, hey, great to see, like, how can I help you today? And then you're like, okay, I don't know what to say. And then you end up, you kind of walk away and you're like, well, I didn't see the magic in that. And so it's a real challenge figuring out how you, I mean, we all have a hundred different ways that we use ChatGPT and AI tools in general. But teaching people what those can be and then bringing them along as the model changes month by month by month and suddenly gains these capabilities way faster than we as humans gain the capabilities. It's it's a really interesting set of problems, and I know it's one that you all solve in in different ways as well.

• 1:47:47Speaker 14

  I have no question. Who feels like they spent a lot of time in O1 and they would say like, "I feel definitely smarter than that thing."

• 1:47:58Speaker 14

  Do you think you still go by O2?

• 1:48:03Speaker 14

  No one, no one taking the bet of like being smarter than that too. So, one of the challenges that we face is like, we know how to go do this thing that we think will be like,

• 1:48:15Speaker 14

  "It is probably smarter than all of us in like a broad array of tasks." And yet we have to like, still like fix bugs and do the "hey, how are you?" problem. And mostly what we believe in is that if we keep pushing on model intelligence, people will do incredible things with them. You know, we want to build the smartest, most helpful models in the world and people then find all sorts of ways to use that and build on top of that.

• 1:48:42Speaker 14

  It has been definitely an evolution for us to not just be entirely research focused and then we do have to fix all those bugs and make this super usable. And I think we've gotten better at balancing that. But still, as part of our culture, I think we trust that if we can keep pushing on intelligence, so it's all forward to you to run down there, it'll...

• 1:49:06Speaker 14

  People will build just incredible things with that capability.

• 1:49:09Speaker 12

  Yeah, I think it's a core part of the philosophy and you do a good job pushing us to always, well, basically incorporate the frontier of intelligence into our products, both in the APIs and into our first-party products. Because it's, it's easy to kind of stick to the thing you know, the thing that works well, but you're always pushing us to like, get the frontier in even if it only kind of works because it's gonna work really well soon. So, I always find that a really helpful push. You kind of answered the next one. You do say please and thank you to the models. I'm curious, how many people say please and thank you?

• 1:49:43Speaker 14

  Isn't that so interesting?

• 1:49:44Speaker 12

  I do too.

• 1:49:47Speaker 12

  I kind of can't, I feel bad if I don't.

• 1:49:49Speaker 14

  I am too.

• 1:49:50Speaker 12

  Okay, last question and then we'll go into audience questions for the last 10 or so minutes. Do you plan to build models specifically made for agentic use cases? Things that are better at reasoning and tool calling?

• 1:50:01Speaker 14

  We plan to make models that are great at agentic use cases that will be a key priority for us over the coming months. Specifically is a hard thing to ask for because I think it's also just how we keep making smarter models. So yes, there's like some things like tool use, function calling that we need to build it that'll help. But most things we just want to make the best reasoning models in the world. Those will also be the best agentic based models in the world.

• 1:50:25Speaker 12

  Cool, let's go to audience questions. How extensively do you dog food your own technology in your company? Do you have any interesting examples that may not be obvious?

• 1:50:37Speaker 14

  Yeah, I mean, we put models up for internal use even before they're done training. We use checkpoints and try to have people use them for whatever they can and try to sort of like build new ways to explore the capability of the model internally and use them for our own development or research or whatever else as much as we can. We're still always surprised by the creativity outside the world and what people do, but basically the way we figured out every step along our way of how to, what to push on next, what we can productize, what the models are really good at is by internal dog food. That's like our whole, that's how we like to go our way through this. We don't yet have like employees that are based off of

• 1:51:20Speaker 14

  O1, but I, you know, as we like move into the world of agents, we will try that. Like we'll try having like, you know, things that we deploy in our internal systems that help with itself.

• 1:51:31Speaker 12

  There are things that get closer to that. I mean, there are like customer service. We have bots internally that do a ton about answering external questions and fielding internal people's questions on Slack and so on. And our customer service team is probably, I don't know, 20% the size it might otherwise need to be because of it. I know Matt Knight and our security team has talked extensively about all the different ways we use models. Internally, or to automate a bunch of security things and, you know, take what used to be a manual process, or you might not have the number of humans to even like, look at everything incoming and have models taking, you know, separating signal from noise and highlighting to humans what they need to go look at, things like that.

• 1:52:13Speaker 12

  So I think internally, there are tons of examples and people maybe underestimate the, you all probably will not be surprised by this, that a lot of folks that I talk to are, the extent to which it's not just using a model in a place, it's actually about using like chains of models that are good at doing different things and connecting them all together to get one end-to-end process that is very good at the thing you're doing, even if the individual models have, you know, flaws and make mistakes.

• 1:52:46Speaker 05

  Thank you. I'm wondering if you guys have any plans on sharing models for like offline usage? Because with this distillation thing, it's really cool that we can share our own models, but a lot of use cases, you really wanna kind of like have a version of it.

• 1:53:03Speaker 14

  We're open to it. It's not on, it's not like high priority on the current roadmap. The, if we had like more resources and bandwidth, we would go do that. There's a lot of reasons you want a local model, but it's not like, it's not like a this year kind of thing.

• 1:53:21Speaker 13

  Okay. My question is, are there any agencies in the government above the local, state, and national level that could really greatly benefit from the tools that you guys are developing, but have perhaps some hesitancy on deploying them because of, you know, security concerns, data concerns, privacy concerns? And I guess I'm curious to know if there are any sort of, you know, planned partnerships with governments, world governments, once whatever AGI is achieved? Because obviously if AGI can help solve problems like world hunger, poverty, climate change, government's gonna have to be involved with that, right? And I'm just curious to know if there is some, you know, planned works when, and if that time comes.

• 1:54:04Speaker 12

  Yeah, I think, I actually think you don't want to wait until AGI, you want to start now, right? Because there's a learning process and there's a lot of good that we can do with our current models. So we've announced a handful of partnerships with government agencies, some states, I think Minnesota, some others, Pennsylvania. Also with organizations like USA. It's actually a huge priority of ours to be able to help governments around the world get acclimated, get benefit from the technology, and of all places, government feels like somewhere where you can automate a bunch of workflows and make things more efficient, reduce drudgery, and so on. So I think there's a huge amount of good we can do now. And if we do that now, it just accrues over the long run as the models get better and we get closer to AGI.

• 1:54:48Speaker 15

  Pretty open-ended question. What are your thoughts on open source? So whether that's open weights, just general discussion, where do you guys sit with open source?

• 1:55:01Speaker 14

  I think open source is awesome. Again, if we had more demand, we would do that too. We've like gotten very close to making a big open source effort a few times. And then, you know, the really hard part is prioritization and we have put other things ahead of it. Part of it is like, there's such good open source models in the world now that I think that segment... The thing we always said in most of our advice, like a really great on-device model. And I think that segment is very well served. I do hope we do something at some point, but we want to find something that we feel like, if we don't do it, then we'll just be missing this and not make like another thing that's like a train to be better on benchmarks. Because we think there's like a lot of good stuff out there now. But like spiritually, philosophically, very glad it exists.

• 1:55:46Speaker 14

  Would like to contribute.

• 1:55:50Speaker 16

  Hi Sam, hi Gavin. Thanks for inviting us to Dev Day. It's been awesome. Hope we like them as work. It's incredible. Why can't advanced voice mode say? And as a follow-up to this, if it's a company, like legal issues, corporate, etc., is there daylight between how you think about safety in terms of your own product, on your own platform, versus giving us developers kind of the, I don't know, sign the right things off so we can make our... I don't want to sing. Could you?

• 1:56:19Speaker 14

  You know, the funny thing is Sam asked the same question. Why can't this thing sing? I wanted to sing. I've seen it sing before. Actually, there are things, obviously, that we can't have it sing. We can't have it sing copyrighted songs. You don't have the licenses, etc. And then there are things that it can't sing. You could have it sing "Happy Birthday", and that would be just fine, right? And we want that, too. It's a matter of, I think, once you... Basically, it's easier with finite time to say no and then build it in. But it's nuanced to get it right. And we, you know, there are penalties to getting these kinds of things wrong. So it's really just where we are now. We really want the models to sing, too.

• 1:57:02Speaker 14

  People were waiting for us to ship Voice Mode, which is like very fair. We could have waited longer and kind of really got the classifications and filters on, you know, copyrighted music versus not. We'll be sending it just shipped in one more. But I think Sam has asked me like four or five times why.

• 1:57:23Speaker 14

  I mean, we still can't offer something where we're gonna be in, like, really bad, you know, hot water developers or first party or whatever. So, yes, we can maybe have some differences, but we still have to comply with the law.

• 1:57:36Speaker 17

  Could you speak a little to the future of where you see context windows going? And kind of the timeline for when... How you see things bounce between context window growth and drag, basically, information retrieval?

• 1:57:48Speaker 14

  I think there's like two different takes on that that matter. One is like, when is it gonna get to like, kind of, normal long context? Like, context like a million or whatever? Like, long enough that you just throw stuff in there, and it's fast enough, you're happy about it? And I expect everybody's gonna make pretty fast progress there, and that'll just be a thing. Long context has gotten, weirdly, less usage than I would have expected so far. But I think, you know...

• 1:58:13Speaker 12

  There's a bunch of reasons for that. I don't want to go too much into it. And then there's this other question of like, when do we get to context like not like ten million, but ten trillion? And when do we get to the point where you throw like every piece of data you've ever seen in your entire life in there? And, you know, like, that's a whole different set of things. That obviously takes some research breakthroughs, but I assume that infinite context will happen at some point, at some point, it's like less than a decade, and that that's going to be just a totally different way that we use these models. Even getting to the like ten million tokens of very fast and accurate context, which I expect measured in like months, something like that, you know, like

• 1:58:56Speaker 12

  people will use that in all sorts of ways and it'll be great. But yeah, the very, very long context, I think is gonna happen and it's really interesting.

• 1:59:07Speaker 12

  I think we may be up time for one or two more.

• 1:59:10Speaker 18

  Don't worry, this is, if you want to go to your favorite question. So with voice and all of the other changes that users have experienced since you all have launched your technology, what do you see is the vision for the new engagement layer? The form factor and how you actually engage with this technology to make our lives so much better?

• 1:59:34Speaker 12

  I love that question. It's one that we ask ourselves a lot, frankly. There's this, and I think it's one where developers can play a really big part here because there's this trade off between generality and specificity here. I'll give you an example. I was in Seoul and Tokyo a few weeks ago. And I was in a number of conversations with folks that with whom I didn't have a common language and we didn't have a translator around. Before, we would not have been able to have a conversation. We would just sort of smile at each other and continued on. I took out my phone. I said, ChatGPT, I want you to be a translator for me. When I speak in English, I want you to speak in Korean. You hear Korean, I want you to repeat it in English. And I was able to have a full business conversation.

• 2:00:18Speaker 12

  And it was amazing. You think about the impact that could have, not just for business, but think about travel and tourism and people's willingness to go places where they might not have a word of a language. You can have these really amazing impacts. But inside ChatGPT, that was still a thing that I had to... Like, ChatGPT is not optimized for that, right? Like, you want this sort of digital, you know, universal translator in your pocket that just knows that what you want to do is translate? Not that hard to build. But I think there's... We struggle with the... With trying to build a... An application that can do lots of things for lots of people and that keeps up, like we've been talking about a few times, it keeps up with the pace of change and with the capabilities,

• 2:01:01Speaker 12

  you know, agentic capabilities and so on. I think there's also a huge opportunity for the creativity of an audience like this to come in and like solve problems that we're not thinking of, that we don't have the expertise to do. And ultimately, the world is a much better place if we get more AI to more people. And it's why we are so proud to serve all of you.

• 2:01:23Speaker 14

  I... The only thing I would add is if you just think about everything that's gonna come together. At some point in not that many years in the future, you'll walk up to a piece of glass. You will say whatever you want. They will have like, there'll be incredible reasoning models. Aiden's connected to everything. There'll be a video model streaming back to you like a custom interface just for this one request. Whatever you need is just gonna get like rendered in real time in video. You'll be able to interact with it. You'll be able to like click through the stream or say different things and it'll be off. Doing like, again, the kinds of things that used to take like humans years to figure out. And it'll just, you know, dynamically render whatever you need and it'll be a completely different way of using a computer and also getting things to happen in the world. Then it's gonna be quite wild.

• 2:02:07Speaker 12

  Awesome. Thank you. That was a great question to end on. I think we're at time. Thank you so much for coming to Dev Day.

• 2:02:23Speaker 00

  That's all for our coverage of Dev Day 2024. We want to extend an extra special note of gratitude to Lindsay McCallum of the OpenAI comms team, who helped us set up so many interviews at very short notice and physically helped ensure the smooth continuity of the video recordings. We couldn't do this without you, Lindsay. If you have any feedback on the launches or for our guests, hop on over to our YouTube or Substack comment section and say hi. We're especially interested in your personal feedback and demos built with the new things launched this week. Feel the AGI.

• 2:03:07Speaker 02

  All right, so you wanted to know more about OpenAI's Dev Day and what stood out to us. We're diving into all the developer interviews and discussions, and there's a lot to unpack.

• 2:03:16Speaker 01

  Yeah, it's interesting. OpenAI seems to be like transitioning, moving beyond just building these impressive AI models.

• 2:03:23Speaker 02

  Yeah.

• 2:03:23Speaker 01

  One expert even called them, get this, the AWS of AI.

• 2:03:26Speaker 02

  AWS of AI.

• 2:03:28Speaker 01

  Yeah.

• 2:03:28Speaker 02

  Okay, so what does that even mean when we talk about AI?

• 2:03:31Speaker 01

  So it means, instead of just offering this raw power, they're building a whole ecosystem, the tools to fine tune those models, distillation, you know, for efficiency, and a bunch of new evaluation tools. Oh, and a huge emphasis on real time capabilities.

• 2:03:47Speaker 02

  You know, instead of just giving us the ingredients, it's like they're providing the whole kitchen.

• 2:03:49Speaker 01

  Exactly. They're laying the groundwork for, well, they envision a future where you can build almost anything with AI.

• 2:03:56Speaker 02

  I see. And one of the tools that really caught my eye was this function calling. They used it in that travel agent demo, remember? How does that even work?

• 2:04:05Speaker 01

  So function calling, it's like giving the AI access to external tools and information. Imagine, instead of just having all this pre-programmed knowledge, you can like search the web for you, book flights, even order a pizza.

• 2:04:17Speaker 02

  So instead of a static encyclopedia, it's like giving the AI a smartphone with internet.

• 2:04:21Speaker 01

  Yeah, precisely.

• 2:04:23Speaker 02

  Yeah.

• 2:04:23Speaker 01

  And this ties into their focus on real time interaction, right? They see a future where AI can respond instantly, just like a human would.

• 2:04:31Speaker 02

  Which would be a game changer.

• 2:04:32Speaker 01

  Right. It's like, imagine voice assistants that actually understand you. Or even seamless real time translation.

• 2:04:39Speaker 02

  No more language barriers.

• 2:04:40Speaker 01

  Exactly. That's just the tip of the iceberg, though. They really believe this real time capability, it's key to making AR truly mainstream.

• 2:04:48Speaker 02

  Okay, so OpenAI is building this AI platform, emphasizing real time interactions. How does this translate into like actual results?

• 2:04:56Speaker 01

  Yeah.

• 2:04:56Speaker 02

  You know, real world stuff?

• 2:04:58Speaker 01

  Well, that's where things get really interesting. Let's talk about the O1 model and how developers are using it to like really push the boundaries of what's possible.

• 2:05:06Speaker 02

  So this O1 model, everyone's talking about it. One developer even said they built an entire iPhone app just by describing it as O1. Is that just hype?

• 2:05:16Speaker 01

  I think there's definitely some substance behind all the hype. What's so fascinating about O1, it's not just about the code it generates. It's how it seems to understand, like the logic.

• 2:05:24Speaker 02

  The logic.

• 2:05:25Speaker 01

  Yeah, like this developer, they didn't give O1 lines of code. They described the idea of the app and O1, it actually designed the architecture, connected everything. The developer just took that code, put it right into Xcode and it worked.

• 2:05:37Speaker 02

  Wow, so it's not just writing code, it's understanding the intent.

• 2:05:40Speaker 01

  Yeah, exactly. And this actually challenges how we measure these models, you know? Even OpenAI admitted that these benchmarks, like what was it? Sui-bench.

• 2:05:50Speaker 02

  Sui-bench.

• 2:05:51Speaker 01

  Right, which looks at code accuracy. It doesn't always reflect how things work in the real world.

• 2:05:55Speaker 02

  Right, because in the real world, you don't just need code that compiles. It has to be like efficient, maintainable.

• 2:06:01Speaker 01

  Exactly. It all has to work together. And OpenAI is really working on this with developers. They're finding that UI development, especially in things like React, it needs better evaluation. It's one thing to code a button that works and another to make it actually look good, you know, and be intuitive.

• 2:06:16Speaker 02

  Right, and it seems like this need for real world context, it goes beyond just like evaluating those models. There was a developer working with this code generating AI genie, I think it was called.

• 2:06:27Speaker 01

  Genie, yeah.

• 2:06:28Speaker 02

  And it's more focused on those specific coding tasks. But they found that its performance really changed between different programming languages. Like JavaScript versus C sharp, for example.

• 2:06:39Speaker 01

  And that just highlights how important the data is, right? Just like us, AI needs that variety to learn. If you train it on just one type of code, it'll be great at that, but. Anything new and

• 2:06:50Speaker 02

  It'll fall flat.

• 2:06:51Speaker 01

  Yeah, so it's about making sure these models have a broad diet of data to learn from. That way they're more adaptable and ready for whatever we throw at them.

• 2:06:59Speaker 02

  So we've got AI that can build apps, understand what we want, even write different kinds of code. It's a lot and it feels like things are changing so fast. How can developers even keep up, let alone like build something successful with AI?

• 2:07:11Speaker 01

  Right, that's the question, isn't it? But it's interesting, you know, both OpenAI and the developers building with these tools, they kind of agree on one thing. You gotta aim for what's just out of reach.

• 2:07:22Speaker 02

  So don't wait for the tech to catch up to your like wildest dreams. Focus on what's almost possible right now.

• 2:07:29Speaker 01

  Yeah, build for where things are going, not where they are today. You wait for that perfect AI, you might miss the boat on shaping how it develops. And being the first one out there doing something new.

• 2:07:40Speaker 02

  Riding the wave, not chasing after it.

• 2:07:41Speaker 01

  Exactly, but and OpenAI really emphasized this too. Even with all this amazing AI, you can't forget the basics of building a business.

• 2:07:50Speaker 02

  So just because it's got AI doesn't mean it's automatically going to be a success.

• 2:07:53Speaker 01

  Right, you need a good strategy, know who you're selling to, and it's got to actually solve a real problem. AI is a tool, not a magic wand.

• 2:08:01Speaker 02

  Like, having the best oven in the world won't help if you don't know how to cook.

• 2:08:05Speaker 01

  Perfect analogy. And then there's this other thing OpenAI talked about that's really interesting. Balancing safety with access for everyone.

• 2:08:14Speaker 02

  So making sure these AI tools are used responsibly, but also making them available to everyone who could benefit.

• 2:08:21Speaker 01

  Yeah, they're really aware that focusing on safety, while important, could limit access to some really powerful stuff. It's a tough balance.

• 2:08:30Speaker 02

  It's like that debate around, you know, life-saving medications. How do you make sure they're used correctly, but also make sure people who need them can actually get them?

• 2:08:38Speaker 01

  It's complicated, no easy answers, but it's something they're thinking hard about.

• 2:08:42Speaker 02

  Well, it's clear that all this AI stuff, especially with these new models like O1, is changing how we think about tech, how we use it.

• 2:08:49Speaker 01

  Imagine walking up to a screen and it just creates a personalized experience for you right there, adapts to what you need. That's the potential.

• 2:08:57Speaker 02

  Like having a personal assistant in every device.

• 2:09:00Speaker 01

  It's exciting, but we gotta be thoughtful about it, build responsibly.

• 2:09:03Speaker 02

  So there you have it. OpenAI isn't just building these cool AI models. They're building a whole world around them and it's changing everything. It's gonna be a wild ride, that's for sure, and we're just at the beginning.