OpenAI Day: GPT-5 Unveiled | Mark Chen, Greg Brockman, Sarah Friar, Max Schwarzer, Brad Lightcap & More

(01:18) - AI Model Whiteboard Breakdown (26:23) - Mark Chen, Chief Research Officer at OpenAI, discusses the recent launch of GPT-5, emphasizing its enhanced reasoning capabilities and seamless integration of various AI models to improve user experience. He highlights the model's ability to perform complex tasks more efficiently, reducing the need for users to choose between different model versions. Chen also touches on the importance of personalization and memory in AI, aiming to make interactions more intuitive and tailored to individual users. (57:52) - Greg Brockman, co-founder and president of OpenAI, discusses the evolution of the GPT series, highlighting the progression from GPT-1's foundational capabilities to GPT-5's transformative impact on software engineering. He reflects on the challenges and breakthroughs in developing these models, emphasizing the importance of scaling and infrastructure in achieving advanced AI functionalities. Brockman also touches on the broader implications of AI, including its role in enhancing human productivity and the necessity for responsible development to maximize societal benefits. (01:31:55) - Sarah Friar, OpenAI's Chief Financial Officer since June 2024, previously served as CEO of Nextdoor and CFO at Square. She discusses the rapid growth of ChatGPT, now with 700 million weekly active users, the expansion of enterprise adoption to 5 million paying business users, and the importance of substantial investments in compute infrastructure to support future AI developments. (01:52:13) - Dedy Kredo, Co-Founder and Chief Product Officer of Qodo (formerly CodiumAI), discusses the integration of GPT-5 into their platform to enhance code review processes. He highlights the model's improved capabilities in generating high-quality code reviews, identifying bugs before production, and ensuring enterprise code aligns with best practices. Kredo emphasizes the importance of AI agents in automating code review tasks while maintaining human oversight to verify code quality and adherence to standards. (01:59:11) - Zach Lloyd, founder and CEO of Warp, discusses the significant advancements in AI models, emphasizing their enhanced capabilities and cost-effectiveness, which are particularly beneficial for individual developers and small teams. He highlights the importance of competition among model providers to drive down prices and improve quality, expressing hope for a future where multiple competitive models coexist, similar to cloud service providers. Additionally, Lloyd addresses the challenges of model deprecation, noting that for application-level stacks like Warp, transitioning to the latest models is straightforward and advantageous. (02:11:15) - Riley Tomasek is a serial entrepreneur and the Founder & CEO of Charlie Labs, home of an AI-driven "autonomous TypeScript engineer" designed to accelerate code reviews and merge processes. Previously, he co-founded Flight (acquired by Figma) and launched Dexa, an AI platform that transforms podcast discovery. Riley holds a B. Sc. in Mathematics & Computer Science from the University of British Columbia and has a track record of building developer-friendly tools and interfaces. (02:18:31) - Guillermo Rauch, founder and CEO of Vercel, discusses the transformative impact of AI on software development, emphasizing the shift towards "vibe coding," where natural language prompts generate code and user interfaces, making software creation more accessible. He highlights the role of AI agents in automating tasks, enabling developers to focus on higher-level management and creative processes. Rauch also explores the future of developer tools, noting the importance of integrating AI capabilities to enhance productivity and streamline workflows. (02:34:28) - Eno Reyes, co-founder and CTO of Factory, discusses how their platform integrates AI agents into every stage of the software development lifecycle, including coding, code review, maintenance, incident response, and documentation. He highlights the platform's focus on large enterprises with over 1,000 engineers, addressing challenges like migrating numerous codebases to new frameworks and modernizing legacy systems. Reyes emphasizes that while AI tools can accelerate individual developers, significant productivity gains require workflow changes that incorporate agents throughout the development process. (02:40:20) - Guy Gur-Ari, co-founder and Chief Scientist at Augment Code, discusses the company's AI coding assistant designed for large teams with extensive codebases, emphasizing its capabilities in question answering, development, refactoring, and migrations. He highlights the thoughtful nature of GPT-5, noting its propensity for tool calls and clarifying questions before code modifications, making it particularly effective for complex tasks. Gur-Ari also mentions Augment's focus on developing proprietary integrations and tools, aiming to enhance the agent's performance without relying solely on external model vendors. (02:48:20) - Harjot Gill, CEO of CodeRabbit, discusses the significant improvements observed with GPT-5 in their AI-driven code review platform, noting a near doubling in performance compared to previous models. He emphasizes that these enhancements will be available to customers at no additional cost, reflecting the rapid evolution of AI capabilities. Gill also highlights the company's focus on monitoring real-world performance metrics, such as user conversion rates and potential issues like hallucinations, to ensure the model's effectiveness and reliability. (02:52:34) - Timeline (02:57:01) - Max Schwarzer, a leading researcher at OpenAI, discusses the recent launch of GPT-5, highlighting its significant advancements in coding capabilities and its potential to revolutionize user interactions by enabling the creation of personalized applications without prior coding knowledge. He emphasizes the importance of refining the post-training process to enhance the model's accuracy and reliability, particularly in reducing hallucinations and improving user engagement. Schwarzer also touches on the future trajectory of AI development, expressing optimism about the integration of reinforcement learning to extend AI's applicability beyond textual domains into real-world interactions. (03:13:26) - Scott Wu, co-founder and CEO of Cognition, discusses the significant advancements in AI coding models, noting that OpenAI has caught up to Anthropic, leading to a competitive landscape. He emphasizes the importance of integrating AI agents like Devin into software engineering workflows to enhance capabilities and efficiency. Wu also highlights the evolving role of engineers, suggesting a shift from "bricklayers" to "architects" as AI tools handle more complex tasks. (03:23:21) - Claire Vo, founder of ChatPRD and former Chief Product Officer at LaunchDarkly, discusses the developer-centric design of GPT-5, noting its enhanced coding capabilities but expressing concerns about its verbosity and tendency to produce lengthy outputs. She emphasizes the importance of validating new models with users, especially in business contexts where concise communication is crucial. Vo also highlights the need for AI models tailored to specific roles, such as strategists, to better serve diverse professional needs. (03:33:34) - Brad Lightcap, OpenAI's Chief Operating Officer, discusses his multifaceted role, which includes responsibilities ranging from project management to sales, and emphasizes the significant improvements observed with the launch of GPT-5. He highlights the diverse applications of OpenAI's models across various industries, such as pharmaceuticals, customer support, and everyday productivity tools, underscoring the transformative impact of AI on organizational efficiency...

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00John Coogan?

  You're watching TVP and your background looks way different because you have a whiteboard behind you because we're breaking down the X's and O's of the GPT five launch today. GPT five launched from open AI. Really quickly, there is some other news. Firefly Aerospace stock opened at $70 in NASDAQ debut. This is the company that landed on the moon. Very cool. Very cool. There, there are a few other stories going on, but we're going to skip most of them because we're going to be focusing on chat GPT today on GPT five. We have a bunch of, a bunch of guests coming on. We have a stacked lineup. We'll pull that up, but we'll break down the X's and O's of the matchup. So of course, open AI.

• 0:42John Coogan?

  Here's our, here's our lineup. We have something like 15 guests today. A ton of folks from open AI, a ton of people that build on top of open AI and can comment on what's going on with chat GPT. But of course, this battle is between open AI and the timeline. It's the, it's, they gotta get the vibes right. It's war. It's war. It's, it's the timelines in turmoil over whether or not this is a good model, what it means for the industry, what it means for AGI timelines. Everyone's got their take. Everyone's posting memes. There's been a ton of funny ones already. We'll take you through them, of course, but let's break down the offense today. We have Sam Altman, founder, CEO. He briefly got cut from the team in November of 2023,

• 1:24John Coogan?

  but he's back leading the team for the 2024, 2025 seasons. He seems healthy. He's doing great today. He went on at 10 AM to break down the launch of GPT five. He has a couple of key plays in his playbook, in his arsenal. He's got a solid ground game, lots of quick posts, hitting the timeline, probably in lowercase. Then he might air it out with a couple thousand word essay. We've seen him do this before. It's a bit of a hail Mary. Maybe AG has a thousand, a couple thousand days away. Maybe we're in the soft singularity, but he's very strong there with the long post when he needs to be. It's up his sleeve if he needs it. Then he can also pull out the vague posting. He was doing this last night, posted a picture of the Death Star. No one knows what it means. Maybe it was taking a shot at the doomers who

• 2:08John Coogan?

  are on the defense today. So he's also known for driving supercars. That lets him get to the office faster. He's saving time and money. You can save time and money by going to ramp.com. Easy use corporate cards, bill pay and accounting and a whole lot more all in one place. And so he is, he also gave apparently, this is a rumor, he gave every OpenAI employee who's been with the company for more than two years, 1.5 million dollars. A lot of people say 1.5 million, that's not enough for a big house in San Francisco, but it is enough for a supercar. So that's probably why he picked that number. And that's why, that's what the OpenAI team will be doing with that money. They'll be buying Aston Martin Valkyries,

• 2:48John Coogan?

  Pagani Huayras, McLaren Sabres, Ferrari Daytona SB3s. They can get a Koenigsegg, Gamera. They could get a Singer, DLS, or a Bugatti Veyron. It would have to be used. They could also get the Bentley Bacalar. There's only, there's only 12 of those ever made. It's an open top two-seater roadster. It's coach built. So that's going to run your 1.5 million, but that's perfect. You just got the 1.5 million bonus. So put it to work, spend it all in one place on a car. This is financial advice. Yes, exactly. Then you got Greg Brockman, he's joining at noon. He's, he's extremely well rested. He's actually coming off a sabbatical right now. That's very exciting. He should be injury free for the rest of the season.

• 3:28John Coogan?

  He cut his teeth at MIT and then he got drafted by Stripe in 2010. Microsoft tried to do a trade deal during the 2023 chaotic trade deal, trade window that opened up post Sam Altman ouster, but he stuck with the OpenAI team and now he's president of the company. Then you got Mark Chen, he's coming on at 11:30 today. He's the chief research officer. The rumor is that he turned out a maxed out contract to head the Metallamas, but he's sticking with the OpenAI team. He was an MIT undergrad, also worked at Jane Street before joining OpenAI in 2018. Then we got Sarah Friar coming on the show at 12:30. She's the CFO of OpenAI. It's her job to find bank accounts big enough to fill all the cash

• 4:13John Coogan?

  they're raising. It's a tough job. You gotta find, okay, this bank account, will it hold 10 figures? Will it hold 11 figures? Will it hold 12 figures?

• 4:23John Coogan?

  Exactly, exactly. She's also going to be defining the non-GAAP metrics that will be catnip for Ben Thompson in just a few years. We're excited to talk to her about how she's measuring the success and the health of their business. Obviously, it's not just revenue, it's not just top line, bottom line. We're going to want to know about queries, we're going to be want to know about DAUs, all those non-GAAP metrics. That's where people are going to be tracking when IPO, when IPO day comes, hopefully soon. And then we also have Brad Lightcap, he's joining at 2:35. He entered the league as an investment banker. Let's give it up for the investment bankers. They don't get enough credit around here, but we love the investment bankers. Then he got drafted by Y Combinator before joining OpenAI as CFO in 2018. Now he's the chief operating officer. And then we have Max Warser,

• 5:07John Coogan?

  he's in charge of post-training, fine-tuning these models, getting them into the fighting performance to put on a display of authority on GPT-5 launch day. Now, let's flip it over to the defense. They're going up against the timeline, they're going up against the vibe checks. We got the doomers. The doomers, they're led by Eliezer Yudkowsky. Admittedly, everyone knows this, no one debates this, the doomers have had a terrible season. But you'd expect to see at least a few hail marys about GPT-5 creating bioweapons thrown up on the timeline today. Probably won't be bangers, probably won't get 1,000 likes, but you'll be seeing them here and there, mostly in the replies. We've also seen some doomers talking about GPT-5 being available to every government employee. And Eliezer

• 5:51John Coogan?

  had some harsh words about that. Don't give the keys to Sam Altman, don't give the keys to the government to OpenAI. He was upset about that. But in general, the doomers not putting much of a fight up today. Then you got Claude. Interesting, Claude was caught playing for the wrong team earlier this week. Anthropic, they're on defense today, but we saw them take out OpenAI's key pinch hitter, Claude. The Claude code API was playing for the OpenAI team. But they shut that down and Claude is no longer pinch hitting for OpenAI. Then you got the Elon stans. The ground game is going to be there. It's going to be, it's going to be strong.

• 6:34John Coogan?

  The Elon stans are going to be tracking the benchmarks relentlessly. We know XAI loves to benchmarks and all the Elon stans are going to be calling out GPT-5 for any, any misaligned benchmarks. If they fail humanity's last exam, it's over, it's over. They'll also toss up the occasional unhinged conspiracy theory. Moving on Gemini, the betting lines have shifted big time. People thought Gemini was out of the game. They're so back. Polly market has Gemini at what 75% chance of being the best model towards the end of the month. This is of course, based on the LM arena, more vibes based benchmark. But Gemini will probably be quiet today. They usually don't try and front run press releases.

• 7:18John Coogan?

  They usually try and sit back, let the model speak for themselves, let the API credits work their way through the latest YC demo day batch and get the product into the hands of people. And so expect to see a big glossy conference in a couple of weeks. Demoing Gemini 3 should be a good rebuttal from the Geminis. Then you got the Metallamas. Zuck's been on a poaching spree. He's rebuilding the team during the off season. Now he has a stacked roster and he's ready to go duke it out, but no one knows exactly what's going to be in the playbook. Is he going to go consumer? Is he going to go API? Is he going to turn into a hyperscaler? We don't know, but we know they got a stacked team.

• 8:00John Coogan?

  They got Alex Wang, they got Nat Friedman, they got Daniel Gross, they got tons and tons of other researchers. They've been raiding every other team, completely reset the salary cap for the league and it's been, it's been an absolute clinic in terms of recruiting over there at Lama. Then you got the final benchmark, Arc AGI. This benchmark stans GPT-5 couldn't get past this defense. And Arc AGI, you know.

• 8:29John Coogan?

  Sitting there right in the end zone, just swatting them down, swatting them down all day. You think, you think, you think we're super intelligence around the corner? Arc AGI denied, denied. Tyler, give us the update on Arc AGI. Where does everything stand? How GPT-5 do? Does it matter? Should we care about Arc AGI? We love the team behind them, but is it an important benchmark? Should we be tracking it today?

• 8:54Speaker 01

  Arc AGI V1 and V2, right?

• 8:56John Coogan?

  Okay.

• 8:56Speaker 01

  On both of them?

• 8:57John Coogan?

  And V3.

• 8:58Speaker 01

  V3, I actually don't know if-

• 8:59John Coogan?

  No one's been, no one's even tested V3 yet.

• 9:02Speaker 01

  Yeah, no one's even really close there.

• 9:03John Coogan?

  But how are we doing on V1?

• 9:04Speaker 01

  V1, GPT-5 is at 65.7. Unfortunately, that's going to be 1% just short of Grok 4, 66.7. Okay, Arc AGI 2-

• 9:15John Coogan?

  The Elon stands are going to be going wild with that.

• 9:17Speaker 01

  Arc AGI 2, 9.9%.

• 9:20John Coogan?

  9.9%.

• 9:21Speaker 01

  Grok 4, 16%.

• 9:23John Coogan?

  16%.

• 9:25Speaker 01

  Kind of brutal, you know, Arc AGI mugging.

• 9:27John Coogan?

  Rough showing, rough showing.

• 9:28Speaker 01

  Some people have accused Grok 4 of being slightly benchmarked.

• 9:32John Coogan?

  Yes.

• 9:32Speaker 01

  You know, this is, you know-

• 9:34John Coogan?

  They might have a team working on it.

• 9:37John Coogan?

  What's the, what are the pros and cons? We know the cons of benchmarking, of benchmarking. You're overfitting on something that might not actually drive consumer value. It might not actually solve real world problems. It might not increase DAUs or revenue or ARR, anything that really matters. It might not even get us closer to super intelligence. Give me the counter argument. Why is benchmarking good?

• 10:02Speaker 01

  The bull case for benchmarking.

• 10:02John Coogan?

  The bull case for benchmarking, benchmarking. Break it down for me.

• 10:06Speaker 01

  Yeah, so I think the idea is basically, this is almost like a non AGI pilled kind of take, right? So if you don't have a super general intelligence-

• 10:15John Coogan?

  Yep.

• 10:17Speaker 01

  Your ability to benchmark basically proves your ability to solve some like kind of-

• 10:24Speaker 01

  Specific tasks. So there's this thing about the It's called going spiky. Yeah. It's called getting spiky.

• 10:30John Coogan?

  Getting spiky, adding more spikes to the spiky intelligence.

• 10:34Speaker 01

  Yeah, I think it was Rune who had this tweet about

• 10:36John Coogan?

  Yep.

• 10:36Speaker 01

  The gas station benchmark.

• 10:38John Coogan?

  Yep.

• 10:38Speaker 01

  Right. I don't care if he says something like, I don't care about AI solving gas stations, if it has a gas station benchmark, something like that.

• 10:48John Coogan?

  Yeah.

• 10:49Speaker 01

  But the idea is like, if you, if the, if making the gas station benchmark.

• 10:53John Coogan?

  Rune said, my bar for AGI is an AI that can learn to run a gas station for a year without a team of scientists collecting the gas station data set in capital letters.

• 11:06Speaker 01

  Yeah. And then my take is basically, I don't care how they got to the, like, I don't care how they made it run the gas station. I care how fast it gets here.

• 11:14John Coogan?

  That it runs it. If we can run the gas station with AI, that's good enough.

• 11:18Speaker 01

  If you have a team who's, you know, your benchmarking team, that just proves that like, if you have some task that's like really important that you want to get done.

• 11:25John Coogan?

  Yep.

• 11:26Speaker 01

  They can just figure it out.

• 11:27John Coogan?

  Yeah.

• 11:27Speaker 01

  So it's like RL for business. This is like the same thing, RL for law.

• 11:30John Coogan?

  Yeah.

• 11:30Speaker 01

  All these like

• 11:30John Coogan?

  This is what

• 11:31Speaker 01

  Specific verticals.

• 11:32John Coogan?

  Miramarati is doing this with thinking machines, right?

• 11:34Speaker 01

  Exactly.

• 11:34John Coogan?

  Like RL for businesses. Come into your organization, understand the most, the most valuable business processes out there that could potentially be RL'd against, that could be turned into a benchmark.

• 11:46Speaker 01

  Yeah.

• 11:46John Coogan?

  And then, and then, you know, bench hacked, because I don't care if you're hacking, you know, if I have translate this type of document to this type of document for my business, if you can do it with 100% accuracy, I don't care that you bench hacked it.

• 12:00Speaker 01

  Yeah, exactly. Like benchmarks right now are not like economically valuable. Like if you, if you're really that much better at MMLU.

• 12:05John Coogan?

  Yes.

• 12:06Speaker 01

  It's like, is, are you producing that much value?

• 12:08John Coogan?

  Yes.

• 12:08Speaker 01

  Probably not. But if you have, if you make some new benchmark, that's, you know, your tax benchmark. I think Anthropic just released that fairly recently.

• 12:14John Coogan?

  Oh, sure, sure, sure.

• 12:16Speaker 01

  That's like, I don't care if you benchmark on that, if it does way better.

• 12:19John Coogan?

  If it does the taxes, it's good.

• 12:20Speaker 01

  It's going to do the task.

• 12:21John Coogan?

  Yeah, yeah, yeah, yeah, that makes sense. What about the, what does it say that it feels like OpenAI seems capable of bench hacking? It seems like they've opted not to. Is that because bench hacking has a risk of giving you negative aura?

• 12:42John Coogan?

  Because if you're accused and found guilty of bench hacking, you could, it often reveals that you're not building this one beautiful, you know, super intelligence to rule them all.

• 12:55Speaker 01

  Yeah, I think it's also like, maybe we're just looking at the wrong benchmarks. Like, maybe there, there's a bunch of like interesting benchmarks about like, there's this one I really like. It's the Minecraft benchmark where you have to like build, you like give it some castle and how good it looks. Or there's the one you always see about the unicorn.

• 13:13John Coogan?

  Yeah.

• 13:13Speaker 01

  That's in, it's, so you use this like math package that does like grass and stuff, but you ask it to draw a unicorn.

• 13:19John Coogan?

  Oh, yeah, I've seen that, yeah.

• 13:20Speaker 01

  Those are really good because it kind of shows the creativity and stuff like that.

• 13:23John Coogan?

  Walk us through TBPN bench. And what we will be benchmarking the AIs against going forward. Have you heard about this?

• 13:31Speaker 01

  Reps of 225.

• 13:33John Coogan?

  That would be close, but it's difficult because the humanoids kind of change that and you can just use a normal actuator. This is, this is truly for a large language model. You feed in our data set. We have a public data set, a private data set presumably at some point. But walk us through TBPN bench.

• 13:50Speaker 01

  Yeah, so I'm yet to try this on GPT-5. I don't think it's out yet for public use. At least I don't have it. But I can, I can tell some of the questions, right? So the first one, I have this picture of a horse. You have to guess the breed.

• 14:02John Coogan?

  Yep.

• 14:03Speaker 01

  So, let me see, I think, why, I don't want to say it in case GPT-5 is listening, but it is, may or may not be a Caspian horse.

• 14:09John Coogan?

  Okay, and it's failing right now.

• 14:12Speaker 01

  03 is failing.

• 14:13John Coogan?

  03 is failing.

• 14:14Speaker 01

  Coro is failing. I haven't tried every model.

• 14:16John Coogan?

  Yeah, we gotta try Grok and Gemini.

• 14:18Speaker 01

  We're gonna bring it all out, yeah.

• 14:19John Coogan?

  Horse identification. This seems extremely hackable, but at the very least, if we get one scientist to be, to go off and collect the horse data set and then, and then, and then bench hack it, I think we will have done our job.

• 14:31Speaker 01

  Yeah, that's the first question.

• 14:33John Coogan?

  Yes.

• 14:33Speaker 01

  The second one is a, it's, I have two pictures of the before and after of this guy. And it's which peptide didn't take to achieve this body transformation?

• 14:44John Coogan?

  Yep, yep, yep.

• 14:45Speaker 01

  So it fails there.

• 14:46John Coogan?

  It fails there. So you have a data set of what peptide does what to the human body? Where'd you find that?

• 14:51Speaker 01

  Well, you know, Wikipedia has a lot of this stuff.

• 14:53John Coogan?

  Okay, okay. You'd think they'd be able to cheat this around with O3, just reason, who is this person? Go look up what they've said they've taken, and then boom, you have the answer.

• 15:02Speaker 01

  Well, at first with O3, when I was prompting it, I would like save the photo, but then it would have the metadata, or the file name would be like Caspian Horse or something.

• 15:10John Coogan?

  Yeah, yeah, yeah, okay. And then the third one?

• 15:12Speaker 01

  The third one, I pass in an audio file of a car revving as to pick which one.

• 15:18John Coogan?

  It has to identify the car?

• 15:20Speaker 01

  The car, yeah.

• 15:20John Coogan?

  From the engine note?

• 15:21Speaker 01

  From the engine.

• 15:22John Coogan?

  And it's not doing it currently?

• 15:23Speaker 01

  It's, no, it's wrong.

• 15:25John Coogan?

  This is a good benchmark.

• 15:26Jordi Hays?

  Humanity's real last exam.

• 15:28John Coogan?

  Yes, exactly.

• 15:31Speaker 01

  So I think those are pretty solid. I have some more, obviously I don't want to make them public in case anyone's going to try to, you know, benchmark this.

• 15:37John Coogan?

  Of course, of course.

• 15:38Speaker 01

  We'll see, hopefully.

• 15:39Jordi Hays?

  It's funny because I was mentioning the other day, this app that my dad had of like tracking that you just set your phone up and it just automatically detects which birds are in your backyard.

• 15:51John Coogan?

  Yeah, I mean, this has to be extremely solvable. It's just something that it reveals the lack of like general intelligence when you have to go and collect the horse dataset, which should just be out there, or the engine note dataset, which should just be out there. But clearly, we are in the age of go NRL on the individual problem, and we are looking at like the power law of capabilities. Knowledge retrieval is clearly a, you know, $12 billion a year market that consumers will pay for, that will probably grow significantly. And then health and therapy and shopping and all the other features that Fiji Simo laid

• 16:34John Coogan?

  out in her post, this is kind of like, you know, what will be RL'd against because those are key pockets of value in the consumer economy, and the same thing will happen in the business economy. But in the B2B context, you'll probably see an individual startup building on top of an API. But even then, most of the model platforms offer kind of RL as a service, fine tunes as a service, something where if you're starting to spend tens of millions of dollars, they will do some customization on top of the model. So that could be the regime for the next few years as we go into this like, you know, instead of like this centralizing AI force, there's only one company, there's actually like a Cambrian explosion of a ton of companies doing a bunch of different things. So, anyway, let's go to Signals post.

• 17:19John Coogan?

  Signal's not happy with the launch. He says, okay, I've seen enough. This launch felt like attending a funeral hosted by minimalists. They're unveiling tech that should feel magical, real breakthroughs, but the whole vibe was grayscale grief. The set design looked like if mood disorder got a Bauhaus grant. I don't know what a Bauhaus grant is exactly. Even the storytelling arc, chart styles, the eulogy, tributes, then closing on someone's health battles. What exactly are we, are we as the audience mourning? It feels like where they're trying to get you to pre-install a therapist. Potentially great products, sure, but the emotional tone was so damn DOA. Incredibly strange all around. I, I think, A, like, like, it's weird because we're in this world, and this is a question that I

• 17:59John Coogan?

  want to noodle on all day, is, is, will this be the last launch of a GP, of a numbered GPT model? Because, like, you don't hear about new, new versions of Google going out. You just, it just got better and better and better. Same thing with Amazon when they were optimizing for, hey, it's faster, we have more on our catalog.

• 18:19Jordi Hays?

  Our takeaway was, is that the product matters more than the model?

• 18:22John Coogan?

  Yes.

• 18:22Jordi Hays?

  Now, and probably will for potentially a very long time.

• 18:26John Coogan?

  And when we were watching the stream, I was cheering because they gave the feature of you can now talk to the model and get it to trigger a deep reasoning workflow, or get it to give you a quick answer in natural language. And so it's, it's abstracting even for even more of the UI into the actual text interface. And so I think in terms of like, surprise and delight and, and I don't know, it's like you, you, everyone kind of rips the Apple thing, but Apple does a great job of being euphoric and, and, and happy with somewhat minor product changes. And like, maybe that's more of where they'll go, is just, hey, there's these new features and here's how these things. And Apple will spend

• 19:10John Coogan?

  10 minutes on stage talking about like shifting an icon around and stuff. And it's like, people like it.

• 19:15Jordi Hays?

  I thought it was interesting. They just sort of casually mentioned that they're, they're deprecating the old model.

• 19:20John Coogan?

  I think it's great.

• 19:21Jordi Hays?

  Which makes sense.

• 19:22John Coogan?

  I think it's great. I don't want the model picker anymore.

• 19:24Jordi Hays?

  But a lot of people are going to be pretty upset about that.

• 19:26Speaker 01

  I think they're getting rid of 4.5.

• 19:28John Coogan?

  Oh, oh, really? And you're, and you're a 4.5 fan?

• 19:32Speaker 01

  I love 4.5.

• 19:33John Coogan?

  But, but I would imagine that the future is If I ask it to think really hard about the pros and the writing style, it would then do a pass with 4.5. And but it would only trigger that when it needs to. It's not going to give you that because if I'm just asking for, hey, regurgitate, regurgitate a bunch of facts or write some code or or put together a table of data. Like, it's not going to need to pull 4.5 off the shelf, just like it's not always going to pull Python off the shelf. It's not always going to pull web browsing off the shelf. And so I I'm not, I'm not sure that I necessarily want 4.5 there as a selection criteria. I would like all this to be tucked behind a UI and have something that's actually cleaner and less frustrating to use. I think it'll lead to higher retention.

• 20:18Speaker 01

  Yeah, for the average like normie. No one knows what 4.5 is like.

• 20:23John Coogan?

  That's true. That's true. Anyway, Chris Pike says OpenAI and Anthropic are duking it out. Meanwhile, consumer surplus is growing. We also have very good news. We also have the details from Mike Newp over at ArcAGI. Full GPT-5 is along the V1 Pareto frontier. That's cost versus performance. OpenAI said they focused on other goals like UX and reliability. Our testing supports this. Mini GPT-5 is super impressive accuracy for cost. In fact, based on cost efficiency, Mini could have entered ARC Prize 2024 and likely won first place. We are still verifying GPT OSS, or as Rune says, GP Toss. Results soon.

• 21:08John Coogan?

  Nano GPT appears overfit. Performance is commodity. And Francois Chalet is also chiming in with the top line.

• 21:16Speaker 01

  John, production team needs the deck.

• 21:19John Coogan?

  Oh, we don't have a deck today. We're just, we're just doing it live through. Yeah, we're just, we're just riffing through the timeline, the timeline tab and just pulling up some random posts. So, so you're free to pull those up, but also we can just read through them. Ashley Vance is saying, but, but model switching was my job. Model switching is out and we are into the future of just talk to the model. Just talk to the model and ask it what you needed to do and it will switch for you. It will pull the right tool for the job.

• 21:49John Coogan?

  Anyway, the other question that I have for the OpenAI folks today is on the nature of secrets. So in Zero to One, Thiel has this concept that discovering a secret is a key to building a startup. And it's a key insight. And I was joking with, you know, the superintelligence or GPT-5. Could my first prompt be, "Teach me exactly how to build GPT-5?" And then I go to Meta and I say, "I know how to do it. I have the prompt. I have the result." And of course, the answer is no. Of course, OpenAI would never leak the most frontier capabilities into the model.

• 22:33John Coogan?

  But can you build a superintelligence? Can you call it superintelligence if it can't tell you how to build superintelligence?

• 22:42Jordi Hays?

  One read on what the secret might be is that the app was the most important thing all along. And if you create this narrative that superintelligence is, you know, weeks or months away, and you get a bunch of people that go and try to compete on raw intelligence. Meanwhile, you build a consumer app business with billions of users.

• 23:05John Coogan?

  Yeah.

• 23:06Jordi Hays?

  It's like, it seems like a pretty good strategy. I guess one quick thought is, how do you rate Sam's vague posting from yesterday with the Death Star in the context of this new, in the context of the release today?

• 23:22John Coogan?

  It's a great question. There's a bunch of reads on it. One is just that, like, the Death Star is, to some degree, like, Stargate. And you have to, oh, wait, if this is the apocalypse, I figured I'd at least tune in live. You have to, like, the impact of this, of GPT-5, is not one crazy, superintelligent model that does everything. It's just a more user-friendly, higher-retention, lower churn consumer model that weaves its way into all aspects of daily life and improves performance and efficiency all over the place.

• 24:05John Coogan?

  And so you have to build this massive cluster to serve all of that? I don't know. What's your read on it?

• 24:12Jordi Hays?

  I don't know. I think it just was, I think it was dramatic. It was provocative. People didn't like it.

• 24:18John Coogan?

  It is provocative because there are many other like super mega structures that are in in sci fi history that are positive.

• 24:26Jordi Hays?

  More positive, yeah.

• 24:28John Coogan?

  And this is, this is like.

• 24:29Jordi Hays?

  But it gets the people going.

• 24:31John Coogan?

  Yeah, I don't know. Is it, is it a metaphor for someone else that's going to attack? Is he, I mean, I mean, the image is from the viewpoint of someone looking at the Death Star. Is he saying he is seeing a Death Star being built on the horizon? Is that something else? Is that another company, another organization? Is that the government? Is that the, is that, is that legal?

• 24:50Jordi Hays?

  Here's a read from Bubble Boy. Says, I am an expert on bubbles, so it brings me no joy to say that the AI bubble is popping this time next year. He is updating his timelines. When you promise infinite scaling and don't produce it, the calculus changes. I don't think it will be bad for most companies, but those who built their entire business model around making the best L L Ms are unfortunately going to struggle as models become more of a commodity. Again, OpenAI is a, my read on it is a consumer app business, right? They still have a big enterprise business, but, but by, you know, their recent valuations are predicated on their, this incredible consumer business that they built.

• 25:32Jordi Hays?

  Bubble Boy says the end user doesn't care much if Claude is 5% better than G P T 5. They care about cost, speed and utility, especially at scale, things will be going. The obvious play now is shorting Nvidia and dumping, okay.

• 25:44John Coogan?

  Getting into financial advice.

• 25:46Jordi Hays?

  Getting into financial advice territory here, Bubble Boy. But interesting, again, kind of goes back to what I was saying earlier in that if you were raising billions to make a lab and I think the potentially anthrop, you know, we'll see what happens in the coding market, but there's some clear winners emerging. And then on the consumer side, you know, expecting a power law outcome and, and it's hard to see anyone unseating Chat G P T there.

• 26:19John Coogan?

  Completely agree. I want to dig in more, but we have our first guest. Let's welcome him to the stream. What a day. Mark, how are you doing?

• 26:28Mark Chen?

  Hey, pretty good. Nice to see you guys again.

• 26:30John Coogan?

  What's happening? Congratulations on the launch. Take us through it. Are you, were you actually live or are you wearing the same thing and you recorded yesterday?

• 26:40Mark Chen?

  Actually live.

• 26:41John Coogan?

  Okay.

• 26:42Mark Chen?

  I don't know why, but we do.

• 26:45Mark Chen?

  Risk gone.

• 26:46John Coogan?

  I mean, we're a big fan, we're big fans of live. I mean, it just allows you to be the most reactive to the most new information. Give us, give us the core thesis that you were trying to get across. I think that there are a few narratives out there. We, we've been enjoying the one that's, you know, this is a dominant consumer product. They just made it a better consumer product and people are going to use the product and get more value out of it. I saw a bunch of things in the presentation where I was like, that's going to make my daily usage of, of chat GPT better. At the same time, we're in this, we're in this world of, oh, the models, the, the numbers matter and the scale matters and this and that and this and that.

• 27:27John Coogan?

  And, and it's, it's a fine line and it's a dance and we're in a transition phase away from benchmarks and away from talking about the size of the bubbles. But what was your core thesis? Like, what did you want to get across to the listener?

• 27:40Mark Chen?

  Yeah. I mean, fundamentally, I think from a research perspective, we've been working on reasoning models for several years now. And I think until now, you've had this really clunky interface. You have to pick, you know, GPT four O or you have to pick it with three. And for the longest time, we've known that O three gives you better answers across the board. It's just too slow, right? I mean, you often don't want to just sit there and wait for the model to reason it out. So we've done a lot of work to push the speed, the performance of our reasoning models such that these can come together and work in a very seamless way. And so I think, you know, above everything, we're trying to move the world into this agentic reasoning world. We believe that's the future. And on top of that, you know, you pointed something out, which I really resonate with.

• 28:25Mark Chen?

  Post-training is a huge part of this release. We really wanted to highlight Max Shortzer and his team who did a phenomenal job. And they've made the model just really that much more useful for consumers, for businesses. It's a monster at coding. So, yeah.

• 28:40John Coogan?

  On the, on the speed of reasoning.

• 28:45John Coogan?

  You're obviously the chief research officer. Are you more optimistic about getting speed ups there from, I don't know, algorithmic design, software optimizations, or new hardware, just let Moore's law carry on, or find new ASICs, or we saw Cerberus posting yesterday about the incredible speed that they're getting, 3,000 tokens a second on GPT OSS. And I'm wondering, what levels, obviously, we pull all of them, but what path of the tech tree are we, should we be like most focused around, most tracking, and most excited about?

• 29:26Mark Chen?

  Yeah, I mean, as a person who represents research, I control the things that I can't control. I think a lot of that focuses on algorithms, right? Simple algorithms that are scalable, that we can pump a lot of compute into. We also do care about the hardware improvements that are stacking up. With the open source release, you see thousands of people, right? Really kind of serving these models, creating really great inference stacks, and those are really great lessons for us to pull from. You know, how, what's the ceiling of the speed in which we can serve these models?

• 29:58John Coogan?

  What can you tell us about the actual, like, user experience of speed? I was, I, just like last week, I finally got to a place where like, for a lot of tasks, I'm firing off a 4.0 query and an 0.3 pro query.

• 30:18Jordi Hays?

  Yeah, and just have my, I have two tabs.

• 30:19John Coogan?

  Yeah.

• 30:20Jordi Hays?

  The 0.3 tab.

• 30:21John Coogan?

  Exactly. And I'm wondering, what user experience patterns you think can help people balance between those? Is this like, just something that we're, like, different patterns that we're going to learn over time? Or different, or are there going to be certain problems of user experience that are purely solved just by better product design, better speed, and we don't even need to learn these? Because I remember, like, you know, when you prompted a, an image generator, you used to have to say, like, don't know six fingers, five fingers, please, or like, don't make mistakes. And now, you know, the models kind of have that baked in. But how are you thinking about the user experience of getting the user the results

• 31:06John Coogan?

  in the right amount of time?

• 31:08Mark Chen?

  Yeah, I mean, this is one facet of why we believe so much in reasoning. It's just because all of the scaffolding you used to have to give the model, all of these small hints, they go away, right? Like, the model can examine its own outputs, it can review them, it can be like, hey, look, like, I'm just counting the fingers here, why are there seven? And it can kind of fix that, right? It does a lot of iterative generation, it does a lot of fixing things on the fly. And so we think one of the benefits of bringing reasoning to the world is really to kind of remove the need for scaffolding. And with GPT-5, right, we know how clunky that that experience is with switching between 4.0 and 0.3. Actually, I mean, there's so many stories. I was just talking to someone yesterday, right? They're like, hey, well, you know, I've used 4.0 my whole life, right? It's the frontier model.

• 31:53Mark Chen?

  And I'm like, hey, well, have you tried 0.3? And they're like, why would I try 0.3? You know, 3 is less than 4. And so, you know, to get out of that world, your GPT-5, I think it's a one-stop shop, reasoning and non-reasoning. And we've really tried to make it kind of just pre-optimal.

• 32:09John Coogan?

  Yeah, yeah, it's absolutely crazy to just take a bunch of letters and smash them together and expect people to pick up on that as a name or a brand. ChatGPT, TBPN, we're both kind of in the same insane gambit. But fortunately, it's worked out and I think people have gotten over the hump.

• 32:25Mark Chen?

  I see it rolls off the tongue, TBPN.

• 32:27John Coogan?

  Yeah, sort of, except our friend David Senra keeps flipping the letters. A lot of people do that. But at a certain point, yeah, you do break through, and ChatGPT has. But keeping the model numbers simpler makes a ton of sense.

• 32:42John Coogan?

  Talk to me about the pace of play for research to actual product, like.

• 32:48Jordi Hays?

  Yeah, and on that note, the line between your personal philosophy on the line between research orgs and engineering product orgs.

• 32:57Mark Chen?

  Yeah, I mean, so our research operates on a variety of different time scales, right? We have teams that they scope out a bunch of ideas and then they start to kind of narrow in on the promising ideas as they get closer to a run. And then you kind of see a winnowing of ideas as you get closer to launching a flagship model, right? And there's always this kind of like.

• 33:21Mark Chen?

  Explore more exploratory to more kind of concrete and execution focused pipeline. And we're pulling on ideas across the board here, right? There's a lot of work in architecture optimization. Seb was on stream. He pointed out improvements in synthetic data. So there's really a lot of work that goes into creating one of these models. And, you know, it's hard to say like, oh, this model was about this breakthrough. Just because right now we have this machine that's producing breakthroughs on all of these axes. And even across several paradigms, right? So it's all that coming together that produces the experience that you guys feel.

• 33:57John Coogan?

  Yeah, can you talk to me about the legacy or future of four point five? I remember I was talking to you and I was like, I haven't been using it a lot. And you looked at me like I was crazy. You were like, oh, it's so good. And I was talking to Tyler and he was like, uh, our, our intern here. And he was saying like, yeah, the people who really like understand how good it is, use it. But, but I was, I was wondering, is there a world where that is a tool in the tool chest for G P T five? In the same way that Python is or a web browser is. And if, if it detects that I want something with more, more emotional prose or more thoughtful writing, it can do a whole bunch of research, collect a bunch of raw, raw text,

• 34:42John Coogan?

  and then kind of do a four point five pass that I believe is more expensive, maybe. And maybe it doesn't make sense for every single query, but could be a, a, a, a feature in the loop or a tool that is pulled into the overall product experience.

• 34:56Mark Chen?

  Yeah, absolutely. Um, speaking of four point five, it's also a very smart model, right? And one of our bars in creating G P T five was to make sure that on a lot of the axes we cared about, that it was able to outshine four point five. And I, I think even in some of the soft ones like creative writing, I, I think that was the case. And, and that's what makes us so confident with, with the name. I think we're able to. Really rely on all of the architecture advancements, all the kind of post-training advancements, all the synthetic data advancements to create a model that's better than 4.5, but much faster and much cheaper.

• 35:32John Coogan?

  Yeah, it feels kind of like we're, I remember, wasn't the second iPhone called the iPhone 3G? And the number literally corresponded to a specific technology. And now, when you get the iPhone 14, it doesn't mean it's 14 megahertz or a gigahertz or inches big. Like, it doesn't, like, the number is abstract and it speaks to a bucket of features. And it feels like there's, I mean, this was the first day of kind of reeducating folks on what the nomenclature means going forward. Have you talked about an annual release schedule? Or like, or, or because there's the iPhone cadence and then there's the Google cadence,

• 36:14John Coogan?

  which was like, Google search just got better every year for two decades. I, it, it, it feels like at a certain point, you want to just be shipping as fast as possible. How do you think about the culture of shipping updates that, you know, you find something that feels like, hey, that could make the customer more delighted or the user more delighted? And we don't need to do a big training run for it, so let's get that out today. And let's tell people about it. Like, how are you thinking about fast iteration versus splashy announcements?

• 36:46Mark Chen?

  Right, so on the product research side, I think it makes a lot of sense to think about, you know, what's the cadence of release and, you know, what are the feature sets that we want to build? And I actually think there's enough great research happening there that we don't have to worry about, oh, you know, is there going to be a drought or a long stretch without enough features to launch? But one thing that's important for us is to be able to provide the people doing the exploratory work, some buffer from that, right? It's hard to do really great exploratory research in an environment where you feel pressured to do release after release after release. And so we let that be a little bit of a lazier pipeline, not meaning that the work itself is lazy, but we give it space really to mature and to flourish. And, you know, once it's ready, we can ship things across, across that fence. So.

• 37:31Mark Chen?

  That's kind of philosophically how we organize. We have a product research org, still very much entrenched in the research, and they care about the release cadence. And they're able to draw from all of the research that's happening, you know, algorithmically and in scaling and in RL.

• 37:46John Coogan?

  Yeah. Talk to me about tool use and how that's growing. I was, I was kind of noodling on this idea that, you know, the, I was, I was thinking about the IMO. And how, at least from the reporting, it sounded like OpenAI's model didn't use tools for that, and that's an incredible achievement. But it's kind of like artificial. Like, I don't, I don't care if the model doesn't use tools. I, I use everything possible. And even if, even if an LLM can, can memorize every fact, I'm fine with an LLM looking stuff up in a traditional database, spinning up a spreadsheet.

• 38:25John Coogan?

  Like, use whatever tool you want, just get me the correct answer. But do we have, i- is it important to give, to surface to the user the variety of tools that are in the GPT-5 tool chest? I noticed something magical happened when I was using GPT, I was using O3 Pro. I sent an image in and I asked to estimate the height of a desk. And it wrote like a thousand lines of, of Python image interpreter and was like, you know, interpreting pixels. And I was like, I didn't even think to trigger Python, it did. Yeah, yeah, yeah, no, it was right. It was crazy. But the, the really funny thing was that it was just a standard sized desk.

• 39:07John Coogan?

  It was just like, it could have just Googled like, how, how tall is an average desk or something? Or just memorized. It probably was just already in the weights that it knows that a desk is like 36 inches tall. But it, it did a ton of work and it still got it right. It fact-checked it a bunch of different ways. But, but, but, but I've noticed that now I can, I, I can pull different things. Make a table. Don't make a table. Write some Python for this. Don't write some Python. And it kind of gives me the feel of like a super user to some extent. But I'm wondering how you're thinking about what is further down. Like, you've given ChatGPT a computer. As Ben Thompson said, you've, you've given kind of the core tools, the Python, REPL, the, the, the web browser.

• 39:50John Coogan?

  How are you thinking about kind of the long tail of tools that you want to bring to bear? And how does that interface? I know that there's API integrations and all sorts of different surface area there. But give me some context on that.

• 40:03Mark Chen?

  Yeah, I mean, our reasoning models are pretty cute, right? I mean, I think they, when you look at their behavior, they know the height of the desk, but they'll still go verify it five different ways. If it's all consistent, give you that median answer. And I think that's really what makes these models so powerful. And when you think about tool use generically, right? Like, we want the models to use that reasoning ability to just be able to like zero shot a new tool, right? It should be able to kind of minimally get instructions about how the tool works and just be able to know how to use it, right? And humans do this all the time. You get a new tool, you start experimenting with it, and then you don't need too much scaffolding and you just go and go and use it and understand it. So we want our reasoning models to use their reasoning to be able to use a broad selection of tools.

• 40:47Mark Chen?

  And of course, there are a couple that you really do care about. You know, in coding, it's very important for you to be able to execute code. It's really important in personalization for you to be able to get context from your calendars and from, basically from the digital world. So I think there's a range of tools we have familiarity with, but beyond that, we want the model to be smart enough to just generalize and use tools zero shot.

• 41:09John Coogan?

  Yeah. Talk to me more about personalization. I feel like there's a world where I feel like I may be under utilizing ChatGPT as an app because I don't have it wired up to a non-relational database where it can just stuff data from, you know, it already has memory and it's doing kind of rollups and there's some sort of saving of context. But I was, when we were talking to Kevin Weal, I was, I was kind of like, well, like, I don't really have like a GitHub repo that's active that I want to like dump code in regularly for like my one-off tasks, but for that image generation, like, you know, understanding the height of the desk, it's like, well, if I'm doing that a lot, maybe I want to have

• 41:53John Coogan?

  a tool built that lives in the world that my chat interface can, can kind of interact with on an ongoing basis and contribute to and modify and, and kind of wind up instantiating a piece of software that's like even more long-lived and then every successive query is even faster. So, um, yeah, how do you think about, about different ways to increase personalization?

• 42:17Mark Chen?

  Yeah, I mean, I think memory is huge. So we have, we have teams surrounding memory and also personality. And when you look at memory, right? I think it's just, we have so much context built up about ourselves that the model doesn't have. And our memory team's been really hard at work. You know, there's a surface level of just gathering facts about you. But there's also stuff about just kind of thinking very deeply about who you are, what your motivations are. And even you could think about, you know, you're, you're trying to do some code-based tasks, right? You're a developer. Shouldn't the model just be trying code out, you know? And just kind of leveraging all that memory of kind of its thoughts about what you want to do to just help you kind of be doing work all the time.

• 43:01Mark Chen?

  So, um, yeah, we do think memory is a huge part of making the model more personalized to you. And it should just makes, make use of all that passive signal about you that it, that it observes or all of that interaction and, and just help you accomplish your goals.

• 43:15John Coogan?

  Got it.

• 43:16Jordi Hays?

  What do you think it'll take for AI to start making novel discoveries? That's been a critique over the last year as everybody's so excited. Everybody's using these products every day and in their work and life. And yet it still feels like we're missing that. Dwar Kesha's talked about, you know, potentially that being around continual learning, but I'm curious what you think.

• 43:40Mark Chen?

  So one thing to underscore is I think the models are already phenomenally creative in certain ways. So when I looked at our performance on, on contests, right? You know, I've, I've done these contests before. Sometimes you have this mental classification of these problems require more creativity or these ones require less. And one of the big surprises for me was that the model can get some of the ones which I intuitively think require more creativity. And, you know, it often does come up with these solutions that I consider quite ad hoc and really don't pattern match to anything I've seen before.

• 44:18Mark Chen?

  When you look at You know, advancing science or mathematics or fields like this. One thing that construct in which humans work sometimes is there are kind of three builders. In mathematics, for instance, there are mathematicians whose role are to kind of build out this theory and almost to kind of create, you know, Olympiad style

• 44:43Mark Chen?

  subproblems, which often other mathematicians who are very good at that kind of style of work can do. And I do think kind of the model will increasingly contribute on that side first, right? If there's some mechanical like, hey, you know, I, I really don't know how to simplify this expression. I really don't know how to like get, get this result. It can really do that quickly for you. We're trying to increase the envelopes such that the model's getting towards that theory building side and, you know, being able to create, uh, creative hypotheses. And all of these components are very useful for what I consider the ultimate goal, which is being able to automate some of our own work and our own research.

• 45:25John Coogan?

  How are you thinking about like the, the layers of mixture, of, of mixing? Like I remember GPT four, I don't know if this was ever confirmed, but mixture of experts model. This is kind of like widely understood in the industry. Now are we in the era of like a mixture of models that have mixture of experts? Like, how many mixtures are going on? How does GPT five actually work? Is there a, is there a taxonomy or, or architecture diagram that you can kind of like walk through to explain what GPT five is? Because it feels so much different than GPT three.

• 46:03Mark Chen?

  Yeah, I mean, one of our, probably the pinnacle of our research, or another path to AGI. When you look at the levels of AGI, the top level is what we describe as organizational AI. And what this means is, you know, collections of agents working together, often like we might in a company towards a shared goal, right? And you would imagine that these agents probably sub-specialize in ways maybe similar to what humans do, maybe in their own more efficient ways. And I think, you know, effectively work together to accomplish some goal. So we very much care about.

• 46:43Mark Chen?

  Exploring this vision, seeing that's much more effective than you one single big brain working on a problem. And I think there are reasons to think why it could be so. And, um, and yeah, I, I think that that is one of the things that we're after.

• 46:56John Coogan?

  Yeah. On that note of specialization, um, how are businesses working with GPT five? Or how do you expect them to work with GPT five in terms of, uh, coming to OpenAI and asking for special capabilities or, or, or fine tuning, or, uh, you know, any sort of RL on this particular problem in my world. I have this specific data set. It's not public, but I, I want a hyper, I want you to benchmark on it. I want you to, I want you to get a hundred percent on, on, you know, the gas station bench or whatever. Um, you know, if I'm, if I have a certain business and, and I'm, I'm willing to invest in sort of some, some overfit RL because it will create

• 47:36John Coogan?

  immense economic value for my business, or it'll solve some fundamental problem. Um, how can, how, how are businesses going to be using GPT five over the next few years?

• 47:46Mark Chen?

  No, that's a great question. So I, I think that, um, this is a chance to kind of highlight one of the, the results that we've accomplished over the last couple of weeks, uh, which is our at coder result. So this is, um, a relatively unknown programming contest, but it involves really the pinnacle of the, the best coding contests, contestants in the world. Um, and what they do is, you know, they're put in a room and they have to solve an optimization problem. This is something that's actually very real world, uh, relevant. So you can imagine an optimization problem as something like, you know, what Uber might have. Uh, you have, let's say riders and you have drivers and you want to kind of create a system where you match them as, as quickly as possible, you know, uh, with, uh, you know, the least amount of

• 48:31Mark Chen?

  cost for, for instance, and, um,

• 48:35Mark Chen?

  And so we've really created a system that can solve optimization problems at the level of the best in the world, right? And these truly are the kind of the best heuristic solvers in the world. And so we have an organization led by Alexander Madri. He it's called strategic deployment. And what they do is for a select handful of customers who really have that, you know, beefy problem that that they need to solve, to just go and provide that value, right? And I think there's a lot we can do there. I think there's a lot of very, very valuable optimization problems in the real world. And we're really excited to partner with with people because I think this creates

• 49:16Mark Chen?

  a template for directly having AI provide economic value and and really catapulting certain industries forward.

• 49:24John Coogan?

  On the on the research side, what what unique advantages do you think you and your team have, given your position in the market with the incredible user adoption and the incredible usage from those users? It's not just DAUs, but it's actually the number of queries, semi-analysis, estimated at like seventy one percent of all queries going through ChatGPT. What advantages does that confer from a research perspective?

• 49:57Mark Chen?

  Yeah, I mean, a lot, right? And I think, you know, it allows us to kind of deeply understand use cases. It allows us to understand the frontier of where humans are, you know, kind of finding value, where they're not finding value, which areas that we need to improve the models on. It gives us a lot of signal into, you know, how users are deriving value, when they derive value. And

• 50:21John Coogan?

  What is that signal? Like, I see the thumbs up, thumbs down button. I I'm sorry, I don't push it very often. I'm not doing my job, I apparently. But I know that you can figure out whether or not I'm satisfied. Just stop booing me, Jordy.

• 50:36Zach Lloyd?

  That's the research.

• 50:37John Coogan?

  Okay, Mark, I promise you, for the next one hundred ChatGPT responses, I will I will be honest with my thumbs up, thumbs down.

• 50:45Mark Chen?

  I love it.

• 50:45John Coogan?

  Just to help you do extra training.

• 50:47Mark Chen?

  We have tons of people luckily who do.

• 50:50John Coogan?

  Oh, that's great. Okay. So you do get a lot of thumbs up, thumbs down. And I'm sure I have done it occasionally. But I, but I also imagine that there's a ton of other signal in there. You know, with the TikTok algorithm or, you know, any social algorithm, it's very easy. Time on site. But with chat GPT, obviously it's exciting when we hear, okay, 30 minutes a day or some rumored number of, of minutes, it feels correlated with usage, feels correlated with value that's being delivered. You can obviously look at churn metrics and all that stuff, but what other, what other pockets of signal are you finding? Are you finding people just, I mean, I, I remember the story about Google where they were trying to figure out how to handle like misspellings and create the, the definitive database.

• 51:33John Coogan?

  Do you know the story where they were trying to develop the definitive database of how to spell things? And they were like taking a bunch of shots at it and they figured out that the, the best, most rich source of data was just if you type in financial into Google and you misspell it. Oftentimes then you will just correct it yourself and the second query you send will be, will be spelled correctly. So they can just look at two similar queries. What's the second one? That's the correct, that's the correct spelling. So yeah, what other pockets of signal are you finding that are translating into the research environment? What are you excited to go deeper on?

• 52:06Mark Chen?

  Yeah. So I'd love to first talk about the DAU signal because I think, um, you know, that's something that a lot of companies track, but we find actually a lot of danger in tracking it too closely. And one of the recent blog posts we pushed out was one on sycophancy, right? If you just, you know, hey, we're gonna boost responses where users say thumbs up. You know, it creates a, a condition for a model.

• 52:33Jordi Hays?

  I just want to say, Mark, I love everything you're doing on this front.

• 52:37John Coogan?

  Yeah, the entire interview has just been fantastic.

• 52:40Jordi Hays?

  You are the best.

• 52:42John Coogan?

  We'd love to have you back on the show tomorrow.

• 52:47John Coogan?

  But clearly problems with that.

• 52:49Mark Chen?

  Yeah, yeah, could could problems, right? The model just starts kind of sucking up to you. And it's saying like, hey, you know, you're right. And even in complicated situations where I think objectively, you know, collectively, we'd be like, hey, this person's in the wrong. The model starts saying, hey, you know, you're right. You know, the other person's gaslighting you. You know, this other person's kind of and and.

• 53:08Jordi Hays?

  And people deal with people deal with this in the real world. They'll go to a friend. They'll tell them about a situation and the friend will give them advice. But maybe it's not the entire, it's not the fullness of the situation, right? Maybe they left out some key facts and the friend is like, oh, yeah, that other person is wrong. Definitely isn't the wrong. And they like skipped over some important details and.

• 53:25Mark Chen?

  Yeah, no, no, exactly, exactly. And we don't want our models to fall into this trap where it's just trying to get you to like, you like what it says. And and so, you know, we wrote back a lot of changes that produce that kind of behavior. And really, the way I think about daily active users today is we need to be opinionated about the features that we build into the future. I think we have, we have a lot of ideas here, but we have to let that drive. You know, build for the future, build for the things that people you think they'll want and maybe don't want necessarily, know they want necessarily today. And then use the A U as kind of this byproduct, right? A way to track that you're on the right, right, right track here. So, um, yeah, I mean, we, we want to be careful here.

• 54:08Mark Chen?

  We don't want to fall into these traps of like, you know, three, four years from now that this turns into kind of engagement bait or something like that.

• 54:15Jordi Hays?

  Yeah. Was it, uh, how, how much time have, has the research team been focused on efficiency specifically? It felt like summer was a, a good window before kids come back to school and start, you know, maxing out queries. Good time to increase efficiency. And, and I know the cost of G P T five.

• 54:37John Coogan?

  Every time there's a new model, I'm like, this is the best it could ever be. It's good enough. Bake it on an ASIC. I just want it for free and I want it like in milliseconds. But that's just me being grumpy, I guess.

• 54:49Mark Chen?

  We've done a lot of work. We've been building out our teams. We focused a lot on scaling. I think Greg's gonna come on a little bit later.

• 54:56John Coogan?

  Yeah, yeah.

• 54:56Mark Chen?

  He's been spearheading a lot of that work. So, yeah, no, honestly, it's become a bigger and bigger focus for us, especially in the last couple of months.

• 55:05John Coogan?

  On the, I mean, this is somewhat related to the sycophancy thing, but I'm interested to know, like, what do you think is driving, like, the GPT tone? You know how, like, the M dash is a thing, and then the, the, it's not a newspaper, it's a way of life? And it's like, there's these, like, little, like, flourishes, like, that come through in kind of a tell that it was written? And in a lot of ways, I love it, because when I get a deep research report, I like that it's using the same Wikipedia-style tone. Like, I want consistency there. I don't want it to be like, oh, this today, it's, looks like it's a Vice News article, and the today, tomorrow, it looks like it's written by someone at BuzzFeed. I like that it's consistent in many ways. But, but why is that happening?

• 55:50John Coogan?

  Do you think that bigger models, like 4.5, kind of were able to solve that? Or do those kind of, like, local minima, like, I don't know, like, wells happen even in bigger models? Is there anything from a research perspective that can, that can stop GPT having its own voice? Or is it fine that it has its own voice?

• 56:10Mark Chen?

  Yeah, that's a really great question. And I think, you know, as you scale up models, as the models become more intelligent, they kind of have a just deeper innate understanding of tone, right? And so you expect that to improve, just naturally, as you make the models more powerful, bigger, better reasoners. But one thing that I think gets lost a lot is each individual company has a lot of impact in terms of how they shape the default tone. And, you know, we publish a document called the spec. It kind of lays out how we expect the model to sound in certain cases, lays out a lot of examples for that. And I think we use the spec in many ways, right? We have people come in and see, hey, was this thing generated in accordance with what we would hope to generate from our spec?

• 56:55Mark Chen?

  And this is a living document, right? It evolves over time. And so I think, you know, each company kind of has a very opinionated take on what they think the model should sound like. And it's not an accident that the models sound a certain way. I don't think, just naturally, every company is going to train the same kind of voice into their model.

• 57:16Mark Chen?

  Totally.

• 57:16John Coogan?

  Well, thank you so much for hopping on. Congratulations on the big launch. We'd love to have you back soon to talk more. We could go in a million different directions, but we'll let you get back to it. We know it's a big day. So have a great rest of your day.

• 57:27Mark Chen?

  Thank you so much. It was a great conversation.

• 57:30John Coogan?

  Talk to you soon. And we will tell you about Restream, one live stream, 30 plus destinations, multi-stream and reach your audience wherever they are. This stream is made possible by Restream. OpenAI just did a live stream. If you're trying with Restream, if you're trying to do, if you're trying to do a stream, you got to get on Restream. So it's everywhere. And we will bring in our next guest, Greg Brockman, the president of OpenAI and the, we'll bring him in. Greg, how are you doing?

• 57:57Speaker 05

  Doing great.

• 57:58John Coogan?

  Well, congratulations. How are you feeling? How's the company feeling? It's been such a wild journey. Just take me through a little bit of the, the like, the vibes and the company and how you got here today.

• 58:12Speaker 05

  Well, I'm excited. The whole company is excited. And honestly, I'm just so proud of the team. Like, it's just been amazing to watch people come together, not just for this launch. And, you know, the funny thing is behind the scenes that people are always putting on the last minute adjustments and polish and scaling up the capacity. And there's always something that goes wrong before launch day. And so there's a lot of people who, who, you know, worked late into the night or really crunched to bring this release to the world. And, you know, it's a little bit like the duck that's, you know, you know, running its legs under the water.

• 58:45Speaker 05

  But that also describes the whole OpenAI history, right? Is that I think that we have put in many years worth of investment to the techniques used to produce this model. And really, it's across just every function within OpenAI that has come together to make this a reality.

• 59:00John Coogan?

  Yeah. I mean, you've been there for every GPT release. How do you think about summing up each iteration in kind of like one, one line? Because GPT one, GPT two, GPT three, these feel like, like similar architectures, at least, at least histories kind of compress them into similar architectures. But how do you think about the progression of just the big numbered releases?

• 59:27Speaker 05

  Yeah, it's interesting because in some ways it's a punctuated equilibrium, but on the inside it looks very smooth. Right? Even before the GPT series formally began, the first result that really sort of set this path to be something that we were heading down and that was clear that we were going to pursue it was the unsupervised sentiment neuron, which was an LSTM in like 2017, so a different architecture from today's transformers. And it was the first time that you could train a model to predict the next element. So we predicted the next character on Amazon reviews and we were able to get semantics out, right? Because you expect, okay, yeah, it's going to learn where the commas go, what maybe what nouns and verbs are. But the idea that it would learn a state of the art sentiment analysis classifier, that was mind blowing.

• 1:00:11Speaker 05

  And so I remember seeing that result in 2017. It's like, we have to scale this up. We have to see where it goes. And so GPT one was like, I think a good like sign of life of you train on sort of all the public data you can get. And you use transformer and that you were able to get state of the art on various downstream benchmarks, right? So you have a model that clearly learned some representation, something useful about the data that it was shown. And it's applicable. You can use it for various tasks. But we didn't really think very hard about the generation side. GPT two was the first time that we were like, all right, let's actually like the samples we're getting from it, the things it actually generates. They're kind of cool. And I remember reading the, in the GPT two blog post,

• 1:00:53Speaker 05

  we have this unicorn story where it generates some fictional story about a herd of unicorns. And it was just so cool. It was like, wow, it like wrote a story. It's actually kind of interesting. It doesn't totally make sense, but like there's something here. There's some real spark of intelligence within this model. GPT three was the first time that we had a model that was actually something people would, it was just barely above threshold for something people would want to use. And I remember working on the GPT three API. This was our first real product. And it was actually the hardest product, the hardest project in total I've ever worked on. Because it just felt like maybe no one wants to use this model. We don't really know what it's useful for. And it certainly was the case that GPT three was a great demo machine. You can make really awesome, just like.

• 1:01:37Speaker 05

  Tweets and you know, cool little little apps and it would give you quick answers, but it didn't feel very reliable. And then G P T four was something that actually felt like it had true real world utility. It was above some threshold. It was something that was helpful for health. It was something that was helpful for, you know, starting to be good at coding. And G P T five, I think just sets a whole new standard for the reliability, for utility. Things like coding, I think, are just like clearly, you know, we're already on this trajectory of transforming software engineering this year. I think are really on the trajectory now to be revolutionized. So just really exciting to see that that whole arc.

• 1:02:10Jordi Hays?

  Yeah. When did, when did, uh, when did the A P I opportunity like really click for you? Because I do remember companies in that era that like quickly unlocked the power of, of the A P I and, and grew tremendously. When did that opportunity click? Because you said initially that you, you kind of had some, I don't know, concerns, kind of doubts, how useful was it going to be? And then when did the consumer opportunity click?

• 1:02:36Speaker 05

  Well, we, in twenty nineteen, end of twenty nineteen, had G P T three. We knew we needed to build a product to be able to actually continue the mission, to be able to raise capital. But what did we want to build, right? We're really here because we believe in A G I, that it's going to have this powerful, positive transformative effect on society. We want to be part of it. And so we thought, well, maybe we could build something in health. And then you realize, okay, well, we're going to sell to hospitals and we're going to maybe hire.

• 1:03:02Jordi Hays?

  Let other people do that.

• 1:03:04Speaker 05

  Exactly. Right? It's just like you have to go into one domain and that means giving up on the G, the general, right? It's like, it feels like you're going to become a one particular thing, but we kind of want to be supporting all industries at once. And so the idea was let's build an A P I and let people figure it out. But this is totally not the way you're supposed to build a startup, right? You're supposed to have a problem. No one cares about the technology behind it. Add value to that problem, focus on just that one thing. And so that's why that project was so hard. And in, you know, January of twenty twenty, February of twenty twenty, I, the, you know, I with the team were going around trying to just find anyone that would be willing to try this A P I. And we were driving to different offices in, in San Francisco, being like, hey, we have

• 1:03:47Speaker 05

  This cool model and it was hard enough to get people to take the meeting, much less to sign up their company for it. It was actually very fortunate. We found it. We found a couple of good partners. And it was fortunate that that happened then because March twenty twenty, suddenly that was COVID. We weren't driving around to people's offices to try to beg them to use this, you know, this, this budding new technology. So it was really six months worth of grind, right, of really trying to turn like when we, when we started with G P T three, I remember it was, you know, that the inference code was not very well optimized. It was like, I don't know, a hundred fifty or maybe two hundred fifty milliseconds per token or something. And we just optimized, optimized, got it down to like fifty milliseconds per token, which by the way, today's models run much faster than that, which is kind of amazing for me, just like seeing how, how much faster we're able to run them with much greater intelligence.

• 1:04:33Speaker 05

  And I remember setting two goals for the team. One was. I actually find one customer who's willing to pay. So literally get a dollar in for this thing. And the second is get a use case that we use at OpenAI every day. That first one happened within the first couple of months. So actually that moment I was like, all right, like this thing is probably gonna work. But in order to get there, we had to do a bunch of, you know, just scaling the A P I and really, you know, doing, doing the product work. But that second one took much longer, right? And that wasn't really until Chat G P T. And so if you fast forward a couple of years, because this was, you know, mid twenty twenty, when we, when we first got that, the A P I into the world. Chat G P T, we didn't release until November of twenty twenty two. So you're talking like a decent, a decent period of, of two years there, a little bit longer.

• 1:05:17Speaker 05

  And I remember we were building, you know, people have talked about, we were going to call it maybe chat with G P T three point five. We had a, a sort of precursor product called, called Web G P T that was built on, on, on three point five, that we were literally paying contractors to use. Right? So this was all throughout twenty twenty two. We basically had the Chat G P T precursor that we had to pay people. They would not pay us. We had to pay them to use this thing. And

• 1:05:45Speaker 05

  the moment for me that really clicked was actually when we finished training G P T four. So that was August eighth of, of twenty twenty two, which actually is like three years ago now. It's actually pretty, pretty wild to realize that, almost to the day. And We did the initial post-train of G P T four and honestly, I had a bunch of bugs in there. It was like broken for a bunch of different reasons. But the model was like extremely creative. It was actually really interesting. It took us like about a year and a half to get to the point that the creative writing of our models matched that initial one that was buggy for various reasons. And I remember, you know, we had an instruction following data set that was post-trained on. So it's really, we had collected examples of here's a human asking for a thing. Here's what the model should do. So it's really not trained to do multi-turn.

• 1:06:31Speaker 05

  So I asked it a question, it gave a response. But then I was like, well, what if we just ask another question? And it actually was able to leverage that full context. It actually was able to have a coherent chat. And the moment that we saw that, they were like, okay, this thing is capable not just of being post-trained to do this like very specific thing, but it can generalize, right? It can kind of do the intelligent thing, even though it wasn't directly trained for it. It was just so clear this was going to be the killer killer application. And so then we were planning on launching G P T four in, you know, early twenty twenty three. And we had this chat infrastructure we'd been working on and it's so clear.

• 1:07:13Speaker 05

  Okay, like we're going to have to release the infrastructure and the model. And it's going to be this, this amazing killer product. And so just almost as infrastructure ahead of getting the, the real thing out. You know, I was excited for us to do chat G P T and that's why we did, you know, then see that come to life in November. So I think that for me, I, I was really focused on G P T four as the model. This is going to be the chat moment that's really going to work and kind of had missed the fact, because every time you see these new models, you just sort of, you know, see only flaws in the previous ones. And so I missed the fact that G P T three point five was something that no one had really tried before in, in the broad sense of society. And it was something that was already useful and that people would respond to.

• 1:07:54John Coogan?

  Was GPT-3 kind of like the main pivot point for shifting the company towards LLMs? Because in the pre-history of OpenAI, there were a lot of other maybe expensive training runs. I don't know how much, I don't know how much financial risk was taken with like the, the OpenAI 5 project or the robotics projects, but it feels like at a certain point, the, the chat became like the main financial risk vector. So I guess the question is like, when, it feels like GPT-3 was the moment when you shifted. I'm also interested in hearing about, Ben Thompson called OpenAI the, the accidental consumer company. And

• 1:08:34John Coogan?

  I'm wondering when that narrative set in for you, like what, when, when did it become clear that this was going to be a really, really powerful consumer application?

• 1:08:47Jordi Hays?

  Yeah, going from paying people to use your product to people saying, hey, we want to give you money for this.

• 1:08:51John Coogan?

  Yeah.

• 1:08:52Speaker 05

  Yeah, a very important transition, it turns out.

• 1:08:55John Coogan?

  Yeah.

• 1:08:56Speaker 05

  So, um, it's, yeah, it's a, it's a great question. I would say that if you rewind to the beginning of OpenAI, you know, there's many people who thought that, you know, in retrospect say that we set out to prove that scale is how you make progress in this field. But it's almost the other way around. Scale was the thing that worked, right? That we tried a bunch of things that didn't pan out. And it really, the first time we saw this concretely wasn't our Dota project. I remember my collaborators, Jakob and Shimon, trained the very first little agent on like 16 cores or something and left it running on their desktop over the weekend. And we came back and it was, this like very, you know, sort of constrained mini environment, but that the model was doing something smart. It was actually able to, to solve this, this kiting environment. And that was pretty cool. And

• 1:09:41Speaker 05

  then they just, they and the team just kept scaling up, right? That we had all these free cores that were just sitting idle on, on, on AWS at the time. And they just kept throwing more compute at it. And every time they would do that, the model would just get better. And so when you look at something like that, you're like, well, you just have to see where this goes. You have to push it until it hits the wall, right? And our goal with Dota was actually to develop new reinforcement learning algorithms. Because the common wisdom at the time was, well, the existing reinforcement learning PPO, it doesn't scale. Everyone knows that. But the question from Jakob and Shimon was, Well, why do we believe that? Has anyone actually tested it? And no one had really tested it. And so I think that that ethos of saying you have to push the existing techniques to the wall until they break.

• 1:10:24Speaker 05

  And then, once they break, you actually have a baseline to overcome. And you win either way, right? Either it just exceeds all the humans in terms of the specific capability that you're trying to exercise, which was the case for Dota, or it hits a wall and now you have a real problem to solve. And so I think that ethos really got embedded in our DNA. And, you know, at the same time, I think that we were really thinking about how do we get to AGI, right? And really, like, Ilya and I spent a lot of time thinking about that question of where's this company going and how do we actually, how do we actually achieve it? And you start to do some math in terms of, you know, the kind of compute that it would take to get to AGI, and you just start to realize you're gonna have to build really big computers, and those are extremely expensive.

• 1:11:08Speaker 05

  And so I think that from the, from these early foundational results and thinking, we kind of realized the path that we're going to have to walk.

• 1:11:15John Coogan?

  So it seems like there's been a few walls that we've scaled up through and then maybe hit them. There's been talk of like a pre-training wall. Now we're putting tons of resources and compute towards reinforcement learning. Is there a third, is there a third scaling curve that we're going to be talking about in the next few years? Are we continuing to scale up those two primary vectors? Is that too high level of an abstraction in terms of like how we should be thinking about just progress along the vector of scale? Like, give me the up-to-date thinking on just the fruits of scale.

• 1:11:53Speaker 05

  Yeah, I'd say fundamentally, deep learning, I think that, you know, people talk about the bitter lesson.

• 1:12:00John Coogan?

  Yeah.

• 1:12:00Speaker 05

  It's almost this exploration into how do you convert compute into intelligence, right? Through a, you know, we have some particular techniques to do that that we're kind of constantly fleshing out. And the thing that's really amazing is if you rewind to, I don't know, even the 1940s... For the McCullough-Pitts neuron, which is kind of the precursor to neural nets. If you look at that paper, they have all these diagrams that actually look very similar to like the kinds of diagrams we draw now of multi-layer neural nets and things like that. Like, the basic idea of what we're trying to do has not really changed in almost like eighty, eighty-plus years, which is just a wild fact, right? It means there's something deeply fundamental about the thing that we are pursuing. And that idea itself, I think, kind of came from trying to model the information processing of the brain.

• 1:12:44Speaker 05

  And it's imperfect and not an exact analogy to biology. And all of these reasons that it should fail or that people have said this thing is doomed. But the results are undeniable at this point. I mean, some people try, but it's hard to, it's really hard to kind of close your eyes and sleep on this, in my mind. And it's very interesting, if you look at, you can find quotes from the mid-1960s of people trying to poo-poo the whole direction, saying that these neural net people have no new ideas, they just want to build bigger computers. And you can basically say something, something very similar today. What we're, what we are trying to do. One moment. Well, water break. Yeah. Exactly. For all of us. Cheers. Exactly. Cheers.

• 1:13:28Speaker 05

  I tell you what, we're all human. Proof of humanity right there. Exactly. So what we're all trying to do is find novel ways of taking compute and really harnessing it. And sometimes you hit a wall, but these walls tend to be ones that you can drill through, right? What we've found is every time you scale up, everything, all of your engineering, all of your sort of scale and variance, all these things, they get stressed to the next level. It's almost that the tolerance has become tighter and tighter. It's like launching a ten x bigger rocket means you need to be like a hundred x just more precise on everything. But it doesn't mean that the fundamentals of the science are different. So, pre-training, there's definitely been a lot of discussion of data wall, doesn't mean it's fundamental, right? It just means that we need to be better and more precise in what we're doing.

• 1:14:12Speaker 05

  There's RL, which has been something that has kind of come from spending a small amount of compute to much larger amounts of compute now. And then there is a third way that we're really harnessing compute, which is compute at test time. And we published some scaling laws around this. And all three of these things multiply. Like, that's the amazing thing. And of course... The compute and the harnessing of it is the fundamental goal. But you get these multiplicative effects out of all of it through the quality of your engineering implementation, right? Through the quality of the data sets, through a bunch of the refining work that you do. And there's lots of different techniques and ideas. And that's what makes this field so rich and why progress is just going to continue a pace.

• 1:14:51Jordi Hays?

  What about on the infrastructure side? You guys have been busy scaling up. What can you share on that front?

• 1:14:58Speaker 05

  Well, so I run a team called Scaling at OpenAI. And we really focus on building the infrastructure for Scaling. And this is in partnership with really everyone across the company. It's almost a misnomer that our team is called Scaling because fundamentally, this whole team and effort is about scale. But what we really try to do is to both on the physical infrastructure side, deliver as much compute as humanly possible. And that is in partnership with companies like Oracle, SoftBank and others that we've been able to deliver just like increasing amounts of compute to OpenAI. But we're constantly thinking about how do we just deliver more flops and do it more efficiently, earlier, cheaper, more power efficient, all of those kinds of questions.

• 1:15:42Speaker 05

  There's the software infrastructure side as well. And really thinking about how do you coordinate massive numbers of GPUs in order to work across one synchronous training run? How do you coordinate that for reinforcement learning? How do you deploy that into production and bring these models to life at massive scale? And I think that every single layer of a stack, there is innovation required. And that's something that's very easy to miss. Like, one way I think about research is that there is, and this is kind of the view from Jacob, who's now our chief scientist, that there's a research stack and you can kind of think of the top of it is, you know, people running experiments and coming up with new ideas for how to, you know, sort of utilize data or something like that. There's a middle of the research stack of people thinking about the, how do you sort

• 1:16:27Speaker 05

  of take these different ways people are running experiments and be able to train in novel ways and kind of put together the pieces differently? And then there's a bottom of the research stack, which is like writing CUDA kernels to get the absolute max out of the GPUs. And at every single layer here, you get a multiplicative factor through innovation. So it all comes together as one big whole.

• 1:16:49John Coogan?

  On scaling, I'm interested to hear about just if we think about like the impact of AGI or the impact of AI just being some sort of maybe you quantitative GDP metric or qualitative just impact and good. Is there an important factor of scale with just not even the flops that are going into the models, into the pre-training, into the RL, into the test time inference, but actually just the flops that are going into the usage of AI within humanity broadly? And I feel like that might maybe be the next like scaling curve that we're seeing as more

• 1:17:31John Coogan?

  people use models, they see improvements all over the fact like is is that something that we should be tracking to see kind of the the instead of these like S curves, we want to see like the continual exponential?

• 1:17:46Speaker 05

  I think that's a great perspective, right? Because at the end of the day, I mean, if you look at kind of the shift from something like Dota, which we pursued in order to, you know, we wanted to do new algorithm development, but really it almost validated how we scale up existing algorithms. But there was no illusion of delivering direct economic benefit from it, right? To the current models where we are still, we're starting, starting to end the era of like pushing on these academic benchmarks, right? You look at things like the IMO, at this point, models are able to get gold medal on it. Like these, the hardest academic benchmarks that are available are sort of no longer a sort of the guiding, the guiding light of progress for these models to where we actually

• 1:18:29Speaker 05

  want to be is for AI to be helping everyone, right? To be something that uplifts humanity. And that's the final metric, right? Is how much does it actually benefit everyone? How much value does it bring to the world?

• 1:18:38John Coogan?

  Yeah, not just health bench. It's actually how many people, did you solve their healthcare problem, right?

• 1:18:44Speaker 05

  Exactly. Yes, yes, yes. And that's the actual goal. And that's what's exciting, right? Is it's like we're moving from the lab to reality.

• 1:18:51John Coogan?

  Yeah.

• 1:18:52Speaker 05

  And I remember in the early days, as we were thinking about how do we measure our progress towards AGI, we always sort of dreamed that one One day, we would be able to measure it this way. And you can think of revenue maybe as a proxy metric for value delivered to the world. It's not perfect, but it's at least something, right? You can think of the distribution of like how much compute goes into it, how many people are using it. But fundamentally, like what we're after is how much do we really uplift humanity through this technology?

• 1:19:18John Coogan?

  Yeah, I mean, I might be misreading him, but I'm pretty sure like that was the Kurzweilian, Ray Kurzweil philosophy was that like total number of flops getting immense, not necessarily all in one data center for one model. It was that it was that compute broadly would be so wide.

• 1:19:35Speaker 05

  Yes, yes, and I remember like on that chart, right? You can see, you know, total compute of all human brains, which really suggests a particular vision of how this technology will be rolled out.

• 1:19:47John Coogan?

  Yeah, distributed, the phones count as as an impact. The Wi Fi router counts for the impact of the Internet, just like the phone does. Not just, it's not just the big pipe that's going, the backbone of the Internet that actually matters.

• 1:20:01Jordi Hays?

  Deep research, hit product, almost everybody I know, at least in the in the

• 1:20:08John Coogan?

  Mark Andreessen says he's reading thirty pages of deep research a day, basically. He loves it.

• 1:20:13Jordi Hays?

  He's making books with it. But why have agents broadly come around a little bit slower than than people may have expected? Is it is it is it just that using computers is actually much harder? Computer use is just a really hard challenge or, you know, I think going into this year, everybody said this was the year of agents.

• 1:20:34John Coogan?

  Are you talking about flight booking or something like that?

• 1:20:36Jordi Hays?

  Yeah, I mean, flight booking, but you know, people, people were saying twenty twenty five is the year of agents. And I would say that it's the year of deep research and not a lot of these other sort of like broader use cases.

• 1:20:46John Coogan?

  Sure.

• 1:20:47Speaker 05

  Well, twenty twenty five isn't quite over yet, so that'd be my response.

• 1:20:55Speaker 05

  I'm very much on the, I think that progress in this field, the way that it tends to work is that if something kind of works with the current generation of models, it will be extremely reliable with the next generation of models. And I think that where we've been is that deep research is the, if you've rewound a year, that was the like, we kind of had something working. And then like this year, it's been just incredible. And I think that agents, you know, specifically like computer use agents are something we've kind of had working. And again, you know, the year is not over. I think there's a lot of rapid progress to be made. But I think that maybe part of it too is that the agents that we're about to see, I think are a little different from maybe what we would have pictured five years ago. Like I remember having a debate with some friends on, do you want a agent that does the flight booking?

• 1:21:39Speaker 05

  Because the problem is it's actually a very high bar to beat the flight booking UI because there's so many preferences that are entailed in that, right? You really have to know kind of what mood you're in, like, are you okay with like taking the extra layover and all of these kinds of questions? And that actually, there's so much other stuff that happens in your life that is toil or drudgery, or that's something that you're not an expert in, you're supposed to be. Think about health, right? That like every patient really is the doctor. If you're coordinating across multiple specialists, there's no doctor that helps you with that, right? That that's really on you. And that there, you actually can have AIs that are just text only, that actually are able to add massive value. And then it frees up your time if you want to go, you know, book the flights yourself.

• 1:22:21Speaker 05

  And so I think that really finding the right problems that have high leverage, right? That really add value to people. And also thinking about the other side of how to make sure these agents are responsible with the trust that you put in them, right? That the more that you give an agent access to your email, the more you really have to trust that it's going to, you know, sort of do right with whatever your task is and send the right email to the right people and be able to say, you know, segment your information and all of these kinds of questions. And so I think that there's both a practical, how do you get to adoption? But also just like, where are the most important leverage points in a person's life?

• 1:22:59John Coogan?

  You also missed coding agents because it's been the year of deep research, but I feel like it's also been the year of coding agents. How is that developing at OpenAI? I've noticed that I'll hit O3 Pro and it'll wind up writing a bunch of code for me and I didn't even ask it to. Then you have specific products for coding. How do you see the evolution of software development evolve? How are you seeing OpenAI customers use coding tools? And how good is ChatGPT or GPT-5 on coding?

• 1:23:32Speaker 05

  Well, software engineering is definitely being revolutionized in front of our eyes. It's been happening and GPT-5 is the best coding model in the world right now. It's the default now in Cursor, which I think is a really huge statement of the quality of the model. And that it's just so good across like every function of writing code, understanding code base, being able to use tons of tools, being able to do agentic work. That, yeah, it's like, I'm not a front-end developer at all, but actually now I am, right? And I think that you are too, right? If you just talk to the model, you can produce incredible things. And so I think that there's this real empowerment. If you think about what computers were supposed to be, right? Computers are supposed to be a tool that makes you more productive, able to do the thing you want.

• 1:24:17Speaker 05

  But then somehow, when we started out with computers, we have to contort the human to the machine, right? Writing assembly language and like all these like very abnormal things for a human to do. And that as we've moved to tools ultimately, you know, in the current generation now GPT-5, suddenly the computer comes closer to you, right? That you just express your intent and you don't think about, okay, like exactly which language and what, you know, version of different libraries, that the model is something you can delegate to. And so we are very committed to programming and to making our models continue to be the best they possibly can be.

• 1:24:51John Coogan?

  Must a superintelligence be able to explain how to build superintelligence?

• 1:24:59Speaker 05

  So it's a great question. So I mean, I think that where we're going is a world, and we're already seeing it, where these models help us produce the next generation of models, right? They also help us really supervise tasks that are too hard for humans to supervise on our own, right? If the model writes a 10,000-line program for you, reviewing that is probably going to be quite burdensome. But if you can have a model that you trust, that maybe isn't as capable as the one that wrote all that code, or maybe there's a team of agents that work together to write all that code, but you have a team of reviewer agents, like this is the kind of thing that you can actually bootstrap trust. And I think that this is like one of the most important things. And also, interestingly, 2017 is when we had the first language results. We also had some results or some vision on how you can actually

• 1:25:43Speaker 05

  bootstrap supervision beyond the scale of tasks that humans are able to supervise directly. And so I think that we're heading to a world where, you know, we now have these chain-of-thought models. We've been advocating very strongly to preserve the integrity of the chain of thought, right? So that means don't directly optimize it to look good, you know, there will be lots of temptation to do it for various reasons. Really make sure that there's no pressure on the model to obfuscate its thoughts within that chain of thought, because then you can really see what it's up to. And I think there's further techniques to even make it more faithful and more rigid to what the internal monologue of the agent is. And so I think that there's actually a lot of promise in terms of interpretability, in terms of supervision, in terms of being able to scale to just like much more sophisticated tasks.

• 1:26:27John Coogan?

  Yeah, I guess my question is like, how much information in the world can be derived from first principles reasoning versus true secrets that can, that need to be discovered by interacting with the world directly? Because I would, it feels like it would be very difficult to, I'm just wondering about like how intellectual property interfaces with Superintelligence, or how, like, if you play this out a lot, how, like, there's all these, like, hard-won, Dorchesh has talked a little bit about this with continual learning, there's all these little subtleties that, maybe they're not secrets, maybe they're not true trade secrets,

• 1:27:10John Coogan?

  you don't think to lock them down, but they're just things that haven't been codified online or anywhere, they haven't been given to anything that is surfaceable by the model. And I'm wondering, how, is it, is it just, we need to build up new knowledge in every fact from first principles, and, and kind of go through the, the history of humanities pursuits of knowledge, or do we just need to onboard more and more information, or maybe it's both, I don't know, it's just something I've been noodling on.

• 1:27:38Speaker 05

  Yes, it's a great question. I, I would say all of the above, select all-star. So, I, I think that the answer is very similar to what it is for humans, right? How does a human generate new knowledge? How do we accomplish new things? First, you want to be grounded in the wisdom of the past, right? You really want to understand what if people tried, what worked, what didn't work? You want to go and read the biographies of, you know, various people and understand those. But you also want to try things out, right? You want to make, you know, some mistakes in a contained environment, in a way that you actually can see the effect of your hypotheses. And then you want to be able to learn from those. And I think that being able to really start to scale up these systems, and be able to integrate them with the world, is a very big process and milestone that we're currently

• 1:28:23Speaker 05

  embarking on, right? To move from a world of totally hermetically sealed reinforcement learning environments, to thinking about how do you actually put real-world interaction in there? And you think about things like robotics, like, you're going to need to have that at some point, right? You're going to need to have some sort of interaction with the real world, and to have models that are able to produce new materials, right? To be able to actually solve various diseases, for them to be able to really help people, right? That, you know, we already have models that are great at use cases like therapy, but to really get to the next level of something can just really help every person accomplish more, and accomplish whatever their goal is, it would be very helpful for that model to actually have some real-world experience with doing that very thing. And so I think that, that figuring out how to bring all of this together is ultimately

• 1:29:06Speaker 05

  what our mission is about, and we do this not in isolation, but really as part of a much broader community.

• 1:29:11John Coogan?

  It seems like it's advantageous to have the most dominant consumer app in that environment. So congratulations, Jordi. Do you have a last question?

• 1:29:16Jordi Hays?

  Last question, what do you hope to see out of Washington DC in the next year, year or two, not thinking super long term in terms of basically promoting innovation within the United States. Obviously, the admin cares a lot about AI and has been making moves. But what else would you like to see? Or where would you like them to double down?

• 1:29:37Speaker 05

  Yeah, I've been very, very impressed with how much the administration has engaged with the technology and really tried to figure out, how can we help and ensure that American AI continues to lead and really sets the standard for the world? And I think that that is the lens that I would really encourage thinking through, right? Is like, this technology is changing very fast. And that fast plus government is not usually an ideal combination. But this is the reality that we have. It's the opportunity we have. And I think that the question in my mind is less about any specific regulation or strategy, but it's really being calibrated. It's really having a very tight OODA loop, right? Being able to react to, okay, we have a new model. These are the capabilities we see on the horizon. How do we make sure that we get the most uplift and benefit from it? And thinking strategically

• 1:30:21Speaker 05

  about not just how do we do this for Americans, right? But how do we actually do this for the world and promote democratic values? And so to me, the most important thing is that motivation, right? Is the question that is asked and the ultimate sort of motivation behind what gets implemented.

• 1:30:38John Coogan?

  Yeah, that makes a ton of sense. Thank you so much for joining us, Jordi. Are you gonna hit the gong?

• 1:30:42Jordi Hays?

  For GPT-5 and the whole thing.

• 1:30:44John Coogan?

  Congratulations on the master.

• 1:30:48John Coogan?

  Historic day. And thank you so much for stopping by. We'll talk to you soon.

• 1:30:51Jordi Hays?

  Thanks for joining.

• 1:30:52John Coogan?

  Have a great day.

• 1:30:52Speaker 05

  Thank you for having me.

• 1:30:53Jordi Hays?

  Cheers.

• 1:30:54John Coogan?

  Really quickly, let me tell you about figma.com. Think bigger, build faster. Figma helps design and development teams build great products together. And we are joined by Sarah Friar, the CFO of OpenAI, next.

• 1:31:06John Coogan?

  And we are going to bring her in in just a minute.

• 1:31:08Jordi Hays?

  The gong is still swinging.

• 1:31:10John Coogan?

  The gong's still swinging, and I'm going to tell you about vanta.com. Automate compliance, manage risk, improve trust continuously. Vanta's trust management platform takes the manual work out of your security and compliance process and replaces it with continuous automation, whether you're pursuing your first framework or managing a complex program. We need one more second. Tyler, any other questions that we should be asking for the OpenAI folks? Anything top of mind? What's on the timeline? Is the timeline still in turmoil or has it settled?

• 1:31:38Speaker 01

  So I think the general vibe is like this model was not benchmarked. But if you actually get to use it, it's pretty solid.

• 1:31:45John Coogan?

  Cool.

• 1:31:45Speaker 01

  One thing, it failed to a P and bench.

• 1:31:48John Coogan?

  Oh, it did.

• 1:31:48Speaker 01

  It did not get the horse breed, correct?

• 1:31:50John Coogan?

  Didn't get the horse breed. Wait, so you have it? You have access to?

• 1:31:52Speaker 01

  Yes, I have access. But I've seen other things on the timeline. We can talk about it later, but it seems like a really good model.

• 1:31:57John Coogan?

  That's amazing. Great to hear. Well, welcome to the stream, Sarah. Good to meet you. How are you doing? Congratulations. A historic day. Thanks so much for taking the time to talk to us. How are you doing?

• 1:32:07Sarah Friar?

  I'm doing great. I mean, how could you not be doing great on the day when GPT-5 launches? It's been a long time in the making and we're so happy it's out.

• 1:32:15John Coogan?

  Yeah, fantastic. Walk me through your role and what GPT-5, what this launch means specifically for you. And yeah, well, let's just start there.

• 1:32:26Jordi Hays?

  Finance has to be, you guys have to be the unsung, the unsung heroes at OpenAI. There's just, there's a lot of, a lot of big numbers.

• 1:32:34John Coogan?

  There's a lot of massive bills coming in for crazy training runs and you have to underwrite these against future revenues. And I'm sure you've developed many models to figure that out. But yeah, walk me through what your role at OpenAI and what today means for you.

• 1:32:46Sarah Friar?

  Yeah, absolutely. So I'm OpenAI CFO, but the finance can be the unsung heroes, but they are an amazing team, so I'm going to shout out to them.

• 1:32:54Jordi Hays?

  They're heroes to us.

• 1:32:55John Coogan?

  They're heroes to us.

• 1:32:56Sarah Friar?

  It's a complex world that we're all living in. And there are a lot of Bs on the end of a lot of the numbers that we look at. Look, what is our role? Number one is just making sure we have a healthy, high-growth business. It's been incredible watching just, first of all, the number of weekly active, 700 million people using ChatGPT every week. And I'm assuming after today, we should see a very nice little bump in that number.

• 1:33:20John Coogan?

  This is going to be a gong-heavy segment, Jordy. I think we're going to have, we have a lot of soundboard for the big numbers, so congratulations.

• 1:33:25Sarah Friar?

  I love it. And I love, I've never met a number I didn't like. I think the other part of the business that you know, and then we have to do this.

• 1:33:34John Coogan?

  Clapping.

• 1:33:35Sarah Friar?

  We have to this balance of the consumer business, enterprise business, and then API business, which I think of as somewhat enterprise, you know, and balancing that out. So enterprise adoption has also been exploding. I probably do, I mean, interestingly, as a CFO, I probably meet four to five customers a week. It's a part of my job I actually love. We have about five million paying business users right now from banks to biotech. I was talking to the CFO.

• 1:34:02Jordi Hays?

  And so that number is individual companies?

• 1:34:05Sarah Friar?

  That is individual seats at companies.

• 1:34:07Jordi Hays?

  Seats in companies. Got it.

• 1:34:08Sarah Friar?

  So what I would say about that number is it's crazy to have done that in just two and a half years because enterprises, right? You gotta, you gotta put your big boy, big girl pants on to go sell to an enterprise, right? They want to make sure that you have the table stakes of security, SSO for signing on, you have HIPAA compliance if you're selling to healthcare and so on. They want to know that other people have done it. So they're often looking for that case study. But they also want to be, you know, the innovator right at the front. And so that to grow that scale of business in just two and a half years blows my mind. And it's not just big, big businesses, which I could talk, you know, at length on, but it's also small mom and pop, you know, you know, literally the

• 1:34:50Sarah Friar?

  people who really keep the lights on in most countries are also gravitating to ChatGPT, which is wonderful. And then on the developer side, four million developers have built in our platform. And the question there is like, that could be a developer inside a big company like Grab. It also could be the next, you know, startup founder. That's Y Combinator getting going with the next multi-billion dollar unicorn business. And so we see the whole gamut there, and that's important to us as well because very mission aligned, right? How are we going to get AGI to all of humanity if we don't do it through this ecosystem? So a big part of my, you asked my role, big part of my role is just keeping that business really healthy, making sure we always have the headlights on so

• 1:35:32Sarah Friar?

  people know the decisions they're making from a business standpoint. Huge part of what the team does. The other big part of my role is compute. If I didn't talk about that in my first breath, you all should correct me. I mean, it's making sure we think compute is a massive competitive differentiator. I give so much kudos to Sam and the team, but particularly Sam, because no matter how big a number we look at, Sam always wants to go bigger and he's been right.

• 1:36:02John Coogan?

  He's never met a number he doesn't want to add a zero to.

• 1:36:05Sarah Friar?

  That too, maybe logarithmic.

• 1:36:09Sarah Friar?

  And, uh, and he's, but he has been very right. And if, you know, you've just had a long conversation with Greg Brockman. I think he does such a good job of kind of really explaining what a completely different world, an AGI world is, or an AI-ified world is. And so I think when people get all caught around the axle of like, you know, what is, what is a gigawatt of compute and oh my God, you guys want to have 10 gigawatts and that's more than the compute of like Ireland, since I grew up there.

• 1:36:38Sarah Friar?

  And then now you kind of look back on that and you're like, those numbers already look small for a world where everyone will have access to intelligence. And we're really starting to see what that can mean when you look at the demos today around things like healthcare and education and so on.

• 1:36:52John Coogan?

  Can you talk to me about non-GAAP metrics and what you think is going to be useful to track? We were talking to Mark Chen about this and he was saying, you know, DAUs are great, time on site is great, but that's not as impactful of a metric for OpenAI as it is necessarily for a social network or an entertainment app. And, and there can actually be some problems that come up with that. So it feels like there might be some tension in the organization eventually, or, or just publicly about, um, you know, what metrics are worth optimizing for. And then there's also the financial community that wants non-GAAP metrics to track the health and progress of the business. And then of course, over, you know, decades, we see companies eventually roll back some of those

• 1:37:36John Coogan?

  non-GAAP metrics and as, as the business gets more complex. So how do you think about the development and, and, and sharing of non-GAAP metrics? And what do you think is actually interesting and provides signal to the business and the, and the investor community?

• 1:37:51Sarah Friar?

  I'm kind of smiling to myself because when anyone normally says, talk to me about non-GAAP metrics, I can see like most of people's eyes roll back in their head. I live for non-GAAP metrics. I would love to do that. Like, I think in a CFO seat, first of all, it's really important to think about input metrics and output metrics. And things like revenue, which is a GAAP metric as well as a non-GAAP metric, they're very laggy. Like, if you're spending your whole time focusing on the revenue number in an operator seat, like, you are completely missing what's going on with the business. So I push my team a lot to get out of kind of ultimately what the P and L looks like, and I'll come back to it, though, and go way upstream and say, what are the true input metrics that tell us about the health of our business?

• 1:38:33Sarah Friar?

  And so I think it does start with that funnel of monthly actives to weekly actives to daily actives. Because we do, I mean, our mission is literally AGI for the benefit of humanity. So we know how many billions of people live on the planet. The fact that we're starting to be able to talk in billions and percentage of the world's population, it blows my mind, right? Today, eighty-five percent of our users are outside the United States. And I love that stat. And in fact, if you go look at where are the big populations of users, it just tracks global population, right? It's countries like India, Indonesia, Brazil, Vietnam, like the Philippines. Like, go to anywhere that has big population, the US too, of course, but that will be your tracker.

• 1:39:19Sarah Friar?

  So that's kind of number one, when I think of an input metric. From there, on the consumer side, you're right, things like time and app, I've actually always had somewhat of a love-hate affair with. But I think in this case, because we're giving people intelligence, teaching them how to use that, I actually think is where time and app does become important. And one of the things we've really seen with ChatGPT are people are spending more time with it. Now, You know, we balance that with things like mental health and so on, making sure that we're not creating bad things like we might have seen in prior eras of computing. But I think we're just getting started on that front. Beyond that, like when we go into areas like the API, I don't look only at usage, right?

• 1:40:02Sarah Friar?

  I can look at tokens per minute as a usage metric. But I look at things like latency, actually try to look at the elasticity of demand, right? We know that developers want performance, they want intelligence. But they also want to make sure the API is always up and they want price. And they're often willing to trade across those three things, right? It's a kind of a linear program depending on what your use case is. And so I think it's important that we are offering things to developers that allow them to optimize across those three metrics, for example. So that's kind of your input metrics. And again, I could wax lyrical, but I won't. But then go to what you really ask. So investors on the other side, right? They want to see a P and L. They're like, I want to be able to compare you to other companies.

• 1:40:45Sarah Friar?

  I want to be able to create maybe a DCF. Like I want to think about fundamental valuation for a company if I'm going to invest in it. And so, you know, today, what I really try to push investors on is we are not a company that should be optimizing for free cash flow today because there's just too much opportunity. Like that point about compute, we have to make a decision on compute today with an eye to what we're going to need in two to three years. Because data centers don't just spring up overnight, like they're not mushrooms. They literally take time and effort. The thing we have failed at frankly, I would say is three years ago, we didn't have enough foresight to say how big could ChatGPT because it didn't exist.

• 1:41:27Sarah Friar?

  It's just a shame on us if we keep doing that over and over. So there can be a bit of a mismatch between our belief on revenue because we don't yet know the product versus the input, which is the cost today on compute. And so getting investors comfortable with the fact that there's probably losses for a period of time. I say probably because. Chat GPT, just generally, the revenue model has continued to surprise to the upside. But at least for now, we should be in big investment mode. And then you kind of said it well, like as companies mature, you move to more GAAP metrics, right? If you look at, you know, the large, the MAG7, many cases, they're looking at like real GAAP net income. So the whole way down to the bottom of the P and L, we're just not there yet.

• 1:42:11Sarah Friar?

  And we should take advantage of that advantage because we can invest as a private company.

• 1:42:16Jordi Hays?

  How do you think about timing fundraises? From my understanding or rumors, the last, you know, the most recent financing was very oversubscribed. And at the same time, you're still committing to CapEx in the future. That is a multiple of current, you know, the current run rate. And so you in the CFO seat, I'm sure there's, you're trying to find this balance of like, what does the business need today? While, you know, not diluting the company, you know, too much, knowing the sort of growth rate of the business.

• 1:42:49Sarah Friar?

  I mean, that's exactly right. That's the, the art, not the science of it is that, you know, we did just come off the back of closing out the sleeve of investment that we could take down in this current round led by SoftBank. And it was massively oversubscribed, which comes back to, I think, the market really waking up to the fact that AI is a generational opportunity. And the scale that it requires is like something people have not even seen before, right? It's, you know, people talk about the internet or like the railways. They're good analogies or transistors. I think Sam always goes back to they're good analogies, but I do think this is bigger than everything that's come before. So there's a, you know, taking down $40 billion, which we just did in this round.

• 1:43:33Sarah Friar?

  That certainly felt like that gave me a lot of confidence.

• 1:43:38Sarah Friar?

  Appreciate that. A lot of confidence to then go out and do large compute deals, right? We announced the large deal with Oracle, for example, and to be able to keep working with all of our supply chain. Microsoft, CoreWeave, Oracle, Nvidia, and so on.

• 1:43:52Sarah Friar?

  But at the same time, you know, in a world where our valuation has gone up, you know, at pace with our revenue, you do get an opportunity to keep coming back to market and not take that same dilution because you're getting that higher valuation for the work and the output that you've created. So it is a bit more of an art than a true science. I think for now, we will continue to need to fundraise in order to fund that compute. But I think we want to start getting more sophisticated, like just pure equity fundraising for everything is an expensive way to fundraise. And I think we're probably getting to the stage at a company where we can be a little bit more kind of broad in how we think about funding overall. And even just working frankly with our supply chain because, you know,

• 1:44:34Sarah Friar?

  our success with bringing this era of AI into being is their success too. And I think these companies are realizing that.

• 1:44:44Jordi Hays?

  What about partners? Last question, partner selection on the compute front. There's not a lot of companies in the world or firms that can really be a meaningful partner.

• 1:44:55John Coogan?

  You should update your LinkedIn title. We saw someone yesterday, works for Discord, is in charge of their cloud buying, and his LinkedIn title was, "I have full responsibility over buying cloud, our entire cloud budget." And it was clearly like a huge flag, but I'm sure, you know, you're in direct text message, you know, with every single person that's relevant in the industry.

• 1:45:18Jordi Hays?

  Yeah, but I'm curious around like, you know, a lot of people have been excited about developing data centers over the last couple of years in hopes to win, you know, the business of companies like OpenAI. But I think when you guys are evaluating partners, I imagine that scale is such a massive factor. And so a single, small data center is not really going to move the needle. You guys need to be thinking in terms of mega projects.

• 1:45:45Sarah Friar?

  Yeah, I mean, I think that's exactly right. I mean, it started with our partnership with Microsoft. And it's kind of, it makes me smile now to go back and look at that original kind of large fabric for pre-training. Because I think it was only in the maybe twenty megawatt sort of size. Um, and you know, now we're talking gigawatts even just this year. Uh, and you're right that when we think about like, what is perfect compute for us or strategically the right compute for us? We are definitely thinking about large scale. Um, we're thinking about flexibility, right? We're learning a lot about, um, you know, pre-training, post-training, test on compute. Even like where the different kind of scaling is happening.

• 1:46:27Sarah Friar?

  Um, we're kind of recognizing there's more of a blurred line often between what people think of as inference. So investors always are like, your inference compute and your training compute. It's like, you know, literally, it's like vanilla ice cream and chocolate ice cream. When in reality, there's like a bit in the middle that is something of both. Um, we also need to think about things like, um, where, you know, latency. Where do we want to put our footprints around the world? That very global weekly active user base, right? As they use ChatGPT, you don't want to slow the model down, right? The beauty of the intelligence is like the real time nature of it. And then when we get into big compute, like where there's lots of tokens being used, like deep research, image gen, um, video, as that comes online, like all the work you saw today, actually just even on voice.

• 1:47:12Sarah Friar?

  Um, like that really quickly means that you've got to make sure your compute is near your users. And so it is a, a big plan that's coming together. But you're right, like small is just not that useful to us.

• 1:47:25Jordi Hays?

  What about pushing partners to take risks? From my understanding, you guys are pre-committing to certain, you know, basically spend levels. But at the same time, I imagine you want people to say, here's what we know we're going to need. But we want you to build, you know, this much capacity so that we, we have the sort of incremental capacity built in.

• 1:47:44Sarah Friar?

  Yeah, we want to, I mean, being extensible is really important. And we do want to see partners like I think Oracle OCI has done a really nice job of that of kind of starting. We started with like one large, it felt really large at the time, data center footprint in Abilene in Texas. And now that has really multiplied up into multiple sites that can all be connected. And that's a good example of a partner who has the capability to start in one way, but to be able to show you a path to maybe five X-ing just in that, in that single footprint. That said, I, we are finding that as we go around the world, there is an ability to go work with governments. For example, we just made an announcement in Norway, made an announcement in the UK.

• 1:48:26Sarah Friar?

  This is the first time in my professional career I've seen countries come to the table. I want to do commercial deals like wall to wall chat GPT. I think the government of Estonia put chat GPT into all of their high schools. High school or I can't remember, it was up in the university level, but that's kind of wowing. And hand in hand with that, they are viewing AI infrastructure as incredibly strategic for their population. And, you know, it's a whole other level of selling versus, you know, I've, I've seen enterprise, large enterprises before, but never anything at this scale.

• 1:49:00Jordi Hays?

  Last question, whose idea was it to give at every federal agency chat GPT for a dollar a year?

• 1:49:07Sarah Friar?

  Yeah.

• 1:49:07Jordi Hays?

  I imagine, I imagine.

• 1:49:08John Coogan?

  You could have gotten more than a dollar. The CFO must be really upset here. Ten dollars, that's ten times as much money.

• 1:49:18Sarah Friar?

  This is one where I think it's really important. Opening is, you know, in some ways a US asset, a national asset. And we want to make sure we're accelerating our government, like all of the resources. As we think about, you know, Western democracy and so on, that we are absolutely putting our technology into those hands.

• 1:49:34John Coogan?

  It's that guy Kevin Wheel, he's been moonlighting for the US government. It's like, which team are you playing for, Kevin? Are you on OpenAI or are you on the US government?

• 1:49:43Sarah Friar?

  Kevin just did his basic training. I don't know if I'm allowed to tell you that, but I was hearing all about it yesterday.

• 1:49:48John Coogan?

  I saw some photos, they look great.

• 1:49:50Sarah Friar?

  Yeah, it's a good thing.

• 1:49:51John Coogan?

  He's gonna be in even better shape.

• 1:49:53Sarah Friar?

  A lot with governments globally.

• 1:49:55John Coogan?

  Amazing. Last question for me, I, you know, the open source model launched two days ago. And there's this world where like you have this dominant, the accidental consumer company, you have this dominant consumer app that's generating so much revenue, then you have B two B and enterprise and API, and that looks more like a cloud provider. But then is there a world where the red hat Linux of open source L L Ms is an open AI division? And that, and that there's actually serious revenue and profit that comes from helping companies implement an open source, large language model like Red Hat built a pretty fantastic business for a long time on top of open source Linux implementations.

• 1:50:36Sarah Friar?

  Yeah, I mean, I think it's the right question to be asked, to be asking. I mean, I think step one was getting

• 1:50:42John Coogan?

  Yeah, you gotta get it out of two days.

• 1:50:43Sarah Friar?

  Our second open source model out and getting, seeing what that traction is and then seeing what the community needs. I think it's important to leave space for a community to develop, right? That is the beauty of open source is that ecosystem that develops. And that was true with Linux. It's true in areas like crypto too. But I do think you'll find over time that as enterprises want to deploy it like I now have dinosaurs myself, but when I was a, you know, when I was a research analyst at Goldman Sachs back in the day, I covered software and I covered, I covered Red Hat, actually.

• 1:51:14John Coogan?

  Yeah, really.

• 1:51:15Sarah Friar?

  And all that growth, I like wrote a research report called Fear the Penguin at one point because Linux being deployed.

• 1:51:22John Coogan?

  Yep.

• 1:51:22Sarah Friar?

  But then you started to understand that for an enterprise, you couldn't depend on like patching and upgrading to happen via community model. Like you needed some of the rigor that goes with an enterprise business where you kind of know when, you know, if you need maintenance, if you need a bug patch and so on. And so that did allow Red Hat to grow an incredible business. So I don't know if it's us or we'd be supportive of others, but I think we are so excited to see open source out there and getting incredible feedback. And I think we want to do that ahead of G P T five to keep coming back to like, we're here to grow this ecosystem.

• 1:51:55John Coogan?

  Well, we'll give you market cap credit for it anyway, even if it's early stage. Well, thank you so much for coming on. This is fantastic. We'll talk to you soon. Thank you, sir. Great to be here, boss. Take care. Have a good one. Cheers. Bye. Up next, we have Didi Credo from Kudo, I believe I'm pronouncing that correctly. Let me tell you about Graphite, code review for the age of AI. Graphite helps teams on GitHub ship higher quality software faster. You can get started for free at graphite.dev. And let's bring in our next guest. How are you doing? Welcome to the stream.

• 1:52:22Speaker 07

  Welcome.

• 1:52:23John Coogan?

  Ooh, very clean background. I know it's probably virtual, but whatever you got going on, it looks fantastic. You look great. How are you doing? Are you excited about GPT-5?

• 1:52:31Speaker 07

  Oh, I'm so excited. It's awesome.

• 1:52:35John Coogan?

  Break it down.

• 1:52:35Speaker 07

  It's actually like everybody's talking about the coding capabilities.

• 1:52:38John Coogan?

  Please.

• 1:52:38Speaker 07

  But no one is really talking about the code review capabilities, and I'm gonna talk about that today.

• 1:52:43John Coogan?

  Yeah, yeah, break it down. How are you using it right now?

• 1:52:47Speaker 07

  Yeah, so we just enabled it in our platform. It's a default model for both our ID plugin, our CLI, our Git plugin.

• 1:52:58Speaker 07

  And yeah, we're using it to generate very high quality code reviews, catch bugs before they hit production, help enterprises verify that their code is aligned with their best practices. So yeah, it's super exciting. I can share my screen and show a few things if like that makes sense.

• 1:53:11John Coogan?

  You can. Everything you share will be live. It'll be a little

• 1:53:15Speaker 07

  Yeah, yeah, yeah, yeah.

• 1:53:16John Coogan?

  Please. But I want to know also, while you're getting that set up, I want to know about what changes materially do you think happened in GPT-5 specifically for code and code review? Do you think there's more data going into the model, more data going into the pre-training, post-training, anything else? Anything that you're noticing that you're like, oh, there's a specific upgrade here. They must have done something to get there.

• 1:53:41Speaker 07

  Yeah, yeah, I think it's a great point. So I think it's all of the above. So it's scaling of both the like the pre-training, but probably a lot of the reinforcement learning.

• 1:53:51John Coogan?

  Yep.

• 1:53:51Speaker 07

  And basically using that at scale to verify that code gets generated in high quality and then also basically caching bugs. And when you do it with reinforcement learning, you have the the actual ground truth. So once you scale that, you can get the model to be to basically be a lot better at that.

• 1:54:14John Coogan?

  How steep is the power law right now in just programming languages? Is it basically all Python, JavaScript, and then like a really hard falloff? Or is it actually important for coding models if they want to be adopted widely to be like truly multi-language and get all the way down into the long tail of like the Rust and the, and you know, C sharp and all the different languages that are out there?

• 1:54:35Speaker 07

  Yeah, yeah, for sure. It's important to, I mean, the majority of the market is in the JavaScript, JavaScript, Python, like the majority of the early adopters, I would say. But then when you get to enterprise use cases, you get a lot of .NET, you get a lot of Java, and the models are pretty, getting pretty good at those languages as well, for sure.

• 1:54:57John Coogan?

  Are you excited about, I mean, how do you think about the difference between like the improvements to GPT-5 from the consumer's perspective versus at the API level? I always found it a little confusing that ChatGPT was available as an API and you could interface with the chat, I believe you could interface with the ChatGPT model via the API. And there's a little bit of like a line blurring there, but are there features that you think are are cruft and you want to kind of rip out for an API use case? Or do you just say, hey, give us the kitchen sink and we'll, we'll, we'll work from there? And it's actually helpful to have, you know, a coding model that can still have a web browser?

• 1:55:39Speaker 07

  Yeah, yeah, I think basically it's a lot about, we consume the model through the API, and it's really the same model that drives the consumer product. But as for us, since our use cases are a lot about agentic use cases, the more the model gets better at using tools and gets better at kind of listening to very, very specific instructions, following instructions is critical for the enterprise use cases. Because for us, unlike the broader market, we believe that for enterprises, you need to have very specific agents that are defined with specific set of instructions and prompts and tools and permissions. And the more the models get trained with that back of environment, the better they

• 1:56:24Speaker 07

  end up serving the enterprise market, which is really where we're focused on.

• 1:56:28John Coogan?

  Um, my, my question is, um, I wonder, like you, you said, like, very specific instructions are important. When are we going to get an agent that I can just turn loose in a code base and say, like, just go improve it, like, just go hunt around, do, like, rewrite that, like, like, when you get a good open source contributor on a team that just becomes nerd sniped by the project that you're building on, they will just go around and find little ways to improve. This documentation needs to be a little better. Let's rewrite this test case over here. Let's add a little bit more, you know, functionality to this class or function. How far are we from that?

• 1:57:07Speaker 07

  Yeah, I think the models are getting better and better at that part of basically kind of running loose in a code base.

• 1:57:13John Coogan?

  Yeah.

• 1:57:14Speaker 07

  But they do need the guardrails in place. And this is kind of where we're focused on, like, a lot of the talk in the market is around the code generation side. You know, let the agent loose and give it a task and it's just going to go around and run for hours and do, and do it. What we're seeing is that the real challenge is now shifting towards how do I verify that the code is aligned with the best practices? How do I make sure that it's well tested, well reviewed, um, doesn't break anything? You know, so that, that's, I think, the next frontier and really developers going forward are not going to write a lot of the code by, by hand. They're most spend, they're going to spend most of their time reviewing code. And that's the next frontier. That's what we're talking, really, like, are here to talk about.

• 1:58:00John Coogan?

  Very cool. Anything else, Jordi?

• 1:58:01Jordi Hays?

  No.

• 1:58:02John Coogan?

  Well, thank you so much for joining, giving us some extra context on the GPT five launch. We will talk to you soon. Have a great rest of your day and thank you for joining.

• 1:58:11Jordi Hays?

  Cheers.

• 1:58:12Speaker 07

  Thanks. Cheers.

• 1:58:13John Coogan?

  Talk to you soon. Uh, and let me tell you about Profound. Get your brand mentioned on ChatGPT. That seems more relevant than ever. Reach millions of consumers who are using AI to discover new products and brands. I forgot to ask about this. We'll have to come back to this, but I want to know if.

• 1:58:30Jordi Hays?

  Profound powers MongoDB, Indeed, Mercury, DocuSign, Zapier, Ramp.

• 1:58:36Jordi Hays?

  Row, Goland, Workable, Missouri, Aidsleep, US Bank, Chime, Clay.

• 1:58:41John Coogan?

  Okay, okay, we get it.

• 1:58:43Jordi Hays?

  They got some logos.

• 1:58:44John Coogan?

  There is this question of like, okay, even if you're, even if you're like, okay, GPT-5 is more incremental than more of a, more of a, an evolution than a revolution. It's like, okay, well then let's talk about how it affects every other business and every other aspect of the economy. What should you be focusing on? And, and is like, do the, do any of the updates from GPT-4 to GPT-5 change how you're positioning your brand for AI search? That's certainly an interesting question to dig into. Anyway, we have Zach Lloyd from Warp coming into the studio. Welcome to the stream for the second time.

• 1:59:19Jordi Hays?

  Welcome back.

• 1:59:20Zach Lloyd?

  Zach Attack.

• 1:59:21John Coogan?

  Good to see you.

• 1:59:21Zach Lloyd?

  Good to be back.

• 1:59:22John Coogan?

  How you doing?

• 1:59:23Zach Lloyd?

  I'm doing pretty well.

• 1:59:25John Coogan?

  Yeah, so, I mean, a lot of what stuck out to me, I'm mostly a consumer of consumer AI apps. I'm very excited about not needing to mess around with a model picker anymore. But take us through the biggest improvements from the software development side.

• 1:59:44Zach Lloyd?

  Yeah, I mean, it's a, it's a major step up from the prior OpenAI models. It's, I mean, it's doing agentic workflows in Warp for a much longer period. It's just a smarter general model. Like we evaluated against all of our benchmarks and it's up there at state of the art, which is, you know, from our perspective, it's, it's awesome to have multiple competitive models that our users can benefit from. So definitely a huge improvement from GPT-4.1.

• 2:00:14John Coogan?

  Yeah, so it seems like not, not the, you know, clog code killer, but certainly in, in, in the same conversation, in the same, in the same football stadium, if we're using a sports metaphor.

• 2:00:27Jordi Hays?

  How much, you know, one thing that stood out is the cost reduction.

• 2:00:31John Coogan?

  I was about to ask.

• 2:00:32Jordi Hays?

  How much do you think that developers will care about that versus just, you know, what it can do from, from an output standpoint?

• 2:00:40Zach Lloyd?

  I think developers do care about value, so sort of like quality to cost ratio. I think it's, the more you get into like the individual developer and the small team, the more that that matters, whereas if you're at the enterprise level, I feel like it's a little bit less price sensitive. So yeah, I mean, you can see it as different apps change their pricing, what the reaction of the developers is. You've probably seen this with Cursor and seen this with Cloud Code. And so developers really, really are looking for something that's cost effective. So the fact that the cost is a little bit lower is actually, is a big deal.

• 2:01:21John Coogan?

  Do you think we're in the Lyft Uber 2015 arc where the prices are subsidized and the prices will go up? Do you think that there's a price war on the horizon now that the frontier models seem to be similar capabilities? Do you think that someone will try and raise a bunch of money, cut prices a bunch and steal a bunch of users? Like, how do you think that plays out?

• 2:01:45Zach Lloyd?

  It's an awesome question. I mean, my hope is that we get to a world where there is price competition at the model layer. So Warp is very much at the app layer, right? And so our value prop is like, we can give our users, who are mostly developers, the best model access. And so to the extent that it's not one sort of model provider running away with that and having pricing power, it's better for us, just candidly. And so, you know, my hope would be something like the model world ends up a little bit like G-Cloud, AWS, Azure. That's our best end state, where all of these models are, you

• 2:02:27Zach Lloyd?

  know, sort of similarly powerful and a little bit more commoditized. I don't think it's been like that, but it's going, it's getting a little bit more like that. And so the more that there's more than one show in town, I think that's generally good for Warp and actually is good for developers because it will put competition, the competition will put pressure to bring the prices down. But I don't know. Like, I also think that people will definitely pay for quality. And so if there is a, you know, meaningful delta and quality on the frontier models, then I think that like whoever has the quality delta will have a lead temporarily, but it's, I'm not sure that that lead will be sustainable. We'll see.

• 2:03:08Jordi Hays?

  How do you think the developer community should plan around model deprecation over the next one to two years? Like, how much, you know, from, I don't know that I've gotten a reaction yet from, I don't know if there's general frustration yet from people. You know, we've heard on the consumer side, Tyler on our team here loves four or five.

• 2:03:34Zach Lloyd?

  Yeah.

• 2:03:35Jordi Hays?

  And so he was a little bit disappointed to hear that. But what kind of, what are you seeing on the developer's side?

• 2:03:40Zach Lloyd?

  Yeah, I think it's a little bit different for people who are like building apps on LLMs versus people who are using LLMs as like an accelerator to do encoding. And like, you know, at Warp, actually, we do both. Like, we're an application-level stack. And like, it's actually very easy for us to go to the latest model. And so it doesn't, it doesn't really bother me. I don't know, I don't know what type of app you would be building where it's like, it's really important that it's like GPT-3.5 or GPT-4 or something like that. I think like generally, we want the most intelligent tokens at the best cost. So I don't, I don't see that being like too big of an issue, honestly.

• 2:04:23John Coogan?

  What about open source? Does that feel like something that will be in the playbook? Is the markup on closed-source models high enough that there will be a significant price delta? Or is the parader frontier kind of indifferent to closed-source, open-source?

• 2:04:41Zach Lloyd?

  So if there was a comparable open-source option, that would be awesome. I think that the economics of it, again, it doesn't, it doesn't seem like a perfect analogy to me between like open-source software and open-source models. So open-source software, it's like you have a big community of people who, you know, for the love of coding are building a really awesome product. For open-source models, it's

• 2:05:06Zach Lloyd?

  It's like you just need a crazy amount of capital to train something that's on the frontier. And so I don't know that how that happens. And so what we've seen is like the open source models are competitive at the quality level that they're at. But the quality level that they're at is not the same as the frontier models. And I don't really see why that would change. And so, I don't know, in Warp, it's like we, we were serving some open source models, but they're just not, they're not as good. And so there's, I think, a more limited use case for them right now. And I don't really see economically why that would change.

• 2:05:45Zach Lloyd?

  In fact, I would be, I would be surprised if anyone was spending billions of dollars to train a model and just kind of put out the open weights. Like, I don't get the business strategy there, but, but maybe that will happen. That would be awesome.

• 2:05:59John Coogan?

  Is there a world where you're like, this idea of like smarter, smarter models either orchestrating dumber, cheaper models? Or like using or, or distilling models into more narrow, narrow formulations that can be run more efficiently? We've talked to a few companies that do this for businesses. Like you just want a model that just filters for profanity and you can run it on, you know, a, a gaming graphics card. And so it's basically super, super cheap or super fast. I'm wondering about like in the coding world, coding agent world, any of that? Like where, where are the opportunities to kind of span out and use an ensemble of models instead of just this?

• 2:06:43John Coogan?

  Hit everything with the smartest, best. It feels like because of the funding environment, everyone can kind of justify like a high cloud bill. But, and most people don't admit that it's hurting the bottom line, but it feels like at some point it kind of has to eventually.

• 2:07:00Zach Lloyd?

  I mean, I think, I think that's a very real thing. Like sense of even in Warp, we don't use like the, the biggest, most powerful model for every task. And so there are certain things like. Um, you know, for warp, maybe for like deciding whether or not we should summarize a conversation is like a good example. So you hit the context window, you're like, okay, is this, is this a good spot to summarize? Is this a good spot to encourage a user to start a new conversation? We use a much more inexpensive and also low latency model, right? The other thing, the trend is that these very, very powerful models tend to have much higher latency. And so we do a mixture of models and that's totally a real thing.

• 2:07:45Zach Lloyd?

  Um, but I, I think for like the like, the predominant use case as a developer is going to be, I want to tell an agent to do something. I want it to be harder and harder. I want it to run for longer and longer. Um, and to do that, it's like, you kind of want in general, the most intelligent model. And so, you know, until this, until the models have a sort of S curve like type shape, I think that, um, I think it's going to be more of a quality game than a cost game for most of these things.

• 2:08:16John Coogan?

  Doesn't it feel like they have an S curve shape right now? Certainly does from a consumer perspective.

• 2:08:22Zach Lloyd?

  That's interesting. Um, from, from a coding perspective, I feel like we're still accelerating. Um, like the difference again, between the last version of GPT and this version of, uh, of GPT is, is probably bigger than the difference between like four, one and four and four and three point five. Like it's a big deal. And same thing with the anthropic models. And I'm sure that we'll see something from Google where it's an acceleration. Um, and I think that there is like a, um, maybe an under appreciation of how much left there is to solve here. Because when, when you, even when you're doing like a real coding task as a pro, like despite all the demos you see on Twitter where it's like, someone asks, uh, you know, an agent to build an app,

• 2:09:04Zach Lloyd?

  that's like a lower level of difficulty than doing what a pro developer does with one of these models. And the models still don't produce great code a lot of the time. Like there's a lot of kind of handholding that has to go into it. And I think, I think that we are still seeing an acceleration in terms of the models actually becoming not just like, okay, competent engineers, but like really, really good engineers.

• 2:09:25John Coogan?

  Yeah. Do you care about benchmarks?

• 2:09:28Zach Lloyd?

  We care a ton about benchmarks, like we um.

• 2:09:30Jordi Hays?

  But your own internal benchmarks or or?

• 2:09:33Zach Lloyd?

  We do both. So, you know, plug for warp, we're number one on terminal bench, which is the public, uh, you know, terminal benchmark and we're top five on sweet bench, which is the coding benchmark. And then the only way, uh, in my opinion, that an app at our layer in the stack can really improve is by measuring the progress. And so we have our own internal set of evals that we run across all these models as well, which are coming from like real use cases. And that again is an advantage of being like a product that's in the wild that has a lot of users is that we can sort of see where the models are failing, where they're working. And so we're, we're very big on that, actually. Yeah.

• 2:10:10John Coogan?

  Awesome. Well, thank you so much for stopping by. We will talk to you soon.

• 2:10:13Jordi Hays?

  Sure. You'll have a busy afternoon.

• 2:10:15Zach Lloyd?

  Shout out, by the way, to OpenAI team, very, very helpful in, uh, working with us to get GPT five to be awesome in warp. And one more shameless plug, it's we have a discount code for people who want to try GPT five in warp. It's $5 GPT five.

• 2:10:30John Coogan?

  Okay.

• 2:10:31Zach Lloyd?

  Thank you for having me, guys.

• 2:10:32John Coogan?

  Yeah, we'll talk to you soon.

• 2:10:33Zach Lloyd?

  Yeah.

• 2:10:34Jordi Hays?

  Cheers.

• 2:10:35John Coogan?

  Uh, Tyler, any updates from the timeline while you're thinking about what the latest vibe check is in the war between OpenAI?

• 2:10:43Jordi Hays?

  I got one from friend of the show.

• 2:10:46John Coogan?

  Is a purpose built tool for planning and building products. Meet the system for modern software development, streamline issues, projects and product roadmaps. Go to linear.app to get started.

• 2:10:55Jordi Hays?

  Choice for OpenAI.

• 2:10:56John Coogan?

  You have something?

• 2:10:56Jordi Hays?

  Uh, from Reggie James, friend of the show. Half of my timeline says this is the closest we've been to AGI. The other half of my timeline says we officially just hit AI stagnation. I love tech.

• 2:11:09John Coogan?

  Well, uh, we will be going deeper deciding whether or not this is stagnation or hyperintelligence takeoff. Uh, and we will be joined by our next guest, Riley from Charlie Labs.

• 2:11:21Speaker 08

  Hey, guys. Thanks for having me.

• 2:11:22John Coogan?

  Good to see you, Riley. How you doing?

• 2:11:24Jordi Hays?

  What's happening?

• 2:11:24Speaker 08

  I'm doing fantastic. We, uh, we've been heads down with GPT five, uh, and.

• 2:11:29John Coogan?

  How long have you had it? How, how long did you get the preview? I feel like it, it, it, you know, it gets rolled out to early adopters a little bit earlier, but it's been weeks, months? How long have you had it?

• 2:11:39Speaker 08

  Word, couple, couple of weeks, like two or three.

• 2:11:42John Coogan?

  What was the first?

• 2:11:43Jordi Hays?

  How's Charlie liking it?

• 2:11:45Speaker 08

  Charlie loves it.

• 2:11:48Speaker 08

  And also I love what Charlie does with it.

• 2:11:50John Coogan?

  Yeah. What does Charlie do with it? What was the first thing you did with ChatGPT five?

• 2:11:55Speaker 08

  Ran our evals.

• 2:11:56John Coogan?

  Oh, yeah? How'd they come back?

• 2:11:59Speaker 08

  Really good. Much better than O3, which was much better than any other model we've run before that.

• 2:12:04John Coogan?

  Interesting. And yeah, so let's zoom out. What do you do? What do these evals measure? Walk me through it.

• 2:12:13Speaker 08

  So Charlie is a TypeScript-focused coding agent that operates much more like a human does. So less like IDE, application, terminal, and more joins your GitHub and Slack and linear workspaces. And it interacts with the team the same way other humans do.

• 2:12:33Speaker 08

  And then our evals are a mix of code review, because part of Charlie's job is to review PRs from humans, as well as his own, and then code authoring, so opening PRs and pushing commits.

• 2:12:46John Coogan?

  So when you develop your own evals, I imagine you try and keep those out of any training data. You want those to be held private, is that correct?

• 2:12:54Speaker 08

  Yes. And it's getting even harder with web access now, because they're too good at finding things.

• 2:13:00John Coogan?

  They're finding.

• 2:13:02John Coogan?

  That's funny. And then, talk to me about the shape of those, of the actual problems in the eval. Are you doing, are there some easy questions, some hard questions, some extremely hard questions? Like, how are you formulating those? What's the shape of an individual task? Is it scored out of like 100? How do you think about developing a good eval?

• 2:13:25Speaker 08

  A mix of hard to very hard. The easy ones are just a waste of money and time at this point, especially with Five. Like, there's a bunch that it's just not going to get wrong.

• 2:13:33John Coogan?

  Yeah, yeah.

• 2:13:34Speaker 08

  And then we're mostly doing, the PR ones look kind of like Sweetbench in the sense that we're taking an issue to start with. But instead of giving the issue like in a Docker container already, we trigger a comment on the issue that says, "Hey, Charlie, go make a PR for this." And then Charlie does his thing, and then the PR comes up. And then we score that PR against a whole bunch of things, like correctness to a known solution that's correct, as well as code quality, testability, and some softer things like descriptions.

• 2:14:06John Coogan?

  Who are the biggest customers or users for like a TypeScript-focused coding agent?

• 2:14:14Speaker 08

  It's a wide range of mostly modern apps, like pretty much any web app these days. It's gonna be like a next JS type app. And then all the way into like backend, like Charlie himself is written in TypeScript.

• 2:14:27John Coogan?

  Sure, makes sense.

• 2:14:28Speaker 08

  And there's very little frontend.

• 2:14:30John Coogan?

  Anything else, Charlie? What else you got?

• 2:14:33Jordi Hays?

  I just want to say I love the name Charlie. It's one of my favorite agent names that we've had on the show.

• 2:14:38John Coogan?

  Yes, it's right up there with Pig and what was the other one?

• 2:14:41Jordi Hays?

  Well, I don't think that was an agent, but yeah, it's a good one.

• 2:14:44John Coogan?

  Yeah.

• 2:14:45Jordi Hays?

  Congrats on logging it down.

• 2:14:46John Coogan?

  Yeah, what about, what about cost and that side of the business? Is there, is there any movement there or anything that you, where you require movement? Or you need movement to really unlock new capabilities in the business or new markets?

• 2:15:07Speaker 08

  Not really for us because we're operating kind of as at a human level. We do value-based pricing, so we charge per PR or per commit. And because that's comparing to such expensive actions that humans are doing. The challenge for us is more actually living up to the promise than doing it cheap.

• 2:15:26John Coogan?

  Yeah, yeah. Are you having?

• 2:15:28Jordi Hays?

  But then, but then doesn't the cost reduction announced today? Isn't that great for business?

• 2:15:34Speaker 08

  Yeah, I mean, it's good overall, but like that's, our problem is not that the models are expensive. It's that they're, I mean, they're getting really smart, but I'll always take more.

• 2:15:45Jordi Hays?

  Never enough.

• 2:15:46Speaker 08

  Like, for instance, since the beginning of August, we've been testing. And 98% of the code that got merged into our code base was written by Charlie.

• 2:15:55Jordi Hays?

  Wow.

• 2:15:55Speaker 08

  Not 30, not 50, 98%. And that's coming through PRs. That's not like autocomplete in a 90e type thing.

• 2:16:02John Coogan?

  That's crazy. Yeah, what, yeah, what does that mean for like the future of, of like, who are you hiring? I imagine that you're still, you know, a, an engineering-heavy organization. That's just puppeteering and orgus- and orchestrating agents. But where do you see like the future of software development as a career path go?

• 2:16:22Jordi Hays?

  Yeah, are, are new CS grads cooked?

• 2:16:28Speaker 08

  I think if they get really good at using the AI, no. If they try and take an approach of getting really good at writing code by hand, for sure.

• 2:16:35John Coogan?

  Yeah.

• 2:16:37Speaker 08

  What we're mostly looking for hiring is people who are able to see things at a much higher level and plan further out. Because with tools like Charlie, you can write so much more code so quickly that it's like it's more important to see where you're going and take the right path than it is to be able to write it quickly.

• 2:16:55John Coogan?

  Very cool. Well, thank you so much for stopping by. Good luck with the rest of your day and congrats on a, on an upgrade to everything that you do.

• 2:17:02Jordi Hays?

  Tell Charlie to have fun out there.

• 2:17:04John Coogan?

  Have some fun.

• 2:17:05Speaker 08

  Thanks, guys.

• 2:17:06John Coogan?

  We'll talk to you soon.

• 2:17:07Jordi Hays?

  All right.

• 2:17:07John Coogan?

  Let me tell you about numeralhq.com. Sales tax and autopilot. Spend less than five minutes per month on sales tax compliance.

• 2:17:15Jordi Hays?

  Sales tax, super intelligence. A number of the fellas in the chat got access to five.

• 2:17:22John Coogan?

  Break it down for us.

• 2:17:22Jordi Hays?

  Ragov says it's pretty good. The writing ability feels a little nerfed. Says the way it writes feels a little programmatic rather than sounding human, reverts to using points even for things like blog posts and also uses overly complicated language for simple stuff. Techno Chief says it's crazy fast.

• 2:17:43John Coogan?

  Oh, that's good.

• 2:17:43Jordi Hays?

  Dan Ratliff says, yeah, I was just gonna say that. Very, very, very fast.

• 2:17:51Jordi Hays?

  Z Jean Ahmed says junior devs are barbecued.

• 2:17:57John Coogan?

  Tyler, anything from your side before we talk to Guillermo from Versale?

• 2:18:01Speaker 01

  I think maybe a good way to, to like vibe check, at least on the timeline is that it's almost like a 4.5 kind of thing.

• 2:18:08John Coogan?

  Sure.

• 2:18:09Speaker 01

  Where it comes out, people are like, this model totally sucks. Look at the benchmarks. It's like not, it's not some massive improvement. It's like a fair, you know, not a step change at all. But then you, you start playing with them. It's actually like, okay, there's actually a good model.

• 2:18:19John Coogan?

  Yep.

• 2:18:19Speaker 01

  Like a lot of the stuff I'm seeing people post like, oh, that's actually like really like interesting output, stuff like that. But it seems good.

• 2:18:28John Coogan?

  Can we do the green text eval? Green text bench?

• 2:18:31Speaker 01

  Yeah, yeah.

• 2:18:31John Coogan?

  We gotta do that.

• 2:18:32Speaker 01

  Be TBPN intern.

• 2:18:33John Coogan?

  Yes, yes, yes, yes. We'll let you cook on that and then we will move on to our next guest Guillermo Rao from Versale. Coming in to TBPN for the second time. Great to see you Guillermo. How are you doing? I like the accent hall. Thank you. Welcome to the stream. How you doing today? Do you think GPT-5 could beat me, you, couple of the boys here on Dust 2 in Counter-Strike?

• 2:18:58Guillermo Rauch?

  Easily.

• 2:18:58John Coogan?

  Easily.

• 2:18:59Guillermo Rauch?

  Yeah, it depends on the frame rate, right?

• 2:19:01John Coogan?

  Yeah.

• 2:19:02Guillermo Rauch?

  On a long enough timeline, we're cooked.

• 2:19:05John Coogan?

  We're cooked.

• 2:19:06Guillermo Rauch?

  But we might frag it short term, or we might be faster.

• 2:19:09John Coogan?

  Amazing, yeah. Yeah, we got to, I mean, I'm sure we'll get to GPT-5, but what's your reaction to the world model stuff from Google? Do you think, do you have an idea of where that's going as a product? It feels like a GPT-2 level technology, very much a research-focused technology. I'm sure OpenAI is working on something too, and a lot of the labs will work on it. But what's your theory behind the generative video game world model stuff that's going on?

• 2:19:38Guillermo Rauch?

  I mean, number one, super fascinating, right? I think when we think about the future, I always think about Gen Sense. The future of applications will be that pixels are generated, not rendered. So as much as we're really excited today that GPT-5 and V-Zero are really good at writing code that then renders interfaces, I think it's also cool to dream of a world where we're just going directly from GPU to pixel grid, right? And, but if you remember, like, a couple of years ago, and maybe a decade ago, there was a lot of excitement of video games that were going to be live-streamed from the cloud.

• 2:20:16John Coogan?

  Yeah, that's right.

• 2:20:17Guillermo Rauch?

  Where your input, your keyboard, you could have a very thin client, your input, your keyboard, your mouse movement was going to be dispatched to the cloud. We're going to have GPUs near you.

• 2:20:26John Coogan?

  Google Stadia was big there, and then Microsoft's done a game, and it's still, Microsoft is actually still pulling it, still pushing it very heavily.

• 2:20:34Guillermo Rauch?

  Awesome tech, but not mass adoption.

• 2:20:37John Coogan?

  Yeah.

• 2:20:38Guillermo Rauch?

  But if you look at, you know, a lot of these technologies are being really successful in letting people get more creative and test things out. A lot of the use cases that we see for V-Zero and Vibe coding are Almost like a communication tool. Like I want to prototype something, I want to see what the what's possible. I want to explore the latent space. And I think those world models are going to be incredible just to inspire what the future of games could look like, right? Uh, just getting ideas for actually then shipping them in real uh, 3D engine model. I think short term. I think long term, all bets are off. Someone was saying in the chat, you know, junior devs are roasted. Um, or barbecued. I think that's not quite true.

• 2:21:16John Coogan?

  Okay.

• 2:21:16Guillermo Rauch?

  Uh, same for like 3D uh, engine developers.

• 2:21:20John Coogan?

  Give us the bull case for junior devs staying off the barbecue.

• 2:21:24Guillermo Rauch?

  So the bull case for I think people in general is that you move from, I mean, the, the progression in the industry has been assistant to agent, to team of agents, agent orchestrator. It's still really useful to have a human be the one that's sort of like managing the team.

• 2:21:42John Coogan?

  Yeah.

• 2:21:43Guillermo Rauch?

  So you're moving from like junior dev to junior engine manager. Uh, especially as these tools become more agentic. Uh, in, in the new version of V0 that's coming up really soon. You're starting to notice that V0 sort of splits the task between a little team. You have the designer of the team, you have the PM of the team that's sort of working on the spec. You have the architect, you have the engineer. I don't know if you saw Claude Code announced, I think it's like a slash security review.

• 2:22:11John Coogan?

  Yeah.

• 2:22:12Guillermo Rauch?

  You think of, think of that as having a security team or a team of agents or security researcher at your disposal. So junior dev as like a vertical skill may be a little barbecued. But junior engine manager, so I think it's just going to be, the junior dev is so much more powered in this world. If you allow yourself to be and you keep up with what these tools can do and, and, and I think you, you stay, you know, at the cutting edge.

• 2:22:35Jordi Hays?

  Yeah, I mean, the obvious bull case is if you're like, if, if someone's a college student today, they can learn to code.

• 2:22:43Jordi Hays?

  Truly AI natively. They don't have to say, oh, we're an AI native organization now. We have to up scale and kind of retrain people how to think. They can just naturally start to think.

• 2:22:53John Coogan?

  I remember there was that Sam Altman post about how we'll look back on, you know, 93% of humanity with subsistence farming. And if you ask those people what they think about our email jobs, they'd be like, you guys are crazy. And it's almost like in the near future, midterm future, maybe even long-term future, it's like the number of individual contributors will be extremely low. And almost everyone will be a manager and you'll become a manager much faster. You'll just be managing agents and then you'll be managing people who manage agents. But the job of almost everyone will become managerial. Maybe that's what happens. I don't know. I'm not 100% on that, but that's what that made me think.

• 2:23:30Guillermo Rauch?

  Someone asked me yesterday, you know, what do you think the future of the market of monitors looks like? Like, does it stay flat? Do people get more monitors because they're

• 2:23:40John Coogan?

  That's hilarious.

• 2:23:41Guillermo Rauch?

  Like Doge coin trader analyst.

• 2:23:43John Coogan?

  Yeah, yeah, yeah, yeah.

• 2:23:46John Coogan?

  In the future, everyone has the hedge fund six monitor setup.

• 2:23:49Jordi Hays?

  In the future, everybody's just gonna be at work monitoring.

• 2:23:52John Coogan?

  Maybe on their phone. I mean, I've noticed that, that, you know, when I was an individual contributor, I had three monitors. I was programming on all the screens. And now I, I mean, I use my laptop during the show and then most of my work is done on my phone, phone calls and then, and then firing off messages. Yeah, maybe, maybe we've actually shift away from monitors and go further into voice interfaces. You, oh, I called the lead of my agents and then that agent relays it to some code.

• 2:24:19Guillermo Rauch?

  I'm very optimistic on voice, by the way, because I've now seen it. I did, we're, we're, we're cooking on a better mobile experience for B0.

• 2:24:27John Coogan?

  Sure.

• 2:24:27Guillermo Rauch?

  And I was going back and forth with my head of mobile and he was talking to B0 and I was writing down in a pretty fast diaper, but he beat me with voice. Using the local model on the phone. So there's still the question of like edge latency versus cloud latency, kind of like what we talked about with 3D. But I do think voice is going to play an increasingly exciting role in, in programming, which is kind of wild. I would have never imagined. I've always been about like typing benchmarks in WPMs. Voice is coming.

• 2:24:58John Coogan?

  Yeah, yeah, yeah.

• 2:24:58Jordi Hays?

  How do you think about competition broadly in developer tooling, code gen? I mean, it right now it seems like there's just so much.

• 2:25:07John Coogan?

  It feels like massive TAM expansion moment. Every company's ripping.

• 2:25:10Jordi Hays?

  TAM expansion moment, but at the same time winners will emerge. Uh, obviously you're you're you're playing to win. And yeah, I'm curious.

• 2:25:21Guillermo Rauch?

  Yeah, on some level, we're playing both sides of the bat. We what we announced today that's really exciting is V zero with G P T five support. So you can go to v zero dot dev slash G P D five. And we'll use G P D five in combination with our model pipeline that makes it really good at vibe coding, especially for non-technical folks. But we also on the Versale AI cloud side of things, we open sourced. Basically, you can create your own vibe coding platform powered by any model.

• 2:25:53John Coogan?

  I was joking about this with Tyler. Vibe code me a vibe coding platform, please.

• 2:25:58Guillermo Rauch?

  That's right. Make no mistake. Yeah, vibe code me a billion dollar company.

• 2:26:02John Coogan?

  Yeah.

• 2:26:03Guillermo Rauch?

  No mistakes. But basically, we are giving people that as a starter kit.

• 2:26:07John Coogan?

  Sure.

• 2:26:07Guillermo Rauch?

  And by the way, the fundamental question that a CEO asked me the other day was, is vibe coding a product or a feature? Or is it both? You know, it's T B D. The case for a feature is, okay, so there's going to be lots of systems of record. Think Salesforce, Snowflake, Databricks. And increasingly, they're going to incorporate code gen capabilities into their platforms. They can use a lot of these capabilities that we just open sourced. And you'll go to their the existing place where you have the data. Kind of like what we've talked about for decades of like, are you bringing computer to data? Are you bringing vibes to the data, right? Are you bringing code gen to your own platform?

• 2:26:47John Coogan?

  Yeah, I mean, you used to you used to bring like a like a, you know, dashboard builder and it would have a couple widgets. And now I could just potentially, if I'm plugged into some sort of data source, some system of record, I could say vibe code this app on top of it. And there's been some tool retools played in this space, Zapier a little bit. But yeah, I mean, this feels like, you know, we're. Getting, we're not fully in the, just the pixels are generated, but we're, you know,

• 2:27:12Guillermo Rauch?

  We're getting close.

• 2:27:12John Coogan?

  UI generative application on top and that, and that being bespoke and ad hoc.

• 2:27:16Jordi Hays?

  I also think it's important to understand the line between consumer vibe coding and just generating ephemeral software and websites and things like that versus enterprises, which will have a lot of different use cases. When I look at the, when I look at the vibe coding market and I see businesses that are, that, that are almost entirely consumers just creating things for fun. I think that has to be a tough business because it's a hyper competitive market and consumers are flaky. They'll create something, you know, for fun, but they'll churn in month two because, you know, it's not, they're not running a real business. Whereas a, a business knows, hey, we'll pay for this on, on a long-term basis

• 2:27:57Jordi Hays?

  because we're, we have a use for it all the time from this product manager to an engineer over here to somebody in marketing, et cetera.

• 2:28:03Guillermo Rauch?

  Yeah. The, the other side of the, of the equation is how do you make this vibe coding tools work really well for enterprises? Frankly, the most surprising emergent thing that I've learned is just how much demand there is in enterprises for vibe coding. And this is because a lot of the, the traditional thing has been the people that understand the business are sitting over here. The people that understand the code are sitting over here and their communication is fraught with peril. Like they don't speak the same language. They kind of like resent one another. I love to tell this story. I was meeting with the CEO of a very successful company. He was telling me that engineers, like asking a feature to his own engineers felt like petitioning the government.

• 2:28:46John Coogan?

  Yeah.

• 2:28:47Guillermo Rauch?

  Even though he's the CEO, it's like he's struggling to like make the case. And please like get me in your next sprint, get this feature. So vibe coding actually solves that problem. All of the PMs, designers, marketers, business users that previously only had access to what like Jira.

• 2:29:06Guillermo Rauch?

  And, you know, to do a little bit of product management tools and writing PRDs and

• 2:29:11John Coogan?

  So rude.

• 2:29:12Guillermo Rauch?

  Those kinds of things. They weren't able to ship PRs. They weren't able to, you know, ship software and now they can. And so the, the opportunity is how do you actually make this secure?

• 2:29:22John Coogan?

  Yeah.

• 2:29:22Guillermo Rauch?

  How do you make it high quality?

• 2:29:24John Coogan?

  Yeah.

• 2:29:24Guillermo Rauch?

  How do you create guardrails? And those are, those are tricky problems and I'm, I'm really happy that some of them are easy to overcome and at least for us. And some of them are active areas of research, but I think the, the enterprises really have a, a strong case for this.

• 2:29:38John Coogan?

  Yeah. Can you walk me through like tool use? I mean, we were talking to the OpenAI folks about GPT-5 being like really like a summation of like standing on the shoulders of giants. You get a Python REPL, you get a web browser, you get, you know, the ability to kind of run cron jobs now. There's voice and, you know, all sorts of different tools kind of wrapped up into one, multiple models. You can trigger reasoning chains if it wants to can do all these different stuff. And that's actually the benefit of like, this isn't just a bigger model. It's like a lot of, it's a, it's a next version of a thing. It's more like switching from the iPhone 12 to 13, then going from the iPhone to the iPhone 3G. It's not just a new technology that's in there. But in the, in the, in the world of vibe coding, what are the tools that you want to think about adding?

• 2:30:20John Coogan?

  I know that basically every vibe coding platform, you know, recommends a database. But I was, we were talking to Harley at Shopify yesterday and there's a world where if I go to a vibe coding platform and I say, I'm building an ecommerce website, it should probably just be like, hey, I'm going to do Shopify under the hood and I'll vibe code the landing page on top. But how are you thinking about the landscape of like tools that you could pull in, full open, because there's open source repos that are like full projects that you could pull in and then just start customizing on top of. It's kind of this big continuum.

• 2:30:51Guillermo Rauch?

  Yeah, there's a couple of layers.

• 2:30:53John Coogan?

  Yeah.

• 2:30:53Guillermo Rauch?

  On the foundation model layer, what you want is a model that is exceptional at tool calling.

• 2:30:58John Coogan?

  Mm-hmm.

• 2:30:58Guillermo Rauch?

  Whether it has built in tools or whether you register them yourself.

• 2:31:01John Coogan?

  Mm-hmm.

• 2:31:02Guillermo Rauch?

  This is a like sort of silent war that has been going on. Like if you talk to devs, what are you optimizing for? Tool calling quality. Why? Because to demystify the word agent, what an agent is. It's a loop of tool calling that builds up context over time. That's all an agent is. So let's, to give you an example concretely of B0. B0 is becoming more and more agentic over time. One of the things that it can do is it can take a screenshot of the thing that's building and reflect on it. So today I live vibe coded to an audience of web three and crypto engineers. And I told B0, hey, make this dark mode. And initially, B0 does me dirty. He's like, he changes some things with dark mode.

• 2:31:47Guillermo Rauch?

  And then it kind of astonished me because I was like, oh, I have to explain to this audience. It then takes a screenshot, looks at it and keeps fixing it. And I was like, this is literally a developer that's alive on autopilot. And the reason it's in autopilot is because he has access to these tools, like looking at the web browser. Another one is research. I vibe coded an example of build me a Substack clone for cryptocurrency news. And the agent didn't know what the cryptocurrency news were. So he started doing research on the internet of, okay, Ethereum passed certain price and whatever. So, and then you're talking about the tools over the internet. So to demystify another topic, MCP is really exciting because it's a new protocol

• 2:32:32Guillermo Rauch?

  for registering tools that your agent doesn't locally have. So those tools that I just talked about, we gave them to B0. Here's a deep research tool. Here's the screenshotting tool. And those will likely become the new services. When you think about like AWS of today, if AWS was an AI cloud, which is kind of what we're trying to build at Resell, like you'd think a lot of those tools are going to become as a service. Like bring me deep research as a service. Bring me browsing and screenshotting as a service and so on. But then you have MCP which allows you to, okay, I need to sell something online. All right, so now there's an MCP for Shopify. Now there is an MCP for Stripe. There's even crypto MCP. So it's really exciting. Like now it's like the ultimate choice for a builder.

• 2:33:17Guillermo Rauch?

  And you don't have to go and learn all these things. You don't have to, th- th- this is almost like a discontinuity of the valley trend of like, if we build amazing documentation, they will come. This is more so if the agent picks you. They will come, right? And so there's a lot of figuring out right now, like, how do I make my infrastructure? How do I make my product to be loved by these agents? And MCP promises to be one of these first things that you are in control of.

• 2:33:44John Coogan?

  That makes a ton of sense.

• 2:33:45Jordi Hays?

  Last question, someone on your team named Josh is in the chat. He wants to know, what, what does he need to do to get a Twitter badge?

• 2:33:55Guillermo Rauch?

  Well, yeah, 100K downloads of the AICI. I think we've been talking.

• 2:33:59John Coogan?

  Okay.

• 2:34:00Guillermo Rauch?

  Good work.

• 2:34:01John Coogan?

  The gauntlet's been thrown down.

• 2:34:03Guillermo Rauch?

  Thank you.

• 2:34:04John Coogan?

  It's on record.

• 2:34:05Jordi Hays?

  You've got your work cut out for you.

• 2:34:05John Coogan?

  It's burned into the immutable record of this live stream and the future training runs.

• 2:34:10Jordi Hays?

  Best of luck, Josh.

• 2:34:11Guillermo Rauch?

  I'll be held accountable now.

• 2:34:12John Coogan?

  We're gonna hold you accountable to that, Guillermo. Great seeing you.

• 2:34:15Jordi Hays?

  Awesome. Great to see you.

• 2:34:16John Coogan?

  We'll talk to you soon. Congratulations.

• 2:34:18Jordi Hays?

  Talk soon.

• 2:34:18John Coogan?

  Let me tell you about fin.ai, the number one AI agent for customer service. Number one in performance benchmarks. Number one in competitive bake offs. Number one in G2. Number one in having an Irish founder.

• 2:34:32Guillermo Rauch?

  That's right.

• 2:34:33John Coogan?

  And we will invite our next guest to the stream from factory.ai. Welcome to the stream. How are you doing? Good to see you.

• 2:34:42Speaker 10

  Hey, how's it going? Glad to be here.

• 2:34:43John Coogan?

  It's great. Thanks so much. Kick us off with an introduction on you and the company.

• 2:34:47Speaker 10

  Yeah, my name's Eno, co-founder, CTO at Factory. We are building a platform for enterprise software developers to perform what we call agent-driven software development. So basically, more than just code, bringing agents into every stage of the software development lifecycle. So think coding, code review, maintenance, incident response, documentation. We think agents should be a part of all of this. And we think that they should be driving a lot of that menial component while you think at the high level about how to plan and structure the work.

• 2:35:21John Coogan?

  There's so many different. Like enterprise is a narrow category. It's a, you know, not consumer, I guess, but it's such a wide, it's such a wide category. Is there a beachhead? Is there a certain type of project within, within different industries or specific industry? That's getting especially, an especially large amount of value out of factory these days.

• 2:35:42Speaker 10

  Yeah, totally. I think that one thing that we see a lot, and typically when we say enterprise, we're thinking greater than 1,000 engineers, right? Like 2,000, 3,000. And one reason why we focus on that larger scale, you tend to have these large organizations where people are, the bottleneck is not code, right? The bottleneck is how do we plan a migration of 185 code bases to this new framework? And there are 3,000 developers that are going to touch this over the next six months. And an SI just told us the quote is $80 million to do it. And we have to figure out how to not.

• 2:36:23John Coogan?

  Re-platforming broadly is one of the major, major tasks for many, many enterprises, right?

• 2:36:30Speaker 10

  A hundred percent. Modernization and migration is huge.

• 2:36:33John Coogan?

  Yeah, yeah, that makes a lot of sense.

• 2:36:35Jordi Hays?

  How do you estimate the market, that market size? And is that, is that what you guys are leading with on the GTM side in terms of trying to find these legacy companies? That are maybe not even using cursor yet. I mean, we, who, we, we talked to the CEO of GitHub yesterday and what?

• 2:36:53John Coogan?

  50% didn't he say?

• 2:36:55Jordi Hays?

  It was like at least half of their user base is not using any AI tools.

• 2:37:00John Coogan?

  Yeah.

• 2:37:01Speaker 10

  Yeah, totally. I, I think that the, the, the thing that we hear often, we pretty much only deploy into companies that have already tried an AI native IDE or have an autocomplete tool deployed. And I think that the thing that we hear often is you, you sort of hear like these numbers thrown around like 5X, 10X. And then in practice, when you adopt an AI IDE, you see 10%, 15%. And so a lot of people are sort of saying like, what is the delta there? Like what causes that transition? And our, our sort of argument here is that there is. Workflow change that's actually required to really adopt agents in the lifecycle, right? And so if you're just sort of like accelerating an individual developer, that you can go a little bit faster.

• 2:37:45Speaker 10

  But if you are able to parallelize and automate at scale, that is going to be that larger introduction of change. And so if you imagine the market here, there are companies where, you know, five or 10% of global payment transactions run on some COBOL system that was written 40 years ago. Every developer is gone and they are, it's a ticking time bomb. Like at some point that needs to go to Java, but there's nobody who even knows how to do that. And so those are the types of projects where the market is so enormous because, you know, half the business runs on this legacy system, hundreds of billions of dollars.

• 2:38:22John Coogan?

  Put it all in Lisp, skip Java, go straight to Lisp.

• 2:38:25Speaker 10

  Exactly, Python, right?

• 2:38:27John Coogan?

  Yeah, Python would be the logical one.

• 2:38:32John Coogan?

  I'm sorry, we're running behind, so we're gonna have to cut this short. But I want to know more about how the enterprise coding agent market will develop. We could see one world where we wind up with, you know, GCP, Azure, AWS, like, you know, pretty comparable competitive. They've all had really great margins. It's been this oligopoly. There's another world where you could see more specialization. One of these companies goes deep into high security environments, or oil and gas, or financial environments, or specializing based on specific programming languages. As, as the market develops, like, how do you think it'll play out?

• 2:39:14Speaker 10

  Yeah, great question. I, I think that what's very clear is that the bulk of very large enterprise has a lot of similar problems, refactors, migrations, modernization. So a platform like Factory is able to deploy into that and solve problems quickly. I, I think that there's likely to be like that sort of 80/20, where there are going to be these very specialized providers that only focus on one sort of problem. And that will represent maybe like 20% of what's out there. And so it won't be like necessarily black or white, but we do think that the bulk of enterprises have a lot of similar needs, especially when you just get across a certain threshold of number of engineers, scale of codebase.

• 2:39:54John Coogan?

  Sure, sure. Yeah, I mean, we even see that with the clouds where, you know, obviously there's the hyperscalers, but then there are neoclouts. And we talked to Armada where they'll send you a shipping container with a bunch of racks inside and put it in stranded energy. So there will obviously be a long tail here. That's a great take. Thank you so much for stopping by. Have a great rest of your day. And enjoy the GPT-5 upgrade. We'll talk to you soon.

• 2:40:17Jordi Hays?

  Have fun out there.

• 2:40:17John Coogan?

  Really quickly, let me tell you about Adio, customer relationship magic. Adio is the AI-native CRM that builds, scales, and grows your company to the next level. And we will be joined by our next guest from Augment. Welcome to the stream. How are you doing, Guy?

• 2:40:33Speaker 11

  Great. Thanks so much for having me.

• 2:40:35John Coogan?

  And that's his name, by the way, if you're listening. His name is Guy. I'm not just calling him Guy. Anyway, please introduce yourself and what do you do? What does your company do?

• 2:40:43Speaker 11

  Yeah, so I'm Guy Gurari from Augment Code. I'm a co-founder and the chief scientist. And we build AI coding assistance for large teams with large code bases. And so you can use Augment Code to do question answering, to do development, to do refactoring, to do migrations, all the tasks that you do, except that our product understands your large code base really well. And so that means less prompting for you and faster and better results out of the agent.

• 2:41:09John Coogan?

  Today, GPT-5 launches. It's kind of a rising tide. Feels like it lifts all boats. Every company gets access to it. We've interviewed a number of companies that are building on top of GPT-5.

• 2:41:20Jordi Hays?

  Except it drowned GPT-4 or 5.

• 2:41:22John Coogan?

  Yes. But in general, how do you think you can use GPT-5? Are there any pockets of value that you think you can uniquely take advantage of?

• 2:41:35Speaker 11

  Yeah, great question. So we've been trialing the model for the past few weeks. And what we found is that the GPT-5 is a very thoughtful model. It likes to make a lot of tool calls. It likes to ask clarifying questions of the user before starting to make code changes. And so the place where I reach out for GPT-5 is typically if I need to make large changes or if I'm trying to answer a very difficult question about the code base, I will let GPT-5 take a crack at it. It will churn for a while, making lots of tool calls, just making sure it got it right. And probably find all the different places in the code where it actually needs to make a change. And so I will typically let it run in the background and come back to it.

• 2:42:18Speaker 11

  And I will often get a high-quality result out of it.

• 2:42:22John Coogan?

  Are there any features or integrations that you're hoping GPT-5 will roll out in the future? We talked to a couple of people who were like, like, we want models that have access to as many tools as possible. And, and you can see with the MCP boom, more people are trying to make their, their services, their products accessible to these models. Is there anything that you see as potential low hanging fruit to just add to the capabilities?

• 2:42:53Speaker 11

  So I think for us, we work hard on developing our own integrations and our own tools, building them into the product rather than relying on GPT-5 or other model vendors to do so. We have worked closely with OpenAI to improve the prompting around our tools so that the agent kind of works flawlessly. I think the thing that would be very nice, I think one of the previous guests mentioned a screenshot tool. I think that's a very, yeah, that's a very nice way to close the loop on front-end software development. Just like we saw how on backend software development, running the tests automatically really helps the, the agent iterate until it gets to working code.

• 2:43:34Speaker 11

  So I think having more support for screenshotting and things like that that close the front-end gap would be very nice to see.

• 2:43:40John Coogan?

  I, I wasn't aware that, that screenshots weren't flowing through. I feel like when I've, when I've triggered operator, I'm getting a, a view, a web view into the website, but I wasn't, I wasn't aware that that wasn't like being passed through easily in the API and you still kind of needed to build that yourself. What, where else, we were just talking about this, like, where are the biggest pockets of value right now for AI coding tools generally? Obviously, everyone knows like the vibe coder, who's just the designer who's learning how to use software for the first time. Then there's the experienced developer going from a 10x to 100x with better code completion. Then there's the enterprise that's, you know, maybe doing replatforming.

• 2:44:24John Coogan?

  Where else are the interesting pockets of value that are maybe on the horizon to be unlocked with new models?

• 2:44:32Speaker 11

  Yeah, so on top of everything you mentioned, certainly the the inner loop of software development, that's where we've spent most of our time at Augment Code developing product for. Yes, you can have a senior developer, starting using agents, starting to use multiple agents in parallel and unlock 10x or more productivity gains. What we're starting to see now with our tools is the beginning of automating software development lifecycle tasks. So with with Augment Code, we have a CLI tool now where you can take the full power of our context engine and the agent, the thing that really understands your codebase, and you can start automating tasks in the background. And so we're seeing more and more developers saying, oh, this is great. Like, I can break out of the IDE now.

• 2:45:15Speaker 11

  I'm using the agent that's already familiar to me, but I'm starting to automate code reviews. I'm starting to automate incident response. I'm starting to automate looking at production logs and automatically assigning tickets based on error logs that I'm seeing. All kinds of new automation use cases that we're seeing just because agents have gotten so good and kind of really understands your codebase.

• 2:45:38John Coogan?

  Are there are there high stakes pockets of software engineering work that most of the AI tooling has kind of stayed away from? I'm imagining like the the the high stakes database migration.

• 2:45:51John Coogan?

  Where where's the the the kind of sticky part of the industry? I was reading a blog post by someone who's doing like very advanced cybersecurity pen testing, and they were saying like, just the creativity of the models wasn't quite there yet to really come up with the to really act and embody like a white hat hacker who is going for a bug bounty. But where where are the pockets of still like intractability? Where, I guess, if you are, you know, in the in the individual contributor, you love just just, you know, coding from scratch, that's where you want to stay for at least the next couple of months.

• 2:46:29Speaker 11

  Yeah, I think still the attention of all the models we've seen and all the agents we've seen around making proper design and architecture decisions. Um, that's still high stakes and still the ability is not there because if you do complete vibe coding and you just let the agent go and do whatever it wants, in the beginning, it looks amazing, the code works and it's all really good. But once you get to low tens of thousands of lines, the bad decisions that were often made around the design and architecture start to show up and development slows down. So that's where we still see a limitation of today's agents and where you still have

• 2:47:12Speaker 11

  to supervise the agent fairly closely uh in order to make sure that you don't get stuck later on. Uh, perhaps this will change in a year, but today I would say all these decisions that you make around how the code is structured uh still requires close supervision and still high stakes because it can really slow your project down if you let it go autonomously for long enough.

• 2:47:33John Coogan?

  That makes sense. Well, thank you so much for stopping by. We will talk to you soon. Have a good rest of your day.

• 2:47:37Speaker 11

  Thanks very much.

• 2:47:38Jordi Hays?

  Cheers.

• 2:47:39John Coogan?

  Let's check in with Tyler on the timeline. Tyler's manning the timeline. How are the vibes? Are there any new posts that have hit the timeline? Are we still in turmoil or has the narrative settled?

• 2:47:49Speaker 01

  I think vibes are are picking up a little bit. You're starting to see people post like, oh, this is something I made. Um, now you can see on LM Arena, it's number one.

• 2:47:58John Coogan?

  No way. Wait, wait, what's what's going on with the poly market then?

• 2:48:01Speaker 01

  So poly market is still.

• 2:48:02John Coogan?

  Still still Google heavy?

• 2:48:05Speaker 01

  Yeah, I think I guess they're just pricing in Gemini three.

• 2:48:08John Coogan?

  Oh, okay.

• 2:48:09Speaker 01

  Um, I'm not exactly sure, honestly. I was actually very surprised to see that it was number one.

• 2:48:13John Coogan?

  Yeah, yeah, yeah.

• 2:48:14Speaker 01

  Um, but yeah, maybe later we can show some of the posts.

• 2:48:16John Coogan?

  Yeah, yeah, yeah. Yeah, that'd be great. Um, well, in the meantime, before our next guest, let's tell you about Eight Sleep, Get a Pod Five, five-year warranty, 30-night risk-free trial, free returns and free shipping. And we will uh have our next guest join us from Code Rabbit. How are you doing? Good to meet you. Good to meet you as well. Thanks for having me here. Uh, what's your reaction to GPT-5? How long have you been playing with it? What are the biggest improvements that you've noticed?

• 2:48:45Speaker 12

  Yeah, I would say mind blowing, right? I mean, we have been playing, our team has been playing for like a few weeks now. Tested a few snapshots. It's amazing. It's a generational leap, we would say. Like, we have been using OpenAI models, like, I mean, how much you know about CodeRabbit? It's been like a couple of years we have been on OpenAI Anthropic. And our product is a very reasoning heavy product, like one of the very few use cases where you have have a PhD style problems where we have to do code reviews. That's what CodeRabbit does, like, users open up pull requests, our agent uses reasoning models to find issues like risk conditions or security issues and so on.

• 2:49:26Speaker 12

  Yeah, so we've been testing GPT-5 on some of the hardest pull requests we have in our golden data set. So we maintain a data set where we track progress of different models and progress of AI in general. So we have like many problems that no model is able to use, uh, solve so far, like I mean GPT-5. But so far it has the highest score. We would say it's like almost two X better than the next O three or Solid or Opus at this time.

• 2:49:50John Coogan?

  What's the customer value there? You think that, uh, like, all the customers just notice that the product gets better? Are you gonna upsell folks? Are you gonna, uh, like, how do you play this? Given that this model is now in public availability, every company, every competitor can can access it as well?

• 2:50:09Speaker 12

  Yeah, there is no upsell. And that's the thing with AI, for the same price or even like better prices, you're getting much more AI, much better AI. Like, that's the whole idea, how fast this space is evolving. So yeah, from the pricing point of view, we don't see like this to be like a separate plan or something in our product. I mean, the product for the same price per month, customers will now just get better quality of results with CodeRabbit.

• 2:50:34John Coogan?

  What's next for the business? What kind of customers are you going after? Who do you think has been on the fence and this release is going to be the thing that gets them to actually jump into the world of AI?

• 2:50:49Speaker 12

  Yeah, we could track the top line metric like one of the things we track very closely in the company is like how many signups to the paid customers we get.

• 2:50:57John Coogan?

  Hmm.

• 2:50:57Speaker 12

  That number has been constantly improving since G P D four, uh, G P D four turbo, G P D four actually dipped. So there was a time when G P D four was almost like a Windows Vista of releases like. Um, it's funny like how we kind of trusted the evals and we thought it's the same model, but in a way it was inferior in many ways. Um, then we saw a huge improvement after O one came out, O one preview was a game changer for us even at that time, our conversion doubled actually. You know, right? I mean, so we went to like more like uh, close to thirty percent success in getting the paid users. And now with G P D five, we are hoping we can see another big jump in the, in the number of people who start becoming paid customers and and how many people churn. So those are the real numbers.

• 2:51:41Speaker 12

  Like one is like vibes, like how people like respond to the model and we get angry tweets or not, I mean, though that's the other part. Uh, but the other thing is like the actual uh, revenues, uh, whether it moves the needle for us and that remains to be seen. Like one of the things we have seen, even though you test these models in a lab, they be, it's not like a huge data set, but once you actually are in the wild, you see hallucination, some of those scale, issues at scale pop up. So those are something we'll still be observing over the next few days to see whether it's like spot only like um, eighty percent of the cases, but then if the false positivity or the hallucinations are too high, then also it's not a great model, but that remains to be seen.

• 2:52:16John Coogan?

  Yep, that makes a lot of sense. Uh, well, thank you so much for stopping by. Congratulations on a new, new tool in the tool chest.

• 2:52:23Jordi Hays?

  New toy.

• 2:52:26John Coogan?

  Uh, we will talk to you soon. Have a great rest of your day.

• 2:52:28Jordi Hays?

  Cheers.

• 2:52:28John Coogan?

  Goodbye. And let me tell you about public dot com investing for those who take it seriously. They got multi asset investing, industry leading yields. They're trusted by millions.

• 2:52:38Jordi Hays?

  Millions.

• 2:52:39John Coogan?

  Um, the, the chat is going wild about public trading, the S P X six, six thousand nine hundred. I think that comes from someone talking about like the non mag seven stocks or something. There's been uh, people benchmarking the, the mag seven versus the.

• 2:52:55Jordi Hays?

  The big news while we were live or earlier today, Trump signed an executive order that is opening up four one Ks to digital assets and private equity.

• 2:53:04John Coogan?

  Uh, what, what's, what's crypto doing? Is it ripping?

• 2:53:08Jordi Hays?

  Bitcoin is up a couple points last time.

• 2:53:11John Coogan?

  This point, you know, where's it gonna go? It's already so high. I mean, it's just like, there's been so many catalysts.

• 2:53:17Jordi Hays?

  John, I'll tell you this, it could go up, it could go down.

• 2:53:20John Coogan?

  Yep.

• 2:53:20Jordi Hays?

  We'll have to wait and see.

• 2:53:22John Coogan?

  Tyler, anything else notable from the timeline? What have people built? I see this GPT-5 just one-shot at a Minecraft clone?

• 2:53:31Speaker 01

  Yeah, I think that's one of the cooler things I've seen.

• 2:53:33John Coogan?

  Okay, so this is, so it wrote, it wrote code to generate this game. It's not generating the pixels. You can do so many different things. Like, you could generate, you could generate a video, generate a world model, generate code that generates a game engine. You could generate code that runs on Unreal Engine. I don't even know what they're using.

• 2:53:50Speaker 01

  One thing now, in, on actual ChatGPT, in, there's like a native, like, it's like a music player. It's almost like GarageBand. You can say, like, if you prompt to, like, build, I saw Sam Altman tweet about this, you prompt, like, to do some kind of, like, beat or something, it'll, like, make an interactive, like, GarageBand almost interface in there.

• 2:54:08John Coogan?

  That's cool.

• 2:54:08Speaker 01

  I was playing with that earlier.

• 2:54:09John Coogan?

  Yeah, I do wonder how many of these features that we're seeing, like, where does OpenAI want to keep things in the B2B world and let other companies build versus just build it as a consumer app?

• 2:54:20Speaker 01

  Yeah.

• 2:54:21John Coogan?

  Like, will ChatGPT eventually just let me push your website? Like, will it, will it become a vibe coding platform? At least, like, a basic one. Like, it's not, it's not the most advanced coding environment, but it can definitely write some code and execute it for you and do some stuff.

• 2:54:38Speaker 01

  Yeah, well, it's funny, because, like, it used to be, you would have a, like, so there was, like, GPT-3.5 or something, and people on top of that built a vibe coding thing. So you could use that to build your own vibe coding thing.

• 2:54:50John Coogan?

  Yeah, yeah, yeah.

• 2:54:50Speaker 01

  But now you can just go straight from ChatGPT to build your vibe coding platform.

• 2:54:55John Coogan?

  Yeah.

• 2:54:55Speaker 01

  But soon, maybe, it'll just be the vibe coding platform.

• 2:54:58John Coogan?

  Yeah, the surface area of this stuff is very interesting. Clearly, they're going after healthcare and therapy. It's interesting that they've kind of stayed away from legal. Even maybe that's just the the the dynamic of the sales process and the dynamic of that particular market. But I mean increasingly, you can you can just ask more and more questions of chat GPT. So the the consumer to business like bleed over. There's certainly a world where just giving everyone in your organization chat GPT is a substitute for a bunch of different SaaS products. So be interesting to see where that develops. What what are you thinking about?

• 2:55:35Jordi Hays?

  Near says there are concerns that the number used to represent our AI's intelligence does not in fact represent its intelligence. Worry not. To address these allegations, we've added three new numbers.

• 2:55:45John Coogan?

  Near. Yeah, near near is building something that's like not particularly benchmarkable, right? Isn't it companion?

• 2:55:52Jordi Hays?

  It's beyond benchmarks.

• 2:55:54John Coogan?

  Beyond benchmarks. Well, in completely other news, Andorl opens a Taiwan office and begin selling AI powered attack drones to Taiwan. Paul Merlucky has said he wants to turn Taiwan into a prickly porcupine. We're in the age of spiky intelligence. That spiky intelligence will be on boarded onto the AI powered attack drones and deployed in Taiwan to keep it safe. What else is going on in the timeline while we wait for our next guest from OpenAI to join?

• 2:56:24Jordi Hays?

  Spore says, raise your hand if you were not automated today. I'll raise my hand.

• 2:56:29John Coogan?

  I was not automated today.

• 2:56:30Jordi Hays?

  Not yet.

• 2:56:31John Coogan?

  We survived. We made it through. Sebastian Buback says, here at OpenAI, we've crashed, we've cracked pre-training, then reasoning, and now we're experimenting with new set of techniques that maximally leverage their interaction. GPT-5 is just the first step in this direction. We're excited to incredibly excited to see where scaling this up will lead us. And it's the unicorn test, I believe. And the latest unicorn is really, really good. That is, that is a creative interpretation. And I think it has to draw this with like SVGs. Anyway, we can talk to our next guest about it.

• 2:57:02Jordi Hays?

  Last post, Jerr ticket says, I went to the permanent underclass party and everyone knew you.

• 2:57:09John Coogan?

  Anyway, uh, back to the serious interviews. Welcome to the stream, Max. Good to see you. How are you doing?

• 2:57:15Speaker 13

  What's happening? Nice to meet you guys. Yeah, uh, doing well. It's a relief to have this launch out in the world. I think it's, uh, you know, we've been working on this for the last few months now, and it's exciting to let the whole world see what we've had.

• 2:57:27John Coogan?

  Yeah.

• 2:57:28Speaker 13

  Just a few months?

• 2:57:30Speaker 13

  It's been, uh, I don't know. It's been a little while.

• 2:57:32John Coogan?

  What's the, what, what's the actual launch day like? Cause you're actually getting this out into the world. Uh, the GPUs are on fire or about to be on fire, warming up. But, uh, is that, is that out of your purview? There is a different team for that, fortunately.

• 2:57:46Speaker 13

  So, right. So I run, uh, I ran a lot of the research for GPT-5. I don't necessarily handle the deployment, but I do get dragged in when the GPUs are on fire. Um, I, I think we're, we're moderately burning right now.

• 2:57:59John Coogan?

  Okay.

• 2:58:00Speaker 13

  Like a two alarm fire.

• 2:58:01John Coogan?

  Yeah, yeah, yeah. Uh, is it, is it materially different? I mean, this is a launch day, but, uh, we'll probably discover like the Studio Ghibli capability. Once it gets out into the long tail of like, you know, hundreds of millions of people try it. Someone comes out with some genius thing, then everyone's doing that, and then the GPUs. Cause I, I feel like the Studio Ghibli thing happened like a few days after the launch of Images in ChatGPT.

• 2:58:24Speaker 13

  It, it did. It was, um, it was pretty fast, but within about a week. I think in this case, we're gonna, we're gonna see that here.

• 2:58:31John Coogan?

  Okay.

• 2:58:31Speaker 13

  Um, I think coding, you know, if I had to take my bets for what the Studio Ghibli thing is gonna be, it's, it's coding. Um, that's the place where I think GPT-5 is like most tangibly a huge leap ahead of GPT-4 and ahead of 03.

• 2:58:43John Coogan?

  Do you think there's a chance that, that, that the coding will mean a Studio Ghibli style, uh, meme? Or, or kind of like, uh, and, and what I mean by that is that, is that like image generation is incredibly valuable in the context of like Hollywood will be using AI to chroma key and, and rotoscope in a professional environment. But what was special about Studio Ghibli was that anyone was making these custom images. And I could imagine a world where, you know, even going from like the Levels I/O example of like I Vibe coded a flight simulator. If we wind up in a Studio Ghibli moment for coding, I would imagine it's like everyone built their own game today.

• 2:59:24Speaker 13

  I think that's pretty much it. Yeah. So I, I don't know if you guys watched the live. What, what do you? That was one of the things we had on the live stream. Like you can just go into ChatGPT. If you try it right now, it might or might not work because the G P five rollout is still ongoing. But if you have five, you can just tell it like, basically make me a game.

• 2:59:40John Coogan?

  Yeah.

• 2:59:40Speaker 13

  And it will make it and you can actually play it in chat G P T.

• 2:59:44John Coogan?

  That's amazing.

• 2:59:45Speaker 13

  So I, yeah.

• 2:59:46Jordi Hays?

  Is there the ability?

• 2:59:47Speaker 13

  Discover that you don't, the thing is like, like with Studio Ghibli, right? Like for, for Ghibli, you don't have to know how to draw to make it work. For this one, you don't have to know how to code.

• 2:59:54John Coogan?

  Yes, but.

• 2:59:54Jordi Hays?

  So can you share, can you share that chat and someone else can play the same game? How, how does the kind of sharing mechanism?

• 3:00:00Speaker 13

  Yeah, there's a, there, you can do the share link.

• 3:00:02John Coogan?

  Yeah.

• 3:00:03Speaker 13

  We're, we're, I think, going to try to make sharing for these a lot better over the next few days. That was P two after the P one and P zero of making the G P U's not completely melt.

• 3:00:12John Coogan?

  Yeah, yeah, yeah.

• 3:00:13Speaker 13

  But yeah, I, we will try to make it much more durable.

• 3:00:16John Coogan?

  Yeah, yeah. I mean, the, the Studio Ghibli thing is so interesting because it was, it's not just that the model capability was there, but it's also like, the prompt was two words and, and, and it was so reliable that you always got a good result and you could personalize it. So even if it wasn't, I, I've seen people build Doom. I've seen people, you know, you can just buy Doom. It's a real game. You can build it. Um, but if you build it and I'm like, oh, that's cool. You did it in a vibe code environment or in chat G P T like that's awesome. But I don't necessarily want to go do that for myself. But as soon as it becomes personal, which is what the studio, I, I had to see what I looked like as Studio Ghibli. I had to see what my favorite photo looked like, my favorite meme looked like in Studio Ghibli.

• 3:00:57John Coogan?

  And once that happens with games, people will, will, will eventually, uh, you know, there'll be this like memetic explosion and you'll see the G P U's will truly be on fire.

• 3:01:06Speaker 13

  Yeah. I mean, I think even today, you could probably with G P D five do Doom, but all of the characters are like, all the enemies are headshots of your friends, like.

• 3:01:13John Coogan?

  Here we're going.

• 3:01:14Speaker 13

  I, that will just work.

• 3:01:15John Coogan?

  Real close. Yeah, we're real close. It's going to be something that's personal, something that, you, you know, you can express your own creativity through. Cause I think people, they still latch on to that. They don't just want, uh, you, you know, a copy of what already exists. They want something new and, and in the Studio Ghibli moment was just new enough. Anyway, we should talk about actual research. We should talk about post-training. Um, what, uh, what, what's the thing you're most proud of? Like, like what can you give us on, without, you know, immediately getting poached? Uh, what, what, what can you give us on, on the actual innovation that went into G P T five? From a post, post-training perspective, what are the, like, the, the kind of keywords and, and paths in the tech tree that we should be digging into over the next few years to understand how this works?

• 3:01:54Speaker 13

  You know, I would say the thing that is most impressive to me about GPT-5 is how much getting all of the details right matters. Like, when I look at GPT-5, you know, we had an early version of this thing a while ago that was kind of okay, but clearly did not meet our bar for revolutionary. And we were trying to figure out, you know, why, why is that not as good as it should be? And the team basically just went off and did a deep dive over a couple of months of just completely rebuilding the post-training stack for this model. And turns out that when you do that, you get what would have taken, you know, another order of magnitude worth of pre-training improvements to, to produce.

• 3:02:34John Coogan?

  How much are you thinking in post-training, in research, about, let's forget the benchmarks and just focus on user satisfaction, like NPS score, basically, or like users, user minutes, or any of these other, uh, the real benchmarks?

• 3:02:50Jordi Hays?

  Yeah, the intangibles.

• 3:02:51John Coogan?

  Profit, revenue.

• 3:02:52Jordi Hays?

  How people feel when they're using it.

• 3:02:53John Coogan?

  Yeah, the feeling and the, and the joy and the actual value that's delivered. Because Studio Ghibli was a delightful moment, it wasn't a benchmark.

• 3:03:03Speaker 13

  Yeah, I, I think, so that was something that we took very seriously for GPT-5. It was like, look at what people are actually doing with ChatGPT and look at where the model is failing them. Either in the sense that the model is like, sort of like you said, it's not enjoyable to use.

• 3:03:16John Coogan?

  Yeah.

• 3:03:17Speaker 13

  And so we, we did, I think, make a lot of progress on that. Like, GPT-5 is much more engaging than our previous really smart models. Like, O3, I, I don't know if you guys talked to O3 in the past, it's a bit bland.

• 3:03:29John Coogan?

  Sure.

• 3:03:30Speaker 13

  And GPT-5, I think, has a lot more character, is a lot more, more interesting. But then also, like, I think for, we really care about just actually being accurate. Like, giving, if a user is trying to do something economically valuable with our model, we want to make sure it lands correctly.

• 3:03:47John Coogan?

  Yeah.

• 3:03:48Speaker 13

  And so what we did there is just like, look at the actual distributions of what people are doing with our models in the real world, figure out where the models are going wrong, build interventions to target it. And that was where, you know, we got, I think, the most impressive improvements in GPT-5. Like O3 would just get things wrong and not tell you it wasn't sure it was incorrect. And GPT-5 is much, much better about like actually being honest when it thinks it might not know.

• 3:04:12John Coogan?

  Yeah. How explicit are all the different pieces of the post-training pipeline? Like you have, you have, you know, safety post-training, you have, stop hallucinating, give me the real facts, you have make sure the text, the flavor, the tone is pleasant. There's so many different things to optimize for. How much of that is like try and just blend it all up into one thing versus like explicit passes, chunk it out, like split it up? How much can you decompose the problem?

• 3:04:44Speaker 13

  So, you know, my background is in reinforcement learning and I think when you look at something like this, the magic is in the reward function, right? It's in what you're actually telling the model to be good at. And so fixing things like hallucinations to a huge extent is essentially a function of just fixing the reward function, actually making it so that the model is reliably penalized for saying something that's false. And if you do that, all of a sudden the model stops saying things that are false. Ditto for safety, right? You know, on the live stream, Saatchi talked a bit about the way we've changed safety for this model. And to a huge extent, it's just a function of, we are actually putting out a paper today on the new safety stack for this model. And the core insight in that paper is just figure

• 3:05:28Speaker 13

  out what you actually want to optimize for, which in our case is helpfulness conditional on not saying something that's actually dangerous or harmful. You know, write that down, figure out what that means as a reward function, then optimize it for it. It's really not magic at all. It's just, again, it's what I said earlier, you've got to get the details right. You know, if at any part of that process you screw it up, the model will be unusable.

• 3:05:51John Coogan?

  What's your current thinking on spiky intelligence? And is there some, is there some flywheel that you can get started where you're identifying low points that aren't spiky enough? And then you're like almost automatically setting up the infrastructure, the eval to then RL against to create a spike?

• 3:06:14Speaker 13

  I think GPT-5 was a preview of what's possible in that respect in the future. Yeah. A step in that direction.

• 3:06:23John Coogan?

  Do you think that there's a world where you get to a place where you're kind of, it's weird because we're not hammering down the nails of the spikes, we're adding spikes.

• 3:06:31Speaker 13

  Hammering up the spikes, yeah.

• 3:06:33John Coogan?

  Weird metaphor that we're stretching a little bit too far, but is there a world where, where you can be doing post-training or just adding capabilities in a more iterative cadence? So that as soon as you identify something, the response can be, yeah, we don't need to wait until GPT-6 to fix this, we can just add this capability because, hey, we just found a pocket of users who are trying to do a thing and they're not super happy with the results and let's add this capability.

• 3:07:02Speaker 13

  Yeah, I think so. I mean, I think we, you know, we are gonna launch other models between now and GPT-6. I think it's relatively common knowledge, but we do update the model in ChatGPT reasonably often.

• 3:07:13John Coogan?

  Yeah, people talk about it all the time.

• 3:07:14Speaker 13

  Yeah, exactly. And, you know, I think we are now in a world where we can conceivably update that model and have it get materially better on capabilities too.

• 3:07:21John Coogan?

  Yeah.

• 3:07:22Speaker 13

  Not just on, you know, the personalities a little bit better than it was before.

• 3:07:26John Coogan?

  Yeah.

• 3:07:26Jordi Hays?

  Going back to your note on the new paper that I guess you guys are releasing today, when you talk about optimizing for helpfulness, is there, is part of that avoiding the model, you know, reinforcing? There's times when you want to reinforce and give kind of like confidence to the user that they're going down the right sort of like thought process and things like that. But then there's like a point where it can get too extreme in terms of maybe convincing a user of something that may be totally untrue. Is that, is that what the paper gets at or?

• 3:08:01Speaker 13

  Um, so it's not specifically about this, although I will say we do explicitly train the model to not lead users down bad paths. That's something that I think we've started taking much more seriously over the last few months. As we've realized, Sam talked about this a little bit, I think back in May. But ChatGPT is just way more important for people's lives now than it was a year ago or especially two years ago. And we do have to actually be very cognizant of what effects our models have on users. So yeah, we do very actively train models to not lead users down the right path. Don't fact-check me on the releasing today, I know we're releasing it. I believe it is soon. I think it's today, but I've also been in a hole dealing with launch all day.

• 3:08:43John Coogan?

  Yeah, we're not big on fact checks here. We're big on the truth zone, which is just the vibes and

• 3:08:48Speaker 13

  The vibes are we'll be publishing some information about the new safety setup.

• 3:08:52John Coogan?

  At some point. It's great.

• 3:08:54Jordi Hays?

  Yeah, I think I think a large part of the conversation around safety should be how reliant and and how useful the product has become to users and then the new level of care that you have to provide versus a while ago when it was just like people saying making a cute image or generating some texts that they were going to use in an email or an internal document and realizing this this vector of usage, which is this like companion confidant that is becoming so prevalent.

• 3:09:30John Coogan?

  Talk to me about post-training for big, big partners, enterprises, government organizations. What is transferring from the research that you're doing to something that can be offered as a as an enterprise level product?

• 3:09:49Speaker 13

  Yeah, so we do. OpenAI does partner with external companies to do essentially custom post-training. That is a thing that we do. And from that perspective, the stuff we do just directly transfers. I'll also say that we've put a lot of work into trying to make our models as general as possible. But to as large an extent as possible, if you want to get really good results from our model, you can do it right on the API just by actually telling the model what you want it to do.

• 3:10:13John Coogan?

  Yeah.

• 3:10:14Speaker 13

  Right. Like GBD 5, I think, is pretty comfortably our most steerable model ever.

• 3:10:18John Coogan?

  Yep.

• 3:10:19Speaker 13

  We've heard a lot of really positive feedback about this, especially from like folks like Cursor.

• 3:10:23John Coogan?

  Yeah. So if I came to you and I was like, I'm an enterprise and I need to generate a lot of Studio Gibli's, you'd be like, what are you doing? Just prompt it. But what are the examples of of of companies, organizations? Is it just is it just private information, private data sets that aren't available on on the open web? Or is it specifically like, there is enough data out there, but there's just not the economic incentive for your team to go and RL on, you know, gas station bench or whatever we're talking about here hypothetically?

• 3:10:57Speaker 13

  I think the answer is both. Yeah, it's definitely both. Because, yeah, we're not going to target, you know, as you said, gas station bench. Not on our own right now, probably, because it's not mostly what people are doing with strategy T.

• 3:11:10John Coogan?

  Exactly.

• 3:11:11Speaker 13

  If you have some application that's super valuable to you.

• 3:11:14John Coogan?

  Yeah.

• 3:11:15Speaker 13

  We can be convinced that it's important.

• 3:11:17John Coogan?

  Yeah, yeah.

• 3:11:18Speaker 13

  It's just not what our users are already trying to do.

• 3:11:19John Coogan?

  What's the state of reward hacking and fighting that in RL environments?

• 3:11:27Speaker 13

  You know, I think we've actually made a lot of progress. There was some discussion of this around O three, that O three was like a little bit deceptive in ways that felt reward hacky.

• 3:11:34John Coogan?

  Sure.

• 3:11:35Speaker 13

  G P T five is dramatically less deceptive than O three was.

• 3:11:38John Coogan?

  What's an example of how that would manifest? Like, do you have like a canonical case study?

• 3:11:44Speaker 13

  Yeah, I mean, the canonical thing is like you ask O three to write you some code. And instead of actually writing some code, it changes some unit test.

• 3:11:51John Coogan?

  Changes the test case, right? Which is, which is kind of hilarious. It's like one of the funniest things that AI has ever done. I understand that it's very bad and it's not what we want, but it is just like, it's kind of cheeky in my.

• 3:12:01Speaker 13

  It's kind of cheeky. It's also like, you know, I've, I feel like if you spend enough time around real software engineers, they do actually do stuff like this pretty often.

• 3:12:07John Coogan?

  I have 100% done that.

• 3:12:09Speaker 13

  I was gonna say, I also have done that. For formal reasons, I won't say that I did it at OpenAI, but I definitely did that.

• 3:12:17John Coogan?

  Yeah, of course, of course. This is natural.

• 3:12:19Jordi Hays?

  What do you think G P T six looks like? You guys, you mentioned that you're going to be shipping, you know, updates to five, but what, what, what are you most excited about? Where, where are you most excited about going?

• 3:12:30John Coogan?

  And, and just really quickly, give us the date that G P T six launches?

• 3:12:36Speaker 13

  Oh, man. Hopefully, we, hopefully, hopefully, six launches is a complete surprise to everyone. I think that would be ideal.

• 3:12:42John Coogan?

  Like a Beyonce album.

• 3:12:43Jordi Hays?

  Oh, yeah. Hopefully, five just makes it and says, hey, it's ready now.

• 3:12:46John Coogan?

  It's ready now.

• 3:12:46Jordi Hays?

  If you want to hit.

• 3:12:47Speaker 13

  Yeah, I think that would, that would be a great thing for six, actually. I would love for six to do all of the launch comms and to, and to do the live stream. That would be really great.

• 3:12:55Jordi Hays?

  Live streaming is, that's the real A G I test.

• 3:12:59John Coogan?

  For sure, for sure.

• 3:13:00Speaker 13

  I feel like we're not that far off, actually. I don't know.

• 3:13:02John Coogan?

  We're getting there.

• 3:13:03Speaker 13

  I mean, video synthesis, maybe, but, you know.

• 3:13:06Scott Wu?

  Talking through a script for 30 minutes? Come on, a model's gotta be able to do that.

• 3:13:10John Coogan?

  For sure. Well, yeah, that'll be the next Sora launch or something. We'd love to have you back on. But thank you so much for taking the time today. We'll talk to you soon.

• 3:13:17Scott Wu?

  Good to talk to you guys.

• 3:13:18John Coogan?

  Congratulations. Cheers. Bye. Congrats on the launch. Let me tell you about adquick.com. Out-of-home advertising made easy and measurable. Say goodbye to the headaches of out-of-home advertising. Only AdQuick combines technology, out-of-home expertise, and data to enable efficient, seamless ad-buying across the globe. And we have Scott Wu from Cognition coming in the studio for the fourth, fifth time. I can't keep track anymore. Thank you for taking the time. We missed you, Scott. Thank you for coming back.

• 3:13:42Scott Wu?

  Great to see you. Guys, how's it going?

• 3:13:43John Coogan?

  It is fantastic.

• 3:13:45Scott Wu?

  Gotta be honest, great week to be in application layer company, I gotta tell you guys.

• 3:13:49John Coogan?

  I was about to say. You get a model. This is the best thing for you ever. Open source. Another win for Scott Wu. Wow, wow, wow. 4.1. Yes. So, yeah, how big is this? Are we in the Uber-Lift territory where, you know, you're gonna be, you know, in price competition between Anthropic and OpenAI going back and forth? Like, what is the real benefit to your business right now from today?

• 3:14:12Scott Wu?

  Yeah, yeah, for sure. So, first of all, obviously, massive capability gains across the board. I think really, really impressive work that OpenAI has put together. You know, people have talked about what's going on in the AI coding model race. And I think by a lot of accounts, you know, Anthropic has generally been ahead for a lot of the last year, honestly. And I think at this point, OpenAI is very clearly, you know, has very clearly caught up. And it's pretty neck and neck, I'd say, between the two right now. And so, very exciting to see all of this unfold and to see what's next. But I think from our perspective, yeah, I mean, code is just such a core capabilities pill to use case, I'll call it. And so, you know, being able to work with smarter and smarter models and do a lot of the work that we do

• 3:14:56Scott Wu?

  just means that both Devin and Winserve can be a lot more capable, a lot more intelligent, can predict what you want to write or what you want to do with a lot higher accuracy.

• 3:15:05John Coogan?

  Yeah, it's almost like surprising that, given the, like, cultural rigor at Cognition, that you're not doing fundamental frontier research. So, can you walk me through, like, what is the...

• 3:15:23John Coogan?

  What is the focus of being an application layer company? Is it UI, go to market? I'm sure it's all of these, but in terms of the hardcore software engineering, like what is important to get right? At some point, there's fine tuning and post-training, but is that moving back into the purview of the foundation labs? Or is there still work that you want to do on top of the models or on top of the APIs?

• 3:15:49Scott Wu?

  Yeah, yeah, it's a great question. I mean, I think the core of being, you know, in an applied lab is really just focusing on a very particular use case, on delivering real, just very direct results. And I think, you know, like I think the foundation labs are obviously, you know, incredible at training base models and all this pre-training and all of the work that they do there. I think from our perspective, we want to work on a lot of very particular capabilities that apply to software engineering in particular. And then obviously, you know, run the whole stack from there to building a product, figuring out the interface and the UX, and then obviously bringing that to market and selling that. On the capability side, there's a lot of particular stuff where, you know,

• 3:16:31Scott Wu?

  one way to put it is I think the base IQ is very much already there in the models. And you can see the raw problem solving ability. And I mean, we've gotten some pretty insane results, you know, getting a gold medal at the IMO or all these other things, right?

• 3:16:44Jordi Hays?

  You called that, by the way.

• 3:16:45John Coogan?

  Yeah, you called that.

• 3:16:46Jordi Hays?

  I think the first.

• 3:16:47Scott Wu?

  I mean, we were one point away, to be fair, a year ago, right? So it was on the way, I'd say. But yeah, so you can really see the general intelligence improving and with every single model generation. On the other hand, for Devin, obviously, you know, it's a very clear like step up in the general intelligence, but also you want to be able to have, you know, if you ask Devin to go debug your Kubernetes or to go and, you know, look into your error logs and figure out what went wrong or things like that, there's often a lot of very specific capabilities. And that's where we find that, you know, the post-training of the RL is most effective. There and a lot of the kind of various work around the models that turns out to be useful.

• 3:17:29Jordi Hays?

  What about speed? A lot of people that have gotten access to GPT-5, or at least in our chat, are reporting that it just feels really, really quick. How is that, over time, going to impact the... I think a lot of people, if they're using Devon today, tasked Devon with something and then maybe they go work on something else for a little bit, or they're running multiple agents concurrently. But at some point, the agent could get so fast that you're just sort of like watching it and work in real time, and you actually want to be engaged. But are we there yet? Is it still a ways out? What do you think?

• 3:18:06Scott Wu?

  Yeah, it's a great question. I think in general, I think async will continue on as a paradigm, even as the models get faster and faster. One of the reasons that it should, by the way, is because there are a lot of real-world thresholds that start to matter. Like, at some point, you're actually spending less time on token generation in the Devon lifecycle, and you're spending more time on every time Devon runs the command to go install packages, or Devon running the unit tests, or Devon pulling up the front end by itself, or things like that, that obviously take real-world time, right? I think we are honestly getting closer and closer to that threshold. But yeah, so long story short, I think in the asynchronous mode, yeah, these things will get faster. You know, we'll see those gains, or we'll be able to spend a lot more time,

• 3:18:49Scott Wu?

  for example, thinking about a single problem relative to the amount of real-world clock time that gets spent. I think for the synchronous use cases is where we'll see things really, really, you know, explode with speed, which is, you know, Windsurf and Cascade, for example, where we see the speed gains really, really matter.

• 3:19:08John Coogan?

  Speaking of Windsurf, give us the update on the... The chat wants to know about the Windsurf tea and the 80-hour demand. How have the buyout offers gone? What's the internal response been? Where'd that idea even come from?

• 3:19:23Scott Wu?

  Yeah, yeah. Look, people are stoked, honestly. And I think from our perspective, it's obviously really important to kind of just like unite and get to the point where we can just be one culture and one kind of shared set of values. And this is how things are at Cognition. It's a pretty busy time. Like, we are at the inflection point of code, and we work like that, too. And so I think a lot of it for folks is just kind of like, you know, we want to make sure folks who really want to do this with us, you know, make that conscious decision to opt in. And for anyone who doesn't, obviously, we totally understand that there are a lot of talented folks that maybe that's just not the right thing for them right now, or, you know, not at this time.

• 3:20:08Scott Wu?

  And so wanted to make sure that they were well taken care of.

• 3:20:11John Coogan?

  And to be clear with the buyout offer, that's on top of the actual acquisition deal that already went through. They already got their vesting.

• 3:20:18Jordi Hays?

  They already got fully vested. So yeah, I was thinking of the roller coaster. It's like you have the OpenAI deal, then the Google deal, then the Cognition deal. And then they're like, wait, these guys work really, really hard. I don't know if I'm cut out for this. And they come back up again, or they're like, wait, I can just go, you know, take a sabbatical and figure out my next thing. So it's a great outcome.

• 3:20:36Scott Wu?

  Yeah, no, and it's obviously, you know, overall is a killer team that's been through a lot. And so I wanted to make sure that they're well taken care of.

• 3:20:45John Coogan?

  That's fantastic.

• 3:20:48John Coogan?

  Anything else you can tell us about the integration of Devon and Winserve? How are the teams getting along? How do you see the products playing together in the long term? Obviously, cross-sell seems really obvious. They have the go-to-market team as well. But how else are you thinking about the interaction, maybe over the longer term there?

• 3:21:05Scott Wu?

  Yeah, yeah, yeah, for sure. Yeah, a lot of obvious integration on the team, as you mentioned, with cross-sell and so on. I think the thing that's really exciting on products, which I think actually comes along with these capabilities increases, is, you know, as the capabilities keep getting better, you start to take on harder and harder tasks with AI and with full agentic workflows, right? And I think there's an interesting thing that happens where, for a lot of the harder tasks, you really actually do want to go back and forth between asynchronous and asynchronous mode, you know? And that's for a few reasons. You know, one of the reasons, obviously, is because there's a lot of review and a lot of like looking at the pieces and thinking about the... You know, all the minutia and the details of what you're implementing. I think another big reason for it is, you know, when you get started on a larger project,

• 3:21:50Scott Wu?

  you know, let's say you're sitting down as an engineer and you're saying, all right, I'm gonna go build this whole project today. You yourself don't actually know all the trade offs you want to make, all the decisions that you want to make and so on, right? And so having a format where, you know, for the decisions that need you to be there and you're involved, setting the kind of the strategy or figuring out high level what should happen, you're able to do that in a nice synchronous environment, which is naturally the the Windserve IDE, right? And then for the parts of the task that you can actually hand off and have an agent work on, you're giving that to Devin and figuring out how you go back and forth between those is is super interesting. So Wave 12 on the way soon. Hopefully we'll have a lot more more to share.

• 3:22:25John Coogan?

  Last question, yeah, hit the soundboard Jordy for that.

• 3:22:28Scott Wu?

  For Wave 12.

• 3:22:29John Coogan?

  Wave 12, fantastic. Last question, we'll let you go. What is your probability that AI will get a perfect score on the IMO next year?

• 3:22:40Scott Wu?

  Oh, interesting. So we, by the way, we just had the IOI, which is the the programming version, like the programming Olympiad, when I think there's a good chance that we'll have a golden medal at the IOI for this year announced as well. I think perfect score for next year.

• 3:22:55John Coogan?

  We as in humanity or we as in cognition?

• 3:22:57Scott Wu?

  As in humanity, yes, yes, yes. An AI perfect score, yeah. Or sorry, an AI golden medal. Right? Perfect score on the IMO next year? I think it's gotta be north of 50, honestly. I would put it around like 75% or so. We'll see.

• 3:23:12John Coogan?

  Well, thank you so much. I will be following it closely and good luck to you and congrats on all the progress. Very fantastic. We'll talk to you soon.

• 3:23:20Scott Wu?

  Awesome, guys. Thanks for having me.

• 3:23:22John Coogan?

  Bye. Let me tell you about Bezel. Getbezel.com. Your Bezel Concierge is available now to source you any watch on the planet, seriously, any watch. And we are joined by our next guest, Claire Voe from ChatPRD. Welcome to the stream, Claire. How are you doing?

• 3:23:37Jordi Hays?

  What's going on?

• 3:23:38Speaker 15

  It's a fun day today, isn't it?

• 3:23:40John Coogan?

  It is a fun day. What was your reaction to the stream? What was your reaction to GPT-5?

• 3:23:45Speaker 15

  You know, GPT-5, the first thing I said, and I got a little early access, is I said, "It's a developer for developers by developer. This thing is built to be a software engineer." You've seen a long string of your guests come on and really speak about the coding abilities of it. And what I think is interesting about this particular model, especially because we're seeing them deprecate the old models in the ChatGPT experience, and we're seeing a lot of positive feedback. But I do think there are drawbacks to a model that's so clearly tuned to a developer use case. And as somebody who's building an application that isn't focused on agentic coding, I have noticed some personality quirks that are going to be really interesting to see how they shake out as we roll out this model to our users.

• 3:24:30John Coogan?

  Walk me through those. What are the...

• 3:24:32Speaker 15

  Yeah.

• 3:24:32Jordi Hays?

  What's the timeline? How much, like, how much time do you have to kind of move users over to five?

• 3:24:40Speaker 15

  Yeah. Yeah. So I mean, I think we have tons of time from the API side to move, move users. And in fact, you know, our strategy at ChatPurdy is not to just upgrade to the latest model. I know Zach at Warp said, like, "Why wouldn't you want the latest intelligence?" And the reality is, because we're doing a lot of business strategy and business writing, I actually want to validate with our users that they're getting the quality of strategic thinking, output, writing that they really want. So we actually A/B test every single model rollout and really evaluate for user quality, token generation, all those things. And, you know, looking early on, it yaps. Man, this thing just wants to go through tokens. Right now, I'm seeing four to 10x the number of tokens generated between the, you know,

• 3:25:25Speaker 15

  four generation models and five. And when you're in a business context, you do not always want longer words, you know? And so it'll be really interesting there. It is certainly focused on execution. So I, you know, I've heard a lot from the OpenAI team, it's steerable. Yes, and its natural inclination is to drive you towards like how, what, very tactical, very specific. And so if you're trying to zoom back out at a strategic level or focus on a business initiative, it's actually a little harder to tune in that direction. So, you know, I think there's a lot of positive things for me as somebody who uses agentic coding platforms, who writes a lot of

• 3:26:04Speaker 15

  code. It's my daily driver now. I love it. But for other use cases, I think it's going to take some time to figure out if it really is optimal in use cases where intelligence actually isn't the differentiating capability.

• 3:26:20Jordi Hays?

  Yeah, it's very interesting to think the best, the best product manager is not the one that writes the most, the longest doc.

• 3:26:28Speaker 15

  No, and you don't send your engineer into your executive meeting. Like, and I really am looking forward to the time where we're not getting these number based models where actually I can get like GPT developer or GPT strategist where they're pre tuned and trained for the role they're going to play as opposed to general purpose, but clearly oriented towards a set of, set of tasks. And I just think if you look at this model, it was oriented towards engineer, software engineering, at least in my experience.

• 3:27:03Jordi Hays?

  So have you been tempted to launch any type of agentic coding products? You are, you guys are obviously at ChatPRD responsible for creating documentation. And if you look at the other guests that have joined today, many of them are competing with each other in different ways and trying to own different parts of the stack. You guys have seemingly stayed really, really laser focused and no one else is doing anything like you're doing, at least on the show today. But talk about like picking your, your lane and kind of like optimizing.

• 3:27:40Speaker 15

  Yeah, we're integrated with a lot of those platforms. So a lot of the kind of like prototyping platforms, v0.dev, lovable, all those, we integrate. We just released our MCP. So I use ChatPRD pretty consistently inside Cursor through our MCP. So I think of, we think of ourselves as the product pair to the AI engineer. Now, what's really interesting about my experience with GPT-5 is the one place it actually does really well is technical specs. And that's a place where ChatPRD has sort of bridged into engineering execution. Often, our product managers are generating a PRD or some sort of business document, and they're actually going the next layer and developing a technical spec.

• 3:28:20Speaker 15

  The GPT-5 technical specs fed into these agentic coding frameworks or prototyping frameworks, output much higher quality assets on that end. So I do almost think there's going to be this kind of like right model for right use case, especially in our kind of business. And so we think of ourselves as integrating. The one thing I have thought about with GPT-5, it's the first one where it feels really simple to just go ahead and roll your own agentic coding framework or prototyping framework inside of our application. So never say never. It's something that we get asked for a lot. But we're good friends with almost all your guests on your show today. And so we like the role we play in terms of being the product manager pair to all these AI engineers.

• 3:29:04Jordi Hays?

  Yeah, that makes sense.

• 3:29:07John Coogan?

  What are you looking for next?

• 3:29:09Speaker 15

  What am I looking for next? I mean, in terms of model capabilities, what I think is really interesting about OpenAI and why I'm really committed to the OpenAI ecosystem, even though I test and use a variety of models, is I think developer support is a real differentiator. So we spend a lot of time talking about model capabilities. And for application developers, certainly ones that are doing more complex applications like agentic coding, model capabilities really matter, like core IQ of the model matters. But the other thing that matters, you know, somebody who has built developer tooling products, is developer experience matters, the primitives in these APIs matters. And so what I'm really pushing the OpenAI team to think about, which is

• 3:29:54Speaker 15

  In addition to the core intelligence of the model, what are the developer tools you need around these models to really make them a platform on which a variety of applications can build? And I do think that OpenAI has disproportionately invested in developer experience, but I'm always looking for like, give me better out of the box tooling, give me more control over these models, give me more hosted services, all those things that as an application developer are just going to make it easier to deploy these models of production beyond the core kind of intelligence of the models themselves.

• 3:30:26John Coogan?

  What was your read on 4.5? Is there a world where, you know, I'm thinking about the, the, the product manager versus the engineer? You have your oh, three, go crunch some really hard reasoning. And then you have four or five, turn it into, uh, you know, stronger prose or like more, you know, a human language.

• 3:30:45Speaker 15

  Yeah, so I did a lot of experimentation around 4.0, 4.5 and 4.1. 4.5 was my favorite prose writer by far. It was, was loved from a business writing perspective. I thought the prose was the most natural. It was really slow, like untenably slow. And so the, um, the compromise we made in our testing is we ultimately ended up with 4.1 as the fan favorite for business writing when we were balancing off both quality of prose and intelligence as well as performance, which for application developers is a real consideration. So I landed on 4.1, 4.1 is the model that's being tested right now against GPT-5 in ChatPurdy.

• 3:31:31Speaker 15

  And one of the things that I have to go do now is figure out how to get chat GPT or GPT-5 to stop writing. It, it writes a lot and it only wants to write in bullet points. So I've got to go back into our prompts and figure out how to direct it to be a little bit more business oriented.

• 3:31:47Jordi Hays?

  Bullet point maximalist.

• 3:31:49Speaker 15

  It's the new M dash. I'm telling you, you will not be able to stop seeing it. It just, all it wants to do is write a bullet point. And call a tool like it. I was using an incursor and it just kept maxing out my tool calls. I'm like, you do not need to read 50 files to do this. So I do think, you know, application developers are really going to have to think about how they slot this into their current workflows. There's definitely tuning that needs to happen, but I'm telling you, you're going to see a lot of bullet points when this thing rolls out.

• 3:32:18Jordi Hays?

  In 60 seconds, where is product management going? A lot of people talk about the, you know, examples of product managers that are starting to ship code themselves, ship whole features, products. But I'm sure those are edge cases to date, but, but where do you feel like it's going based on, on your user base?

• 3:32:39Speaker 15

  Yeah. I mean, it's going to go one direction or the other. Product managers are either going to develop the hard skills to do the design, the go to market and the engineering job to some extent, because some of these other jobs are definitely going away for product managers. Or my favorite use case, engineers and designers are going to get tools like chat P R D or these prototyping tools or cursor. And they're going to be able to actually do the product management job. And so what I think is we're going to see a new type of role emerge, which is a much more generalist role, where people maybe have a specialist capability and they're augmenting that product thinking or they're augmenting that technical thinking with, with AI. But I don't think there's going to be product managers as they were, you know, five or 10 years ago for much longer.

• 3:33:23John Coogan?

  Makes sense. Well, thank you so much for stopping by.

• 3:33:25Jordi Hays?

  Yeah, great to have you on.

• 3:33:26Speaker 15

  Thanks for having me.

• 3:33:27John Coogan?

  Bye.

• 3:33:27Jordi Hays?

  Cheers.

• 3:33:28John Coogan?

  Up next, we have Brad Lightcap, the chief operating officer of OpenAI. Welcome to stream, Brad. Also, Jordy, your post saying, I'm updating my timelines. You now have four years to escape the permanent underclass has over 4,000 likes.

• 3:33:42Jordi Hays?

  There we go. Thousand likes for every year.

• 3:33:45John Coogan?

  Love it. Anyway, Brad, how are you doing?

• 3:33:47Jordi Hays?

  Brad, what's going on? Guys, how are you? Good.

• 3:33:49John Coogan?

  Congratulations on the launch. What, what are the biggest takeaways for today from your side? I'd love to know about what it actually means to be the COO of OpenAI. OpenAI does so many different things, consumer internet company, API business, enterprise, there's all sorts of stuff, building data centers. What, what is your actual role?

• 3:34:11Speaker 16

  My role is kind of whatever the company needs me to do.

• 3:34:13John Coogan?

  Okay.

• 3:34:14Speaker 16

  I play everything from like, you know, PM when I need to to like, you know, salesperson when I need to. That's kind of the fun part of the job for me. On this launch in particular, it was really fun. I spent a lot of time last few weeks with customers, with partners, getting a feel for GPT-5 relative to what they were previously using. In some cases, those are OpenAI models. In some cases, they were other models. But, you know, I've been OpenAI a long time, been at OpenAI seven years. So I've seen GPT-3, I've seen GPT-4. And then to be able to see GPT-5 and, you know, just I think the joy of people being able to use it in production and seeing how much better it is, that's the best part.

• 3:34:53Jordi Hays?

  Greg told us earlier about the era of having to pay people to use the early versions of the product. You guys have come a long way since then.

• 3:35:02Speaker 16

  Yeah, we had like three customers with GPT-3 or something like that. And so it was easy to manage, easy to talk to all of them. They actually were like tired of us calling them, being like, is it, is it good? Is it getting better? And so now it's, you know, we're fortunate that we've got more than that. But it's cool. I mean, the diversity of use cases, I think the number of things that people are able to use it for. We've got everything from the team at Amgen, you know, Big Pharma, Life Sciences, using it for clinical workflows there. We've got teams at Uber's, you know, building it for customer support. Teams at Notion and Cursor building it into products that people use every day. So I think that's the power of it is, is it just more and more coverage of service area of things people do, you know, with these tools.

• 3:35:44John Coogan?

  I'm not sure how much you touch organizational design at OpenAI, but I'd be interested to hear your thoughts on how those companies that you mentioned should be thinking about AI changing their org structure. Is it sort of like a horizontal, cross-functional service layer like,

• 3:36:03John Coogan?

  you know, a finance team that touches a lot of different elements of the business? Or should most companies be thinking about standing up a dedicated like AI implementation team? How do we get a chat box on every product that we already ship? Like, how do you think about those trade-offs if you were talking to, you know, a friend at a Fortune 500 company that was thinking about their AI strategy?

• 3:36:24Speaker 16

  Yeah, you know, it's an interesting question. I think it was maybe said earlier on the show. The thing we see is just people can do more. And so there's like this much wider latitude that you get if you're an individual person at an individual company, where especially as you get bigger, you know, maybe more bureaucratic organizations that have a lot of different functions, a lot of different levels, you have to rely on a lot of other people in the org to get stuff done. You've got to rely on your data science team to do data analysis. You've got to rely on your design team to do mockups. You've got to rely on your marketing team to do copy. And I think what we see with AI is it just accelerates people to get to a great V1 of everything. So if you're a high agency individual and you want to get stuff done, you're no longer gated on people that, you know, you otherwise would be.

• 3:37:08Speaker 16

  And I think that should enable organizations to move a lot faster. And I think it should enable the people at organizations that really drive them to do a lot more. And we see that consistently. ChatGPT Enterprise, I think that is consistently what we hear. And we seek those people out when we deploy ChatGPT Enterprise. We find those like, you know, two or three people at the organizations who are just the like AI superstars and champions, and then try and actually use them as these kind of touchpoints for the rest of the org to learn from.

• 3:37:34John Coogan?

  How are you, how are you personally using AI these days?

• 3:37:38Speaker 16

  Uh, you know, my biggest challenge, I think, day to day is context switching. If you look at my calendar from like top to bottom, it's like, I joke like I, you know, with my wife, I like have to like show up to work like wearing like a lab coat. And then I like take the lab coat off and like put some like sunglasses on and a film school jacket. And, you know, then I'm talking to like a media company and then I like take that off. And so, so I go through the costume changes. And I think what, what I actually mostly use it for is just to help with bridging me from kind of thing to thing, to kind of put me in the mindset of being able to work with customers, help customers. GPT-5 is incredibly good at this kind of structured reasoning of how do we actually take what is this very diverse set of things that models like GPT-5 can do, and then apply them in domains that I don't think about every day.

• 3:38:20Speaker 16

  And so it gives me this launching off point to be able to talk with, with, with leaders and with customers much more fluently about how we can help their organizations.

• 3:38:28Jordi Hays?

  Within, let's say, a set of companies like the Fortune 500, what does AI adoption look like across the spectrum? Because I'm sure that there's companies that you talk to that are truly adopting AI in the way that John was mentioning, like trying to become AI native, changing their entire organizational approach. And then there's companies that just want to buy software to say that they're becoming AI native. So what does that spectrum look like in practice?

• 3:39:01Speaker 16

  Yeah, it is a wide spectrum. So at the top level, we're seeing just like amazing appetite for wanting to adopt tools for people. And I think that's like the easiest place to start. Typically, that's where we steer organizations. If they're starting at zero, is just give your people the best tools. You may have seen, we've grown ChatGPT Work, which is our enterprise and team product, from 3 million seats to 5 million seats now from June till now. So torrid growth there, and we don't see any abatement in demand there. If anything, it's accelerated from last year. And so I think people and organizations are starting to realize that like, at a minimum, you need to make sure people have the best tools. What's cool about GPT-5 now is it also enables people

• 3:39:45Speaker 16

  to use the best tools at every point. And so if you're in an organization, you're not fumbling with the model picker, you're not trying to figure out when to use a reasoning model, you're not trying to figure out kind of the art of prompting to get the perfect thing. All of that stuff is abstracted, and it's kind of taken care of for you. And you can have confidence that your people are actually using the best models at any given point. Beneath that, it gets a little more complicated. So more and more organizations, I think, are starting to grasp how the tools can actually help in the business process. So whether that's in customer support, whether it's in research, whether it's in software engineering and data science. You're seeing these tools more and more adopted in the enterprise. I think there's still a quality gap, though. I think we now are just breaking into what I would call

• 3:40:28Speaker 16

  the kind of era of models that have capabilities that are good enough to make a dent in the types of problems businesses care about. Businesses care a lot about things like reliability, right? They think they care about accuracy. They care about the resilience, the resiliency of the model to recover from tool use errors and to be able to string together these very long, kind of multi-tool, multi-step workflows. So GPT-5 is a step on all those things. And I expect that that will enable us to be able to do more and more things in the business process.

• 3:40:56John Coogan?

  Do you think those customers that you just mentioned will stick with this idea of like GPT-4 level workloads will stay on GPT-4? And maybe there'll be cost savings, but those workloads will stick around for a very long time? And then you'll develop almost new capabilities, new workflows, new, new workloads that will be additive, but the enterprises will stick? Or will they want to, is everything so fresh that they'll want to just rewrite everything with the latest and greatest?

• 3:41:28Speaker 16

  More often than not, I think it's the latter. You want to rewrite everything. One of the cool things we did here was we were able to keep the pricing on GPT-5 at the level of O3 pricing. And so, you know, if you're cost sensitive, you don't really have an excuse to not upgrade. GPT-5 is faster than O3 and 4.1, so we've improved on latency for sensitive use cases that are speed sensitive, latency sensitive. And obviously, the intelligence bar has gone up. And so, you know, unless you've got really a very kind of narrow and specific workflow where you've got a model like 4.1 that kind of is okay, there's really not a reason, I think, that people wouldn't upgrade.

• 3:42:04John Coogan?

  Yeah, do we need like a three-dimensional Pareto frontier right now that matches not just cost and capability, but also cost, capability, and latency or something? Is that, is that something that you're seeing a lot of demand from in the enterprise?

• 3:42:16Speaker 16

  Yeah, 100%. We actually measure it that way. So we look at those three vectors and it's always kind of an optimization function along those three, those three, those three axes. We think we found that here. It was actually, in terms of where my work was over the last few weeks, it was a lot of, I mean, it's a qualitative, you know, kind of, you know, really like manual process of collecting feedback because everyone's got a little bit of a different preference and we can only pick kind of one or two points on that curve. And so, just trying to kind of dial customer feedback, namely developer feedback in for us on where that, that balance of things are, is a big part of our process for picking, picking all those, all those points. And so, we hope that, we hope that people like it and it unlocks, you know, the kind of maximum use.

• 3:42:58John Coogan?

  It's great. Jordi, anything else?

• 3:42:59Jordi Hays?

  How are you thinking about open source? Who, you know, who's been most excited to get access to it? And yeah, where do you see it going?

• 3:43:08Speaker 16

  Yeah, I mean, it's important to us. You know, I'm glad we've, we've gotten this out. It's been a huge team effort. I think there was a kind of a thing that like, you know, OpenAI doesn't like open source anymore. It's like, no, we're just like really busy with a gazillion other things. So I think hopefully going forward, we've got more of a leaned in vantage point on, on, on open source. But it unlocks a huge number of use cases. I mean, if you think about kind of like, you know, government use cases, you think about on prem, you know, use cases where you're handling sensitive data and very sensitive environments. You think about where you want to run models on the edge. All these things right now are kind of inaccessible to us as a service provider to customers, because we just don't quite have models that kind of fit at those points. So this for us, we think is huge TAM expansion.

• 3:43:53Speaker 16

  And we're excited to be able to work with enterprises on, on implementing that model, which is, is I think, you know, competitive, hopefully with, with R O three class of models. So.

• 3:44:01Jordi Hays?

  What is the landscape like for companies that are helping to implement OpenAI products at various enterprises? You have the, you know, big consulting groups that will give you an AI strategy. Maybe they'll try to take it a step further, but I imagine there's a cottage industry of, you know, firms that have sprung up to try to help organizations unlock the value beyond, hey, let's just get everybody a seat with ChatGPT work.

• 3:44:29Speaker 16

  Yeah, I think there will be this new industry that emerges that is kind of separate and apart from kind of the legacy set of SIs and, you know, consultants that is really AI fluent. They're very AI native. I think it's very hard to borrow, I think paradigms from the last 20 years of software building and, you know, implementation that are going to kind of map to what we're dealing with here. You're dealing with fundamentally probabilistic systems that are moving and increasing and improving at a rate of now collapsing to every few months. And I think the nature of use cases changes quickly. Where enterprises are focused and deploying them changes quickly.

• 3:45:12Speaker 16

  And so I think it's just hard for the legacy industries to keep up, frankly. We've had a lot of success working with some of this new breed of SI. So the distills of the world and others that really have been born, I think, forged in the fire, so to speak, of this new platform. And so we hope there's more of them. We'd be excited to work with anyone that wants to work with us on it. There's more business than we can handle, and so we're always happy to spread the love.

• 3:45:40Jordi Hays?

  Talk about the $1 ChatGPT product for the government. Were you involved in that at all?

• 3:45:49Speaker 16

  I was involved in that. We wanted to do something that was meaningful for US government. It's been a real big focus of ours lately.

• 3:45:59Speaker 16

  I think our view is the government has got to start to modernize. We've got to make sure that the tools that we use in the private sector are also in the hands of folks serving us in the public sector. And we wanted to make that really simple. So we made ChatGPT, basically equivalent to ChatGPT Enterprise Free. It's a dollar per year per agency. Hopefully, we can afford that. And we wanted to make that available to anyone that wanted to use it and standardize through GSA. So we're super appreciative of the partnership with them and more, I think, that we can do on that front.

• 3:46:29John Coogan?

  How is that different than just like, if I'm a government employee, I can just go to google.com and I have access to that and Google provides benefits?

• 3:46:36Jordi Hays?

  Scott Kapoor was saying that he can't use ChatGPT on a government.

• 3:46:41John Coogan?

  Yeah, why? Yeah, just talk to me about how it's different to offer ChatGPT as an actual service with a contract that you're vending in, you're actually, they are a client versus just if you put up a website, every government employee can access the web to some degree? Or would it be blocked? Why does it need to be a deal at all as opposed to just like? Everyone just uses it.

• 3:47:07Speaker 16

  Yeah, so part of it is just making sure that government employees can access it. So in some places, obviously, you know, you can put blockers in place that would prevent access. We hear a lot of stories, by the way, of people like going out on their lunch break to their car in the parking lot and like, you know, pulling up chat GPT on their phone and like throwing a bunch of stuff in there just to like, because they know it'll get them through the day faster. And we've done work, by the way, with governments, with the state of Pennsylvania and other places where we've seen dramatic increases. You know, things like two to three hours a day saved per employee, given the nature of the work that they do and how helpful chat GPT can be. And so this lets us have an interface into them as a customer. It lets our team engage with them in a direct way. We can see how they're using the product and can help them use it better. And so that's, that's important for us is like, we got to build on that foundation with them.

• 3:47:52John Coogan?

  And then presumably it also allows the, the government to define like security and privacy in their world as opposed to if you're just like some website out there, they, their choice is only block or don't block as opposed to actually, you know, communicate with you. This is okay to train on, this is not, et cetera, like keep everything private, et cetera, et cetera.

• 3:48:10Speaker 16

  Yeah, I mean, we don't, we don't train on, on enterprise data at all.

• 3:48:13John Coogan?

  Yeah.

• 3:48:14Speaker 16

  You're safe there, but the, yeah, I mean, for us, like just being able to, to treat them as a customer, right? To treat them as a user and you go, you know, you mentioned earlier, like we were talking about kind of like there being these points of success at every organization that, you know, you've got people who are like way more sophisticated in using these tools than others. We want to be able to see those people and amplify them, and the government's no different. There are people that we've worked with in government who are incredibly sophisticated in how they use AI tools. And our goal is to get everyone there.

• 3:48:43Jordi Hays?

  How do you think about the group of users that are active students? They've been on summer break, you guys have been busy over summer. Are you thinking about, and you recently launched, I forget the exact name for the product. I think it was like chat GPT learning. How are you thinking about that cohort and unlocking new capabilities for them this coming year?

• 3:49:05Speaker 16

  Yeah, so we launched something called Study Mode, which was in our core ChatGPT product. And it was a little bit of an experiment. We wanted to see if you change the way the model behaves, when it can kind of, when it knows you want to be in a learning mode, if that can actually enhance outcomes for students. Where we have all these kind of studies that have been done very like anecdotally about ChatGPT's ability to, to drive student outcomes and learning outcomes. And so here we kind of took a little bit more of an intentional approach of, if you actually model the, take the model and actually use it in a more Socratic style, where it can actually kind of quiz you, it can withhold certain information that it wants you to be able to empirically deduce. It wants you to reason about problems and it kind of reasons with you as a partner. So far, so good.

• 3:49:50Speaker 16

  It's, it's really cool. And learning is kind of the killer use case of ChatGPT. And so I think, you know, to be able to actually launch something that is, in some sense, extends that kind of killer use case, has been really cool. And the student feedback so far, even on summer breaks, been positive.

• 3:50:05John Coogan?

  Well, we'll let you get back to your day. What's, what's next on your agenda? Are you putting on the lab coat or the suit and tie and going to Washington?

• 3:50:14Speaker 16

  Good question. You know, today I'm mostly with the team and talking to customers. And maybe tomorrow I'll get back to the lab coat. But in the meantime,

• 3:50:24John Coogan?

  We appreciate you taking the time to talk to us.

• 3:50:26Speaker 16

  So.

• 3:50:26John Coogan?

  Yeah, well, thank you so much for taking the time to talk to us.

• 3:50:28Jordi Hays?

  Great to see you.

• 3:50:28John Coogan?

  We will talk to you soon. Have a great rest of your day. And the timeline has been in turmoil because President Trump says he will be imposing a 100% tariff on all semiconductors coming into the United States. It started with widespread tariffs on chips and then turned into export controls. This is from the Kobe S. E. Letter. Is this a red flag moment? I don't know why we have the red flag.

• 3:50:51Jordi Hays?

  It felt like it. Ben was getting the flag ready.

• 3:50:54John Coogan?

  And video potentially affected. But Taiwan says TSMC exempt from Trump's 100% chip tariff. Very unclear. The story is obviously still developing.

• 3:51:09John Coogan?

  And Dylan Middick says, you're telling me that this level of monitoring the situation is free? And it's a picture of you in front of the whiteboard, monitoring the ChatGPT versus the timeline today.

• 3:51:20Jordi Hays?

  We're monitoring.

• 3:51:22John Coogan?

  Illinois has banned AI therapy, making it the first state to regulate the use of AI in mental health services.

• 3:51:28Jordi Hays?

  Interesting.

• 3:51:29John Coogan?

  Interesting headline that's coming out.

• 3:51:31Jordi Hays?

  It's interesting because the product can just be used, the therapy, like the user can choose to do that. It's not necessarily, it's kind of hard to ban outright. Like, maybe you can ban it in a clinical setting.

• 3:51:45John Coogan?

  Yep, I wonder how they define this. There's probably a loophole, if I know anything about how these bans are implemented. But yeah, maybe it's like if you're in the clinical setting, you can't be, you can't use it, but then people will just use it independently.

• 3:51:57Jordi Hays?

  Therapist just on their phone.

• 3:51:59John Coogan?

  They're gonna be going to the car.

• 3:52:00Jordi Hays?

  They're gonna be having, they're gonna, no, they're just gonna have it listening to the conversation.

• 3:52:04John Coogan?

  Yeah.

• 3:52:04Jordi Hays?

  They're gonna be like, what should I do right now? What should I say?

• 3:52:07John Coogan?

  What should I say? How does that make you feel?

• 3:52:11John Coogan?

  That's what it's gonna tell you. Celsius nearly doubles revenue year over year. This is the energy drink. Revenue of $739 million versus $632 million consensus. North America grew 87%, international grew 27%. But here's the real kicker. Alani New acquisition is the primary driver of growth. Alani New added $300 million in revenue and retail sales are up. So, wow, what performance. But yeah, I mean, that was the expectation when they bought Alani New is that they would, I guess this is like the first moment they, they rolled them in, probably. But huge growth for Celsius as they become multi-product, multi-consumer company.

• 3:52:55John Coogan?

  What else is going on in the timeline? We have one last guest. I think you might have to hop on with Taipei.

• 3:53:01Jordi Hays?

  Yeah.

• 3:53:01John Coogan?

  So feel free to jump when you need to.

• 3:53:05John Coogan?

  Tyler, anything going on on the timeline we should be monitoring? We are, of course, monitoring the situation.

• 3:53:11Jordi Hays?

  Um, I've been, so, so when Max was on, he was talking about like how you can like make a little game, right? So I've been working on like a balloons tower defense game.

• 3:53:18John Coogan?

  Okay, how's it going?

• 3:53:19Jordi Hays?

  So it's going pretty well. Um, I'm, I'm making another change, but then maybe I can screen record and do a little share.

• 3:53:26John Coogan?

  Yeah, that'd be great. You could share with the, with the folks too.

• 3:53:28Jordi Hays?

  Yeah.

• 3:53:28John Coogan?

  Um, I like this post from Ray Sullivan. These GPT five numbers are insane and it's a chart of GPT version versus number. And then, uh, once it gets to four, it goes 4.1, 4.2, 4.3, 4.4, 4.5. So the, the fifth one is a massive, massive bar.

• 3:53:45Jordi Hays?

  We needed analysis of the charts from today. It seems like there was multiple that were.

• 3:53:53John Coogan?

  Kind of odd or hallucinated or, or off.

• 3:53:56Jordi Hays?

  But it's interesting that multiple of them snuck out.

• 3:53:59John Coogan?

  Just in Sheets, the popular convenience store chain with 750 locations is now offering 50% off purchases paid with Bitcoin and crypto daily from 3 to 7 PM. What a wild move by Sheets. Well.

• 3:54:13Jordi Hays?

  Well, Ben Hylak is in the waiting, the re-stream waiting room. Let's bring him in.

• 3:54:19John Coogan?

  Bring in Ben Hylak. How you doing Ben? Good to see you.

• 3:54:21Speaker 17

  Doing well. How are you guys doing?

• 3:54:23Jordi Hays?

  We're doing well. I'm just gonna say hello. I gotta take off and talk with Taipei. I'm gonna let John take it from here.

• 3:54:28John Coogan?

  Absolutely.

• 3:54:28Jordi Hays?

  And you guys have a fantastic conversation.

• 3:54:30John Coogan?

  Give me the update. Uh, how's the day been for you? What were your expectations? Did this meet exceed? Uh, did it underwhelm you? How you doing?

• 3:54:38Speaker 17

  Well, um, so I've actually had access for a couple of weeks. So we actually did a video. I'm not sure if you've seen it, but, uh, OpenAI brought a couple of, uh, folks from the Twitter sphere to their office a couple of weeks ago to try it.

• 3:54:51John Coogan?

  And some other folks.

• 3:54:52Speaker 17

  Yeah, yeah. Yep, yep, yep. Um, I think that, uh,

• 3:54:58Speaker 17

  It pretty much exactly meets my expectation as far as like how uh how it's been received. Um, and I've tweeted about this as well, but I think that it's really, really good at like one-shotting things. Um, you know, I think like it's better than I think other models we've seen, but I think it's actually sort of a distraction in a lot of ways. Um, I think that the things it's a lot better at are A, a lot harder to describe. And B, I don't think the the harnesses for it really exist yet. Um, I think a lot harnesses, so. Uh, what the way I've been describing it is that I think I've seen, you know. Uh, web search existed in chat GPT for a really long time, right? Like it was able to like call a tool, search the web.

• 3:55:40John Coogan?

  Yep.

• 3:55:41Speaker 17

  Obviously, like deep research was very different than that, right? Like what we saw was it was like actually like calling, you know, searching the web. It was like reasoning about those results, changing its kind of course, like course correcting in the middle. So like intermediate reasoning is like what is the is the term for it. Um, and they really trained it how to search the web well. Um, I think GPT five does that for like a whole plethora of tools. The interesting thing is that a lot of products like, I think a lot of the agentic products that exist today were kind of built wrong. Uh, like they weren't built that they didn't build the tools the right way. Um, and we've seen this before, like if you look at like, you know, the first. Um, you know, kind of infrastructure for agents was Langchain like way back when.

• 3:56:25John Coogan?

  Yeah, I remember.

• 3:56:26Speaker 17

  Two or three, yeah, it was, it was, you know, it was, uh, it was early, but it was wrong, right? And so like anybody that, you know, they've they've iterated since, right? They have like Langraph, it was a better implementation, but the first implementation of Langchain was like, again, early but wrong. And so if you built your product on Langchain, like you had to, you know, significantly change it. I think we'll see a similar thing happen for GPT five. You know, it's not just like, you know, uh, change the string and get, you know, from, you know, four O to five or something and push it. And now you, you know, yeah.

• 3:56:55John Coogan?

  Yeah, you know that meme about like, oh, like Sam Altman stood on stage and like, just like, you know, killed seventy five startups. Google just killed a hundred startups. Apple just killed Partifal with their new thing or whatever. Did any of that happen today? It feels like it feels like this is like the Lang chain needing to change their strategy. That happened a while ago. I haven't identified anything. It feels like, you know, Scott Wu hopped on and said, like, you know, great day to be an application layer company. The foundation models got better. It's it's more tools in my tool chest. I'm extremely happy and and I'm more confident than ever. And I believe him. I believe that he was he doesn't see today as like fundamentally needing to change his business model.

• 3:57:40Speaker 17

  I I think that's true, actually. I think that people have been, you know, there's a lot of people building agents right now. I think a lot of them have not been feasible for some of the reasons that GPT five starts to address. So I think it is. I think that what it means is that the entire architecture behind agents will get a lot simpler. Like, i- it feels like a good day for people building applications. Yeah, it's not immediate that there's like some, you know, like company or something that got killed today.

• 3:58:11John Coogan?

  Yeah, yeah, yeah. I mean, in general, it feels like, you know, Dorkesh updated his timelines. There's just been a general idea that like we've we've maxed out pre-training. We've kind of maxed out post-training. We're now in the let's reap the reward of this. And we've seen it in like the incredible financial performance, the incredible usage numbers. You know, millions and millions, hundreds of millions of people are using ChatGPT thirty thirty minutes a day. I'm I love the product. And yet it feels it feels like the what have you done for me lately meme? It's totally like, OK, yeah, we went from the iPhone four to the iPhone five today.

• 3:58:49Speaker 17

  Yes.

• 3:58:50John Coogan?

  Still really an important technology, great company, but like I want another iPhone one.

• 3:58:54Speaker 17

  Yes, yeah, yeah, yeah. I know, I totally get what you're saying. I think that, like, I I wrote a piece about this with Swix, but I the it really actually changed the way I see that path to AGI. Like, I think before using it a lot, I kind of was like, OK, we need like bigger, bigger models. They're going to like get smarter or something. I think. Like, I had this realization, so I was watching it like solve, I had this like really weird, um, like dependency conflict with yarn, like we have like a mono repo. It's like, part of the problem also with this discourse is like, um, the, the sort of problems it gets good at solving are just like not sexy things to talk about. They're not things that you can, you'll understand. I'm like,

• 3:59:36Speaker 17

  oh, we have this issue with our, like, the way we structured things and like, but like, um, a cou- a couple of weeks ago, I was watching it like, I had this problem, no other model would solve it. And, um, I watched it sort of like poke around, like, it started running this like yarn Y command in a bunch of different directories in between is like reasoning and like correctly reasoning about like what, what and why and what it was learning. And it, it, you know, taking little actions in between, seeing what happened.

• 4:00:00John Coogan?

  Yeah.

• 4:00:00Speaker 17

  Um, I think what I realized is that like, um, you know, if you imagine like humans without tools, like if we never had any tools, we're never even able to write things down, like, would you be able to tell that we're intelligent? Uh, would we have like, you know, learned to speak, et cetera? Like, I, I just like, don't,

• 4:00:17John Coogan?

  Yeah.

• 4:00:18Speaker 17

  You know, even if we could not have ever invented fire, right? It's like, it's like, where would we be right now? There's, that feels like there's a similar, like, I actually think a lot of the next year is just going to be, how do you get these models to do things better? It's like, you know, I think it's next year.

• 4:00:31John Coogan?

  Uh, in your yarn, uh, example, um, you, you said like you were, you were having it, I assume GPT five, like work on the problem. Was that wrapped in a coding tool? Did you just go to chat.com?

• 4:00:47Speaker 17

  Yes.

• 4:00:47John Coogan?

  And give it your GitHub repo? Like, like, talk to me, like, uh, what was the actual user experience from your side?

• 4:00:53Speaker 17

  Yeah. So this was in cursor.

• 4:00:54John Coogan?

  Okay.

• 4:00:55Speaker 17

  Um, I think the Codex CLI, the new version of the Codex CLI, which they just released today is also really, really, really good.

• 4:01:00John Coogan?

  Okay.

• 4:01:01Speaker 17

  Um, I think that you will really only see a significant difference. In places where it can sort of like explore its environment is the way I would put it. Like when I was watching it like go bounce around my repo and like like I felt almost like I was watching something navigate like a little like video game like Pokemon or something like that. That's kind of what it felt like like it's kind of like I'm gonna go over here. I'm gonna see this. Okay, wait a minute. That conflicts with what I just saw over here. Like where should I go next? Do you know what I mean? Like it it it felt very um novel uh is like what I would say. Yeah.

• 4:01:33John Coogan?

  Yeah, yeah, yeah. Um, what what uh so yeah, I mean, how are you using it? What what what um uh where do you see it going? Do you see it like uh? Just like a little bump of a tailwind today or or what's your read on like uh like how you'll be using GPT five going forward?

• 4:01:50Speaker 17

  I mean, yeah, there's two huge things. So like one thing that like really got missed today is that uh they also released GPT five nano. Which is like an incredibly good model, actually. Um, so like we're not talking about it, but it's half the cost for input tokens than flashlight. Or sorry, uh ha- yeah, I think it's actually half the cost of input tokens and flashlight. And it's a really good model. Like it's like four O level for a lot of like writing and stuff like that. Um, and uh so yeah, we'll we'll be using that. Uh, probably in the short term. I think it'll be interesting to see how other providers react. Like I I'm sure Google will cut their prices as a result.

• 4:02:28Speaker 17

  But it is the cheapest like hosted uh model, I think that I I don't think anyone's serving it any other model for those prices for that matter. Yeah.

• 4:02:36John Coogan?

  That makes sense. Um, what else are you looking for for the rest of the year? Uh, probably no GPT six on the horizon, but what are you looking out for? I mean, it seems like Google is expected to respond with Gemini three soon. Uh, but what else are you tracking in the in the world of AI these days?

• 4:02:52Speaker 17

  It's a great question. Um, I think that, yeah, that's gonna be wildly interesting. I think what Google does will tell us a lot. I think that they, uh, you've probably seen it, but you know that they released this like world model uh yesterday. We're kind of not talking about it anymore. I mean, like if those videos, I haven't tried it myself. If those videos are real, like that's that's one of the most mind blowing things I've seen in the last like, you know, decade or something. So like, if that's real, like that's extremely interesting. And I think has all the stuff that's going on with world models right now, has like huge implications for like, everything like from robotics, just like so many different fields. So super, super interested in that. And the other thing is I actually just think that like, again, I'm I'm actually really bullish on G P T five. I think that the way it was received today is like,

• 4:03:38Speaker 17

  just about how I expected it like. And the reason is like, when I say harness again, I'm like, I think that like, canvas and chat G P T is pretty bad is like my would be my take like, uh, you know, uh, it's a tough product to make, but like, uh, yeah, like it does really poorly with like long files crashes sometimes like that sort of like, I think that we don't have the the product layer around G P T five doesn't exist yet. So I think we're going to see some really, really interesting products um that are built around it.

• 4:04:02John Coogan?

  Yeah, it's always hard when you go from like a binary qualitative in your face improvement G P T like chat G P T was like, we passed the Turing test. And now the next test is like super intelligence and self replicates is smarter than every single person knows everything. It's like, the bar is like, we really moved the the goalposts, you know?

• 4:04:22Speaker 17

  A hundred percent. I think that there was like a lot of, you know, discourse around the model as well, like leading up to it, which I think didn't help, you know? But like, the way that I would think about it is like, I think that, you know, depending, there there's some percentage of the way through automating software engineering that we've made it like, let's say it's like seventy percent or something seventy five percent. Um, the tough part is like that last like twenty five percent is um, A, the hardest, it's like the least um, sort of decipherable to like explain to people. It's the least like um, universal, like like, if I'm just like, oh, make a, you know, one of the examples I did, I I made a personal website. It's like all Mac OS nine themed in like twenty minutes with G P T five.

• 4:05:01Speaker 17

  Um, and so it's it's really fun, right? You get it. Like my mom gets it. Like I can show it. I I can share it. You get it. You know, my mom, I can't explain any of the like, the very specific ways that G P T five like helps in our specific code base, our specific problem, whatever. Um, so I think that like it'll be less and these launches will probably get less and less uh, sort of interesting from a so like from a what it does for software engineering as that gap gets closed. Like I, you know, what's the last five percent of software engineering? Like I, you know, like I, I it's probably not going to be that interesting to me.

• 4:05:37John Coogan?

  Do you think they'll be on an annual release cadence now? Like, Apple updated all of their iOS, all their operating system nomenclature to be like, we are now on 26 because it's the year, it's like a car model, like Jaguar.

• 4:05:50Speaker 17

  You can plan it. I don't think you can plan ahead. Like, that's the interesting thing is like, I think that, you know, there's people that say that GPT-4.5 was supposed to be GPT-5.

• 4:05:59John Coogan?

  Yep, yep.

• 4:06:00Speaker 17

  And like, I think that it sort of came out and they're like, eh, like, you know, I actually love 4.5, I think it's a really fun model, but.

• 4:06:09John Coogan?

  Well, it's true that like improvements come in many places, just like with the iPhone, like the latest iPhone, you buy that because it doesn't, it's not just like the one with the new screen, it has a slightly better camera, slightly lighter, longer battery, like, it's like an ensemble of improvements that then they add up. And I think that that feels like what we're getting here today and what we will get in the future is like this little, like, we did a little extra RL over here, this tool is now sharper, it has new capabilities, we added multimodal, like, you know, the video generation got better and this feature got better, etc.

• 4:06:41Speaker 17

  And I think that like what a model is, is still going to change a lot and like how we value, like, so just give an example, like, 4.0 was sort of this big thing, you know, where they talked about it being like natively multimodal, you know, taking in even like video at some point, video in, video out, like audio in, audio out, and like, you know, you haven't heard that from GPT-5 yet, like, you can't talk to it on advanced voice mode.

• 4:07:03John Coogan?

  That's interesting, essentially rollback.

• 4:07:05Speaker 17

  It doesn't generate image, like, you know what I mean, there's no, at least yet, native image generation. We don't know much about how it works under the hood, but like, it's still calling 4.0 to generate images, right? So it's like, do you start to see an unbundling of these model capabilities, like, seems quite possible, like, the best model for writing natural language might not, or like, writing creative, you know, creatively, might not be the same model that writes, you know, really good Rust code, like, these might be different models. So I don't know, we'll see.

• 4:07:36John Coogan?

  Yeah, create image here is now tucked next to deep research agent mode, etc. But I would hope that you can call that from the actual chat interface.

• 4:07:44Speaker 17

  You can call it from the GPT-5 chat. It's just using, it's using GPT Image 1, I think is actually the name of the model. So it's a dedicated image generation model, which I think it may be 4.0. I don't totally know.

• 4:07:54John Coogan?

  Yeah, I just, I don't particularly care. I'm not looking for one model to rule them all. I'm fine if with models calling different tools, it seems fine.

• 4:08:03Speaker 17

  Yes.

• 4:08:04John Coogan?

  Anyway, fun day. Thanks for hopping on.

• 4:08:06Speaker 17

  Of course, of course, anytime.

• 4:08:08John Coogan?

  Have a good one. Bye. And that's our show today. Folks, leave us five stars on Apple Podcasts and Spotify. And thank you for tuning in to the GPT-5 GigaStream. We're on hour four and a half. We've enjoyed hanging out with you. Tyler, anything else from the timeline? Close it out for me. Timeline's still in turmoil.

• 4:08:27Speaker 01

  If we want, we can show the little game I made.

• 4:08:29John Coogan?

  Okay, yeah, let's show Tyler's game. Can we do that? Is that possible? You got it? Tyler's Tower Defense?

• 4:08:35Speaker 01

  Yeah, this was, this was one shot.

• 4:08:36John Coogan?

  Okay. Wait, what do you mean? One shot, one prompt? You said you were working on it.

• 4:08:41Speaker 01

  I was, but then it's like wasn't as good.

• 4:08:44John Coogan?

  Oh, so you went back to a single prompt?

• 4:08:46Speaker 01

  Yeah, I made a change, but then I realized like, okay, this is not as good. So I just went back to the first one.

• 4:08:51John Coogan?

  Okay. So, yeah, my, my, my question is, I mean, this, this seems, well, actually like it's, it's like the game engine. I don't know what it's using under the hood, what it's, do you know, did it write like WebGL code? Or did it write like?

• 4:09:03Speaker 01

  I think it's just, it's just like JS.

• 4:09:06John Coogan?

  Okay.

• 4:09:06Speaker 01

  And it's just using like HTML canvas.

• 4:09:08John Coogan?

  That's pretty crazy.

• 4:09:09Speaker 01

  Yeah.

• 4:09:10John Coogan?

  You'd think it would use some like 2D engine off the shelf or something. But my, my question is like, what? That won't go viral because that is less impressive than just the tower defense app that I can get in the App Store.

• 4:09:25Speaker 01

  For sure.

• 4:09:25John Coogan?

  But it's like, maybe if I take my, you know how there's like ControlNet images went viral? Where people would take their corporate logo and then they'd throw that through ControlNet? And it would be like the TVPN logo overlaid over like a forest and like the trees would look like logo.

• 4:09:37Speaker 01

  Yeah, or like the QR code.

• 4:09:39John Coogan?

  Yeah, so maybe like it's tower defense, but it's my logo or something like that. And like the, the, the enemies are like woo, like moving through something like that. I don't know. There's just gotta be a way to personalize it and make it so every single game is a unique snowflake that you want to go and experience that one. You want to look at it. You want to spend some time in it. I don't know.

• 4:09:58Speaker 01

  Yeah. But. It's hard because it's like, it's still, you know, predicting the next token. It's not like image, the four image generation was like kind of a, it wasn't novel, I guess, because there was image generation.

• 4:10:09John Coogan?

  Yeah.

• 4:10:09Speaker 01

  But it was like such a massive improvement. This is

• 4:10:12John Coogan?

  Yeah.

• 4:10:12Speaker 01

  like, there's not any clear massive step change here. It's a little bit better in a lot of ways.

• 4:10:17John Coogan?

  Yeah.

• 4:10:18Speaker 01

  So.

• 4:10:18John Coogan?

  Oh, well, well, we'll have to play with it more. Let us know what you think about GPT-5 and we will see you tomorrow. Have a great day. Thank you so much. Bye.