Inside OpenAI Enterprise: Forward Deployed Engineering, GPT-5, and More | BG2 Guest Interview

Open Source bi-weekly convo w/ Bill Gurley and Brad Gerstner on all things tech, markets, investing & capitalism. This week, guest host Altimeter’s Apoorv Agrawal explores how OpenAI is reshaping enterprise with Sherwin Wu, Head of Engineering OpenAI Platform, and Olivier Godement, Head of Product OpenAI Platform. From T-Mobile’s AI voice support to Amgen’s drug breakthroughs to Los Alamos’ air-gapped supercomputer—this episode dives into the real world of AI at scale. Enjoy another episode of BG2! Timestamps: (00:00) Intro (01:50) OpenAI’s Enterprise Mission: Beyond ChatGPT (06:00) Case Study: T-Mobile <> Voice & Support (11:30) Case Study: Amgen <> Accelerating Drug Development (13:45) Case Study: Los Alamos National Lab (17:00) Why 95% of AI Deployments Fail? (20:30) Physical vs Digital Autonomy: Scaffolding & Infrastructure (26:00) GPT-5: Release, Benchmarks vs Behavior (30:00) GPT-5 Feedback: Instruction Following, Hallucinations, Code Quality (33:00) Multimodality: Text, Voice, and Video (35:30) Audio: Realtime API vs Stitched Audio (38:00) Model Customization & Reinforcement Fine-Tuning (RFT) (43:00) Rapid Fire: Long/Short Picks (1:03:00) Highlights and Lowlights @ OpenAI Show Notes: T-Mobile Partnership: https://www.t-mobile.com/news/business/t-mobile-launches-intentcx-with-openai Amgen Partnership: https://openai.com/index/gpt-5-amgen/ Los Alamos Partnership: https://www.lanl.gov/media/news/0130-open-ai MIT AI Report: https://mlq.ai/media/quarterly_decks/v0.1_State_of_AI_in_Business_2025_Report.pdf Produced by Dan Shevchuk Music by Yung Spielberg Available on Apple, Spotify, www.bg2pod.com http://www.bg2pod.com Follow: Brad Gerstner @altcap https://x.com/altcap Bill Gurley @bgurley https://x.com/bgurley BG2 Pod @bg2pod https://x.com/BG2Pod Apoorv Agrawal @apoorv03 https://x.com/apoorv03 Sherwin Wu @sherwinwu https://x.com/sherwinwu Olivier Godement @oliviergodement https://x.com/oliviergodement

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  We literally had to bring the weights of the model physically into their supercomputer.

• 0:04Apoorv Agrawal?

  In San Francisco, you could take a car from one part of SF to the other fully autonomously. As opposed to the digital world, I can't book a ticket online right now. Physical autonomy is ahead of digital autonomy in 2025.

• 0:16Speaker 00

  I think AI agents are like really in day one here, like ChatGPT only came out in 2022. And the slope, I think, is incredibly steep. I actually do think self-driving cars have a good amount of scaffolding in the world. You have roads, roads exist, they're pretty standardized, you have stoplights. AI agents are just kind of dropped in the middle of nowhere.

• 0:33Apoorv Agrawal?

  We'll start with long, short game.

• 0:35Speaker 00

  I'm short on the entire category of like tooling eval's products.

• 0:40Speaker 02

  Healthcare is probably the industry that will benefit the most from AI. I think I'm a JPL.

• 0:46Speaker 00

  You're definitely a JPL.

• 0:47Speaker 02

  The first one was the realization in 2023 that I would never need to code manually like ever, ever again.

• 1:01Apoorv Agrawal?

  Hey folks, I'm Apurva Gavirval. And today at the OpenAI office, we had a wide-ranging conversation about OpenAI's work in enterprise. I have with me the head of engineering and head of product of the OpenAI platform, Sherwin Wu and Olivier Godin. OpenAI is well-known as the creator of ChatGPT, which is a product that billions across the world have come to love and enjoy. But today, we dive into the other side of the business, which is OpenAI's work in enterprise. We go deep into their work with specific customers and how OpenAI is transforming large and important industries like healthcare, telecommunications, and national security research. We also talk about Sherwin and Olivier's outlook on the next, what's next in AI, what's next in technology, and their picks, both on the long and short side. This is a lot of fun to do. I hope you really enjoy it.

• 1:43Apoorv Agrawal?

  Well, two world-class builders, two people who make look building easy. Sherwin, my palter 2013 classmate, tennis buddy, with two stops at Cora and Open Door through the IPO before joining OpenAI, before ChatGPT. You've now been here for three years and lead engineering for all OpenAI platform.

• 2:05Apoorv Agrawal?

  Olivier, former entrepreneur, winner of the Golden Lama at Stripe, where you were for just under a decade, and now lead all of the product at OpenAI Platform. That's right. Thanks for doing it. Thank you.

• 2:17Speaker 00

  Thanks for having us.

• 2:18Apoorv Agrawal?

  As a shareholder, as a thought partner, kicking ideas back and forth, I always learn a lot from you guys. And so it's a treat. It's a real treat to do this for everybody. You know, I'll open with, people know OpenAI as the firm that built ChatGPT, the product that they have in their pocket that comes with them every day, to work, to personal lives. But the focus for today is OpenAI for enterprise. You guys lead OpenAI Platform. Tell us about it. What's underneath the OpenAI Platform for B2B for enterprise?

• 2:49Speaker 00

  Yeah, so this is actually a really interesting question too, because when I joined OpenAI around three years ago to work on the API, it was actually the only product that we had. So I think a lot of people actually forget this, where the original product from OpenAI actually was not ChatGPT. It was a B2B product. It was the API we were catering towards developers. And so I've actually seen the launch of ChatGPT and everything downstream from that. But at its core, I actually think the reason why we have a platform and why we started with an API is, it kind of comes back to the OpenAI mission. So our mission obviously is to build AGI, which is pretty hard in and of itself, but also to distribute the benefits of it to everyone in the world, to all of humanity. And it's pretty clear right now to see ChatGPT doing that,

• 3:34Speaker 00

  because my mom, maybe even your parents, are using ChatGPT. But we actually view our platform, and especially our API and how we work with our customers, our enterprise customers, as our way of getting the benefits of AGI, of AI, to as many people as possible, to everyone in every corner of the world. ChatGPT obviously is really, really, really big now. It's, I think, like the fifth largest website in the world. But we actually, by working through developers using our API, we're actually able to reach even more people in every corner of the world and every different use case that you might have. And especially with some of our enterprise customers, we're able to reach even use cases within businesses and reach end users of those businesses as well. And so we actually view the platform as kind of our way of fully expressing our mission,

• 4:18Speaker 00

  of getting the benefits of AGI to everyone. And so. Concretely, though, what the platform actually includes today. The biggest product that we have is obviously our developer platform, which is our API. You know, many developers, the majority of the startup ecosystem builds on top of this, as well as a lot of digital natives, Fortune 500 enterprises at this point. We also have a product that we sell to governments as well and the public sector. That's all part of this as well. And also an emerging product line for us in the platform is our enterprise products. So what we actually might sell directly to enterprises beyond just a core API offering.

• 4:55Apoorv Agrawal?

  Fascinating.

• 4:56Speaker 02

  And maybe to double down, like, I think B2B is actually quite core to the OpenAI mission.

• 5:04Speaker 02

  What we mean by distributing AGI benefits is, you know, I want to live in a world where, you know, there are like 10x more medicines like, you know, going out every year. I want to live in a world where, you know, education, like, you know, public service, civil service, you know, are like, you know, increasingly like optimized, you know, to everyone. And, you know, there are like a large category of use cases that only go through B2B, frankly, unless you enable like the enterprises. And, you know, we talked to Palantir, I think that's probably the same fee as at Palantir. It's like, hey, like, those are the businesses who are actually like, you know, making stuff happen in the real world. And so if you do enable them, if you do accelerate them, like, that's how essentially you benefit, you know, to distribute AGI.

• 5:43Apoorv Agrawal?

  Yeah. Well, maybe we can double click into that, Olivia. You know, the reach for chat is obviously wide.

• 5:50Speaker 02

  Yeah.

• 5:50Apoorv Agrawal?

  Billions of users, but for enterprise, it's maybe tell us about it. Maybe we go deep into a customer example or two. And what is an organization that we have helped transform, maybe? And at what layers?

• 6:03Speaker 02

  So if I were to step back, like, we started our B2B efforts with the API like a few years ago. Initially, the customers were startups, developers, indie hackers, extremely technically sophisticated people, like, you know, who are building like, you know, cool new stuff essentially, and taking massive like, you know, market ethical risk. So we still have a bunch of customers in that category, and we love them, and we keep building with them. On top of that, you know, over the past couple of years, we've been working one more with traditional like enterprises and also like digital natives. Essentially, I think basically everyone woke up like with GPT on like, those models are working. There is a ton of value and they could see essentially many use cases in the enterprise. Couple of examples which I like the most.

• 6:48Speaker 02

  One which is very both fresh and you know is quite cool. We've been working a lot with T-Mobile.

• 6:53Apoorv Agrawal?

  T-Mobile.

• 6:54Speaker 02

  So T-Mobile, leading like US telco operator. T-Mobile has like, you know, a massive customer support load. Like, you know, people asking like, you know, hey, I was charged like that amount of money. What's going on or you know, my, my cellphone like is not working anymore. A massive like, you know, share of that load is like, you know, voice calls. Like people want to talk to someone. And so for them, like, you know, to be able to essentially automate like more and more and, you know, to help like people like self-serve in a way. Like, you know, debug their, their subscription was pretty big. And so we've been working with T-Mobile pretty much for the past year. At that point, to basically automate like not only like text support, but also voice support. And so today, like, you know, there are like features like in the T-Mobile app. That if you call, actually handled by open, you know, models behind the scenes.

• 7:39Speaker 02

  And you know, it does sound like supernatural, like, you know, human sounding latency quality wise. So that one was really fun. A second one, which is very-

• 7:48Apoorv Agrawal?

  Just on that, can I, can I ask you a follow-up question? So we've got text models, we've got voice models, maybe even video models someday that are deployed at T-Mobile.

• 7:56Speaker 02

  Yeah.

• 7:57Apoorv Agrawal?

  But what above the models or, or, or adjacent to the models might we have helped T-Mobile with, for example?

• 8:03Speaker 02

  Yeah, there is a tongue we're doing. The first one is, you know, you have to put yourself in the shoes of an enterprise buyer. Like their goal is to automate, you know, or reduce, like, you know, optimize customer support. And you're going from like a model, like tokens in, tokens out. To that you case, it's hard. And so, you know, first like there's a lot of design, like, you know, system design. We do have actually now forward deployed engineers who are helping us quite a bit.

• 8:26Apoorv Agrawal?

  Forward deployed engineers.

• 8:27Speaker 02

  Yeah.

• 8:28Speaker 00

  Borrow the term from Palantir. Yeah, it's a great term.

• 8:31Speaker 02

  Were you an F D at Palantir?

• 8:33Speaker 00

  I was not an F D. I was on, I think they called it the dev side, right? It's like software engineering. I was also only an internet at Palantir. But yeah, it's a great term. I think it accurately describes what we're asking folks to do, which is like embed very deeply with customers. And, and honestly, like build things specific to their systems. They're deployed onto these customers. But yeah, we are obviously growing and hiring that team quite a bit because they've been very effective like at T-Mobile.

• 8:55Speaker 02

  For years of my life.

• 8:57Apoorv Agrawal?

  Yeah, forward deployed.

• 8:59Speaker 02

  Yeah.

• 8:59Apoorv Agrawal?

  But go ahead. So forward deployed engineering.

• 9:01Speaker 02

  Forty twenty years and the sort of like systems and like integrations they're doing is, you know, first, like, you know, you have to orchestrate those models. Like those models are not just, you know, those models like know nothing about like, you know, the CRM, like, you know, and like what's going on. And so you have to plug the model to like many, many different nectar tools. Many of those like tools, like in the enterprise, do not even have like APIs or like clean interfaces. Right. It's the first time they've been exposed like, you know, to a third party system. And so there is a lot of, you know, standing up like, you know, API gateways, like tools connecting.

• 9:32Speaker 02

  Then you have to essentially like define what good looks like, you know. Again, like it's a pretty new exercise for everyone, like, you know, defining like a golden set of evals is, you know, easier than it sounds, harder than it sounds. And so we've been spending like a bunch of time with them.

• 9:48Apoorv Agrawal?

  Eval's are important.

• 9:48Speaker 02

  Eval is super important.

• 9:49Speaker 00

  Especially like audio evals. I know audio evals are like extra hard to grade and get right. And but like the bulk of the use case here is actually audio and you have like, I don't know, five minute like call transfer, how do you actually know that the right thing happened? It's a pretty tough problem.

• 10:02Speaker 02

  Yeah, it's pretty tough. And then, you know, actually nailing down like the quality of the customer experience, like, you know, until it feels natural. And here latency and interruptions, they're really like, you know, important part. We shipped in GA, an API, a real time API. I think it was last week.

• 10:21Apoorv Agrawal?

  Couple of weeks ago.

• 10:22Speaker 02

  Yeah.

• 10:23Speaker 00

  It was just last week, I think.

• 10:24Speaker 02

  Which is like a beautiful work of engineering. You know, there was a really cracked team behind the scenes. Which basically allows us like to get like the most like natural sounding like, you know, voice experience without having like these weird interruptions on your lag where you can feel that essentially the thing is off. So yeah, cobbling all that together, you know, and you get like, you know, a really good experience.

• 10:46Apoorv Agrawal?

  Yeah, that's a lot more than just models.

• 10:47Speaker 02

  Yeah.

• 10:48Speaker 00

  Yeah, I was gonna say, one actually really great thing that I think we've gotten from the T-Mobile experience is actually working with them to improve our models themselves. So for example, the last real time, the real time GA last, last week, we obviously released a new snapshot, the GA snapshot. And a lot of the improvements that we actually got into the model came out of, you know, the learnings that we had from T-Mobile. It brings in a lot of other change from other customers, but because we were so deeply embedded into T-Mobile and we were able to understand what good looks like for them, we were able to bring that to some of our models.

• 11:16Apoorv Agrawal?

  That makes sense. So this is a large customer with tens of millions of users, if not hundreds of millions. And the before and after is on the support side, both tech support internally and then their customer support.

• 11:26Speaker 02

  Yeah.

• 11:26Apoorv Agrawal?

  Makes sense?

• 11:27Speaker 02

  Yeah.

• 11:27Apoorv Agrawal?

  Is there another one that you guys can share?

• 11:30Speaker 02

  I like a lot Amgen.

• 11:33Apoorv Agrawal?

  Amgen, the healthcare business?

• 11:34Speaker 02

  Amgen, yeah. So we are working quite a bit with healthcare companies. Amgen is one of the leading healthcare companies. We specialize into drugs for cancer or inflammatory diseases. They're based out of LA. And we've been working essentially with Amgen to essentially speed up the drug development and commercialization process. So the North Star is pretty bold. And it's really interesting, similarly, we embedded pretty deeply with Amgen to understand what are their needs. And it's really interesting, when I look at those healthcare companies, I feel like there are two big buckets of needs. One is pure R and D.

• 12:15Speaker 02

  It's like you're seeing a massive amount of data and you have super smart scientists who are trying to combine, test out things. So that's one bucket. A second bucket is much more common across other industries. It's like pure admin, document authoring, documents reviewing work, which is, by the time your R and D team has essentially locked the recipe of a medication, getting that medication to market is a ton of work. You have to submit to various regulatory bodies, get a ton of reviews. And when we looked at essentially those problems, what we knew, what models we're capable of, we saw a ton of benefits, a ton of opportunities to automate and augment essentially the work of those teams.

• 12:59Speaker 02

  And so, yeah, Amgen has been a top customer of DPC5, for instance.

• 13:02Apoorv Agrawal?

  Wow. I mean, this could be hundreds of millions of lives if a new drug is developed faster.

• 13:07Speaker 02

  Yeah, exactly.

• 13:09Apoorv Agrawal?

  Huge impact.

• 13:09Speaker 02

  So that's, I think, one good example of a kind of impact on which you need to enable enterprises to do it.

• 13:17Apoorv Agrawal?

  Right.

• 13:17Speaker 02

  You know? And so I think we're going to do more and more of those. And yeah, frankly, on a personal level, it's a delight. If I can play a tiny role essentially, doubling the kind of medication that people get in the real world, that feels like a pretty good achievement.

• 13:32Apoorv Agrawal?

  Huge.

• 13:33Speaker 02

  Huge, huge. I know you had one.

• 13:35Speaker 00

  Yeah, so one of my favorite deployments that we've done more recently, actually, is with the La Salamos National Labs. So this is the government national research lab that the US government is running in La Salamos, New Mexico. It's also where the Manhattan Project happened back in the 40s and 50s, back when it was a secret project. So after that, they ended up formalizing it as a city and a program, and then now it's a pretty sizable national laboratory. This one is very interesting because, one, just the depth of impact here is unimaginable. For me, it's on the scale of Amgen and some of these other larger companies. But obviously, they're doing a lot of actual new research there, so a lot of new science. They're doing a lot of stuff with our Defense Department and defense use cases as well.

• 14:20Speaker 00

  So very intense, you know, very intense stuff. But the other thing that's actually very interesting about this one was that it's also a story of a very bespoke and new type of deployment that we've done. So because they are so, they're a government lab, they're so restrictive and high-security and high-clearance with a lot of their things, we couldn't just do a normal deployment with them. They couldn't, you know, you can't have people doing national security research just hitting our APIs. And so we actually did a custom on-prem deployment with them onto one of their supercomputers called Binado. And so this actually involves a bunch of very bespoke work with some FTEs, also with a lot of our developer team, to actually bring one of our reasoning models, O3,

• 15:02Speaker 00

  into their laboratory, into an air-gapped supercomputer, Binado, and actually deploy it and get it installed to work on their hardware, on their networking stack, and actually run it in this particular environment. And so it was actually very interesting because we literally had to bring the weights of the model physically into their supercomputer in an environment, by the way, where you're not allowed to have, you know, it's very locked down for a good reason. You're not allowed to have, like, cellphones or any electronics with you as well. And so I think that was a very, very unique challenge. And then the other interesting thing about this deployment is just how it's being used, right? So the interesting thing is, because it's so locked down and on-prem, We actually do not have much visibility into exactly what they're doing with it.

• 15:46Speaker 00

  But we do have, you know, they give us feedback. Yeah, yeah. They actually do have some telemetry, but it's within their own systems. But we do know that it's being used for a bunch of different things. It's being used for aiding them in terms of speeding up their experiments. They have a lot of data analysis use cases, a lot of notebooks that they're running with reams of data that they're trying to process. They're actually using it as a thought partner, which is something that's pretty interesting to me. O3 is pretty smart as a model. And a lot of these people are tackling really tough, you know, novel research problems. And a lot of times they're kind of using O3 and going back and forth with it on their experiment design, on like what they actually should be using it for, which is something that we can't really say about our older models.

• 16:30Speaker 00

  And so, yeah, it's just being used for a lot of different use cases for the national lab. And the other cool thing is it's actually being shared between Los Alamos and some of the other labs, Lawrence Livermore, Sandia as well, because it's the supercomputer setup where they can all kind of connect with it permanently.

• 16:46Apoorv Agrawal?

  Fascinating. I mean, we've just gone through three pretty large-scale enterprise deployments, right? Which might touch tens, if not hundreds of millions of people. But there was this, on the other side of this is the MIT report that came out a couple of weeks ago. 95% of AI deployments don't work. A bunch of, you know, scary headlines that even shook the markets for a couple of days. Like, you know, put this in perspective, like for every deployment that works, there's presumably a bunch that don't work. So maybe we can, you know, maybe talk about that. Like, what does it take to build a successful enterprise deployment, a successful customer deployment? And the counterfactual, based on all your experience serving all these large enterprises?

• 17:29Apoorv Agrawal?

  I think at that point, I may have worked with like a couple of hundreds, I think, enterprises. So, okay, I'm going to a pattern match.

• 17:38Speaker 02

  What I've seen being like clear leading indicator of success. Number one is like the the interesting combination of like top down like buy in and like enabling like you know very clear group of like a tiger team essentially. Like you know, at the enterprise, which is sometimes a mix of like OpenAI, like you know, enterprise employee. So you know, typically, like you know, you take like T-Mobile, like the top leadership was like extremely bottom in, like it's a priority. Right, right. But then letting the team like you know organize and be like, okay, if you want to start small, start small. Right. You know, and then we can scale it up essentially. So that would be part number one. So top down buy in and a bottom, call it tiger team. Tiger team, you know, people like, you know, a mix of like technical skills and like people who just have like the organizational knowledge, like institutional knowledge, you know.

• 18:23Speaker 02

  It's really funny, like in the enterprise, like customer support is a good example, like what we found is that the vast majority of the knowledge is in people's heads. Right. Right, which is probably like a thing that you know, we have these like in general. But like, you know, you take like customer support, you would think that you know, everything is like perfectly documented, like you know, et cetera. The reality is like the the standard like operating procedures, like SOPs, are larger than people's head. And so unless you have that tiger team, like mix of like technical and like, you know, subject matter expert, really hard like to get something underground. That would be one.

• 18:56Speaker 02

  Two would be evals first, like whatever we define like good evals, like that gives like a clearly clear common goal for people to hit. Whenever like, you know, the customer like fails to come up with good evals, it's a moving target. You never know essentially, you know, if you've made it or not. And you know, evals are much harder than what it looks to get done.

• 19:16Speaker 00

  And evals also oftentimes you need to come bottom up, right? Because all of these things are kind of in people's heads, in the actual operator's heads. Like, it's actually very hard to have a top down mandate of like, you got like, this is how the evals should look. A lot of it needs the bottoms up adoption.

• 19:29Speaker 02

  Right. Yeah, yeah, yeah. And so we've been building quite a bit of tooling on evals. We have like a universe product and you know, we're working on more to essentially solve like, you know, that problem or you know, make it as easy as we can. The last thing is, you know, you want to hill climb essentially. You have your evals, the goal is to get to 99%. You start at like, you know, 46, you know, how do you get there? And here, frankly, I think oftentimes, like you know, a mix of like.

• 19:56Speaker 02

  I would say like almost wisdom from people who've done it before. Like, you know, a lot of that is like, you know, like art sometimes more than science. And like, you know, knowing like the quirks of the model, the behavior. Sometimes we even need to fine tune ourselves, the models, you know, when there are some clear limitations. And, you know, being patient, getting your way, you know, up there and then, you know, ship.

• 20:17Apoorv Agrawal?

  Can we go under the hood a little bit? You know, one of the things that we think about a lot is autonomy more broadly, right? What is the makeup of autonomy on one side? You know, in San Francisco, you could take a car from one part of SF to the other fully autonomously. No humans involved, no, you press a button.

• 20:33Speaker 00

  We love the Waymos.

• 20:34Apoorv Agrawal?

  Right? They've done billions of rides. I think it was like, what, three and a half billion rides? This is on the Tesla FSD. I think Waymo's done like million, tens of millions of rides. That's a lot of autonomy.

• 20:45Speaker 00

  Yeah.

• 20:46Apoorv Agrawal?

  In the physical world, as opposed to the digital world, I can't book a ticket online right now. There's all sorts of problems that happen if I have my operator try to book a ticket. And it's very counterintuitive because the bar for physical safety is so much higher. The bar for physical safety is higher than the human's capability because lives are at stake.

• 21:06Speaker 00

  Yeah.

• 21:07Apoorv Agrawal?

  The bar for digital safety, not that high because all you're going to lose is money. Nobody's life is at stake, but yet physical autonomy is ahead of digital autonomy in 2025. What seems counterintuitive, like, why is that the case at, you know, at a technical level? Why is it that what should sound easier is actually a lot harder?

• 21:29Speaker 00

  Yeah, so I think there are kind of two things at play here. And I really like the analogy with self-driving cars because they've been, they've actually been like one of the best applications of AI, I think, that I've used recently. But I think there are two things in play. One of them is honestly just the timelines. Like, we've been working on self-driving cars for so long. I remember when I, you know, back in like 2014, it was kind of like the advent of this. And everyone was like, oh, it's happening in like five years. Turns out it took like, I don't know, 10, 15 years or so for this thing. So there's been a long time for the technology to really mature. And I think there's probably like dark ages, you know, back in like 2015 or 2018 or something, where it felt like it wasn't gonna happen.

• 22:05Apoorv Agrawal?

  The trough of disillusionment.

• 22:06Speaker 00

  Yes, yes, yeah. And then now we're, you know, finally seeing it get deployed, which is really exciting. But it has been like, I don't know, 10 years, maybe even 20 years from the very beginning of the research. Whereas I think AI agents are like really in day one here, like ChatGPT only came out in 2022. So like around three, like less than three years ago. But I actually think that what we think about with AI agents and all that really, I think, started with the reasoning paradigm that when we released the O1 preview model back in late last year, I think. And so I actually think this whole reasoning paradigm with AI agents and the robustness that those bring has only really unfolded for like a year, less than a year, really.

• 22:47Speaker 00

  And so I know you had a chart in your blog post, which I really like, which, you know, the slope is very meaningfully different now. Like self-driving started very, very early. Slope seems to be a little bit slower, but now it's reaching the promised land. But man, like we started super recently with AI agents and the slope, I think, is incredibly steep. And we'll probably see a crossover at some point. But we really have only had like a year really to explore these.

• 23:10Speaker 02

  Do you think we haven't crossed over already when you look at like the coding work in particular?

• 23:14Speaker 00

  Yeah, it's a good point. It's like, you know, your chart actually shows AI agents as below self-driving, but like, you know, it was like, what is the Y-axis? Like by some measures, like I would not be surprised, actually, if, you know, AI products or AI agent products is making more revenue than Waymo at this point. Like Waymo is making a lot, but like, just look at all the startups coming up. Look at, you know, ChatGPT and how many subscriptions are happening there and all of that. And so maybe we have actually crossed and, you know, a couple of years from now, it's going to look very, very different.

• 23:40Apoorv Agrawal?

  Yeah, the Y-axis is tangible felt autonomy. Perfectly objective.

• 23:47Speaker 00

  How do I feel about?

• 23:48Apoorv Agrawal?

  Exactly vibes more than revenue, but revenue is a good one. We should probably redo that with revenue.

• 23:53Speaker 00

  The, there's a second thing I wanted to mention on this as well, which is the scaffolding and the environment in which these things operate in. So I actually remember in the early days of self-driving, a lot of the like researchers around self-driving were saying that the roads themselves will have to like change to accommodate self-driving, right? There might be like sensors everywhere so that the self-driving cars can interact with it, which I think is like, you know, retrospect overkill. But I actually do think self-driving cars have a good amount of scaffolding in the world for them to operate in. It's like not completely. Like unlimited. You have roads, roads exist, they're pretty standardized, you have stoplights. People generally operate in like pretty normal ways. And there are all these traffic laws that you can learn. Whereas AI agents are just kind of dropped in the middle of nowhere.

• 24:38Speaker 00

  And they kind of have to feel around for them. And I actually think, you know, going off of what Olivier just said too. My hunch is some of the enterprise deployments that don't actually work out. Likely don't have the scaffolding or infrastructure for these agents to interact with as well. A lot of the like really successful deployments that we've made, a lot of what our FDEs end up doing with some of these customers is to create. Almost like a platform or some type of scaffolding connectors, organizing the data. So that the models have something that they can interact with in a more standardized way. And so my sense of self-driving cars actually have had this in some degree with roads over the last. Over the course of their deployment. But I actually think it's still very early in the AI agents space. And I would not be surprised if a lot of these, a lot of enterprises, a lot of companies just don't really have the scaffolding ready.

• 25:23Speaker 00

  So if you drop an AI agent in there, it kind of doesn't really know what to do. And its impact will be limited. And so I think once this scaffolding gets built out across some of these companies, I think the deployment also speed up. But again, to our point earlier, I think there's no slowdown, there's no, you know, things are still moving very fast.

• 25:38Apoorv Agrawal?

  That's great. Well, you know, I've thought about autonomy as a three-part. Structure, you've got perception, you've got the reasoning, the brain. And then you've got the scaffolding, the last mile of making things work. Maybe we can dive into the second part, which is the reasoning, which is the juice that you guys are building with GPT-5. Most recently, huge endeavor, congrats. The first time you guys have launched a full system, not a model or a set of models, but a full system. Talk about that. I mean, the full arc of that development, what was your focus? I mean, honestly, the benchmarks all seem so saturated, like clearly it was more than just benchmarks that you were focused on. And so what was the north star? Like, tell us about GPT-5, soup to nuts.

• 26:22Speaker 02

  It's been the work of love of many people for a long time.

• 26:27Speaker 02

  And to your point, I think GPT-5 is amazingly intelligent. You look at the benchmark, like, you know, the Sweetbench and the likes, you know, it is going pretty high. But I think to me, equally important and impactful was, I would say, the craft, like, the style, the tone, the behavior of the model. So, you know, capability is intelligence and, you know, behavior of the model. On the behavior of the model, I think it's the first model, like, large model release, for which we have worked so closely with a bunch of customers for like month and month, essentially, to better understand, like, what are the concrete locks? Like, what are the concrete blockers of the model? And often, like, you know, it's not about, like, you know, like, having a model which

• 27:09Speaker 02

  is way more intelligent, a model which is faster, a model that better, like, follows instruction, a model that is more likely to say no, you know, when, you know, he doesn't know about something. And so that, like, super close, like, you know, customer feedback loop on GPT-5 was pretty impressive to see. And I think, like, all the love that GPT-5 has been getting, like, you know, in the past, like, couple of weeks, I think people are starting to share that, essentially, the builders. And once you see it, like, it's really hard, essentially, to come back to a model which is, like, extremely intelligent, but more, you know, an exclusively, like, academic, essentially, way.

• 27:43Apoorv Agrawal?

  Yeah. Are there trade-offs that you made as you were going through it? Like, maybe what are the hardest trade-offs you made as you were building GPT-5?

• 27:51Speaker 00

  I think a very clear trade-off, which I honestly think we are still iterating on, is the trade-off between the reasoning tokens and how long it thinks versus performance.

• 28:00Apoorv Agrawal?

  Yeah.

• 28:00Speaker 00

  Because, and I honestly, this is something that I think we've been working on with our customers since the launch of the reasoning models, which is, these models are so, so smart, especially if you give it all this, like, thinking time. I think the feedback I've been seeing around GPT-5 Pro has been pretty crazy, too. It's just like, like, you know, these unsolved-

• 28:19Speaker 02

  Andre had a great tweet last night.

• 28:20Speaker 00

  Yeah, yeah, I saw that Sam retweeted it. But like, these, like, unsolved problems that none of the other models could handle, you throw it to GPT-5 Pro and- It just like one shots it is pretty crazy. But the trade off here is you're you're waiting for 10 minutes. It's quite a long time. And so these things just get like so smart with with with more inference time. But on the like product builder, on the API side for some of these like business use cases. I think it's pretty tough to like, you know, manage that trade off. And for us, it's been difficult to figure out where to where we want to fall on that spectrum. So we've had to make some trade offs on like, how much of the model think versus like how intelligent should it get? Because as a product builder, there's a latency, there's a real latency trade off that you have to deal with. Where, you know, your user might not be happy waiting 10 minutes for like the best answer in the world. It might be more okay with the substandard answer and like no wait at all.

• 29:05Apoorv Agrawal?

  Yeah, I mean, even between GPT 5 and GPT 5 thinking. I have to toggle it now because sometimes I'm so impatient, I just want it ASAP.

• 29:13Speaker 00

  Yeah, I think there's an ability to skip, right? And yeah, it's right. Where it's like, I'm impatient, I just want the more simple answer.

• 29:19Apoorv Agrawal?

  That's right, that's right. Well, four weeks in, GPT 5, how's the feedback?

• 29:24Speaker 00

  Yeah, I think the feedback has been very positive, especially on the platform side, which has been really great to see. I think a lot of the things that Olivier mentioned have been, you know, come up in feedback from customers. The model is extremely good at coding, extremely good at kind of like reasoning through through different tasks. But especially for like coding use cases, especially at the, you know, at the, when it thinks for a while. It'll usually solve problems that no other models can solve. So I think that's been a big positive point of feedback. The kind of robustness and the reduction in hallucinations has been a really big positive feedback.

• 29:58Apoorv Agrawal?

  Yeah.

• 29:59Speaker 00

  I think there's an eval that showed that hallucinations basically went to zero for a lot of this. It's not perfect, there's still a lot of work to be done, but.

• 30:05Apoorv Agrawal?

  That's a big one.

• 30:06Speaker 00

  I think because of the reasoning in there too, it just makes the model more likely to say no, less likely to hallucinate answer. So that's been something that people have really liked as well. Other bit of feedback has been around instruction following. So it's really good at instruction following. This almost bleeds into like the constructive feedback that we're working on where. Heard that it's so good at instruction following, that instruction following that people need to tweak their prompts, or it's almost like too literal in.

• 30:30Speaker 02

  That one is an interesting trade off, actually. Because you know, when you ask people, developers, like what do you want? Like you want the model to follow instructions, of course. But once you have a model which, who is like, that is like extremely literal, essentially, then essentially it forces you to express extremely clearly what you want. Otherwise the model, you know, may go sideways. And so that was interesting feedback.

• 30:49Speaker 00

  It's almost like the monkey paw where it's like developers and platform customers ask for better instruction following. So like, yes, we'll give you really good instruction following, but it was like, you know, follows it almost to a T. And so it's obviously something that the team is actually working, working through. I think a good example of this, by the way, is some customers would have these prompts. I remember when we were testing GPT-5, one of the negative feedback that we got was the model was too concise. We were like, what's going on? Why is the model so concise? And then we realized it was because they were reusing their old prompts from other models. And with the other models, they have to like, you have to like really beg the model to be concise. So they're like 10 lines of like, be concise, really be concise, also keep your answer short. And it turned out when you give that to GPT-5, it's like, oh my gosh, this person really wants it to be concise.

• 31:32Speaker 00

  And so their response would be like one sentence, which is too terse. And so just by removing the extra prompts around being concise, the model behaved in a much better way and much closer to what they actually end up wanting.

• 31:43Speaker 02

  Turns out, writing the right prompt is still important.

• 31:45Speaker 00

  Yes, yes, yeah. Prompt engineering is still very, very important. Yeah. On constructive feedback for GPT-5, there's actually been a good amount as well, which we're all, we're all working through. One of them that I think is, I'm really excited for the next snapshot to come out to fix some of this is code quality. And like small like code, like paradigms or like idioms that they might use. I think there are like feedback around the types of code and the patterns in which it was using, which I think we're working through as well. And then the other bit of feedback, which I think we've already made good progress on internally, is around the trade-off of the reasoning tokens and thinking and latency around intelligence. I think especially for the simpler problems, you don't usually need a lot of thinking. The thinking should ideally be a little bit more dynamic.

• 32:31Speaker 00

  And of course, we're always trying to squeeze as much reasoning and performance into as little reasoning tokens as possible. So I had mentioned that that curve kind of going down as well.

• 32:39Apoorv Agrawal?

  Yeah. Well, huge congrats. I mean, it's been, I know it's a work in motion for a bunch of our companies. They've had incredible outcomes with GPT-5. One of them is expo cybersecurity business, just like a huge.

• 32:49Speaker 00

  Yeah, I saw the chart from that. It was pretty crazy.

• 32:51Apoorv Agrawal?

  Huge, huge upgrade from whatever they were using prior to that.

• 32:55Speaker 00

  I think they're going to need a new eval soon.

• 32:57Apoorv Agrawal?

  That's right. They're going to need a new eval. It's all about evals. On the multimodality side of it, obviously, you guys announced the real-time API last week. I saw T-Mobile was one of the featured customers on there. Talk about that, like how obviously the text models are quite leading the pack. But then we got audio and we got video. Talk about the progress on the multimodal models. When should we expect to have like the next big unlock and what would that look like?

• 33:23Speaker 02

  It's a good question. The teams have been making amazing progress on multimodality, on voice, image, video. Frankly, the last generation of models have been unlocking like quite a few cool use cases. One of the feedback that we've received is, you know, because like text was so much leading the pack on intelligence, like people felt taking part in voice that the model was somewhat a little less intelligent. And you know, until you actually see it, like it does feel weird, like you know, to have like, you know, to have a better answer like on text versus voice. And so that's pretty much a focus that we have at the moment.

• 33:55Speaker 02

  I think we like filled like part of that gap, but not the full gap for sure. So I think, you know, catch up, I would say, you know, with the text, like, you know, would be one. A second one, you know, which is absolutely fascinating is the model is like excellent at the moment on like, you know.

• 34:14Speaker 02

  Easy casual conversation, like talk to your coach, your therapist. And we basically had to teach the model like to speak essentially better like in actual like, you know, work economically valuable setups. Give an example, like the model has to be able to understand what an SSN is and you know, what it means to spell SSN. And if one digit is actually like, you know, fuzzy, it should actually has to repeat versus, you know, guess. You know, there are lots of like, you know, intuitions like that that someone, you know, of course has, you know, of our voice that we are currently like teaching the model. And that's like an ongoing work actually with our customers, you know, until we actually confront the model to like actual like customer support calls, actual sets call. It's really hard like to get a feel, you know, for those gaps. So that's a top like, you know, priority as well.

• 34:59Apoorv Agrawal?

  This is a completely off script, but an interesting question that comes up in voice models, particularly the real time API is, you know, previously people were doing, they were taking a speech input, convert that to text.

• 35:11Speaker 02

  Yeah.

• 35:12Apoorv Agrawal?

  Then have some layer of intelligence.

• 35:13Speaker 02

  Yeah.

• 35:14Apoorv Agrawal?

  Then you would have a text to speech model that would sort of play it back.

• 35:17Speaker 02

  Yeah.

• 35:18Apoorv Agrawal?

  And this would be, it would be a stitch of these three parts. But the real time API, you guys have integrated all of that.

• 35:24Speaker 02

  Yes.

• 35:25Apoorv Agrawal?

  And how, you know, how does it happen? Because a lot of the logic is written in, in text. A lot of the Boolean logic or you call it any function calling is written in text. How, how does it work with the real time API? Is that?

• 35:37Speaker 02

  That's an excellent question. So the reason why we should the real time API is that we saw that for this teach model.

• 35:45Apoorv Agrawal?

  The stitch model.

• 35:46Speaker 02

  Yeah.

• 35:46Apoorv Agrawal?

  Oh, the real time.

• 35:47Speaker 02

  The stitch.

• 35:48Speaker 00

  Like a stitch together.

• 35:49Speaker 02

  Stitch, you get like speech to text, thinking, text to speech.

• 35:52Apoorv Agrawal?

  Yeah.

• 35:53Speaker 02

  Like we saw essentially a couple of issues. One, slowness, like, you know, more hops essentially.

• 35:57Apoorv Agrawal?

  Yeah, yeah, yeah.

• 35:58Speaker 02

  Two, loss of signal, like, you know, across each model. Like the speech text model is less intelligent.

• 36:05Apoorv Agrawal?

  Yeah, you'd lose emotion.

• 36:06Speaker 02

  Exactly.

• 36:07Apoorv Agrawal?

  Accent, tone.

• 36:08Speaker 02

  Exactly.

• 36:08Apoorv Agrawal?

  Right? Pauses.

• 36:10Speaker 02

  Yeah. And you know, when you, when you are doing like actual voice, like phone like calls, essentially like those signals are like so important, like for essentially to the stem.

• 36:19Apoorv Agrawal?

  Yeah.

• 36:21Speaker 02

  One of the challenges that we have is what you mentioned, which is, you know, it means like a slightly different architecture essentially for text versus voice. And so that's something that we are actively working on. But I think it was the right call to start essentially with, let's make like the voice experience like natural sounding to a point where essentially you're. Feeling comfortable like putting in production and then working backward like to unify like the sort of the orchestration logic essentially across modalities.

• 36:46Speaker 00

  And to be clear, like a lot of customers still stitch these together. It's like kind of what worked in the last generation. But what we're interesting seeing is more and more customers moving towards the real time approach because of how natural it sounds, how much lower latency. It is especially as we up level the intelligence of the model. But also even like taking a step back, I will say it's like pretty mind blowing to me that it works. Like the fact that like, I think it's mind blowing that these L M's work at all. Where you just train it on a bunch of texts and it's just, you know, auto aggressively coming up with the next token and it sounds super intelligent. That's like mind blowing in and of itself. But I think it's actually even more mind blowing that this speech to speech setup actually works correctly. Cause you're literally taking the audio bits of from it, from someone speaking, streaming or putting it into the, the model and then it's generating audio bits back. And so, uh, to me, it's actually crazy that this works at all.

• 37:31Speaker 00

  Let alone the fact that it can understand accents and tone and pauses and things like that. And then also be intelligent enough to handle a support call or something like that.

• 37:38Apoorv Agrawal?

  I mean, you've gone from text in text out to voice in voice out. That's pretty crazy. We have a bunch of companies in our portfolio that are using these models, you know, Parlo on the customer support side. LiveKit on the infra side and, you know, there's a bunch of use cases. We were starting to see that, that, that a speech to speech model could, could address. Obviously a lot of the harder ones still, still running on what you're calling the stitched model. But I, I, I hope the day is not far when it's all on real time API.

• 38:06Speaker 02

  It's gonna happen at some point.

• 38:08Apoorv Agrawal?

  Right, right, right, right. And actually, maybe that's a good segue into talking about model customization. Because I suspect that you have such a wide variety of enterprise customers. I think you mentioned what hundreds of customers or, or maybe more. Each of them has a different use case, a different problem set, a different, call it envelope of, of, of parameters that they're working in, maybe latency, maybe, maybe power, maybe others. How do you, how do you handle that? Talk about what OpenAI offers enterprises who need a customized version of, of, of a great model to make it great for them.

• 38:38Speaker 00

  Yeah, so model customization is actually been something that we've invested very deeply in on the API platform since the very beginning. So even, you know, pre-ChatGPT days, we actually had a supervised fine-tuning API available and people were actually using it to great effect. The most exciting thing actually, I'd say, around model customization, it obviously resonates quite well with customers because they want to be able to bring in your own custom data and create your own custom version of, you know, O3 or O4 mini or something, or GPT-5 even, suited to their own needs. It's very attractive, but the most recent development I think is very exciting has been the introduction of reinforcement fine-tuning, which is something we announced late last year, I think in the 12 Days of Christmas. We've GA'd it since and we're continuing to iterate on it.

• 39:22Speaker 02

  What is it? Break it down for us.

• 39:23Speaker 00

  Yeah, so it's called, it's actually funny. I think we made up the term reinforcement fine-tuning. It's like not a real thing until we.

• 39:30Speaker 02

  It's stuck now. I see it on Twitter all the time.

• 39:33Speaker 00

  I remember we were discussing it and I was like, I don't know about RFT, guys.

• 39:36Speaker 02

  You're not kidding.

• 39:37Speaker 00

  Yeah, yeah. So reinforcement fine-tuning. So it really, it's introducing reinforcement learning into the fine-tuning process. So the original fine-tuning API does something called supervised fine-tuning, call it SFT. It is not using reinforcement learning. It is, you know, it's using supervised.

• 39:56Speaker 02

  Supervised learning, yeah.

• 39:57Speaker 00

  And so what that usually means is you need a bunch of data, a bunch of prompt completion pairs. You need to really supervise and tell exactly the model how to, how it should be acting. And then when you train it on our fine-tuning API, it moves it closer in that direction. Reinforcement fine-tuning introduces like RL or reinforcement learning to this loop. Way more complex, way more finicky, but in order of magnitude more powerful. And so that's actually what's really resonated with a lot of our customers. It allows you to, if you use RFT, the discussion is less of like creating a custom model that's specific to your own use case. It is, you can actually use your own data and actually crank the RL, yeah, turn the crank on RL to actually create a like best-in-class model for your own particular use case. And so that's kind of the main difference here.

• 40:42Speaker 00

  With RFT, the dataset looks a little bit different. Instead of prompt completion pairs, you really need a set of tasks that are very gradable. You need a grader that is very objective that you can use here as well. And so that's actually been something that we've invested a lot in over the last year. And we've actually seen a couple, a good number of customers get really good results on this. We've talked about a couple of them across different verticals. So Rogo, which is a startup in the financial services space. They have a very sophisticated AI team. I think they hired some folks from DeepMind to run their AI program. And they've been using RFT to get best-in-class results on parsing through financial documents, ask questions around it, and doing tasks around that as well. There's another startup called accordance that's doing this in the tax space.

• 41:27Speaker 00

  I think they've been targeting an eval called Taxbench, which looks at CPA-style tasks as well. And because they're able to turn it into a very gradable setup, they're actually able to turn the RFT crate and also get, I think, soda results on a tax bench just using our RFT product as well. And so it has kind of shifted the discussion away from just customizing something for your own use case to really leveraging your own data to create a best-in-class, maybe best-in-the-world model for something that you care about for your business.

• 41:56Speaker 02

  Yeah, I feel like the base models are getting so good at instruction-following that for behavior, like steering, you don't need to find you at that point. You can describe what you want, and the model is pretty good at it. But pushing the frontier on actual capabilities, my hunch is that RFT will pretty much become the norm. If you are actually pushing in your field intelligence to a pretty high point, at some point, you need to hire, essentially, with custom environments.

• 42:27Speaker 00

  Fascinating. And even going back to the point earlier around top-down versus bottoms-up for some of these enterprises, a lot of the data that you end up needing for RFT require very intricate knowledge about the exact task that you're doing and understanding how to grade it. And so a lot of that actually comes from bottoms-up. I know a lot of these startups will work with experts in their field to try and get the right tasks and get the right feedbacks to craft some of these datasets.

• 42:51Apoorv Agrawal?

  Without further ado, we're going to jump into my favorite section, which is a rapid-fire question. We had a lot of great friends of ours send in some questions for you guys. We'll start with Ultimater's favorite game, which is the long-short game. Pick a business, an idea, a startup that you're long. And the same short that you would bet against that there's more hype than there's reality. Whoever's ready to go first, long, short?

• 43:16Speaker 00

  My long is actually not in the AI space, so this is going to be slightly different.

• 43:20Apoorv Agrawal?

  Wow, there we go.

• 43:21Speaker 00

  My short is, though, in the AI space. So I'm actually extremely long esports. And so what I mean by esports is the entire, like, professional gaming industry that's emerging around video games. Very near and dear to my heart, I play a lot of video games, and so I watch a lot of this. So obviously, I'm pretty in the weeds on this. But I actually think there's incredible untapped potential in esports and incredible growth to be had in this area. So concretely, what I mean are like, you know, a really big one is League of Legends, all of the games that Riot Games puts out. They actually have their own professional leagues. They actually have professional tournaments, believe it or not. They rent out stadiums, actually, now. But I just think it's like, if you look at kind of what the youth and like what younger

• 44:03Speaker 00

  kids are looking and where their time is going, it's predominantly going towards these things. They spend a lot of time on video games.

• 44:09Speaker 02

  They watch more esports than soccer, basketball, you know?

• 44:11Speaker 00

  Yeah, yeah, yeah, yeah, a growing number of these, too. I've actually been to some of these events, and it's very interesting.

• 44:18Speaker 02

  He's very committed to his long.

• 44:19Speaker 00

  Yeah, yeah, I'm extremely long in stuff.

• 44:22Apoorv Agrawal?

  And so they're booking out stadiums for people to go watch electronic sports?

• 44:27Speaker 00

  Yeah, yeah, yeah. It's a, I literally went to Oracle Arena, the old Warriors stadium, to watch one of these, I think, before COVID.

• 44:34Apoorv Agrawal?

  Wow.

• 44:35Speaker 00

  And then the, so it's just...

• 44:37Apoorv Agrawal?

  Before COVID, wow, that's five years ago, six years ago.

• 44:39Speaker 00

  So I actually, I've been following this for a while, and I actually think it had a really big moment in COVID, like, everyone was playing video games. And I think it's kind of like come back down. So I think it's like, undervalued, you know, it's like, I think no one's really appreciating it now. But it has all the elements to like, really, really take off. And so the youth are doing it. The other thing I'd say is it is huge in Asia, like, absolutely massive in Asia. It is absolutely big in Korea, in China as well. Like, you know, we rented out Oracle Arena, I think, or like, the event I went to was an Oracle Arena. My sense is in Asia, they rent out like the entire stadiums, like the soccer stadiums, and the players are already like celebrities. So anyways, you know, as like, you know, the, I know like Korean culture is really making its way into the US as well, I think that's another tailwind for this whole thing. But anyways, esports, I think, is something you should keep an eye out on, because there's

• 45:24Speaker 00

  a lot of room for growth.

• 45:25Apoorv Agrawal?

  Very unexpected.

• 45:26Speaker 00

  Yeah.

• 45:27Apoorv Agrawal?

  Good to hear. Short?

• 45:28Speaker 00

  My short? My short's a little spicy, which is I'm short on the entire category of like tooling around AI, AI products. And so this encapsulates a lot of different things. Kind of cheating because some of these, you know, I think are starting to play out already. But I think like two years ago, it was maybe like evals products or like frameworks or vector stores. I'm pretty short those. I think nowadays, there's a lot of additional excitement around other tooling around AI models. So RL environments, I think, are really big right now as well. Unfortunately, I'm very short on those. Not really, I don't really see a lot of potential there. See a lot of potential in reinforcement learning and applying it.

• 46:11Speaker 00

  But I think the startup space around RL environments, I think, is really tough. Main thing is, one, it's just a very competitive space. There's just a lot of people kind of operating in. And then, two, if the last two years have shown us anything, the space is evolving so quickly and it's so difficult to try and like adapt and make and understand what the exact stack is that will really, you know, carry through to the next generation of models. I think that just makes it very difficult when you're in the tooling space because, you know, today's really hot framework or really hot tool might just not get used in the next generation of models.

• 46:43Speaker 02

  So I've been noticing like the same pattern, which is the teams that build like breakout like startups in AI are extremely pragmatic. Like they are not super like, you know, intellectual, but like the perfect world, etc. And it's funny because I feel like, you know, our generation has basically started in tech in a very like stable, like, you know, moment where, you know, technology had been building up like, you know, for years and years with like SaaS, like clouds, etc. And so we were in a way like raised, like, you know, in that very stable moment where it makes sense at that point to, you know, design like very like, you know, good like abstractions and toolings because, you know, you have a sense where it's going. But it's so different today. Like, no one quite knows what's going to happen next year or two. So it's almost impossible like to define like the perfect tooling platform.

• 47:27Apoorv Agrawal?

  Right, right, right, right. Well, that's, there's a lot of that going around right now.

• 47:31Speaker 00

  Yes.

• 47:33Apoorv Agrawal?

  Spicy, a lot of homework there. Olivia, over to you, sir.

• 47:36Speaker 02

  Long short? Long short? I've been thinking a lot about education for the past month in the context of kids.

• 47:45Speaker 02

  I'm pretty short on any education which basically emphasizes a human memorization at that point. And I say that having mostly been through the education myself. But you know, like, I learned so much on like, you know, history facts, like, you know, legal things that are. You know, some of it like does shape your way of thinking. A lot of it frankly is just like, you know, knowledge tokens essentially. And those knowledge tokens, you know, turns out like, you know, LLMs are pretty good at it. So I'm I'm quite short on that.

• 48:15Apoorv Agrawal?

  That's right. You won't need memory when strategy is bionic. You can just think about it straight into your head.

• 48:19Speaker 02

  Exactly, exactly. What am I long at?

• 48:24Speaker 02

  Frankly, I think healthcare is probably the industry that will benefit the most from AI in the next like year or two.

• 48:33Apoorv Agrawal?

  Oh, say more.

• 48:34Speaker 02

  I think like all the ingredients are here for a perfect storm. A huge amount of like structure and structured data, you know, it's basically the heart of, you know, like the the pharma companies. The moles are excellent at digesting, processing that kind of data. A huge amount of like admin, like, you know, heavy, like documents, heavy, like, you know, culture. But at the same time, like companies which are very technical, very R and D friendly. Like, you know, companies like, you know, who's like sort of technology in a way is at the heart of what they do. And so, yeah, I'm pretty bullish on the.

• 49:08Apoorv Agrawal?

  This is like life sciences. So you mean life sciences, research organizations that are producing drugs? Gotcha.

• 49:13Speaker 02

  Exactly, yeah.

• 49:14Speaker 00

  Yeah, it's almost like, you know, over the last twenty, thirty years, these these like pharma or like biotech companies have have basically, if you look at the work that they're doing, like only a small amount of it is is actual research. And so much of it ends up being admin and like, you know, documents and and things like that. And that area is just so ripe for, you know, something to happen with AI. And I think that's what we're seeing with Amgen and some of these other customers.

• 49:36Speaker 02

  Exactly.

• 49:37Speaker 00

  And it's also like not what they want to do. I think it's good that we have some regulations there, obviously, but like just means that they have like reams and reams of things that kind of go through. And so. You know, like, when you have a technology that's able to really help like bring down the the cost of something like that, I think it'll just, you know, tear right through it.

• 49:53Speaker 02

  And I think once governments and like, you know, institutions are going to realize that, like, if you step back, like, it is probably one of the biggest battle like to like human progress, right? You step back in the past decade, like, you know, how many like true like breakthrough drugs have there been? Like, you know, not that many, like, you know, how life would be different, like, if you double that rate, essentially. So once you realize what it takes, yeah, my hunch is that we're going to see quite a bit of momentum in that space.

• 50:18Apoorv Agrawal?

  Wow. All right. Lots of homework there as well.

• 50:21Speaker 02

  Yeah.

• 50:22Apoorv Agrawal?

  Next one. Favorite underrated AI tool, other than ChatGPT, maybe?

• 50:26Speaker 02

  I love Granola.

• 50:28Speaker 00

  Oh, man, you stole my, you stole my answer.

• 50:30Speaker 02

  I love so much Granola, like.

• 50:32Apoorv Agrawal?

  Two votes for Granola.

• 50:33Speaker 02

  There is something like, yeah.

• 50:34Speaker 00

  Hey, what about ChatGPT Record?

• 50:37Speaker 02

  I like ChatGPT Record as well, but there are some features of Granola which I think are really done well, like the whole, like, you know, integration with your Google Calendar is excellent.

• 50:45Apoorv Agrawal?

  Yeah.

• 50:46Speaker 02

  And just, you know, the quality of like the transcription and like the summary is pretty good.

• 50:51Apoorv Agrawal?

  Do you just have it on? Because I know your calendar is back to back. You just have Granola on.

• 50:54Speaker 02

  So the funny thing is that I don't use Granola internally. I use Granola for my personal life mostly.

• 50:59Apoorv Agrawal?

  I see.

• 51:00Speaker 02

  Yeah.

• 51:01Apoorv Agrawal?

  I see. On dates, I'm joking.

• 51:03Speaker 00

  I was gonna say, yeah, Granola is actually gonna be mine. So two votes for Granola. I was gonna say the easy answer for me is Codex, as a software engineer. It's just like, so, it's gone so good recently. Codex Kly, especially with GPT-5. Especially for me, I tend to be less time sensitive about like, you know, the iteration loop with with coding. And so leaning in a GPT-5 on Codex, I think, has been really, really.

• 51:25Apoorv Agrawal?

  Interesting. What about Codex has changed? Because, you know, Codex has also been through a journey. Codex has been around for a bit. I remember, like, it's been launched for like a year, more than, over a year ago. It's like, what's, what's changed about Codex?

• 51:35Speaker 00

  Yeah, I was actually gonna say.

• 51:36Apoorv Agrawal?

  Codex Kly has been around for a bit.

• 51:37Speaker 00

  I feel like it's been less than a year for Codex.

• 51:40Speaker 02

  A few months, I would say.

• 51:41Speaker 00

  The time dilation is so crazy in this field. It feels like it's been around for a very long time.

• 51:44Speaker 02

  A year ago with GPT-4O, like, you know, I made that demo, like, that feels like ages ago.

• 51:49Speaker 00

  Oh, one hadn't even come out yet. 12 Days of Christmas hadn't happened yet.

• 51:52Speaker 02

  The voice demos.

• 51:53Apoorv Agrawal?

  Maybe it was a naming thing, okay, but anyway, yeah.

• 51:55Speaker 02

  Oh, there was a Codex model.

• 51:56Speaker 00

  That's what I'm thinking about. There was a Codex model?

• 51:58Speaker 02

  We are, yeah, we are. You're not to blame for that confusion.

• 52:01Speaker 00

  Also, I think the GitHub thing was called Codex.

• 52:04Apoorv Agrawal?

  That's right.

• 52:04Speaker 00

  Yes, yes.

• 52:05Apoorv Agrawal?

  That's right.

• 52:05Speaker 00

  But I'm talking about our coding product within ChatGPT, which is the Codex Cloud offering, and then also Codex Kly. So actually, maybe if I were to narrow my answer a little bit more, it's Codex Kly, which I've really, really liked. I like the local environment setup. The thing that's actually made it really useful in the last, I'd say, like, month or so is, one, I think the team has done a really good job of just, like, getting rid of all the paper cuts, like the small product polish and, like, paper cut things. It just, it kind of feels like a joy to use now.

• 52:31Apoorv Agrawal?

  Nice.

• 52:31Speaker 00

  I feel more reactive. And then the second thing, honestly, is GPT-5. Like, I just think GPT-5 really allows the product to shine. It's, you know, at the end of the day, this is kind of a, this is a product that really is dependent on the underlying model. And when you have to, you know, like, iterate and go back and forth with the model, like, four or five times to get it right, to get it, to, like, you know, do the change that you want, versus having it think a little bit longer and it just, like, one-shots and does exactly what you want to do. You get this, like, weird, like, bionic feeling where you're like, I feel so mind melded with the model right now and, like, perfectly understands what I'm doing. And so getting, like, that kind of like dopamine hit and, like, feedback loop constantly with Codex has made it kind of like an indispensable thing that I really, really like.

• 53:11Apoorv Agrawal?

  Nice.

• 53:12Speaker 00

  And the other thing I'd say Codex is just really good for me is, so I use it in my, for, like, personal projects. I also use it to, like, help me understand codebases, like, as a, as a engineering manager, now I'm not as in the weeds on the actual code. And so you're actually able to use Codex to really understand what's happening with the codebase, have it, like, ask questions and have an answer about things and really catch up to speed on things as well. So, like, even the non-coding use cases are really useful with Codex Kly.

• 53:39Apoorv Agrawal?

  Fascinating. Sam had this tweet about Codex. Usage ripping, I think like yesterday. So I wonder what's going on there, but you're not alone.

• 53:50Speaker 00

  Yeah, I think I'm not alone, just judging from the Twitter feedback. I think people are really realizing how great of a combination Codex祥andGPT-5 are. Yeah, I know that team is undergoing a lot of scaling challenges, but I mean, the system hasn't gone down for me, so props to them. But we are in a GPU crunch, so we'll see how, you know, how long that goes.

• 54:08Apoorv Agrawal?

  Awesome, awesome. All right, the next one. Will there be more software engineers in 10 years or less?

• 54:17Apoorv Agrawal?

  There's about what 40, 50 million?

• 54:18Speaker 02

  Full-time professional software engineers? That's what you mean, like full-time, like actual jobs?

• 54:23Speaker 00

  Yeah, because it's a hard one, because like I think without a doubt, there's going to be a lot more software engineering going on.

• 54:28Speaker 02

  Yes, of course.

• 54:29Speaker 00

  There's actually a really great post that was shared, I think in our internal Slack. It was like a Reddit post recently, I actually think that highlights this. It was a really touching story, it was a Reddit post about someone who has a brother who's non-verbal. I actually don't know if you saw this, it was just posted. It's a person on Reddit posted, they have a non-verbal brother who they have to take care of. The brother, like they tried all these types of things to help the brother interact with the world, use computers, but like vision tracking didn't work because I think his vision wasn't good. All the tools didn't work. And then this brother ended up using ChatGPT. I don't think he used Codex, but he used ChatGPT and basically taught himself how to create a set of tools that were tailor-made to his non-verbal brother. Basically, a custom software application just for them. And because of that, he now has this like custom setup that was written by his brother

• 55:15Speaker 00

  and allows him to like browse the internet. I think the video was like him watching The Simpsons or something like that, which is really touching. But I think that's actually what we'll see a lot more of, like this guy's not a professional software engineer. His title is not software engineer, but he like did a lot of software engineering, probably pretty good, good enough definitely for his brother to use. So the amount of code, the amount of like building that'll happen, I think is just going to go through an incredible transformation.

• 55:38Apoorv Agrawal?

  Right.

• 55:38Speaker 00

  I'm not sure what that means for software engineers like myself. Maybe there's, you know, equivalent or maybe there's even more.

• 55:43Speaker 02

  Of course, more shall we?

• 55:44Speaker 00

  Yeah, more of me.

• 55:45Speaker 02

  We need way more of you.

• 55:48Speaker 00

  But definitely a lot more software engineering and a lot of code.

• 55:50Speaker 02

  I buy that completely. I buy completely the thesis that there is a massive software shortage. Like in the world, like we've been sort of accepting it, you know, for the past 20 years. But like the goal of software was never to be that super rigid, super hard to build, you know, artifact. It was to be like, you know, customized, like malleable. And so I expect that we'll see like way more, a sort of a reconfiguration of like people's like, you know, jobs and skill set. Where way more people code, like, you know, I expect that product managers are going to code like more and more, for instance.

• 56:23Speaker 00

  Yeah, you made your PMs code recently, if you remember.

• 56:25Speaker 02

  Oh yeah, we did that. That was really fun. We started like essentially not doing like PRDs, like product requirements documents. You know, classic PM thing, you write like five pages, like my product does X, that are. And you know, PMs have been basically vibe coding prototypes. And one, it's pretty fast with GPT-5 and like Codex.

• 56:44Speaker 00

  Yeah, just a couple hours, I think.

• 56:45Speaker 02

  Freaking fast.

• 56:46Speaker 00

  Yeah.

• 56:46Speaker 02

  And second, like it sort of conveys like so much more information than a document. Like you get a feel essentially for the feature, like is it right or not? So yeah, I expect that sort of behavior, we're going to see more and more.

• 56:59Apoorv Agrawal?

  Yeah. Instead of writing English, you can actually now write the actual thing you want.

• 57:02Speaker 00

  Yeah, yeah.

• 57:03Speaker 02

  Yeah.

• 57:03Apoorv Agrawal?

  And yeah, that's amazing. Advice for high school students who are just starting out their career?

• 57:09Speaker 02

  My advice is, I don't know, maybe it's evergreen, like prioritize critical thinking above anything else. If you go in a field which requires like extremely high critical thinking, like, you know, skills, I don't know, math, physics, or you know, maybe philosophies in that bucket. You will be fine regardless. If you go in a field that sort of turns down that thing. And again, gets back to like memorization, like, you know, pattern matching, I think you will probably be less future-proof as a, you know.

• 57:40Apoorv Agrawal?

  What's a good way to sharpen critical thinking?

• 57:43Speaker 00

  Use ChatGPT and have it test you.

• 57:47Apoorv Agrawal?

  That's actually.

• 57:49Speaker 02

  Having like, you know, world-class tutor who essentially knows how to put the bar like 20% about what you can do all the time, you know, is actually probably like a really good way to do it.

• 58:02Apoorv Agrawal?

  Yeah. Nice. Anything from you, sir?

• 58:06Speaker 00

  Mine is, I think it's just, I think we're actually in such an interesting, like, unique time period where the like younger, so like, maybe this is a more general advice for not just like high school students, but just like the younger generation, maybe even like college students. It's like, I think the advice would be, don't underestimate how much of an advantage you have relative to the rest of the world right now because of how AI-native you might be or how like, you know, in the weeds of the tools you are. My hunch is like, high schoolers, college students, when they come into the workplace, they're gonna have actually a huge leg up on how to use AI tools, how to actually transform the workplace. And my push for like some of the younger, I guess, high school students is like, one, like, just really immerse yourself in this thing. And then two, just

• 58:49Speaker 00

  like, really take advantage of the fact that you're, you're in a unique time where like, no one else in the workforce really understands these tools as deeply, probably as you do. A good example of this is actually, we had our first intern class recently at OpenAI, a lot of software interns. And some of them were just like the most incredible cursor-power users I've like ever seen.

• 59:08Speaker 02

  They were so productive. I was shocked in a good way.

• 59:12Speaker 00

  Yeah. I was like, yeah, I know, I know we can get good interns, but like, I don't know they'd be like this good. And I think part of it is like, they've grown up using these tools, for better or worse, in college. But I think the meta-level point is they're, they're so like AI-native. And even like, I don't know, me and Olivier, we're like kind of AI-native. We work at OpenAI, but like, we haven't like been steeped in this and kind of grown up in this. And so, the advice here would just be like, yeah, leverage that, like, you know, don't be afraid to kind of like, go in and spread this knowledge and take advantage of it in the workplace, because it is a pretty big advantage for them.

• 59:43Apoorv Agrawal?

  Yeah, I can't remember who said this to us at Bouncer. But every intern class was just getting faster, smarter, like laptops, like smarter every generation.

• 59:51Speaker 00

  You sure didn't peak in 2013 when I was an intern.

• 59:54Apoorv Agrawal?

  That's right.

• 59:55Speaker 02

  He's a real spy.

• 59:57Speaker 00

  That's summer 2013, yeah.

• 59:59Speaker 02

  Two guys like you.

• 1:00:00Apoorv Agrawal?

  That's right. That's right, that's right, that's right. Well, lots happened, you know, lots happened since you guys joined OpenAI, right? What, three years and almost three years? In your OpenAI journey, what has been the rose moment, your favorite moment? The bud moment where you're like most excited about something, but still opportunity ahead? And the thorn, toughest moment of your three-year journey?

• 1:00:23Speaker 02

  The thorn is easy for me. What we call the blip, which is, you know, the coup of the board. Like, that was a really tough moment. It's funny, because, you know, after the fact, it actually reunited quite a bit the company. Like, there was a feeling, OpenAI had a pretty strong culture before, but, you know, there was a feeling of camaraderie essentially that was even stronger. But, you know, sure, like tough on the day off.

• 1:00:44Apoorv Agrawal?

  It's very rare to see that anti-fragility. Most orgs after something like that break, break apart. But I feel like OpenAI got stronger, OpenAI came back.

• 1:00:53Speaker 02

  It's a good point. I feel it made OpenAI stronger for real now, essentially, when I look at the other fact. When I look at, you know, other, like, you know, news, like departures, or, you know, whatever, like, you know, bad news essentially, I feel the company has built, like, you know, a thicker skin and, you know, an ability to like recover like way quicker.

• 1:01:12Speaker 00

  I actually think, I think it's definitely right. Part of it, too, I think, is also just the culture. I also think this is why it was such a low point for a lot of people. So many people just at OpenAI care so deeply about what we're doing, which is why they work so hard. You just care a lot about the working, almost feels like your life's work, like it's a very audacious issue and thing that you're doing. Which is why I think the blip was like so tough on a lot of people. But also is what I think helped bring people back together and why we were able to hold together and get that thick skin as well.

• 1:01:38Apoorv Agrawal?

  Yeah.

• 1:01:39Speaker 00

  Yeah. I have a separate worst moment, which was the big outage that we had in December last year, if you remember. Yeah, you remember.

• 1:01:47Apoorv Agrawal?

  I do.

• 1:01:48Speaker 00

  It was like a multi-hour outage, really highlights to us how essential of almost like a utility the API was. So the background is, I think, we had like a three, four-hour outage sometime in... November or December last year. Really brutal, pure self zero, no one could hit ChatGPT, no one could hit the APIs. It was really rough. That was just really tough, just from a like, you know, customer trust perspective. I remember we like talked to a lot of our customers, we kind of like

• 1:02:17Speaker 00

  post mortem them on what happened and kind of our plan moving forward. Thankfully, we haven't had anything close to that since then. And I've been actually really happy with all the investments we've made in reliability over the last six months.

• 1:02:31Speaker 00

  But in that moment, I think it was really, it was really tough.

• 1:02:34Speaker 02

  Yeah. On the happy side, like on the roses,

• 1:02:38Speaker 02

  I think I have two of them. The first one would be GPT five was really good, like the sprint up to GPT five. I think really like, you know, showed like the best of OpenAI. Like, you know, having like cutting edge, like science research, like, you know, extreme like customer focus, like, you know, extreme like, you know, infrastructure and inference, like, you know, talent. And the fact that we were able like to ship like such a big model and scale it to like, you know, many, many, many, many tokens, you know, per minute, like almost like immediately, I think speaks to it. So that one I really.

• 1:03:13Speaker 00

  With no adages.

• 1:03:14Speaker 02

  With no adages.

• 1:03:15Speaker 00

  Yeah, really good reliability, yep.

• 1:03:17Speaker 02

  Like I remember when we shipped like GPT four turbo like a year ago, a year and a half ago, we were terrified by, you know, like the influx of traffic. And I feel we've really gotten like much better at, you know, shipping like those, you know, massive updates.

• 1:03:31Speaker 02

  The second rose, like, you know, happy moment for me would be the first Dev Day was really fun. It felt like a coming of age, like OpenAI, like, you know, we are embracing that, you know, we have like a huge community developers. You know, we are going to ship models, new products. And I remember basically seeing like, you know, all my favorite people, OpenAI or not, you know, like in the room essentially nerding out on, you know, what are you building, like, you know, what's coming up next. It felt really like, you know, a special moment in time.

• 1:03:59Speaker 00

  That was actually gonna be mine as well, so I'll just pay you back off of that, which is the very first Dev Day 2023, November. I actually, I remember it, so I mean, obviously, a lot of good things have happened since then. There's just a very, I don't know why, for me, it was a very memorable moment, which was, one, it was actually quite a rush up to Dev Day, we shipped a lot. So our team was just really, really sprinting, so it was like this high, you know, high-stress environment kind of going up. To add to that, you know, of course, because we're OpenAI, we did a live demo on Sam's, in Sam's keynote of all the stuff that we shipped. And I just remember being in the back of the audience, sitting with like the team, and like waiting for the demo to happen. And once it finished happening, we all just like let out a huge sigh of relief. We were like, "Oh my God, thank you."

• 1:04:42Speaker 00

  And so there's, I think there's just like a lot of like, you know, buildup to it. For me, the most memorable thing was I remember right after Dev Day, all the demos worked well, all the talks worked well, we had the after-party, and then I was just in away mode, driving home at night with the music playing. It was just like such a great end to the Dev Day. That was, that was what I remember. That was my rose for the last, last year.

• 1:05:00Apoorv Agrawal?

  Love it. That's awesome. I assume you guys are, but please tell me if you're AGI-pilled, yes or no? And if so, what was the moment that got you there? What was your aha moment? When did you feel the AGI?

• 1:05:11Speaker 02

  I think I made J-Pilled? I think I made J-Pilled.

• 1:05:14Speaker 00

  You're definitely AGI-pilled.

• 1:05:15Apoorv Agrawal?

  I am, okay.

• 1:05:19Speaker 02

  I've had a couple of them. The first one was the realization in 2023 that I would never need to code manually like ever, ever again. Again, I'm not the best coder, frankly, you know, I chose my job like, you know, for a reason. But realizing that, you know, what I thought was a given, that we humans would have like to write like basically machine language like forever, is actually not a given. And that, you know, the pace of pace is huge, feeling the AGI. The second like feel the AGI moment for me was maybe the progress on voice and multimodality. Like, you know, text, like at some point, you get used to it, like, okay, you know, the machine can write pretty good text.

• 1:05:59Apoorv Agrawal?

  Yeah, voice makes it real.

• 1:06:01Speaker 02

  But once you start actually talking, like, you know, to something that really understands your tone, like, you know, understand my accent, like in French, it felt like sort of a truly moment, like, okay, machines are going beyond like cold, mechanical, deterministic, like, you know, like logic, to something like much more like emotional and like, you know, tangible.

• 1:06:25Apoorv Agrawal?

  Yeah, that's a great one.

• 1:06:27Speaker 00

  Yeah, mine are. So I do think I am AGI-pilled. I probably gradually became AGI-pilled over the last couple of years. I think there are two, and for me, yeah, I think I actually get more shocked from the text models. I know the multimodal ones are really great as well. For me, I think they actually line up with two, like, general breakthroughs. So the first one was right when I joined the company in September 2022. It was pre-ChatGPT.

• 1:06:54Apoorv Agrawal?

  Yeah, two months ago.

• 1:06:55Speaker 00

  But at the time, GPT-4 already existed internally. And I think we were trying to figure out how to deploy. I think Nick Turley's talked about this a lot early days of ChatGPT. But it was the first time I talked to GPT-4. And it was like going from nothing to GPT-4 was just the most mind-blowing experience for me. I think for the rest of the world, maybe going from nothing to GPT-3.5 in chat was maybe the big one, and then going from 3.5 to 4. But for me, and I think for a lot of, maybe some other people who joined around that time, going from nothing to, or not nothing, but like what was publicly available at the time, going from that to GPT-4 was just incredible. Like, I just remember asking, throwing so many things at it. I was like, there's no way this thing is gonna be able to give an intelligible answer. And it just like knocks it out of the park. It was absolutely incredible.

• 1:07:37Speaker 02

  GPT-4 was insane. I remember GPT-4 came out when I was interviewing with OpenAI. And I was still working on the fan. Should I just join? I saw that thing, I was like, okay. I mean, I mean, guys, there is no way I can work on anything else at that point.

• 1:07:50Speaker 00

  Yeah, yeah, yeah. Yeah, so GPT, yeah, GPT-4 was just crazy. And then the other one was, like, is the other breakthrough, which is like the reasoning paradigm. I actually think the purest representation of that for me was deep research and throwing like, like, asking it to like really look up things that I didn't think it would be able to know and seeing it like think through all of it, like be really persistent with the search, get really detailed with the write-up and all of that. That was pretty crazy. I don't remember the exact query that I threw it, but I just remember like, I feel like the field AGI moments for me are like, I'll throw something at the model that I was like, there's no way this thing will be able to get. And then it just like knocks it out of the park. Like, that is kind of the field AGI moment. I definitely had that with deep research with some of the things I was asking it.

• 1:08:31Apoorv Agrawal?

  Yeah. Well, this has been great. Thank you so much, folks. You guys are building the future. You guys are inspiring us every day and appreciate the conversation.

• 1:08:42Speaker 00

  Yeah, thank you so much.

• 1:08:42Apoorv Agrawal?

  Thank you.

• 1:08:53Apoorv Agrawal?

  As a reminder to everybody, just our opinions, not investment advice.