Ep 59: OpenAI Product & Eng Leads Nikunj Handa and Steve Coffey on OpenAI’s New Agent Development Tools

Two weeks ago, OpenAI released its set of tools to help developers build agentic systems. Today on Unsupervised Learning, Nikunj Handa (Product Lead) and Steve Coffey (Eng Lead) answer some of the biggest questions around how developers should be thinking about building in the agentic paradigm in 2025.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  I'm Jacob Afaron and today on Unsupervised Learning, we had a really wide-ranging discussion. We talked about how developers should think about where these agents do and don't work, as well as computer use models and how those are being used. We talked about how enterprises should be building for the agentic future, as well as what will differentiate application builders who are building on top of these models. And we hit on AI infrastructure, what the needs are still for developers, and where there's still room for startups to compete. This was a ton of fun to do this right after a really compelling release from OpenAI. I think folks will really enjoy this. Without further ado, here's our episode.

• 0:31Jacob Effron?

  Well, thank you both so much for coming on the podcast. Really appreciate it.

• 0:33Speaker 01

  Yeah, good to be here.

• 0:34Speaker 02

  Awesome to be here.

• 0:35Jacob Effron?

  Yeah, I mean, congratulations. Never a dull moment at OpenAI, but I feel like the last month has been like even crazy by your standards, the amount you've shipped.

• 0:41Speaker 02

  Yeah.

• 0:42Speaker 01

  Yeah, it's been quite a journey, hasn't it?

• 0:47Jacob Effron?

  I can certainly imagine. Well, I feel like there's a ton of things we'll want to dig into in all the stuff you've released lately. But maybe to start just at the highest level, I'd love to hear kind of your long-term vision for how we as consumers will like interact with agents in like, you know, the next five, 10 years.

• 1:00Speaker 02

  Yeah, I mean, right now we see it all happening in services like ChatGPT. You've got deep research, you've got operator, people are like specifically going to the spot. I think the most exciting thing about releasing models and APIs that are underlying these agentic products is that we're going to see them in more and more products across the web. So computer use coming to, you know, a browser that you like to use or operator automating like a task that you do day to day at work and doing all the clicking and filling out forms and all the research for you. I think it's just going to become more and more deeply embedded into products that you use today, day to day. And that's what we're most excited about, at least in the API platform,

• 1:44Speaker 02

  is to like just like disperse this thing and have it be everywhere.

• 1:47Speaker 01

  Yeah, I think one of the cool things about working on the API platform is you actually don't know what people are going to want to build. It's very verticalized, right? So like on ChatGPT, you first party, we kind of have an idea of what people will want to do. But in the API, it's just like they, people know their domains way better than we ever could, right? And so it'll be really interesting to see, yeah, how these products like and these model capabilities make their way into verticals.

• 2:06Jacob Effron?

  Yeah, is there a particular like agent that you'd be like, you're like waiting for? You're like, God, I just can't wait until I have, you know? Everyone always likes a travel agent for some reason. I don't know if there's, there's one that's top of mind for you guys.

• 2:18Speaker 02

  My my top one is an API designing agent.

• 2:22Speaker 01

  Yeah.

• 2:22Speaker 02

  The amount of time like Steve and I.

• 2:25Speaker 01

  Back and forth.

• 2:25Speaker 02

  Yeah, we're just like going through every single parameter name that we can think of.

• 2:29Speaker 01

  Should it be param config or config param?

• 2:32Speaker 02

  Yeah.

• 2:34Speaker 02

  Yeah, that would be amazing. We have like some deep research like thing that looks at like the best API design things and it gets really.

• 2:41Speaker 01

  Yeah, we could like just be fine tuning it on all the APIs that we really like. That's actually a really good idea.

• 2:46Jacob Effron?

  Yeah, I hope someone takes your API and then gives you a product back that.

• 2:49Speaker 01

  That'd be great. I would love that.

• 2:50Jacob Effron?

  That's the dream, right?

• 2:51Speaker 01

  Yeah, exactly.

• 2:52Jacob Effron?

  I guess, you know, a question a lot of folks are asking is, you know, we're obviously in the very early innings of these agents and they're kind of accessing the web and communicating with each other in ways that have been built for, you know, this previous paradigm. There's all sorts of futuristic ways that that folks think about how these agents may access the web and kind of communicate with each other. There's even that viral YC demo where agent realizes talking to an agent and they switch to something that's easier to exchange information. Like, how do you guys think about how this evolves? And obviously, I'm sure the developers will take you in all sorts of directions, but any early inklings of of how this might work?

• 3:20Speaker 02

  Yeah, for sure. I think on agents communicating or getting information from the web, we've already seen a big change. We've gone from this world where an agent would do a single turn, decide whether it wants to search the web or not, get information from the web and synthesize a response. That was like what I think 2024 was about. 2025 is already about products like deep research where the model is like getting information from the web, thinking about what it got, reconsidering its stance, getting something else from the web, opening multiple web pages in parallel to like try to try to save time.

• 3:56Speaker 02

  And this whole like chain of thought tool calling or calling tools in the reasoning process is like a significant shift in terms of like how agents access information from the web. And you can totally imagine some of these webpage extraction details being replaced by other agents in the near future, where I don't even know if like this agent needs to know that it's talking to an AI agent on the other end. It's just like an endpoint that it calls and it's like, oh, it got some very useful information that it uses to make its decision or backtrack or do something completely different. So, yeah, I think it's all going to be pretty seamlessly embedded in this like chain of thought process where tool

• 4:38Speaker 02

  calling is just happening between both the internet and your your private data and your private agents. So that's where I see it going like pretty much like in the coming months.

• 4:49Jacob Effron?

  Is this something that you think companies like should be, because obviously one version of the world is they can, you know, just wait for agents to start accessing their, you know, sites. Another version is like they should be building actively toward this and like, you know, create the agent themselves that makes it easier for a consumer agent to hit it. Like, how should folks that are, you know, running these, you know, running product at some of these companies be thinking about this?

• 5:07Speaker 02

  I think the developers are already doing this. So we put out the agent SDK for this very reason, because people are creating these multi-agent, like swarms of multiple agents to solve these business problems. So if you look at like a customer support automation problem, you have one agent that's like looking after your refunds, another that's sort of looking after billing and shipping information, something else that sort of makes a decision on like pulling the FAQ or escalating to a human. And so we already see this like multi-agent architecture be very popular. And we want to make it much easier for developers to to build on it. And that's why we built the agent SDK. Now,

• 5:48Speaker 02

  when do you start exposing these agents to the public internet and how that becomes useful is is going to be very interesting. I don't think we've seen too much of that, but it makes so much sense that that will happen at some point. And my advice to companies or and products would be like, just like build these AI agents internally to to solve real problems that your company is facing today. And whenever it's sort of like becomes apparent that like exposing this to the internet for someone else to communicate with you makes sense, you know, that that'll just happen. And I don't think we're too far from it, but yeah, that I think it'll just happen in the coming months.

• 6:27Speaker 01

  Yeah, totally. I think what's really interesting too is that like, you know, before we've seen most of the data that a model is seeing is either your own data, chat history, file search. And I think like what's really interesting, especially with these tools that are much more connected to the web is that we'll see a lot more model, a lot more data going into the model that's actually from around the web and not just data that you're providing, which is really interesting.

• 6:48Jacob Effron?

  As developers are thinking about incorporating, you know, and using these APIs, what heuristics do you guys use for where like agents do and don't work today? And how would you kind of advise folks?

• 6:57Speaker 02

  Let's like take a step, a little bit of a step back. So 2024, what most agentic products looked like was a very like clearly defined workflow with about, you know, less than 10 tools.

• 7:10Jacob Effron?

  Yeah.

• 7:11Speaker 02

  It's like about a dozen tools at most, and this like very well orchestrated go from here to there to there to there. And that's how a lot of companies like built a bunch of really cool coding agents, built a bunch of really cool customer support automation projects, deep research projects, et cetera.

• 7:28Speaker 02

  In 2025, we've gone to this model where everything is happening in this chain of thought. It's like the model is in its reasoning process is clearly smart enough to figure out how it should call multiple tools and then also like figure out that it's going down the wrong path, take a U-turn and then try something else. And I think you've gone away from the whole deterministic workflow building process. And OpenAI has like been working on on tools like reinforcement, fine tuning, et cetera, to like make this something that developers can use themselves. And I think the next step after this is going to be, how can you get rid of that, you know, 10 to 15 tool constraint that you have? Like, how could you just expose this thing to hundreds of tools?

• 8:12Speaker 02

  Have it like figure out which is the right one to call and then make use of, make use of those tools. I think that's like really the next unlock. And then this thing becomes like, it's like, it has all the superpower it needs. It has the compute, it has the way of like, like reasoning about different tool trajectories, and it has access to a lot of tools. So that's what I'm like really excited about in the coming months is removing the number of tools constraint. But yeah, it's kind of hard to make that work with today's models, but I think that's going to change.

• 8:43Speaker 01

  Yeah. Yeah, I think also just like increasing the available runtime that these models have to go off and like do what they need to do. I mean, if you're a human, you can go off and like work on something for a day and use as many tools as you need to get the job done. And I think now like we've seen runtimes that for models, especially like deep research that are in the minutes. But you know, being able to get these things to go into the hours and into the days is going to yield some really powerful results.

• 9:05Jacob Effron?

  Last year, you had to put like such specific guardrails and chain things so closely together because you couldn't let things go off the rails. It seems like now you, you know, you're even more flexible in what you can allow. And then obviously the dream is just like, yeah, go off here. It's like the hundreds of tools that you could use across every task, you know, go, go figure it out.

• 9:20Speaker 01

  Yeah, exactly.

• 9:20Speaker 02

  Totally. I, I think it's not like, let's see how the, the next generation of models generalize to like all of the use cases that developers are going to have. There's also this sort of reinforcement fine tuning technique where you're creating these tasks and graders. And if developers can create their, their own tasks and graders and get the model to find the right path, the right tool calling path to solving a particular problem that's very unique to that developer's domain. That would be amazing. So I'm really excited about the next series of models that are going to come out. And like our early like results from reinforcement fine tuning. All that comes together to make like agents that are actually very useful.

• 10:02Speaker 01

  Yeah. The really cool thing about that is you're really steering the model in its chain of thought. And you're kind of teaching it how to think about your domain, which is just like a really powerful kind of mental model. When you think about it, you're like, okay, how do I think about like, how do you, how do you basically like train a model to be like a legal scholar essentially? Or train it to be like a medical doctor or something like this, really like training the way that it thinks in the same way that, you know, four years of university would train you to think in a specific way. So like, I think the reinforcement fine tuning thing is a great example of like. Where you're going to see like a really interesting verticalization.

• 10:32Jacob Effron?

  Yeah.

• 10:32Speaker 01

  For these models, yeah.

• 10:33Jacob Effron?

  And for that, I mean, how have you thought about, I feel like one of the classic problems or people talk about there is, you know, you can provide. I'm sure folks want something like off the shelf that makes it easy to do the kind of like grading and evaluation. And at the same time, like some of these domains are so hyper-specific in their own, you know, problems. How have you thought about like the infrastructure level, the right level of tooling to provide to folks that are doing that kind of fine tuning in like a domain like legal or healthcare?

• 10:54Speaker 01

  Yeah, I'd say it's like still a work in progress, you know, where like, I think right now the things we're exposing is basically. We're allow it, we're kind of giving developers a way to build their own graders. So for example, if you are, if you have an eval that you show does 50% on a medical task, right? You can build these graders, let's say, that can cross-reference a model's chain of thought or something else that it's outputting based on. Like a, some sort of like known ground truth, like a medical textbook or something like this, right? And so over the course of fine tuning, you can sort of like steer the model in those, in the direction to be able to produce like better and better outputs. And just, yeah, just kind of be able to just steer it in that way.

• 11:37Speaker 01

  So we're kind of providing the, the basic building blocks, really mostly just like these, these really flexible graders that allow you to like. Take a model output and then grade it against some sort of ground truth or execute some sort of code to prove like, oh yeah, this is mathematically correct. We're not just, you know, checking that this string equals this string, right? Where there's actually some like mathematical correctness to it.

• 11:57Jacob Effron?

  Yeah, I mean, it feels like the biggest question across the board in so many aspects of AI right now is like, what actually can be graded? I mean, I feel like it's the big question and like test on compute and what you can scale like. And obviously I think, you know, if you take healthcare and law, for example, you know, one critique of, of some of these evals is like, well, cool. Like, you know, being a lawyer is not passing the bar, like being a doctor is not like passing these medical exams. Anything like you've seen folks on the ground doing that you feel like is a creative way to actually like best use this type of approach?

• 12:21Speaker 02

  Honestly, like after having talked to the folks who are, who've built things, things around operator and deep research internally, like it's. It's pretty challenging right now to do this stuff and takes a lot of iteration. I don't think I've seen anything out there that's like productized.

• 12:40Speaker 02

  Grading and task generation in a way that just like nails it for your domain. I think this is like the biggest problem to be solved this year. And if not, like it might even go into like next year. The technique is going to come out, but how are you going to actually build really good tasks and graders? Is something that's going to be pretty challenging. Yeah, I I know it's possible now. These these products exist, so you like know that like it's possible to build something like deep research. There's been some replications of that around the internet as well. So you like have enough proof over here. It's just about how do you productize it so that like almost anyone can make use of it? That's going to be hard.

• 13:19Jacob Effron?

  What about um computer use? Like, how do you classify for developers today? Like, you know, how they should think about using that, where it works, where it doesn't?

• 13:25Speaker 02

  Computers, well, like surprisingly, a lot of like cool use cases. You know, initially, we thought that a lot of computer use use cases will be around legacy applications that don't have APIs. And people have been trying to like automate this thing for ages and they haven't been able to. And that's definitely the case. Like, we we definitely have had a couple of customers try it out in sort of the medical domain, where they have these like super manual tasks that people are just like clicking through across three or four different applications to to to do things. And that works really well. But you've also seen, we've also seen examples of companies that are using it to do like research on Google Maps.

• 14:05Speaker 02

  So I think Unify GTM is one of the companies that had used it earlier during our alpha phase. And they basically, they would have like climate tech startups ask questions like, has this company expanded its charging network? And so what the agent would do is like open up Google Maps, turn on Street View and like go to places and see whether they're like more chargers or not.

• 14:33Speaker 02

  And like, I'm like, okay, Google Maps does have an API. I actually don't know if Street View has an API, but it's probably really hard to like figure out which exact location and which direction to look at maybe. And so the all of these like. You can pretty much automate anything. It's kind of cool, right? So you could start there and then you could maybe think about API approach after that.

• 14:54Speaker 01

  Totally. I mean, there's a whole, like, many, many domains, like, don't map to JSON, right? Like, you can't serve them over the web in plain text. So like, these kind of use cases where you need some sort of a combination between, like, vision and text ingestion, like, I think are really, really well-suited for Kuah. Yeah, that's a really interesting example. I didn't.

• 15:10Jacob Effron?

  Yeah. Yeah, the unified use case is fascinating.

• 15:11Speaker 01

  That's really cool.

• 15:12Jacob Effron?

  I was struck by, obviously, you really, you had a bunch of alpha testers and whatnot. And so you released this and then the next day, like, I feel like every big company was like, this is an awesome thing we built with, like, this API.

• 15:20Speaker 01

  Right.

• 15:20Jacob Effron?

  Any particular, you know, even just in the, in the week or so since it's been out, like, any particular favorites that, like, you didn't expect or kind of cool ways people have been using these?

• 15:27Speaker 02

  Oh, that's a good one. Uh, post-alpha, let's think. Well, the computer use ones are the coolest. I think you, you have, I, I really am excited about the, the platform players on computer use as well. Like, if you think about the other tools that we have. So we have web search, we have file search, and we have computer use. Web search, you have a bunch of companies that provide APIs for people to be able to get data from the web, put it into the model's context. File search is, like, pretty mature. Honestly, you have the vector database industry. And on computer use, like, I think things are super early. Um, the main thing people want to do, or businesses want to do, is, like, take these, uh, like, Docker containers or these, uh, VMs in the cloud and then, like,

• 16:11Speaker 02

  put their software in it, put their authentication into it so that they can go and automate things. And there are a couple of really cool ones. There's Browserbase that, that provides this, like, service. There's, like, a YC startup called Scrappybara that has, I think, one of the better developer experiences around, um, making computer use models work really well with hosted virtual machines. Uh, and, you know, I'm, I'm like a developer platform person. So for me, it's like looking at those platform plays and like, all right, like, what's the thing that people are going to build on top of that is, is very exciting. And so, yeah, I'd say, like, those are my top two, um, Browserbase and Scrappybara. I'm pretty excited to see what they do.

• 16:50Speaker 01

  Yeah. I thought, I thought too, Arc was doing some pretty cool stuff. Like, they were, they were basically building a, a tool where they We're going to have, basically you could just like open a tab and give it an instruction and then it kind of goes and does something in the background. I think it's it's very much an operator like use case, but it's really baked into the product. I mean, it's just a web browser, you you're using it, right? It's not like necessarily like baked into a tab in your web browser. It's really just like part of the browser itself, but that was that sort of native integration was really cool.

• 17:14Speaker 02

  Yeah, I think they're calling it D I or something, right?

• 17:17Speaker 01

  Yeah.

• 17:17Speaker 02

  That was there.

• 17:17Jacob Effron?

  Super cool.

• 17:18Speaker 02

  Yeah.

• 17:18Speaker 01

  That's awesome.

• 17:19Jacob Effron?

  Is there anything you kind of noticed so far that like maybe some of the most sophisticated users are doing with the APIs? They're like, God, I wish like we could disseminate this more broadly. If only we were on a podcast and we could tell the world, this is a good way to use some of these things. Anything you've noticed patterns that some of the most sophisticated folks are using?

• 17:34Speaker 02

  Yeah, for for the tools, like it still feels like pretty early. I think during the alpha phase, we definitely found folks who were, you know, they try to get the model and the tool to do the thing that they're they're trying to get it to do. And if that doesn't work, they try a bunch of prompt engineering and then that doesn't work. They make this a step in the workflow. And I think like by going through those like steps, they typically get what they want. It's like, hey, web search, the tool is not giving me exactly what I need. But can I make it part of my workflow where this is just one of the steps that gets information from the web? And then like I pass it on to something else, either deterministic or another L L M step.

• 18:14Speaker 02

  On net, I'd say it's pretty early right now and we're going to discover a lot of this in the coming weeks.

• 18:20Speaker 01

  Yeah, I think one to invert the question a little bit, like one thing that I'm really glad that like we were able to ship is sort of like in the agency S D K. This idea that we're going to sort of split the concerns of what your job is or what your task is across many different agents. It's very much analogous to sort of like the single processor computer versus the multi processor computer, right? Like you just allow each agent to focus on one task and you give it all the context. And then your efficacy on those tasks goes way up, right? Because you're not trying to prompt engineer one agent to do a hundred different things, right? You're kind of just like spreading that across. So I was really glad to see us sort of like. I'm not sure if we invented that paradigm or not. I'm assuming we didn't, but like just to just to like shift that as a really first class pattern. I was really, I thought that was really cool.

• 19:01Jacob Effron?

  Yeah. No, it's interesting because I feel like you alluded to the fact that like, hey, if it's not working, you can kind of just like add it as a step. And I feel like one interesting, you know, quandary that we have on the investing side is like, it feels like, you know, a lot of people, whatever the current capabilities of the model are, they kind of build whatever scaffolding they need to make them work. And sometimes you're like, well, that that gets you the product in the market now and gives you a product that is valuable. At the same time, if you went to a beach and waited three, six months for the models to get better, they may just be able to do it, right? With with your, you know, hundred tools to one thing versus like chaining the steps together. And so, you know, I'm curious like how you think about like, you know, the kind of steps that people are building around the models, like, does that all get obviated over time? Or is like some of that useful?

• 19:41Speaker 02

  I think that this is the most like agent or agent and tool orchestration is like the most important thing right now because my opinion is that the models are much further than where most AI applications are like making use of of things. And like, there's so much value to be extracted from these models that building things around models to make them work really well is an it's an extremely important thing that AI startups should be doing and AI products should be doing.

• 20:14Speaker 02

  Yeah, it's like seen it time and time again where even on customer support automation, which has like been a thing that's been around as a concept for a while, we had a couple of companies like really crack it in late 2023 and 2020, early 2024. And and the adoption has been like kind of slow, you know, like you don't see that many companies move as fast as the first like 10, 15, 20 companies moved. And it just shows like how important it is to be good at orchestrating, to be meticulous about like looking at your traces, figuring out how to prompt engineer, having like an eval set so that your prompt doesn't degrade something else. Like this is so hard today. It's crazy how hard it is.

• 20:58Speaker 01

  Yeah.

• 20:58Speaker 02

  And so I would, I would tell people like, that's the exact thing to be focusing on is how to make these models work.

• 21:04Speaker 01

  Yeah, 100%. And I think too, like, you know, just the, just the idea of splitting up your task among many different agents is like, just makes debugging the whole workflow way easier, right? Because if you have a really capable model and it has 100 instructions and you change a few tokens, right? It might drastically change the outcome of your eval, right? But if you just have one, you know, handoff agent, you have one triage agent, you have one this, like tweaking each one of those becomes a lot more isolated where you're not, you know, the blast radius is much smaller as you're sort of like hill climbing on your eval.

• 21:35Jacob Effron?

  I think when you were on latent space, you mentioned that you like, you know, over time, want to add more knobs to make, you know, things more customizable for developers. What do you think that like, looks like over time? And, you know, how did you think about this kind of tension of providing something that's like relatively easy to use out of the box versus like the ultimate amount of customizability?

• 21:50Speaker 01

  Yeah, totally. I mean, this sort of like idea of APIs as ladders is really something that we took from first principles when we were designing the responses API. And I think it really comes down to, you know, a couple of things, right? Like you want to give a lot of power out of the box, you want to make doing the, the simple thing really easy. And then you want people to be able to get a little bit more reward for every effort, every amount of effort that they put in. And so for us, this looks like a great example, I think, actually is file search, where it's actually really easy to use just out of the box. You upload some documents, you don't even have to do it in the API, you could just do it in the other website. You pop in your vector store ID and it just works, right? And now let's say, okay, well, this actually isn't quite working for my use case. Well, okay, now I actually have knobs to go in and tweak the chunk size, right? The default is 400, maybe I want it to be 200, maybe I want it to be 1,000, right?

• 22:35Speaker 01

  So I have those knobs, they have sensible defaults, right? And so I can go in a little bit deeper and get a little bit more reward for everything I'm putting in. And, you know, it goes way deeper than that in the file search example, right? You have metadata filtering, you have the ability to customize the reranker, right? But this stuff isn't, we don't force you to set all those things right up front, right? We kind of like give those things to you and expose them, they're in the docs, you can find them. But if you're just kicking the tires with the API, you don't want to think about, you're like, what the heck's a reranker, right? You know, so, um, yeah, that's kind of what we think about it is like, make it as simple as possible. I think we actually spent a pretty long time trying to get the quick start for calling the API down to like four lines of code of curl. And we were like really obsessing over that, like it should be this simple.

• 23:17Speaker 01

  But then there's also 50 more prompts that you could set if you want to, and they'll have like reasonable defaults.

• 23:22Jacob Effron?

  Over time, like what other knobs might you want to add?

• 23:26Speaker 01

  It's a good question.

• 23:26Speaker 02

  Oh, yeah. I mean, for tools like for web search, you want to basically add site filtering. That's been a big ask right now. You just have to search the whole internet.

• 23:36Speaker 01

  Yeah.

• 23:36Speaker 02

  Or you can prompt your way into it.

• 23:38Speaker 01

  Specific location too on web search. Where now you can set the city, you can set the country, but like actually setting down to like the block or the even court.

• 23:47Speaker 02

  Which is like super important for for weather, for events, type of queries.

• 23:51Speaker 01

  Especially SF micro climates, right?

• 23:53Jacob Effron?

  Yeah, seriously.

• 23:54Speaker 02

  Yeah. I actually like one of the things we're really excited about doing with responses API is building all the features into it that we had in the assistance API, but not forcing users into it. So we released the assistance API, I think, November twenty twenty three. It had this like full concept of storing your conversations, storing your like model configurations in a in a assistant object, etc. And you found that like the the hill to climb to like get started was was a lot. With responses, we're taking the the other approach where you're starting off with like a single API call and a single endpoint in one concept you have to learn. And then maybe you want to store your conversation with us. So you can opt into using like the equivalent of the threads object.

• 24:38Speaker 02

  And maybe you want to store your model configuration with us. So like you opt into like an assistant type of object. And those things you just like plug in, like it's just one parameter you configure. And that's a knob that you have to to like, you know, have OpenAI host the thing for you. So yeah, I I think that's another set of knobs we like in the short term, we really want to get to.

• 24:59Speaker 01

  Yeah, exactly.

• 25:00Jacob Effron?

  Reflecting back on some of the previous APIs you've released, like obviously these ones are meant largely to supplant those. Like any learnings or like things you like, hey, we got that really right. Or actually, we kind of missed the mark on that and we've kind of fixed it in this current iteration.

• 25:11Speaker 01

  Yeah, totally. I mean, I think the thing that we really got right with the assistance API especially is tool use. You know, that's where we really figured out, like, we saw a ton of usage, especially with the file search tool. That's where the API really found market fit, right? Was people wanting to bring their own data to the API and have the model search over it. But what we got wrong is a lot of the things that that McCoon said, really, it was just too hard to use, right? You didn't have you no way to opt out of the context storage. A lot of people didn't like the context. They wanted more of a chat completions interface where they were able to provide their own context on each turn of the model. But also the chat completions interface is quite limiting, right? The API can only output one thing and the model does many things, right? And so you want it to be able to do a bunch of stuff in the background and then kind of give you the results of all of its all of its thinking and all of its doing.

• 25:56Speaker 01

  And so, you know, we really tried to like, take the best parts of assistance API, sort of the tool use and their like, multiple outputs and all of that stuff and the really the ease of use of chat completions and bring those things together.

• 26:07Jacob Effron?

  Makes a ton of sense. How should developers think about like this kind of suite of developer tools now and like, you know, the MCP landscape?

• 26:13Speaker 02

  Yeah, I think they're like, they're probably solving different problems, right? Like, so the responses API is focused on making these multi-turn interactions with models really good. So a model should be able, we're providing a foundation for the model to be able to call itself multiple times, so have multiple model turns and call tools multiple times, so have multiple tool turns to get to a final answer. So that's like, we've, we've set the building block, which is the responses API. MCP is sort of like how you use tools and bring tools to models. And I think these things are honestly pretty complimentary in some sense.

• 26:59Speaker 02

  And, and we have to figure out what, what, what we do on the tools registry and the tools ecosystem side. But, you know, MCP is super cool and that's, that's, yeah, that's, that's something we have to like figure out in terms of how we bring that to our ecosystem as well.

• 27:14Jacob Effron?

  Yeah, one thing I'm struck by is obviously like, I feel like in the, you know, first years post chat GPT, there was a lot of AI infrastructure companies that popped up that were trying to do, you know, aspects of what you're, you know, you've released now Asian orchestration, vector databases. How do you think about kind of like the opportunity for standalone AI infrastructure companies and like where, you know, where it makes sense for those to exist on top of what you guys are building and where it might not make as much sense?

• 27:36Speaker 02

  Yeah, I think on our, our side, like, you know, we're working with our users and listening to what their asks are. And they want like a one-stop shop for, you know, The things that they want LLMs to do, they want it to be able to search their data and search the internet. And so we've taken a step in that direction. That being said, I feel like the AI infra companies are building like low level, very powerful APIs that are, you know, infinitely flexible. And and there's always going to be like a big market for that kind of stuff. And I think we just got to build the thing that our users are asking for, which are these like more out of the box tools. And we're taking a different approach to this, this whole space.

• 28:20Speaker 02

  But there'll be vertical, specific AI infrastructure companies. I think there's certain companies that build like VMs just for the coding startups, coding AI startups on there. So that they can like test their code and like spin down the VM as quickly as possible. I think they call it run loop or or something. I've heard of them.

• 28:38Speaker 01

  Yeah.

• 28:38Speaker 02

  So it's going to be like verticalized AI infra, which is like, seems like it makes a lot of sense to keep doing that.

• 28:44Speaker 01

  Totally. Yeah. It's like stuff that we're not always going to want to be in the business of doing, right?

• 28:47Speaker 02

  Yeah.

• 28:48Speaker 01

  I think too, there's like a whole class of like LLM ops companies that are like doing some like really interesting things like helping you like manage your prompts and helping you like manage your billing and understanding like where your usage is going. And I think that that sort of stuff is like really cool too. It's not necessarily like low level infrastructure, but it's still stuff that developers care about.

• 29:04Speaker 02

  Yeah, in a multi-model fashion, do multi-provider and all of that.

• 29:07Speaker 01

  Exactly. Yeah, like open router, things like that.

• 29:09Speaker 02

  Yeah.

• 29:09Speaker 01

  Yeah.

• 29:09Jacob Effron?

  Yeah. I mean, and obviously, you know, it sounds you guys spend probably most of your days talking to developers and getting their wishlist. I'm sure you, it sounds like you got a lot of it into this, this current generation APIs, but I'm sure there's always more to do. Like, how do you think about, you were kind of talking about evals earlier as like the problem, but like, how do you kind of think about the stack range problems that are like still left, that are, that make, working with these models painful today? You know, for developers and kind of what some of the most important things to be solved are?

• 29:34Speaker 02

  Yeah, I think like tools, tools is definitely a very big thing for us to figure out. We have the foundational building block. We need to build the tools ecosystem on top of it. There's obviously great work on the MCP side over here and like that is like top of mind for us to to figure out what what we do on that front. We also have.

• 29:55Speaker 02

  You know, like the computer use VM space is pretty early, and I think that that's another big one. Like, how do you get enterprises to like securely and reliably deploy these virtual machines in their own infrastructure? And and observe them and all the things that the computer use models are doing on top of that. I feel like these models, these computer use models are going to get so good so quickly. Because we're just at like the G P D one or two of that of that paradigm. And this thing is going to be incredibly useful. So I'm like very curious to see how the infra on that front takes off.

• 30:33Speaker 01

  I I mean, I think that one of the things that was really interesting to me during the alpha period was like all the different environments that people wanted to try out the computer use tool in. Like, we saw folks, it, the the model works best in a browser environment, right? It's kind of like what it was trained on, but people were trying to use it with iPhone screenshots and Android. And I was like, wow, that's so interesting. I hadn't even thought about doing that. And so I think that like, you know, the sky is going to be the limit on like what people will want like. Or is there going to be a company that just does sort of like iPhone V M's or like sort of like, uh, the, you know, like a, um. There was a company that used to do just like testing frameworks for like I O S, things like that. But just now it's for A I models, like stuff that's like really interesting. Because like, you know, different flavors have been to, you know, all of that stuff. It's really just a huge amount of fragmentation.

• 31:17Speaker 01

  And so it's going to be really interesting to see how the community kind of like steps up to fill the gaps there. Yeah.

• 31:22Speaker 02

  Yeah. Also, like, you also think people doing all like, I think this is a startup trying to do cybersecurity work. So trying to find vulnerabilities in other sites and services using computer use.

• 31:33Speaker 01

  Poke around for thirty minutes.

• 31:34Speaker 02

  Yeah, which is, uh, it's pretty super interesting.

• 31:37Speaker 01

  Yeah.

• 31:38Jacob Effron?

  Yeah, that's really interesting. I mean, I guess obviously one of the fun parts of your job must be, uh, you're obviously probably really tightly integrated with the research team, see the models as they come through. Like, any things that you're looking out for on the model side, like I'm sure you you get the next computer use model or the next, you know, uh, models that are used for agents. Like, any milestones or capabilities that you're like, God, when we can do X, like, anytime I get the new model, I try X. And if we could do that, like, that would be so game changing for our developers.

• 32:01Speaker 02

  Yeah, ah, that's an interesting one. I actually have a bunch of prompts that I've gotten from a bunch of VC startups. And they're always like, this thing never works. And I actually like have them saved as like what we call presets or prompts in the openai dashboard. And each time something new comes, I like try like three or four of them. They're they're all like pretty much focused on like agentic tool use. And there's like six or seven different tools that are pretty straightforward. And I'm just like looking for these like reliable executions of them from turn to turn.

• 32:38Speaker 01

  Yeah.

• 32:38Speaker 02

  And I'm pretty optimistic like with our next series of models, but there's certain there's certain of them that like it just doesn't get right.

• 32:46Speaker 01

  Yeah.

• 32:47Speaker 02

  Um, I'm I'm also like really keen on finding much smaller and much faster models. Like faster than than Foro Mini for sure, that that are like pretty good at these tool use things. So if you think about, you know, the the workhorse models or the supporting models that sit around the old ones of the world. That can like do these like really quick classifications and guard railing and all. I think that there's a lot of room for improvement on those type of things. And yeah, just like the fastest, smallest classifier would be really, really cool to like work on.

• 33:21Speaker 01

  Totally, especially because they're so fine-tunable.

• 33:23Speaker 02

  Yes.

• 33:23Speaker 01

  Right? And you can just like really tailor those things like to your heart's content on like a specific use case. So yeah, that would be really cool. I'd have a fleet of those. For me, it's diffs. I just want the model to be able to spit out a diff that can apply cleanly to my code and it'll just work and I don't have to budget to get it to. That's going to be huge. That's going to be really, really huge. The models don't really like to understand. They don't really understand line numbers that well.

• 33:46Jacob Effron?

  What was your reaction? Obviously, there there was some really impressive agent work out of China recently. And I think that like, you know, it kind of always seemed that the most cutting edge agents would go alongside the most cutting edge models. But obviously, and I mean, you know, I think they're using anthropic models and whatnot, but I feel like it might have challenged that paradigm a little bit. And so curious kind of your reaction to to some of those demos.

• 34:06Speaker 02

  My reaction was like, this is what we've been saying internally is that like the capabilities are there in the models, but like so few people are able to make use out of it. I think it's it's crazy that like it's still like this.

• 34:21Speaker 02

  We we need to make it easier for developers and everyone to be able to build more powerful things with the models without like being, you know, like in. Yeah, like exceptional AI and ML people. And so I just feel like it validates the fact that give people the right tools, give people the right models. Help them, put them together with things like the agent SDK, make these things observable so that more and more people can build things like what we saw come out of China. Yeah, that's that's my take on it.

• 34:55Speaker 01

  I think just making the flywheel spin way faster from, you know, evals to production to fine tuning and back again, like that is such a powerful loop that we just need to make way simpler.

• 35:05Jacob Effron?

  Yeah.

• 35:05Speaker 01

  Yeah.

• 35:06Jacob Effron?

  What do you like think are the key things to make that simpler?

• 35:09Speaker 02

  I think the biggest thing to figure out, honestly, like we've got a.

• 35:12Speaker 01

  We have a good answer.

• 35:15Speaker 02

  I mean, the research team does it at OpenAI all the time, like the the model is getting better at chat, it's getting better at like doing all the deep research things. The next operator model is going to be so much more powerful at doing computer use things. How do you productize that is like the the thing that we need to figure out. Obviously, like with a lot of toil and like really closely observing your traces and like creating the right evals and graders. Like it it works for sure. We just have to prioritize this and and we need to figure out.

• 35:46Speaker 01

  Yeah.

• 35:46Speaker 02

  How to make this easy.

• 35:47Speaker 01

  It needs to be about 10 times easier than it is today. It's definitely doable, you know, you can you can create an eval, but it's hard, a lot of work to create an eval. And so I think that's the biggest thing for me is just like, how do we make that process of evaling your task, your workflow a lot easier?

• 35:59Jacob Effron?

  No, I mean, it's funny, I am struck by it feels like we have a new model and people spend like, you know, six, nine months trying to discover the use cases, they probably discover one percent of like what these models can actually do, and then it's like on to the next one. And so it's a it's pretty wild. I mean, obviously, you know, I think we we all kind of feel like we're on the precipice of this like super large change and, you know. Feels like, you know, we're going, especially as you make these tools easier, you know, agents are going to be increasingly ubiquitous. If I'm just like a normal, you know, enterprise or consumer CEO today, I haven't really thought about this so much. Like, what would you be doing in those people's shoes? If you're running a company that like, you know, probably in this agentic future has some way of interacting with these models?

• 36:37Speaker 02

  It's got to build, it's got to start, like, start with, start exploring these frontier models. Start exploring the, the computer use models. Take a couple of workflows internally and try to like, get a feel from like, for, for building these multi-agent architectures to, to automate things end to end. I feel like that's the, the most actionable and actual thing that you can do right now. On the tool side, like, figure out which of your, your, your, like, manual workflows need a tool interface and, and start doing that. Like, I feel like, like, the, the whole, like, digital transformation and automation thing that, that had its, like, thing during the cloud days is like coming back right now.

• 37:22Speaker 01

  Totally.

• 37:23Speaker 02

  And so sometimes I talk to users who are like, we, we want to automate this whole thing. But 90% of the work to be done is to like, figure out how to get programmatic access to certain tools that you're using. And like, the LLM portion is just like tiny in the middle and I'm like, this is, this is a very different problem for us. And like, yeah, you can like solve it with computer use right now and, and try to like get, get it to direction. But really just like finding ways to like automate your, your applications. Trying out the frontier models is probably like the main thing I'd recommend.

• 37:56Speaker 01

  Yeah, I think it's like really interesting being a developer in this era because, you know, for a long time, like, we have, as developers, been constantly automating away the bottom 20% of our job. Whether it's through better frameworks or better programming languages or what have you. And so I think that like, for me, if I were running a company, I would just be asking my employees like, what's your least favorite thing that you do on a daily basis? And like, let's try to figure out ways to automate that. That's going to make like everybody happy. It's going to, you know, increase productivity, of course. And so, yeah, that's how I would think about it.

• 38:24Jacob Effron?

  Have you guys done that?

• 38:26Speaker 01

  No, I'm not.

• 38:26Speaker 02

  They're so busy.

• 38:32Jacob Effron?

  I love that. I mean, look, a fascinating conversation. We always like to end with kind of a quick fire round where we stuff a bunch of overly broad questions in the last five minutes. And so maybe to start, I'd love your take on like, one thing that's overhyped and one thing that's underhyped in the AI world today.

• 38:45Speaker 02

  My answer is like, agents are both overhyped and underhyped. We've been talking about agents for like, a couple of years.

• 38:56Jacob Effron?

  We've gone through like, two full hype cycles.

• 38:57Speaker 02

  Yeah, I know. The same time, like, underhyped because like, hey, the companies that actually figure it out and build deep research-like things are fully automated, like, some really manual tasks are are able to just do so much. So yeah, that's my take on it.

• 39:12Jacob Effron?

  Yeah. I mean, obviously, you guys are so close to the cutting edge here. I'm curious, like, what's one thing you've changed your mind on in the AI world in the last year?

• 39:19Speaker 02

  I think it's the, for me, it's definitely the the power of these reasoning models has been, like, I don't know, like, they were always, we were always like, aware of this reasoning thing coming. And I I did not appreciate how that combined with tool use is going to create things like operator and deep research. And just seeing that it's possible to like, move away from this workflow, like, setup that every company was doing to this completely agentic product that figures out to use in its chain of thought and actually delivers, like, really, really powerful results. It that that's been like, the biggest, like, shift for me. And then, like, seeing early results of our reinforcement, fine tuning,

• 40:04Speaker 02

  Alpha, those are, you know, that that's been like, the biggest shift for me, in terms of like, how it's possible to do this.

• 40:10Speaker 01

  Yeah, for me, it's just fine tuning broadly, you know, it just like, the power of being able to like, I kind of thought that all the knowledge that you could put into models kind of like, baked in when it when it comes off of the off of the GPUs and but being able to like, really add a bunch of your own custom information and seeing how much that moves the needle for a specific task is pretty impressive.

• 40:26Jacob Effron?

  What do you think will be like, the biggest differentiator of of application builders long term? Because like, the question in venture, is it like, you know, kind of deep knowledge of the models and like, how to, you know, how to really build these agents? Is it just like, knowing a domain super well, so you know what to build? Like, what do you guys think of that?

• 40:40Speaker 01

  I think it's kind of a combination. And then there's this idea of like, if you have whatever special sauce it takes to be able to really bring the AGI out of the models that like, we think is in there. I don't know what they do, whether it is, if it's prompt engineering, or workflow orchestration, or something else, right? Like, I think that is going to be like, a huge differentiator.

• 40:59Speaker 02

  For me, it's like being really good at orchestrating. Like, I feel like that's gonna be the biggest.

• 41:03Jacob Effron?

  Yeah. What do you, what do you mean exactly about that?

• 41:05Speaker 02

  Like, bringing together your tools and data with a bunch of model calls, um, with a bunch of models, either in the fashion of it being reinforced and fine tuning and like calling these tools in the chain of thought, or in terms of like chaining together multiple L L Ms and being really good at like doing that quickly, evaluating and improving it. I think that's like the biggest skill that, that would move people forward, uh, like in the next year or two.

• 41:32Jacob Effron?

  Awesome. What do you think are some of the most like underexplored applications of these models today?

• 41:36Speaker 02

  Well, I haven't like seen anything crazy on the scientific research side. I think that would be really, like when the O series model started, like that was the, the main hope and expectation was that there'll be a step change in how quickly scientific research happens. And I think we've heard, we've seen some like early reports around that, but very curious to see how, how that changes.

• 41:59Speaker 01

  I think that like so much of, so much criticism about like the AI industry as a whole has been that the interfaces are not quite right yet. And I think like especially for a, a space like academia, you know, where everything is kind of the same way it's been for a long time. I think like finding the right interface for that is going to be really key and really drive a lot of adoption there.

• 42:16Jacob Effron?

  Yeah.

• 42:17Speaker 02

  Robotics too, maybe.

• 42:18Speaker 01

  Yeah.

• 42:18Speaker 02

  It's like, it's probably time for something big to happen.

• 42:22Speaker 01

  Yeah.

• 42:23Jacob Effron?

  The origins of OpenAI.

• 42:24Speaker 01

  Yeah.

• 42:24Jacob Effron?

  Good old Ruby's Cube. Um, do you think model progress will be more the less, uh, more or lesser the same, uh, as last year, this year?

• 42:31Speaker 02

  Oh, it's gonna be more.

• 42:32Speaker 01

  I think it's gotta be more. Yeah. Especially as like, I mean, it's a, it's a feedback loop, right? Especially as like we, the models are trying to teaching us how to, how to like improve them with better data and, and things like that. It's like something we, we do a lot on the research side.

• 42:44Jacob Effron?

  Which AI startup or like categories are you most excited about right now, like outside of OpenAI?

• 42:48Speaker 01

  I came from a travel background. Uh, I was doing a travel company right before I joined OpenAI. So I'm just like really excited to see sort of like somebody really crack that. I think the travel industry is like super entrenched and there's only a handful of like big players. And so I'm really excited to see who builds the actual AI travel agent. Um, I think.

• 43:04Jacob Effron?

  Everyone's favorite demo for agents.

• 43:05Speaker 01

  Exactly. Yeah. But like nobody, it's not like, uh, there's not a product that people are using there. So I'm really excited.

• 43:10Jacob Effron?

  Why doesn't it work yet?

• 43:11Speaker 01

  I, I don't know. I, I'm gonna go figure that out right after this.

• 43:19Speaker 02

  I use granola a lot. Have you heard of that?

• 43:21Jacob Effron?

  Yeah, of course.

• 43:22Speaker 02

  Yeah, that's my favorite AI tool these days. And I, every meeting, I'm in a very meeting-heavy role, so it helps a lot.

• 43:31Jacob Effron?

  Yeah, no, great product. Well, look, I think there's a ton of interesting threads for folks to pull on. Obviously, a ton of great stuff that you guys recently shipped. I want to leave the last word to you. Where can our listeners go to learn more about the APIs? About really any place you want to point them. The floor is yours.

• 43:46Speaker 02

  Yep, our docs, platform openly.com docs. And also like the OpenAI Devs channel on Twitter or the account on Twitter.

• 43:58Speaker 01

  And the community forum is always a great place to check out. I don't know the domain of that one.

• 44:01Speaker 02

  Is it forum dot?

• 44:03Speaker 01

  Forum dot.

• 44:03Speaker 02

  Community dot openai dot com.

• 44:04Speaker 01

  Just Google OpenAI Community Forum before you find it.

• 44:08Speaker 02

  Yeah.

• 44:08Jacob Effron?

  Or ask ChatGPT for it.

• 44:09Speaker 01

  Or ask ChatGPT.

• 44:11Jacob Effron?

  Awesome. Well, thank you both so much. This has been a ton of fun.

• 44:13Speaker 02

  Awesome.

• 44:13Speaker 01

  Thanks so much.