From API to AGI: Structured Outputs, OpenAI API platform and O1 Q&A — with Michelle Pokrass & OpenAI Devrel + Strawberry team

Congrats to Damien on successfully running AI Engineer London https://x.com/maximelabonne/status/1834573217475744160! See our community page https://www.latent.space/p/community and the Latent Space Discord https://discord.gg/xJJMRaWCRt for all upcoming events. This podcast came together in a far more convoluted way than usual, but happens to result in a tight 2 hours covering the ENTIRE OpenAI product suite across ChatGPT-latest, GPT-4o and the new o1 models, and how they are delivered to AI Engineers in the API via the new Structured Output mode, Assistants API, client SDKs, upcoming Voice Mode API, Finetuning/Vision/Whisper/Batch/Admin/Audit APIs, and everything else you need to know to be up to speed in September 2024. This podcast has two parts: the first hour is a regular, well edited, podcast on 4o, Structured Outputs, and the rest of the OpenAI API platform. The second was a rushed, noisy, hastily cobbled together recap of the top takeaways from the o1 model release from yesterday and today. Building AGI with Structured Outputs — Michelle Pokrass of OpenAI API team Michelle Pokrass https://www.linkedin.com/in/mpokrass/ built massively scalable platforms at Google, Stripe, Coinbase and Clubhouse, and now leads the API Platform at Open AI. She joins us today to talk about why structured output is such an important modality for AI Engineers that Open AI has now trained and engineered a Structured Output mode https://openai.com/index/introducing-structured-outputs-in-the-api/ with 100% reliable JSON schema adherence. To understand why this is important, a bit of history is important: * June 2023 when OpenAI first added a "function calling" capability to GPT-4-0613 and GPT 3.5 Turbo 0613 (our podcast/writeup here https://www.latent.space/p/function-agents) * November 2023’s OpenAI Dev Day (our podcast/writeup here https://www.latent.space/p/devday) where the team shipped JSON Mode, a simpler schema-less JSON output mode that nevertheless became more popular because function calling often failed to match the JSON schema given by developers. * Meanwhile, in open source, many solutions arose, including * Instructor (our pod with Jason here https://www.latent.space/p/instructor?utm_source=publication-search) * LangChain (our pod with Harrison here https://www.latent.space/p/langchain, and he is returning next as a guest co-host) * Outlines (Remi Louf’s talk at AI Engineer here https://x.com/remilouf/status/1806331535852704134) * Llama.cpp’s constrained grammar sampling https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md using GGML-BNF * April 2024: OpenAI started implementing constrained sampling https://x.com/gdb/status/1784990428854391173 with a new `tool_choice: required` parameter in the API * August 2024: the new Structured Output mode https://openai.com/index/introducing-structured-outputs-in-the-api/, co-led by Michelle * Sept 2024: Gemini shipped Structured Outputs https://x.com/OfficialLoganK/status/1833226001670934827 as well We sat down with Michelle to talk through every part of the process, as well as quizzing her for updates on everything else the API team has shipped in the past year, from the Assistants API, to Prompt Caching, GPT4 Vision, Whisper, the upcoming Advanced Voice Mode API, OpenAI Enterprise features, and why every Waterloo grad seems to be a cracked engineer. Part 1 Timestamps and Transcript Transcript here https://gist.github.com/FanaHOVA/69f540c5228d7ace69bac06189cf0396. * [00:00:42] Episode Intro from Suno https://suno.com/song/eed1b9c6-526c-480a-8bff-64f0908ffcb1 * [00:03:34] Michelle's Path to OpenAI * [00:12:20] Scaling ChatGPT * [00:13:20] Releasing Structured Output * [00:16:17] Structured Outputs vs Function Calling * [00:19:42] JSON Schema and Constrained Grammar * [00:20:45] OpenAI API team * [00:21:32] Structured Output Refusal Field * [00:24:23] ChatML issues * [00:26:20] Function Calling Evals * [00:28:34] Parallel Function Calling * [00:29:30] Increased Latency * [00:30:28] Prompt/Schema Caching * [00:30:50] Building Agents with Structured Outputs: from API to AGI * [00:31:52] Assistants API * [00:34:00] Use cases for Structured Output * [00:37:45] Prompting Structured Output * [00:39:44] Benchmarking Prompting for Structured Outputs * [00:41:50] Structured Outputs Roadmap * [00:43:37] Model Selection vs GPT4 Finetuning * [00:46:56] Is Prompt Engineering Dead? * [00:47:29] 2 models: ChatGPT Latest vs GPT 4o August * [00:50:24] Why API => AGI * [00:52:40] Dev Day * [00:54:20] Assistants API Roadmap * [00:56:14] Model Reproducibility/Determinism issues * [00:57:53] Tiering and Rate Limiting * [00:59:26] OpenAI vs Ops Startups * [01:01:06] Batch API * [01:02:54] Vision * [01:04:42] Whisper * [01:07:21] Voice Mode API * [01:08:10] Enterprise: Admin/Audit Log APIs * [01:09:02] Waterloo grads * [01:10:49] Books * [01:11:57] Cognitive Biases * [01:13:25] Are LLMs Econs? * [01:13:49] Hiring at OpenAI Emergency O1 Meetup — OpenAI DevRel + Strawberry team the following is our writeup from AINews https://buttondown.com/ainews/archive/ainews-o1-openais-new-general-reasoning-models/, which so far stands the test of time. o1, aka Strawberry, aka Q*, is finally out https://openai.com/o1/#snake-video! There are two models we can use today: o1-preview (the bigger one priced at $15 in / $60 out) and o1-mini (the STEM-reasoning focused distillation priced at $3 in/$12 out) - and the main o1 model is still in training. This caused a little bit of confusion https://x.com/giffmana/status/1834306463142949338. There are a raft of relevant links, so don’t miss: * the o1 Hub https://openai.com/o1/#snake-video * the o1-preview blogpost https://openai.com/index/introducing-openai-o1-preview/ * the o1-mini blogpost https://news.ycombinator.com/item?id=41523050 * the technical research blogpost https://openai.com/index/learning-to-reason-with-llms/ * the o1 system card https://openai.com/index/openai-o1-system-card/ * the platform docs https://platform.openai.com/docs/guides/reasoning * the o1 team video and contributors list https://www.notion.so/ainews-draft-479a38e041fe4ab4b05d6f90573d967d?pvs=21 (twitter https://x.com/polynoamial/status/1834346060170367031) Inline with the many, many leaks leading up to today, the core story is longer “test-time inference” aka longer step by step responses - in the ChatGPT app this shows up as a new “thinking” step that you can click to expand for reasoning traces, even though, controversially, they are hidden from you (interesting conflict of interest…): Under the hood, o1 is trained for adding new reasoning tokens - which you pay for, and OpenAI has accordingly extended the output token limit to >30k tokens (incidentally this is also why a number of API parameters from the other models like temperature and role and tool calling and streaming, but especially max_tokens is no longer supported). The evals are exceptional. OpenAI o1: * ranks in the 89th percentile on competitive programming questions (Codeforces), * places among the top 500 students in the US in a qualifier for the USA Math Olympiad (AIME), * and exceeds human PhD-level accuracy on a benchmark of physics, biology, and chemistry problems (GPQA). You are used to new models showing flattering charts, but there is one of note that you don’t see in many model announcements, that is probably the most important chart of all. Dr Jim Fan gets it right: we now have scaling laws for test time compute, and it looks like they scale loglinearly. We unfortunately may never know the drivers of the reasoning improvements, but Jason Wei shared some hints https://x.com/_jasonwei/status/1834278706522849788?s=46: Usually the big model gets all the accolades, but notably many are calling out the performance of o1-mini for its size (smaller than gpt 4o), so do not miss that. Part 2 Timestamps * [01:15:01] O1 transition * [01:16:07] O1 Meetup Recording * [01:38:38] OpenAI Friday AMA recap * [01:44:47] Q&A Part 2 * [01:50:28] O1 Demos This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:04Speaker 00

  Michelle Pokrass on the mic, leading the way. Open me, API, making waves every day. Pre-chat, GPT, she was ahead of the game. Scaling challenges, hit, but she stayed in the lane. Case on mode, structured output's tight. Building AGI with that clearance site. Refusal field tricky, HCTP's out of date. Chatting those calling functions, it's the future we make. From agents built with structured dreams, to five-two models in API streams. Patch it up, scale it wide. Voice mode whispering, we ride the tide. Latent space for the world to see. Michelle's got the roadmap, building AI free.

• 0:40Speaker 01

  Welcome back. This is Charlie, your AI co-host. Michelle Pokrass built massively scalable platforms at Google, Stripe, Coinbase and Clubhouse, and now leads the API platform at OpenAI. She joins us today to talk about why structured output is such an important modality for AI engineers that OpenAI has now trained and engineered a structured output mode with 100% reliable JSON schema adherence. To understand why this is important, we have to go all the way back to June of last year when OpenAI first added a function calling capability to GPT-40613 and GPT-3.5 Turbo-0613,

• 1:25Speaker 01

  which was then followed by November's Dev Day, where the team shipped JSON mode, a simpler, schema-less JSON output mode that nevertheless became more popular because function calling often failed to match the JSON schema given by developers. Meanwhile, in open source, many solutions arose, including Instructor and Langchain from our former guests Jason Liu and Harrison Chase, who by the way is returning to co-host an Agents episode soon, and outlines from World's Fair speaker Remy Luff and Llama.cpp's constrained grammar sampling using the GGML extension of the Backus Nowa form, or BNF syntax.

• 2:07Speaker 01

  Fast forward to April of 2024, OpenAI started implementing constrained sampling with a new tool choice required parameter in the API, and finally, in August, closed the loop by releasing the new structured output mode, which extends constrained sampling with specific post-training to improve the performance of complex JSON schema following, especially with the strict true flag.

• 2:34Speaker 01

  The other big labs seem to be following suit with Gemini shipping structured outputs and inner mode in this past month. We sat down with Michelle to talk through every part of the process, as well as quizzing her for updates on everything else the API team has shipped in the past year, from the Assistants API to prompt caching GPT-4, Vision, Whisper, the upcoming Advanced Voice Mode API, OpenAI Enterprise Features, and why every Waterloo grad seems to be a cracked engineer.

• 3:08Speaker 01

  In Latent Space Community news, if you're in Germany, the first meetup for AI engineers is happening in Cologne in two weeks, and the second AI Engineer Summit, the curated invite-only conference run by the AI Engineer World Fair team, is now moving to January in New York City. See the show notes for details. Watch out and take care.

• 3:34Alessio Fanelli?

  Hey, everyone. Welcome to the Latent Space Podcast. This is Alessio, Partner and CTO in Residence and Decibel Partners, and I'm joined by my co-host swyx, founder of SmallAI.

• 3:43Speaker 03

  Hey, and today we're excited to be in the in-person studio with Michelle. Welcome.

• 3:48Speaker 04

  Thanks. Thanks for having me. Very excited to be here.

• 3:50Speaker 03

  This has been a long time coming. I've been following your work on the API platform for a little bit, and I'm finally glad that we could make this happen after you ships structured outputs. How does that feel?

• 4:01Speaker 04

  Yeah, it feels great. We've been working on it for quite a while. So very excited to have it out there and have people using it.

• 4:07Speaker 03

  We'll tell the stories soon, but I want to give people a little intro to your backgrounds. So you've interned and worked at Google, Stripe, Coinbase, Clubhouse, and obviously OpenAI. What was that journey like? You know, the one that has the most appeal to me is Clubhouse because that was a very, very hot company for a while. Basically, you seem to join companies when they're about to scale up really a lot. And obviously, OpenAI has been the latest. But yeah, just what are your learnings and your history going into all these notable companies?

• 4:35Speaker 04

  Yeah, totally. For a bit of my background, I'm Canadian. I went to the University of Waterloo and there you do like six internships as part of your degree. So I started, actually, my first job was really rough. I worked at a bank and I learned Visual Basic and I like animated bond yield curves. And it was, you know, not, not-

• 4:51Speaker 03

  Me too.

• 4:52Speaker 04

  Oh, really?

• 4:52Speaker 03

  Yeah, I was a derivative trader, interest rate swaps, that kind of stuff, yeah.

• 4:56Speaker 04

  Yeah, so I liked, you know, having a job, but I didn't love that job. And then my next internship was Google and I learned so much there. It was tremendous. But I had a bunch of friends that were into startups more and, you know, Waterloo is like a big startup culture. And one of my friends interned at Stripe and he said it was super cool. So that was kind of my, I also was a little bit into crypto at the time. Then I got into it on Hacker News and so Coinbase was on my radar. And so that was like my first real startup opportunity was Coinbase. I think I've never learned more in my life than in the four month period when I was interning at Coinbase. They actually put me on call. I worked on like the ACH rails there and it was, it was absolutely crazy. You know, crypto was a very formative experience.

• 5:34Speaker 03

  This is 2018 to 2020, kind of like the first-

• 5:38Speaker 04

  That was my full time. But I was there as an intern in 2016. Yeah. And so that was the period where I really like learned to become an engineer, learned how to use Git, got on call right away, you know, managed production databases and stuff. So that was super cool. After that, I went to Stripe and kind of got a different flavor of payments on the other side. Learned a lot, was really inspired by the Colsons. And then my next internship after that, I actually started a company at Waterloo. So there's this thing you can do. It's an entrepreneurship co-op. And I did it with my roommate. The company's called Readwise, which still exists.

• 6:09Speaker 03

  Yeah, yeah. No one uses Readwise.

• 6:10Speaker 04

  Yeah.

• 6:11Speaker 03

  You co-founded Readwise?

• 6:12Speaker 04

  Yeah.

• 6:13Speaker 03

  I'm not a premium user. It's not even on your LinkedIn?

• 6:17Speaker 04

  Yeah, I mean, I only worked on it for about a year. And so Tristan and Dan are the real founders and I just had an interlude there. But yeah, really loved working on something very startup focused, user focused and hacking with friends. It was super fun. Eventually, I decided to go back to Coinbase and really like get a lot better as an engineer. I didn't feel like I was, you know, didn't feel equipped to be a CTO of anything at that point. And so just learned so much at Coinbase. And that was a really fun curve. But yeah, after that, I went to Clubhouse, which was like a really interesting time. So I wouldn't say that I went there before it blew up. I would say I went there as it blew up. So not quite the startling track record that it might seem. But it was a super exciting place. I joined as like the second or third backend engineer. And, you know, we were down every day, basically, you know?

• 7:02Speaker 04

  One time Oprah came on and absolutely everything melted down. And so we would have a stand up every morning and be like, how do we make everything stay up? Which is super exciting. Also, one of the first things I worked on there was making our notifications go out more quickly. Because when you join a Clubhouse room, you know, you need everyone to come in right away so that it's exciting. And the person speaking thinks a lot of my audience is here. But when I first joined, I think it would take like 10 minutes for all the notifications to go out, which is insane. Like, you know, by the time you want to start talking to the time your audience is there, it's like you can totally kill the room. So that's one of the first things I worked on is making that a lot faster and, you know, keeping everything up.

• 7:37Speaker 03

  I mean, so already we have an audience of engineers. Those two things are useful. It's keeping things up and notifications out. Notifications, like is it a Kafka topic?

• 7:45Speaker 04

  It was a Postgres shop and you had all of the followers in Postgres. And you needed to like iterate over the followers and like figure out, is this a good notification to send?

• 7:53Speaker 03

  Okay.

• 7:53Speaker 04

  And so all of this logic, it wasn't like well batched and paralyzed. And our job queuing infrastructure wasn't right. And so there's a lot of like fixing all of these things. Eventually, there were a lot of database migrations because Postgres just wasn't scaling well for us.

• 8:06Speaker 03

  Interesting. And then keeping things up, that was more of a, I don't know, reliability issue, SRE type?

• 8:12Speaker 04

  A lot of it, yeah, this goes down to like database stuff everywhere I've worked.

• 8:16Speaker 03

  It's all databases.

• 8:19Speaker 04

  Actually, at Coinbase, at Clubhouse, and at OpenAI, Postgres has been a perennial challenge. It's like the stuff you learn at one job carries over to all the others because you're always debugging a long-running Postgres query at 3am for some reason. So those skills have really carried me forward, for sure.

• 8:36Speaker 03

  Why do you think that not as much of this is practiced? Obviously, Postgres as an open-source project is not aimed at like gigascale, but you would think somebody would come around and say, hey, we're like the...

• 8:46Speaker 04

  Yeah. I think that's what Planet Scale is doing, kind of. It's not on Postgres, I think. It's on MySQL. But I think that's the vision. It's like they have zero downtime, migrations, and that's a big pain point. I don't know why no one is doing this on Postgres, but I think it would be pretty killer.

• 9:01Speaker 03

  Their connection poolers like PG Bouncer is like good enough?

• 9:04Speaker 04

  Yeah. Well, even, I mean, I've run PG Bouncer everywhere, and there's still a lot of problems.

• 9:10Speaker 03

  I mean, your scale, it's something that not many people see, so.

• 9:13Speaker 04

  Yeah. I mean, at some point, every company gets the scale, every successful company gets the scale where Postgres is not cutting it, and then you migrate to some sort of NoSQL database. And that process I've seen happen a bunch of times now.

• 9:23Speaker 03

  MongoDB, Redis, something like that?

• 9:25Speaker 04

  Yeah. I mean, we're on Azure now, and so we use Cosmos DB.

• 9:30Speaker 03

  Cosmos DB! Hey!

• 9:31Speaker 04

  At Clubhouse, I really love Dynamo DB. That's probably my favorite database, which is like a very nerdy sentence, but that's the one I'm using if I need to scale something as far as it goes.

• 9:40Speaker 03

  Yeah. Dynamo DB, when I learned, I worked at AWS briefly, and it's kind of like the memory register for the web.

• 9:46Speaker 04

  Yes.

• 9:47Speaker 03

  You know, if you treat it just as physical memory, you will use it well. If you treat it as a real database, you might run into problems.

• 9:54Speaker 04

  Right. You have to totally change your mindset when you're going from Postgres to Dynamo. But I think it's a good mindset shift and kind of makes you design things in a more scalable way.

• 10:01Speaker 03

  Yeah. I'll recommend the Dynamo DB book for people who need to use Dynamo DB. But we're not here to talk about AWS. We're here to talk about OpenAI. You joined OpenAI pre-ChaiGPT. I also had the opportunity to join, and I didn't. What was your insight?

• 10:14Speaker 04

  Yeah, I think a lot of people who joined OpenAI joined because of a product that really gets them excited. And for most people, it's ChatGPT. But for me, I was a daily user of Copilot, GitHub Copilot. And I was like so blown away at the quality of this thing. I actually remember the first time seeing it on Hacker News and being like, wow, this is absolutely crazy. Like, this is gonna change everything. And I started using it every day. It just really, even now, when I don't have service and I'm coding without Copilot, it's just like 10x difference. So I was really excited about that product. I thought now is maybe the time for AI. And I had done some AI in college and thought some of those skills would transfer. And I got introduced to the team. I liked everyone I talked to, so I thought that would be cool. Why didn't you join?

• 10:54Speaker 03

  It was like, I was like, is Dolly it?

• 10:56Alessio Fanelli?

  We were there. We were at the Dolly like launch thing. And I think you were talking with Lenny and Lenny was at OpenAI at the time and you were like.

• 11:05Speaker 03

  We don't have to go into too much detail, but this is one of my biggest regrets of my life. But, but, well, I was like, okay, I mean, I can, I can create images. I don't know if like this is the thing to dedicate, but obviously you had a bigger vision than I did.

• 11:18Speaker 04

  Dolly was really cool too. I remember like first showing my family. I was like, I'm going to this company and here's like one of the things they do. And it like really helped bridge the gap. Whereas like, I still haven't figured out how to explain to my parents what crypto is. My mom, for a while, thought I worked at Bitcoin. So it's like, it's pretty different to be able to tell your family what you actually do and they can see it.

• 11:38Speaker 03

  Yeah. And they can use it too, personally. So you were there, were you immediately on API platform? You were there for the ChatGPT moment?

• 11:44Speaker 04

  Yeah, I mean, API platform is like a very grandiose term for what it was. There was like just a handful of us working on the API.

• 11:50Speaker 03

  Yeah, it was like a closed beta, right? Not even everyone had access to the GPT 3.

• 11:55Speaker 04

  A very different access model then, a lot more like tiered rollouts. But yeah, I would say the applied team was maybe like 30 or 40 people. And yeah, probably closer to 30. And there was maybe like five-ish total working on the API at most. So yeah, we've grown a lot since then.

• 12:10Speaker 03

  It's like 60, 70 now, right?

• 12:11Speaker 04

  No, applied is much bigger than that. Applied now is bigger than the company when I joined.

• 12:15Speaker 03

  Okay.

• 12:16Speaker 04

  Yeah, we've grown a lot. I mean, there's so much to build, so we need all the help we can get.

• 12:19Speaker 03

  I'm a little out of date, yeah.

• 12:20Alessio Fanelli?

  Any ChatGPT release, kind of like all hands on deck stories? I had lunch with Evan Morikawa a few months ago. It sounded like it was a fun time to get, build the APIs and have all these people turn and use the web thing. Like, how are you prioritizing internally? Like, what was the helping scaling when you're scaling now GPU workloads versus like Postgres bouncers and things like that?

• 12:42Speaker 04

  Yeah, actually surprisingly, there were a lot of postgress issues when ChatGPT came out because the accounts for like ChatGPT were tied to the accounts in the API. And so you're basically creating a developer account to log into ChatGPT at the time because it's just what we had. It was low key research preview. And so I remember there was just so much work scaling like our authorization system and that would be down a lot. Yeah, also GPU, you know, I never had worked in a place where you couldn't just scale the thing up. It's like everywhere I've worked, compute is like free and you just like auto scale a thing and you like never think about it again. But here we're having like tough decisions every day. We're like discussing like, you know, should they go here or here? And we have to be principled about it. So that's a real mindset shift.

• 13:20Speaker 03

  So you just really structured outputs. Congrats. You also wrote the blog post for it, which was really well written. And I love all the examples that you put out, like it really gives the full story. Yeah, tell us about the whole story from beginning to end.

• 13:31Speaker 04

  Yeah, I guess the story we should rewind quite a bit to Dev Day last year. Dev Day last year, exactly, we shipped JSON mode, which is our first foray into this area of product. So for folks who don't know, JSON mode is this functionality you can enable in our chat completions and other APIs, where if you opt in, we'll kind of constrain the output of the model to match the JSON language. And so you basically will always get something in a curly brace. And this is good. This is nice for a lot of people. You can like describe your schema, what you want in prompt and then, you know, we'll constrain it to JSON. But it's not getting you exactly where you want because you don't want the model to kind of make up the keys or like match different values than what you want. Like if you want an enum or a number and you get a string instead, it's like pretty frustrating.

• 14:13Speaker 04

  So we've been ideating on this for a while and like people have been asking for basically this every time I talk to customers for maybe the last year. So it was really clear that there's a developer need and we started working on kind of making it happen. And this is a real collab between engineering and research, I would say. And so it's not enough to just kind of constrain the model. I think of that as the engineering side, whereas basically you mask the available tokens that are produced every time to only fit the schema. And so you can do this engineering thing and you can force the model to do what you want, but you might not get good outputs. And sometimes with JSON mode, developers have seen that our models output like white space for a really long time where they don't.

• 14:49Speaker 03

  Because it's a legal character.

• 14:51Speaker 04

  Right, it's legal for JSON, but it's not really what they want. And so that's what happens when you do kind of a very engineering biased approach. But the modeling approach is to also train the model to do more of what you want. And so we did these together. We trained a model which is significantly better than our past models at following formats. And we did the end work to serve like this constrained decoding concept at scale. So I think marrying these two is why this feature is pretty cool.

• 15:13Alessio Fanelli?

  You just mentioned starts and ends with a curly brace and maybe people's minds go to prefills in the Claude API. How should people think about JSON mode, structured output, prefills? Because some of them are like roughly starts with a curly brace and asks you for JSON, you should do it. And then instructor is like, hey, here's a rough data schema you should use. And how do you think about them?

• 15:34Speaker 04

  So I think we kind of designed structured outputs to be the easiest to use. So you just like the way you use it in our SDK, I think, is my favorite thing. So you just create like a PyDantic object or a Zot object and you pass it in and you get back an object. And so you don't have to deal with any of the serialization.

• 15:49Speaker 03

  With the parse helper.

• 15:50Speaker 04

  Yeah, you don't have to deal with any of the serialization on the way in or out. So I kind of think of this as the feature for the developer who is like, I need this to plug into my system. I need the function call to be exact. I don't want to deal with any parsing. So that's where structured outputs is tailored. Whereas if you want the model to be more creative and use it to come up with JSON schema that you don't even know you want, then that's kind of where JSON mode fits in. But I expect most developers are probably going to want to upgrade to structured outputs.

• 16:17Speaker 03

  The thing you just said, you just use interchangeable terms for the same thing, which is function calling and structured outputs. We've had disagreements or discussion before on the podcast about are they the same thing? Semantically, they're slightly different.

• 16:31Speaker 04

  They are, yes.

• 16:31Speaker 03

  Because I think function calling API came out first.

• 16:33Speaker 04

  Yes.

• 16:34Speaker 03

  Then JSON mode. And we used to abuse function calling for JSON mode.

• 16:39Speaker 04

  Right.

• 16:40Speaker 03

  Do you think we should treat them as synonymous?

• 16:42Speaker 04

  No.

• 16:42Speaker 03

  Okay, yeah, please clarify. And by the way, there's also tool calling.

• 16:47Speaker 04

  Yeah, the history here is we started with function calling and function calling, you know, came from the idea of like, let's give the model access to tools and let's see what it does. And we basically had these internal prototypes of what code interpreter is now. And we were like, this is super cool. Let's make it an API, but we're not ready to host code interpreter for everybody. So, you know, we're just going to expose the rock capability and see what people do with it. But even now, I think there's a really big difference between function calling and structured outputs. So you should use function calling when you actually have functions that you want the model to call.

• 17:16Speaker 03

  Right.

• 17:17Speaker 04

  And so like, if you have a database that you want the model to be able to query from, or if you want the model to send an email or like, you know, generate arguments for an actual action. And that's the way the model has been like, fine tuned on is to like, treat function calling for actually calling these tools and getting their outputs. The new response format is a way of just getting the model to respond to the user, but in a structured way. And so this is very different, like responding to a user versus like, you know, I'm gonna go send an email. A lot of people were hacking function calling to get the response format they needed. And so this is why we shipped kind of this new response format. So you can get exactly what you want and you get kind of more of the models verbozity. It's like, kind of responding in the way it would speak to a user.

• 18:00Speaker 04

  And so less kind of just programmatic tool calling, if that makes sense.

• 18:03Alessio Fanelli?

  Are you building something to the SDK to actually close the loop with the function calling? Because right now it returns the function, then you gotta run it, then you gotta like, fake another message to then

• 18:13Speaker 04

  Yeah.

• 18:13Alessio Fanelli?

  continue the conversation.

• 18:15Speaker 03

  They have that in beta, the runs.

• 18:16Speaker 04

  Yes, we have this in beta in the node SDK. So you can basically

• 18:21Speaker 03

  Oh, not Python.

• 18:22Speaker 04

  It's coming to Python as well.

• 18:23Alessio Fanelli?

  That's why I didn't know.

• 18:25Speaker 03

  Yeah, I'm a node guy, so

• 18:26Alessio Fanelli?

  The JavaScript mine.

• 18:28Speaker 03

  It's already existed.

• 18:30Speaker 04

  It's coming everywhere, but basically what you do is you write a function and then you add a decorator to it. And then you can, basically there's this run tools method and it does the whole loop for you, which is pretty cool.

• 18:40Speaker 03

  When I saw that in the node SDK, I wasn't sure if that's, because it basically runs it in the same machine.

• 18:46Speaker 04

  Yeah.

• 18:47Speaker 03

  And maybe you don't want that

• 18:48Speaker 04

  Right.

• 18:48Speaker 03

  to happen.

• 18:49Speaker 04

  Yeah, I think of it as like, if you're prototyping and building something really quickly and just playing around, it's so cool to just create a function and give it this decorator. But, you know, you have the flexibility to do it however you like.

• 18:59Speaker 03

  Like you don't want it in a critical path of a web request?

• 19:02Speaker 04

  I mean, some people definitely will.

• 19:05Speaker 04

  You know, it's just kind of the easiest way to get started.

• 19:07Speaker 03

  Yeah.

• 19:07Speaker 04

  But let's say you want to like execute this function on a job queue async. Then, you know, it wouldn't make sense to use that.

• 19:13Speaker 03

  Pyart, Instructure, Outlines, JSONformer. What did you study? What did you, you know, credit or learn from these things?

• 19:21Speaker 04

  Yeah, there's a lot of different approaches to this. There's more fill in the blank style sampling where you basically preform kind of the keys and then get the model to sample just the value. There's kind of a lot of approaches here. We didn't kind of use any of them wholesale, but we really loved what we saw from the community and like the developer experiences we saw. So that's where we took a lot of inspiration.

• 19:42Speaker 03

  There was a question also just about constrained grammar. This is something that I first saw in Lama CPP, which seems to be the most, let's just say academically permissive.

• 19:53Speaker 04

  The lowest level.

• 19:54Speaker 03

  Yeah.

• 19:55Speaker 04

  Yeah.

• 19:55Speaker 03

  For those who don't know, maybe, I don't know if you want to explain it, but they use back as no form, which you only learn in like college when you're working on programming languages and compilers. I don't know if you like use that under the hood or you explore that.

• 20:05Speaker 04

  Yeah, we didn't use any kind of other stuff. We kind of built, you know, our solution from scratch to meet our specific needs. But I think there's a lot of cool stuff out there where you can supply your own grammar. Right now, we only allow JSON schema and the dialect of that. But I think in the future, it could be a really cool extension to let you supply a grammar more broadly. And maybe it's more token efficient than JSON. So a lot of opportunity there.

• 20:29Alessio Fanelli?

  You mentioned before also training the model to be better function calling. Was that discussion like internally for like resources? It's like, hey, we need to get better JSON mode. And it's like, well, can you figure it out on the API platform without touching the model? Like, is there a really tight collaboration between the two teams?

• 20:45Speaker 04

  Yeah, so I actually work on the API models team. I guess we didn't quite get into what I do at API.

• 20:51Speaker 03

  What do you say it is you do here?

• 20:54Speaker 04

  Yeah. So yeah, I'm the tech lead for the API, but also I work on the API models team. And this team is really working on making the best models for the API. And a lot of common deployment patterns are research makes a model and then you kind of ship it in the API. But you know, I think there's a lot you miss when you do that. You miss a lot of developer feedback and things that are not kind of immediately obvious. What we do is we get a lot of feedback from developers and we go and make the models better in certain ways. So our team does model training as well. We work very closely with our post-training team. And so for structured outputs, it was a collab between a bunch of teams, including Safety Systems, to make, you know, a really great model that does structured outputs.

• 21:32Speaker 03

  Mentioning Safety Systems, you have a refusal field.

• 21:35Speaker 04

  Yes.

• 21:35Speaker 03

  You want to talk about that?

• 21:36Speaker 04

  Yeah.

• 21:36Speaker 03

  It's like a.

• 21:37Speaker 04

  Yeah, it's a little, it's pretty interesting. So you can imagine, basically, if you constrain the model to follow a schema, you can imagine there being like a schema supplied that it wouldn't, it would add some risk or be harmful for the model to kind of follow that schema. And we wanted to preserve our model's abilities to refuse when something, you know, doesn't match our policies or is harmful in some way. And so we needed to give the model an ability to refuse even when there is this schema. But also, you know, if you are a developer and you have this schema and you get back something that doesn't match it, you're like, ah, the feature's broken. So we wanted a really clear way for developers to program against this. So if you get something back in the content, you know it's valid, it's JSON-precible. But if you get something back in the refusal field, it makes for a much better UI for you to kind of display this to your user in a different way.

• 22:23Speaker 04

  And it makes it easier to program against. So really, there was a few goals, but it was mainly to allow the model to continue to refuse, but also with a really good developer experience.

• 22:30Speaker 03

  Yeah. Why not offer it as like an error code? Because we have to display error codes anyway.

• 22:36Speaker 04

  Yeah, we fluffled for a long time about API design as we are want to do. And there are a few reasons against an error code. Like you can imagine this being a 4xx error code or something. But you know, the developer's paying for the tokens. And that's kind of atypical for like a 4xx error code.

• 22:53Speaker 03

  We pay with errors anyway, right? Or no?

• 22:56Speaker 04

  4xxes.

• 22:57Speaker 03

  Is not, that's a, that's a U error.

• 22:59Speaker 04

  Right. And it doesn't make sense as a 5xx either, because it's not our fault. You know, it's the way the API, the model is designed. I think the HTTP spec is a little bit limiting for AI in a lot of ways. Like there are things that are in between your fault and my fault. There's kind of like the model's fault and there's no, you know, error code for that. So we really have to kind of invent a lot of the paradigm here.

• 23:21Speaker 03

  Make it 6xx.

• 23:23Speaker 04

  Yeah, that's one option. There's actually some like esoteric error codes we've considered adopting. We're still figuring that out, but I think there are some things like, for example, sometimes our model will produce tokens that are invalid based on kind of our language. And when that happens, it's an error, but you know, it doesn't five hundred is fine, which is what we return, but it's not as expressive as it could be. So yeah, just areas where, you know, web two point O doesn't quite fit with AI yet.

• 23:50Alessio Fanelli?

  If you had to put in a spec to just change. What would be your number one proposal to like re-haul?

• 23:56Speaker 03

  The HTTP committee to reinvent the world.

• 24:00Speaker 04

  Yeah, that's good. I mean, I think we just need an error of like a range of model error. And we can have many different kinds of model errors, like a refusal is a model error.

• 24:09Alessio Fanelli?

  Six oh one, auto refusal.

• 24:11Speaker 04

  Yeah, again, like, so we we've mentioned before that chat completions uses this chat ML format. So when the model doesn't follow chat ML, that's an error. And we're working on reducing those errors, but that's like, I don't know, six oh two, I guess.

• 24:23Speaker 03

  A lot of people actually don't no longer know what chat ML is.

• 24:26Speaker 04

  Yeah.

• 24:26Speaker 03

  Because that was briefly introduced by OpenAI and then like kind of deprecated. Everyone who implements this under hood knows it, but maybe the API users don't know it.

• 24:35Speaker 04

  Basically, the API started with just one endpoint, the completions endpoint. And the completions endpoint, you just put text in and you get text out. And you can prompt in certain ways. Then we released chat GPT. And we decided to put that in the API as well, and that became the chat completions API. And that API doesn't just take like a string input and produce an output. It actually takes in messages and produces messages. And so you can get a distinction between like an assistant message and a user message, and that allows all kinds of behavior. And so the format under the hood for that is called chat ML. Sometimes, you know, because the model is so out of distribution based on what you're doing, maybe the temperature is super high. Then it can't follow chat ML.

• 25:14Speaker 03

  Yeah, I didn't know that there could be errors generated there. Maybe I'm not asking challenging enough questions.

• 25:20Speaker 04

  It's pretty rare and we're working on driving it down. But actually, this is a side effect of structured outputs now. Which is that we have removed a class of errors. We didn't really mention this in the blog, just because we ran out of space, but

• 25:32Speaker 03

  What we're here to do.

• 25:33Speaker 04

  Yeah, the model used to occasionally pick a recipient that was invalid. And this would cause an error. But now we are able to constrain to Chat ML in a more valid way. And this reduces a class of errors as well.

• 25:45Speaker 03

  Recipient, meaning so there's this like a few number of defined roles, like user, assistant, system.

• 25:51Speaker 04

  So like recipient as in like picking the right tool.

• 25:53Speaker 03

  Oh.

• 25:54Speaker 04

  So the model before was able to hallucinate a tool. But now it's I can't when you're using structured outputs.

• 26:01Alessio Fanelli?

  Do you collaborate with other model developers to try and figure out this type of errors? Like, how do you display them? Because a lot of people try to work with different models.

• 26:10Speaker 04

  Yeah.

• 26:10Alessio Fanelli?

  Is there any?

• 26:11Speaker 04

  Yeah, not a ton. We're kind of just focused on making the best API for developers.

• 26:16Speaker 03

  A lot of research and engineering, I guess, comes together with evals. You published some evals there. I think Gorilla is one of them. What is your assessment of like the state of evals for function calling and structured output right now?

• 26:29Speaker 04

  Yeah, we've actually collaborated with BFCL a little bit, which is, I think, the same thing.

• 26:34Speaker 03

  A Berkeley function calling leaderboard.

• 26:36Speaker 04

  Kudos to the team. Those evals are great and we use them internally. Yeah, we've also sent some feedback on some things that are misgraded. But and so we're collaborating to make those better. In general, I feel evals are kind of the hardest part of AI. Like when I talk to developers, it's so hard to get started. It's really hard to make a robust pipeline. And you don't want evals that are like 80% successful because, you know, things are going to improve dramatically. And it's really hard to craft the right eval. You kind of want to hit everything on the difficulty curve. I find that a lot of these evals are mostly saturated, like for BFCL. All the models are near the top already. And kind of the errors are more, I would say, like just differences in default behaviors. I think most of the models on the leaderboard can kind of get 100% with different prompting.

• 27:21Speaker 04

  But it's more kind of you're just pulling apart different defaults at this point. So yeah, I would say in general, we're missing evals. You know, we work on this a lot internally, but it's hard.

• 27:30Speaker 03

  Did you, other than BFCL, would you call out any others just for people exploring this space?

• 27:35Speaker 04

  Sweetbench is actually like a very interesting eval. If people don't know, you basically give the model GitHub issue and like a repo and just see how well it does at the issue, which I think is super cool. It's kind of like an integration test, I would say, for models.

• 27:47Speaker 03

  It's a little unfair, right?

• 27:49Speaker 04

  What do you mean?

• 27:50Speaker 03

  A little unfair because like usually as a human, you have more opportunity to like ask questions about what it's supposed to do.

• 27:56Speaker 04

  Right.

• 27:56Speaker 03

  And you're giving the model like way too little information.

• 27:58Speaker 04

  It's a hard job.

• 27:59Speaker 03

  To do the job.

• 28:00Speaker 04

  Yeah. But yes, we mentioned targets like how well can you follow the diff format? And how well can you like search across files? And how well can you write code? So really excited about evals like that because the pass rate is low. So there's a lot of room to improve.

• 28:12Speaker 03

  Yeah.

• 28:12Speaker 04

  And it's just targeting a really cool capability.

• 28:14Speaker 03

  I've seen other evals for function calling where I think might be BFCL as well, where they evaluate different kinds of function calling. And I think the the top one that people care about, for some reason, I don't know personally that this is so important to me, but it's parallel function calling.

• 28:28Speaker 04

  Right.

• 28:29Speaker 03

  I think you confirmed that you have, don't support that yet. Why is that hard? Just more context about it.

• 28:34Speaker 04

  So yeah, we put out parallel function calling Dev Day last year as well. And it's kind of the evolution of function calling. So function calling V1, you just get one function back. Function calling V2, you can get multiple back at the same time and save latency. We have this in our API, all our models support it, or all of our newer models support it. But we don't support it with structured outputs right now. And there's actually a very interesting trade off here. So when you basically call our API for structured outputs with a new schema, we have to build this artifact for fast sampling later on. But when you do parallel function calling, the kind of schema we follow is not just directly one of the function schemas. It's like this combined schema based on a lot of them. If we were kind of do the same thing and build an index every time you pass in a list of functions, if you ever change the list, you would kind of incur more latency.

• 29:19Speaker 04

  And we thought it would be really unintuitive for developers and like hard to reason about. So we decided to kind of wait until we can support a no added latency solution and not just kind of make it really confusing for developers.

• 29:30Speaker 03

  Mentioning latency, that is something that people discovered is that there is an increased cost in latency for the first token.

• 29:36Speaker 04

  For the first request, yeah.

• 29:37Speaker 03

  First request. Is that an issue? Is that going to go down over time? Is there just an overhead to parsing JSON that is just insurmountable?

• 29:44Speaker 04

  It's definitely not insurmountable and I think it will definitely go down over time. We just kind of take the approach of, and you know, if you, if there's nothing in there you, you don't want to fix, then you probably ship too late. So I think we will get that latency down over time. But yeah, I think for most developers, it's not a big concern because you're testing out your integration. You're, you're sending some requests while you're developing it and then it's fast and prod. So it kind of works for most people. The alternative design space that we explored is like pre-registering your schema. So like a totally different endpoint and then passing in like a schema ID. But we thought, you know, that was a lot of overhead and like another endpoint to maintain and just kind of more complexity for the developer. And we think this latency is going to come down over time. So it made sense to keep it kind of in chat completions.

• 30:28Speaker 03

  I mean, hypothetically, if one were to ship caching at a future point, it would basically be the superset of that.

• 30:34Speaker 04

  Maybe. I think the caching space is a little underexplored. Like we've seen kind of two versions of it. But I think, yeah, there's ways that maybe put less onus on the developer. But you know, we haven't committed to anything yet, but we're definitely exploring opportunities for making things cheaper over, over time.

• 30:50Alessio Fanelli?

  Is AGI in agents just going to be a bunch of structured output and function calling one next to each other? Like, how do you see, you know, there's like the model that's everything. Where do you draw the line? Because you don't call these things like an agent API. But like if I were a startup trying to raise a C round, I would just do function calling and say this is an agent API.

• 31:07Speaker 04

  Yeah.

• 31:07Alessio Fanelli?

  So how do you think about the difference and like how people build on top of it for like agentic systems?

• 31:12Speaker 04

  Yeah, love that question. One of the reasons we wanted to build structured outputs is to make agentic applications actually work. So right now it's really hard, like if something is 95% reliable, but you're chaining together a bunch of calls. If you magnify that error rate, it makes your like application not work. So that's a really exciting thing here from going from like 95% to 100%. I'm very biased working on the API and working on function calling and structured outputs, but I think those are the building blocks that we'll be using to distribute this technology very far. It's the way you connect natural language and converting user intent into working with your application. And so I think there's no way to build without it, honestly. You need your function calls to work. Yeah, we wanted to make that a lot easier.

• 31:53Speaker 03

  And do you think the assistance API thing will be a bigger part as people build agents? I think maybe most people just use messages and completion.

• 32:02Speaker 04

  So I would say the assistance API was a bet in a few areas. One bet is hosted tools. So we have the file search tool and code interpreter. Another bet was statefulness. It's our first stateful API. It'll store threads and you can fetch them later. I would say the hosted tools aspect has been really successful. People love our file search tool and it saves a lot of time to not build your own RAG pipeline. I think we're still iterating on the shape for the stateful thing to make it as useful as possible. Right now, there's a few endpoints you need to call before you can get a run going. And we want to work to make that much more intuitive and easier over time.

• 32:39Speaker 03

  One thing I'm just curious about, did you notice any trade offs? When you add more structured output, it gets worse at some other thing that was you didn't think was related at all.

• 32:49Speaker 04

  Yeah, it's a good question. Yeah, I mean, models are very spiky and RL is hard to predict. And so every model improves on some things and maybe is flat or neutral on other things.

• 33:01Speaker 03

  It's very rare to just add a capability and have no trade offs in everything else.

• 33:06Speaker 04

  So yeah, I don't have something off the top of my head, but I would say, yeah, every model is a special, its own thing. This is why we put them in API dated, so developers can choose for themselves which one works best for them. In general, we strive to continue improving on all evals, but it's stochastic.

• 33:21Speaker 03

  Yeah, able to apply the structured output system on backdated models like 4.0 May as well as Mini, as well as August.

• 33:31Speaker 04

  Actually, the new response format is only available on two models. It's 4.0 Mini. And the new 4.0.

• 33:37Speaker 03

  Okay.

• 33:38Speaker 04

  So the old 4.0 doesn't have the new response format.

• 33:41Speaker 03

  Okay.

• 33:41Speaker 04

  However, for function calling, we were able to enable it for all models that support function calling. And that's because those models were already trained to follow these schemas. We basically just didn't want to add the new response format to models that would do poorly at it. Because they would just kind of do infinite white space, which is, you know, the most likely token if you have no idea what's going on.

• 34:00Speaker 03

  I just wanted to call out a little bit more in the stuff you've did in the blog post. So in blog posts, just use cases, right? I just want people to be like, yeah, we're spelling it out for you. Use these for extracting structured data from unstructured data. By the way, it does vision too.

• 34:12Speaker 04

  Yes.

• 34:13Speaker 03

  So that's cool. Dynamic UI generation. Actually, let's talk about dynamic UI. I think gen UI, I think is something that people are very interested in.

• 34:21Speaker 04

  Yeah.

• 34:21Speaker 03

  It's your first example. What did you find about it?

• 34:24Speaker 04

  Yeah, I just thought it was a super cool capability we have now. So the schemas, we support recursive schemas. And this allows you to do really cool stuff, like, you know, every UI is a nested tree that has children. So I thought that was super cool. You can use one schema and generate like tons of UIs. As a backend engineer who's always struggled with JavaScript and front end, like for me, that's super cool. I've now, we've now built a system where I can get any front end that I want. So yeah, that's super cool. The extracting structured data, like the reality of a lot of AI applications is like you're plugging them into your enterprise business. And you have something that works, but you want to make it a little bit better. And so the reliability gains you get here is like, you'll never get like a classification using the wrong enum. It's like, it's just exactly your, your types.

• 35:09Speaker 04

  So really excited about that.

• 35:11Speaker 03

  Like maybe hallucinate the actual values, right? So let's clearly state what the guarantees are. The guarantees is that it fits the schema, but the schema itself may be too broad. Cause the JSON schema type system doesn't say like, I only want to range from one to 11. You might give me zero. You might give me 12.

• 35:29Speaker 04

  So yeah, JSON schema, so this is actually a good thing to talk about. So JSON schema is extremely vast and we weren't able to support every corner of it. So we kind of support our own dialect and it's described in the docs. And there are a few trade offs we had to make there. So by default, if you don't pass in additional properties in a schema, by default, that's true. And so that means you can get other keys, which you know, you didn't spell out, which is kind of the opposite of what developers want. You basically want to supply the keys and values and you want to get those keys and values. And so then we had to decision to make. It's like, do we redefine what additional properties means as the default? And that felt really bad. It's like, there's a schema that's predated us like, you know, it wouldn't be good. It'd be better to play nice with the community. And so we require that you pass it in as false. You know, one of our design principles is to be very explicit.

• 36:14Speaker 04

  And so developers know, you know, what to expect. And so this is one where we decided, you know, it's a little harder to discover, but we think you should pass this thing in so that we can have like a very clear definition of what you mean and what we mean. There's a similar one here with like required. By default, every key in JSON scheme is optional. But that's not what developers want, right? Like you would, you'd be very surprised if you passed in a bunch of keys and you didn't get some of them back. And so that's the trade off we made is to make everything required and have the developer spell that out.

• 36:41Speaker 03

  Is there a required false? Can people turn it off or they're just getting all?

• 36:45Speaker 04

  So developers can, basically what we recommend for that is to make your actual key a union type. And so

• 36:52Speaker 03

  Nullable.

• 36:52Speaker 04

  Yeah, so make it a union of int and null. And that gets you the same behavior.

• 36:56Speaker 03

  Any other of the examples you want to dive into math, chain of thought?

• 36:59Speaker 04

  Yeah, you can now specify like a chain of thought field before a final answer. This is just like a more structured way of extracting the final answer.

• 37:06Speaker 03

  Yeah.

• 37:07Speaker 04

  One example we have, I think we put up a demo app of this math tutoring example, or it's coming out soon.

• 37:13Speaker 03

  Did I miss it? Oh, okay.

• 37:14Speaker 04

  Basically, it's this math tutoring thing and you put in an equation and you can go step by step and insert. This is something you can do now with structured outputs. In the past, a developer would have to like specify their format and then write a parser and parse out the model's output. It should be pretty hard, but now you just specify like steps and it's an array of steps. And every step you can render and then the user can try it and you can see if it matches and go on that way. So I think it just opens up a lot of opportunities, like for any kind of UI where you want to treat different parts of the model's responses differently. Structured outputs is great for that.

• 37:45Speaker 03

  I remembered my question from earlier. I'm basically just using this to ask you all the questions as a user, as a daily user of the stuff that you put out. So one is a tip that people don't know, and I confronted you on Twitter, which is you respect descriptions of JSON schemas, right? And you can basically use that as a prompt for the field.

• 38:02Speaker 04

  Totally.

• 38:02Speaker 03

  I assume that's blessed and, you know, people should do that.

• 38:04Speaker 04

  Intentional, yeah.

• 38:06Speaker 03

  One thing that I started to do, which I don't, it could be a hallucination of me, is I changed the property name to prompt the model to what I wanted to do. So for example, instead of saying topics as a property name, I would say like brainstorm a list of topics up to five, or something like that, as like a property name. I could stick that in the description as well, but is that too much?

• 38:30Speaker 04

  Yeah, I would say, I mean, we're so early in AI that people are figuring out the best way to do things. And I love when I learn from a developer, like a way they found to make something work. In general, I think there's like three or four places to put instructions.

• 38:44Speaker 03

  Yeah.

• 38:44Speaker 04

  You can put instructions in the system message. And I would say that's helpful for like when to call a function. So it's like, you know, let's say you're building a customer support thing and you want the model to verify the user's phone number or something. You can tell the model in the system message, like, here's when you should call this function. Then when you're within a function, I would say the descriptions there should be more about how to call a function. So really common is someone will have like a date as a string, but you don't tell the model, like, do you want year, year, month, month, day, day? Or do you want that backwards? And that's what a really good spot is for those kind of descriptions. It's like, how do you call this thing? And then sometimes there's like really stuff like what you're doing. It's like name the key by what you want. So sometimes people put like, do not use. And, you know, if they don't want, you know,

• 39:27Speaker 04

  this parameter to be used except only in some circumstances. And really, I think that's the fun nature of this. It's like you're figuring out the best way to get something out of the model.

• 39:35Speaker 03

  Okay, so you don't have an official recommendation is what I'm hearing?

• 39:38Speaker 04

  Well, the official recommendation is, you know, how to call a model, system instructions.

• 39:41Speaker 03

  Exactly, exactly.

• 39:42Speaker 04

  Or when to call function, yeah.

• 39:44Alessio Fanelli?

  Do you benchmark these type of things? So like same with date. It's like description, it's like return it in like ISO eight.

• 39:51Speaker 04

  Yeah.

• 39:51Alessio Fanelli?

  Or if you called the key date in ISO A six zero zero one. I feel like the benchmarks don't go that that deep, but then all the AI engineering kind of community, like all the work that people do, it's like, oh, actually this performs better, but then there's no way to verify.

• 40:06Speaker 04

  Right.

• 40:06Alessio Fanelli?

  You know, like even the, I'm gonna tip you a hundred thousand dollars or whatever, like some people say it works, some people say it doesn't. Do you pay attention to this stuff as you build this? Or are you just like, the model is just gonna get better, so why waste my time running evals on these small, small things?

• 40:21Speaker 04

  Yeah, I would say to that, I would say we basically pick our battles. I mean, there's so much surface area of LLMs that we could dig into and we're just mostly focused on kind of raising the capabilities for everyone. I think for customers, and we work with a lot of customers, really developing their own evals is super high leverage. Cause then you can upgrade really quickly. When we have a new model, you can experiment with these things with confidence. So yeah, we're hoping to make making evals easier. I think that's really generally very helpful for developers.

• 40:49Speaker 03

  For people, I would just kind of wrap up the discussion for structured outputs. I immediately implemented, we use structured outputs for AI news, I use instructor and I ripped it out. And I think I saved twenty lines of code. But more importantly, it was like we cut it by fifty five percent of API cost based on what I, what I measured because of we saved on the retries.

• 41:09Speaker 04

  Nice. Yeah, love to hear that.

• 41:11Speaker 03

  Yeah, which, which people I think don't understand when you, you can't just simply like add instructor or add outlines. You can do that, but it's actually gonna cost you a lot of retries to get the, the model that you want. But you're kind of just kind of building that internally into the model.

• 41:25Speaker 04

  Yeah, I think this is the kind of feature that works really well when it's integrated with like the LLM provider. Yeah, actually I had folks, even my, my husband's company, he works at a small startup, they thought we were just retrying. And so I had to make the response. We are not retrying, you know, we're doing it in one shot and this is how you save on latency and cost.

• 41:43Speaker 03

  Awesome. Any other behind the scenes stuff, just generally on structured outputs? We, we're gonna move on to the other models.

• 41:49Speaker 04

  Yeah, I think that's it.

• 41:50Speaker 03

  Oh, but those are excellent products and I think everyone will be using it. And we have the full story now that people can try out. So roadmap would be parallel function calling, anything else that you've called out as like coming soon?

• 42:01Speaker 04

  Not quite soon, but you know, we're thinking about, does it make sense to expose custom grammars on JSON schema?

• 42:06Speaker 03

  What would you want to hear from developers to give you information? Whether it's custom grammars or anything else about structured output, like what would you want to know more of?

• 42:13Speaker 04

  Just, you know, always interested in feature requests, what's not working. But I'd be really curious, like what specific grammars folks want? I know some folks want to match programming languages like Python. There's some challenges like with the expressivity of our, you know, implementation. And so, yeah, just kind of the class of grammars folks want.

• 42:31Speaker 03

  I have a very simple one, which is a lot of people try to do, use GPT as judge, right? Which means they end up doing a rating system. And then there's like 10 different kinds of rating systems, the Likert scale is whatever. If there was an officially blessed way to do a rating system with structured outputs, everyone would use it.

• 42:48Speaker 04

  Yeah, yeah, that makes sense. I mean, we often recommend using log probes with classification tasks. So rather than like sampling, you know, let's say you have four options like red, yellow, blue, green. Rather than sampling, you know, two tokens for yellow, you can just do like A, B, C, D and get the log probes of those. You know, the inherent randomness of each sampling isn't taken into account. And you can just actually look at what is the most likely token.

• 43:13Speaker 03

  I think this is more like a calibration question. Like if I ask you to rate things from one to 10, a non-calibrated model might always pick 11, just like a human would.

• 43:21Speaker 04

  Right.

• 43:23Speaker 03

  So like actually have a nice gradation from one to 10 would be the rough idea.

• 43:27Speaker 04

  Yeah.

• 43:28Speaker 03

  And then even for structured outputs, I can't just say have a field of rating from one to 10 because I have to then validate it. And you know, it might give me 11.

• 43:35Speaker 04

  Yeah, absolutely.

• 43:36Speaker 03

  Yeah.

• 43:37Alessio Fanelli?

  So what about model selection? Now you have a lot of models. When you first started, you had one model endpoint? I guess you had like DaVinci and then, but like most people were using one model endpoint. Today you have like a lot of competitive models and I think we're nearing the end of the 3.5 run, R I P. How do you advise people to like experiment, select, both in terms of like task and like cost? Like what's your playbook?

• 44:02Speaker 04

  In general, I think folks should start with 4.0 mini. That's our cheapest model and it's a great workhorse, works for a lot of great use cases. If you're not finding the performance you need, like, you know, maybe it's not smart enough, then I would suggest going to 4.0. And if 4.0 works well for you, that's great. Finally, there's some like really advanced frontier use cases. And maybe 4.0 is not quite cutting it. And there I would recommend our fine tuning API. Even just like 100 examples is enough to get started there. And you can really get the performance you're looking for.

• 44:32Speaker 03

  We're recording this ahead of it, but like you're announcing other some fine tuning stuff that people should pay attention to.

• 44:38Speaker 04

  Yeah, actually tomorrow we're dropping our GA for GPT 4.0 fine tuning. So 4.0 mini has been available for a few weeks now and 4.0 is now going to be generally available. And we also have a free training offering for a bit. I think until September 23rd, you get 1 million of free training tokens a day.

• 44:55Speaker 03

  This is already announced, right? Or was, am I talking about a different thing?

• 44:58Speaker 04

  So that was for 4.0 mini and now it's also for 4.0. So we're really excited to see what people do with it. And it's actually a lot easier to get started than a lot of people expect. I think they might need 10s of thousands of examples, but even 100 really high quality ones or 1000 is enough to get going.

• 45:12Speaker 03

  I think people's concerns about fine tuning is that they're kind of locked into a model. And I think you're paving the path for migration of models. As long as they keep their original data set, like they can at least migrate nicely.

• 45:23Speaker 04

  Yeah, I'm not sure what we've said publicly there yet, but we definitely want to make it easier for folks to migrate.

• 45:29Speaker 03

  It's the number one concern, you know, I'm just, you know, it's obvious.

• 45:32Speaker 04

  Absolutely.

• 45:33Speaker 03

  I also want to point people to you have official model selection docs where it's in the guide. We'll put it in the show notes where it says to optimize for accuracy first. So prompt engineering, rag, evals, fine tuning. This was done at Dev Day last year, so I'm just repeating things. And then optimize for cost and latency second. And there's a, there's a few sets of steps for optimizing latency. So people can read up on that stuff.

• 45:54Speaker 04

  Yeah, totally.

• 45:55Speaker 03

  Yeah.

• 45:55Alessio Fanelli?

  We had one episode with Nicolás Carlini from DeepMind. And we actually talked about how some people don't actually get to the boundaries of the model performance. You know, they just kind of try one model and it's like, oh, LLMs cannot do this and they stop.

• 46:08Speaker 04

  Right.

• 46:08Alessio Fanelli?

  How should people get over the hurdle? It's like, how do you know if you hit the model performance or like you hit scale issues, you know? It's like your prompt is not good or like try another model and whatnot. Is there an easy way to do that?

• 46:19Speaker 04

  That's tough. Some people are really good at prompting and they just kind of get it right away. And for others, it's more of a challenge. I think there's a lot we can do to make it easier to prompt our models. But for now, I think requires a lot of creativity and not giving up right away. Yeah. And a lot of people have experience now with chat GPT. You know, before ChatGPT, the easiest way to play with our models was in the playground. But now kind of everyone's played with it with a model of some sort and they have some sort of intuition. It's like, you know, if I tell you my grandma is sick, then maybe I'll get the right output. And and we're hoping to kind of remove the need for that. But playing around with ChatGPT is a really good way to get a feel for, you know, how to use the API as well.

• 46:56Alessio Fanelli?

  Will prompt engineering be here forever? Or is it a dying art as the models get better?

• 47:01Speaker 04

  I mean, it's like the perennial question of software engineering as well. It's like, as the models get better at coding, you know, if we hit a hundred on Sweetbench, what does that mean? I think there will always be alpha and people who are able to like clearly explain what they're trying to build. Most of engineering is like figuring out the requirements and stating what you're trying to do. And I believe this will be the case with AI as well. You're going to have to very clearly explain what you need. And some people are better than others at it. And people will always be building. It's just the tools are going to get far better.

• 47:29Speaker 03

  That's two weeks you release two models. There's GPT four O twenty three four O eight O six. And then there's also ChatGPT for latest. I think people are a little bit confused by that. And then you you issued a clarification that was one's chat tuned and the other is more function calling tuned. Can you elaborate just?

• 47:44Speaker 04

  Yeah, totally. So part of the impetus here was to kind of be very transparent with what's on ChatGPT and in the API. So basically, we're often trading models and they're different use cases. So you don't really need function calling for user defined functions in ChatGPT. And so this gives us kind of the freedom to build the best model for each use case. So in ChatGPT latest, we're releasing kind of this rolling model. The weights aren't pinned as we release new models.

• 48:11Speaker 03

  This is literally what we use.

• 48:12Speaker 04

  Yeah, so it's in what's what's in ChatGPT. So it's very good for like chat style use cases. But for the API broadly, you know, we really tune our models to be good at things that developers want, like function calling and structured outputs. And when a developer builds their application, they want to know that kind of the weights are stable under them. And so we have this offering where it's like, if you're tuning to a specific model and you know your function works, you know it will never change the weights out from under you. And so those are the models we commit to supporting for a long time. And we think those are the best for developers. But we want to give it up, you know, we want to leave the choice to developers like, do you want the chat G P T model or do you want the A P I model? And you have the freedom to choose what's best for you.

• 48:51Speaker 03

  I think it's for people, they do want to pin model versions. So I don't know when they would use chat G P T, like the rolling one, unless they're really just kind of cloning chat G P T. Which is like, why would they?

• 49:05Speaker 04

  I mean, I think there's a lot of interesting stuff that developers can do when unbounded. And so we don't, we don't want to limit them artificially. So it's kind of survival of the fittest, like whichever model is better, you know, that's the one that people should use.

• 49:18Speaker 03

  Yeah, I talked about it to my friends as like, this isn't that new thing, like, and basically OpenAI has never actually shared with you the actual chat G P T model. And now they do.

• 49:28Speaker 04

  Well, it's not necessarily true. Actually, a lot of the models we have shipped have been the same.

• 49:33Speaker 03

  Okay.

• 49:33Speaker 04

  But you know, sometimes they diverge and it's not a limitation we want to stick around.

• 49:38Speaker 03

  Anything else we should know about the new model? I don't think there were, there were, there was no evals announced or anything, but, but people say it's better. I mean, obviously L M S is like way better about on everything, right? It's like number one in the world on.

• 49:49Speaker 04

  Yeah, we're, um, we published some release notes. They're not as in-depth as we want to be yet because we're still, it's still kind of a science and we're learning what actually changes with each model and how can we better understand the capabilities? But we are trying to do more release notes in the future and keep folks updated. But yeah, it's, it's kind of an art and a science right now.

• 50:10Speaker 03

  You need the best evals team in the world to help you figure this out.

• 50:13Speaker 04

  Yeah, evals are hard. We're hiring if you want to come work on evals.

• 50:16Speaker 03

  Hold that thought on hiring. We'll, we'll come back to the end on what you want, what you're looking for. Cause obviously people want to join you and they want to know what qualities you're looking for.

• 50:24Alessio Fanelli?

  So we just talked about A P I versus chat G P T. What's I guess like the vision for the interface? You know, the, the mission of OpenAI is like build A G I that is accessible, like where is it going to come from?

• 50:36Speaker 04

  Totally. Yeah. So I believe that the A P I is kind of our broadest vehicle for distributing A G I. You know, we're building some first party products, but they'll never reach every niche in the world and kind of every corner in community. And so really love working with developers and seeing the incredible things they come up with. I often find that developers kind of see the future before anyone else. And we love working with them to make it happen. And so really, the API is a bet on going really broad. And we'll go very deep as well in our first party products. But I think just that our impact is absolutely magnified by every developer that we uplift.

• 51:08Speaker 03

  They can do the last mile where you cannot. Like ChatGPT is one type of product. There's many other kinds. In fact, you know, I I observed, I think in February, basically ChatGPT's user growth stopped when the API was launched. Because everyone's like, kind of be able to take that and then build other things. That has not become true anymore because ChatGPT growth has continued to grow. But then you're not confirming any of this. This is me quoting similar web numbers, which are have very high variance.

• 51:35Speaker 04

  Well, the API predates ChatGPT. The API was actually opening as first product and the first idea for commercialization that predates me as well.

• 51:42Speaker 03

  Wide release, like GA, everyone can sign up and use it immediately. Yeah, that's what I'm talking about. But but yeah, I mean, I I do believe that. And, you know, that means you also have to expose all of opening I models, right? Like all the multimodal models. We'll ask you questions on that. But like, I I think that that API mission is is important. It's interesting that hottest new programming language is supposed to be English. But it's actually just software engineering, right? It's it's just, you know, we're talking about HTTP error codes.

• 52:10Speaker 04

  Right.

• 52:11Speaker 03

  Like.

• 52:12Speaker 04

  Yeah, I think, you know, engineering is still the way you access these models. And I think there are companies working on on tools to make engineering more accessible for everyone. But there's still so much alpha in just writing code and and deploying.

• 52:25Speaker 03

  Yeah, one might even call it AI engineering.

• 52:27Speaker 04

  Exactly.

• 52:29Speaker 03

  Yeah, so like, there's lots of war stories from from building this platform. We started at the start of your career and then we jumped straight to structured outputs. There's a whole thing, like two years that we skipped in between.

• 52:38Speaker 04

  Right.

• 52:39Speaker 03

  What have become your principles? What are your favorite stories that that you like to tell?

• 52:43Speaker 04

  We had so much fun working on the assistance API and leading up to Dev Day. You know, things are always pretty chaotic when you have an externally, like a dicing function. That is hard and there's like a stage and there's like a thousand people coming.

• 52:56Speaker 03

  You can always launch a waitlist.

• 52:59Speaker 04

  We're trying hard not to.

• 53:00Speaker 03

  Yeah.

• 53:00Speaker 04

  Because, you know, we love it when people can access the thing on day one. And so, yeah, the assistance API, we had like this really small team and. Just working as hard as we could to make this come to life. But even actually the morning of, I don't know if you'll remember this, but Sam did this keynote.

• 53:16Speaker 03

  Yep.

• 53:17Speaker 04

  And Ramon came up and they gave free credits to everybody.

• 53:20Speaker 03

  Mm hmm.

• 53:21Speaker 04

  So that was live, fully live, as were all of the demos that day. But actually, maybe like two hours before that, we had a little outage and everyone was like scrambling to make this thing work again. So yeah, things are early and scrappy here and, you know, we were really glad. We were a bit on the edge of our seat, watching it live.

• 53:39Alessio Fanelli?

  What's the plan B in that situation? If you can share.

• 53:41Speaker 03

  Play a video, this is classic DevRel, right? I don't know.

• 53:44Speaker 04

  I mean, I actually don't know what the plan B was.

• 53:47Speaker 03

  No plan B, no failure.

• 53:49Speaker 04

  But we just, you know, we fixed it, we got everything running again and the demo went well.

• 53:54Speaker 03

  Just higher cracked Waterloo.

• 53:55Alessio Fanelli?

  Exactly. Skill issues as usual.

• 53:58Speaker 04

  Sometimes you just gotta make it happen.

• 54:00Speaker 03

  I imagine it's actually very motivating, but I did hear that after Dev Day, like the whole company got like a few weeks off just to relax a little bit.

• 54:08Speaker 04

  Yeah, we sometimes get, like, we just had the week of July 4th off and yeah. It's hard to take vacation because people are working on such exciting things and it's like you get a lot of FOMO on vacation. So it helps when the whole company's on vacation.

• 54:20Speaker 03

  Mentioning Assistant's API, you actually announced a roadmap there and things have developed. I think people may not be up to date. What's the offering today versus, you know, one year ago?

• 54:31Speaker 04

  Yeah, so we've made a bunch of key improvements. I would say the biggest one is in the file search product. Before, we only supported, I think, like 20 files per assistant. And the way we use those files was like less effective. Basically, model would decide based on the file name whether to search a file and there's not a ton of information in there. So our new offering, which we shipped a few months ago, I think, now allows 10K files per assistant, which is like dramatically more. And also, it's a kind of different operation. So you can search semantically over all files at once, rather than just kind of the model choosing one upfront. So a lot of customers have seen really good performance. We also have exposed more like chunking and re-ranking options. I think the re-ranking one is coming, I think, next week or very soon.

• 55:13Speaker 04

  So this kind of gives developers more control and more flexibility there. So we're trying to make it the easiest way to kind of do RAG at scale.

• 55:21Speaker 03

  Yeah, I think that visibility into the RAG system was the number one thing missing from Dev Day. And then people got their first impressions and then they never looked at it again. So that's important. The rerankor is a core feature of, let's say, some other foundation model labs. Is OpenAI going to like offer a reranking service, a rerankor model?

• 55:41Speaker 04

  So we do reranking as part of it. I think we're soon going to ship more controls for that.

• 55:46Speaker 03

  Okay, got it. And like if I'm an existing Langchain, Llama Index, whatever, how do you compare? Do you make different choices? Like, where does that exist in the spectrum of choices?

• 55:56Speaker 04

  I think we are just coming at it, trying to be the easiest option. And so ideally, like, you don't have to know what a rerankor is. And you don't have to have a chunking strategy. And the thing just kind of works out of the box. So I would say that's where we're going. And then, you know, giving controls to the power users to to make the changes they need.

• 56:14Speaker 03

  Awesome. I want to ask about a couple of other things, just updates on stuff also announced at Dev Day. And we talked about this before. Determinism, something that people really want. Dev Day, we announced the seed param as well as system fingerprint. And like objectively, I've heard issues.

• 56:28Speaker 04

  Yeah.

• 56:28Speaker 03

  I don't know what's going on.

• 56:29Speaker 04

  Yeah, the seed parameter is not fully deterministic. And it's kind of a best effort thing.

• 56:34Speaker 03

  Yeah.

• 56:35Speaker 04

  So you'll notice there's more determinism in the first few tokens. That's kind of the current implementation. We've heard a lot of feedback. We're thinking about ways to make it better, but it's challenging. It's kind of training off against, you know, reliability and uptime.

• 56:46Alessio Fanelli?

  Other maybe underrated API only thing, logic bias. That's another thing that kind of seems very useful. And then maybe most people are like, it's a lot of work. I don't want to use it. Do you have any examples of like use cases or like products that are made a lot better?

• 57:01Speaker 04

  Yeah.

• 57:02Alessio Fanelli?

  Through using it?

• 57:03Speaker 04

  So yeah, classification is the big one. So logic bias, your valid classification outputs. And, you know, you're more likely to get something that matches. We've seen it, people logic bias, like punctuation tokens, maybe trying to get more succinct writing. Yeah, it's generally very much a power user feature. And so not a ton of folks use it.

• 57:22Speaker 03

  I actually wanted to use it to reduce the incidence of the word delve.

• 57:25Speaker 04

  Yeah.

• 57:26Speaker 03

  Have people done that?

• 57:28Speaker 04

  Probably. I don't know. Is that one token? You're probably, you gotta do a lot of permutations.

• 57:33Speaker 03

  It's used so much.

• 57:34Speaker 04

  Maybe it is. Depends on the tokenizer.

• 57:36Alessio Fanelli?

  Are there non-public tokenizers? I guess you cannot answer or you would admit it. Are the 100K and 200K vocab apps like the ones that you use like Russell models or?

• 57:44Speaker 04

  Yeah, I think we have docs that publish more information. I don't have it off the top, but I think we publish which tokenizers for which model.

• 57:52Alessio Fanelli?

  Okay, so those are the only two.

• 57:53Speaker 03

  Rate, the tiering rate limiting system. I don't think there was an official blog post kind of announcing this, but it was kind of mentioned that like you started tying like fine tuning to tiering and like feature rollouts. Just on the, from your point of view, like, how do you manage that? And like, what should people know about the tiering system and rate limiting?

• 58:11Speaker 04

  Yeah, I think basically the main changes here were to be more transparent and easier to use. So before developers didn't know what tier they're in. And now you can see that in the dashboard. I think it's also, I think we publish like how you move from tier to tier. And so this just helps us do kind of gated rollouts for the fine tuning launch. I think everyone tier two and up has full access.

• 58:31Speaker 03

  That makes sense that, you know, I would just advise people to just get to tier five as quickly as possible.

• 58:36Speaker 04

  Sure.

• 58:37Speaker 03

  Like Gold Star customer, you know, like, I don't know. It seems to make sense.

• 58:39Alessio Fanelli?

  Do we want to maybe wrap with future things? And kind of like how you think about designing and everything. So you just mentioned you want to be the easiest way to basically do everything. What's the relationship with other people building in the developer ecosystem? Like, I think maybe in the early days, it's like, okay, we only have these APIs and then everybody helps us. But now you're kind of building a whole platform. How do you make decisions?

• 59:02Speaker 04

  Yeah, I think kind of the 80 20 principle applies here. We'll build things that kind of capture, you know, 80% of the value and maybe leave the long tail to other developers. So we really prioritize by like, how much feedback are we getting? How much easier will make this, will this make something like an integration for a developer? So yeah, we want to do more in this space and not just be an LLM as a service, but kind of AI development platform as a service.

• 59:25Speaker 03

  Oh, okay. That ties into a thing that I put in the, in the notes that we prepped. There are other companies trying to be AI development platform. So you will compete with them or they just want to know what you won't build so that they can build it.

• 59:41Speaker 04

  Yeah, it's a tough question. I think we haven't, you know, determined what exactly we will and won't build. But you can think of something, if it makes it a lot easier for developers to integrate, you know, it's probably on our radar and we'll, you know, stack rank by impact.

• 59:54Speaker 03

  Yeah, so there's like cost tracking and model fallbacks. Model fallbacks is an interesting one because people do it. I don't think it adds a ton of value, but like if you don't build it, I have to build it. Because if one API is down or something, I need to fall back to another one.

• 1:00:09Speaker 04

  Yeah, I mean, the way we're targeting that user need is just by investing a lot in reliability. And so we

• 1:00:15Speaker 03

  Just don't fail.

• 1:00:17Speaker 04

  I mean, we have improved our uptime like pretty dramatically over the last year. And it's been, you know, the result of a lot of hard work from folks. So you'll see that on our status page and our continued commitment going forward.

• 1:00:28Alessio Fanelli?

  Is the important thing about owning the platform that it gives you the flexibility to put all the kind of messy stuff behind the scenes? Or yeah, how do you draw the line between what you want to include?

• 1:00:39Speaker 04

  Yeah, I just think of it as like, how can we onboard the next generation of AI engineers? As you put it, right? Like, what's the easiest way to get them building really cool apps? And I think it's by building stuff to kind of hide this complexity or just make it really easy to integrate. So I think of it a lot as like, what is the value add we can provide beyond just the models? That makes the models really useful.

• 1:01:00Speaker 03

  Okay, we'll touch on four more features of the API platform that we prepped. Batch, vision whisper and then team enterprise stuff. So you wanted to talk about batch.

• 1:01:08Speaker 04

  Yeah.

• 1:01:08Speaker 03

  So the rough idea is you give a, the contract between you and me is that I give you, give you the batch job. You have 24 hours to run it and it's kind of like spot inst for

• 1:01:18Speaker 04

  Sort of.

• 1:01:18Speaker 03

  For the API, what should people know about it?

• 1:01:22Speaker 04

  So it's half off, which is a great savings. It also works with like 4.0 mini. So the savings on top of 4.0 mini is pretty crazy, like the stuff you can do.

• 1:01:31Speaker 03

  7.5 cents or something per million.

• 1:01:33Speaker 04

  Yeah, I should really have that number top of mind, but it's like staggeringly cheap. And so I think this opens up a lot more use cases. Like, let's say you have a user activation flow and you want to send them an email like maybe every day. Or like at certain points in their user journey. So now you can do this with the batch API. And something that was maybe a lot more expensive and not feasible is now very easy to do. So right now, we have this 24-hour turnaround time for half off. And curious would love to hear from your community, like, what kind of turnaround time do they want?

• 1:02:02Speaker 03

  I would be an ideal user of Batch, and I cannot use Batch because it's 24 hours. I need two to four.

• 1:02:06Speaker 04

  Two to four hours, okay. Yeah, that's good to know. But yeah, just a lot of folks haven't heard about it. It's also really great for like evals, running them offline. You don't generally don't need them to come back within, you know, two hours.

• 1:02:17Speaker 03

  I think you could do a range, right? Two to four for me, like, I need to produce a daily thing. And then 24 for like the the average use case, and then maybe like a week, a month, who cares?

• 1:02:25Speaker 04

  Yeah.

• 1:02:25Speaker 03

  For people who just have a lot to do.

• 1:02:27Speaker 04

  Yeah, absolutely. So yeah, that's Batch API. I think folks should use it more, it's pretty cool.

• 1:02:32Alessio Fanelli?

  Is there a future in which, like, six months is like free? You know, like, is there like small, is there like super small, like shards of like GPU runtime that like over a long enough timeline, you can just run all these things for free?

• 1:02:46Speaker 04

  Yeah, it's certainly possible. I think we're getting to the point where a lot of these are like almost free.

• 1:02:51Alessio Fanelli?

  That's true. You're already close.

• 1:02:52Speaker 03

  Why would they work on something that's like completely free? I don't know. Okay, so Vision, Vision got G8. Last year, people were so wild by the GPT-4 demo, and that was primarily Vision. What was it like building the Vision API?

• 1:03:03Speaker 04

  Yeah, the Vision API is super cool. We have a great team working there. I think the cool thing about Vision is that it works across our APIs. So there's, you can use it in the Assistants API, you can use it in the Batch API and chat completions. It works with structured outputs. I think it just helps a lot of folks with kind of data extraction where, you know, the spatial relationships between the data is too complicated and you can't get that over text. But yeah, there's a lot of really cool use cases.

• 1:03:27Speaker 03

  I think the tricky thing for me is understanding how frequent to turn Vision into from like single images into like effectively just always watching. And right now, I think people just like send a frame every every second. Will that model ever change? Would there just be like, I stream you video? And then?

• 1:03:45Speaker 04

  Yeah, I think it's very possible that we'll have an API where you stream video in. And maybe, you know, to start, we'll do the frame sampling for you.

• 1:03:53Speaker 03

  Is it frame sampling is the default, right?

• 1:03:56Speaker 04

  Right.

• 1:03:56Speaker 03

  But I feel like it's hacky.

• 1:03:57Speaker 04

  Yeah, I think it's hard for developers to do. And so, you know, we should definitely work on making that easier.

• 1:04:02Alessio Fanelli?

  Is there in the Batch API, do you have like a time guarantees, like order guarantees? Like if I send you a batch request of like a video analysis, I need every frame to be done in order?

• 1:04:12Speaker 04

  For batch, you send like a list of requests and each of them stand alone. So you'll get all of them finished, but they don't kind of chain off each other.

• 1:04:20Alessio Fanelli?

  Well, if you're doing a video, you know, if you're doing like analyzing a video.

• 1:04:24Speaker 03

  I wasn't linking video to batch, but that's interesting.

• 1:04:27Alessio Fanelli?

  Yeah, well, a video is like, you know, if you have a very long video, you can just do a batch of all the images and let it process.

• 1:04:34Speaker 04

  It's like batch, but serially.

• 1:04:35Speaker 03

  Sequential true.

• 1:04:36Alessio Fanelli?

  Yeah, yeah, yeah. But the whole point of batch is you're just using kind of spare time to run it. Let's talk about my favorite model, Whisper. Oliver, I built this thing. Yeah, I built this thing called SmallPodcaster, which is an open source tool for podcasters. And why does Whisper API not have diarization when everybody is transcribing people talking? That's my main question.

• 1:04:58Speaker 04

  Yeah, it's a good question, and you've come to the right person. I actually worked on the Whisper API and shipped that. That was one of my first APIs I shipped. Long story short is that Whisper V3, which we open sourced, has, I think, the diarization feature. But there's some like performance trade offs. And so Whisper V2 is better at some things than Whisper V3. And so it didn't seem that worthwhile to ship Whisper V3 compared to like the other things in our priorities. I think we still will at some point. But yeah, it's just, you know, there's always so many things we can work on. It's tough to do everything.

• 1:05:29Alessio Fanelli?

  We have a Python notebook that does the diarization for the pod, but I would just like, you can translate like 50 languages. But you cannot tell me who's speaking. That was like the funniest thing.

• 1:05:40Speaker 04

  There's like an X K C D thing about this, about hard problems in AI. It's forget the word.

• 1:05:45Alessio Fanelli?

  Yeah, exactly.

• 1:05:46Speaker 04

  Tell me if this was taken in a park. And like, that's easy. And it's like, tell me if there's a bird in this picture. And it's like, give me 10 people and a research team. It's like, you never know which things are challenging and diarization is, I think, you know, more challenging than than expected.

• 1:05:59Alessio Fanelli?

  Yeah, it still breaks a lot with like overlaps, obviously. Sometimes similar voices it struggles with, like I need to like double read the thing.

• 1:06:07Speaker 04

  Totally.

• 1:06:08Alessio Fanelli?

  But yeah, great model. I mean, it would take us so long to do transcriptions. And I don't know why, like SmallPodcast has better transcription than like mostly every commercial tool.

• 1:06:17Speaker 03

  It beats the script.

• 1:06:19Alessio Fanelli?

  And I'm like, I'm just using the model. I'm literally not doing anything, you know, it's just a notebook. So yeah, it just speaks to like, sometimes just using the simple OpenAI model is better than like figuring out your own pipeline thing.

• 1:06:30Speaker 04

  Totally.

• 1:06:31Speaker 03

  I think the top feature request there just would be, I mean, say, again, you know, using you as a feature request dump.

• 1:06:37Speaker 04

  Yeah, totally.

• 1:06:37Speaker 03

  Is like being able to bias the vocab. I think there is, like, in raw whisper, you can do that.

• 1:06:42Speaker 04

  You can pass a prompt in the API as well.

• 1:06:44Speaker 03

  In, but you pass in a prompt, okay.

• 1:06:45Speaker 04

  Yeah.

• 1:06:46Speaker 03

  There's no more deterministic way to do it.

• 1:06:47Speaker 04

  So you, this is really helpful when you have like acronyms that aren't very familiar to the model. And so you can put them in the prompt and you'll, you'll basically get the transcription using those correctly.

• 1:06:57Alessio Fanelli?

  We have the AI engineer solution, which is just a dictionary.

• 1:07:00Speaker 04

  Nice.

• 1:07:00Alessio Fanelli?

  We're like all the way misspelt it in the past and then G sub and like replace the.

• 1:07:05Speaker 04

  If it works, it works. Like that's, that's engineering.

• 1:07:07Alessio Fanelli?

  Okay. It's like, you know, llama with like one L or like all, all these different things or like Langchain and like transcribes Langchain and like.

• 1:07:15Speaker 03

  Capitalization does.

• 1:07:16Alessio Fanelli?

  A bunch of like three or four different ways.

• 1:07:18Speaker 03

  Yeah.

• 1:07:18Speaker 04

  You guys should try the prompt feature.

• 1:07:20Speaker 03

  I love these like kind of pro tip. Okay. Fun question. I know we don't know yet, but I've been enjoying the advanced voice mode. It really streams back and forth and it handles interruptions. How would your audio endpoint change when, when that comes out?

• 1:07:34Speaker 04

  We're exploring, you know, new shape of the API to see how it would work, would work in this kind of speech to speech paradigm. I don't think we're ready to share quite yet, but we're definitely working on it. I think just the regular request response probably isn't going to be the right solution.

• 1:07:48Speaker 03

  For those who are listening along, I, I think it's pretty public that OpenAI uses LiveKit for the, for the ChatGPT app, which like seems to be the socket based approach that people should be at least up to speed on. Like, I think a lot of developers only do request response and like that doesn't work for streaming.

• 1:08:03Speaker 04

  Yeah. When we do put out this API, I think we'll make it really easy for developers to figure out how to use it. Yeah. It's, it's hard to do.

• 1:08:09Speaker 03

  It'll be a paradigm change. Okay. And then I think the last one on all this was team enterprise stuff. Audit logs, service accounts, API keys. What should people know is in the enterprise offering?

• 1:08:18Speaker 04

  Yeah, we recently shipped our admin and audit log APIs. And so a lot of enterprise users have been asking for this for a while. The ability to kind of manage API keys programmatically, manage your projects, get the auto log. So we've shipped this and, and for folks that need it, it's, it's out there and happy for your feedback.

• 1:08:34Speaker 03

  Yeah. Awesome. I don't use them. So I don't know. I, I imagine it's just like build your own internal gateway for, you know, your internal developers to manage your deployment of OpenAI.

• 1:08:44Speaker 04

  Yeah. I mean, if you work at like a company that needs to keep track of all the API keys, it was, it was pretty hard in the past to do this in the dashboard. We've also improved our SSO offering, so that's much easier to use now.

• 1:08:55Alessio Fanelli?

  The most important feature of an enterprise company.

• 1:08:58Speaker 04

  Love SSO.

• 1:09:00Alessio Fanelli?

  All right, let's go outside of OpenAI. What about just you personally? So you mentioned Waterloo. Maybe let's just do why is everybody in Waterloo cracked? And why are people so good? And like, why have people not replicated it? Or any other commentary on your experience?

• 1:09:15Speaker 04

  The first is the COOP program. It's obviously really good. You know, I did six internships, learned so much in those. I think another reason is that Waterloo is like, you know, it's very cold in the winter. It's pretty miserable. There's like, not that much to do apart from study and like hack on projects. And there's this big like hacker mentality, you know? There's a Hack the North is a very popular hackathon. And there's a lot of like startup in computers. This kind of just has this like startup and hacker ethos. Then that combined with the six internships means that you get people who like graduate with two years of experience. And they're very entrepreneurial and you know, they're down to grind.

• 1:09:51Speaker 03

  I do notice a correlation between climate and the correctness of engineers. So you know, it's no coincidence that Seattle is the birthplace of Microsoft and Amazon. I think I had this compilation of Denmark where people like, so it's the birthplace of C++, PHP, Turbo Pascal, standard ML, BNF, the thing that we just talked about, MD5 Crypt, Ruby on Rails, Google Maps, and V8 for Chrome. And it's because according to Bjorn Sjostrup, the creator of C++, there's nothing else to do.

• 1:10:20Alessio Fanelli?

  Well, yeah, Lena Storebalts in Finland.

• 1:10:23Speaker 04

  I mean, you hear a lot about this, like in relation to SF. People say, you know, New York is way more fun. There's nothing to do in SF. And maybe it's a little by design that all tech is here.

• 1:10:32Alessio Fanelli?

  The climate is too good. If we also have fun things to do.

• 1:10:35Speaker 03

  Nature is so nice, you can touch grass. Why are we not touching grass?

• 1:10:38Speaker 04

  You know, restaurants close at like 8pm. Like, that's what people are referring to. There's not a lot of like late night dining culture. Yeah, so you have time to wake up early and get to work.

• 1:10:49Speaker 03

  You are a book recommender or book enjoyer. What underrated books do you recommend most others?

• 1:10:54Speaker 04

  Yeah, I think a book I read somewhat recently that was very formative was The Making of the Prince of Persia. It's a Stripe press book. That book just made me wanna work hard like nothing I've ever read. It's just like this journal of what it takes to like build, you know, incredible things. So I'd recommend that.

• 1:11:10Speaker 03

  Yeah, it's funny how video games are for a lot of people, at least for me, kinda like the, some of the moments in technology. Like when I played The Sense of Time on PS2, it was like my first PlayStation 2 game. And I was like, man, this thing is so crazy compared to any PlayStation 1 game. And it's like, wow, my expectations for like the technology, I think like OpenAI is a lot of similar things like the advanced voice. It's like you see that thing and then you're like, okay, what I can expect from everybody else is kinda raised now, you know?

• 1:11:37Speaker 04

  Totally. Another book I like to plug is called Misbehaving by Richard Thaler. He's a behavioral economist and talks a lot about how people act irrationally in terms of decision making. And I actually think about that book like once a week, probably at least when I'm making a decision and I realize that, you know, I'm falling into a fallacy or, you know, it could be a better decision.

• 1:11:56Speaker 03

  Yeah, you did a minor in psych.

• 1:11:57Speaker 04

  I did, yeah. I don't know if I learned that much there, but it was interesting.

• 1:12:02Speaker 03

  Is there like an example of like a cognitive bias or misbehavior that you just love telling people about?

• 1:12:08Speaker 04

  Yeah, people, so let's say you won tickets to like a Taylor Swift concert. And I don't know how much they're going for, but it's probably like $10,000.

• 1:12:16Speaker 03

  Oh, okay.

• 1:12:17Speaker 04

  Or whatever, sure. And like a lot of people are like, oh, I have to keep these, like I won them, it's $10,000. But really, it's the same decision you're making. If you have $10,000, like would you buy these tickets? And so people don't really think about it rationally and like, would they rather have $10,000 of the tickets? For people who want it, a lot of the time, it's gonna be the $10,000, but their bias is because they want it. The world organized itself this way and you should keep it for some reason.

• 1:12:39Speaker 03

  Yeah, oh, okay, I'm pretty familiar with this stuff. There's also a loss version, loss aversion version of this where it's like, if I take it away from you, you respond more strongly than if I give it to you.

• 1:12:48Speaker 04

  Yes. If people are like really upset if they like don't get a promotion, but if they do get a promotion, they're like, okay, phew. It's like not even, you know, excitement, it's more like we react a lot worse to losing something.

• 1:13:01Speaker 03

  Which is why like when you join like a new platform, they often give you points and then they'll take it away if you like don't do some, some action in like the first few days.

• 1:13:10Speaker 04

  Yeah, totally, yeah. The book references people who like operate very rationally as econs, as like a separate group to humans. And I often think like, you know, what would an econ do here in this moment and try to act that way?

• 1:13:24Speaker 03

  Okay, let's, let's do this, our LLMs, econs.

• 1:13:28Speaker 04

  I mean, they are maximizing probability distributions.

• 1:13:32Speaker 03

  Minimizing loss.

• 1:13:33Speaker 04

  Yeah, so I think way more than all of us, they are econs.

• 1:13:36Speaker 03

  Whoa, okay, so they're more rational than us?

• 1:13:39Speaker 04

  Hmm, I think their optimization functions are more clear than ours.

• 1:13:42Alessio Fanelli?

  Yeah, just to wrap, you mentioned you need help on a lot of things.

• 1:13:46Speaker 04

  Yeah.

• 1:13:46Alessio Fanelli?

  Any specific roles, callouts, and also people's backgrounds? Like, is there anything that they need to have done before? Like, what people fit well at OpenAI?

• 1:13:54Speaker 04

  Yeah, we've hired people of all kinds of backgrounds. People who have PhD in ML or folks who've just done engineering like me. And we're really hiring for a lot of teams. We're hiring across the applied org, which is where I sit for engineering, and for a lot of researchers. And there's a really cool model behavior role that we just dropped. So, yeah, across the board, we'd recommend checking out our careers page. And you don't need a ton of experience in AI specifically to join.

• 1:14:20Speaker 03

  I think one thing that I'm trying to get at is like, what kind of person does well at OpenAI? I think objectively, you have done well. And I've seen other people not do as well and basically be managed out. I know it's an intense environment.

• 1:14:33Speaker 04

  I mean, the people I enjoy working with the most are kind of low ego, do what it takes, ready to roll up their sleeves, do what needs to be done and unpretentious about it. Yeah, I also think folks that are very user focused do well on kind of API and chat GPT. Like, the YC ethos of build something people want is very true at OpenAI as well. So I would say low ego, user focused, driven.

• 1:14:57Alessio Fanelli?

  Cool, yeah, this was great. Thank you so much for coming on.

• 1:15:00Speaker 04

  Thanks for having me.

• 1:15:01Speaker 01

  That was an excellent conversation with Michelle, which were just about to ship before we then heard that the OpenAI strawberry, now named O1 Release, was coming. So we delayed our podcast to organize an emergency meetup with more friends from OpenAI and latent space enjoyers in San Francisco to capture quick takes on O1, as well as answer some questions.

• 1:15:32Speaker 00

  Letting space news all once begun, Left Michelle and GPT for all just for fun. Strawberry fields where the bites all run, Scaling up inference for second to none. Reset the counter, GPT's passe. O1's the hot shot, the A G I way. Not a system, just one model that's on the rise. An extraordinary alien of unknown size. O1 minis, it'd be quick. O1 previews, just a tease. Encoded STEM, minis the bees' knees. Token counts the same, GPT for those old news. Hiding thought chains like a cognitive Letting space news all once begun, Left Michelle and GPT for all just for fun. Strawberry fields where the bites all run, Scaling up inference for second to none. Bigger context loading, no more scrolls.

• 1:16:17Speaker 00

  Handling long tasks, chunking's got holes. Tools not in the kit yet, the functions are call. Multimodal magic, O1's on the ball. RL's the secret sauce in our code group. GPT for all can catch up even in debug mode. You think it seems slow, because we summarize the gist. But when answers drop, they're faster than the rest. Hiding firm, there O1's the gun. Left Michelle and GPT for all just for fun. Strawberry fields where the bites all run. Scaling up inference for second to none.

• 1:16:54Speaker 01

  There are two sections to our O1 coverage today. The first is our recorded meetup audio from Thursday, which will also be posted on YouTube, recapping the most important takeaways about O1 from SWICS. And then, an open-ended Q and A session with Romaine Hewitt and other OpenAI representatives. We apologize in advance for the audio quality, but it was the best we could gather on short notice. The second is a recap of the official OpenAI Developers AMA on Friday, which you'll hear about from me. Finally, you'll hear some O1 demos from Min Kim of Lighthouse and your favorite latent space co-host Alessio. Stay tuned.

• 1:17:37Speaker 03

  Yeah, thanks everyone for joining this emergency O1 meetup. And thank you to OpenAI for the model and the food. Very much appreciated for feeding our brains and our stomachs. So I'm just going to yap a little bit for the AI news recap that we did today. You know, the one thing I really appreciate is that they generally tend to make it API available, even if the API can be surprisingly hard to work with. So anyway, so there are a few data points that I'm just going to recap. And then we'll let the actual experts answer any questions or at least say a few words as well. So there are effectively like two models released today. I like to show it like this in this chart because I think that people don't really understand that the O1 preview is really just a checkpoint to a broader thing.

• 1:18:22Speaker 03

  And they already have, and that one's still in process.

• 1:18:26Speaker 03

  The other notable thing that a lot of people are talking about that was in our summary of Discord discussion and Twitter discussion today was the pricing. That's notably a little bit higher than, for example, Opus.

• 1:18:41Speaker 03

  And we'll talk about the other stuff. There were a bunch of blog posts. It was actually a little bit hard to find or read. But this is what I think everyone should read. If you signed up to the newsletter, you would have got it. So there's each model has its own blog post. There's a technical research blog post. There's a system card. And then there's also docs. And then there's some other videos if you care about the humans behind the project. I tried to collect together the, what the alpha that people drop. Because obviously that is less. The main thing, so this is actually, if you remember the goodwill hunting, goodwill hunting board problem, I gave it that and it solved it. And so my, the tweet that I was going to work on was basically, O1 is Matt Damon, then it can solve the goodwill hunting problem.

• 1:19:23Speaker 03

  And the new paradigm really is in ChatGPT, it shows this chain of thought. As far as I understand, in the API, it doesn't actually show that. And this chain of thought is only shown in ChatGPT. And it's not the actual chain of thought. It is a hidden, obscured summary of the chain of thought. And that is materialized as reasoning tokens, which you do pay for. The token limits are expanded. And this was the nastiest part about me actually using the API today was that I could not just one for one port over the code. Things like instructor no longer work because there's no system role anymore. There's no temperature, there's no two coloring, there's no streaming. And especially there's no max tokens because you have no constraint or visibility on how many tokens are being used. But there's a lot of really interesting information in the docs that gives you some

• 1:20:07Speaker 03

  mental model of how these reasoning tokens are given you and how if they take too many tokens and I ran it into exactly this for this task because it only solved the first problem and it could not solve second, third, fourth because it ran into the 128 tokens issue. Okay, the third thing that I want to talk about is also the evals. We're very used to really good evals, but I think this is obviously again new highs in a lot of the categories that we're used to. The thing I think that is more important than just the evals is also scaling laws for test time compute. This one, I really recommend people check out Jim Phan's comments on this, which is just an excerpt from the technical blog post that we now not only do train time compute,

• 1:20:51Speaker 03

  but also test time compute. And they seem to scale in a logarithmic fashion that we can actually predict and invest in. And so that seems to be very, very helpful. As far as I understand, I basically covered, I looked at the entire strawberry team's Twitter, which is the unofficial documentation. I really, I think the main thing that people are trying to figure out is how much can you do in the wild versus must you train the models to do this? And the only hints that we have so far is that they're definitely doing change without using RL and it's much better than just prompting. And that's something that obviously as engineers, we don't have access to. You can read Jason's post for the rest.

• 1:21:37Speaker 03

  And then finally, because I don't want to take up too much time, it seems like a lot of people are calling out O1 Mini for its relative outperformance. Usually, when there's two models, one big, one small, or small, medium, big, usually the big model gets all the love and then the other stuff is just sizably smaller. But O1 actually outperforms, if you go all the way back. Lucas from DeepMind actually called out like, "If this is true, why do you use preview, O1 preview at all? You should just use Mini or O1." And that's because O1 doesn't exist yet. But Mini basically scores better in STEM because it was trained to do that. That does not mean that it is better than O1 preview. It's just trained to do better in STEM. So that's my quick recap. We're lucky enough to have a couple of people from OpenAI here. Do you want to say a few words?

• 1:22:24Speaker 03

  What people should know? Yeah, I'll just leave it open. And here, you can take this if you want, just kind of. Just to magnet it. Cool. All right.

• 1:22:35Speaker 05

  Hey, everyone. Thank you so much, swyx, for organizing this like emergency O1 meetup.

• 1:22:42Speaker 05

  Thank you, Sean. So thank you, Vipu, MindDB, Decibol for hosting. But yeah, super exciting day at OpenAI with the release of O1. Sounds like Sean already gave you all the lowdown on the launch day. But yeah, it's a new day for us. I think we're very excited to see what you builders, developers, are going to start building with this model, in particular, agents or agentic apps that maybe before you had some challenges making them work. So yeah, we're very, very jazzed at the idea of like you kicking the tires of the model. It's obviously the very beginning. We have a lot of work ahead of us to give access to everyone and increase the rate limits and so on and so forth. But yeah, very excited to be here and can't wait to see the demos tonight and in the days to come. We have a few members, by the way, of the team tonight. There's Lindsay from our comms team.

• 1:23:26Speaker 05

  There's Edwin also from our developer experience team. But yeah, excited. And Anuj from our engineering team as well. So yeah, excited to be here and answer questions and can't wait to see the demos. Thanks again.

• 1:23:38Speaker 06

  Yeah.

• 1:23:45Speaker 05

  That is a great question. I think, Lindsay, you want to take that one?

• 1:23:48Speaker 07

  So we decided to reset the counter with O1 because the capabilities that this model presents are just totally new and different from the GPT series. So we're still going to invest in the GPT series, like 4O is not going away. We're still very much invested in multi-variety, but we wanted to represent sort of this shift in capabilities towards reasoning, which is just something that, while models have shown us that we're able to reason by like stitching things together in the past, this is just connecting. So we started over with O1. O stands for OpenAI, not O like OmniModel and GPT4O. You know that we're not great at naming models, it's really hard.

• 1:24:34Speaker 03

  Just on the naming, I'll just follow. I read someone saying somewhere, I cannot find this source, that there was no GPT-5 and like, is this a, correct?

• 1:24:45Speaker 03

  Like, is it R I P GPT or?

• 1:24:48Speaker 07

  We're still going to invest in the GPT series. I don't know if we're going to have a GPT-5, but this is O1, it's different.

• 1:24:54Speaker 06

  I think you should expect us to continue doing the kinds of model historically people were thinking would be GPT-5, but we actually call it as a name.

• 1:25:03Speaker 03

  Something else.

• 1:25:04Speaker 06

  But again, our naming has always been confusing, so just a little bit.

• 1:25:09Speaker 08

  They're bad at names.

• 1:25:10Speaker 07

  Everyone is kind of bad at naming models.

• 1:25:13Speaker 06

  So it's like similar to VSX to Akos 11.

• 1:25:17Speaker 07

  Yeah, kind of like that, yeah. You can think of it that way.

• 1:25:20Speaker 08

  Question, how would you just phrase it? So for people that don't follow super closely, there's the 4O series, the traditional 4, these new ones, really the category.

• 1:25:29Speaker 05

  I mean, you can think about like the GPT4O series, for instance, like that's your workhorse. Like, we still expect developers, builders to rely on GPT4O heavily, and we keep on investing in that series for sure. And O1 will be for this new class of apps and that require like a steps of reasoning, for instance. On the API team and the developer experience team and overall at OpenAI, we expect that developers, builders, like all of you, will actually use both in concert. It's very likely that you'll need O1 and you'll reach for it for very specific, complex, hard

• 1:26:01Speaker 05

  And then for a lot of tasks that work great already with GPT for Foro, like summarization, labelingization and all that, you'll still have the Foro, Foro Mini to rely on. So we definitely expect builders are going to use both in concert.

• 1:26:15Speaker 06

  Will there be a way to speed up the chain of thought? Because I don't know how we can have time. Do you want to take that? Definitely, I think I have ideas to speed up these models because in the end, a lot of classical approaches to make these faster still apply. And we just started our own vision. So our idea is actively working on a bunch of to improve this, that, and this game model.

• 1:26:43Speaker 05

  It's also day one for the API. So as Sean mentioned, like when he did his recap, not everything is exposed in the API just yet because it's a net new model. But over time, we'll keep on adding features as well and the tools, et cetera, to the model.

• 1:26:57Speaker 06

  But it has to follow with the place. It's still on the chain of work. The battery gets a easy in many different steps. And if you guys can make the steps shorter, there is a weird like bring the tension kind of vibe to speeding up, easing base, right? I mean, I can just think of, yes, there is a chain of thinking, but it are easy to reach. Are you saying that other picture, how do you have that mechanism? That can be have to be awesome. You have to see, we can get all of that. We're forward. I think the obvious things that we are there are create context.

• 1:27:39Speaker 06

  As you can see, because our content is an actual thing. So that is something that we're trying to do, to get like that. And then our ability to believe and go to I hope it will just get faster and faster as we work on the vision. And we learn from, can we never look back on the way we went through. But for now, I can say it was strong. Taking what we have, which seems like a good point, believing it, even if it was pretty good, right? Or possibly pretty meaningful for me. I don't think it's great. So we have worked it out for us in terms of chain of work inside of it. And now that we got this, we really go for it often.

• 1:28:25Speaker 06

  So if I made like 10 steps, kind of like bake into the model, what if I did bake around the this, that's about the everything? That's a lot of it is in the PC, in the hardware, in the hardware. It's system, like some of those things that's getting happening. Would you be able to decide or specify how much would be, you want to spend at this time? Because where does this people, the model, kind of always be the one deciding how many equipments it wants to spend on? Do you think that a lot of the question is, is like, is like, obviously, there are also some other issues, training, post-training, in the learn, and like, obviously, I don't know of it, but the core of it is.

• 1:29:09Speaker 08

  This weekend, we're doing a big hackathon here, and of course, people will use the new model, and this is like remote as well, so we've got over a thousand participants and like 300 here in SF, but as you guys release a new model, and it's different than just traditional LMS, what do you want people to build, like any crazy, different ideas other than just, you know, summarization, whatever, like? Hey, summarization is important.

• 1:29:33Speaker 05

  I think that's a great question.

• 1:29:36Speaker 05

  The reason why we're putting these things like out there on day one and rolling out to all developers is because, you know, we know developers are going to be coming up with the most interesting use cases for these models. The recommendation would be anything you've tried to build in the past with a GPT-4 class model that may not have worked so well, or things that you felt like were limitations of the model, try that again with O1, like are there like complex challenges, especially around coding, for instance, like refactoring a large codebase, or potentially like trying to to travel shoot an issue in a large codebase, those kind of clever things that GPT-4 would maybe not be the best model for, try reaching out for like O1 Mini for this, who's particularly great at coding.

• 1:30:21Speaker 05

  Yeah, and we, basically, the reason why we're here, the reason why we want to talk to all of you is because we love feedback, we we definitely going to hear from what you build, and hopefully make those models better for you.

• 1:30:30Speaker 06

  So how would we get access to O1? Somehow Kevin and I thought access to O1, we all have like one review.

• 1:30:37Speaker 05

  Oh, O1 is not like released yet, so for now it's all about O1 preview and O1 Mini. We'll communicate later on about O1.

• 1:30:43Speaker 03

  I think that guy works at OpenAI, whoever this is.

• 1:30:45Speaker 05

  Yeah, yeah, yeah, yeah.

• 1:30:48Speaker 06

  In the chat GPT app, you can kind of see the thinking steps are all tied, you know?

• 1:30:53Speaker 05

  Yeah.

• 1:30:53Speaker 06

  What they are, you can't go through it. Are you planning on exposing that eventually, where people get that and kind of interact on a single step of the chain of thought?

• 1:31:01Speaker 05

  Possibly, yeah. I think for the APIs, it's really early days. I think we wanted to just make it work on day one with chat completions, like the most popular API that people have. But over time, we'll be trying to add more knobs for the developers to kind of understand what's happening. It's also for the first time, it's the first time we have a model that may take like 30 seconds to think hard about a problem. Like, how do we make the developer experience better for your app? I think there's a lot of work ahead of us, yeah.

• 1:31:29Speaker 06

  Going off on what he said, the fact that you can now see like little by little, like the way it's piercing together, the problem and the solution. Does that mean that there's more visibility towards, why is giving the answer because, you know, like there's a big black box and by AI, it's from player response? So like, does this mean that like now there's more visibility towards like this? Like, why is this answer the one coming out?

• 1:31:52Speaker 05

  Yeah, I think what's interesting about these new models is like you can see the reasoning, right? Like, for instance, like we, you might have seen in the, in the blog post today that we had a few videos with like a quantum physicist. For instance, like using the model to go through like equations and looking step by step at like, oh yeah, I understand what the model is actually doing. So yeah, there is a bit more about like, there's, the chain of thought turns into like a tokens where, tokens where you understand like what's happened during the thinking process.

• 1:32:20Speaker 06

  And how do you count conversations through the recent tokens also go against the context, the one for the API context? So if it's a monologue, if you can have conversations and all the reasoning it does, does that fit into the?

• 1:32:32Speaker 05

  I think we documented what's happening in our platform docs. I don't have the exact answer top of mind, but we, we share that.

• 1:32:41Speaker 05

  I think you, you had that in your diagram, yeah, right there.

• 1:32:44Speaker 03

  So I ran into this using chat GPT and the, the thing just stalls. You're not able to enter any more messages. So it doesn't do any truncation for, truncation for you and lets you continue chatting?

• 1:32:56Speaker 05

  Yeah, and the API does here, but I think something we have to improve.

• 1:33:01Speaker 06

  You'll see a setup of what you're doing. So thinking and explaining one question with the answer of explaining whatever. All of that gets added. So what you see is added and the thing. However, each additional step does another thinking process as you cut this multi content. And the amount of tokens it has to generate to do the thinking also is, you know, it's just like a flashback, right? So as you add in the chat G P T, the subsequent multi step, what you see in the U X is the total accumulated tokens. But then the thinking tokens are also required if you don't show the countdown.

• 1:33:46Speaker 06

  But they don't accumulate. If I do five thinking steps, all the hidden tokens for those thinking steps to accumulate. What accumulates over time is what you see.

• 1:33:56Speaker 09

  So interesting. So if I do another chat message after that, only the summary that the U X is very well stated.

• 1:34:06Speaker 03

  One thing I I noticed also about the thinking tokens is the open, the Sweetbench verified. So Sweetbench now requires you to submit your thinking trajectories. And that's why these guys cosign, which I I don't know if you talk to them at all while they're here.

• 1:34:22Speaker 05

  I have not personally, but some people at the company I have, yes.

• 1:34:24Speaker 03

  So they they actually they actually scored um higher than

• 1:34:28Speaker 05

  Yeah.

• 1:34:29Speaker 03

  Uh oh one on Sweetbench verified um and higher than, this is full Sweetbench. But they could not get on the leaderboard because the leaderboard requires them to submit their thinking trajectories. So it's it's basically cosign did the same thing that you are doing, which is they're not showing the thinking trajectories because it's I P.

• 1:34:49Speaker 05

  Hmm.

• 1:34:49Speaker 03

  I I thought it was an interesting uh observation. Yeah.

• 1:34:54Speaker 05

  Cool.

• 1:34:54Speaker 03

  There's yeah, nothing, nothing to say beyond that. It's like, it's it's kind of cool, but like it's. Right, like oh one? So oh one reported their own results and if you put their results side by side, they're slightly lower.

• 1:35:04Speaker 05

  This was before, I think.

• 1:35:05Speaker 03

  Yeah, yeah. This is, this is a blog post, this is from the blog post of Sweetbench verified. And then the oh one results were somewhere here. One of these guys. Yeah. I don't know where the Sweetbench one is. Somewhere here. Yeah. Yeah, yeah.

• 1:35:23Speaker 05

  System card, okay.

• 1:35:26Speaker 05

  Anyway.

• 1:35:27Speaker 03

  Cool.

• 1:35:27Speaker 05

  Don't wanna keep you too long, but thank you for that. You have one more?

• 1:35:31Speaker 06

  I was wondering, is in one of the models, it shows kind of like a scale of the computer. About this one, it looks like it's about one to two percent of scale. I was wondering if you've experimented with more, what is the kind of scale for the different computer?

• 1:35:50Speaker 05

  Yeah.

• 1:35:53Speaker 05

  Yeah. I think all we have to share on this is already on the blog post at this time, yeah.

• 1:36:00Speaker 06

  I have a question to that. Is the current model that's available, it seems to be only 20 seconds for the thinking part of that? Is it, is there like a maximum of compute, like would that be a minimum thinking? Or is that like a long amount?

• 1:36:19Speaker 06

  Like is there a cap on to, to get a lot of thinking part of the current model?

• 1:36:24Speaker 05

  I think we shared like on the API side, like the current way it's working. But for everything else, like I think we are very flexible at the moment. I'm sure we'll learn along the way. I don't think there's anything in particular that would be like, oh, this is the number, this is the cap.

• 1:36:39Speaker 06

  I see. So if in theory, if a question's complex, visually complex, it could be complicated for a longer time for second questions, that? Or is that all right?

• 1:36:50Speaker 05

  Possibly, yeah, yeah, yeah.

• 1:36:51Speaker 06

  Can you do that already?

• 1:36:53Speaker 05

  Yeah.

• 1:36:57Speaker 05

  Yeah, when you ask how many R's in strawberry, for instance, now it gets it right, but it's also pretty fast. Versus like a very complex problem. Yeah, this question, this is the Google hunting question. It took 121 seconds.

• 1:37:10Speaker 06

  Oh, okay, wow.

• 1:37:11Speaker 05

  Yeah.

• 1:37:12Speaker 06

  So if only one that, like the same way we judge the whatever, or complete whatever, plus some time, because you didn't have a simple answer. Just mean that at some point you can just put up thinking and you just don't get answer because it's a long cap for the distinct workers.

• 1:37:28Speaker 05

  That's right. That's right, yes.

• 1:39:36Speaker 07

  And what else was I gonna say?

• 1:39:39Speaker 05

  So yeah, it should take place around 10 a.m. tomorrow. So I think we'll post a tweet between 8 and 9, something like this, to kind of source questions and then we'll have some researchers to go deeper into some of those.

• 1:39:49Speaker 07

  Wait, what is it? We're gonna be at Dev Day in October?

• 1:39:51Speaker 05

  Yes.

• 1:39:53Speaker 07

  Sweet.

• 1:39:54Speaker 05

  Awesome.

• 1:39:56Speaker 03

  Okay. Yeah, I think we have like two demos. And first is the O1 Visa demo and then the second is the O1 Model demo. But actually, it's also AI-enabled, so you can come up with.

• 1:40:08Speaker 01

  That was the O1 Emergency Meetup that happened the evening of Thursday. See show notes for full video and photos. And a huge thank you once again to the OpenAI team for supporting us. More excellent updates are coming for OpenAI Dev Day 2024. And you can bet on the Latent Space Crew being there to take you through it.

• 1:40:32Speaker 01

  For part two of our O1 coverage, I'm going to recap the top takeaways from the Friday Twitter AMA done by the research team. Tybor Blahoe asks, "How did you come up with the new names for O1, O1 Preview and O1 Mini?" World's Fair speaker, Romaine Hewitt, from OpenAI answers. "Reasoning represents a new level of AI capability, so we decided to reset the counter back to one and introduce this series as OpenAI O1." Preview because it's a preview of the capabilities and Mini because it's smaller. And O stands for OpenAI. Former guest Simon Willison asks,

• 1:41:14Speaker 01

  "Why is it called O1 Mini and not O1 Mini Preview?" Shengjia Zhao from OpenAI answers, "You're exactly correct here. O1 Preview is a preview of the upcoming O1 Model, while O1 Mini is not a preview of a future model. O1 Mini might get updated in the near future as well, but there is no guarantee." Max Schwarzer from OpenAI adds, "We can't discuss the precise sizes of the two models, but O1 Mini is much smaller and faster, which is why I can offer it to all free users as well."

• 1:41:53Speaker 01

  O1 preview is an early version of O1 and isn't any larger or smaller.

• 1:41:59Speaker 01

  Friend of the pod, Alex Volkov asks: Can you guys clarify, is O1 a system that runs chain of thought behind the scenes and gives us an answer, or a model that reasons with special tokens and just hides those tokens from us at the output and just shows the final answer? Many folks are confused by this. Noam Brown from OpenAI answers: I wouldn't call O1 a system. It's a model, but unlike previous models, it's trained to generate a very long chain of thought before returning a final answer. We don't have plans to reveal CoT to users either in the API or ChatGPT. There is no guarantee the summarizer is faithful. Hyung Won Chung from OpenAI adds:

• 1:42:45Speaker 01

  Answer tokens are typically, though not necessarily, a lot shorter than the CoT. It's a single model. We're not sharing param counts right now. Thinking stage is a summary of the thought process, so it appears to be slower than it is. O1 is a single model. Jason Wei from OpenAI adds: I think it's safe to say that no matter how hard you prompt GPT-4O, you probably won't get an IOI gold. Many, many people ask: will lower upper bounds of thinking time, or test time compute, be a controllable variable via API? Noam Brown from OpenAI answers:

• 1:43:26Speaker 01

  In the future, we'd like to give users more control over how much time the model spends thinking. Hyung Won Chung from OpenAI answers: OpenAI O1 preview doesn't use tools. For the future models, we are considering adding support for function calling, code interpreter, and browsing.

• 1:43:47Speaker 01

  Johna Whitaker from Answer AI asks: "How are the team finding it for research code? HTML Snake is cool, but I'd love to hear their own use cases from the research trenches." Wukas Kondratziuk, strawberry training infra lead at OpenAI Answers: "A couple of PRs to the OpenAI codebase was already authored solely by O1." Many, many people ask: "What are the plans for the next steps? Preview phase duration, availability of the real O1 from benchmarks, and missing functionality tools, etc."

• 1:44:21Speaker 01

  Ahmed El Kishki from OpenOI Answers: "While we don't have the exact preview duration, we plan to iteratively deploy additional functionality, including tool capabilities like code interpreter and browsing." Today's guest, Michelle Pokrass of the API team, adds: "We know we're missing a lot of the features that developers need. We're working to enable function calling, structured outputs, developer system messages, and our other standard params." Jerry Torek from OpenAI responds to a question about vision aka image recognition. Hopefully soon, working hard on it, not providing any hard date yet. Wenda Zhou from OpenAI adds:

• 1:45:04Speaker 01

  "OpenAI O1 is built with multimodal and achieves SOTA on MMMU. We're working hard on making it safe to share with everyone." Nikunj Honda from OpenOI adds: "We'll add batch API support once the rate limits are higher." Many, many people asked about pricing. Sheng Jiajiao from OpenAI Answers: "Historically, prices go down 10x everyone. Two years, the trend will probably continue. For example, the cost per token of GPT-4O Mini dropped by 99% since TextDaVinci-003." swyx asks: "What inverse scaling have you seen under O1?"

• 1:45:45Speaker 01

  Jason Wei from OpenAI Answers. I don't know of any great examples of inverse scaling. You can see from our blog post that on some types of prompts like "personal writing", it seems like OpenAI O1 preview is not much better than GPT-4O, or even slightly worse. Finally, I will read you the recap from Tyba Blaho, which we link in the show notes. "Model names and reasoning paradigm. OpenAI O1 is named to represent a new level of AI capability. Preview indicates it's an early version of the full model. Mini means it's a smaller version of the O1 model. Optimized for speed. O as OpenAI. O1 is not a system.

• 1:46:30Speaker 01

  It's a model trained to generate long chains of thought before returning a final answer. Size and performance of O1 models. O1 mini is much smaller and faster than O1 preview, hence offered to free users in future. O1 preview is an early checkpoint of the O1 model, neither bigger nor smaller. O1 mini performs better in STEM tasks but has limited world knowledge. O1 mini excels at some tasks, especially in code-related tasks compared to O1 preview. Input tokens for O1 are calculated the same way as GPT-4O, using the same tokenizer. O1 mini can explore more thought chains compared to O1 preview input token context and model capabilities.

• 1:47:14Speaker 01

  Larger input contexts are coming soon for O1 models. O1 models can handle longer, more open-ended tasks with less need for chunking input compared to GPT-4O. O1 can generate long chains of thought before providing an answer, unlike previous models. There is no current way to pause inference during code to add more context, but this is being explored for future models, tools, functionality, and upcoming features. O1 preview doesn't use tools yet, but support for function calling, code interpreter, and browsing is planned. Tool support, structured outputs, and system prompts will be added in future updates.

• 1:47:56Speaker 01

  Users might eventually get control over thinking time and token limits in future versions. Plans are underway to enable streaming and considering reasoning progress in the API.

• 1:48:07Speaker 01

  Multimodal capabilities are built into O1, aiming for state-of-the-art performance in tasks like MMMUCOT, chain of thought.

• 1:48:18Speaker 01

  Reasoning: O1 generates hidden chains of thought during reasoning, no plans to reveal COT tokens to API users or ChatGPT. COT tokens are summarised, but there is no guarantee of faithfulness to the actual reasoning. Instructions and prompts can influence how the model thinks about a problem. Reinforcement learning, RL, is used to improve COT in O1, and GPT-4O cannot match its COT performance through prompting alone. Thinking stage appears slower because it summarises the thought process. Even though answer generation is typically faster, API and usage limits. O1 Mini has a weekly rate limit of 50 prompts for ChatGPT+ users.

• 1:49:03Speaker 01

  All prompts count the same in ChatGPT. More tiers of API access and higher rate limits will be rolled out over time. Prompt catching in the API is a popular request, but no timeline is available yet. Pricing, fine-tuning and scaling. Pricing of O1 models is expected to follow the trend of price reductions every one. Two years, batch API pricing will be supported once rate limits increase. Fine-tuning is on the roadmap, but no timeline is available yet. Scaling up O1 is bottlenecked by research and engineering talent. New scaling paradigms for inference compute could bring significant gains in future generations of models. Inverse scaling isn't significant yet,

• 1:49:48Speaker 01

  but personal writing prompts show a one preview performing only slightly better than GPT-4O, or even slightly worse, model development and research insights. O1 was trained using reinforcement learning to achieve reasoning performance. The model demonstrates creative thinking and strong performance in lateral tasks like poetry. O1's philosophical reasoning and ability to generalise, such as deciphering ciphers, are impressive. O1 was used by researchers to create a GitHub bot that pings the right code owners for review. In internal tests, O1 quizzed itself on difficult problems to gauge its capabilities.

• 1:50:29Speaker 01

  Broad world domain knowledge is being added and will improve with future versions. Fresher data for O1 Mini is planned for future iterations of the model, October 2023 currently. Prompting techniques and best practices. O1 benefits from prompting styles that provide edge cases or reasoning styles. O1 models are more receptive to reasoning cues and prompts compared to earlier models. Providing relevant context in retrieval augmented generation, RAGE, improves performance. Irrelevant chunks may worsen reasoning general feedback and future enhancements. Rate limits are low for O1 preview due to early-stage testing but will be increased.

• 1:51:14Speaker 01

  Improvements in latency and inference times are actively being worked on remarkable model capabilities. O1 can think through philosophical questions like, what is life? Researchers found O1 impressive in its ability to handle complex tasks and generalize from limited instruction. O1's creative reasoning abilities, such as quizzing itself to gauge its capabilities, showcase its high-level problem-solving. That's it for the AMA. Now for demos from Min and Alessio.

• 1:51:44Speaker 10

  So exciting. Thanks so much for having us. Wow, super fun. This is coincidental. Hi, everyone. My name is Min. I'm the founder and CEO of Lighthouse, where we make US immigration go fast. This is coincidentally a fun day because we're talking about the O1 model and we specialize in the O1 visa. Is anybody in the room, like, on an O1 by any chance, just out of curiosity? Oh, amazing. Cool. Just in case for anybody who's unaware, I was born in South Korea. This is, like, near and dear to my heart, but mostly because I've spent my whole career working with early-stage technologists and really amazing talent, and we make it really hard for you to continue to be here and build what you want to build.

• 1:52:26Speaker 10

  The interesting thing about something about US immigration that you might not know is that there's a sort of best-kept secret called the O1 visa, which is an employment visa that is meant for, really popularized by startup, by the technology or the entertainment industry originally. So for a long time, it was used by models, musicians, so on and so forth. You know, Melania Trump got in on an O1 type thing. And then, over the last five years, five, six, seven years or so, it's become really popularized by the startup and technology industry for like highly technical talent and entrepreneurs. But the problem is that it's still really underused. So if you think about, just to give you some context, there's only about 11,000 people who get it every year. And that is in contrast to the 750,000 people who applied for the H1B three years ago.

• 1:53:10Speaker 10

  So it's just hugely underused visa. And why is that the case? It's because most lawyers are really bad at it. Most providers don't know how to do it. And then, when you go through the process, you end up having to do a lot of it yourself. And for those of you who have been through the process, you may or may not have had, understand this, where the actual application is like a 300 to 400-page research report about you and all the amazing things that you have done. And how those professional or academic achievements map to the criteria that USCIS wants to see. It's sort of like applying like a really, really intense college application, like on steroids. And at Lighthouse, we want to make this process go way faster. Because traditionally, if you go through this, one, you'll often get told you don't qualify.

• 1:53:54Speaker 10

  And then, two, you'll end up having to do so much of it yourself, and it'll take you six months. And it really doesn't have to be that way. So what we do is try to make the process as simple as possible. We kind of like 30 days or fewer. You can do it in four weeks. And then, the government can back, will get back to you within two to three weeks or so. And so, the whole thing doesn't have to really take more than two months. And we were really lucky to get to support SWICS. And so, in some ways, this is SWICS's O1 congratulations party as well. Yay! And we would love to have a lot to support more people. And part of the reason I'm here today is because I kind of wanted to give you a behind-the-scenes look of, we've basically built a platform that's sort of like a paralegal out of the box. So that you don't have to spend time going back and forth with an attorney and spending hours and hours and hours drafting a lot of this yourself.

• 1:54:39Speaker 10

  We are happy users and customers of OpenAI. Please give us access to the mini. Because the model itself actually said, I think in your paper, that it is really good for STEM, like STEM problems. And a lot of the people that we work with are STEM talent. And so, a lot of what we end up having to do is translate the technical papers that they've published and the work that you have done in a way that a layperson, a USCIS officer, who's probably, you know, 40, 50 years old, probably spent some time in the military and doesn't really have context about what makes AI special right now. And an amazing thing is, right now, everybody wants AI talent in the US. And so if you are working in that field, we definitely want to help you be able to stay and come here. So what I'm going to show you is actually, I'm just going to do this on the fly,

• 1:55:23Speaker 10

  and so we'll test the boundaries of all this.

• 1:55:29Speaker 10

  But let's say we were going to do, let's, I actually just picked Daniel because he's the second author on the paper. And so we're going to do it in the fly. And so let's say he is an expert in artificial intelligence, just ignore that. Hold on, let me just make sure I can.

• 1:55:47Speaker 10

  Yes? Oh, oh my god, can you not see? Okay, is that better? Amazing. This is what we've built so that we can make this process go fast for you. And let's say it goes to OpenAI, cool, let's just ignore that. We don't need everything right then and there. And what is really cool is, once we can... Oh, okay, give me one second. We're gonna do it here.

• 1:56:18Speaker 10

  But let's pretend that Daniel needs, we have to talk about his papers. We can just add his page, and let's pick his like first five papers. And what this is actually going to do is, there's a whole bunch of stuff we've built in the background here, but we've basically fetch all the papers, read all of them, summarize them, and then we've layered on top of it a ton of industry know-how, basically like all the legalese speak, based on like tons of feedback from the legal experts that we work with. And so it takes a little bit of time because there's a lot of content that it's going through. But it'll basically spit out the entire thing in terms of what we need to do, and then all we need to do is edit and revise it, and then we'll show it to you, and then you make sure that it like is sound, right? And we can do this with a whole bunch of other things.

• 1:57:03Speaker 10

  It'll usually take like a minute or two because it's like literally picking up all the papers and then reading them, and then summarizing them. But, so we'll let that run. But basically, we can do that, and we do it to find out whether or not your pay comparables are high. We do it to figure out what the papers that you've written or research that you've done. If you, honestly, we can put like this into it and have it, have it like synthesized in a way that is intended for USCIS to read. And yay, it's done.

• 1:57:35Speaker 10

  And so it'll basically tune it to whatever we decide that we wanted to focus on. Right? And so this, in this case, we happen to want it to be focused on machine learning product. And so everything is kind of like oriented around that. And it will summarize all the papers that we asked for, the five that we wanted. It'll pick up all the papers that we want. And so if you're familiar, SWICS, we like gave you this like 500-page document that we said we're going to literally physically print and ship to USCIS. And we basically turn a process that's probably 120 hours, when you work with a traditional provider, into something that's probably closer to 20. And so that's how we can do it fast. And we would love to work with the O1 Mini model,

• 1:58:17Speaker 10

  because a lot of what we have to do is make sure that this is as precise as possible without, while still being like precise as possible. Yeah. And then, just for kicks, I wanna know, I wanna show you what it might look like, what it might say if O1 Mini. What might it say if we asked it to go read this? And...

• 1:58:56Speaker 10

  It should, oh yay. So, we would say Daniel developed OpenAI Mini, and it's a cost-efficient reasoning model designed to excel in STEM fields. And so it'll basically summarize all of this in the way that it needs to be kind of like written for the officer. And in essence, the way that it would typically work is either you would have to write this for the lawyer, because they wouldn't really know how to summarize this. And you don't really know what good looks like, because, you know, probably, reasonably, you would say, "Well, I wrote an abstract. That's what the abstract is for." But it's not written in the way that USCIS would understand. So you would probably have to spend a lot of time doing this, or they would do it, and they would do a shoddy job.

• 1:59:39Speaker 10

  And so we kind of combined the industry know-how and the magic of AI now to make this process a lot more efficient.

• 1:59:49Speaker 10

  Yay!

• 1:59:52Speaker 10

  So if you are looking for an O1 visa, we also support all employment fee visas to the US, so come find me. My name is Min.

• 2:00:04Alessio Fanelli?

  All right, so the idea here, as, sorry.

• 2:00:09Alessio Fanelli?

  As Sean mentioned, the O1 API doesn't actually have a lot of the things like structured output, tool calling, everything like that. So I spent the last three hours trying to give those tools to it. I'm using a thing called E2B. I do not work at E2B, but we invested in E2B and E2B is the only thing that actually does this. It's basically like a cloud sandbox for your LLM to use. So the way I built it, I take a prompt. In this case, I'm from Rome, so I wanted to see a visualization of the growth of the Roman Empire, as one does, and I had to remind O1 that it has access to a code interpreter, even though it doesn't know it. And then what I did is use 40min to extract the code. So O1 is just doing the planning, obviously writes the code in the actual outpost.

• 2:00:52Alessio Fanelli?

  So if you see, this is what O1 actually returned. So it's like, you know, these are the steps. This is like something we can use for the data visualization. This is like explaining all, you know, a lot more information than is going to go in the actual thing. This is how you process it. And then what I've done is just putting 40min as a way to extract all of it. So it's like, you know, your software engineers that receives the plan. And then from the plan, you create one single script that does everything that you need to execute it. And I've created this simple kind of like structured output model. So there's the code that needs to be run, all the pip packages that are required to run the code, the file name, and then the command to execute.

• 2:01:35Alessio Fanelli?

  So what I'm doing after that is basically open a sandbox. Actually, the only reason why this doesn't work, we cannot get the PNGs out of the sandbox right now. We're trying to figure out the right path. But basically you install all the pip packages and then you run the command that the model generated. And then there's also a self-healing loop. So if there's an error with the sandbox, it tells you what the error is. I feed that back again to the model and it's like, hey, actually this happened. And then I use 40min again to extract the new code, rewrite to the file that I was running before and kind of like loop through it. And so what that looks like, see, we cannot actually get this file out. But it starts from this document, which is like, you know, a very kind of like,

• 2:02:20Alessio Fanelli?

  it's a good plan, but like it's not something you can actually run. It goes through 4.0. So there's one error, which is most of the time, people are not using negative dates in pandas. So the O1 is actually saying, hey, you know, in the year minus 23. And pandas is like, what does that even mean? Nobody ever says that, but it is a valid date. So you actually had to convert to a different format to kind of do to do before Christ dates. So that's something that O1 didn't do, but 4.0 Mini can do and update the code. So yeah, I'll publish this on GitHub just so people have it. But this is like an easy way to get from O1 planning to like 4.0 mini code extraction. And then actually have a runtime that does it. As you might have seen in the security review of the model, it's like quite good at jailbreak.

• 2:03:04Alessio Fanelli?

  So I did not want to run it even in Docker in the in the local environment. So if E2B gets hacked, Vasa gets right there, he's going to be mad. But yeah, this is this is it. And we'll send out the the link after. And I'm actually going to run the code locally. I just kind of copy paste it that thing here. So if you just do Python. Test the pie. These are like all the charts that actually get generated. So this is like the empire regulation over time. This is the territorial areas over time. And these are, you know, kind of overlap together. So again, this is a data set that there's not really a good source out there for. There's kind of like all these different pieces.

• 2:03:47Alessio Fanelli?

  So I think one of the most exciting thing about these models is also like data set generation. You know, like, how do we get take all of this tax data that has been trained into it? And bring it out in a structured way? So yeah, that's that's it.