OpenAI DevDay: Beyond the Headlines with Logan Kilpatrick, OpenAI's Dev Relations Lead

We’re deep diving into OpenAI DevDay with Logan Kilpatrick, Dev Relations Lead at OpenAI. If you need an ERP platform, check out our sponsor NetSuite: http://netsuite.com/cognitive. We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. SPONSORS: SHOPIFY: https://shopify.com/cognitive for a $1/month trial period Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive NetSuite has 25 years of providing financial software for all your business needs. More than 36,000 businesses have already upgraded to NetSuite by Oracle, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you're looking for an ERP platform ✅ head to NetSuite: http://netsuite.com/cognitive and download your own customized KPI checklist. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. X/SOCIAL: @labenz (Nathan) @OfficialLoganK (Logan) @CogRev_Podcast TIMESTAMPS: (00:00:00) - Episode Preview (00:02:08) - How many startups did OpenAI kill? (00:05:50) - Current employee count at OpenAI (00:06:59) - OpenAI's mission being focused on developing safe AGI to benefit humanity (00:07:10) - How the GPT Store relates to AGI and progressing agent development (00:08:22) - OpenAI's strategy to release AI iteratively so society can adapt (00:10:50) - Safety considerations around the OpenAI Assistant release (00:11:30) - Capability overhangs and is the internet ready for agents? (00:14:13) - Why certain agent capabilities like planning aren't enabled yet by OpenAI (00:15:28) - Sponsors: Shopify | Omneky (00:17:34) - GPT-4-1106 Preview designation (00:21:50) - 16k fine-tuning for 3.5 Turbo (00:25:13) - GPT-4 Finetuning and how to join the experiment (00:27:53) - Custom models: $2-3 million pricing to build a defensible business (00:29:48) - Bringing costs down to bring custom models to more people (00:30:19) - Sponsors: Oracle | Netsuite (00:33:53) - Copyright shield (00:35:42) - OpenAI doesn’t train on data you send to the API (00:36:37) - New modalities and low res GPT vision (00:37:26) - GPT Vision Assessment for Aesthetics (00:42:30) - WhisperLarge v3 (00:44:15) - Text-to-speech API: the voice strategy and AI safety (00:51:45) - Log probabilities coming soon (00:53:45) - The evolution of plugins to GPTs: the challenges with plugins (00:55:33) - GPT Instructions, expanded knowledge, and actions (01:00:18) - How is auth handled with GPTs (01:01:04) - Hybrid auth (01:02:50) - GPT Assistant API Billing (01:07:58) - AI Safety (01:10:28) - OpenAI Jailbreaks and Bug Bounties (01:11:57) - The OpenAI roadmap for a year from now The Cognitive Revolution is brought to you by the Turpentine Media network. Producer: Vivian Meng Executive Producers: Amelia Salyers, and Erik Torenberg Editor: Graham Bessellieu For inquiries about guests or sponsoring the podcast, please email [email protected]

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  How many startups do you think you guys killed yesterday?

• 0:02Logan Kilpatrick?

  There's so much opportunity for startups being created. And I think like the reality is if your if your startup was like a super, super thin wraparound something like you didn't really have a startup. You just had some like project that you were working on that was building off of an API. And like my takeaway from yesterday is the barrier to entry for new startups and for people to build like extremely compelling products has never been lower. And there's never been more amazing technology to build with. I think we we probably created thousands and thousands of new startups. If you take a step back, if you're a venture backed, or even if you, you know, are like later series, you know, ABC, whatever it is, and have raised enough money to do this and have cash like,

• 0:44Logan Kilpatrick?

  it actually makes a lot of sense. This is an opportunity to like truly build a defensible business. Like you can have a model that's, you know, trained on, I think, in order of like billions of tokens, which is like not something that anybody else has access to. And again, it requires a sufficient investment to make that happen. But I do think that it's like, this is what people have been asking us for. Like, how can we build a defensible business using your products and services?

• 1:08Nathan Labenz?

  Hello and welcome to the cognitive revolution where we interview visionary researchers, entrepreneurs and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Logan Kilpatrick from OpenAI. Welcome to the cognitive revolution. How's it going, Nathan? I'm excited to be here. It is an exciting time. You guys have just launched a whole smorgasbord of stuff at OpenAI. And so congratulations on that. And thanks for taking a little time to dig into the details and all the little nuances of it with us.

• 1:50Nathan Labenz?

  I wanted to kind of start with a couple of big picture questions and then really get into the weeds. So starters, obviously going into yesterday's event, the sort of meme sphere and certainly coming out of it too, was like, how many startups is OpenAI going to kill with this release? So, you know, kind of a tongue in cheek starter question, but if you had to put a number on it, how many startups do you think you guys killed yesterday?

• 2:12Logan Kilpatrick?

  I've been seeing this trend on Twitter and I get such a visceral negative reaction to it. I really do think there's like, there's so much opportunity for startups being created. And I think like, the reality is if your startup was like, a super, super thin wraparound something, like, you didn't really have a startup, you just had some like, project that you were working on that was building off of an API. And like, of course, as things change, like, you'll be disrupted in some capacity. Like, true genuine startups like, are thinking about from the ground floor, how do I build a defensible business? And like, you have to go and like, build an innovative product to do that. So I, my takeaway from yesterday is, the barrier to entry for new startups and for people to build like,

• 2:57Logan Kilpatrick?

  extremely compelling products, has never been lower. And there's never been more amazing technology to build with. I think we probably created thousands and thousands of new startups. And gave the ones who are building on our platform today, you know, lower prices, a ton of new features. They can go and compete with, you know, other big companies. So I'm really excited. And I think there's never been a better time to build something cool, using AI.

• 3:22Nathan Labenz?

  Yeah, well, no doubt about that. So we're going to get into all the tools. Just another kind of angle on that same sort of question. It does seem like there is a bit of a shift, where, you know, not even a year ago yet, and it's crazy to think about, right? ChatGPT, I understand was kind of motivated at OpenAI by this sense that like, hey, these models are getting pretty good. But we're not seeing kind of the traction or the sort of everyday use that we would, you know, expect to see or hope to see. So like, let's put a product out there and try to kind of kickstart that. That obviously really worked. And now this, you know, year later, it seems like there's a bit of a shift toward more of an Apple-like, not first but best, maybe you could say, kind of product direction, where, you know, the community has been out there kind of exploring all these things of RAG and, you know, agent frameworks and all that stuff.

• 4:08Nathan Labenz?

  And here comes OpenAI with like, presumably, by a significant margin, I would imagine, you know, the best-in-class version of that. So how would you kind of frame that shift? And is that something you think we'll see more of in the future from OpenAI? Very curious about that.

• 4:25Logan Kilpatrick?

  I think part of it is, and the community has an advantage in the sense, I think this is probably the same thing for Apple. Like, I think we would love to be first to a bunch of things. I think the reality is, like, for us to build something, it has to have, like, you have to be able to scale it to the hundreds of millions of weekly active users in ChatGPT and like really make sure that it fits into like, what is the core strategy from a product perspective? So there's always like a million and one different things that we could be doing. And it's just hard to ship things quickly when you are building at that scale. Like, there's just like a lot of really difficult technical challenges to solve. So it's not like we've been sitting there like, you know, you know, scratching away at like every last little detail, being like, ah, we want to make sure that we're, you know,

• 5:08Logan Kilpatrick?

  putting out the very best thing and we don't want to move quickly. It's like, really feels like, at least, at least to me internally, that like, we're always in an all-out sprint all the time. And like, as soon as something is ready in the pipeline, like, it's coming out to people. The reality is, like, the team has just been growing and there's not enough, like, human hours in the day to do all, do all the stuff. I think people forget, like, it's not like we didn't ship anything for the last nine or ten, nine or 12 months. Like, there's been a constant slew of things. It's just, there's a lot of work that, that has to go into making these things happen.

• 5:39Nathan Labenz?

  Yeah, even just the last month or two, I mean, it wasn't that long ago that 3.5 fine tuning dropped and, you know, people were like, boy, if they're releasing this, the head of Dev Day, you know, Dev Day is going to have some real fireworks. What is the employee count at OpenAI now? Do you know the current number off the top of your head?

• 5:54Logan Kilpatrick?

  Yeah, I know, I know the general number. I think when I joined last year, it was something around like 300 and I think we're, we're well and beyond that. I think the, the only general comment that makes, I'm not sure how public facing the, the internal numbers are, is that I think the estimate on LinkedIn is, is a little bit higher than what it actually is in, in practice. But we've, we've grown significantly. It's been awesome to see, and like, throughout it is, we needed to, because there's so much work that has to happen. But, yeah, it's, it's interesting to see the, the shift that's happened with going from a, you know, when I, again, when I joined it, I wasn't even that early, a 300-person company, to now something that's, that's much, much larger than that. And obviously is going to continue to grow in, in the next, the next year, so. That'd be interesting.

• 6:38Nathan Labenz?

  Yeah, so obviously, one of the keys to scaling an organization is having a strong sense of kind of mission and high level strategic clarity. You know, I'm sure not unrelatedly, OpenAI recently updated its core values. And there is now a statement, even I'd say more explicitly than previously, that the company is focused on AGI. AGI focused, we are committed to building safe, beneficial AGI that will have a massive positive impact on humanity's future. Anything that doesn't help with that is out of scope. All these things that got launched yesterday, a lot of details, you know, a lot of features, the GPT store. How are you guys thinking about that as being on the path to AGI? Because I could certainly imagine somebody saying like, wait a second, a GPT store?

• 7:23Nathan Labenz?

  Is that really on the, you know, kind of critical path to AGI? But I know that's obviously something you've thought deeply about as a team. So I'd love to kind of hear how this is all contextualized as part of that grand mission.

• 7:34Logan Kilpatrick?

  Yeah, it's a good question. So I think like, to put it like very precisely, like OpenAI is not, we don't exist to make the best chat platform. We don't exist to make the best API. Like, these are, these are mechanisms in which help get us to the point of AGI that benefits humanity. I think the piece around ChatGPT and the piece around the API is, is another angle of our philosophy, which is we need to get this technology out to people iteratively. And you heard Sam say this actually in the keynote yesterday, where assistants and GPTs are really our first shot at and moving in the direction of agents. And we didn't want to go, you know, all the way to the sort of autonomous agent type workflows

• 8:16Logan Kilpatrick?

  that you, you see and some other products and services today, really in the vein of we need time for, for society and for people to adapt to these changes. And I think that's, that's really, ChatGPT today is really like the mechanism to allow people to see where this technology is going and how to adapt. Like, I think Sam, Sam also made this comment sometime in the last few years, but like, it is entirely possible that OpenAI could have continued to do the research that we were doing and build. You know, what could become AGI in the confines of OpenAI not sharing it with the world? And in five years, we just sort of emerge with this thing that works and like that. That's actually like a plausible outcome like that.

• 8:59Logan Kilpatrick?

  I actually think that someone, whether us or someone else, could actually do that. Um, the reality is like, that's not the best thing for humanity. Us showing up with this thing that's now all of a sudden super powerful and no one's ever seen chat GPT. No one's ever used GPTs or any of the future products. It it would cause a tremendous amount of turmoil for for everybody. I'm like, that that's not great. Um, so I think that's that's really the the strategy is like the core research is what's going to enable AGI to happen. Um, but chat GPT and our API are the way of making sure that that technology benefits everyone and everybody gets a chance to like see what's coming.

• 9:35Nathan Labenz?

  And shape it too, that the two quotes that sit out to me were, we think it's important to give people a chance to experience and adjust to this technology, which you're just speaking to. And then also, we want more people to shape how AI behaves. I thought that was also uh pretty telling. Obviously, there's been a number of different things there, but the GPT store, you know, is kind of a seemingly a way to really open up access to how you can, you know, shape the behavior of an AI assistant without even necessarily needing to, you know, get into the API docs. So that's that's definitely very interesting as well. So I'm I'm actually grown, you know, I'm definitely somebody who's pretty concerned about AI's safety, big picture.

• 10:13Nathan Labenz?

  But I've definitely grown a lot more convinced over time by this kind of over capabilities, overhang argument that like things have to kind of get into the public sooner rather than later. Or else, you know, it gets to be super disruptive when there's that huge or potentially disruptive when there's that huge delta. Uh, so let's kind of get into the weeds and help close the uh capability overhang gap. The big themes that I saw, I started to wonder if you would add anything to it. Multimodality, obviously huge. I'll call it accessibility via price drops across the board and also the turbo speed up. I'll call it platform expansion uh as kind of a shorthand for saying, you know,

• 10:52Nathan Labenz?

  these adjacent formerly kind of adjacent complementary things like vector database and runtimes, which, you know, have existed a little bit in in chat GPT, but haven't been to the API. Now that's kind of available for everyone. And then finally, agents. And I I would I will we'll get into it all, of course, but I I'm curious, maybe for starters, like you said something interesting a second ago on.

• 11:16Nathan Labenz?

  Not going all the way to autonomous, like, how are you guys thinking about how far these, you know, assistants, not yet maybe full agents? How far are they kind of intended to go right now in terms of their autonomy?

• 11:28Logan Kilpatrick?

  The context around how far the limitation of these models, how far the assistants can go and how far GPTs can go today. Really, there's no mechanism for them to do like self-planning. And if you look at like a lot of the like traditional like agents products, AutoGPT, BabyAJI, whatever they are, they do this whole like planning process and they say, I'm going to go and execute these tasks and I'm going to sort of follow up afterwards. GPTs don't have that that capacity today. It's really still the core, like, send a message, get a response, send a message, get a response. So the like long-term queuing up of actions, I think there's also the the sort of missing link towards the more agent workflow is like, I'm going to go and run these things in the background,

• 12:10Logan Kilpatrick?

  maybe without you self-directing me to do it, like, I'm kind of just going to go and randomly check and see whether your email's been updated or, you know, whatever, whatever that might be. So I think those are really the missing, the big missing pieces right now. I think with the assistants API, you could probably build some of these yourself if you really wanted to. I do think like we'll eventually get there in the API, we'll eventually get there with with GPTs, assuming that it can be done in a safe way. I I think there's also like that question of reliability. Like, I I don't think that anybody has really like hit the home ground yet with with agents products, whether it's in an API or a consumer product. And I think that's another missing piece of like, it's probably just going to take us a little bit more time

• 12:52Logan Kilpatrick?

  and others as well to figure out like, how do we actually do this reliably? Like, make sure that it really provides value. I've tried a bunch of the agents products and I'm just like, this basically takes as much work as if I was going to do it myself. So like, what's what's really the point of of using this? Like, it's just kind of, if anything, you're just like making yourself, at least at the current state today. My feeling personally is like, you're making yourself like liable for more risk. Like, I'm giving my credentials to some system where I don't really get to see every single step that it's taking. And, you know, it might not really do the things that I want. And I think there's just like a little, there's like some small little missing piece of refinement for that workflow to get really, really good. And then people will just, yeah, I I think it's going to be the explosion of of of those agents.

• 13:37Logan Kilpatrick?

  I also think like the the challenge today with agents is the internet is and society are really not. Ready at the moment, I think, for like a hundred million or a billion agents. Kind of like, you know, the chat GPT has a hundred million active, weekly active users. If all of a sudden, every single one of those weekly active users has access to quote unquote agents who can go out and do things on their behalf, like the internet and products and services are just not ready for that. And I think it's going to take a little bit of time and hopefully GPTs are like a nice way for companies and people to start thinking about this.

• 14:12Nathan Labenz?

  Interesting. Is this sort of a conscious kind of decision to say, we're going to, you know, not train this kind of planning capability past a certain point right now? Or is it just that, you know, OpenAI also hasn't quite, you know, figured out how to do it? I mean, most of that kind of suggested that it probably is like within the scope of what OpenAI could do, at least in the near term, if not already.

• 14:39Logan Kilpatrick?

  Yeah, I, I think it's, I think it's a little bit of both. Like my guess is there's just some like core technology things that need to be figured out, like even with what was released in the last couple of days, like it was an all out sprint to get there. So like, you know, the scope had to be constrained in a lot of ways. I, I do think that there is, there needs to be like deeper thought put into like actually rolling these things out and making them accessible. So I think that's like a huge angle of it. I do think the other angle is like actually making sure that the core technology works really well. And I, I'm, I'm less concerned about that piece just because our, our team is incredible and can continue to, to make amazing things. Um, I think the, the deeper challenge is like getting everyone else in the world on board with this idea and comfortable with it. And I think that's, that's just a very, a very difficult problem.

• 15:21Nathan Labenz?

  No doubt. There, there's a lot of adjustment that people are going to need to make over the not too long time horizon.

• 15:28Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 15:33Nathan Labenz?

  So let's kind of start, you know, with some fundamentals and then, you know, work our way back up, you know, I think kind of the complexity ladder towards some of the highest end. You know, higher up the stack as we go, starting just with the models. GPT-4 11.06 preview. First question, what is meant by preview? I guess I noticed that there's like 100 requests per day limit right now. So I guess that's probably the main thing, right? Obviously you can't deploy that. Anything else that we need to be aware of on the previews designation? And any sense of timeline for when that limit goes up?

• 16:05Logan Kilpatrick?

  Yeah, the preview designation is just intended to tell people that this isn't the final version of the model. So this was a like a essentially an early release candidate of what the finalized model will actually look like. So there's still like a bunch of like actual modeling work, post-training work and other things that need to happen to get this model like to the standard of something that we would normally release in the API. So that's, that's why that preview designation exists. I'd imagine in the order of like weeks, hopefully less than months. It's, it's available as like the regular general version and no longer accessible through, through preview. You know, we, we need to make sure that we get feedback from people as we release these new models and making sure that this,

• 16:47Logan Kilpatrick?

  especially given how, how many changes are baked into this model, that it's still performing to the level that people would expect. So hopefully you can try it as a drop-in replacement for some of your existing use cases. See how it performs, make sure there's no major regressions. And then if there are major regressions, come and yell to us and let us know so that we can, we can explore and make sure that the model is, is going to be generally good.

• 17:09Nathan Labenz?

  You know, yesterday, the, the sort of highest level statement on performance was just, it's better than GPT-4. How is that being characterized? Like I didn't see, maybe this was in a breakout session. Funny enough, neither you nor I were there in person yesterday for personal reasons. But is there like a model card coming out? Are we going to, should we expect to see like the MMLU score and the, you know, bar exam score in the same way that we got for the original GPT-4? What's the outlook there?

• 17:39Logan Kilpatrick?

  Yeah, I think so. One example is, and I don't know how specifically this generalizes to for Turbo, but 3.5, we can use this as a proxy. The quote is, "For instance, our internally evals show a 38% improvement on format following tasks such as generating JSON, XML, and YAML." The point of feedback and the question around releasing benchmarks, I think, is one that we've heard, and I think with our last model API release, I think there was, we didn't do a good job of, like, fully characterizing, you know, what are the performance differences between these models? The TLDR is, I think we want to release more of these benchmarks.

• 18:20Logan Kilpatrick?

  I don't know yet if the full plan is to release a bunch of benchmarks for the, like, when this model comes out of the preview version, I would hope so, but don't have a definitive answer right now. I also think the nice part is, like, hopefully people can run some of these evals as well and, like, keep us honest, and I think people have done that with previous model iterations as well.

• 18:40Nathan Labenz?

  Yeah, it'll take a couple days worth of your 100 requests per day to get the MML U Suite run, but I am sure somebody is on it somewhere, Dan Hendrickson and team, I'm sure, have already started pooling their requests and, you know, it can't be too long now before we'll have that score. So on the vision side, it seems like the note there is, it's basically the same model under the hood, it's just kind of a presentation, like, system message difference, so we basically should expect one and the same. Given that, is there a reason that it's too, I guess it's a system message, but I'm a little confused as to why it's even presented as, like, two different models via the API?

• 19:21Logan Kilpatrick?

  Yeah, the big background context is it's two separate, like, tech stacks, if you will, to get it to work right now, and there just wasn't enough time to, they used to be completely independent, separate systems, and there wasn't enough time to unify things, which is why they're being presented as two different options, because there's, like, some, like, very slight nuanced differences, but behind the scenes, it is the same thing, it will end up being unified into one, and people should look at it and expect it to behave in the same way, with the caveat that with vision, there's, like, a very small addition to the system message, which could, again, technically, like, throw off some of the prompts that people have in some cases.

• 20:04Nathan Labenz?

  Next up, 3.5 Turbo 16K fine tuning. This is the first fine tuning that supports function calling, correct?

• 20:13Logan Kilpatrick?

  So when we originally released 3.5 Turbo fine tuning, it didn't support function calling. And then we fast followed like a couple of weeks later with function calling. So the last version did support it. This is the first version that supports 16K out of the box.

• 20:27Nathan Labenz?

  Definitely very interested to crack into that. We're using 3.5 fine tune now with the Waymark, you know, core script writing model. And, you know, we've got a lot of stuff that we're throwing at it. So we do, you know, sometimes have to even truncate our inputs to make sure we stay within the 4,000. But the 16,000 is going to obviously open that way up and create the possibility for like downstream, you know, iteration, chat style, back and forth, function calling. So that'll be a big, you know, expansion of possibility just for us with our, you know, pretty narrow product. So excited to get into that in more detail.

• 21:02Logan Kilpatrick?

  It's been crazy to see how much value people have gotten from 3.5 fine tuning. I think 16K has been like the number one ask for people. And it, yeah, I think now, especially with, with extra context, you're probably getting even closer to GPT-4 level. Like I think we've, we've already had a bunch of customers. I think we shared some of these use cases out of people getting like GPT-4 like performance using 3.5 Turbo. I think now it's the gap is even going to be closer because you have more tokens to work with. You can give more intricate examples. And I think that was like one of the core limitations before. It's like there just wasn't enough context in some cases to like really give everything that you wanted the model to do, plus improved instruction following and everything else that's coming in this latest 3.5 Turbo version.

• 21:46Logan Kilpatrick?

  I think it's, yeah, the, the gap between the models in some cases continues to shrink, which is really cool.

• 21:52Nathan Labenz?

  Yeah, I wonder if you have any big tricks? And I know you guys are doing a, I guess I don't know if you're co-sponsoring it, but there's a scale webinar tomorrow that I'm looking forward to attending that's on kind of best practices for fine tuning 3.5. For me, the number one thing that has driven performance is training it on reasoning.

• 22:11Nathan Labenz?

  So the huge difference between if I just give it good examples and ask it to, you know, follow, you know, obviously, the implicit ask of fine tuning is learned from these examples. But then layering in the reasoning and explaining why it's writing, in our case, you know, the script for the video in a particular way is a huge delta in performance. Any other tips along those lines that are like, kind of simple, but like, drive a lot of value in the 3.5 fine tuning?

• 22:38Logan Kilpatrick?

  Yeah, we actually had a breakout session on this at Dead Day. It was recorded and it should hopefully be available in like the next few days, hopefully by the end of this week. I'll defer people to that recording. I know John, who's one of the folks on our fine tuning team, and Colin, who actually is maybe the person who's doing that webinar tomorrow, were the two speakers for that. And I, I saw a bunch of the chatter online. People seemed really excited about it. I didn't, I haven't actually seen the talk, so I'll defer to them because they're the, they're the sort of experts in the space right now. But they went through a bunch of like customer use cases and the pitfalls and all the things like that. So it should hopefully be the perfect answer for people looking for, for tips like that.

• 23:17Nathan Labenz?

  Cool. Okay. Well, then the next, there's even two tiers of, you know, kind of fine tuning possibility beyond that. GPT-4 fine tuning coming soon. Is this the 128K that will be the base? I assume so. Can you get me into the program? And more generally, like, what kind of frontier use cases are you looking for as you're looking, as you're, you know, going to pull people into that experimental program?

• 23:44Logan Kilpatrick?

  Yeah, the context for the experimental program is, is just wanting to make sure that there's a bunch of safety concerns, obviously, because GPT-4 is much, much more powerful than, than 3.5 and the way that it's trained and, and, you know, with more context and things like that. So we just need to be a little bit more initially cautious about, about rolling that out wider. It's also more expensive. So I think people, you really need to be somebody who is, who is pushing on the boundaries. And part of the concern was like, a bunch of random developers are going to show up and like, kick off a bunch of fine tuning jobs and not really have perspective on like, the cost for, for GPT-4 fine tuning, which is more expensive than 3.5, as you would, as you would assume. And then be like, oh, wow, I just paid all this money.

• 24:28Logan Kilpatrick?

  And by the way, like, I probably am not a machine learning expert and like, did it make my training data right? And like, now I just paid all this money for something that's not super great. So I think there's like, from a Product perspective, a little bit of concern of that as well as like it, you just have to be more thoughtful as you're spending more money and and making a more powerful model about like how you're actually training it. And I think like it takes folks who are like really thoughtful about this, like you and and the folks at Waymark to to do it well and like actually have like maybe a little bit of data science and machine learning background. I don't think there's like, there's no specific use cases that we're we're looking for on the the four

• 25:09Logan Kilpatrick?

  early access side. It's really about just like deploying it safely and and specifically to people who have already used 3.5 fine tuning, like it, you should really try 3.5 fine tuning for your use case, especially now with 16K. So you won't even have the option to opt in um to to like apply for for fine tuning. If you haven't used 3.5, and this has been one of the mistakes that we've made in the past with all our like general wait lists that we've done is like, everybody just says, yeah, I'm I'm interested in this. Like, that would be cool to do for fine tuning without actually ever having spent the time or looked at the data and all those things. And like, it just makes it really, really difficult for us when a million people say that they want something. And we're like, well, we don't, it's like, how do you filter through those people?

• 25:53Logan Kilpatrick?

  And it's a lot of work. So hopefully this new approach of like gating it based on actually having used previous fine tuning will make sure that we get like a much higher signal to noise ratio from customers.

• 26:04Nathan Labenz?

  And then beyond that, custom models, uh, this was kind of the highest end thing I saw uh via Twitter that two to three million dollars is kind of the entry level price expectation. I was surprised. My initial guess was 10 million dollars just to start. Um, just given how much, you know, kind of demand I would expect you guys to get for that from companies for whom, you know, money is presumably not much of a constraint. I'm I'm priced out of that at Waymark, but you know, there's a lot of companies there with some pretty healthy balance sheets. So any guidance for kind of what you're looking for there beyond obviously people with the money to spend?

• 26:41Logan Kilpatrick?

  Yeah, I think part of the context with custom models is really wanting to push the boundaries of like, what is actually possible if you if you put in a bunch of your proprietary data? And I think there's companies who are building their core product around AI are going to be well suited for this. And I think like, if you take a step back, if you're a venture backed, or even if you, you know, are like later series, you know, ABC, whatever it is, and have raised enough money to do this and have cash, like, it actually makes a lot of sense. You have, this is, this is an opportunity to like truly build a defensible business. Like, you can have a model that's, you know, trained on, I think, in order of like billions

• 27:23Logan Kilpatrick?

  of tokens, which is like not something that anybody else has access to. And again, it requires a sufficient investment to make that happen. But I do think that it's like, this is what people have been asking us for, like, how can we build a defensible business using your products and services? And I think custom models is going to be one of the ways to do that. I'm hopeful that like, I feel like I deeply resonate with the voice of like the average developer. And obviously, most people who are hacking on a project or are sort of getting their feet wet in this space, don't have two to three million dollars. So I'm hopeful that we'll learn a ton of stuff from this process and like figure out a way that we can bring those costs down to make it more widely accessible to more companies, more startups, and people who are interested in pushing the limits of what's

• 28:07Logan Kilpatrick?

  possible with these models without putting in two to three million dollars. But I think we just need to learn a bunch of stuff still. It takes like, you know, actual OpenAI researchers are going to help you make those models better from end to end. So it's a huge investment from our side, especially given like, there's so many things that we could be doing. But I do think it's important to help people with with these problems.

• 28:30Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors. I thought it was a bargain. It's like if you are an enterprise that is sitting on a huge amount of data that you've kind of been collecting and not really knowing what to do with, and you've got $100 million plus sitting on a balance sheet. It's like, there's really, I can't come up with anything better to spend than to go in and do something like this, right? I mean, you talk about what is going to transform your business, what is a threat to every business, but also what is a massive opportunity for every business? I don't know how, you know, public company CTOs, CIOs are not like basically all, you know, on the waitlist for this product. And, and I'm not paid to say that, but it just seems like a pretty clear value prop at not that extreme of an entry point.

• 29:13Logan Kilpatrick?

  You know, if you're somebody who's just pushing into this space today, like to go and procure, hire a bunch of people to do the, the actual like model training, like get the resources, all those things like you'd probably spend on the, and you're like a genuine technology company. You'd probably spend in the order of like at least a few million dollars in CapEx anyway, and likely end up with something that's like not as performance in, in many cases as, as what we can hopefully give you for, for two to three million. So I do think there's that angle of it as well. We could sort of outsource some of that stuff to us.

• 29:50Nathan Labenz?

  Yeah, zero doubt about that. I mean, that doesn't go that far in machine learning, you know, researcher salaries in today's world. And, um, you know, good luck, I would say, competing. So, uh, a lot to go, so keep moving on. So the copyright shield is another big thing. I'm kind of just interested in a little bit of the strategy on this and if there are any footnotes, like I believe I saw with Google's policy, which is kind of similar, that there was a carve out, if I understand correctly, like if you're trying to create, you know, if you're trying to kind of jailbreak and create copyrighted stuff and then it like doesn't cover you. I wonder if there's anything similar here. And also just kind of wondering like, I guess, what is the strategy for this? Is this like you want to get into the litigation like sooner rather than later? Or you like feel totally confident that everything you need to license is licensed?

• 30:36Nathan Labenz?

  I can kind of see multiple different reasons for trying to put something like this out there.

• 30:39Logan Kilpatrick?

  Yeah, I don't want to touch on the specifics because I'm looking at the blog post now and we don't link out to like a specific page right now. I think we'll have something that's forthcoming that like goes into the nitty gritty. So I won't make any, any sort of definitive statements about it. I think generally the context on why we want this is we want people to feel comfortable that they're, that they can build with us and not have to worry about copyright claims. And I think that, you know, other providers have done this. I think it's important for us to do this as well. So that I think going back to the comment you made about like public company, you know, CTOs and stuff like that. I think there's a lot of people who want to build with this technology and they perhaps have other folks who, who they work with in a legal department or something else like that. Who are, who are worried about the risks involved.

• 31:20Logan Kilpatrick?

  And it's a, it's a limitation on who can use this technology if we don't provide things like this. So the hope is we provide this, we're able to work with more people and bring more people into the ecosystem. I don't think anybody wants to be in litigation, but it's, I think it's important. It's an important service to our customers.

• 31:37Nathan Labenz?

  In general, I would agree, nobody wants to be in litigation, but it does feel like there's maybe a moment here where, you know, the, the leading developers kind of feel like we need to be party to some of these things to, you know, help shape the, the precedents. I know some of that is already happening. Now would maybe be a good time for you to remind everybody that you don't train on data via the API. We've gone how many minutes and you haven't said that yet, so here's your chance.

• 32:00Logan Kilpatrick?

  Yeah, we don't train on data that you send to the API. I think it's, it's super important. I think in the context of assistance, we don't train on any of that data either. If you upload files, all that good stuff like, again, in a lot of ways, it's, it's a downside for, for certain customers. Cause there's people who, who want actually the models to improve for their use cases. We do have a way to opt in if folks are interested in that, but in general, if you don't want people, if you don't want us to have your data, we won't use it. We don't want to use it. We want people to be comfortable building with the platform.

• 32:30Logan Kilpatrick?

  And I'm very happy that we we made that change in March. I think it's, you know, I had so many conversations with people who, even though we were, it was like a very tiny sliver of the sampling of the data. It's still, you know, there are folks who weren't comfortable with it, and I'm very glad that we we don't have to, we don't have to deal with that challenge today.

• 32:47Nathan Labenz?

  Cool. So let's get into some of the new modalities. I think probably the one that's most exciting for me is image understanding. It just seems like, first of all, it's really good, you know, from what I've seen in chat GPT, amazing. And, you know, for thinking about Waymark, you know, one of our longest standing challenges has been, we collect all these images off the web for the businesses that are going to make video and we use those. But how do you know which ones are good and bad and whatever? It's tough. So we've, you know, incrementally improved, but this seems like it should unlock a notable step improvement. The price is pretty interesting, like pretty aggressive, I would say, you know, kind of maxes out at about one cent per image.

• 33:27Nathan Labenz?

  And then there's this low resolution option. You can kind of go all the way down to basically, it looks like 12 images per cent. If you do the kind of cheapest version, love to get a little bit better color on like, what is that low res thing like suitable for? And also the question on aesthetics, almost no image understanding model that we've seen has a sense of like, how beautiful is this? It's all about the content and very little about kind of the quality. So my, yeah, two questions there are, what's the low res thing for? How do we think about kind of that, that, you know, order of magnitude, price range? And then is there an aesthetic component to it?

• 34:04Logan Kilpatrick?

  Yeah, on the, on the low res version, the, the principal idea is that we take an image, we reformat it to five 12 by five 12. So the easy comparison, I think this is actually in the documentation right now that as I was writing it in, I had a wedding in Mexico this weekend. The idea is if, if you as a human could take an image and reformat it to five, five 12 by five 12 and answer the same question, the model should be able to do it as well. If what you're asking about in that five 12 by five 12 cropped image is some tiny little detail that's shown in one little pixel somewhere, might not be able to pick it up. But if you're like, you know, here's a five 12 by five 12 image.

• 34:45Logan Kilpatrick?

  It's a picture of a highway with a car on it and you ask like, you know, is there a car in the picture? The model should be able to do that. Like you and me as humans would be able to do that. The model should be able to do it as well. So it really depends on like, what is the input image size that you're working with? And what is the level of detail that you need the model to actually go through? With high res mode, it goes through on like a patch by patch basis. And depending on the image size and dimensions, it depends on the number of patches that are created. And it goes in and it again, looks at those five twelve by five twelve, but it like zooms in on each individual section. Which is why the cost is higher because it's processing additional additional patches. So again, the reasoning for folks is look at it as you would as a human.

• 35:29Logan Kilpatrick?

  If you can answer that question, you should be able to do it with low res. If you can't and you need the high res version, think about it from that like patching perspective.

• 35:39Nathan Labenz?

  Five twelve by five twelve is the low res or the low res is smaller than that?

• 35:43Logan Kilpatrick?

  Low res is five twelve by five twelve.

• 35:44Nathan Labenz?

  Wow, okay.

• 35:45Logan Kilpatrick?

  So it's not super small. Like I think five twelve by five twelve is like reasonable. You could take a lot of images and like still understand the gist if you reformat them to five twelve by five twelve. Especially given the price perspective, like in a lot of cases, it might be worth like just seeing if five twelve by five twelve gives you the result that you want for certain use cases before you go and pay more money for for high res mode.

• 36:06Nathan Labenz?

  One angle I've been kind of taking to all these different things is just kind of comparing the prices to other things that are out there. And on the image side, it seems to be lower than basically anything else I've seen that's hosted. For example, if you go use lava thirteen B on replicate. It seems to be, you know, and it's they kind of charged by like the GPU second. But it seems like you can do like three images, three images percent, I should say. With the low res, it's more like twelve percent, but you do still have the output tokens too of like what's generated. That's kind of included. So it seems like it's like competitive or maybe like even a little bit cheaper than other options. Whatever that does not seem to be the case is the Dolly three integration.

• 36:49Nathan Labenz?

  If I understand correctly, that still is at four cents per image. And that's like definitely way higher than, you know, a lot of the sort of open semi open source, you know, or kind of hosted open source options. I'm curious if there's a driver there. Is that like a licensing deal that's gone on in the background? Presumably it's not compute, right? Because like I would think Dolly three turbo would would exist if it was a matter of compute.

• 37:16Logan Kilpatrick?

  It's a good question. I don't also just to go back to your question before about aesthetic stuff within vision. I haven't tried that out. I would imagine it's like it has the same level of reasoning and capabilities as like base GPT four. So like you could probably get the model to like make general assessments of whether something is beautiful or not. I don't know if like, again, it's not going to be like real because it doesn't, you know, it doesn't understand in some sense. You know, the saying is beauty is in the eye of the beholder. I don't know what the, how it will react to certain things. You could, you could probably ask it those questions. But on the question of pricing for Dolly three, I would imagine we're, we're. And this is just like the strategy in general is like we're priced as aggressively as we can be.

• 38:01Logan Kilpatrick?

  Given the technology, given the compute that's required. And you see this in every API release that we do where we're always lowering prices. We're always trying to make things more competitive for our customers. I don't know what the specific drivers are for for Dolly three. I know that it's like, again, at the end of the day, it's like a really, really powerful model. So I imagine that there's significant costs that are associated with us generating those, those images. Hopefully, hopefully it gets cheaper, but yeah, I don't have any. I also think there's like a ton of safety system stuff that's happening around Dolly three. And some of the other like image modality models. So that could be a driver as well, but not a hundred percent sure.

• 38:40Nathan Labenz?

  Gotcha. Okay. So also announced yesterday, Whisper Large V three, not yet available in the API. But interesting to note that English is not the lowest word error rate. So definitely a big focus on kind of lots of different languages, so much so that some other languages, I think English was like, you know, on two benchmarks was like in the number seven spot and maybe the number three or four spot. So I don't know if you have any comment on that, but that was kind of interesting.

• 39:06Logan Kilpatrick?

  Yeah, I'm super excited because, again, one of the core challenges of language models and just like the internet in general is like there is a disproportionate focus on on English and it just makes these products and services that are built around language like much less accessible in places where English isn't the the first language. And I think Whisper focusing on non-English is is a super positive step. And I think it's something that we'll have to do more broadly across all of our models is is really make them sort of world-class in all these languages that are on that exist around the world. So I'm excited for people to get these V3 in the API. So I think it's it's just like a lot of work to spin that model yourself and like run it and do all the things that

• 39:51Logan Kilpatrick?

  are required. And we we've gotten such positive reception from putting V2 Whisper in the API. People people really liked it and made it so that anybody could use it and not have to worry about like all the GPU stuff that's required to to go and run it yourself. But I also think it's important to make some of these things open source. So I'm glad that we did that as well.

• 40:09Nathan Labenz?

  Yeah, cool. I've seen some interesting demos already too of like people changing language kind of mid, you know, audio file and it, you know, handling that seamlessly from what, you know, people are reporting. So some very cool possibilities there. Obviously, the other side of understanding audio is generating audio. So the text-to-speech API, here, pricing seemed pretty aggressive again, comparing the 1.5 cents per 1,000 characters or 3 cents per 1,000 characters for the HD audio to 11 labs. It's like a full order of magnitude cheaper. They're like in kind of 15 to 30 cents per 1,000 characters from what I was seeing today. So that's a notable price drop that's definitely going to probably

• 40:53Nathan Labenz?

  support some additional price drops if I had to guess. I wonder if there's if you have any comments on the kind of the strategy of the voices? This is something where I've I've commented many times that You know, while it may have been sort of an accident or kind of, you know, not expecting the thing to take off as much as it did. The name ChatGPT, I think, is really kind of a service to the public in that you can't really, or it's not easy to anthropomorphize ChatGPT. It just sounds like a robot, and I think there's something really good about that, that like, it's a constant reminder that, you know, it's not a human, even though it may, you know, pass the Turing test or whatever else. I feel kind of the same way hearing the voices. It feels to me like they are at a really nice balance point.

• 41:35Nathan Labenz?

  And I've been a, you know, connoisseur shopper of different, you know, text-to-speech solutions. It feels like it's at a really nice balance point where it's like, still slightly uncanny valley and doesn't sound human. Like, I think I will know, you know, when I'm hearing it. But at the same time, it's like very pleasant and kind of natural. Is that what you were going for with that? To try to kind of, you know, still make clear to people like when this is an AI voice?

• 42:03Logan Kilpatrick?

  I'm not 100% sure, to be honest with you. I think you are probably, as a, as a connoisseur of these voices, probably like uniquely in the position where you could make that assessment. Kind of like how I feel these days when I see like text and it's kind of sometimes easy to tell whether or not it was generated by the computer or not. I think if you look at like the average person who is like not deeply in the AI space, like you could probably show them those voices and convince them that it was a real human. I think that's one of the challenges of, of TTS is that we, and this is explicitly part of our usage and term limits today is you have to tell people that this is not a real human voice. So regardless of whether or not it's, it's, it is not intended to be like extremely

• 42:48Logan Kilpatrick?

  human-like to convince, to, to fault humans that they're talking to a human. We want this to be clear that you're talking to an AI. Just like, again, with the terms of service of all of our, of our products, we're not, we don't want people to be misled. And, and again, I don't, I can't speak to whether or not that was specifically part of the strategy for making those voices. But yeah, they are, they are extremely good. If you, I was playing around with a bunch of them and I think we'll, we'll have more in the future as well, but the existing ones just do a good job of. Striking a nice balance, as you said, I think there's going to be so many cool use cases that people come up with for for TTS. It'll be it'll be exciting here. I was just talking to my my younger brother today and he was working on a um a blog post and I was like,

• 43:34Logan Kilpatrick?

  use TTS like have given audio version of it too. And like, it's just it's going to be so easy. I think the thing that eleven labs, to my understanding at least, and I haven't played around with the product a ton, but um the nice part for them is like, I I think they have like a whole like front end setup where like you you really as somebody who's never programmed before could go in and and generate like take your text and turn it into audio. And I think people underappreciate like the costs associated with doing things like that. And that could be part of the reason why they have um some of the higher costs than than us potentially. And I I haven't actually looked at the numbers myself, but I I trust you. So there's an there's always those non trivial

• 44:11Logan Kilpatrick?

  expenses and like we have some of this. If you look at why chat GPT is more expensive than API, well, there's more going on than just like the bare metal API powering that experience for people. Um, so I I think they'll continue to be competitive and like at the end of the day, our TTS is only available through an API. So like the it requires like a sufficient level of customer experience using or customer knowledge to use a curl request or a Python request or what have you.

• 44:38Nathan Labenz?

  Yeah, I think there's a couple of the things too that that mean that the text to speech companies are not killed by this release. First, the voice cloning is a huge one. You know, I cloned my voice with both eleven labs and Play HT. And that's just not something OpenAI is offering and seems like it's, you know, probably enough of a kind of hairy ball wax that you just like might not get there for a while if ever. Um, and then there's also kind of the emotional direction, which I could see that being something OpenAI might eventually take on. But you know, if you're trying to make games or trying to make characters and like, you know, really trying to bring kind of an energy to it, that's like not your sort of kind of NPR narrator or like call center agent energy, you know, then that's something that's just not in scope right now, but those other companies are already doing, you know, quite well. So

• 45:22Nathan Labenz?

  I definitely think they will continue to have, you know, a very significant, you know, amount of usage for those reasons. Stitching these modalities together, is there like an Omni API coming? Like, it seems like, you know, right now, I have to like call Whisper and then hopefully I can like stream my Whisper tokens into the language model somehow. And then, you know, it will then stream me back tokens and then I gotta send that to the text to speech. But it seems like integrating that into one API where, you know, a stream audio in and a stream audio out, maybe with kind of the text almost as like logs, is the future? Is that kind of what you would envision the future looking like as well?

• 46:01Logan Kilpatrick?

  Yeah, I would imagine we do something like that. And mostly because like right now, and this is the case across all of our APIs, like if you integrate multiple things together, you pay the round trip penalty many, many times. And that's like just the least good experience that you could have. So it makes it kind of sense. And like, again, I don't know when that would come, but I would imagine we end up doing something like speech to speech.

• 46:26Nathan Labenz?

  It seems like that is definitely on the natural path. Reproducible outputs? I thought that was really interesting. Is that essentially equivalent to me caching responses on my side? Or like, what does it add, if anything, beyond kind of if I were to implement my own cache on generations?

• 46:44Logan Kilpatrick?

  The challenge is, even if you cache the generations previously, the generations might be different for the same prompt. So this is fixing that. So like if you're, it's attempting to fix that. It's still in a beta right now, but now you can pre-specify the seed and you shouldn't for a given, if a given input is the same, with the seed being the same, you should get the same output. And the way that you verify that all of these things should be the same is the system fingerprint. Because again, there's much more going on than just that seed parameter behind the scenes. Tons of things baked in. The system fingerprint for two outputs, if it's the same, it should give you the same, the same output. And this just makes like the whole process more reproducible. So I don't think caching wouldn't solve this problem today.

• 47:30Logan Kilpatrick?

  Again, you could, you could cache the inputs and the output and then like give people the same thing, but then that's. Um, yeah, like you could have done that, but I think it's it's in the vein of solving a different problem, which is people want to be able to like thoroughly test these systems. And you can't really do that if you don't know what the expected output is supposed to be like. It's just hard to make unit tests and stuff like that.

• 47:53Nathan Labenz?

  Probably biggest surprise to me was the log probabilities coming soon. I had never expected to see that again for at least for the best models because my thought was you get those log probes. Now you're, you know, opening up much easier, you know, imitation learning for everybody that, you know, wants to kind of match GPT four. I assume that, you know, that's still the case, right? So does this mean that basically there's just no concern about anybody matching GPT four?

• 48:24Logan Kilpatrick?

  Generally, we are concerned about like the leakage of of intellectual property. I think it's specifically going to be log probes for output tokens, which is like slightly different than like pure log probes for input and outputs. I think that slightly changes it. I also think like at the end of the day, like developers have really asked for this and it you there's certain use cases that just like aren't not really possible without log probes. So um, it's mostly it's mostly for those people, like for the people who have asked for the people who want to build products that that require this. Um, I'm happy that we're doing it. There's there's been a ton of of interest for like the last six to 12 months. It's just like how to do it in a way that balances all the other things is is the hard part.

• 49:08Nathan Labenz?

  My read on that is that it really does kind of reflect that all of these like we match GPT four things basically are not close. And, you know, it's because if it was happening that way, you know, presumably it would be a different decision. But, you know, I take this as a as an update that, yeah, all that stuff is kind of bullshit. But you don't have to say that I'm saying that.

• 49:28Logan Kilpatrick?

  I trust your your opinion, but I'm not sure if um.

• 49:31Nathan Labenz?

  Narrow domains, you know, I, yeah, a lot of a lot of caveats there, but we got more fish to fry. So GPT is an assistant. So this is kind of obviously the big, you know, star of the show. Coming into this, you know, the the earlier generation of kind of the expansion of chat GPT was plugins. Plugins, you know, Sam A and others had said, you know, hadn't really achieved product market fit. Because for starters, like. What is conceptually the evolution that you think takes plugins which weren't quite working to GPTs which, you know, hopefully will really work?

• 50:05Logan Kilpatrick?

  Yeah, I think part of the challenge with with plugins was that A, you had to be a developer to make them. B, it was, you know, there was a bunch of like system slash product things that that we just never had time to like perfectly optimize. There were certain things that just like weren't visible to the user that probably should have been and other things like that. I think GPTs is like this, this perfect evolutionary story of where plugins are today. Really, you get like all of the value of of custom instructions of all these different modalities. And like this is, you know, if I go back as somebody who was running the the chat GPT plugin store, people were like asking like, how do we get code interpreter with my plugin? How do I get, you know, X, Y, and Z with my plugin and and really go

• 50:50Logan Kilpatrick?

  deep on on making this like a truly custom experience, owning the experience end to end? You can now do that with GPTs and you can still have the best of both worlds where you you take the API that you were using with with chat GPT plugins and make that accessible as an action inside of it. So it's, I think it's like a slight refactor of the overall framing of the problem. But intuitively, if you're somebody who had a plugin, who had users, you can now go and make a GPT and you should be able to do the similar things with a ton more features built in. Like, again, it was a real trade off for customers to be like, oh, I want to use X, Y, and Z plugin, but now I'm sort of losing out on a bunch of the modalities that I had in my in my other chat sessions. And the

• 51:35Logan Kilpatrick?

  GPT really sort of puts a nice bow on solving that problem.

• 51:39Nathan Labenz?

  So the kind of three elements of this, and then it gets, you know, more nitty gritty on the API side, of course, but instructions, expanded knowledge, and actions.

• 51:51Nathan Labenz?

  Instructions are pretty straightforward. That's your classic prompt engineering. Expanded knowledge is where the platform expansion of bringing the vector database, you know, into the platform is obviously being powered. So people can, you know, upload their own documents, et cetera, et cetera. And then the actions, as I understand it, there's basically three kinds of actions. One is using the retrieval to actually go get the knowledge out of the data store. There was a really interesting tweet posted from a breakout session that kind of showed that like all the tricks had been kind of thrown at that. From your hypothetical document embeddings to your reranking to whatever, you know, at least per the metrics on that slide, it looks like it's going to perform really well.

• 52:34Nathan Labenz?

  So I'm looking forward to getting into that more. Then there's code interpreter, which is where the system can write and execute code on OpenAI's runtime. And then there's the ability to call functions back to the either, you know, that the GPT kind of defines and makes available, or, you know, on the API side that the developer defines and makes available. So that's kind of all just foundation laying. I think people, you know, largely know that. How do you kind of see these things coming together? One thing that I was initially playing with it, and I was like, okay, here's the DALL-E three GPT, and here's like some other GPT, but can I have them together?

• 53:16Nathan Labenz?

  You know, or if I, with the demos too, it was like, okay, here's Zapier, and you know, there's a couple of different things happening there. What if I want to use like Zapier and something else, you know, and my like Google Calendar or and my Slack? Do I have to like tie all those things through Zapier weirdly? Or can I have like all these different things kind of as, because that was one thing that the plugins did seem to have a little bit, at least more clear as of now.

• 53:38Logan Kilpatrick?

  I, I hear that. I think that that's true. I think with plugins, you could sort of enable two or three things at the same time and have those available in the same GPT. You can do this with GPTs today. You would just use multiple actions. I think the, the current product experience, you have to sort of know what the URL for those APIs are. If you were going to use like third-party plugins. So it's, it's really more so designed today with actions of like you using your own API and your own, your own plugin manifest or open API file. I think it's it's possible that that changes over time to like allow more for like the integrating of Gmail. At the end of the day, like, people really want that. They want to be able to say, like,

• 54:24Logan Kilpatrick?

  I have this super powerful single GPT that does, you know, these five different things for me and I go in and I check the different modalities that I want, or I check the different features and services that I want. We're just not there today. I think eventually we'll get somewhere that that has that because that's that's what consumers are going to want. Whether it's actions or some different iteration of that, I don't know for sure, but it's definitely something that makes a ton of sense to have. Like, you don't want every single developer, every person making a GPT to have to go reprogram like their integration with the Gmail API, for example. Like, you just want something that sort of works across all of them and you can just one click check that it works and then you're good to go.

• 55:02Nathan Labenz?

  Yeah. So for now, if I'm an enterprise and I've got G Suite over here and I've got Slack over here, basically I can go and just set all those different things up as

• 55:15Nathan Labenz?

  actions, you know, kind of mix and match on my own, but I don't have like a higher level, you know, click this, click that and just kind of naturally import. But we but you can do it if you, you know, have the kind of wherewithal to go list out all the actions that you need.

• 55:31Logan Kilpatrick?

  You can do it and you could also technically use some of the third-party plugins to do this as well. I think the real challenge, and this goes back to like why hopefully GPTs are a better evolution of this is like, you just have to be when you don't control the plugin yourself or you're it's not you like writing the code and stuff like that, you just have less control over like what's happening with this data that I'm using or that I'm sending back and forth. And I think there's like a strong incentive for like Gmail hasn't actually made a plugin. It's any any email plugin is like a third-party provider. So you really have to be comfortable with like, I'm making my data accessible to some third-party and like the chances are you don't know who that third-party is really strongly. There should be, I think, a bias for

• 56:16Logan Kilpatrick?

  companies, especially making these things to think about like, you should probably make the Gmail integration yourself. It makes a lot of sense.

• 56:24Nathan Labenz?

  I'm very, very careful about who I allow into my Gmail and G Suite more broadly, certainly. How are you handling off on this stuff? That was, you know, kind of skipped over in the demo, but you know, with the Zapier one, for example, how do I like connect this to my Zapier and then? On that end, I assume like all the, you know, if Zapier is integrating to Slack and Gmail, whatever, like that's kind of within the Zapier platform. Is that right?

• 56:49Logan Kilpatrick?

  Yeah, for the Zapier case, that's true. I think Zapier has done some interesting things. I don't want to say getting around this, but to enable it in like a more seamless way. You do so like each individual endpoint as part of an action, you can specify the auth for. And actually, I think, again, going back to like the differences between plugins and actions. This is one of the positive things about, or very, very positive, one of the, one of the many positive things about actions is that you now have access to like hybrid auth. So you can have certain endpoints that are, that don't require auth. You can have certain endpoints that require auth. You can also set certain actions as consequential or inconsequential. And that controls whether or not the user is prompted to like explicitly allow a certain

• 57:34Logan Kilpatrick?

  action to happen, so that the model's not like doing things on your behalf without your consent. So I think both of those things are like core features that plugin developers have been asking for. Like, how can I make sure that users are knowingly sending data somewhere? And how can I make sure that I don't gatekeep everybody at the very first step and force them to sign in and allow them to sort of, you know, get their feet wet with my product? And then when they want to do something like save it or what have you, it forces them into auth. So I think both of those things are now, are now possible with, with actions.

• 58:07Nathan Labenz?

  That saving mechanism, is there anything like that for the retrieval that's built in? Like, could I update a document or sort of add to the files?

• 58:19Logan Kilpatrick?

  Yeah, for, for GPTs, you can go in and edit them after you create it. So you can actually change anything. So it's, it's the instructions can be iterated on. The, the files can be iterated on. You can go in and change your, your custom, your custom actions. If you want to, you can turn off certain capabilities. So it's, it's all fully customizable, even after you've initially created the, the first incarnation of that.

• 58:40Nathan Labenz?

  The retrieval, again, it looks pretty amazing and the annotations are cool as well. There's some really nice developer conveniences that have been added to, like the files, you know, a few, several new tabs within the kind of just platform website. I'm looking forward to when files have a little preview of the first, how many lines of the file as well. It's just a, you know, maybe breadcrumb, not that that one's unobvious, but.

• 59:03Nathan Labenz?

  A question on the billing. So it's 20 cents per gigabyte per assistant per day of content backing, right? Is that, is like one gigabyte the minimum? Or is there, is there some smaller gradation of storage that is billable?

• 59:21Logan Kilpatrick?

  It's a good question. I don't know off the top of my head. That's also specifically for file retrieval. So it's specifically for uploaded files that get embedded. I would imagine that that's like, there's also a breakdown of, like if you only upload half a gigabyte, I'd imagine you're being charged half that, half that price. I'm guessing it's not just like, you get charged 20 cents regardless of how much data you put in there. And I imagine it's, it's broken down. I could be wrong about that. So I'll, I'll test and follow up afterwards, but I haven't actually tried it to know off the top of my head what the billing looks like.

• 59:57Nathan Labenz?

  I guess vision over time would be maybe more kind of control over retrieval as well. Right now it seems like it's very black box, like as far as I could tell, I don't have any way to kind of say what I want my chunk size to be, or you know, how many records to pull. That does have some implications if I'm like pulling a bunch of records and then, you know, using that in, in context because those are tokens, right? So am I right that it's pretty black boxy right now? And I guess more to come?

• 1:00:27Logan Kilpatrick?

  Yeah, you should have some level of control because you could tell the model and the, and the user message or in the system message, like how you want it to perform in those cases. And like, at the end of the day, it's like, it's still GPT four, GPT four turbo in this case. So it will follow those instructions. So if you say like, always only answer questions based on the first three images that I upload, like it should be able to follow those instructions and ignore some of the, some of the other stuff. So you do have that level of control from like a chunking perspective. Maybe that's something that we'll end up releasing. What tends to happen is like, we end up making the thing that is going to be like most usable for most people. And you see this actually, fine tuning is a good example of this where we actually like automatically set

• 1:01:09Logan Kilpatrick?

  the hyper parameters for you. And in certain cases, you can override those and like change them yourself, but like we're doing it based on the best practices of us fine tuning many, many, many models and people just tend to like. More often than not modify those parameters and like it ends up being detrimental to their like it's a net negative for them instead of being like a net positive. So I I would imagine it's possible that like the patching or chunking for the images ends up being something that we we give people more control over, but it's it's not possible today.

• 1:01:40Nathan Labenz?

  There the fine tuning UI also within the platform site, definitely a really nice upgrade. I love being able to see the loss curve even just to get, you know, a little bit of a a sense for how long is it dropping? Where does it start to flatten off? You know, why is this one run like just generally lower than the other? Definitely a lot for me to kind of dig into more there and and better understand what's happening with fine tuning. And we can use fine tuned um three, five uh sixteen K function calling for assistance. Is that right? Or is that not yet?

• 1:02:12Logan Kilpatrick?

  You should be able to do it for assistance. I'll I'll double check that it's um that it's compatible today, but you fundamentally should be able to.

• 1:02:20Nathan Labenz?

  Yeah, clearly this is where it's going. Cool. So a couple just questions on like best practices or conceptual stuff for the assistance. You know, I'm doing a bunch of prototyping stuff with this company Athena in the executive assistant space. We've got all these clients. They're all very different. And so I'm kind of wondering here, is this a situation where I create one like Athena assistant and then I provide like runtime context differently, you know, with kind of our client profile file and maybe some recent history or whatever. But it's all one assistant with, you know, a thread per client? Or should I be thinking about it more as like one assistant per client? And how would I know the difference?

• 1:02:58Logan Kilpatrick?

  Yeah, that's a really good question. I think it fundamentally depends on like whether or not each assistant would have different different instructions and different capabilities. You could make images or you could make files accessible at a specific thread levels like you you can if if that's the only thing that you want to change between them, then that would make sense. I do think it probably. You probably would want to like tailor the assistant on like an individual basis, or at least is the thing that I would I would consider trying. You just have to be more general and like it'd be more useful to like take all the context that you have about somebody and like put or about the whether it's the actual person who's the assistant or the person who's the the client and like put that into the

• 1:03:43Logan Kilpatrick?

  the system instructions um for the assistant specifically. So you probably just lose a little bit of customers customization if you try to make like a general assistant, but I do think that would be like a worthwhile starting place. And then, you know, having an option to like go and and customize deeper potentially as a as like a next step.

• 1:04:02Nathan Labenz?

  Okay, cool. Makes sense more definitely a lot of discovery ahead on that front. Safety, you mentioned safety. There wasn't too much talk about safety yesterday at the keynote. Could you give us any information on like the red teaming that went on behind this? I mean, you know, I know OpenAI has obviously been very engaged in policy, has made White House commitments. So per those White House commitments, if nothing else, like clearly there was some red teaming. You know, I don't know if there was like a report submitted to the government or if there will be any sort of like system card coming, especially, you know, with this kind of assistant paradigm. But it does seem like, you know, probably a lot went into that and I'd love to hear anything about it that you can share.

• 1:04:43Logan Kilpatrick?

  Yeah, it's a good question. If if you look broadly at like what assistance is, it's it's not fundamentally something that's like a different capability than has existed previously. It's really like chat completion strung together with a database and then a bunch of other modalities that we already had accessible other places now accessible in assistance specifically. So we've red teamed and done a lot of the safety work like for each of those individual modalities on the I I think like the thing that I can publicly point to um from a safety perspective is around vision. So we did release like a a vision safety card um GPT four with with vision that folks can go out and read. It's uh available on the OpenAI website and there's like a full eighteen page uh breakdown

• 1:05:26Logan Kilpatrick?

  that goes through all the all the questions and challenges around vision. I think it's it's actually really interesting if you want to get a sense for like the things that we've thought about, the high level problems of like trying to make a general reasoning system. And it's um it's incredible to see the work. And I I think there's also like a a page that goes through like all of the people that we've worked with to make vision come to light. And if you think about like You know, what was the limitation from getting Greg's demo of Vision working back in March to actually making it accessible as a first party and third party product in our APIs? Like, all of that safety work really is like one of the huge things that we've done. And again, it follows the same trend of what happened with GPT-4,

• 1:06:11Logan Kilpatrick?

  where, you know, GPT-4 was ready long before the model was actually released to the public. And there's just so much safety work and thought that goes into how to build systems in a scalable and safe way. I'm happy and yeah, huge kudos to the teams that have done the work and all the right teaming. That's happened.

• 1:06:28Nathan Labenz?

  Cool. Yeah, I'm going to go in and see what I can do on the jailbreak front. A while ago, there was a like bug bounty program or kind of a teaser of that. Is there, is that actually happening? Do I get paid for jailbreaks now?

• 1:06:40Logan Kilpatrick?

  Jailbreaks is a good question. I don't know if jailbreaks fall into the considerations of the bug bounty program. I believe the bug bounty program is constrained to like you find an exploit to like access one of OpenAI's systems or something like that. That program is definitely live and accessible. And we definitely find bugs and people definitely get bounties, which is awesome. I don't know if jailbreaks are a part of that or not. It has, we use the platform that everybody uses via Crowd something. I forget the name escapes me at the moment, but it's the same platform that every company uses for this. So it has all the eligibility details and it, I believe, will say whether or not jailbreaks are one of the things.

• 1:07:22Logan Kilpatrick?

  My understanding is that they're not because it's not, it's like more of like a system level, like security vulnerability, rather than like a model behavior thing.

• 1:07:33Nathan Labenz?

  At the next, you know, roundtable about this, put one in for me on moving jailbreaks into the eligibility category for that. I honestly think it would serve everybody pretty well because lord knows there's just insane surface area here. And, you know, you need a lot of people to kind of go out and explore weird spaces. So a little incentive, I think, is pretty helpful.

• 1:07:53Logan Kilpatrick?

  I think that's fair. Yeah.

• 1:07:55Nathan Labenz?

  So the last thing is, you know, at the end of the keynote, Sam said, this is all going to look quaint compared to what OpenAI is cooking up for us now. I mean, that's pretty insane. Yeah, obviously, I know you, you know, aren't going to tell us what we're going to have a year from now. And I guess my real question is, to what degree do you guys even know that, right? I mean, the is, do you have a roadmap that is kind of well established in your own minds for the next year? Or is it going to be the kind of thing where you've raised the platform level? Obviously, now another thousand, you know, kind of things, if not a million things are going to kind of, you know, blossom on top of that platform and then take all that into account and kind of figure out what the next, you know, platform raises. I guess it's probably both, but how do you think about that balance?

• 1:08:39Logan Kilpatrick?

  Yeah, I think the research roadmap is both more laid in stone, but also less, less predictable. And I think that's going to drive a lot of the like, what ends up being released is like, what research are we able to do? And what are the new sort of breakthroughs that we're able to achieve that then enables us to bring products to market? So I think that that's like the biggest open question. The research team knows where they're making bets as far as like, you know, because they actually have to allocate compute to these things. So they they know in the top of the head of like, you know, where are the 10 places we're making bets? And what what could those things potentially yield? I think the real question is like, what do those things actually work and result in like a amazing new product modality, whatever it is, and making that accessible.

• 1:09:23Logan Kilpatrick?

  So I think it's, I have less of that insight because they're actively doing the research right now and we we don't yet know. But yeah, it'll be every, every time after we do a big release like this, I'm like, there's nothing left to ship. There'll be nothing and I'm I just, just, you know, talking to you today and seeing the announcements yesterday and listening to a lot of the conversations, like, there's so many things that are yet to come that are going to be super, super important. And it's, it's a lot of obvious things, too. It's like not, not things that are like crazy groundbreaking, but are going to lead to people to be able to do the, the incredible crazy groundbreaking products. So I'm excited about that.

• 1:10:02Nathan Labenz?

  Fast times at OpenAI always and in the AI space more broadly as well. We'll be digesting this for, you know, weeks, if not months to come. And for now, I just want to say again, thank you for all the time and all the clarifying answers. Logan Kilpatrick, thank you for being part of the cognitive revolution. Thanks for having me. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.