Ep 24: OpenAI Head of DevRel Logan Kilpatrick on The Best ChatGPT Use Cases, Future of Agents, and Google Gemini

OpenAI's inaugural DevDay sparked excitement in the AI community, with several product releases and ChatGPT hitting the milestone of reaching 100M weekly active users. On this week’s episode of Unsupervised Learning, we sat down with the Head of Developer Relations at OpenAI, Logan Kilpatrick. Logan shared with us how OpenAI prioritizes product builds internally, the interesting use cases he's seen for several OpenAI products, where OpenAI is headed, and what the Gemini release means for the ecosystem.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  This is Unsupervised Learning, Red Points AI Podcast. I'm Jacob Effron, and today we had an awesome guest in Logan Kilpatrick, the first DevRel hire at OpenAI. Logan has what I think is truly one of the most interesting jobs in the world, and our conversation certainly reflected that. We talked about the breadth of offerings OpenAI has, and how people are building on top of them today, how OpenAI prioritizes what they're going to build. We talked about agents in the future and what it means for the Internet. And we also hit on Google Gemini and what the model means for the ecosystem. I think folks are really going to enjoy this one.

• 0:27Jacob Effron?

  Logan, thanks so much for coming on. Really appreciate you having me here.

• 0:30Speaker 01

  Yeah, I'm super excited. This is going to be an awesome conversation.

• 0:32Jacob Effron?

  Awesome. Well, to start, I have to ask, I think we just celebrated the year anniversary of ChatGPT. If I remember correctly, I think you were like literally hired pre-ChatGPT, but then started like when there was a million users or something. So pretty crazy transformation there. How do you use ChatGPT like in your life?

• 0:49Speaker 01

  It's a lot of coding things. I think like honestly, I spend like a decent portion of my time trying to make our developer platform actually better. And I am not somebody who is like classically true. Like I have a computer science background, was a software engineer, but I'm not like a web development expert. So I think there was a lot of things that I don't know. And I have like the idea of exactly what I want for developers when they show up on our platform, but it's oftentimes hard to translate that into code. And like half of the features that I ship are like 90% ChatGPT code. And it really like, as an engineer, has given me like so much more freedom to go beyond what I would normally be able to do without having to like spend 13 hours like reading the documentation for React and all these other things. So it's been awesome. That's, that's like a bulk of like true coding things is still like what I get the most value on a day-to-day basis.

• 1:34Jacob Effron?

  I love that. I mean, one thing I'm struck by with your job of kind of, you know, the developer side of the house is there's just such a breadth of products you guys have. And obviously, a lot of these are incredibly widely used. I'm kind of curious, are there things that you have shipped at OpenAI where you're like, I don't think a ton of people are using this today, but like in the future, it's, it's going to be something that a lot more people use. It's funny, I remember we actually talked in March and you were like embeddings, watch out for that. Like that's something that, you know, is underused today. As you look across the product suite, is there a particular set of things that you're like, I think, you know, in a year from now, there'll be, be a lot more.

• 2:06Speaker 01

  Yeah, I think maybe it's not unexpected, but I think the assistance API is what is going to end up being like a long term thing that it just takes a while to ram people. But I think part of the reason why is we released a beta product. There's like a bunch of things that we still need to build to make it like an incredible product experience, like the reg strategies, giving developers more customization over that. But it's very clear from people's reaction that like they want this, like they don't want to have to deal with all the nitty gritty things themselves. I think there's a lot of customers who like do want that and we should give them the knobs or they should use like our embeddings API and Llama Index or Langchain or whatever it is to like continue to like build a more complicated product. But I do think if you look at where we are today, there's going to be so many more experiences

• 2:49Speaker 01

  a year from today that are powered by assistance, just because like we're going to build a bunch of great tools. You already have code interpreter, like it's, it's really exciting. And to think back six months ago from today, the fact that like you couldn't use code interpreter in the API and now like that's the thing that people were like most excited about in many cases with chat GPT and it's available to like everyone now to go and build into your product. And I'm really excited to see what people build. I think it's going to be incredible.

• 3:14Jacob Effron?

  That's awesome. I thought you were going to say multimodal because I remember you did a talk recently where you're like, next year is the year of multimodal.

• 3:19Speaker 01

  Yeah, I think there will be a bunch of multimodal stuff. I think we're still early on the vision use cases. I do think that like what you can use the vision model with today, there's a bunch of domains where it's like constrained. I think it'll get better over time, but I think a lot of the really, really cool stuff requires like a very detailed, the model has to have a very detailed understanding of like the positional relationship between objects and the image. And I think the model does a good job at this today, but I think there's a lot of use cases that don't work well until it like is perfect at it. And I think this is like the conceptual difference today with like GPT 3.5 and GPT 4. Once you make that jump, all of a sudden, all these new use cases are.

• 4:02Speaker 01

  Are unlocked because it's so much more robust. And I think the same thing is going to be true for vision where like today, we're kind of in like the GPT 3.5 vision era. And I think once we can make the jump to the next level, like that's really when like so many more use cases will be robust. So that's, that's the piece that I'm waiting for.

• 4:18Jacob Effron?

  You know, a lot of our listeners are probably using, you know, GPT 3.5, GPT 4. And, you know, I think a question we get a lot is, you know, which one should I start with or where should I begin? And I thought one thing that was interesting in in Dev Day is you guys obviously announced the ability to fine tune GPT 4. But I think it's like rolled out to people that have already tried to fine tune GPT 3.5. And so I feel like this, the sequencing is probably clear in your head of like, you know, in which use cases and, you know, what ordering of trying should one go through. But curious like how you'd articulate that to folks who are building in the space today, like where, where should they get started on the, on the opening offerings today?

• 4:52Speaker 01

  Yeah, it's a great question. I think fine tuning is a slightly different story and we can talk about that as well. I think it really tends to be that like, you just have to do a little bit more work to like finely craft your prompt. If you want something to work like really, really well with 3.5 turbo. I think that model is super capable and there are a bunch of use cases that it works great for. I think after you get out of the scope of like three or four instructions in a single prompt, I think that's where it really starts to break down. And that's really where GPT 4, like you can see the delta and the performance. So I think as you build more complicated requests for the models, switch over to GPT 4. But I feel like now, especially with after Dev Day, all the price drops that we had, it's like, it's so much more affordable to use those models. I feel

• 5:37Speaker 01

  like a lot of people just like dive straight into GPT 4 today, just because it's like not worth the extra effort of having to try to finagle your way into doing it. And for people who are trying to stick with 3.5, they go and fine tune it and they can, we shared in the blog post when 3.5 fine tuning came out that we, a lot of the early customers that we were working with were getting GPT 4 level of performance using GPT 3.5 by fine tuning and doing some prompt engineering and they were getting like token saving toss and all these things. It's, it's exciting.

• 6:07Jacob Effron?

  Do you have like, I mean, it sounds like one of these rules of thumbs for you is, you know, when you're having to input like Or four specific requirements into a prompt like any other rules of thumb you use from like, yeah, it might be time to to try the larger, you know, more costly model.

• 6:19Speaker 01

  Yeah, I think like a lot of basic things like the knowledge cutoff piece is another big one. I think there's also like a bunch of model level improvements. And this is all talked about in our documentation that like haven't made their way to 3.5 that only exists in in the four turbo preview model today. So I think if you want some of those things, like I think there's probably some delta in performance between function calling, for example, which is like another huge use case that people care a ton about, but also another another great area where you can go and do fine tuning, you can fine tune 3.5 for function calling. It works really well. You can actually like if you had like 50 functions, you can fine tune a bunch of data using, give a bunch of function calling data and actually like fine tune the functions out

• 7:03Speaker 01

  of the prompt so that you don't because you have to pay for the tokens of the functions for every call and you can literally make those go away. And then the model essentially just like hallucinates the functions, which is like a weird, a weird byproduct, but it's like great for developers because you don't have to pay the 50 function token cost every single time you send a response or send a prompt.

• 7:22Jacob Effron?

  Makes total sense. And, you know, one other announcement I thought was really interesting that you guys had is you you introduce these custom models, right? And I feel like everyone's kind of intrigued where that fits into the strategy long term, you know, who the right folks are to to be building those like, how do you think about both today and in the future where that fits into to the kind of open AI offerings?

• 7:39Speaker 01

  Yeah, I think today you really have to have a lot of data and I think you have to have a lot of data in a domain that the model's not already super great at. So it's usually like, you know, you can imagine the legal space or the medical space areas where like the model just like might not have the best access to the data and like your company does have access to great data and you could think of a thousand different companies where this where this might work well for them. I think. It probably will end up being that a lot of the customers that we work with, a lot of the early use cases with custom models, the goal actually is let's take all those learnings and let's make it more accessible. Because I think like $2 to $3 million for a model, or however much it ends up costing, is great for those customers who have enough money to pay for it. But I think

• 8:24Speaker 01

  there's a lot of developers who like want to benefit and like startups who want to benefit from this technology who like don't have $2 to $3 million in CapEx that they want to go and throw at trying to make one of these models. And especially with in a lot of cases, like how great the open source models are, and you can use those and fine tune. And like, it's also nice because in those examples, you don't need the same order of magnitude of tokens. Like, I think it's, I don't remember off the top of my head, but I think it's like, you need like billions of tokens to do one of those custom models. And like, most people don't have good enough data where it like actually makes sense to do that, unless you have a bunch of like trove of information. So it'll be interesting.

• 9:04Jacob Effron?

  And do you think, I mean, can I imagine two versions of the future? One is, obviously, as your underlying models get better, you know, the amount of people that you're like, oh, the base model doesn't do my space as well, you know, decreases. On the other hand, it feels like there's a big desire from enterprises to train on their own data, to, you know, have, that's part of their proprietary advantage. Like, do you imagine, are we at like, peak need for custom models now? Or is that something, you know, how do you see that evolving over time?

• 9:29Speaker 01

  That's a great question. I think the future is always going to be really great base models. I think the base models can continue to get really great. I think they'll be more steerable, which will help this problem in a lot of ways. I still think there'll always be the need for custom models. I think also just like, it gets into like the really nitty gritty details of the performance. And I think the custom models, you know, over time could become more efficient in some ways because you could, and again, I don't know the specific details of how we're doing all these engagements with people. But I think you could like remove certain data from, from training sets that like, just the model doesn't need to have. So you get a more compute efficient model, because you can then pump in all the data that like you really care deeply about. So I

• 10:12Speaker 01

  think there'll be a bunch of things that like, the Base model improvements will never solve the problem fully and you'll you'll still need those custom models. And I think the real question is whether or not that custom model offering like ends up being something that we can get into an API and like make affordable for for all developer. I mean, it'll still probably always be more expensive just because of the process, but I'm excited for that day and I hope we we push in that direction.

• 10:35Jacob Effron?

  Totally, like you bring your own data and kind of just figure it out yourself.

• 10:38Speaker 01

  Exactly.

• 10:39Jacob Effron?

  Though I guess obviously you still need the billions of tokens to to make it, you know, useful to do on your own.

• 10:43Speaker 01

  It also takes a lot of expertise and I think this is part of the reason why we've wanted people to fine tune 3.5 turbo before they went to four is like it's just it actually starts to require like some level of of like machine learning, understanding to like go and do that well. And I think a lot of people, at least we've seen with 3.5, like you might not actually get great results if you didn't make great data. And like because of how accessible these APIs are, like this could be your first foray into machine learning and like never have trained a model before. And I think hopefully the custom model program actually does like help solve this problem in some sense. Cause like our research teams like actually hand hold you, you could say I have a bunch of data, but we don't have, you know, the machine learn

• 11:25Speaker 01

  world-class machine learning team that's going to help us train these models and our team comes in and actually helps them do that, which is really nice.

• 11:31Jacob Effron?

  Totally. I have to imagine that has like the world's longest waitlist.

• 11:34Speaker 01

  I don't know. You'd be surprised. I feel like if there's companies that you're thinking of that this would be great for them. I'm happy to connect. I think it'd be interesting.

• 11:41Jacob Effron?

  Yeah, no, that's awesome. I'm sure we'll get some pings on that as when this releases. You know, one thing I'm always intrigued by when you're in the team's job is just your product footprint is so large now. You serve so many different types of users and use cases. How do you guys prioritize what to build? Obviously both the product resources, you have limited GPUs. How do you allocate those? How do those decisions get made?

• 12:01Speaker 01

  Yeah, that's a good question. I think this is our leadership team does an incredible job of trying to balance all of the, there's so many. It's one of the most difficult environments to like make those decisions. Cause I think it's like, on one hand, there's a bunch of things that are like really, really clear that developers want. And and one like very tactical example of this is like API key base usage. People want to go into the usage dashboard and see what the what the spend is per API key. And like, it's actually not like a extremely difficult engineering feat to make that happen. Like the teams can do that. It's just the trade off against that and like reliability as an example. And I think There's a bunch of things that continue to be the North Star. I think one of them is like providing a world-class service to our customers.

• 12:44Speaker 01

  And that requires that we continue to make a bunch of investments in reliability. And I think if you can't do that well, none of the other stuff really matters. I think the other piece of it, as well as we've been so constrained from an engineering resource perspective, that we've really had to focus on like shipping new capabilities. Like, we have all this stuff lined up. We have the like actual research work done. It's just about productionizing that. And to be honest, like that oftentimes takes priority over like the features of like building dashboards and monitoring and alerts and stuff like that. But I think as the team grows, like we'll just have more time naturally to do that work. And those are also the things that I think the like rough edges that we just now have the time to go and fix. And hopefully we'll ship a bunch of those things in 2024 and like really have like

• 13:28Speaker 01

  a true enterprise platform that people can be really excited about using.

• 13:31Jacob Effron?

  I'm always struck by it seems like some of the things that, you know, are seventh or eighth in the announcements on Dev Day, like, you know, even the text to speech stuff is like, there's entire companies that that's just all they do. And it's amazing, uh, it's amazing the breadth you're able to cover.

• 13:42Speaker 01

  That text to speech model is so incredible. I think it's, I've been playing around with the demos for, for months and months and months now. And it is just, it's so good at, at doing what it does that I'm, I'm, yeah. The future for that product line is going to be so exciting because like there's just, there's an infinite amount of use cases where those voices just end up being like really, really helpful. And 11 Labs has done an incredible job. Like they have such like wide distribution. And I think there's, um, it's not clear to me if that's like the case for developers, like developers. I feel like 11 Labs has like been a very consumer play and they've done a great job of making it accessible to people who are making content. So it'll be interesting to see like whether they retain a bunch of those users or whether people come over to use our API. But hopefully it's our API.

• 14:27Speaker 01

  I think it's great. Hopefully we'll see.

• 14:29Jacob Effron?

  That makes, uh, makes sense. I mean, I guess I'm curious, you know, obviously there are so many things people come to OpenAI for. How do you think about when it makes sense to use open source models or when it makes sense to, to not be in the OpenAI ecosystem?

• 14:42Speaker 01

  Yeah, that's a great question. I think it's it's a little bit, you know, I think in some ways people want to own their own destiny. And I think I hear people when they when they talk about like, at the end of the day, if you have an open source model, like you own the the weights, you own the IP. And I think like, there's a bunch of business use cases where that makes a lot of sense. I my personal opinion is that I continue to think that like the models we ship are always going to be better than than open source models. I think there's it's maybe less apparent today because of like just the orders of magnitude of these models. Like at a certain point, there will be points where like, it's just really, really

• 15:18Speaker 01

  costs intensive to like train these really, really large models. And there's just so much engineering work that goes into that. And I'm a deep believer in the open source community. Like I before I was at OpenAI, I spent a long time helping build the Julia ecosystem and the open source Julia programming language. So open source runs deep in my veins, but it's just the the reality of it is that the large models from these companies will will always end up being better. Again, I think like if you want to own the weights and and that's like critical for your business use case, I think it makes sense. I also think that because of the weights being open source for some of these models like Llama, for example, the types of customization and things you can do with fine tuning is just a little bit like more than we offer in our like standard fine tuning offerings. I think that's another

• 16:03Speaker 01

  use case, but I think that will sort of like slowly shift over time. We'll be able to like do more of all the different fine tuning and like training techniques that people want, like give them, you know, again, not not committing to this, but like, you know, people want RLHF, the models and stuff like that. And I think like you could technically do that with open source models today, and it's not something we support. And I think those are those are all examples of things that I think make a lot of sense for us to do in the future.

• 16:26Jacob Effron?

  I feel like one popular narrative is like, you know, OpenAI will always have the state of the art model, and that will always be able to push the boundaries of what can be done. But maybe what you could do with an OpenAI model a year ago, you can then use an open source model off the shelf. And I think there's there's still this question about like, okay, so if you're doing kind of GPT three level capabilities, is it going to be cheaper using the open source model? Or you guys keep slashing prices. So it honestly might be. But you know, I think that's that's I'm curious for your take on that for some of the, you know, less, I guess it was cutting edge a year ago. And now it seems like there's a bunch of models that can do those use cases.

• 16:56Speaker 01

  Yeah, I think it depends on the sophistication of the user. Like, I think there's a lot of people, like even for me today, and maybe I should do this and figure out how difficult it is, but like just spinning up some like going on replicate or something and like doing all that work that's required.

• 17:11Speaker 01

  Seems like it could be a hurdle for people. And I think there's like a bunch of like dev X value, like just the experience of just being able to go and use an API and not have to worry about getting GPU allocation, which is impossible to get, or like using one of these other, these other providers. So I think that's, that is part of the equation as well.

• 17:28Jacob Effron?

  Makes sense. And I'm curious, like, what tools do you see people most commonly use alongside OpenAI when they're kind of shipping their products?

• 17:35Speaker 01

  That's a, that's a great question. I think there's a whole slew of, of observability products that are out there and, and different. And I won't, I won't name specific names because I, because I'm an angel investor. I know the founders or whatever it is. I don't want to.

• 17:48Jacob Effron?

  I feel like if OpenAI endorsed one on the podcast, they would just talk about that forever. They'd be like, we're, we're the official one that's been stamped.

• 17:53Speaker 01

  Yeah, there's a lot. And to be honest with you, I think they give a lot of value. Like, I think I was, I was meeting with someone on our team, one of the PMs, and he was saying to me, he came from Stripe and he was saying how it's like his mental model of using APIs was like so deeply built on top of what Stripe offers, which they have this like dashboard where every time you can do something, you can see in the dashboard, like the, the logs and everything like that. And he was like, it was impossible for me to use the OpenAI API because we don't have that. And he was like, that was one of his reasons for going to these observability companies is just to be able to like see the requests as they come in through, through the proxy services. And I think like that makes a ton of sense. I think at the end of the day, like we should build that for developers because it's what people want. It's extremely useful to have.

• 18:37Speaker 01

  I think like from a, a more tooling, like less observability, more like actually building features perspective, the Llama indexes of the world, the Lang chains of the world, people are using a ton. It'll be interesting. Haystack is another one. It'll be interesting to see how that changes over time. Like, I think Lang change is a great example of. Their open source project has done incredible, like Langchain itself has done incredible. I like Harrison a lot, so I hope Langsmith is really successful, but it'll be interesting to see like, do developers and enterprises actually end up adopting those tools? Or like, where's, yeah, where does the value end up being accrued? And I think like, having an open source project is great, but at the end of the day, they need to make money and build a real business.

• 19:20Speaker 01

  And it'll be interesting to see if Langsmith is like, what, what gets the most traction from that perspective as well.

• 19:27Jacob Effron?

  Totally. So I guess, yeah, you have, you know, obviously you see kind of developers hanging together projects. And then you see kind of the more enterprise side of things. On the enterprise side, like, are they using surrounding tools today? Or is it mostly like, you know, they use the APIs or they use kind of OpenAI off the shelf?

• 19:40Speaker 01

  Yeah, they, the people are definitely using those tools. I think it, again, it, enterprises are so interesting to see, like, everybody is so different. I think there's a lot of companies that like, even companies that have a lot of technical sophistication that are using these tools, just because like, whomever is the engineering manager, or like, the engineering director is like, made the decision that it's not worth their end time to like, go and build those things. But I think there's also a lot of companies that are like, very uncertain about taking on those dependencies, especially given like, how the whole open source space, when it's like a venture backed company as well, I think always is like, much different than a lot of like, like, traditional open source projects that are like, truly open source and like, built by the community and things like that.

• 20:23Speaker 01

  So I think a lot of companies have like, gone and rebuilt their infrastructure. And I think it's, my, my take is it's surprising how far you can get, or like, how,

• 20:34Speaker 01

  it's not that much work to go and rebuild a lot of this infrastructure yourself. I think at the present moment, I think like, two years from now or something like that, maybe that'll be like, completely different and everybody will just use these off the shelf tools or some SaaS product or whatever it is. But I think today you could actually get away with rebuilding most of the things that you need. And like, this is how I, like, when I use our API, this is mostly what I do, to be honest. Like, all the abstraction layers, I just go and build myself, because it's not a ton of work in most cases.

• 21:01Jacob Effron?

  And I guess, you know, I feel like you have this fascinating seat, seeing how folks are building this. If you were to like, leave OpenAI and start a company, like, is there like, a particular space? You're like, oh man, I really wish someone was solving that problem. Or as you think about blockers that developers and enterprises have.

• 21:16Speaker 01

  It's a good question. I don't think about leaving OpenAI often.

• 21:19Jacob Effron?

  No, I would never.

• 21:22Speaker 01

  That's a good question. I maybe we'll I'll think about it and we can we can come back to it. I'm not sure if there's something like I think the reality is like the Lang chains and Lama indexes of the world have solved a lot of these problems that developers have. I think like even like prompt layer is another example of like a company not solving like a different lens of the problems and like managing all your different problems as they come in. I think the thing actually that I'm most excited about today is I do have an answer for the question is around evals. Like I think somebody should really go and make a a company that's like natively only trying to solve the eval problem because I think it's like a fundamental challenge for people using LLMs. A new model comes out, whether it's from OpenAI or some other provider.

• 22:05Speaker 01

  I don't know as the model, as the end user of that model, how it's going to impact my use case. And really, the only way to do that is to have a bunch of robust evals that like you go and build. And the process of making evals today is a huge pain, pain in the butt. Like it's not fun to it's like not fun work to do. And I think somebody needs to come and build like a really, really compelling product. And it's actually interesting to me that I haven't seen someone. I'm curious if you've seen any companies that you're talking to that have solved that problem, but I have not personally seen anyone solve that problem. And it seems like it's one of the most impactful things, especially in today's world where the model iteration cycle is so quick and there's always a new model coming out that's

• 22:47Speaker 01

  supposed to be better. And you just don't know if it's actually better. And it requires so much human time to figure out like, is this actually better for my use case?

• 22:55Jacob Effron?

  No, it's interesting. I mean, there's probably like, you know, 25, 30 really early stage companies going after that. But I don't know if anyone's quite cracked it yet in many ways because it's hard to automate a lot of that. And I think there's both obviously, you know, evals comparing models to models and then on your own use cases. And it was interesting, we had Linus from Notion on a few weeks ago. And what he was saying is like, it's so hard to build a product here because the biggest benefit of these eval tools is when you actually go see the places where it fails, and you just learn so much from going in the details of those that like trying to automate that away in any in any way is actually like gets rid of that that whole beneficial experience.

• 23:27Speaker 01

  Interesting. That's a that's a great perspective to have. That's, that's cool. I could imagine that being true because I feel like the edge cases like is where you're learning the most about how the models work and also like how they've changed and things like that. That's, that's a cool perspective.

• 23:40Jacob Effron?

  Well, I mean, speaking of of of models that we're trying to figure out where they where they fit in. You know, I'd be remiss not to ask you about Google Gemini, I guess, literally just dropped this morning. So I guess I know it's it's fresh. But curious for your take on kind of like where that will end up fitting into the ecosystem.

• 23:54Speaker 01

  Yeah, I'm excited for Google. I think like all the stories about how hard it is for them to like get product out the door for whatever reasons, if that if that's actually true or not, I think it. It takes a lot to ship something on the order of magnitude of of Google. And I was reading Jeff Dean's tweet this morning, and it seems like. They put an incredible amount of work and thought into getting this model out. So I'm really, I'm really excited. I think it's a the biggest question for me from like a consumer and developer side is like. Will people actually use their APIs? Will they use Bard? I don't know the answer to that question, but I think it's it's awesome that they're doing it. And. I'm personally happy because at the end of the day, I think we're all we're all Google customers as well. Like, I'm a I'm a Gmail user. I'm a Docs user. I'm a.

• 24:34Speaker 01

  I'm a Google search user. So I think it's it's cool that they're they're pushing on that innovation. And I'm hopeful for me as a customer that I get to to benefit from that stuff as well.

• 24:42Jacob Effron?

  I mean, one thing you brought up earlier about where you thought was going to kind of be most used next year was the assistance API, and I'd love to get into like GPTs and the assistance APIs. I think it's just it's fascinating. Maybe I'd actually love to start with just the journey over the last year. I mean, I feel like your first foray into this space was via plugins. It seems like maybe that didn't have quite the traction folks wanted. And so I'd love to hear just a little bit about maybe learnings from that experience and also how it kind of evolved into where you are today.

• 25:06Speaker 01

  Yeah, that's a great question. I think. Plugins had a really ambitious. Goal and mission. I think in a lot of ways, it was. It was kind of framed as a product release, and I think my my reframing of it in hindsight was more of of like a kind of like we did with ChatGPT originally as like a research preview. I think like. Just the very essence of the idea that you would like. Put this publicly hosted, accessible to the Internet. AI plugin dot manifest file, which kind of described what you wanted a model to do. With your open API spec, I think it was like. From a product strategy. You know, maybe not like the best implementation from a from a developer, from a consumer angle.

• 25:49Speaker 01

  But I think from like a research perspective, like it was, I think it was net positive that that's the direction that we went.

• 25:57Speaker 01

  It's interesting, there was a lot of limitations with plugins. I think mostly, honestly, like plugins could have been a more successful product service area. I think there's just so much resource constraint inside of OpenAI that like a lot of the folks who helped ship plugins, like as soon as that product was released, like ended up having to go help ship browsing and went go ship code interpreter and then the thousand other things that came after that. And it becomes really easy to to sort of lose sight of the some of the things that you released before. And then I also think there's plugins has all these like security, privacy questions. And the beautiful part is that most of those problems have now been solved with GPTs. Like,

• 26:41Speaker 01

  I think a lot of the problem areas that we had around like taking consequential actions and like needing to like prompt the user for consent and a bunch of the different off challenges that we had with plugins have now been solved with GPTs and it's a much better it's a much better interface because it's so many of the requests that developers had was like, I want to have browsing plus my plugin. That wasn't possible. I want to have like code interpreter interacting with my plugin, which wasn't possible. And now you can sort of check, check, check, add in whatever your custom action is, which is essentially a plugin. And it just works out of the box, which I think is is great. And the GPT store coming in the future is going to be like, again, there were so many discoverability challenges, as everybody pointed out with the

• 27:24Speaker 01

  plugin store with just basic search and a couple of categories that I think the GPT marketplace is really going to hit a home run on solving that problem. So it'll be awesome.

• 27:34Jacob Effron?

  I mean, I'm curious so far, like anything, you know, well, one, just like trends and things you've seen people building like in the in the time that it has been out, like what how are people kind of are using them?

• 27:44Speaker 01

  Yeah, I my instinct thus far is that most of the use cases are really around like sharing prompts. I think, to be honest with you, today, that's like a majority of it. I think it also goes back to the reality that there is a bunch of value still in prompt engineering. Like you can get it like I've seen some of the most popular GPTs and around coding and building websites and like you can get a Pretty far value add from just having a great set of instructions. I haven't seen, and maybe just because I haven't been looking hard enough, like a bunch of incredible examples of people using custom actions. I'm sure people are doing this, but I think they're still just like the everyday person can't, who doesn't know how to code. Like it's just a little bit of

• 28:28Speaker 01

  a frictionful experience to go in and add a custom action because you have to like put in the open API spec and understand some of those nuances. So I'll be interested to see if there's a world where there's, you know, you can add even more actions from, you know, a plugin catalog or something like that in the future. I think it'll be, it'll be interesting to see if we end up getting to that point.

• 28:47Jacob Effron?

  It's interesting. I feel like people have been talking about the death of prompt engineering for a while. Like, you know, yeah, it's useful now, but at some point it won't be. And it persists in being quite useful.

• 28:55Speaker 01

  I do think it will go, it will go away in some, like I think the fundamental nature of prompt engineering is communication. I think humans are, especially when we have to use our fingers for some reason, just like lazy communicators. And I think the model will, will hopefully evolve to do something like what Dolly is doing, Dolly three is doing today where you give it the prompt and then in chat GPT and the API today, you actually end up getting a revised prompt where we have this like layer of translation between your original request and what the model ends up seeing. And I hope that we end up with something like that, just because my instinct is you'll always need this like prompt, like more, you'll always need to communicate better. And I think

• 29:37Speaker 01

  that's the reframing of prompt engineering is like, if you just communicate what you actually want in a very concise way, it works. And the question is, can we reduce the friction for people so that you don't have to type out like all the verbose stuff and get the model to try to do all the things that you really want it to do?

• 29:55Jacob Effron?

  Well, I guess now you can speak it, you can draw it and, you know, put it in. There's all sorts of new ways to communicate. I guess, has there been a particular example you've seen of GPTs or using the assistance API where you're like, that's kind of how we were hoping people were going to use it, or anything come to mind that you thought was a particularly cool implementation?

• 30:11Speaker 01

  I haven't actually seen somebody build this yet, but it's something that I'm really excited for somebody to build. So maybe this is a slightly adjacent answer to the question, but I really want a text-first assistant experience. I think that's going to be like, you know, you can imagine the world where you use the assistance API and connect it with Twilio or one of the other text messaging providers and can bring sort of that, bring that assistant chat GPT experience to so many service areas that like weren't possible today. And the reality is like, you don't do all of your work in a web browser, you don't do all your work in chat GPT as a, as an API service area. And there's so many different areas where I just want AI to be able to assist me and not have to leave

• 30:56Speaker 01

  that experience. And I think assistance is a great way of doing this. And the fact that you can now like pass around thread IDs between some of these different objects, like I really, I haven't seen somebody build an assistant that I can text yet, but that is what I want. Like I, I have so many of my ideas when I'm talking to other people on, on via text message or something like that. And, or even via email, I think is another interesting one. I think this is why Grok and Twitter will end up being useful is because people are throwing around so many interesting ideas on Twitter all the time that I think having like an AI native experience that's like adjacent to that will be interesting and useful in some capacity.

• 31:35Jacob Effron?

  Super interesting. How do you think you'd engage with that? Like, you know, you know, if you were in kind of in email or in Twitter or in some sort of text experience, like how would the kind of AI assistant come in?

• 31:44Speaker 01

  Yeah, I want to be able to like text and be like, hey, like do X, Y, and Z, kind of like, you know, how you would imagine, like I've seen so many of these virtual, like AI executive assistant or personal assistant things. And the, the big challenge with all those is I need to, they're not like coming to my existing workflows. They're like, you need to now go to my aiassistant.com website and like import your Gmail and like do all Weird stuff that like that's like a new experience for me. It's really hard to get into that habit versus like I'm so used to just pulling out my phone and like texting someone and be like, Hey, I need help with X, Y, and Z thing. Like, can you help me? And I think that experience is just going to be so much better when you can like send a text message or send an email.

• 32:29Speaker 01

  And I think those those two particular service areas I'm intrigued by most because the number of you don't have to re-educate users on how to text or send an email. And I think you need to re-educate users today. Like, this has been one of the fundamental challenges of ChatGPT. And then the team has done a bunch of stuff to try to address this. But you show up on ChatGPT and you're like, I don't really know what I'm supposed to do next. And I think it's just hard to, you can imagine in a world where you're texting an assistant, you know, I loop in my grandma. It's a group chat with me, my grandma and my my AI assistant. And I'm like, hey, grandma, like, whatever your question is, like, go and ask it here. Like, you need help sending an email. You're trying to find an answer to something like, you can just do it right here. And you you like have that multiplayer

• 33:14Speaker 01

  experience. And there's like someone there to help you. And like that that experience is just not possible in ChatGPT today. That that experience is impossible on, you know, aiassistant.com with whatever the the the personal assistant experiences. So I'm really excited about that.

• 33:27Jacob Effron?

  Yeah, it's super interesting. And hearing you speak, I'm kind of struck by the fact that like, you know, at OpenAI, you have these two different, you know,

• 33:35Jacob Effron?

  approaches almost one being like ChatGPT is like this consumer front end of like, come here and you can like literally do everything. Like, you have so many things that you guys can do from ChatGPT. And then, on the other hand, as you say, there's so many things you do outside of of that world. And it's really convenient to have a ChatGPT like experience embedded in those in those other experiences. Like, how do you kind of, I mean, does that create a tension at some point? Or like, you know, or even as you think about five, 10 years, like, how much of what we do and how we engage with these AI assistants, do you think happens through like the OpenAI homepage or like, you know, ChatGPT page versus like just hidden behind in other applications?

• 34:10Speaker 01

  I think it's For OpenAI to have the order of magnitude of success relative to Google in the sense of like, people start their search experience on Google.com and that's like the only place that they do it, I think is a, is not a long shot, but I think it's just a higher hill to climb than being present where our, where our customers and users already are, like not, not everyone is always going to want to go to chat GPT, chat at OpenAI.com or use the iOS app. I think being accessible in other service areas, and I think this actually goes back to, this is essentially a Microsoft strategy with Copilot. They're building that Copilot experience, which uses GPT-4 and a bunch of other models, directly

• 34:52Speaker 01

  into the service areas that the Microsoft customer base is already using. And I think that, that strategy resonates with me. I think it makes a lot of sense. Of course, it's easy, easier for Microsoft to do because they, they have those means of distribution. I think it's different for us because we sort of have to build all of those means of distribution and connect in all those different places that don't exist for us today.

• 35:11Jacob Effron?

  Totally. Well, I really hope someone listening builds this all because I think like there's a lot of really cool, I love like linking up with Twilio too. I think that'd be super interesting. You know, on this GPT's topic, I was listening to another podcast you were on and, you know, you talked about how you guys have put some limitations around agents taking action and that, you know, I think you said literally that we may not be ready for what happens to the internet when, you know, there's these autonomous agents that are, that are all over the place. I'm curious like how you think about the timeline to have maybe agents with more wide-ranging capabilities. And then I love to dig into that more. Like, what do you think the consequences are for, for the internet as a whole?

• 35:43Speaker 01

  I think there's, I think there's a bunch of internet infrastructure work that actually needs to happen. Like for me today as a, as a developer, as somebody who I feel like I'm close to the pulse of what's happening, like

• 35:54Jacob Effron?

  I would say so.

• 35:55Speaker 01

  I, I don't know what the safeguards that have been built in to like help authenticate humans versus AI agents on the web. And I think there's, there's so, there's so many people on the internet that I think there's a lot of, like, it's going to take year, on the order of years for that to be possible. And I think my hope is that we end up Moving towards that process gradually enough that people have time to adapt. Because I think there is a world where, you know, I think we have a bunch of safety mitigations built into our vision models and stuff like that. But I think you could use models to get past all of those things that you would normally use to prove that you're a human on sites. And there's a lot of things, if you think about it, that you don't want to show AI agents

• 36:40Speaker 01

  certain information. And as a developer of a website or as somebody who creates something, you probably want like a different door for AI agents. It's also just like a really, in a lot of ways, a really inefficient way to use these models if they have to like jump through all the same like hoops, hoops that like we as humans are like well adjusted to. It's kind of, it would be kind of odd if that ends up being the default sort of experience for agents to go and retrieve information from places online. So I, it feels like it has to be on the order of years just because of some of these infrastructure limitations. And maybe the only way that ends up accelerating is if companies like, you know, the sort of guardians of the internet in some way, like Apple and Google and all these other

• 37:23Speaker 01

  like really large corporations end up getting together and building some consortium about like how these tools interact. And I could imagine that being, that being the case. And I actually think I was, I was having a conversation with Parag from the former CEO of Twitter, and he was saying that there actually is something that these companies are working on. And I'm pretty sure it's like an open standard or something and might already be out there. So I need to do some research and figure out what that is. But I'm curious if that ends up being the solution to part of this problem.

• 37:50Jacob Effron?

  I love that idea of having kind of two different paths for like a human user versus an agent user. And it also sounds like basically you guys probably could push the capabilities of these agents, but are, you know, you're waiting till that's kind of in place to make sure that they end up getting used responsibly and there's kind of a clear, a clear infrastructure that allows them to be used responsibly.

• 38:07Speaker 01

  I think the responsibility angle is one of it. I also think like being, having humility on it, I think it's like a hard, it's hard to do it well, just because there's like anytime you go out and do and request anything from the internet, even as me as a human using the internet, like things don't always work as I would expect. Like more often than not, we have this experience as users of the internet. So I think it's just. Like hard to build a really reliable and robust experience. I think I think that's also an angle of it. I don't think there's the the perfect product experience that we're just sitting on waiting.

• 38:38Jacob Effron?

  I wasn't sure, so you know, I had to ask.

• 38:39Speaker 01

  Yeah, I think that I think there's some I think there's some real engineering challenges to solve, some real product experience challenges to solve in addition to all of the safety and and the safety work that needs to happen.

• 38:49Jacob Effron?

  Totally. I feel like we went through like an entire hype cycle with agents in the last like six months.

• 38:53Speaker 01

  It's it is. It's crazy. I wonder if anyone actually got any value from from like some of the I I do think it's positive that it like forced people to really start thinking about these problems. I think had Auto GPT and Baby AGI and a bunch of those other agent products not gone sort of like commercial like a consumer viral, it wouldn't have forced a lot of people to try to think about this problem and like figure out and and in a lot of ways probably actually slow down some of the push that they were doing just because of like the potential for for misuse with these with the agents that are out there.

• 39:29Jacob Effron?

  Yeah, totally. You know, makes that makes sense. I mean, obviously, I feel like with Demo Day, you know, between GPTV, DALL-E 3, text to speech, like, there's so much stuff that people are now playing around with. What I I'm curious like if there are any specific examples you've seen where you're like, yeah, that's exactly how we were hoping folks were going to use that.

• 39:47Speaker 01

  A great example, I think, is have you seen TL Draw that went viral in the last week or so?

• 39:54Jacob Effron?

  Yeah, on Twitter.

• 39:55Speaker 01

  Yeah, yeah. So so the the interface for folks who haven't seen it is essentially you load into TL Draw, it's a it's a basic blank slate and you can draw stuff and it essentially takes what you drew and converts it automatically into like a functional app. And I think it's it's like end to end using a bunch of the different pieces that we've built and like orchestrated together over the last year. And if you go back to the original GPT-4 demo that that Greg Brockman gave.

• 40:20Jacob Effron?

  The famous drawing.

• 40:21Speaker 01

  In March, I was sitting right there as he was as he was giving that demo, watching him give it. And it's just so mind blowing to see him take that example, all of the work that had to happen to make vision ready to to go out into the world and be used safely and in robot. And reliably and then now to see like a developer and probably in the order of a couple of days or a few hours was able to like take that build it into a real application hundreds of thousands or millions of people were able to use it. I think that's like the the perfect success story in my mind of like making this technology accessible to as many people as possible and like actually building real things. And again, I think TL draw is like intentionally supposed to be like a little bit gimmicky, but there's a lot of use cases now where I could see like building off of that

• 41:06Speaker 01

  that are actually really, really, really useful to people.

• 41:09Jacob Effron?

  Yeah. What's one that that you that you go build?

• 41:12Speaker 01

  I not that I would go build this, but I think a similar one to this. Have you seen any of those like people sketching art and then having the art go and like use a generative image model to actually go and create it. I think like just that lens for me, I feel like I'm somebody who's always been so like creatively limited by like, I'm just like, I feel like I can do things, but I don't know like what it's going to look like from like an art perspective. And I think it's so cool to be able to be like, here is what I could create if I, you know, kept working on this like crappy

• 41:44Jacob Effron?

  Start with a little bit and then see different places.

• 41:46Speaker 01

  Yeah, I feel like that it free in a lot of ways. I feel like Dolly has pushed me to like free my mind of what's possible. Like I feel like I can imagine more because I'm inspired by this AI art that's been created. And I think that's like, that feels super empowering to me. I like it. And I think there's hopefully a bunch of other cool use cases like that.

• 42:05Jacob Effron?

  That's really cool. I don't know if there's any tool that can help my poor artistic skills, but that's I love the idea of being able to explore different possibilities.

• 42:10Speaker 01

  It can at least inspire you. I'm like, there's, there's a lot of work you need to do.

• 42:14Jacob Effron?

  A gap between the inspiration and execution.

• 42:15Speaker 01

  Yeah.

• 42:17Jacob Effron?

  That's a, that's awesome. I guess you mentioned like we're still so early days of GPT four V. It sounds or GPTV. It sounds like you were saying that, you know, there's still a lot of things that need to get solved, like.

• 42:29Jacob Effron?

  You know, it almost seems like the next version, you feel like there'll be just like a ton more of a broad applicability. Like, I'd love to hear just a bit more about that, like what you've noticed about what it can do well, what it can't do well.

• 42:38Speaker 01

  Yeah, I think the biggest thing is like people, like you could imagine a great use case for, I'm a, I'm a huge Canva user. I've always used Canva to make things because I'm, again, going back to my lack of artistic skills. I've made like hundreds and hundreds of things on Canva before because they give you incredible templates. And I think like, that's a great example for Canva where the, the vision model today cannot really do a perfect job of understanding all of those spatial relationships. And I think there's a bunch more examples like that. Like if you try to do OCR with like, taking pictures of, of spreadsheets or receipts or whatever, it, it gets a lot of the stuff right most of the time, but I think it, it misses like some of the structure and it'll tell you things are in the wrong place.

• 43:23Speaker 01

  And you can imagine for like the Canva example, like that it has to work in, in the spatial relationship between these things. And, and what I really want to do is be able to go into Canva and be like, hey, you know, reformat this to like fit something and like see the like movable objects, essentially like the crossover of Canva and Dolly and see the movable title and everything like that. And now I can just click and drag and edit all that stuff. Cause if you, if you look at Dolly today, like the text output is, is not always perfect. And it, you know, oftentimes it's like a really easily fixable thing. It's like a couple of characters that I need to fix. So I'm hopeful that the model ends up being able to enable experiences like that because it's, otherwise it's, you know, the, for Dolly, for example, like,

• 44:07Speaker 01

  I don't even know what the solution would really be to like fix some of those text things. Like you really can't do it. You have to like really have a lot of skills to like go into Photoshop and like edit something like that. So.

• 44:17Jacob Effron?

  Totally. I feel like how, yeah, how you make the outputs of Dolly like editable is, is a fascinating and very important question.

• 44:22Speaker 01

  Yeah. I mean, I don't know if we'll, we'll solve it or Canva will solve it, but I think there's a lot of people who have a vested interest in making that work well from a, like a Photoshop might solve it. Who knows? We'll see.

• 44:33Jacob Effron?

  And then, I mean, obviously, you spent all day with developers and enterprises. And I feel like you guys, over the last year, have just been systematically, you know, whatever objections one might have had or reasons people weren't building, you're just like shooting them down one by one. And, you know, even I noticed in Dev Day, you guys, you know, provided protection against lawsuits and their copyright, which is something that comes up a lot on the enterprise side. You know, going into 2024, if you were to kind of stack rank the reasons that like enterprises and developers aren't building with, you know, with LLMs in general, what are like the main objections you see or the main blocks you see for more folks using this?

• 45:05Speaker 01

  Yeah, I think the largest one is robustness and reliability. And I think today, you actually likely need to like build some of these like more intricate orchestration frameworks and like use third-party tools like Guardrails AI and some of these like a bunch of, there's tons of like compliance LLM companies that are out there. And I think like the reality is you need to use those today. Like if you really want to go into production and feel the confidence that, you know, something bad's not going to happen, I think you have to do that. And I'm hopeful that like from a pure platform perspective, we'll continue to like upstream solve a bunch of those problems.

• 45:46Speaker 01

  And then it's just a matter of like, my fear is because people are going to be so accustomed to these problems, there even when we end up solving them, people are still going to be like, oh.

• 45:56Jacob Effron?

  I want the extra Guardrails check.

• 45:57Speaker 01

  Yeah, I want my, I want to build all these extra things. So I think I'm hopeful we'll be able to solve that and communicate that to developers. I think the other one is just like, and this can, this seems like it's always going to maybe be a problem with how the AI ecosystem is going, but like latency is huge. Like there's a lot of use cases where people don't, can't have the model sit there and or can't have their user sit there for like seven seconds and wait for a response. And there's just a lot of work that we need to continue to do internally from a model development perspective, from a, from an inference perspective, to make that work better. And I hear customers when they say like, this isn't, you know, this isn't fast enough for my use case. And I'm hopeful that by the end of 2024, nobody will be saying that anymore because the models will be super, super fast and we'll have enough GPUs finally.

• 46:43Speaker 01

  Video keeps keeps doing their job. So hopefully that will will not be an objection anymore.

• 46:49Jacob Effron?

  Thought there's this whole class of use cases where you like want to stay in your creative flow and that like extra, you know, second of of generation like takes you out of that.

• 46:56Speaker 01

  It really does. I think it's it. In some ways, L L M's are are like the I heard someone say this, that it's like the a clone of human thought. And in many ways, it doesn't move at the speed of thought. And I think that's that can be so jarring and and a lot of a lot of experiences. In some ways, though, I actually I and this this takes me to the a broad topic of like the different user interfaces and and U X experiences. I think in a lot of ways, like not having an instant response will be useful in a bunch of a bunch of circumstances. And I'm curious to see who ends up like coming up with a bunch of thoughtful design patterns around this because I I feel like we're all going to be trained even more so than just like going on Google

• 47:37Speaker 01

  and finding an answer something. It's going to be like you can really get the answer to what your question is like even faster. And when you don't have that, our attention spans like go down so significantly. So it it's fascinating.

• 47:48Jacob Effron?

  Totally. Yeah, I think with the you know, the TikTok generations only only so.

• 47:52Speaker 01

  I didn't want to pick on TikTok, but.

• 47:55Jacob Effron?

  No, I think the the U X pattern of streaming the answer, you know, one word at a time seems to be a clever way to get around a lot of that stuff.

• 48:01Speaker 01

  Very true.

• 48:02Jacob Effron?

  Well, we always like to kind of end our interviews with a quick fire where we just get your thoughts on a on a few things. So maybe first, what's overhyped and underhyped in A I today?

• 48:12Speaker 01

  Overhyped, prompt engineering, underhyped, observability. I think more people like if you're actually building something, you should be using, you should find some way to do observability, whether you build it yourself or use some off the shell product. I think it's hard to use these models and not understand like the everything that's happening.

• 48:33Jacob Effron?

  I guess I'm curious, like throughout the last year at Open A I, maybe an example of one thing you thought would work that you guys released and it just didn't.

• 48:40Speaker 01

  Hmm.

• 48:41Jacob Effron?

  Because it seems like it's just been one nonstop string of wins.

• 48:45Speaker 01

  Yeah, I mean, I think maybe plugins is the best example of this. Like, I think plugins was was close to what we wanted, but it was just missing like a few core things. I think the nice part was we were able to turn it into a win with with GPTs, I feel like. But I think that's probably the best example of when plugins first came out. I really thought that like that was kind of going to be the solution that people wanted. Like, it felt like that initially because people were so excited and like, this is the next app store, et cetera, et cetera. And it didn't end up being the case. And I think a lot of it comes back to like just the experience of using and building plugins like wasn't there yet. We just needed more time to continue to refine that.

• 49:26Jacob Effron?

  Yeah, and I guess on the counter side, is there something that you guys didn't think would be a big thing? I know ChatGPT is the canonical example, but maybe something else in the past year that maybe was a footnote in release and ended up being this like, you know, much bigger, worked much better than you would have thought.

• 49:39Speaker 01

  I think function calling function calling is a great example of like, I wouldn't have intuitively thought that this is something that you would need to build or want to build. But it's actually is like what enables like most really interesting production use cases is people doing function calling. So I think that's been my biggest surprise. I just wouldn't have thought that that's like something that we would need to build for people.

• 50:02Jacob Effron?

  Totally. That makes makes sense. And I guess like, you know, I'd be curious, even just to hear about the evolution of like, you know, I think when you started, it was like just you doing all things DevRel at OpenAI. Like, how is that, you know, how is like the team? And, you know, you've obviously seen this crazy journey of the past year. What is like, how has the team evolution been like at OpenAI in terms of, I mean, obviously you've grown a lot in people, but like, what's that look like today?

• 50:25Speaker 01

  It's fascinating. I think this is my my first experience seeing something like this play out of of like how quickly the company has grown, how much. How much work there is to do. And I think I was, I don't know if I was making this comment to you or I was making this comment to someone else earlier today, but it seems like even with more people coming on board, there's still an endless amount of work. Like, there's so much work that needs to happen. And I'm, I'm just, I'm, I'm so happy that we've gotten so many incredible new people onto the team. I now sit in the developer experience team. We have a whole API experience team as well. We have, like, I think when I joined originally, there was like six or seven people who are doing all of the engineering work for the entire API. And now we have like, we have so many folks from

• 51:10Speaker 01

  capabilities to fine tuning to a bunch of our enterprise offerings. So it's just awesome because

• 51:17Speaker 01

  people want so much from us. And like, we just weren't able to do what people like, a lot of the basic things that people were asking. And it feels great to know that we now have an incredible team that is pushing all the time to make this stuff happen. So it's, it's interesting. But it is also in some sense, like a jarring experience to go through the, like, I can't imagine there's a lot of companies out there that have like gone through this crazy growth trajectory and just to like, see how that plays out internally. And I think we've, in a lot of ways, done a good job of, of navigating this. And we were talking before off camera about the whole SAM and board situation. I think the very positive thing of that situation, the positive outcome is like,

• 52:02Speaker 01

  really bringing together the team. And I think it would have been easy to over time. And I didn't actually feel this way, but I think it's possible to assume that like, over time, just because of how much the company has grown, like, people just kind of go off on their own, you know, their own little areas. And like, there's nothing really bringing them back to the core mission. I think, like, that was kind of a nice shake and wake up that, like, we are really all in this mission together. And like, we have to stay focused together. And if you have, if you're one of those people who joined recently and kind of, you were like, holy, holy crap, like, I just joined in this, everything kind of flashes before your eyes. And now you sort of feel like you, you know, bonded by that trauma in a lot of ways. So I do think that's one of the positive outcomes.

• 52:46Jacob Effron?

  I mean, according to display at the heart, the open eyes, and then that is people, the letter, I mean, it's really impressive.

• 52:51Speaker 01

  Yeah. It was a, it was an experience.

• 52:54Jacob Effron?

  I can, I can imagine. I'm excited for the TV show they eventually make about, uh,

• 52:58Speaker 01

  A good documentary.

• 52:59Jacob Effron?

  Who's gonna play you?

• 53:00Speaker 01

  I, I won't be played, thankfully. I hope that I'm not played.

• 53:05Speaker 01

  My success outcome would be if no one is playing me.

• 53:07Jacob Effron?

  Very fair. Um, I mean, I feel like you, you know, you obviously spend so much of your day talking to developers, enterprises. I feel like there's so many folks out there that like, maybe they use chat GPT. They're like, I'd call AI curious. They know this is like the next big wave. But they haven't like fully dove in yet. And I feel maybe feel a little overwhelmed with just how fast the space is moving. Like, you must have these conversations. What do you, what advice do you give to, to those kinds of folks and how to get started, where to get started?

• 53:31Speaker 01

  I think it's, I think it's tough because I, I actually imagine that this is like a vast majority of people. Like I do think a lot of people have heard about chat GPT and we, we released some of the numbers at developer day, but we have a hundred million weekly active users on chat GPT. That's such a small number of, of people. If you actually zoom out a little bit and there's, at the end of the day, this technology is going to so profoundly impact the entire world. Um, I imagine most people, to be honest with you, listening to this podcast are probably folks who have tried chat GPT before, but I think for, for people who have tried chat GPT.

• 54:05Jacob Effron?

  Yeah, I think there's still, you know, a set of people who have, but maybe haven't necessarily like figured out, well, okay, how do we put that into our enterprise or?

• 54:12Speaker 01

  Yeah, it's a great question. I would also say like, go and take this and like, talk to people who you know, and like, be the, be the ambassador for, for AI in some sense would be my pitch. But I think some of the, the easiest ways is like, literally write down the problems that you have. Like if you can do an audit of like, here are the five things that I spend the most time doing in my job or my daily life that like I hate doing and don't really enjoy and wanna get rid of in some sense. Or here are the five things that like, I'm deeply passionate about and I really wish that I was better at. And the, the art example is, is perhaps a good one. And me writing code is another great example. Like I, I wanna be the best engineer that I could possibly be. And these tools really, really help amplify that. I think. If you're somebody who's a developer and you're not like, I actually meet people today

• 54:55Speaker 01

  who are developers who aren't using chat GPT and they aren't using GitHub Copilot. And it blows my mind. I think every developer should be using these tools like it's really, it's table stakes. Like, I think you could put somebody head to head who's an average developer versus like some of the best people in the world. And like, the average developers are going to win with AI. And that means like, you have to use this technology. It just makes, to me, it feels like I have the freedom to to go and build anything that I can come up with in my head. And I didn't used to feel that way two years ago. And I think it's just, it's incredible to think where the models are today. And you fast forward two years and like, you're really going to be able to, it'll be interesting for the venture capital space. Like, it'll be really, you really will be able to build whatever the

• 55:39Speaker 01

  product or services that you want, because these models are going to help you so much. And I think, yeah, the world is just not ready for that yet. But I think chat GPT was the most searched Wikipedia page. So I think people are looking, which is, yeah, which is great. So I think that's, that's an important first step, figuring out the problems you have, figuring out what it is that you want to be better at, and finding a way to integrate AI into like your, your daily workflows. I think that it has to be a habit that you like, go to these tools first, whether it's chat GPT or something else, so that you can really start to see around the corner of how this technology is going to impact your, your life and career and all those things.

• 56:16Jacob Effron?

  I almost wish that like, you know, people would document like, here's some of the best writers that use, you know, AI tools and how they use those, or the best artists or, you know, the best coders that use, that have found ways to use this stuff. And it feels like such like tribal knowledge now, where like, you know, a friend will show you or something. But I think, you know, finding ways to spread that more ubiquitous would be really cool.

• 56:33Speaker 01

  This sounds like a great newsletter that you should start. Go find, go find the most incredible people in the world who are using these tools and share those stories. I feel like that would honestly be super useful. Because in a lot of sense, like, I hear people on like not know like I think this goes back to that that core UX challenge of like you show up in chat GPT and like I don't really know what I want to do with this. I meet people all the time who and I live in Chicago. So I'm outside of the like bubble and excitement about AI. And most people are like, I don't know how this actually helps me in my everyday life. But like, I do know what my problems are. And I feel like there needs to be, whether it's something that's built in a chat GPT or something else that like translation mechanism from here are the problems I actually have to, how do I go in and solve

• 57:18Speaker 01

  these things in my real life? And my bet is actually that it's going to be, you know, companies like Apple in a lot of ways who have the user experience of a billion people around the world or whatever it is already locked into their ecosystem and they can sort of start to show people what's possible. And you have like a little, you know, I think people are already sort of familiar with this with Siri, but you know, once Siri becomes actually useful and good, I think that'll that'll pique a lot more people's interest to like, what can I do with this technology? So I think they and a lot of other companies like them have an important role to play, which is why I go back to like, I'm excited about Gemini. I think it's going to show a lot of consumers around the world what's possible with this technology and they're going to go out and explore what are the tools that I can use to like best leverage this technology.

• 58:01Jacob Effron?

  Well, I think that's an awesome note to end on. I guess the last thing I will leave you the last word of of just where can folks go to learn more about all the stuff you're doing in OpenAI about you. This is your place to show point, point folks to wherever you'd like.

• 58:13Speaker 01

  Platform. Openai.com. You can see all the cool stuff that that we're working on from an API side. And if folks have cool GPTs that they made, send them to me on Twitter. I'm very curious to see what what folks are working on.

• 58:25Jacob Effron?

  Awesome episode with Logan. What were some of the things that stood out to you? I think it's just, I mean, I'm always just struck by just the breadth of what OpenAI offers. And, you know, I think his You know, and it just seems like it's going to continue that way. You always wonder like are there parts of the offering they might not do as much on. I think what I found really interesting is when we were talking about open source models and he was saying, look, a lot of the reason people like open source models today is that you, you know, you, there's more control you have with fine tuning. You can do reinforcement learning on them. And he kind of hinted like, but you know, later we'll probably allow reinforcement learning on our own models. And, uh, you know, you've seen it even with, obviously, one of the promises of open source models is it'll be cheaper and faster, but like certainly, you know, with the price cuts they've had at GPT 3.5 turbo, like a lot of that's, you

• 59:08Jacob Effron?

  know, it is just the breadth of what they have is, is really compelling. And I, I do kind of wonder, uh, how long they just keep controlling like every part of this ecosystem.

• 59:18Speaker 02

  Totally. Yeah, it was, the breadth also stood out to me and it really seems like they want to be this AWS of AI where every single thing a developer has, they're going to offer some version of it. They're going to have fine tuning. They're going to have, um, RLHF, they're going to have models, they're going to have all the different steps that you would need. And, uh, I just, I thought it was interesting just how big or how comprehensive they want to make the platform. And then I think there's an interesting follow up question onto that, which is in which one of those verticals are there opportunities for startups to build something specialized and maybe really good in the same way that we saw, you know, Snowflake go and own data warehousing, um, from, from AWS and others.

• 1:00:01Jacob Effron?

  I'm glad we got our obligatory Snowflake mention in.

• 1:00:02Speaker 02

  Got to get Snowflake in. It didn't start snowing though. Um, but, uh, but yeah, I don't know. I think about like which ones of those verticals are potentially interesting ones for standalone startup tools. Um, but yeah.

• 1:00:16Jacob Effron?

  I mean, I think he even mentioned like observability and monitoring is like something that they're not going to do. And then, you know, they're obviously looking at other tools. Um, and then, yeah, you know, it's interesting. The other thing that I really thought was fascinating was his, you know, the idea of how people are going to build on top of this assistance product. Um, and, you know, his, his take that essentially a lot of people are going to want to engage with these models and these kind of experiences in applications that are already in. And so that like, you know, you can kind of see, it's funny. I remember when we talked to him, what nine months ago, and he was like, embeddings guys, like embeddings are the future. And like, lo and behold, everyone's doing like their rag pipelines and whatnot. Uh, I feel like he's like, look, like assistance kind of in people's products is going to be this game changing experience. And I, it's always fun talking to him because I feel like you're kind of feeling, you're seeing what's, what will just be very popular in six, nine months.

• 1:01:00Speaker 02

  Totally. Yeah. And I think it's interesting. From a business standpoint, it would make sense that they want you to kind of continue to leverage your own data because then I think the more that you're using your own data and that's in the OpenAI ecosystem, this solution just gets stickier and stickier. And so I think with embeddings and some of these, the models you were talking about, like I think that just their gravity starts to grow.

• 1:01:22Jacob Effron?

  Totally.

• 1:01:23Speaker 02

  Yeah.

• 1:01:24Jacob Effron?

  But they are in an interesting position, and I think I even asked him this straight up. It's like, on the one hand, they have a consumer product, ChatGPT, which like, you know, in an ideal world, it's like your homepage to every AI experience you want and like every, you know, your assistant for everything. And then the other hand, you have this assistance product that's kind of enabling you to live in a host of other products. And, you know, I'm not sure they're immediately in tension, but you see over time, like they're, you know, the product teams that are working on each of these may have, uh, may have different priorities.

• 1:01:51Speaker 02

  Totally. Yeah. Well, I think this, this kind of goes back to the strategy of do it all. They're gonna, they're gonna have an API and they're gonna have the consumer phasing app and they're gonna have, and all of this is gonna be fed into, uh, you know, some massive AI model, uh, which I think it'll be fun to watch evolve. He seemed very excited for model eval startups and evaluation, which we've seen a ton of those. And it seems like it's been a really hot space for new startups just starting this year. And I wonder if that will be something where we start to see some of those scale next year as people get farther and more mature and they're building on top of Mistral and OpenAI and those sorts of things. So, um, I don't know, maybe model eval, something to watch for 2024.

• 1:02:36Jacob Effron?

  Anything else in the broader AI world that's caught your eyes these days?

• 1:02:39Speaker 02

  Uh, I thought how Mistral dropped their model was super cool. How they had, they first just released the, the model weights and that was on Friday, went viral on Hacker News. Then they did the official announcement the Monday after the weekend that went viral on Hacker News again. And so I thought that was really cool developer marketing where they got to kind of double dip on the announcement. And it also just seems like a great model that is both, has amazing performance and, um, and people are raving about. So I thought that was cool.

• 1:03:09Jacob Effron?

  Yeah, I mean, I was at NERPS' big AI conference in early December, and it's funny. You know, I think during the conference last year, ChatGPT was released, and so that was the talk of town. And then this year, all everyone was talking about was Mistral, and they were kind of, the team was there and had their cool gear on and. You know, it's cool to see just the extent to which those, the quality of those models and the general open source ethos has taken the, uh, the world by storm.

• 1:03:31Speaker 02

  Did you go to the conference?

• 1:03:32Jacob Effron?

  I did, yeah.

• 1:03:33Speaker 02

  Oh, nice. How was it?

• 1:03:34Jacob Effron?

  It was, it was really cool.

• 1:03:35Speaker 02

  It was cool.

• 1:03:36Jacob Effron?

  There's a lot of, a lot of good energy.

• 1:03:37Speaker 02

  Yeah, I've heard good things. I've never been, but um, yeah, it seems like the de facto AI conference these days.

• 1:03:44Jacob Effron?

  Lot more VCs these days.

• 1:03:45Speaker 02

  Lot more.

• 1:03:45Jacob Effron?

  I mean, I say that as part of the problem, so.

• 1:03:47Speaker 02

  Yeah, that's, yeah, so you're a part of the problem.

• 1:03:49Jacob Effron?

  Always.

• 1:03:50Speaker 02

  Yeah.

• 1:03:51Jacob Effron?

  To the extent our listeners haven't checked it out to you, one AI tool I think is just really cool is, I don't know if you, have you played around with Suno at all? Like the, the latest release they have on the music side?

• 1:04:00Speaker 02

  Um, I have not played around with it, but fun fact, it is powered by Modal.

• 1:04:06Speaker 02

  So, you know, shameless plug, uh, but, uh, yeah, I, I've heard.

• 1:04:10Jacob Effron?

  Everyone's gonna think I just like set you up for a way up there.

• 1:04:14Speaker 02

  I haven't played with it yet, though.

• 1:04:15Jacob Effron?

  It's really, so, uh, it's really cool. We might have to start using it for our theme music, like literally you can just prompt with like, you know, write a song about how great a podcast host Pat is as a country song or something.

• 1:04:26Speaker 02

  That sounds like a hit.

• 1:04:27Jacob Effron?

  It generates, it would be a banger, uh, and it like generates the lyrics for the song and then like sounds like your average country song. I was, uh, it's funny, I, I like nerd out on this stuff and I show my wife most of it and she's like 90% of it is like, what is this? But this one was a huge hit last night. We were playing around with it for a while.

• 1:04:43Speaker 02

  So it passes Julie's bar for.

• 1:04:44Jacob Effron?

  Yeah, which is way higher than my bar. Obviously, she'd be a good investor.

• 1:04:47Speaker 02

  Yeah, that makes sense. Um, cool. Yeah, no, I, I definitely would love to play with it. I think the generative music is so cool. Like it is really, really fun when it works. So, yeah, what do you think of Gemini?

• 1:04:56Jacob Effron?

  Well, it's funny, like I, you know, uh, the day we had the interview with Logan, Gemini came out in the morning and I was like, I wonder if he's gonna say anything or whatnot. And I think, you know, there was like an entire hype cycle around Gemini in between that interview and, and our release here where first, you know, this demo came out and it looked amazing. And then people were like, wait, is this real or is this kind of like a bunch of shots put together and, and many attempts? Uh, and then obviously, you know, I think folks are just kind of getting access to the model. But I, I really liked Logan's take on it, which was, you know, uh, obviously it's great to have other things out there. It feels like. It feels like one of the big themes for 24, now to get a second prediction, is multimodal. It feels like that, and that obviously is a huge part of the Gemini model and GPTV and all this other stuff.

• 1:05:38Jacob Effron?

  And I think we're gonna see a lot of, you know, we're just starting to experiment with what happens when you input images or videos into these models and output some combination of text, image and video. And just the different things you'll be able to do, I think are really, really cool.

• 1:05:50Speaker 02

  The multimodal demos are really, really impressive. And I think that was the most impressive thing about Gemini, where you could, you know, the guy was drawing the duck and then I could see the duck being drawn in real time. And I do think that's where the future is going, and then there's so many cool applications. That's bringing it back to Iron Man, that's where you get to Iron Man vision, so.

• 1:06:12Jacob Effron?

  I'm shocked you went to Iron Man and not to Plugging Live Kit, but I guess that is a...

• 1:06:16Speaker 02

  Yeah. There we go, too.