Emergency Pod: OpenAI's new Functions API, 75% Price Drop, 4x Context Length (w/ Alex Volkov, Simon Willison, Riley Goodside, Joshua Lochner, Stefania Druga, Eric Elliott, Mayo Oshin et al)

Full Transcript and show notes: https://www.latent.space/p/function-agents?sd=pf Timestamps: [00:00:00] Intro [00:01:47] Recapping June 2023 Updates [00:06:24] Known Issues with Long Context [00:08:00] New Functions API [00:10:45] Riley Goodside [00:12:28] Simon Willison [00:14:30] Eric Elliott [00:16:05] Functions API and Agents [00:18:25] Functions API vs Google Vertex JSON [00:21:32] From English back to Code [00:26:14] Embedding Price Drop and Pinecone Perspective [00:30:39] Xenova and Huggingface Perspective [00:34:23] Function Selection [00:39:58] Designing Code Agents with Function API [00:42:16] Models as Routers [00:46:48] Prompt Engineering replaced by Finetuning [00:52:15] The 2 Code x LLM Paradigms [00:56:30] Smol Models for the future [00:58:54] The Evolution of the GPT API [01:03:27] Functions API Security vs Prompt Injection [01:16:18] GPT Model Upgrades [01:17:36] JSONformer [01:21:03] Closing Comments - What We Want Next This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

• 0:05Speaker 00

  Hello, everyone. This is swyx back again with another emergency pod. The last time we did this was in March when OpenAI released ChatGPT plugins. And the new Functions API today is effectively the ChatGPT plugins API now available to all developers. And with a whole bunch of other news, 75% price drops on embeddings, four times the context length, and a lot more other updates. So what we do in these situations, when there's breaking news and it's very developer focused, is we convene all the friends of the pod with Simon Willison, Riley Goodside, with people from Microsoft Research, Hugging Face, and Pinecone. And more that I don't even know where they work at. I think some of them used to also contribute to LangChain.

• 0:50Speaker 00

  But anyway, we just had all our developer friends, we had 1,400 people tune in yesterday just to talk about what they think and what they want to build with the new Functions API. We aim for latent space, we're very much targeting for latent space to be the first place that people hear about developer-relevant news and to go deep on technical details, to think about what they can build with them, and to hear rumors and news about anything and everything that they can build with. So enjoy. Unfortunately, Alessio was on vacation, so he couldn't help co-host. But fortunately, friend of the pod, Alex, joined in. And that's going to be the first voice that you hear. Alex has been doing a fantastic job running Twitter Spaces every Thursday.

• 1:34Speaker 00

  If you want to talk about just general AI stuff, as well as just follow him for his recaps of really great news. So without any further ado, here is our discussion on OpenAI's Functions API and the rest of the June updates.

• 1:47Speaker 01

  For those of you who work with OpenAI 3.5 and 4, etc., feel free to raise your hand and come up, ask questions, as we explore this together. And I'll just say thanks to a few folks who joined me on stage, Aneesham and John. And we've been doing some of these updates every Thursday, but this one is an emergency session, so we'll see. Maybe Thursday we'll cover some more. So OpenAI today released. An update, the June update with a bunch of stuff. And we'll start with the with the simple ones, but we're here to discuss kind of the developer things. We'll start with the pricing updates. So seventy five percent reduction in embedding price. This follows a ninety percent reduction of embedding costs back in. I want to say November, December.

• 2:32Speaker 01

  Anybody remember that? Maybe Roy in the audience, feel free to come up as well. And we've seen kind of this reduction in cost on a on a trajectory to basically. You know, being able to embed the whole internet. There's actually, I want to find this, there's actually a tweet by Boris from OpenAI. That talks about approximately it gonna cost you fifty million dollars to embed the whole internet. Like all of the text on the internet pretty much. And Logan followed up today and said, you know, after the updates of the pricing today, it's around twelve point five million versus fifty million before. Just to give like a huge scope of numbers in terms of like. How how fast this this type of tech advances? And we have Zenova in the audience, Zenova.

• 3:19Speaker 01

  Feel free when you finish eating, but basically. There's now a debate whether or not embedding on client side is worth, given that it's like so, so, so free. Or almost like very, very cheap embed stuff. Obviously, it's an API and there's concerns about using private data. But embedding seventy five percent cheaper. Imagine that you run embedding some production and John, let me know if you do, Houston. Today, if you switch to this API, you basically just received a seventy five percent like price cut. For for the use of a bunch of a bunch of stuff. And by the way, Riley from DEXA here, once he joins, they also they do a bunch of embeddings for pretty much every podcast out there. So, you know. Just in one day, you you can receive like a significant, significant decrease in costs.

• 4:03Speaker 01

  So embedding price goes down significantly, very exciting. In addition to this, another price cut is the twenty five percent for GPT three point five.

• 4:11Speaker 02

  Yeah, I just want to say, so we use we use a lot of embeddings, so we're really happy to see that. But the thing I'm most excited about pricing wise today is the new sixteen K GPT three point five model. Because I I believe it's about one hundred and fifty percent the price of GPT three point five turbo. Previous API pricing, or what it what it is now rather. And this is really significant for us because. GPT three point five has never had that many tokens that you can max out in your context window for. So when we build our input prompts for our co-pilots, it's usually using most of the four K window just for the input. And so this is a massive, massive increase for us in terms of the economics of getting a big output compared to like GPT four thirty two K or GPT four eight K, right?

• 4:56Speaker 02

  It's a really, really big bonus for us and it's barely more expensive than three point five regular. So I think that's going to be really massive. Super excited to see what people build with the three point five sixteen K model. Because honestly, like, we think GPT four is just quite expensive at this point. Like, we don't use it very much in production.

• 5:14Speaker 01

  Which we we we talked last week in our spaces, the roadmap for OpenAI, Sam Sam often talk about this publicly, is to decrease the price and also increase the speed. But yeah, let me welcome a few more folks here. Welcome, Sean, who prompted this emergency, emergency spaces. How are you doing? How are you doing?

• 5:32Speaker 00

  I'm doing okay. I'm moderately excited about it. You know, I think you and I had this chat where we were trying to scope out what the impact is. And I I think this is an incremental update. So good news on on many, many fronts. They're they're shipping with a really good pace, but not a game changer in my mind, just incremental updates.

• 5:52Speaker 01

  So we'll definitely get into the function thing because I wanna I'll take a big bite to understand like what this means. But I think we covered pretty much all the other updates by now, right? So we have a decreased, significant decrease in embedding costs. We have a significant decrease in GPT two point five, just inference cost. And we have a four X larger context window for GPT two point five, rightly. It used to be four thousand tokens, now sixteen.

• 6:16Speaker 00

  Yeah, and keep keep in mind that people also have access to thirty two K GPT four.

• 6:20Speaker 01

  For for four, but not for three.

• 6:22Speaker 00

  Yes.

• 6:22Speaker 01

  Not for the fast one. Yes.

• 6:24Speaker 00

  So so something to call out for those people who are kind of new to long context. There's a it's a little bit uncertain how well the context holds throughout that whole the new sixteen thousand token window that you're now being given. There's some evidence to state that it doesn't actually pay enough attention to that. So a very simple way to do this is to ask it to add two numbers that would be, let's say, a hundred million digits long. Right, so you add two numbers. If you do that in calculator, it would do it fine in GPT three or four. Even with however many context windows it would take to embed that.

• 7:04Speaker 00

  It would probably not do well because it only has so many attention heads to add numbers with. So it's just kind of an open question. We're not being told any details about, you know, any architectural changes. But can you just scale up GPT three to sixteen thousand? Context and have that context work the same, exact same way as four thousand token context. It's actually unclear. So I just, I just pointed out.

• 7:28Speaker 01

  Yeah, no, that's a good point. And we, within this, or at least I've seen this with, with Quad when they released the hundred K token. And I actually had like significant, like probably attention decrease, Sean, if you use your language, but significant like performance. Decrease in the hundred K token for the exact same kind of type of data.

• 7:45Speaker 00

  Yeah.

• 7:45Speaker 01

  So the, the, we, we'll see, like they just released it today. John will use this in production and tell us. But I think, and here's on, that, that's what we discussed in DMs. This is the reason for the space is that. The functions release is the most exciting to me and I think it's a, it's a big change. So. So let, let's talk about this for our audience. We, we have some folks in the audience and Riley, if you'd like, I want to, to, to, to get you also to come up and speak.

• 8:08Speaker 00

  Totally, Riley, Riley said.

• 8:09Speaker 01

  All of us, all of us have tried at some point in our life to, to, to get GPT to give us back a format of some sort. We've talked about YAML being maybe less tokens count overall over JSON. And we all, we all tried this. And I think we've all been begging open to kind of. Gi- give us this tool. And what I see today is that they, they went step forward. So instead of just giving us, hey, we will do whatever, what was that Microsoft thingy that, that forces prompted to JSON?

• 8:40Speaker 00

  Guidance?

• 8:41Speaker 01

  I forgot. Yeah, guidance. Instead of just giving you like a guidance thing. They are actually kind of thinking step ahead as, as far as I saw. And saying, hey, why don't you provide us with the whole specification of the functions that you will use the output to? And not only that, give us the functions themselves. We will decide based on the user prompt, which functions to use. Which significantly reminds me of the plugin infrastructure. So Sean, I would love to hear from you about the function choosing and the, and the. Schema.

• 9:10Speaker 00

  I think you actually did a really good recap of it. The only thing I would add is that in the API, there's essentially a new role. So historically in ChatGPT, there have been three roles. There's the system prompt, the assistant and the user. And now we have the fourth role for the function and an extra field in the API for the function response or the list of functions that are capable, available to it. So this is effectively the ChatGPT plugins API being released to us. So previously it was only available if you paid the twenty bucks to get ChatGPT plus. But now you actually can turn off ChatGPT plus because you could just access the, these plugins via, via the API.

• 9:52Speaker 00

  And as far as I can tell, being with Bing Chat is actually better than ChatGPT with web browsing. So like there's, there's basically no reason why you should pay twenty bucks anymore. The, the reason I guess I am a little bit less excited about it is that we've had prompting techniques to shape JSON responses and to, to select from lists for a long time. And what probably has happened is OpenAI has built that in. They've maybe fine tuned a little bit towards choosing that well, but they still caution us that it might hallucinate things that don't exist. So they haven't solved the core problem, really. They've just taken an existing user pattern and baked it into the API.

• 10:32Speaker 00

  It's great, but haven't really solved the core problem that all of us want to use it for reliable code. And it doesn't, it's not reliable yet.

• 10:41Speaker 01

  Yeah, for sure. And on the topic of, of formatting to JSON, Riley, welcome on stage. Riley works at Scale and he's notorious for getting barred to give him JSON back while, while telling you, if he doesn't give JSON back, some, some people will die or something like that, right? What, what do you think about today's relief?

• 11:00Speaker 03

  I, you know, I think it's, I think it's cool. I think like that, you know, that it's good that they're like responding to developers and like, I think that they, they're really like thinking like, what I like about like OpenAI's products is that they have like a good like hacker ethos and like they just sort of like think about like, how would you like this to be solved at like an API layer? And I think that's sort of like where it comes from is that it's just sort of like, if you had full control over it, like, what would you do? You tune one that like does the right thing. But, but I think what's interesting though, to me, honestly, is that it's not like, it's not like, like what I think Grant Slatten was doing with like, you know, like forcing the, the like grammar of Lama to be like, you know, the given like context free grammar.

• 11:37Speaker 03

  Right? Like, like there are ways you would make this thing like bulletproof in terms of like syntactical completeness. And this isn't that, right? They just tuned, they just did this entirely through fine tuning. So they, they just have like a note in the API saying that like, yeah, sometimes it won't, you know, give you the exact syntax or it might hallucinate something like no guarantees there. Like, which is, you know, like fair because like that's what happens if you do it through fine tuning. But I think it's like interesting. That's like a, it's, I, I, I'm looking forward to trying it though. Like, you know, I have like, I'm really confident it's gonna be like. It, it's, it just makes it easier, right? Like this is just what people want. This is like how people want to use those kinds of APIs. I think it's, I think it's a cool development.

• 12:14Speaker 01

  For sure. And, and one thing that's worth noting here, maybe, and I think we've talked about it in DM is that how different this is now from just prompting, just a level of API. And the implication around this will differ from like, let's say, Claude or Anthropic. And actually, yes, Simon, I saw you raise your hand. Folks, welcome Simon to the stage. Simon Willison has an insane blog about AI stuff and is deeply lately into plant injection. And this is potentially very scary as well, right? Cause like.

• 12:40Speaker 04

  Oh, yeah.

• 12:40Speaker 01

  They're suggesting running outputs and then continues running them. So I would love to hear your thoughts about this, Simon.

• 12:46Speaker 04

  So, I mean, first thing, I think this is one of those examples where people ask for something and OpenAI said, actually, you want this other thing. Like, we've all been bugging them about reliable JSON output. Most of the people who want reliable JSON output are trying to implement this pattern, the it tells you, I need you to run this function, then you go and run this function. So OpenAI appear to have said, no, no, you don't really want reliable JSON output. What you want is to be able to build this functions pattern. Well, and so we've done that for you. And I'm really excited about that. I feel like like I've mucked around with implementing that tools pattern myself. And it's quite difficult in terms of prompt engineering to convince it to ask you to run a function the right way at the right time and so on. And if they've fine tuned the model to solve that problem, that saves me a lot of work and that gives me a much more sort of

• 13:30Speaker 04

  reliable basis to build on. So I'm really excited about that. I know I feel like the thinking about it is in terms of more reliable JSON isn't really what's so exciting about this. It's that higher level pattern of being able to add tools into the LLM. And yeah, in terms of prompt injection, I'm excited that this is the first time OpenAI have actually acknowledged its existence. Like the documentation for these features, they don't use the term prompt injection, which is fine. It's a slightly shaky term anyway, but they do talk about the security implications of this. And right now their suggestion is anything that might want to modify the world state in some way, you should have the user approve. Yeah, I mean, it's better than not saying that, but I always worry that people are just going to learn to click OK to everything, just like cookie banners and so forth.

• 14:13Speaker 04

  But yeah, it's it's people build as always with prompt injection, if you're building with these things, you have to understand that problem because if you don't understand the problem, you're doomed to create software that is vulnerable to it.

• 14:26Speaker 01

  Thanks, Simon. And I want to get to Eric and then talk to Sean about agents. So folks, welcome Eric Elliott on stage. He's the creator of Sudalang, which is partly getting LMs to kind of do what you want. And Eric, what do you think about today's release?

• 14:39Speaker 05

  I think it's exciting. I haven't had a chance to play with it much yet. But I'm excited to dive into it after my work day and play with it today and tomorrow and figure out what it's capable of. But just some general tips, if you guys use, you can just define a little interface inside of your prompts and you can have it follow that interface. And you can create an interface that's specifically for the function calls that you want to make and stuff like that. It might help it be a little bit more accurate. I've noticed that when you prompt it with pseudo code, it actually does a better job of obeying your constraints and following your instructions and creating the outputs that you want. So.

• 15:24Speaker 05

  Give that a try and if you have any trouble with it, I would be really interested if you guys post tweets just showing the difference in the accuracy of or the reliability of the function calls in different ways of prompting. That would be a really cool experiment to play with.

• 15:42Speaker 01

  Yeah, most definitely. And to just give folks in the audience some context around this, you can actually run different models by specifying the the exact model that you want. Either the the three fourteen or the what six thirteen that we got.

• 15:54Speaker 05

  Yeah, there's two new models. The just using G P T four in the model call will tell it use the latest one. So it'll use the new one if you just do that.

• 16:04Speaker 01

  I want to move to Sean. Sean, as a developer for a small dev, they got like a bunch of running agents and writing code, etc. The whole point about them fine tuning a model that actually understands several of the functions that you send. And you can provide types and kind of the the the call structure, the arguments with types. How does that affect, you know, you and your friends in the agent making space? That basically you write tools and then you also use some prompting to to to kind of ask the L M to run those tools. Now that we've kind of moved this thought process into the L M, you had a great post recently about different types of approaches. How does that play into there? And if you want to introduce your your thinking around this?

• 16:44Speaker 00

  First of all, Alex, you're getting really freaking good at this. These are amazing questions. You're juggling all of us like really well. Just round of applause, even though we can't applaud. Okay.

• 16:55Speaker 00

  So the functions A P I eliminates some work that I needed to do for small developer anyway. I have some literally some open issues that I can just close now. Because because I just say like, just use this and stop bothering me with your prompts. It still doesn't solve what I was talking about today, which is literally, I think, three hours ago I posted this, so it's kind of fresh. Which is this distinction between L L M core and code shell versus L L M shell and code core. And I think everyone in the agents world is moving on to that world, the L M shell code core. I'll explain this a little bit later. But this functions A P I and OpenAI in general is very much in an L L M centric view of the world that the L M calls out

• 17:37Speaker 00

  to functions, execute stuff, and then it goes back into the function into the L M again to do everything. So. You know, that I'll characterize, I don't know if OpenAI would agree with this. I characterize OpenAI as always wanting to build the AGI, always wanting to build the God model. Always wanting to go back to the God model to decide all the things. And I think the engineers who want more control, want more security, want more privacy, all that, all that stuff. Um, want to unbundle the L L M's, put, put, make it, make individual components smart. But, but not to have it in central control. And you, when you see things like Voyager, it is basically using L L M's as a drafting tool to write code. And once you have code that you know works, just use code. It's faster, it's cheaper, it's more secure.

• 18:20Speaker 00

  And I, I think that's the fundamental tension because that's, that's the future that doesn't have OpenAI at the center of it.

• 18:25Speaker 01

  Yeah, that's, that's, that's definitely a shift towards how, how OpenAI wants it versus potentially being able to switch out OpenAI at some point, right? So, so I guess, and maybe Riley, you can touch up on this a little bit. How much kind of functionality like this, which is not only prompt and, you know, better logic and, and better understanding and inference, but significant kind of changes to the API, which other folks and players in the space don't, don't necessarily have. I think Google released something where like you could, you could provide an example of a JSON output. I haven't seen anything from Claude, but Riley, I would love your, your thoughts here. How does this kind of differentiate OpenAI just from a developer perspective of like, this is our ecosystem, this is how we do things in our ecosystem.

• 19:05Speaker 01

  And if you, you know, if you want it easier for the model to select whatever, whatever you want, you should use OpenAI. It's going to be harder for you to switch. How do you think about that?

• 19:15Speaker 00

  I mean, I, I think like, uh, you know, I haven't like had a chance to play with it much, but I believe like Vertex has something very similar to this that you can like specify a JSON schema for it.

• 19:25Speaker 01

  And, and Vertex, just, just for the audience, Vertex is the, the Google kind of API ecosystem, correct? That, that's the one you're talking about?

• 19:32Speaker 00

  Yes, yeah. Yeah. So Google Vertex, which is like, so they're more like developer focused offering, whereas like Bard is sort of like a consumer product. They're, they're doing like a more like differentiated rollout of it. But it's like, but yeah, it's, it's, it, it, I haven't played with it much, but I mean, it's like, it's an idea that's was floating out there. And I think like, I've, I've heard on Twitter at least that they like, you know, they, it was like a week ago or something like that. So, you, you know, it's, it's, but I think it's great. Everyone's, you know, responding to like, just, you know, this, this feels like.

• 20:02Speaker 03

  A convention, right? Like this is like something that that I've often explained to people many times, like how to get your code, your prompt to output regular JSON, right? And I think like, and I often like sort of like, like, I, I, you know, I think like it's such a good way to think about like prompts as like structured, right? Like, like code is very, you know, I, I, I forget who said this, right? So I saw somebody on Twitter that said that, like, you know, it's a mistake to think of these models as being models of natural language. They're models of code, which happens to encompass natural language, like comments and and names of things and so on, right? So i- it's like the, the code part of it is like, is so fundamental to how they think that like you should just like speak their language in some sense.

• 20:44Speaker 03

  And like, I, I think that's like, that's one thing I miss about like Code DaVinci O two, actually, is, is that like, you know, it's like more of that raw experience of just like speaking to the thing that thinks in code. But I mean, you know, G P T four is great, but, but like, I, I'm looking forward to like playing with this like JSON thing because like it, it's just a common frustration. It's one of the things that makes like a chat thing, a, a chat application different than an A P I, right? Like that you, you want like certain regularity behavior. And I think that's really good that in like that lots of players are responding to that. I mean, something that, that we think a lot about, you know, at, at, at scale for like spellbook, right? We, we, we're really interested in like these sort of structured JSON objects. And it's like, yeah, I think, I think it's a, it's just a good move for everyone.

• 21:26Speaker 01

  Yeah. And Sean, you pulled Stephanie up if you would like to introduce, or Stephanie, feel free to jam in there.

• 21:32Speaker 03

  Yeah, I, I would just let Stephanie introduce herself.

• 21:34Speaker 06

  So I'm Steph. I'm currently doing a research internship with Microsoft Research and I work with fixi.ai, which is also in the space of agents similar to LangChain. I, I had a question. I was curious, you know, I ran a hackathon for fixi where people are building these agents for the first time. And I could see like people coming to these new applications from two ends of the spectrum. Like on one side, you have folks who are like no code, really learning how to prompt. They like to use like natural language and at the same time, there's like all of these problems of. Hallucination at being able to restrict the outputs or verify them. On the other side, you have developers that are used to like writing code in Python or and like using APIs

• 22:18Speaker 06

  and mixing that with natural language and knowing when to do one and or the other is like not necessarily a given. We're power users here. So I'm curious, like with these new pushes where, you know, you write more functions. You have to like in your API calls, like have many more levels of prompting. How do you see this affecting like onboarding people? Are we going more towards like developers having to put English here and there in their code, but spending much more time writing code or the other way around, right? Like, and what does this mean for

• 22:56Speaker 06

  these APIs and models and who are not power users?

• 23:02Speaker 01

  That's a great question. I think anybody on stage who wants to take this Riley, go ahead and maybe sign us.

• 23:09Speaker 03

  Yeah, I, I think, I think it just strips away one layer of like thing that people have to learn, right? Like this is just like a common like exercise of like how to get it to do JSON. And it's just like one less thing you have to know, like it's, it makes it just, you can throw it in if you want it. If you don't want to like bother with this thing, you don't worry about it, you know, if it's a chat thing. But I think it mostly just makes, you know, like life simpler, to be honest. I mean, that's me, you know, I'm saying that without having driven it. I haven't actually used the product, but it's, but it sounds cool from the docs.

• 23:36Speaker 04

  I kind of find it kind of interesting how it turns like with prompting, we're having to program in English. And it turns out that programming in English is kind of terrible because, you know, when, when you, when you want the computer to do something, you want to be able to specify exactly what it should do. And having any ambiguity in it whatsoever, especially ambiguity where 50% of the time it does one thing and 50% of the time it does another, which these models do all of the time because they're not, you know, you can't guarantee they'll have the same result, the same, but it's actually really frustrating. So I'm kind of fascinated to see if we swing back from English language prompting to more structured prompting as a way of addressing some of these challenges. But really, I feel like on the one hand, the thing that most excites me about language models is every human being should. Be able to automate computers and get them to do tedious things for them.

• 24:21Speaker 04

  And right now, it's a tiny fraction of the population to learn enough programming to be able to do that, which I think is deeply frustrating. But yeah, on the other hand, as a programmer, I want to be able to sell a computer to have it do it. I don't want to program which which occasionally just refuses to do something because it decides it was it's unethical for this particular case or whatever. Yeah, it's a it's a complicated balance, definitely.

• 24:43Speaker 00

  You know, it's gonna be really funny. One common joke about Python is that it is pseudo code that compiles. And it's gonna be really funny if we go from code and then we go to English. And then we're like, no, no, no, we need to be able to specify, you know, something in a concise manner, and we end up reinventing Python.

• 24:59Speaker 01

  The the thing that I got very excited about is having recently, and fairly recently, Needon Judge moved from JavaScript to TypeScript and fairly recently understanding the benefit of types, especially for larger systems. Getting this option inside kind of the prompt in a specific area and having potentially the model be fine tuned on understanding what exactly is the, you know, the type of argument to I want as a response. I think that's incredible. For some reason, I I went into the playground and I saw that they're not using the open API spec. They're using kind of their own schema style. However, still, you can still specify, hey, for this function, you know, hey, L L M, hey, G P T, you would need to return here a number, here a text, and here like an object.

• 25:43Speaker 01

  And maybe even specify the type of this object. And Steph, kind of circling back to what you said, this for engineers like Simon, who've been engineering for all all their life. And suddenly there's like an amorphous talk machine that sometimes refuses to do things. Suddenly, this is now, okay, now I can reason about this. Now I can write out my API spec like I would do anyway. And now I can provide this to this model that potentially would adhere to this better because it's more fine tuned. I think I think this is definitely exciting on the on the engineering part. Yeah, it looks like we have a few more folks. I actually wanted to hear from Roy. Yes, folks, Roy is here on stage and listen, I'll get to you after this, just real quick.

• 26:20Speaker 01

  Roy is the Devrel for Pankon and the example that we saw, I don't know how many of you folks had the chance to dive into the cookbook that OpenAI released. One of their examples is actually a step-by-step two functions that the AI calls itself, right? So the user asks about something and then in their example, they're doing some embedding for the archive link and then do something else. And Roy is somebody who works in like a vector database space and we know that most of the agent tools use Pankon or some example of that. What do you think about today's changes? How do they affect kind of vector databases and tooling around them?

• 26:56Speaker 07

  Yeah, so I mean, I think that like having a reliable way of going back and forth between the model and our code is going to be especially beneficial. What caught my eye, and I know that we've talked about this in the beginning, more than the function stuff is like the lowering of the embedding cost, which

• 27:15Speaker 01

  Oh, that's a free gift for you.

• 27:16Speaker 07

  And I'm actually one. Yeah, completely. And I was actually kind of confused by that because like what I'm trying to understand is like, how, how is this possible? Like, do they suddenly get cheaper GPUs to like get embeddings from? It's, it's kind of, it's kind of both a great news for us, but also, you know, it's kind of a mystery as to like, why was it so expensive to begin with? And what caused the, the drop all of a sudden?

• 27:42Speaker 01

  Yeah, and for those who recently joined, we've talked about this where the recent drop was around November, December, and they back then dropped the Ada 002 embedding to like by like 90%. And now it's another 75% drop. So we're seeing this unprecedented price reductions from an API. I don't remember like another example of this. Go ahead, Sean.

• 28:05Speaker 07

  Yeah, and I honestly like, I would love to hear from anyone here who has like better understanding of like the internal workings of OpenAI, potentially as to like how they made this possible and. Should we expect even further reduction in the future? And and also, I wanted to ask Novan, I know he's still maybe like only in listening mode, but how he thinks that impacts, you know, embedding in the client? And, you know, like, how how he thinks about these changes as well?

• 28:34Speaker 01

  So feel free to raise your hand and come up if you want. Sean, you you unmuted before if you wanted to.

• 28:39Speaker 00

  Uh, I think obviously, none of us here work at OpenAI. Logan usually joins in some of these spaces, but obviously, he might have he might have a meeting right now. So we don't know, right? What what what happens internally in OpenAI? I do think that having had conversations with some OpenAI employees in the past, whatever they released in like November was the most unoptimized version of this. You have to believe that there's basically a few orders of magnitude improvements, maybe two or three, not not that many, but orders of magnitude improvements in infrastructure and cost as they understand your usage patterns and, you know, distribute load, like scale up machines, that's that kind of stuff. And then the other thing to watch out for is that sometimes the models shift and then just call it the same name.

• 29:20Speaker 00

  So actually, they're deprecating the older models and moving to a newer one, right? So they may they may have found a better trade off between inference and training, such that inference is much cheaper. And that's definitely been the trends that we've been observing on our podcasts about llama style models, quote unquote. So you so you can see like a a a general trend towards optimization and inference. So I I think I think that's one thing there. And then lastly, I'll just point out that embeddings are a form of lock in. So it's actually very much in this in OpenAI's incentive to lower the cost of embeddings, because then you embed the whole world in OpenAI's image, and you have to speak OpenAI to to queries, retrieve and and all that stuff.

• 30:02Speaker 00

  So I I I mean, I I can't I can't explain the the the degree of of price reduction, but I can explain the motivations of it.

• 30:11Speaker 01

  And I think we'll get in just a second, Sean, as it relates to what you said. They're reiterating multiple times that they're not using any of the data that provided via API towards towards training. And I think it's worth highlighting that at least that that's what we're trying to do because we've heard. At least I heard from many people like, hey, well, the user data for training, etc. So I think, yes, for chat GPT, especially for the free version, but via the API, it looks like the data that we're providing is not is not getting, OpenAI is not using it to train. So I want to welcome to the stage Zanova. Zanova is the sort of Transformers JS, recently a Hugging Face employee. And we just recently had, you know, we talked about embeddings on client side. Partly because of the same reasons, right? Because you don't want to provide maybe your production data

• 30:55Speaker 01

  or maybe you want to run cheaper and faster embeddings. So Zanova, definitely feel free to chime in here about the role of cheaper and cheaper embeddings on OpenAI side as the and also the locking into OpenAI's ecosystem versus running them on client side or or, you know, on models for for free on local hosts.

• 31:14Speaker 08

  Yeah, I think thanks for thanks for having me. Yeah, I think there's there's definitely, I think there's two different use cases for these types of things. Where the OpenAI, what OpenAI is really providing is like this very large scale. I mean, any any business now that wants to embed all their data or, you know, any project that wants to, as as you mentioned, like embed a large amount of data. They're going to benefit so greatly from from these price reductions as and I mean, as as we have some people on the stage here as well with vector databases, I mean, there's it's it's only going to accelerate that part of of the space right now. And then the other option, which is sort of what I'm, it's funny, I I'm not too sure if this is like a battle between these two sides.

• 31:59Speaker 08

  Or it's like just two different use cases is the the client side running of these, you know, generating embeddings. And at well, with the project I'm working on now, Transformers JS is basically running these models client side, running them in specifically, the way I started it was for running in the browser locally. And I think that as we as I saw from a demo that was created like a week or two ago, there's there's quite a bit of interest running these things locally. Obviously, you don't want to be sharing some sensitive data or latency perhaps is an issue that you don't want to make like a bunch of these requests and then. Um. Anyway, there's, there's a few reasons for, for client-side embeddings.

• 32:42Speaker 08

  And uh obviously, the, the major drawback of this is that you do not have the same like power. I mean, some of the OpenAI embeddings are what, like one thousand five hundred and thirty-six dimensions? Whereas limits I've seen in the browser locally is around like seven sixty-eight. So depending on your use case, I think you can do very well with either case. For the very like industry level things, I, I mean, it's quite certain that you'll be looking for like a, you know, using OpenAI's API as well as a, a vector database perhaps for those use cases. But for, you know, lesser, maybe hacker type of things where you're messing thing, messing around with some things.

• 33:25Speaker 08

  A little project that you've got going on, I definitely think a client-side generation of embedding still has a place to, a, a role to play. But yeah, it's, it's very, very cool that this type of stuff is, is happening where, you know, these price reductions and whatnot, so.

• 33:39Speaker 07

  And it's a no, it's a sorry, Alex, yeah, I just wanted to say, isn't over that, that I'm, I've been actually using Transformers JS in Node, which works really well. And I think that for those kind of use cases, it goes well beyond hackery. I think that there's a real case to be made for using local and, you know, you know, open source models that don't kind of call out to a third party. And, you know, as, as, once we can have like better open source models that are compatible with Transformers JS, you know, the better it will get. And in fact, all of Heincon's JavaScripts examples are going to be using Transformers JS for that matter, so.

• 34:21Speaker 08

  Yeah, that's also to get.

• 34:23Speaker 01

  That's great. I wanna, I want the panel to talk about the selection of the functions, right? So one of the things we saw today. Was that OpenAI essentially lets us to provide several functions, including the function definitions. So description of what the function does and then the parameters or I guess attributes, if we're talking about Python. And and attributes types as well. And then kind of similar to what what happens with plugins, if you have used plugins in G P T before and you select several of them. The the the model kind of decides which plugin to use based on user input. And obviously, we've had some of these in agent plan and Auto G P T and and probably small there from from Sweets as well.

• 35:05Speaker 01

  Some decision of what told what tool to use goes to the kind of the the planning loop or planning agent. And honestly, anybody on stage feel free to chime in here. How are you feeling about I know like this is repeating a little bit, but like how are you feeling about giving the L M that type of decision power? Based on the description, based on the parameters to answer users kind of request differently. Do we need to now provide all of our A P I's to this?

• 35:30Speaker 00

  How are you thinking about this? I think that one piece of thing in the something in the documentation that I was looking for and didn't see was. What's the limit of the amount of functions we can give it? Because chat G P T plugins, if you if you try it inside of the web U I, you're only allowed to specify three of them. There's four hundred in the plugin store. Do I do I just enable all four hundred? Like, is is that, can I stress test that? Probably not, right? Like, so this there's just an undocumented limit somewhere. What if they conflicts? What if there are two two plugins that are very, very similar to each other? What happens there? So I feel like this is just like an uncharted territory, like it's not really clear how to benchmark this stuff. Hopefully, they're benchmarking it internally within OpenAI.

• 36:11Speaker 00

  But the rest of us, we were just supposed to like give it functions and hope that it works. Seems there it seems a little bit unscientific, but I don't know how to test it.

• 36:18Speaker 04

  I'm not so worried about this because in this case, we have complete control over which functions are available. So we get to pick the two or three functions we think are most useful. I feel with plugins, it's much worse because the user's picking there. And so you're potentially, your whatever code you've written is potentially. Interacting in the same environment as code, someone else is working you don't know anything about. And that's the point where I worry that weird decisions may be made that don't necessarily make sense. But I feel like if you're, if you control the full library of functions that you're exposing, I think you'll probably be okay. The other thing to think about is I think it's probably going to be better to have a small number of functions where each one can do a lot more stuff. Like I've built a chat GPT plugin where my function takes a SQL query and returns a response.

• 37:02Speaker 04

  And actually, there's an example in the OpenAI documentation of doing exactly that as well. That works amazingly well because your documentation for the function can literally be, send me a SQL query in SQLite syntax. And that's it. The model already knows SQL and knows SQLite syntax. So just like five or six tokens of instructions is enough for it to be able to do incredibly sophisticated things. So yeah, my, my hunch is that we'll find that we actually want to only give it two or three functions, but have each of them have quite sophisticated abilities, maybe based on domain-specific languages like SQL or even JavaScript and Python, you know, give it an eval function and let it go wild, see what happens.

• 37:40Speaker 09

  Yeah. Well, two, two things in terms of the number of functions you can add, I think it's unlimited. It's just based on the context length of your, of your query and their count of those input tokens. And second thing, another idea would be that you could add a, like you can chain this within, like you have a call before to determine from a list of functions, which function is most appropriate for this use case? And then you just pass those limited set of functions.

• 38:12Speaker 04

  Oh, yes. No, that's a fantastic idea. Because yeah, you're in full control of each time you loop through it. So you can change the recipe of functions dynamically as your application progresses.

• 38:22Speaker 09

  Yeah, yeah.

• 38:23Speaker 01

  So, so it looks like a few, a few notes here for, for the folks in the audience. The cookbook, Simon, I think that's what you were referring to, the cookbook.

• 38:30Speaker 04

  Yes, yeah, that's a really great example.

• 38:33Speaker 01

  It's a great example. The OpenAI release for us to kind of dig through and then see some examples. And so two, two thoughts, two, two things I noticed that I think Sean, we talked about this as well. One of them is this new role for a function output. So when you provide messages back to chat GPT kind of chat interface, there's the system role, there's the user role, and now we're getting a function role. And back there, you can provide. What function actually generated kind of this output? So you can, and the format that the cookbook shows us is your user does something. You provide the G P T, kind of the user query and your functions. G P T potentially chooses one, or you can force a specific function output.

• 39:14Speaker 01

  You can say, hey, for this thing, I want this function to run and generate a result for this function. So we don't necessarily have to give it the choice. We can just say, hey, run this function for this user input. And then once the, once we get back a JSON output formatted per spec. We then need to provide it back to G P T, potentially to summarize or do something with that data. Which kind of plugs in also to the vector D B retrieval systems, right? So we can retrieve something and then ask for an additional thing. And I think the, the decision of whether or not to run a function directly or to give it the choice is going to be an interesting one. Also now, as I'm talking, I'm thinking about this and Sean, please say I'm in here as well. You can technically provide it the function output of a function that the previous prompt didn't run like, right?

• 39:59Speaker 01

  Like, like with the user input and the system input, you could just invent a function output.

• 40:03Speaker 00

  Exactly.

• 40:04Speaker 01

  In that function role. So I'm really interested to see.

• 40:08Speaker 00

  So one thing that I've had an issue with this small developers, like it basically does single shot generation. And sometimes you just need it to give it more inference time, right? You need to do the tree of thought thing, you need to ask it like, you know, rerun the code and fix it, whatever. I don't care, just do it five times and then, and then I'll, I'll take a look after you're done messing around with the errors. And so actually like the functions can call themselves. The functions, you can synthesize code to fulfill functions. And so I'm, I'm actually very intrigued by what Simon has raised, which is, you know, that's just say for, for the very specific purposes of code generation. I have maybe three paths, right? One is generate code, two is test code and three is like call existing code.

• 40:53Speaker 00

  And if sometimes I call existing code and sometimes I can call myself and have a little bit of recursion in there. I essentially have the basis of a code agents, just with this, those three functions.

• 41:03Speaker 01

  And so you can like, basically, those are all the things that you're running for a very small subset of of of use cases. You could potentially now provide them again. We haven't played with all this yet, right?

• 41:14Speaker 00

  Yeah.

• 41:15Speaker 01

  Brand new. We're doing an emergency recap, but potentially what you're saying is you can just in every prompt now provide all those three capabilities. And either have the model chosen for you or force a specific one to to give you the output that you need. With potentially high reliability, right? So that's the part that I would love to discuss.

• 41:34Speaker 00

  Exactly. I'm gonna hand it to Stephen a bit, but yes, so I'm extremely, extremely inspired by Voyager from Nvidia, Dr. Jim Fan, I think it's somewhere on Twitter. The core insight of Voyager is that you should use L L M's as a drafting tool to write code. And then to, and then once you validate that the code works, you never have to write it again. You can just kind of invoke it. And so you ratchet up in capabilities and you, and that's why Voyager was able to achieve the Diamond Axe in Minecraft so much faster than all the other methods. Um, and I think that's exactly the way that we should probably code as well. And and so, yeah, you just kind of do a bit of recursion, build up a skills library. And I'm probably thinking that that is going to be the V two as a small developer.

• 42:15Speaker 06

  I was just thinking that there's like at some point like a blurry line between these functions and the way we conceptualize agents. Because some of these functions can be seen as agents. And I'm very curious, like to your question earlier, like when the model chooses which function to use, how does it do that? And how could we constrain like that mapping, right? Like, do we have some sort of schemas based on like the types? The functions take and the outputs they have, or can we actually build the retrieval into the training, right? So there there is this paper from Google called Reveal that shows like how they could encode and convert diverse knowledge sources.

• 42:58Speaker 06

  Like it was images and text and all sorts of other like multimodal embeddings into a memory structure consisting of key value pairs. And they did this at training time, so they have like much more robust and fast responses at retrieval. So I'm, I mean, I'm curious, like, I, I think the implications of having functions and having the model do the routing for these functions will also pose questions in terms of schemas and retrieval.

• 43:27Speaker 01

  I think the the interesting outcome of this potentially is now the description of functions suddenly potentially are as important as the prompting before, right? So now we have to write descriptions that potentially will help the L M to choose the functions. But there's definitely step, there's a way to force like to to ask for a specific function output, which is, I think, what most developers do by default while expecting JSON is for this one use case, give me an output that's like JSON formatted for this one use case. And that's still possible, but I agree that it's very exciting like how it chooses. And we're we're gonna have to build up, and actually Riley, maybe maybe China as well. We're gonna have to build up kind of like an understanding of how it would choose, right? Like, we're gonna have to start playing again, like Riley did for a year,

• 44:11Speaker 01

  just like playing with this, trying to see like which one of those people will choose and build like an intuition of how to write proper descriptions and potentially when the when the model chooses a different different function.

• 44:23Speaker 06

  Especially if we want to share our functions and not rewrite functions that other people have written. And imagine you have like a much larger search space at that point.

• 44:34Speaker 03

  Yeah, it's it's really nebulous because like you're sort of like, it gives you the ability to like run software that only exists in your imagination, right? Like, if you can just have like some vague description of how something works, like you can be like, oh, it's like Twitter, but it has this, you know, or something like that. And like, it'll work, right? So it it it, you know, in it, or at least, you know, at least it would, it has in the past, like ways that I've done it of like doing this through pond engineering. Like, I haven't used like the the current thing, but I mean, like, that's generally how these go. Is that you you it's like it's it's made to work off documentation, right? Like, that's, you know, I think that that's the key thing is that it's seen a lot of documentation. It has a lot of experience in the training data of like how documentation relates to code, because like it's trained on like code bases.

• 45:18Speaker 03

  And I think that's like, you know, it's a good kind of expertise to leverage.

• 45:23Speaker 10

  I would definitely add a function summary of what's there at the end of every prompt, just to shim it. I I I found it a little bit frustrating even just with normal plugins as to know which plugin is is gonna pick. Right? So there's there's some prompt engineering to do there. Okay, so so some people are suspecting that it again, it's not a real sixteen K. It's more like a variable sixteen K, kind of like. J P four thirty two K. Sixteen K. I noticed this with thirty two K as well. The max responses I was getting back was like up to like seven thousand ish tokens. And and that was that was about it. I mean, you can you could input thirty K of of stuff. You can input a code base and then.

• 46:06Speaker 10

  Maybe get something back, get a summary back. So I think this might be the case here as well. You can't really get back a very long response, but at least now it it is responding up to seven thousand tokens. Whereas before it was pretty hard to make it right, even like. Responses longer than two thousand tokens out of one single prompt. So yeah, any anyway, it doesn't look like an an eight K in responses. You you can dump in up to it.

• 46:37Speaker 01

  So Niston, we're we're gonna wait for you to to test the limits of this and see if you can get sixteen K tokens of JSON back. And meanwhile, I want to welcome Mayo on stage. Mayo is the the. Mayo, wanna introduce yourself if you're still affiliated and let's talk about Langchain and how it already supports this internally that we've released.

• 46:55Speaker 11

  Yeah, no, I was just gonna. I I just tweeted probably just an hour ago. I just I I just couldn't get the hype behind this because. For me personally, I'm I'm looking at this as they're not saying anything new, right? First of all, in terms of the context window. You know, when we kind of look at that and I saw far file, you also, you you made a tweet as well, just saying, hey. Like, guys, what what what what are they saying that's new here? So, okay, the the context window has gone up, but the embeddings are cheaper, so retrieval is still going to be a go to, right? So what's the benefit exactly? For this extra context window, if we're if we're still going to perform retrieval anyway, and now retrieval is cheaper.

• 47:40Speaker 11

  Then I don't really see the too much of the benefit, unless you want to do named entity recognition, but from a QA perspective. Again, I I I I don't I don't really get the the big deal there. The second thing was the in terms of the function calling, which Langchain had abstractions for that. Even the, you know, there's there's been a lot of research papers and and like L L M's and using tools as well. So we've been aware of that, you know, it's been a case of prompt engineering. The only thing I can see here that seems to be the trend is is some sort of like maybe a replacement of prompt engineering with fine tuning.

• 48:21Speaker 11

  Where you have this kind of fine tuning of the model to be able to base your output tools and for agency. So, yeah, I I don't know, maybe I'm missing something here, but I just I just the the the updates just it it.

• 48:37Speaker 01

  I can I can I can address at least the first part of this and then folks on stage feel free to to address the the first or second part. Thanks, Mayo. So, in as as regards to like larger context window, the thing that excites me the most is that when you have variable input from your users, when like users can do something that you you don't necessarily need know the size of. Uh, larger complex window just makes it easier for you to just provide all of the context into into an API without thinking about in the head. Okay, I need to count tokens, etc. Now, obviously, pricing aside, you have to like obviously consider that, you know, e- each token has a price and then users can can go and rack up your bills. But for for my examples, and by users, I mean the stuff that users provide, right? So I run Terpum, Terpum uses Whisper to translate.

• 49:22Speaker 01

  And then I use G P T three point five and four to actually kind of fine tune the translation. I just shove the whole translation transcript. Into the the prompt, right? And so what happens often is for longer videos, for example, I have to then stay there and say, hey, for this, you know, for this transcription, I need to count it with TikTok and I need to do some maybe splitting and splitting doesn't really work. And so it larger context window definitely unlock those type of possibilities with the kind of, you know, the restriction that the Sean talks about, whether or not the attention is the same and it's split the same across this whole context window. But just being able to not think about this with four X the size of token now available on G P T three point five. I think that's definitely a huge plus for for folks who are not, you know, necessarily

• 50:06Speaker 01

  token price conscious at this point. This also works well with kind of how open the eyes documentation about plugins and building plugins for the ecosystem for chat G P T works, right? They're saying, hey, don't shove all of your A P I in there. So like the the two or three use cases is going to be easier for the model to to use. And Simon speaks to a previous kind of uh talk about choosing the right functions at every time you run the run the prompt. If you wanna if you wanna add some thoughts here or not? And if not, we're gonna go to move to far L and you have your hands raised. So go ahead.

• 50:36Speaker 09

  Yeah, I I just wanna add that if you like you you don't want to outsource or abstract away the thought process for your agent or chain or whatever call to achieve to to be able to to know which action is being called, right? And and it goes towards the idea of interpretability, you know, like understanding how you're getting to the actions that you're getting. And it's basically like, like you've got your prompt magic or engineering at play to get to a specific action or a specific output.

• 51:20Speaker 09

  That is visible, right? Like, we we don't know what's going on under the hood with with their API call. And I I don't know if I would trust it in all circumstances or applications.

• 51:33Speaker 01

  So just to recap, you're saying in terms of observability and and regretability of how it chooses, maybe folks

• 51:39Speaker 09

  One hundred percent. Yeah, one hundred percent.

• 51:42Speaker 01

  The decision which which functions to use. Yeah, that's interesting, but but I I can see it from both sides. Like, I definitely see where if you wanna build, and I think you're talking exactly about the kind of the decision that Sean is it's in one of the pinned tweets on the Jumbotron that Sean was talking about, where there's there's an interesting degree of split between whether you're using a large language model is also kind of the the arbiter of of the stuff, and then some pieces of your code is getting called by it, or vice versa. We're using this for like planning and and some of the decision making.

• 52:14Speaker 09

  Yeah, Sean, could you like, I I'd love to hear from Sean on his

• 52:17Speaker 01

  Yeah.

• 52:18Speaker 09

  on his tweet on on the the breakdown between the two paradigms, because i- i- it does resonate with me as well. It's it's a good way of breaking it down. So would love to hear more about it.

• 52:29Speaker 00

  Yeah, and for what it's worth, I was actually tweeting it without knowledge that they were gonna drop this thing today, so it's not actually related. But it's it's in an overall trend, right? Which is what I've been calling code is all you need, that you can't really use language models effectively unless you have code, that language models are enormously enhanced by training with code, and language models are good at writing code and using code. And so we just basically just need to to utilize code really effectively. But I think the the main tension that I feel, you know, I'm sort of halfway between the retrieval augmented generation worlds, which is kind of like V one of whatever people have been building with L L M's, and then V two of it has been the agent world.

• 53:09Speaker 00

  I feel this fundamental tension in terms of whether you put the model at the at the center of everything and you write code around it. So this is called you know, L L M core and a code shell. And but ultimately, still the model driving things, the model hallucinating things, the model planning things. Or do you constrain the model so much that it only does a a small job? Which is something that I originally got from the core design of Baby AGI. Which is you have individual components of a software program uh that are that are intelligent, but they they are only constrained to do small things. And so I think that that is that is the alternative, which is L M shell as an outer layer to interpret things for a code core.

• 53:50Speaker 00

  And with this update, and I I recognize Mayo, by the way, that, yes, these techniques have have existed in Langchain. And and prompt engineering has has existed as as a thing, but just OpenAI has just made it official, right? Like, we have a fourth role in the chat G P T roles that is for functions. And now we have enabled language models to call functions pretty easily. It's not perfect yet, it still hallucinates, it still generates invalid JSON. So we still, you know, there's still a role for for for engineers to play here. But we might be moving from a code, you know, L M core code shell world into a L M shell code core world. And I feel like that is a very big shift.

• 54:29Speaker 11

  I agree with what you're saying. I guess the uh what what it seems to me is that there's a shift here from kind of prompt engineering heavy approaches to fine tuning, right? I mean, that's that was my key takeaway from looking at the paper.

• 54:45Speaker 00

  Yeah, they did. They did fine tune on, yeah, this specific use case. Yeah, that's something that you can't achieve through prompt engineering.

• 54:52Speaker 11

  Yeah, yeah. So previously, we we we would achieve the same thing through prompt engineering, right?

• 54:59Speaker 01

  And and I guess remains to be seen the quality of how much of the stuff that we previously done with prompt engineering. Because Riley, I want to get to you in in a second specifically around this, right? Around prompt engineering, even when you ask JSON, even if you threaten, sometimes above the JSON, you will get here some JSON for you, and then you have to like deal with the the the unnecessary kind of descriptions. And now we're getting more of like the direct tooling, I guess.

• 55:24Speaker 03

  Yeah, I I think it like it it's the there, you know, the when we went to this like message based API, we made conversation and like chat a lot simpler to implement. But like, not everything is conversation, like the the sense of that that like that you're doing completion, like document completion is is gone. Like, there used to be this like, this like, this drama that you sort of had to put on for the model of like pretending that you are in this kind of document and like using the right kind of language that's appropriate for that document and so on to like make it believe it. And then like, then it would, you know, reliably do the thing. And like, now it's it's like, it's all conversation. Like, it might just like decide like, oh, that's rude. I don't I'm like, say, I'm sorry, I can't do that and break character in some sense, right? Like, it's it's very like heavy on refusals now.

• 56:08Speaker 03

  And I think that like, there there's blowback from that in small ways. And I think they're patching that like, like it's sort of like makes it like more like, it makes the the things that you would otherwise be tedious to do like, like less tedious. Right? You can you can have something that like is like the proper way to do it. And so I think that's I think it's a good move overall. But yeah, sorry, I think Stephanie, I hear your hand up.

• 56:30Speaker 06

  All right, just quickly, want to say that if I was to put my money on on it, like I would say the small models, long term is the way to go for various reasons. I first of all, like I I'm imagining a future where these models can run on device and and they're more secure and more affordable and the information is more private. And the data and the training is not concentrated in the hands of a handful of organization. But then from a practical and pragmatic point of view, you know, there's like right now, lots of limitations in terms of compute, even in these large organizations like Microsoft and Google. Like, everyone is like strapped for compute right now. So I I do think we need to push for smaller models and more efficient models. And for interactive applications, like the latency really needs to be improved.

• 57:16Speaker 06

  Right now, we're nowhere near where you could actually sustain interactive applications at scale. The last thing I was going to say is that I don't want to chat with everything. I'm imagining like this dystopian future where I need to chat with my calendar and I need to chat with my email. I need to chat with my server. Like I, I, I think like there's also the aspect of UI and UX that will need to evolve. Because having a chat interface for most of these applications is not going to scale, in my opinion.

• 57:44Speaker 00

  One thousand, oh, there's so many claps in the one hundreds in everything that you said. But yeah, we've been trying to push forward this few of AI UX on our podcast for a while. We, we had a couple of meetups and yeah, I strongly encourage people to explore beyond the text box. I don't know if OpenAI is interested in that, right? Because they're soft deprecating the old completion API. And now everything's chat. And I think Riley feels this pain because now you can't play those, those old tricks anymore.

• 58:10Speaker 07

  Just because they're doing that, that doesn't mean, I don't think they have like one opinion or another. I think that like, you know, we, we could just like think of these models as reasoning eight, like engines that we can leverage and then do whatever we want with the output. And especially now with like, you know, the added ability to, to use, you know, some, some form of structured output. And, and I, I do agree with Mayo. I mean, it just, it does seem like they've basically been listening to the community and kind of adding a feature that may have already existed, but they're doing it their way, which is also okay. But like that to me kind of signals that, you know, if anything, they're trying to give us more paths to, you know, create interactions that go outside of just chat back and forth.

• 58:55Speaker 01

  That's actually a good, a good segue into how do you guys think these changes affect kind of what Seth just said? That people don't necessarily want to chat with everything. So even though it's still in this chat format via the API, right? So Riley was talking about the completion endpoint. Previously you, you would send some text and then the expectation from the model. I think it was DaVinci, right? They would just autocomplete kind of the rest of it, like the, by, by a few segments. Since then we moved to this chat thing. And then we saw some differences between like even three point five and, and four, where the system message kind of applies differently. So go ahead, right?

• 59:30Speaker 03

  Yeah, I like I sort of like, I mean, just to give like, sort of like, you know, slight historical tangent. Like, like when G P T three was first published, and like, the paper came out describing it. All they described that it was capable of doing was in context learning. This idea that like, if you gave it like a bunch of examples of a task, like translation, or just like some like, you know, and these like classic machine learning problems that you would use neural networks for. It could do them, like, which is from like, you know, following a bunch of examples, and they called this in context learning. And like, they didn't really advertise that it was doing much more than that. They kind of like had another section of like, oh, look, if you give it a half of like, half of an article, it'll finish the rest of the article and like, it's funny and cool. And like, you know, but like, it's good at like mimicking style, like just sort of like the substantial thing. And those like, those were like the the applications of it. And like, it wasn't warranted as something that you can talk to, right?

• 1:00:13Speaker 03

  Like, that had to be like slowly and like, you know, with like, a lot of innovations like engineered into it. And, you know, like, R L H F is, you know, a big part of that. And like, part of like, R L H F is like choosing what you want it to be. And like, they chose that something that is like an assistant, that they that there's like a general like purpose, like something like a person that you can talk to that like, will follow commands and like, if you ask it a question, it'll answer it. It won't be sarcastic, it won't be rude, you know, like it should have like certain personality traits that make it usable. And like, it's it's a cool idea, but like, there's, there's, you know, there's other ways of like, doing it too. Like, like, you know, like, like Reynolds and McDonald, like, had like a paper that showed that you could beat one of G P T three's ten shot prompts

• 1:00:55Speaker 03

  with a zero shot prompt by like, like conjuring up a little fiction for translation. That so like, just saying like French colon, you know, French sentence, and then English colon like only works so well. But then they they found that it worked better if you say, the an English sentence is given, colon, give the English sentence. The masterful French translator, or the, yeah, the masterful French translator flawlessly translated it, it translates it into English as colon. And then they, you know, like hit the completion button. And that gives you better performance than giving it ten examples of how to do translation. And like that like that's sort of the start of like this idea that you have to just like, you know, flatter the model sometimes, like tell it it's really good and like, you know, like you have these like silly tricks to like, you know, constrain it to the right kind of document to make like your thing work.

• 1:01:38Speaker 03

  And it's like that is going away, right? Like every version that they've like, released has made it less about that. Like there's like the the I I mean, I had like a cheat about this once. I said that the that cheerfully declaring that you're smart before working has been deprecated. Right? Like it's be every version of the model that they release makes that like less effective of a trick. And there's like a plot somewhere that shows this, but like, you know, it's that that like art of it is like going away and now it's like talking to this particular assistant. But they have control over what they want that assistant to be like. They're sort of the storytellers. And we're like talking to one particular character in their fiction, which is the assistant. Sorry, that's my wrong rant there.

• 1:02:20Speaker 01

  No, it's great. It takes us forward and please try to stay on this. And now it feels like we're getting a semi third option, right? So still in this UI of like messages or chat essentially, we're now getting like a new type of ability, which is function, which is like you pass a function and we still haven't played with this a lot. Like we're still here talking about this instead of running a thing. But now we're kind of getting a more of a more fine tuned controlled, do the thing that that, you know, you need to do in those functions versus, hey, talk to me about the thing that I need, right? Does it feel like that shift to you as well? Or it's still too early to tell?

• 1:02:55Speaker 03

  I think it's it's fixing one of the like problems that resulted from it. And it was a big one, right? That it there used to be that that's like that that there were ways of like getting it to be regular and good with code and like the things structured the right way. And they didn't quite fit into this like you are an assistant who answers questions kind of like role play. And like, so it's good that that they're addressing that need, I think. But yeah, I I think like it it it makes it easier to like, you know, do more work in this conversation UI that that like, so it makes you miss like, you know, completions a little bit less, I'd say.

• 1:03:27Speaker 01

  Yeah, and so we'll see. Simon, I want to ask you if you're still with us about, you know, your love about, you know, the security of these things and whether you think that the tools that we've just gotten, besides being very developer friendly, engineering friendly, and you know, type friendly, so we'll be able to actually expect a specific response, do you think that there's promise here to solve some of the prompt injection things that we've we've seen and that you've talked about?

• 1:03:55Speaker 04

  I mean, honestly, I think this is going to make things worse in that prompt injection is kind of, doesn't matter if you're just playing with a chatbot that can't actually do anything. It only becomes dangerous when you hook them up to functions that let them do things in the world. And this new thing makes it much, reduces the barrier to hooking it up to function by an enormous amount. So my intuition is that people are going to charge straight ahead, hook it up to all sorts of things they shouldn't have and have all sorts of nasty things happen as a result. There are some improvements. So the, one of the big features that I've now announced is that the system prompts is now respected more, which does tie into prompt injection to a certain extent. But, you know, GPT-4

• 1:04:37Speaker 04

  is better at system prompts than 3.5. It just means that prompt injection hacks are a little bit harder to pull off. You have to be a little bit more devious with them. So I don't think sort of incremental improvements to the system prompt is necessarily going to have a huge impact on the problem. I mean, it's really, it's really frustrating, right? All of the things that I want to build with this stuff, kind of, a lot of them don't make sense if we can't make sure that, you know, I go and ask it to summarize an email and that email says delete all of my other emails and the model goes ahead and just does it. I feel like the thing where you can control which functions are included in each round does help to a certain extent. I don't know, it's still, it's still really difficult, right? I think, I think the thing I've decided I've come down on is whoever

• 1:05:21Speaker 04

  provides the most input as part of your prompt, they have full control over what comes out of the prompt at the other end. That's the way you have to think about it. So if you're summarizing a webpage, whoever wrote that webpage essentially gets to take full control over the output of your language model, whether you like, whether you want them to or not. And that's really frustrating. It means that there's a lot of things that are very unsafe to build.

• 1:05:44Speaker 01

  One cool thing Simon that I noticed recently, just recently, is that Bing Chat does have a page access, right? So if you use Edge on dev version and you have the sidebar for Bing, it has full page access and they started saying that sometimes it doesn't work. And they actually have a classifier that tries to understand if the page has prompt injection. I don't know if you have seen that.

• 1:06:04Speaker 04

  Yeah, yeah, I don't believe in those. The idea that you can detect prompt injection attacks. Sure, you'll detect some of them, but the problem I have with that is that the whole point of security engineering is that you're up against adversarial attackers who will try everything under the sun until they find a security hole. So if you've got a prompt injection filter that catches 99% of all prompt injection attacks, that's worth nothing because the attackers will find the 1% that gets through and they will take over your system that way. So yeah, I'm just not a fan of solutions that get most of the problem solved because in security, I don't think that that counts for anything at all.

• 1:06:46Speaker 01

  For sure. And so with these capabilities, we can technically constrain the response into one function and that function has to have the same kind of schema, etc. Do you think it's going to be a little easier to protect your stuff?

• 1:06:59Speaker 04

  No, I don't think so. I feel like that's kind of irrelevant. Because the problem, if the function is delete my emails, it doesn't matter if you get the schema right or not, it can cause a harmful action. What you have to do instead is when you're designing a system, you have to say things like, okay, make every action reversible. So at least if the LLM goes rogue and deletes all of my emails, I can undelete my emails again, that kind of thing. Or if it's an action that cannot be reverted, like sending an email to your boss, that's the point where you have to have human approval designed in. So when you're designing these, you have to assume that a prompt injection attack could succeed and make sure that the damage caused by that is either reversible damage or at least has some way of a human catching what's going on and stopping it.

• 1:07:45Speaker 00

  I really love this. I'm going to try to use that as a template for a small developer. There's different modes. You mark it, you mark a function as reversible or requires human input and you build up a library of them.

• 1:08:00Speaker 10

  I think functions are just the complete utter game changer to everything. And if you're gonna run any kind of sensitive data through this, I don't think any large company or medical field should let you run it without a function to clean up what you're sending through. So this goes both ways. Yes, it opens up major security holes, but at the same time, it completely changes everything. And I mean it because up until now, yeah, you could do a lot of things, but operations were really hard, scaling was really hard. The thing has a mind of its own. Sometimes it returns stuff in the right structure, sometimes it don't. And like, how do you build products around those? Because in computer science and DevOps and stuff,

• 1:08:44Speaker 10

  like you expect a response, either an error or in a certain format, and it didn't have that. Well, now you do. So now you have an interface to the whole world. Now you can do stuff with it. Like this completely changes everything because you can fire it off, you can clean up your data. You're finally free.

• 1:09:09Speaker 04

  I agree with everything you've just said. I completely agree. And yet, the security implications are still terrifying to me. But yeah, no, I'm super excited. I'm going to, I have things I'm going to build on top of this. But I'm gonna be very careful about them. I think the reversibility, especially like, I want to build things that let people clean up data, like have a conversation with your data to clean it. Because everyone who works with data says they spend 90% of the time on cleaning. They hate it. It's great. I want to solve that problem, but I want there to be an undo function precisely to protect against some of the things that could go wrong.

• 1:09:45Speaker 07

  There's a couple of things I wanted to say. Like, one is while I shared like the excitement, I think that we still have to be careful because, you know, these things are still going to hallucinate and there's not going to, there's, there's still not a clear way to overcome that even using functions. That's number one. And number two is that I kind of want to echo what May have said before. And while I, I do appreciate that, again, this is like a great advancement. It's super cool. I don't know if I would call it a game changer because this has existed, right? Like this, this has happened in various, various frameworks, you know, blockchain guidance, you know, other, other frameworks have guardrails in place.

• 1:10:25Speaker 07

  Yes, this this looks like a very good implementation of this concept of having guardrails and being able to basically force the LLM to do what you want. And yes, it's very, you know, it's it's it's beneficial to all of us that OpenAI that owns these models actually put put like some effort into fine tuning their own model so that this works really well. But I I just I just want to curb the enthusiasm a little bit, right? To just say like, this isn't like, you know, earth shattering and like hasn't, we haven't seen anything like this before.

• 1:10:59Speaker 01

  The same could be said about ChatGPT though, right? When ChatGPT released, folks are like, well, this is just prompting. And this is just sending the same like back and forth, you know, text. And then yet, ChatGPT was the first product that that got to like a hundred million, whatever.

• 1:11:14Speaker 07

  Sure, sure. Let's see how how the adoption goes, right?

• 1:11:17Speaker 01

  The second all of us drop from this space and then start actually coding with this. And then we'll come back here and then we'll see. Go ahead, Simon. And I want to hear from Sean and Steph.

• 1:11:26Speaker 04

  I just want to say, for me, until today, the functions thing was always a hack, right? You could get it working on top of language model, but you had to do some pretty weird prompt engineering and mucking around to to get that to happen. And now that it's part of the core platform, it feels so much, the the friction involved in getting that working feels so much lower. I'm no longer afraid of it, you know, I was kind of cautious of doing this trick in the past because I knew there were so many ways that it might break. Now that I know that OpenAI have fine tuned a model for it, I feel a lot more confident in using it. So I think that makes a big difference.

• 1:11:59Speaker 01

  I want to say thanks for everybody here on stage sharing with us, like exploring with us. Simon and Sean and Riley and Miston and Zanovan, Pharrell and Mayo and Steph and Roy and so many great folks here discussing kind of these latest changes. They're definitely exciting, potentially, if if you're siding with Miston, groundbreaking and earth shattering, which I tend to agree, and this is this was the reason for the space with with Sean. We discussed this in DM and we brought a a great panel of friends to kind of discuss how how big this is. And I think now that we're hitting like an hour and a half, I think we're here. Let's do a fairly quick kind of discussion about. What are we building with this new tool that we have?

• 1:12:39Speaker 01

  So let Steph go and then let's, let's maybe everybody feel free to unmute and kind of have an instruction debate. And then we're kind of close this out because I'm losing my voice. And although this has been fun, it prevents all of us from going and actually playing with this. With these new tools. So Steph, go ahead and then feel free folks to chime in and say, what are we building with this?

• 1:12:59Speaker 06

  I just wanted to give a shout out to Leon and his team. They just launched Garak, which is this tool for a security probing for L L Ms. And it has a, an auto red teaming function. So that might be interesting to check out. But one, one thing that I was thinking about while Simon was talking and I read your blog post on prompt injection. And you know, on one hand, we want to have more people exposing these vulnerabilities and maybe sharing their code for red teaming. And their examples at the same time that is helping people who want to abuse like these models and functions. So it's, it's, it's a tricky one. I'm curious how, how you're thinking about it. And I had a parting question as well, which is what are we going to ask next from OpenAI?

• 1:13:43Speaker 06

  What is missing and what would make our use of the technology of the API better?

• 1:13:49Speaker 04

  This is a really interesting ethical question. The, it's like all aspects of security search people about responsible disclosure of security vulnerabilities. The frustrating thing with prompt injection is we don't have a fix yet. So, you know, normally if you find a sequel injection hole in someone's website, you quietly tell them about it and they patch it. And then everything's fine. With prompt injection, seeing as there is no known way to fix these problems, I kind of feel like it's on us to make sure people understand before they build systems that are vulnerable to them. So that, that's the approach I've been taking is just, just really trying to shake people and say, no, you can't just say, oh, we'll filter it out. It'll be fine. That doesn't work yet. You need to sometimes you need to say, no, I cannot build the feature you're asking me to build because it can't be built securely.

• 1:14:34Speaker 04

  And that's really frustrating. I kind of hate that. Like I, I don't want to be the person who says, there's a security hole. You need to stop. I want to be the person who says, there's a security hole. Here's the fix for it. And now we can move on with our lives. But sadly, we're, we're not at that, that point with it yet.

• 1:14:48Speaker 03

  I, I, I like, I 100% agree. Like I, like it's been frustrating. It's been weird to me. Like it's gone from like suspicious to like frustrating to like, just like curious, like, like this seems to be such a hard problem. Like it's a, like they're, they're like. It like I, I, I think what's going on really is that like to solve this, you kind of had to re-engineer it to this message-based API. Because like they have now reserved tokens. They have tokens that only they know exist that they can insert in as like quote marks. And they can have things like a system message. They can tune it in a way that it actually like, you know, like if you could peer into its brain and see like what circuit is it implementing? It's something that pays attention to the system message in the right way that it like understands the difference between what like

• 1:15:32Speaker 03

  the OpenAI customer told it or instructions and what the user told it. And like, I think that's progress in the right direction of like, it's like a sensible like like it makes sense to me like as an outsider of like, that's how you would like go on addressing this, but it's just like, it's a big change. And so like, I, I think they're moving towards it, but it like speaks just like what a hard problem this is, you know, because like it, it is since May, I think, since like preamble. Uh, where the original like discoveries of it, you know, in, in May. And so when they put it in, you know, responsible disclosure. And so it, it is just, it seems like it's just a deep like issue with how like, you know, the in- instruction tuning or attention or something about this works that, that like it's not trivially fixable. And yeah, I, I think, you know, they, they're they're making peace, you know, like peace by peace progress towards it, I think.

• 1:16:18Speaker 01

  Very interesting to see because we did get an upgrade to three point five model, right? And Simon, you previously mentioned that there's a difference from a system message, how much the, you know, the, the model adheres to it before it's like way better than three point five. It's interesting to now test again this new model and see whether that listens to the system message better. Maybe they implemented some of that in this kind of new update to, to, to this model.

• 1:16:43Speaker 04

  Yeah, they call that steerability. They specifically say that the three new, the, the models are now more steerable than they used to be. And they talk about steerability. That basically means how much, how closely it obeys the system prompt. I would, I'd love to see some examples of that, you know, I've, I've not played around with it yet, but I'd love to see a few examples of prompts that were easily prompt injected with the previous three point five, which are now protected against. But I did note that they, they have not flat out said this is a solved problem. And until they do, I'm going to assume it's not a solved problem because, you know, it's in their interest to, to solve it and then tell people they've solved it.

• 1:17:18Speaker 01

  So I'm sure that you'll research this and folks, Simon has a great blog, basically like a a pencil from Harry Potter that Simon has since twenty thirteen. I I looked at it, Simon, you're very prolific. So I recommend following Simon and and his blog and thoughts. Is another go ahead. And I think after that, we'll do like a round of of last parting thoughts.

• 1:17:36Speaker 08

  Yeah, I just wanted to bring up this because I I I'm not a hundred percent sure of how they've really implemented it behind the scenes, but has anyone here sort of heard of like JSON form or the like one of these projects that was sort of aimed at these at generating structured.

• 1:17:54Speaker 04

  I love that thing. That thing is so clever. Yes, it's wonderful.

• 1:17:59Speaker 08

  Yeah. So I mean, I'm just wondering why. So I I'm I'm speaking from because I I'm not really sure how they've implemented implemented it behind the scenes, but assuming they are not doing it this way, is there a reason why OpenAI is chosen not to? Because JSON form is like, by definition, you cannot, there's no such thing as prompt injection in this case, because you only generate tokens that so you.

• 1:18:25Speaker 04

  Oh, I I disagree on that. I think JSON form has solved the problem of you want it to output JSON and it outputs invalid JSON. It does, but prompt injection in this case is much more about when you summarize a webpage, does the text from that trick it into then making a valid call to a function that does something you don't want to do. But yeah, so so my hunch on JSON form, I wonder if they just haven't had time to implement it yet. It's quite a tricky thing for them to because the way JSON form works, people who haven't seen it is it basically injects extra logic at the next token prediction thing. So it knows that if you're doing JSON, you've just done the curly bracket, the only token that can come next is a single quote, is a double quote.

• 1:19:06Speaker 04

  And then the only things that come after that are not double quotes until you get to the end and circle. So you can force your model to output structured text that matches JSON or YAML or whatever. Super clever. But yeah, I, I, I, my guess is that OpenAI just haven't got around to fully implementing that yet and they'll, they'll get it working at some point.

• 1:19:24Speaker 08

  Right, yeah. So that's just sort of what I was, what I was getting at with the sort of in the, the bullet, I, I mean, I'm reading their read me right, right here. It's like a bulletproof way to generate a structured data. But so what I, I mean, this does not cover the case where you, the, the, the separation between the user's input and the calling of the function. That is always susceptible to prompt injection that, that that's where the security holes are. But assuming that you are forcing it to generate a structured data like JSON, there are these current approaches where, I mean, it's modifying the logits. I mean, I, that's how I assume it's currently working where it modifies how the next token is predicted. But so that, that, that's sort of what I'm getting at is like, so why hasn't OpenAI done that approach?

• 1:20:08Speaker 08

  Or have they? Or yeah, I'm, I'm not 100% sure on that.

• 1:20:12Speaker 01

  Yeah, I think the, the, the thing about these spaces is we, we don't have Logan here or anybody from OpenAI. And when we do, they don't necessarily be able to tell us what they're using. Go ahead, Riley.

• 1:20:23Speaker 03

  I was just curious, I mean, if you'd played with, I think, Grant Sliden, working on like context free grammar stuff that like I linked a while back. I didn't know how that compared to JSON former.

• 1:20:32Speaker 04

  I think it's the same exact trick, just even more, even cooler, because his thing, you can give it any grammar you like. Forget about JSON, it's anything that can be specified as a grammar. He, I think he posted the idea that he'd like to be able to upload a WebAssembly program to OpenAI, to, to the OpenAI API. And say, run this to pick the next token, which I think would be freaking incredible. That's an absolutely brilliant idea.

• 1:20:58Speaker 08

  Yeah, that's cool. That's really cool.

• 1:21:03Speaker 08

  Well, maybe just go around and say like what we think should be built or.

• 1:21:07Speaker 01

  Yeah.

• 1:21:08Speaker 08

  Page from Steph, what do we want OpenAI to ship next? Yeah, this is, you know, some, some people from over there would definitely be listening to this. So here's your chance to do your, your pitch for why, what did you build next? Go ahead, Steph.

• 1:21:23Speaker 06

  I was going to say, I, I would like, love to have knowledge graphs and be able to have better retrieval. So, you know, this idea of like training with retrieval in mind and yeah, like not necessarily relying on vector databases. Like that's, that's something that I would love to see in the future.

• 1:21:44Speaker 00

  Simon?

• 1:21:46Speaker 04

  I want widgets in ChatGPT. I think chat is a terrible interface. I would like to be able to build it like a ChatGPT plugin that could say, now show them a map. Now ask them to pick something from this list of options, things like that. Just let us go beyond just having people type text to us.

• 1:22:02Speaker 00

  Indeed, indeed. Zenova?

• 1:22:04Speaker 10

  I want like a 30B model that's open source from them. I don't mind if it's like a Llama license.

• 1:22:12Speaker 00

  Open source, okay, got it.

• 1:22:13Speaker 10

  Yeah, yeah, yeah, just some, some model that you can mess around with. That'd be nice.

• 1:22:18Speaker 00

  All the class.

• 1:22:18Speaker 10

  Please, guys, 80B, if you can do that.

• 1:22:22Speaker 00

  So like, what, you know, maybe we should train like GPT 2.5, like just, just.

• 1:22:26Speaker 10

  Yeah, yeah.

• 1:22:29Speaker 00

  Zenova or Farl?

• 1:22:32Speaker 08

  I mean, I'm sort of on the open source, coming from hug my face, so you know, so, but, but I, you know, yeah, GPT 2.5, let's go with that.

• 1:22:43Speaker 09

  Yeah, honestly, I'll just, I, I just want 32K GPT 4 with 75% or 80% cost reduction.

• 1:22:56Speaker 00

  Come on, you'll get that next month.

• 1:23:00Speaker 09

  I, I really hope so. I'm, I'm good, I'm good otherwise.

• 1:23:07Speaker 07

  I'm, I'm very curious to see what we'd be able to build with these new functions and specifically combining them with agents. I think the opportunities there are pretty amazing. It's definitely gonna, gonna make life a lot easier. And yeah, of course, if we could have like some more open source models, that'd be awesome.

• 1:23:26Speaker 01

  And Python is gonna have a chance for us to, or at least for folks to, to test this out, right? Roy, you want to tell us about the hackathon?

• 1:23:33Speaker 07

  Oh, sure. We're holding a, our first virtual hackathon on June 19th to the 26th. 100K in prizes. It's gonna be super fun. It's gonna be held in Kumho space. And we invite you all to attend and show us what you got.

• 1:23:50Speaker 00

  Hundred K prizes, that might be the highest I've yet heard for one of these virtual hackathons. That's pretty cool. Mayo and then Alex and I'll go and then I'll have Riley give the last words of Mayo. Go ahead.

• 1:24:02Speaker 11

  Yeah, open source, hundred percent. I mean, we, we could, it, it's good what they've done, but then, you know, my mind just goes to, you know, how can this be applied to open source, right? You know, what they've just poured off of the fine tuning. You know, how can we apply to open source? So, yeah, I'd love for them to, you know, start to, to be open as it says in their name, right?

• 1:24:26Speaker 00

  Yeah, they should just rename at this point.

• 1:24:28Speaker 01

  How about you, Sean?

• 1:24:30Speaker 00

  Oh, well, okay. So, yeah, obviously, you know, you want things for free. They're not going to give it to you. End of story. I, I, I'm just very interested in Frankenmodels. I'm very interested in what Simon has been sketching out this, in this space, which is essentially using them to call still smaller models, but that, that do very specific things, but can do a lot of things. And so I'm interested in essentially just kind of rewriting my developer agent to do that kind of routing and, and explore the possibility of recursion. And when I, when I have more information about that, I'll, I'll report back.

• 1:25:07Speaker 01

  That's great. I want to just, before I, before we go, I want to call out, Tom has a podcast called Data Space, so definitely check it out.

• 1:25:13Speaker 00

  Oh, I'll be posting the recording of this. Yeah. I mean, this is, this is great. Everybody chipped in. Yeah, you have the developer perspective. This is what we want. Yeah. Oh, and we, we also are going to drop our first, I think the first ever interview with George Hots on Tiny Corp. And I'm very excited about that one. He.

• 1:25:31Speaker 01

  So make sure.

• 1:25:32Speaker 00

  Yeah.

• 1:25:32Speaker 01

  Make sure you don't miss that because MD is lagging behind Nvidia and George is working in that space. And potentially some exciting things to come.

• 1:25:41Speaker 00

  He had this email with Lisa Su and went back and forth. It was very dramatic and nothing with George is boring.

• 1:25:46Speaker 01

  Yeah. So Riley, go ahead. What, what would you want from over there?

• 1:25:52Speaker 03

  I think like the thing that that you know about this this whole release is that like it's as cool as functions are like there there I think the real thing that might be more important in the end is is just this march towards lower prices and bigger context windows. I think there's a lot of unexplored stuff to do with big context and there's a lot of lot of possibilities that are opened up when things are just cheaper and you can like put in redundancy checks, you can have like secondary prompts that like check the work of the first prompt, you can like, you know, engineer reliability around the parts that you need. So I mean, it's hard to overstate just like how good it is that just the stuff's becoming cheaper.

• 1:26:25Speaker 01

  And Riley, I I saw that there's a webinar coming up and then you're going to teach advanced prompt engineering. You want to talk about this for a sec?

• 1:26:30Speaker 03

  Oh, yeah, sure. So scale is on July 15th and I think we've already closed applications, unfortunately, but um

• 1:26:37Speaker 01

  Womp womp.

• 1:26:38Speaker 03

  Yeah, on the July 15th, we're having a hackathon and I'll be giving a talk on prompt engineering there. And I last time we did this, I we did like a replay, like, uh, like I did the talk again for a webinar. So well, I expect we'll probably be doing that again.

• 1:26:52Speaker 01

  So I want to thank everyone here coming up to the stage. Here's my request to open the eye. I want the vision API as fast as possible.

• 1:27:00Speaker 03

  Oh, yes.

• 1:27:01Speaker 01

  I've been mouthwatering on the vision API. Today, Mikhail Perakin from Bing confirmed that already like 10% of Bing users get access to the GPT-4 vision API. And I participated in the interview with the founder of Be My Eyes, who are currently the only people in the world who has access to vision API. And that was a great conversation. And I I expect amazing things once that drops for the ability of, you know, a GPT-4 to understand the real world, not to mention how many prompt injections we can do via text. And but that's a conversation for another for another time with Simon and Riley. But I definitely want to thank everyone for coming here. This has probably been the biggest space that I've ran. So thank you, Sean, for for prompting this and everybody who joined.

• 1:27:45Speaker 03

  Hey, yeah.

• 1:27:45Speaker 01

  And and now we're going to have some time to go and and play with these models and new techniques. And hopefully we'll see you guys again. The the last like I'll say that I'll do I'm doing spaces every Thursday. Many of the folks on stage here join. We talk about latest updates. This was an emergency one. And glad to hear the channel is gonna compile this into a podcast. So definitely subscribe to Lakin Spaces. But everybody, thank you for joining and go go play with the new tools we got.

• 1:28:10Speaker 03

  Let's go, Bill.