[AIE Summit Preview #2] The AI Horcrux — Swyx on Cognitive Revolution

This is a special double weekend crosspost of AI podcasts, helping attendees prepare for the AI Engineer Summit https://www.ai.engineer/summit next week. After our first friendly feedswap with the Cognitive Revolution pod https://www.latent.space/p/cogrev-tinystories#details, swyx was invited for a full episode to go over the state of AI Engineering and to preview the AI Engineer Summit Schedule https://www.ai.engineer/summit/schedule, where we share many former CogRev guests as speakers. For those seeking to understand how two top AI podcasts think about major top of mind AI Engineering topics, this should be the perfect place to get up to speed, which will be a preview of many of the conversations taking place during the topic tables sessions on the night of Monday October 9 at the AI Engineer Summit. While you are listening, there are two things you can do to be part of the AI Engineer experience. One, join the AI Engineer Summit Slack https://www.ai.engineer/summit/. Two, take the State of AI Engineering survey https://www.surveymonkey.com/r/aiengineering2023 and help us get to 1000 respondents! Links * AI Engineer Summit https://www.ai.engineer/summit/ (Join livestream and Slack community) * State of AI Engineering Survey https://www.surveymonkey.com/r/aiengineering2023 (please help us fill this out to represent you!) * Cognitive Revolution full episode with Nathan https://www.youtube.com/watch?v=5EzAabIUSE8 * swyx’s ai-notes https://github.com/swyxio/ai-notes/blob/main/Resources/Good%20AI%20Podcasts%20and%20Newsletters.md (featuring Communities in README.md) * We referenced The Eleuther AI Mafia https://news.ycombinator.com/item?id=37368264 * This podcast intro voice was AI Anna again, from our Wondercraft pod https://www.latent.space/p/wondercraft#details! Timestamps * (00:00:49 https://www.youtube.com/watch?v=5EzAabIUSE8&t=49s) AI Nathan’s intro * (00:03:14 https://www.youtube.com/watch?v=5EzAabIUSE8&t=194s) What is an AI engineer? * (00:05:56 https://www.youtube.com/watch?v=5EzAabIUSE8&t=356s) What backgrounds do AI engineers typically have? * (00:17:13 https://www.youtube.com/watch?v=5EzAabIUSE8&t=1033s) Swyx’s Discord AI project * (00:20:41 https://www.youtube.com/watch?v=5EzAabIUSE8&t=1241s) Key tools for AI engineers * (00:23:42 https://www.youtube.com/watch?v=5EzAabIUSE8&t=1422s) HumanLoop, Guardrails, Langchain * (00:27:01 https://www.youtube.com/watch?v=5EzAabIUSE8&t=1621s) Criteria for identifying capable AI engineers when hiring * (00:30:59 https://www.youtube.com/watch?v=5EzAabIUSE8&t=1859s) Skepticism around AI being a fad and doubts about contributing to AI * (00:34:03 https://www.youtube.com/watch?v=5EzAabIUSE8&t=2043s) AI Engineer Conference speaker lineup * (00:41:14 https://www.youtube.com/watch?v=5EzAabIUSE8&t=2474s) AI agents and two years to AGI * (00:46:04 https://www.youtube.com/watch?v=5EzAabIUSE8&t=2764s) Expectations and disagreement around what AI agent capabilities will work soon * (00:50:12 https://www.youtube.com/watch?v=5EzAabIUSE8&t=3012s) Swyx’s OpenAI thesis * (00:53:03 https://www.youtube.com/watch?v=5EzAabIUSE8&t=3183s) AI safety considerations and the role of AI engineers * (00:56:24 https://www.youtube.com/watch?v=5EzAabIUSE8&t=3384s) Disagreement on whether AI will soon be able to generate code pull requests * (01:01:07 https://www.youtube.com/watch?v=5EzAabIUSE8&t=3667s) AI helping non-technical people to code * (01:01:49 https://www.youtube.com/watch?v=5EzAabIUSE8&t=3709s) Multi-modal Chat-GPT and the future implications * (01:03:33 https://www.youtube.com/watch?v=5EzAabIUSE8&t=3813s) Nathan living in the same dorm as Mark Zuckerberg * (01:04:44 https://www.youtube.com/watch?v=5EzAabIUSE8&t=3884s) Competitive dynamics between OpenAI and other AI model developers * (01:05:39 https://www.youtube.com/watch?v=5EzAabIUSE8&t=3939s) Play.ht vs ElevenLabs * (01:09:20 https://www.youtube.com/watch?v=5EzAabIUSE8&t=4160s) The tension between platforms and developers building on top of them * (01:11:40 https://www.youtube.com/watch?v=5EzAabIUSE8&t=4300s) The best thing startups can do to compete with foundation model providers * (01:16:26 https://www.youtube.com/watch?v=5EzAabIUSE8&t=4586s) User identity/authentication services like Login with OpenAI * (01:19:20 https://www.youtube.com/watch?v=5EzAabIUSE8&t=4760s) Google vs the other live players * (01:20:46 https://www.youtube.com/watch?v=5EzAabIUSE8&t=4846s) AI Horcruxes / Pendants * (01:22:05 https://www.youtube.com/watch?v=5EzAabIUSE8&t=4925s) The concept of an AI app bundle for consumers and developers This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

• 0:06Speaker 00

  Ladies and gentlemen, it's the Latent Space Weekend Edition. Woo-hoo! This weekend is a special one, as we are gathering many of our former and upcoming guests and over 10,000 of you for our very first AI Engineer Summit, both in San Francisco and on YouTube. We were interviewed by a few of our fellow AI podcasters about the summit, and figured we would cross-post them over the weekend to help you prepare, even if you can't join us in person. Now, we'll have a very current episode recorded with Nathan LeBenz of Cognitive Revolution, where we discussed how to hire AI engineers, key tools for AI engineers, skepticism around AI being a fad, the AI Engineer Conference speaker lineup, and then an hour of AI podcast inside baseball around the future of AI agents,

• 0:50Speaker 00

  multimodal chat GPT, and AI horcruxes. While you are listening, there are two things you can do to be part of the AI Engineer experience. One, join the AI Engineer Summit Slack. Two, take the State of AI Engineering survey and help us get to 1,000 respondents. Both are linked in the show notes, and we would really love to have you. Now, here's Swyx's conversation on the Cognitive Revolution.

• 1:16Speaker 01

  Swyx, welcome to the Cognitive Revolution!

• 1:20Speaker 02

  Thanks. I've been a longtime listener and very excited to be a first-time caller.

• 1:24Speaker 01

  Well, thank you. Glad to have you here, and I'm also a big fan of your work with the Latent Space Podcast, the newsletter, and also looking forward to what you guys are putting together with the AI Engineer Summit, which is coming up in just a couple days. So I'm excited to get into all of that with you.

• 1:42Speaker 02

  Yeah, happy to dive into that. We did a cross-post, I think, a few months ago, and I really liked your deep dive into the Tiny Stories stuff, and that's the one that we featured on our feed. And so I feel like you have the room to go much more in-depth than us. So I really appreciate the work that you're doing, going like these two-hour things with researchers. It's really impressive.

• 2:03Speaker 01

  Thank you very much. I really appreciate it. I guess, for starters, I thought we'd kind of organize this by taking a little bit of like a broadview survey of your work over the last year or so. As far as I know, you've coined this term, AI engineer. And so I guess I wanted to start off by just kind of asking you like, what is an AI engineer? I'm fascinated in general by these like new AI jobs, right? We've got the prompt engineer and a kind of a few different things have been put forward. Seems like the AI engineer though, might have more staying power than like the prompt engineer. So how do you think about that new emerging role?

• 2:38Speaker 02

  Yeah, I definitely think of prompt engineering as like so 2022 and AI engineering as so 2023. And I feel like this is a controversial take a little bit because everyone should be able to use AI. There, there, there is no restriction on, you know, who does and does not use AI. But I do think that people who are choosing to specialize in the AI stack probably deserve a full-time role that describes what they do. And out of all the possible names that people have proposed, like cognitive engineer, LLM engineer, probably the one that is going to win is AI engineer. And so I'm not so much pointing it as observing that this is a trend that's happening and putting all my chips on, on, on red, as, as they say.

• 3:23Speaker 02

  So the AI engineer is a software engineer specializing in AI and the, and the emerging AI stack. And ML engineer is not an ML researcher, both of which are much more established roles and much more on the sort of research oriented and ML ops side of the fence. It is everything to do with what happens after you have a model in production and maybe with a little bit of fine tuning, which is, as everybody knows, just extra training on top of the, the, the vast amount of pre-training that's already been done. And I think it's basically an emerging category for a few reasons. It's more or less just demand and supply. And, you know, I started life as a finance and economics guy. I was a trader and hedge fund guy for quite a few years before I was a developer.

• 4:06Speaker 02

  And I did, I just think it's just pure demand and supply. You know, there's maybe like 5,000 good LLM engineers in the world, uh, LLM researchers in the world, and you cannot hire them like as, as the average company, average sort of whatever. There's just, there's no way you'll, you'll ever actually be able to build this talent in-house. For better or worse, these models are now available as APIs or as open source. Models, there will be a corresponding rise in demand to for engineers who are capable of putting them to use, even though they don't necessarily train them as a on the on some of the foundation model research basis. I do think that there will be a rise in this category of the AI engineer. There's definitely a rise in startups serving that category. And so I pivoted in space, the newsletter, the podcast, and now the conference,

• 4:48Speaker 02

  all towards serving this persona, which I identify to be one of them.

• 4:52Speaker 01

  Yeah, I think I would count myself among the AI engineers as well. I probably, I guess I probably come to it from a somewhat non-standard background, but maybe you can tell me, like, where do you see the AI engineers coming from? Are they mostly software engineers who've kind of just taken an interest in AI and gone down the rabbit hole, so to speak? Or are there other backgrounds that you see as well?

• 5:15Speaker 02

  Yeah, they're going to be mostly software engineers taking interest and getting deeper in AI. In the post that I wrote on the in-space, the key visual to have in mind is this kind of left to right spectrum of research constrained work versus product and customer constrained work. And or basically, how much, how close are you to machine learning research? Or how close are you to applications of that research? And so on the far left, it's the research scientists and ML researchers. A little bit further along is the ML engineers, data scientists as well. I would class within that category. After you've sort of put those models into production, then you get the AI engineers, which are the emergent category, and then finally, the full stack generalist engineers,

• 5:58Speaker 02

  who are working on just like the last mile of UI and UX to deliver to AI products to people. And so I think just like this emerging category of AI engineers, it's much more people on the right side of the spectrum, the software engineers moving left into AI. So they're learning this, they're suddenly learning what tokenizers are and what embeddings are, what better databases are, why prompting chain of thought makes sense rather than zero-shot generation of stuff. And then it is much more likely to be people from the right moving left than people from the left moving right, meaning the ML engineers and data scientists moving right into applications, even though you do get those. So for example, Raza Habib, my most recent guest from Human Loop,

• 6:40Speaker 02

  has a PhD in probabilistic programming with DPO networks, but he's working on an LLM off solution just because he sees that. A lot more people can be served that way. So there are people with all sorts of backgrounds, but I do think it's primarily software engineers.

• 6:54Speaker 01

  We've had a somewhat of a parallel trajectory in the podcasting game over the last few months. Roz was also an early guest because I've been a customer of Human Loop since early this year and definitely have got a lot of value from the platform. Now starting to use it also to, I think, really getting to the part of the vision that they probably had in mind early on, which is the really accessible fine tuning loop that they've enabled. And we just did an episode kind of going down that rabbit hole a little bit, talking about like, I think my number one insight there was using GPT-4 reasoning as part of the fine tuning data set for 3.5 as a way to really, in my experience, dramatically improve the results.

• 7:39Speaker 01

  And that was a, something that Human Loop made much, much easier than it otherwise would have been. It's funny because I had already been a customer for months, but I hadn't used all the latest stuff. And then I was like, kind of thinking to myself, like, how am I going to kind of code up my own loop to do this? And then thought, well, I should check Human Loop and see what all the latest features are. And sure enough, they had done a really nice job of anticipating the need and really streamlining that process. So what do you think are the key skills for the AI engineer? If I were to come to you and say, okay, I've got a background in software development. You know, I've played around with ChatGPT and, you know, almost everybody's like, at least used Copilot or something like that at this point.

• 8:22Speaker 01

  Where do I go from there to make myself employable as an AI engineer? What are the employers, the app developers most need?

• 8:32Speaker 02

  Yeah, totally. I think that's where most people start, which is use the off the shelf models that have the most adoption and that is going to be Copilot and ChatGPT. I do think I have been mapping this out. So if you go to the about page on Lit and Space, we also have an emerging email course that we call Lit and Space University to trigger the Louisiana State University fans.

• 8:55Speaker 02

  And and it's kind of mapping out the the curriculum of what I expect to be the baseline competencies of an AI engineer. Like, just write the job title, like, what what what do you expect people to do? And I think I want to ground this in a very practical manner, because it's a demand and supply issue. Companies are trying to hire these people and people are trying to learn to be useful to companies so that they can uh be part of the AI movement, uh but then also apply their their engineering uh knowledge towards building use useful and interesting things. Um and so uh I'll just kind of list through some of them, but we can go into more details uh as needed. The high level themes are AI UX, AI coding tools, LLM tooling, AI infra and infra's hardware, and that includes fine tuning as well.

• 9:39Speaker 02

  Uh and then finally, the most speculative would be AI agents, which everyone has been talking about, but doesn't have that much practical use. Um so I I think there's definitely a trade off between what people actually use at work versus what people just like to talk about on Twitter and star on GitHub uh and promise AGI without actually doing it. Um so maybe that might be a hot take there. Um in my course, uh so we have a seven day free sort of email course type of thing, and and that's meant to be like, I'm never going to make money from that. That's just meant to be like, uh like, start here if you want to learn AI engineering, your software engineer has never dealt with any of these APIs. Um so day one would be just try the GPT API for the first time, uh and just a a lot of a lot of software engineers just haven't bothered.

• 10:20Speaker 02

  And I think you'll be surprised at how much controls, uh how much how many options there are available. Um thinking about running into the context limit for the first time, which, you know, chat GPT and Copilot kind of hide away for you. Um and just like, get familiar with all those basics. I think it's very important. Day two will be prompt tooling and memory, and that's where you get into like the vector databases and the lang chains and all that. And those are just kind of frameworks that will hopefully help the developments. And and I do encourage people to kind of build some extractions of these themselves before using something off the shelf, you know, so you get a better understanding. Um day three is code generation, day four is image generation, day five is speech to text, day six, fine tuning and running open source models, and then day seven is building an agent. And I think once you've finished that little sampler course or like a tour,

• 11:05Speaker 02

  you have the base capabilities that I expect every AI engineer should have such that whenever any PM or any CEO comes to you with like, hey, I have an AI products that I would like to build. Is this possible? You can figure out if you can build it or you can just tell them like, hey, it's not possible for X, Y, and Z reasons. You know, that that way that you'll be useful as an AI engineer.

• 11:23Speaker 01

  So that's really interesting because this is like, you know, the old XKCD. Somebody's wrong on the internet, right? I almost have my own version of that with people posting G P T four can't do X things when in fact like it definitely can. And they're either in many cases, like using the wrong model or prompting it wrong or doing like a bunch of things that are wrong. So I actually find that people often come to wrong conclusions about what AI can and can't do to it. Obviously, it's a fast moving frontier. So like your value of knowledge decays pretty quickly. I mean, seven days definitely seems like enough, especially if you really go for it to, and this is like a remarkable thing, right?

• 12:06Speaker 01

  The part of what I love about the AI space in general is just that you can kind of jump right into the frontier. I think that's just extremely cool and super fun. And I would think most technologists would agree with that. So I do think like a seven day, you know, intensive is a pretty reasonable amount of time to like get mostly up to speed. I guess if there was one thing I would wonder about, it would be would folks after that kind of intensive have a well-developed sense for what really is or isn't possible? And like, how would you coach people maybe on like continuing to refine their sense of what is and isn't possible? Because I see way too many people just giving up too soon.

• 12:47Speaker 02

  Yeah. Hmm. This is interesting. So first of all, I actually wouldn't describe the course as intensive. It's just an email course. It's meant to take an hour a day. It's probably, and then we leave breadcrumbs to explore with a lot more details and suggestions for side projects and stuff. I think that if you want to be up to speed, there's a vast gulf between, hey, cover the fundamentals and be be cutting edge because the people who are cutting edge are the people who are. Hiding in discords and on Twitter and talking about very niche jargon-y stuff that you won't even see for a few months out. So I don't really know if like that's a realistic goal for most people. I think most people want to make sure they cover the fundamentals and then be able to build most projects that you've seen out there that make money.

• 13:30Speaker 02

  And ultimately, I think that's what most people want. To be cutting edge means you have to go down a lot of rabbit holes. And I'm not exactly sure if I would recommend that for most people because that is a full-time job in and of itself. Which is why, by the way, I call it an AI engineer because I, you know, I do think that this deserves a separate category or job title because it's, this is a full-time job keeping up on things. But I do have a recommendation, which is I have a list of Twitter people to follow. I have a list of Discord communities that I watch. And obviously a list of podcasts and newsletters, which I recommend, which you're definitely on. So all of those are on my GitHub. I have an AI notes GitHub where I, where I linked to all these lists of Twitter, YouTube,

• 14:14Speaker 02

  podcasts and newsletter people. By the way, we're also running a survey of who people listen to. So a little bit of a market share competition going on. And if you, if you want to Google state of AI engineering survey, you can get a sense of who people are listening to. And, and I think that's important to understand, like, like what drives people's attention towards individual projects. That is something that I should keep note off as well.

• 14:41Speaker 01

  Yeah, interesting. I'll have to definitely subscribe to your Twitter list, which I did not know that you had out there. But I do actually get most of my stuff, I think, first from Twitter still, and then definitely have like a ton of discords that I've joined over time as well.

• 14:58Speaker 02

  So I have this little project where it basically scans discords on a daily basis and then summarizes them into an email. And I'm wondering if I should just release that as like a thing that people can subscribe for. Because I think it'll be kind of popular, but also very noisy because people discuss, discuss all sorts of things in Discord. And like, it might not actually make sense.

• 15:18Speaker 01

  Well, I mean, that's kind of the AI challenge in general, right? Is like, how reliable can we make these things? I definitely think if you could get it to work well, it would be valuable. I've got to be in, I don't know, 50 different discords at this point. And it is super noisy in there. And I honestly, kind of, I join them. I'll often kind of scan around, see what's happening, you know, see what discussion is there. And then often I don't really come back very much. So, good God, the number of Discord notifications have become unmanageable. So that just kind of gets tuned out. So I actually wonder if there would be a way to intercept the notifications

• 15:59Speaker 01

  or kind of use that as a particular signal as opposed to like, you know, having to kind of grab and summarize everything. I do think if it worked well and it could kind of create an auto GPT style digest of the news from all these different projects of interest, I definitely think that would be really interesting.

• 16:18Speaker 02

  I might release that as a side project. I don't know how popular it would be, but I think it definitely solves a pain point for myself. It's funny because I was, one of our unreleased episodes that we just recorded is Jeremy Howard from fast.ai. And in his recent post, he had a post recently on sort of single one-shot learning or like sudden drops in the loss curves, trying to figure out like the origins of that. And he mentioned the alignment lab discord. And I always just find these discords of very elite LM people just to emerge from nowhere. Like news research is another one. I'll just throw these names out there for people who want to find it. They're all on my list, so you can go check it out. But like, how do you find them as they're emerging?

• 16:58Speaker 02

  Because like, you know, most people found out about Luther AI after Luther was forming and the core community was already established. And now it's kind of, you know, people have departed Luther into what I think called the sort of Luther mafia recently. Luther is still going, obviously. And I just didn't find like, you want to join these communities when they're early and everyone's still trying to figure it out and building something interesting. And Jeremy said, like, most of this stuff is also happening in private channels, not public channels. So it's just an extra. Wall garden inside of a wall garden. And just joining Discord isn't enough. You have to invest enough to get access into the private channels. This is what I'm saying.

• 17:37Speaker 01

  I need to get myself invited to some more private channels.

• 17:40Speaker 02

  For what it's worth, the bar for the in-space Discord is you show up, you introduce yourself, you're invited in.

• 17:46Speaker 01

  Well, I appreciate your inclusive approach. A few more just kind of general state of the field questions, then I want to kind of shift toward the event that you have coming up soon and talk a little bit about kind of where things are going as well. The tooling for AI engineers, if I understood you correctly earlier, it seems like the pattern is mostly identifying the best available tools and then figuring out how to make those work together as opposed to, you know, obviously you kind of distinguish this from like deep ML research, but even from like self-hosting, it seems like most of these things are kind

• 18:25Speaker 01

  of services that people are figuring out how to put in concert and not too often spinning up their own services at this point. Is that accurate?

• 18:35Speaker 02

  Sure, even though they absolutely can, right? This is the power of an engineer that you can decide to build versus buy at every point in time. And that's always a fun topic of conversation whenever you need to build versus buy.

• 18:47Speaker 01

  When you think about that build versus buy, like that's such a common debate, obviously, and it's like a tricky one for a lot of folks because the tension that I experience is like, I want to be early to market, if not first to market, right? Like at my company, Waymark, it's like, I want to be the, we have this video maker. I want to be the first video maker that does like A, B, and C, X, Y, and Z for you. And to do that, you know, you kind of have to build more than you would have to build if you just waited a little while because, you know, if you have a little patience, then sure enough, you know, the market kind of provides solutions for a lot of these things that you need. So I think it's obviously super contextual, but I wonder how, if you have like any sort

• 19:31Speaker 01

  of framework or kind of general, you know, high level view of how people should be thinking about build versus buy. Especially given how quickly things are coming online for us to be able to buy.

• 19:43Speaker 02

  I generally have a pretty vendor friendly version of this because I have been a developer relations person for five years. And but I'm also very sympathetic to the choose boring technology, try to limit your dependencies as much as possible mindset. And so basically, I typically, what I often say is buy first. And then the moment you understand your domain well enough, be in a position to be able to build yourself. And this way you get the benefits of people setting you on the right path with best practices that they've learned from everyone else. But you're ready at this point to understand that this field is so immature. You may have to rewrite, you may have to rip out something that you started not liking.

• 20:28Speaker 02

  So this was the topic of a lot of discussion this summer with Langchain, where a lot of people did had this exact journey, right? Like they started out building out some Langchain and they ran into a lot of problems that maybe Langchain wasn't well designed for. And then they ripped out Langchain and they said Langchain is crap. And I think that's kind of unfair to Langchain just because they're evolving as well. I mean, the company is less than a year old, you know? So I think it's fine, like everyone, like if you're, if you chose to be in this field, you are choosing to be on the cutting edge. And sometimes the cutting edge cuts you and that's what they do. That's the nature of these things. I would basically buy off the shelf first and understand what people are building out there. Because you plug into existing communities of people who have encountered these problems for far longer and have dealt with them. And thought with them much deeper than you have.

• 21:12Speaker 02

  And then anything that you don't understand or don't, you know, don't appreciate after working with them for a few weeks. Then you can rip them out and build your own, right? But like, I think at this point, we know we have on our show notes like human loop, guardrails and Langchain.

• 21:29Speaker 02

  These are fairly well established frameworks among the community that you should at least understand as table six for what AI engineering is today. Just because there's been a few hundred thousand people ahead of you mapping all these things up.

• 21:40Speaker 01

  What other tools would you put on that list? I mean, we talked a little bit about human loop as kind of, you know, just to set the level there, a, at least the way I think about it, first is kind of a playground in which to develop prompts. So my workflow is I go there and I workshop a prompt until I get it working reasonably well. Then the reason I do it there as opposed to anywhere else is because I can hit save and then immediately I have an API that I can call that basically puts all the prompt management stuff on the human loop platform so that all I have to do as a developer from outside is just provide a couple of variables, you know, one or more, whatever I set up. And that makes it super easy to develop against.

• 22:25Speaker 01

  Then I can also update my prompt without having to do code changes, which is quite nice. And they log everything for me and allow me to, you know, kind of come in later and post-process it, evaluate it, export certain subsets of data, which I might use to, you know, power fine tuning based on like the most successful data points, whatever. That's really cool. We've had Shreya from Guardrails on the show a while back as well. You want to give your take on Guardrails and how that's used?

• 22:53Speaker 02

  Yeah. Guardrails is, I would say, in the output validation side of the fence. And, you know, as the name implies, it puts sort of safety rails on what generative AI does, specifically generated text. And I would say it is competing somewhat with the other orchestration frameworks because everyone, there's only one room for the sort of LLM interface layer and everyone's fighting for it. Human Loop wants to be in that interface so that it can track everything. And so do all the other LLM ops and prompts ops companies is what they'd be calling. In my podcast with Inraza, I called it the foundation model ops companies, but they all effectively have the same thing, right?

• 23:36Speaker 02

  Like we'll track and version manage your prompts and then we'll eval them and then we'll like, you know, we'll track your costs too and track your latency and whatever, right? They all have the same roadmap because it's fundamentally an ops product. And Guardrails and Chain, slightly different. They're much more in the application framework side of the fence with an open source first and monetization second. And guardrails would do things like validate that the SQL generated is correct if you're if you're trying to use AI to generate SQL. They used to have sort of output validation, uh, Langchain also has this, by the way, output validation for valid JSON if you're if you're sort of outputting valid JSON. And that went away when obviously OpenAI came out with its own version of that.

• 24:16Speaker 02

  And so I think the roadmaps with these sort of orchestration or application, now them specific frameworks will evolve. But guardrails is much more focused on safety and production readiness, uh, let's say. And Langchain much more on the orchestration, even though they collide on some features, because all of them are built on each other's features.

• 24:35Speaker 01

  Yeah, that's super interesting. Let's come back to that in a second and get a little bit more into the competitive dynamics. But before we do, and this, um, you know, starts to give you an opportunity to talk about the upcoming AI engineer summit as well. But, you know, that and and more broadly, how would you suggest that people go about looking for AI engineers today? So, you know, whether you're whatever kind of company you you might be, right? Software application company or, you know, I would say honestly, any company that has kind of a lot of operational overhead could probably make great use of an AI engineer, even if they're not putting a, you know, a public facing or customer facing application out there necessarily.

• 25:18Speaker 01

  What do you look for if you're hiring that skill set? You know, it's so new, people are like lost, they they, you know, people are kind of just looking for somebody to tell them what to do. But obviously that creates a lot of opportunities for them to hire somebody that doesn't really know, you know, what to do. So if you don't have the skill set yourself, like where do you go to look for it? How do you evaluate it? And of course, you know, the the upcoming summit, you know, could be one of of those places.

• 25:40Speaker 02

  I honestly, I really should set up a job board. Um, I've been chatting with a couple of companies on like, hey, I need someone to just set up a job board. You know, I don't even have to make money on these things. Um, I just want to make sure that people who are looking for others, whether it's, you know, you're hiring or you're wanting to be hired. Uh, you have a common place to match. Uh, right now it's on Twitter, you know, anyone like actively talking in shipping projects is, is, uh, fair game. Um, within Discord, it's like in the latent space Discord, we have a hiring channel and people can, people post jobs there. And I know people have gotten hired off of there as well. I think it's just like regular channels and communities, um, posting in like the monthly sort of who is hiring job posting boards on, on Hacker News. That's, that's typically the way I would recommend these things.

• 26:24Speaker 02

  And obviously, uh, coming to AI engineering conferences and talking with people who are also attending, uh, that, that tends to be a very high signal filter for, uh, who's, who's very engaged. Um, and I would say like, you know, especially if you yourself are not an AI engineer and you want to hire someone who is an AI engineer. The kind of core competencies that we listed out earlier in this podcast, you know, the, uh, that we have on the about page that we have on the latent space university curriculum, uh, is what I would expect the bare minimum for people to understand, um, for working as an AI engineer. And I don't think it stops there, um, because I do think that a core requirement that you cannot really test for, you just have to sort of observe in an AI engineer is that they're kind of entrepreneurial.

• 27:05Speaker 02

  They, they're, they're kind of, they're comfortable with things that are not that well defined, uh, because prompt engineering is not that well defined. Uh, because the, the model landscape shifts every single month with the release of a new model or whatever. Uh, they have to be able to be on the ball and proactive and not just, hey, we are, you know, I'm a, I'm a sort of data scientist and, uh, and now I do L L M's now and, and, and now I'm an AI engineer. Um, and I don't think that that is the kind of person that will put your business. At the cutting edge of what you can do with generative AI, they have to be a little bit entrepreneurial. They have to proactively come to you with ideas instead of saying, hey, well, you know,

• 27:49Speaker 02

  throw an L L M layer on top of the existing app, which is totally fine. But like, I do think that especially for me, and especially like the the high people hiring that I talk to, they want someone who is a little bit entrepreneurial. So people who can just kind of like shift their own projects and and and get the get attention for them or like use new techniques and put them in practice. And I I do have some in in sort of the rise of the AI engineer blog post. There's some sort of role models that I do highlight, you know, that I think the entire Versal team has been doing an awesome job of showing what it is like to be entrepreneurial, right? Even though they're not a foundation model team, they can put these things to good use and in a way that is appealing to enough people that they get millions of users on their free projects.

• 28:33Speaker 02

  And I think that is something that every company should have. And that is very much the ethos of the kind of AI engineer that I want to encourage.

• 28:42Speaker 01

  Why are not all software engineers jumping at the chance to be AI engineers? It seems like there's something weird going on there. If somebody says to me today, oh, these AIs, they'll never be really useful. They're not that useful. There's no good use cases, whatever. The first thing I would say is code, okay? Like you can, you know, question anything, but you really cannot question the utility of a copilot. And you definitely cannot question the utility of coding with G P T four. It is, you know, for me, I would say a multiple speed up safely. And I maybe am not like the very best software developer in the absence of G P T four. But like, you know, I've got plenty of experience and definitely can make stuff work. Given how.

• 29:28Speaker 01

  Powerful it is for developers in their existing workflows and you know and the fact that it's like kind of been integrated into their tools before most other tool sets. What am I missing? Why is it not the case that like everybody is kind of gravitating this way?

• 29:46Speaker 02

  Yeah, I do think there's some baseline skepticism about whether or not this is a fad. And a lot of people have been burned by effectively crypto. It's a very fair thing to have some skepticism about anything new. It's a very fair thing to have some self doubt over your credentials. Like, do you need a PhD to make progress in this field? And very much like why I'm trying to promote the AI engineer is to encourage the idea that no, you actually don't need credentials to make progress in this field because everyone is effectively uncredited. When transformers themselves are six years old and when GPT-3 itself is three years old. And you know, many of the techniques and the companies that we talk about in this space are all less than one year old and all making tremendous progress in AI. And by the way, you know, one of the biggest promoters of the AI engineer concept is Andrej Karpathy,

• 30:31Speaker 02

  who's who very kindly supported the concept with the idea that there should be more AI engineers than ML engineers. And you'll be very successful without training anything. So I think there's there's some mix of skepticism about AI being a fad. And then there's some mix of self skepticism about whether they can contribute in AI. And then there's always like fundamental misalignments or some fundamental doubts about the stochastic parrots argument, right? Whether or not just multiplying a bunch of matrices can actually approach anything regarding simulated intelligence, which we can always talk about in sort of like those kind of dorm room hallway style conversations like what is consciousness? What is what is intelligence? But I mean, you and I know, like especially we're recording with with AI.

• 31:14Speaker 02

  Like, these things are actual productivity enhancers. And they've gone from single line autocomplete to function autocomplete to code based, entire code based generation. They can help you as tools of thought, you know, with a human in the loop or they can function without humans in the loop, all of which we're going to see as part of the conference, right? And that kind of leads us, the more and more you have this sort of AI in the driving seat, that leads you more and more towards autonomous agents. There's a wide spectrum of AI as sort of autocomplete, all the way to AI as autonomous agents. And I do think you just kind of depict your line of like where you think usefulness currently is and and understand that that will probably move over time. We have about six orders of magnitude more improvements in terms of scaling according to Nat Friedman until the end of the decade.

• 31:58Speaker 02

  So take whatever we have today and project forward.

• 32:01Speaker 01

  Yeah, it's gonna get wild. Kind of pretty hard to imagine what pops out the other end of a million fold more compute than went into GPT-4. So tell me about the speaker lineup. I went through, there's twenty eight accounted speakers shown on the website. I was proud to have had seven on the podcast as former guest, which was pretty cool. And it seems like you're basically kind of creating a lineup that covers kind of all the inputs, basically, that an AI engineer would need, right? From foundation models to these kind of frameworks, to the quality control measures. And then there's like a number of folks from agent startups as well. Run that down and particularly on agents, I'd love to hear your perspective.

• 32:47Speaker 01

  I wonder, is that something that you see as being like the next thing that's gonna come on for the AI engineer to tap into? Or is that kind of a different sort of thing where it's like those more are the AI engineers that are building the coolest new stuff on those lower levels?

• 33:02Speaker 02

  I basically pulled together from my network all the top speakers that I thought would build the interesting things for AI engineers, specifically for more more technical audience, because there's other conferences happening all, you know, all this fall in San Francisco. But they're all very high in general and they spend a lot of time talking about policy, safety, regulation, copyright and all those things. But for builders, I think there wasn't a builder specific conference until mine came along. And then obviously OpenAI had to top it with Dev Day, which we can also talk about, which I think you and I are also going, which is I'm very excited. We should do a live podcast on Dev Day. Yeah, we have people from AutoGPT is our presenting sponsor with OpenAI, with Microsoft speaking,

• 33:45Speaker 02

  Notion speaking, Amazon speaking and GitHub. Like all these are top names, I think, in the AI engineering field. But I also wanted to balance it out with names that you've never heard of. Like Mithun Honsar from the sort of Rust OLM community is speaking. And we but we also, you know, have like the first. I I think one of the world's first demos of Adept, one of the world's first demos of new computer, and just trying to get a mix of like projects that you've never seen before. So like Lindy is a very, uh, very, very hyped agent project that they've never done a public talk here. And and so so I think we're we're trying to be the stage where people kind of launch these new features, new products, projects for the first time to provoke some thought, and then balance it out with people who talk more about sort of active production issues.

• 34:30Speaker 02

  So I think one perception of AI is it's all greenfield. It's all, you know, twenty something year olds building toy projects is not actually in serious work. And so so I want to balance it out with Eugene from Amazon, who is running Amazon Books and with with language models, you know, and and Abi Arian, who's who just finished the O'Reilly book on L L M ops in production. People like that who are actually implementing AI in like large scale production systems. Hex and Prefect are also speaking with us on on how to pivot an existing non-AI company into an AI company. And I think both of them have done fantastic job of that. So I'm kind of curating this list. I do have very large gaps that I'm very conscious of, right?

• 35:12Speaker 02

  So I don't have image generation people, and I don't have that much infrastructure, these tens and replicates of the world, and I would like to feature them next year. I just had a limited schedule of a two day single track conference. The the underlying thing about behind this conference is I'm I'm trying to basically create the ICML for engineers. So ICML, the International Conference of Machine Learning, is like the ultimate, like, if you go to one conference a year, that's the conference you go to if you're a machine learning researcher. There's no equivalent for the engineer. And so what we're trying to do is like try to offer that. And so this thing was going to grow over time. I I kind of see this as like a ten year commitment towards building the ultimate survey of the field in in any given point in time for engineers.

• 35:54Speaker 01

  I'm jealous that you have a demo from Flo and Lindy among a bunch of other cool stuff as well. I had him on as an early guest and continue to follow their, you know, progress from little hints they give out in the public. But I still have not been able to get into that thing even as a an alpha user. So be very, very curious to see what they're about to show off.

• 36:16Speaker 02

  I I'm very excited to. And honestly, you know, I I think you you're doing you're doing a great job with with your reports as well. And I'd love to have you as a as a speaker in in. In the next iteration, but yeah, I maybe I'll talk, I'll spend a bit of time talking about agents, right? Which is a very, very hot topic. So I would say, you know, maybe the surprise now is that AutoGPT is obviously now a company. Being one of the fastest going open source projects ever. I think, I think there's a ton of interest in, in what they're doing and Torin will be speaking. On our stage, I think for his first time ever as a conference speaker. So I think that'll be super exciting. I do think that there's a range of agent type projects, right? So the open source agents definitely tend to be sort of single use.

• 37:00Speaker 02

  I would say like their goal is really trying to optimize for the, what's the one most impressive thing that you can do? In a single, you know, short video recording or Twitter screenshot. And I do think like that, that really is pushing the boundaries in terms of like what is possible. And then obviously being open source, people can actually go through the source code and, and copy ideas off of that. And I think a lot of people have done that with both AutoGPT and Baby AI. And I think the, the closed source agent type companies like the Adets, like the, like the Lindys. Like the new computers and, and you know, the others that are the people working on here. They tend to work on very much more mundane, but daily use type of use cases.

• 37:41Speaker 02

  Because they're trying to work towards like, what would you subscribe for like a $20 a month, $100 a month subscription? I think both approaches are valid and I do think that you need to have representation for, from both. The amount of human intervention is something that people are trying to get a grip of. Because the way that AutoGPT does it is they basically ask you for a confirmation step before they do anything. And like, that's fine, but that's not autonomy. That's just like, you know, assisted prompting, whatever. What you want is to fire and forget. Like that, that is where ultimately these things have to go. You know, whereas in San Francisco, cruises and Waymos are now common modes of transportation. And literally I just get in a car, I never talk to anyone and it just brings me to a destination. That's what I want.

• 38:25Speaker 02

  And we might be permanently for a while, at least five to 10 years away from full self-driving in agents. Which is where self-driving was like 10 years ago. You know, so I do think like this is one of those the most speculative areas. But I have seen some of these demos live, you know, because I'm friends with most of the speakers that they're featuring. And I will say like they are useful today, so they may look trivial now, but there's active research going on towards making them more and more substantial.

• 38:54Speaker 01

  I'd love to hear more about specifically like what use you have been able to get out of any of the agents. I've kind of played around with everything I've been able to get my hands on as well. And I recently called them broadly as a class like still just for fun.

• 39:11Speaker 02

  Yes.

• 39:12Speaker 01

  Although I don't think it will stay that way. In fact, if anything, I would say maybe I would put my timeline to AI agents, you know, kind of crossing whatever chasm they need to cross. Probably short, if I understood you correctly, probably shorter than, you know, multiple years. I think I'd probably put my money more on like, I guess, kind of the anthropic timeline, if you will. You know, Dario has said in a couple of interviews that, you know, they're gonna.

• 39:39Speaker 02

  Two years to AGI, come on.

• 39:41Speaker 01

  Yeah, something along those lines. I mean, I can't confidently rule anything out at this point. I'm not confident that that will happen. It does seem like probably the biggest question in the whole space right now is like, does the current paradigm get to the ability to do sort of insightful work of the sort that currently humans are kind of the only things that are able to do. You could come short of that, though, and still have like very useful agents, right? That can like decompose, you know, somewhat complicated tasks and reliably execute on them. I mean, there's, you know, a lot of different places on this spectrum of AI progress we could zoom in on.

• 40:22Speaker 01

  But I'd love to hear your, your take on both, why you think AGI is farther off than that? And, you know, then coming in a little bit toward AI agents, or take them whatever you want, but also on the AI agent side. I would be thinking like a year from now, they probably will be working well and like reasonably commonplace. But it sounds like you're maybe not expecting it to happen that fast.

• 40:45Speaker 02

  I think this will happen in gradations. And this is one, one maybe difference between our two podcasts. I tend to not discuss AGI. I tend to not discuss timelines, even though obviously there's some implicit assumption of them in everything that we do. Just because like it's so hard to predict the future and like it's not falsifiable in any way. So it's just, it's just a fun dinner topic conversation. I, so I will say like, there are some categories which I'm more interested in than others, right? Like, so the original founding prompt of AutoGPT was I want to increase my network. That kind of category of agents, not interested in. Like that's, that's too general for me. That's, that's too AGI bro sciency. But like, uh, they're very limited scoped agents, which I think are useful today. I, I have a piece on agents, which is relatively popular that I wrote in April.

• 41:30Speaker 02

  And I, and I said, actually, the most useful agents that I use today is one that has no LLMs in it at all. And that's the Savvy Cal or Callan Lee agents. Right? Because I, I, I send you a link and then like you schedule a meeting with me on, at your own convenience. And if you need to reschedule or cancel, uh, it just happens on your side of the fence. And it just kind of happens autonomously without me. And that is kind of the experience I just want for all my agents. And the fact that we don't have that with LM-enabled agents is a problem. And we, we're, we're gonna sort of slowly emerge to get there. The second form of agents, which I think have kind of already proven, uh, proven themselves relatively successful is code interpreter

• 42:11Speaker 02

  or advanced data analysis, as it is now known, because it can generate code, run that code, and then use the output of the code to decide whether or not it needs to fix that code or to stop. And that is the beginnings of a loop that, uh, is basically required for agents to have some level of autonomy in decision-making. And, um, if you sort of break that, break that down even further, you need, uh, some ability for planning and prioritization. You need a broader set of tools. You need memories to go with it and then you need ways to interact with the outside world, whether it's sort of generating text or manipulating some kind of UI. I think those are sort of like the agents research. For those who are interested, definitely read Lilian Wang from OpenAI.

• 42:54Speaker 02

  Her blog on agents, I think, is one of the most comprehensive survey overviews of the research in agents to date. So I think some categories of those will emerge, right? And so one of the people that I forgot to mention, Deddy from Codium AI, that's an Israeli company that raised an $11 billion seed for building the coding agents. They only focus on test generation as an agent. To have an agent sort of independently running around generating tests in your codebase and then for you to extract, reject them. I think that's a relatively small problem that I, yeah, I'd be perfectly happy to agree with you. It will probably be useful and commonplace in a year. But things where, which require a lot more self-driving and a lot more, have a lot more degrees of freedom to fail.

• 43:36Speaker 02

  Those things, I think, will be waiting for them for quite a while. My last observation is, I think you'll be surprised what Linde and NoShin have to show. I can't say more than that.

• 43:47Speaker 01

  Yeah, I can't wait. I feel like you have more insider information than I do and yet I want to take the bull side of a bet perhaps. Like, we need to maybe, you know, think it through to really refine the decision criteria. But I'm trying to zero in on what you think would fail that I think would work at a certain point in the future.

• 44:09Speaker 02

  Yeah, what do you think I would think would fail? And then maybe we can kind of discuss there. Because I'm positive, I just, I just, I'm much more interested in the engineering challenges than timelines.

• 44:19Speaker 01

  I do think people go well past the point of usefulness on their timeline discussions and their, you know, P dooms, if you will. Whenever anybody asks me about my P doom, I always say somewhere between 5 and 95%. And my quick, you know, add-on to that is like, and I'm not sure it's that much worth trying to narrow it down further because 5% is enough to be very concerned about it in my mind. And like 5%, you know, if it is 95%, then the 5% chance of, you know, surviving is like worth fighting for. So kind of anything in there, you know, to me is like likely enough to be a problem and.

• 44:57Speaker 02

  Yeah, 5% makes you a doomer, basically.

• 45:00Speaker 01

  Well, the real doomers would go much higher. I don't know that they would have me in their camp as a bona fide doomer with a 5% number. Because you're multiplying by infinity, right? So 5% rounds to 1. Well, it certainly does make it for me like the issue of our time. Again, I don't try to zero in on a specific number much more than that. And I'd say same thing for timelines. Like, it seems like things could get really crazy in the next two to three years. It also seems still pretty plausible that we just kind of hit some sort of top out where it's like, hey, this paradigm kind of closes in on like expert performance on a lot of things,

• 45:41Speaker 01

  but never really achieves that sort of eureka insight capability. And we kind of stay there for a while until something else happens. I definitely think both of those are true, or not true, but like plausibly true. Going back to the question of like, you know, where our expectations might differ. I mean, I guess the stuff that Lindy has shown would be like the kind of thing that I do expect to work, you know, in the not too distant future. And to, you know, describe those a little bit, it's kind of like text to automation is a lot of what the demos are. Like Flo will post something where he'll have like a pretty simple prompt. It's like, hey, every time somebody emails me, you know, from this domain,

• 46:24Speaker 01

  check this other thing, you know, draft me a response, put that in a calendar invite, you know, send it here, whatever, right? And he kind of just says all this stuff. And then the platform, you know, just judging from the screenshot that he posts, interprets that, sets up this kind of automation workflow. And in theory, then it's like set up, ready to go, almost like you have a Zapier Zap that you've just conjured, you know, with like two sentences. Some of them that he's shown have been reasonably complicated. And, you know, I assume that it doesn't always work, or he'd probably have launched it by now. But I guess I think that that probably will start to work pretty well in the not that distant future.

• 47:09Speaker 01

  And I guess maybe to refine it a little bit more, it seems like once you can fine tune GPT-4.

• 47:17Speaker 01

  Probably should be able to make a lot of those things happen, right? If you can just kind of document the reasoning, the breakdown, the planning. I mean, GPT four is currently doing everything. If you could really just zero it in on like the decomposition and scaffolding of kind of mundane, even if like somewhat complicated tasks, feels like that would be enough to me to get it to work pretty well. I don't know. What do you think about that?

• 47:42Speaker 02

  I think it could work well. We just don't really know. We don't really have benchmarks for planning and prioritization yet. And I I've had actually debates with OpenAI people who are researching this actively. Part of my GPT four point five thesis is that long inference is kind of the next frontier of OpenAI. Like, what can you do if you gave it a year to inference instead of a few nanoseconds or milliseconds? Anyway, so I would say that you're exactly right in your your mental model of where what Lindy does and how that interacts or overlaps with Zapier. And I do think that that is very useful. It's it's still not like that impressive, I guess, maybe.

• 48:24Speaker 02

  But to everyone who's ever had an executive assistant or, you know, wished for an executive assistant, like this is going to approach what you would want as a as a sort of always on personal assistant in your life. And I do think that that will unlock, you know, a level of productivity and quality of life, to be honest, that we just never seen before. And the fact that we can pay, you know, dollars for this, I think is is useful. For what it's worth, I think Lindy's working out like it's not just reliability, but also like economics and security as well. And I do think that one of the pitches of why AI engineering is a thing is because we, the programmers, are going to be the people who put Shogoff in a box, is what I is what I say. Right? Like the like.

• 49:09Speaker 02

  Us fine tuning things down to smaller models, to domain-specific models, is the safe approach. Being able to compose them into usable software systems to put humans first, instead of creating sort of potentially ruinous AGI. Um, I do think that that is a movement that I can stand strongly behind and um is is like, nobody's against it. Effectively, like, we all, we all want this to happen. It cannot happen quickly enough. And when it happens, there's not, there's no safety concerns, uh, except to the point where uh, you let agents loose on the internet with, with uh, with no permissioning. Um, and so far, I think everyone, including Flo, uh, who, who recently had a, had a nice, uh, safety discussion turn. Um, I think everyone's being like very responsible about it.

• 49:54Speaker 02

  Like, if you interviewed Connor Lee, Connor's like one of the most like safety-minded people, like he split from Uber there because they, they weren't safe enough. And he has like this very strong criticism of the other GPT team. Uh, and I think he's just never met them. Like, all of them are very concerned about safety. And, and I think, uh, it is the engineers who will put Shoggoth in the box. That's, that's my bottom line. Um, and, and understanding how to, how to wield these tools for human benefit and not human ruin. I think, uh, it, it, like, the, the responsibility lies very heavily on the engineers.

• 50:27Speaker 01

  I think I agree with everything you're saying there. I often describe myself as a adoption accelerationist and a hyper scaling pauser. In other words, like, let's get the benefits that we can from our current systems. Or, you know, I think we're gonna go a little bit farther than current, and that's probably okay, you know? But I'm very nervous about what happens if you put a million times more compute into a model than was put into GPT-4. Cause I just feel like we have zero ability to, to know. Reminds me a little bit too of Eric Drexler's comprehensive AI services vision. You know, just the idea that if you have narrow, perhaps superhuman, but still just fundamentally narrow systems doing all kinds of jobs.

• 51:13Speaker 01

  In a way that can kind of become a buffer for the world against more generalist systems. And, you know, as such, like, it's, seems really good to try to create these narrow systems that hopefully work really well. Is there something that, going back to like, what do we expect to work and not work? What do you expect to not work? Is there something that you would say like, yeah, two, three years from now, I still think, yeah, we're definitely not going to have an AI that you can get to do X?

• 51:40Speaker 02

  Oh, God. That's so.

• 51:42Speaker 01

  I have a tough time answering this one, so.

• 51:44Speaker 02

  Yeah, yeah, putting upper bounds on these things. Well, no, one of the reasons I pivoted is very, very hard into this thing is I have to sort of updating algorithm. Whenever I've been wrong, I ask what else I could be wrong about. And then I progressively update. It's kind of my sort of Adam optimizer version of momentum based learning. But so effectively, I, so I think I'm very skeptical about code agents that do PRs. So there's like sweep.dev and then there's a, there's a, there's a bunch of other agent type companies that will promise, you know, file an issue will turn into a PR, right? And it always, always works on like very, very simple sort of copy change demos and it never works on more complicated things.

• 52:26Speaker 02

  And I do think basically like the, the problem of software architecture and translating user requirements into code. I would, I would be happy saying like, that's a five years out thing and not, it's not an immediate solve thing. I mean, I, I still think it's a very worthy problem because AI engineers, instead of AI, like, AI engineers, job title, but like AI, like primarily AI engineer. I do think that's a very useful goal and worthwhile goal because, you know, engineers are expensive. And if you can pay an AI bot, that's a hundred times cheaper than a regular human engineer, but is maybe 10% of the capabilities, then you'll probably use it for quite a few situations. I just think like, yeah, the, the whole promise of like issue to PR is probably going to be overhyped for a long time

• 53:09Speaker 02

  just because the complexity of what software is. You know, I'm very happy to eat my words because I, and there are quite a few friends pursuing this, this goal. And more power to them, right? Like all, all human progress is, is based on unreasonable people.

• 53:24Speaker 02

  I think Meta GPT, I don't know if you've covered that paper recently. I think Microsoft also had a similar paper come out where effectively you have AIs play different roles in a sort of agile scrum process in working towards a common outcome in software. Has it shown better human eval results than GPT four? But again, like you're fundamentally not describing the full complexity of what software systems want to optimize for. And I don't think we have nearly enough training data to do that. Like it's very much in the world of intelligent design rather than evolution.

• 54:00Speaker 01

  I think we can maybe refine this a little bit more offline, but I think I would take the other side of that bet. I mean, there's, you know, how big of a pull request and, you know, how big of a code base and I think obviously for it to be interesting, you'd have to have enough tokens that it would be like beyond what a Claude two can currently handle. But even so, you know, it does feel to me like I would say in two to three years, it does feel like I would expect us to get there. And still on the basis of like, maybe there's no like eureka moments. I had an episode not long ago with the Med Palm M authors from Google, where it's like, they're really closing in on expert level

• 54:42Speaker 01

  medical question answering, the ability to take a x-ray or another scan and provide like a radiology report at very close to a human level. Forty percent of the time their Med Palm M x-ray report was preferred to the actual radiologist. So it's still under fifty, but it's like, damn, that's getting really close, you know? So if we can get that far already on problems that hard, I do feel like in a, you know, in a two-year timeframe, some pretty complicated. Kind of prompt to PR type things should also be possible. So we can maybe firm that up and put ourselves on record for a little friendly wager.

• 55:24Speaker 02

  Look, there's, there's, there's a, there's a bit of undefined thing here, which is like, how much do you have to specify things in order for a decent PR to come out? And will, will that prompting effectively approach a new programming language? Right? Like, you know, English is the new hottest new programming language. But like to spell things out in some level of detail to which the, the, the L M's will get it. Like, are you, are you, are you basically just, just programming, but in a different, higher level assertion that is kind of like a domain specific language for L M's? So yeah, that, that is open for debate. And I do think like some of the approaches, I think I've seen G P T engineer kind of approach that with their sort of pipeline YAML thing. Does that count? I don't know, right? Like, is that's kind of pseudo prompting, that's pseudo programming already.

• 56:08Speaker 02

  In my mind, I want a non-technical person to just like manage a software project by themselves. Right? Just filing issues. I would, I would say that's my bar for like issue to PR is all. Non-technical person. But if you need a technical person to like come up with like some interesting YAML file and like prompts in specific ways, then it's not, your hand is too much on the, the, the, the way skill at that point.

• 56:30Speaker 01

  Yeah, okay, I think that makes sense.

• 56:32Speaker 02

  I, I have one more thing, one more topic to discuss, which is probably going to be one of the most talked about talks that is coming up, which is OpenAI just confirmed their talk with me. And we're going to have the first public Dolly three and G P T for vision talk. I, I'm pretty excited about that because that, that is the signal of the finally, we're moving into multimodality for everything. It's going to be a wild world, worlds. And I don't really know how to handle it because most of the people I have on, they're all in the tech domain and we're going to be very off balance when, when new modalities come out.

• 57:06Speaker 01

  Yeah, I got a little glimpse of that last week in trying to add a feature to a React app. And it was a fascinating experience. I have never, prior to this, you know, session last week, programmed in React at all. I've done JavaScript before, but it's been a while since I've really been into it. And so, you know, I, my first question was, can you explain to me the general structure of a React app? And then it did, and then it kind of, you know, I, there were like a couple of file types in my repo that I asked it about, and it explained to me, you know, what those are and said, oh, well, that means you've got these libraries in place, and that's what their purpose is. And then I asked it

• 57:47Speaker 01

  if it could give me a command that I could use to print out the tree structure of all my files, so I could paste that back in, so it could see the structure of my project, so it helped me do that. And then, you know, basically just kind of worked my way through to a working module. And along the way, because the new image, I didn't really have to do this, but along the way, took a couple of screenshots, fed those in as well. And I was like, man, this is really amazing. You know, it's kind of understanding like what the defect is in the screenshot and, you know, modifying the code based on, in part, on the screenshot. It definitely is still at the point where like, if you looked through this transcript, you'd be like, Nathan, you qualify as a technical person here. This is not, you know, at the level that you're describing. But you can see, you can kind

• 58:31Speaker 01

  of squint at it and see your way there with these screenshots where like, I want it to be something else, and here's what it is doing, you know, and if it can start to deduce or infer, you know, the, the right fixes based on just like, the visuals of the app not working, then it does seem like, you know, you're not too far away from a non-technical person. Maybe not as efficiently, but still kind of being able to like, get, you know, something along the lines of what they want, you know, which today is just like unthinkable. So.

• 59:02Speaker 02

  For, for those interested, I think McKay Wrigley has been making a lot of us jealous with the GPT-4 vision access, with some examples of what he can do from like, you know, sort of the vision to code modality. It's not just vision, right? It's also voice. And part of my update that I, that I did for September, and this is something that Ben Thompson of the Strict Techery Podcast, I think he's, he's now talking on the Sharp Tech Podcast. He had, he had, so he had a private demo of both modalities, and I've seen both in person as well. And I kind of agree with this point, which is vision is more. Impressive individually, but voice is the thing you're gonna use every day, all the time. It's kind of an interesting inversion of expectations, because I would count vision as my most important sense out of the six senses, or five senses.

• 59:46Speaker 02

  But voice, the ability to hear with, it's hands-free and it's just always on and understanding what you want. I think that's kind of a thing that I see a lot of people here in San Francisco building with. And so my perception, I used to dismiss voice. I was like, oh, like AutoGPT has like an eleven labs, that's just for fun. Like, who wants an agent talking back when I can just kind of read the text by myself way faster? But the fact that it's hands-free, the fact that it's just always on, and the fact that you can imitate some of human expressions, I think it's gonna be remarkable. And I don't know if you've seen, have you seen the Russian language teaching demo of GPT-4 voice mode, or ChatGPT voice mode?

• 1:00:26Speaker 01

  No, I haven't.

• 1:00:27Speaker 02

  So Greg Brockman tweeted this, I mean, maybe like two, three days ago. Duolingo should not exist anymore. This thing is gonna teach you languages on a personalized basis, much better than any other app in existence. Any human teacher, just because it's way more patient and way more forgiving, probably can understand, you know, multiple languages way better than you can. So I think it's gonna be a very, very interesting time for understanding all these new values, because I'm sure they're working on more than they haven't told us.

• 1:00:56Speaker 01

  Yeah, so you're kind of the, you know, Duolingo shouldn't exist anymore. That is kind of what I wanted to take your temperature on next, because I feel like in some sense I've seen like a version of this movie before, where, you know, going back to early Facebook days, for example, I was, I just happened to be in the same dorm as Zuckerberg in college, and I actually thought it seemed pretty dumb at first. I was like, I'm gonna go online and put a picture of myself. So I was a late adopter, actually, of Facebook on campus.

• 1:01:28Speaker 02

  Fun fact, Matt Walsh from Fixie, he used to be Zuckerberg's CS50 professor. I don't know if he's ever told you that.

• 1:01:34Speaker 01

  That did not come up, actually. That's fascinating.

• 1:01:37Speaker 02

  So in the social network, like the class that Jesse Eisenberg walks in on and then like solves the question and he walks out, that was him, that was Matt Walsh.

• 1:01:46Speaker 01

  Okay. Small worlds. Yeah, that's really funny. You know, Facebook initially had this like, come one, come all, build on our platform. You know, you get access to everybody's friends. Everything's better with friends. It's all going to be social. And then it kind of became clear over time that like, actually, we're going to build the most high value use cases natively on the platform. And, you know, the app developers can kind of like get a little bit of stuff around the edges. But today, basically, you can like login with Facebook and that's really it, right? There's not like a lot of utility in the Facebook platform. I'm wondering if we're starting to hit a moment where OpenAI is going to go a similar

• 1:02:28Speaker 01

  direction and start to kind of eat these adjacent use cases that are currently other companies. And so like companies we've talked about, right? Like human loop. I love the product. We're both fans, but it sure seems like from an OpenAI standpoint, you know, they just kind of released the like fine tuning jobs. Yeah. It was like, well, what are you doing there? That's starting to look a little like human loop, right? And then, so there, I mean, there's a ton of stuff that they could do there and it makes total sense. And it's like, not, they wouldn't obviously be doing it because they're hostile to anybody else in the ecosystem, but just because they're asking their users, you know, what do we, what do you need? And they're just executing on that. And these are things that people need.

• 1:03:11Speaker 01

  And then I'm kind of like, the same thing might start to happen with, for example, text to speech, you know, that we've got some really good companies that have created amazing stuff. I've cloned my voice with both PlayHT and Eleven Labs. They both sound awesome. Do you have a favorite? I don't know if you're willing to go on the record. Yeah, I think it does depend on use case. I would say Eleven Labs is probably the favorite for ease of use right now. But if I was doing something more artistic or kind of creative, or I wanted more emotion. Yeah, Play seems to have a lot of diversity. Yeah, the, the creative direction that you can give it is a little bit more open. So I think, you know, if you're analyzing this from kind of a who's OpenAI posing the

• 1:03:54Speaker 01

  biggest problem for, in that analysis, it would be Eleven Labs probably because OpenAI presumably is going to be more reluctant to like. Create the voice that's gonna, you know, feed into your horror movie or your video, you know, your shooter video game or whatever, where you're gonna want kind of a wide range of emotions, at least in the short term. But it seems like quite clear that they're headed for a, I would, I would imagine, tell me if you think this is off, but it seems like, you know, when I think about this dev day coming up in a month, I'm like, well, geez, they just launched voice in the app, they just launched vision in the app, you know, they just launched this preview of fine tuning, they've got vector database built in natively to ChatGPT Enterprise.

• 1:04:38Speaker 01

  Seems like all these things are coming to the API.

• 1:04:41Speaker 02

  Sorry, I haven't heard about the vector database. What details are there?

• 1:04:45Speaker 01

  I don't have a lot of details, but they have said publicly that, you know, you're gonna be able to connect ChatGPT to your information, right? As an enterprise customer. Basically, what I understand that that translates into is you can connect your Google Drive or your Notion or your Dropbox or whatever to their system. I understand they're gonna support a number of them out of the box, you know, kind of with their own connectors, and then they, you know, presumably will have some way for people to connect their own, you know, random data sources into the system as well. And, you know, as a ChatGPT enterprise user, then you can like query all your company's

• 1:05:28Speaker 01

  Google Drive assets as part of the ChatGPT enterprise experience. I don't know exactly where that is in terms of whether it's launched or not, but they've sort of said it's coming, and that much, I think, you know, is not like super secret. Beyond that, I don't have a lot of details. But I guess I just feel like all this stuff is coming to the API, right? And it seems like they're gonna do this to serve the AI engineer, because, you know, the AI engineer doesn't really want to have to go piece five different services together. Like, you know, today, if I want to create a voice assistant, I've got to like transcribe, send something into the language model, get text back. Convert that to audio.

• 1:06:10Speaker 01

  And you know, what would be a lot better is if I could just send audio into OpenAI and get audio streaming back at me, right? And similarly, like today, and I've done this and you've done it too, right? We've got to go select a vector database provider and figure out how we want to chunk our stuff and how it gets loaded in there. And then what's our query strategy? And you know, that is its own art, you know, to figure out like, how do we generate the query for the vector database to get the right stuff? Because often like the question the user asks and the actual contents that you're trying to, you know, find similarity with, those maybe not, don't line up so well. So there's like, whatever, all these kinds of different tricks and techniques. And again, I, what I really just kind of want is like, one place to be like, here's

• 1:06:53Speaker 01

  my docs, here's my question, I'm putting it in as an image and an audio file. And by the way, I want my response back as streaming audio and just have kind of OpenAI handle everything, right? Like, does that seem right? It seems like everything's, they're going to kind of just eat all these sort of things around them. And the AI engineer benefits, but a lot of these companies, you know, I wonder like, for a lot of the speakers that you have, as much as they have led, you know, how do they end up not having kind of been the, you know, community R and D department for OpenAI?

• 1:07:33Speaker 02

  Look, every large enough platform eventually comes into this tension with their developers. And OpenAI is just coming into that right now. Apple has this long history of Sherlocking, which is an official term, I think, where, you know, some, some very successful community app gets killed effectively by an official Apple version. And I do think like, you know, OpenAI has promised to not compete with, with its own developers. They have said that, funny enough, in the deleted transcript of the interview that Sam did with Raza, which is still publicly available, they promised to not compete with developers, but I don't think their promise is ironclad. I don't think they want to stand behind that.

• 1:08:18Speaker 01

  That reminds me of the dumb and dumber scene, not to call anybody dumb, but the, you know, these are just as good as money. These are IOUs. And it's like, yeah, that was easy to say at that roundtable, but because they're at 1,000 people now and, you know, they've, they've got product managers.

• 1:08:35Speaker 02

  600.

• 1:08:35Speaker 01

  600? Oh, okay.

• 1:08:37Speaker 02

  Yeah, it, I mean, I, I happen to have very life stats, so I know. I would say that they always want to focus their people on the bigger things, right? Where, you know, there was this recent discussion about the OpenAI phone. This is what I've been calling it with Jony Ive. Like, working on a phone, that's big and ambitious, you know, working on, you know, A/B testing, fine tuning infrastructure for the enterprise, not that ambitious. Leave that to the lesser engineers in the community to, to pursue and, and, you know, I think go, go for the big things and obviously try to solve problems that everybody has. So I would say like, I think, yeah, OpenAI does, does exist in attention with the developers that go on top of it. Probably some of the companies will be Sherlock. I, I don't think 11 Labs would be a threat.

• 1:09:22Speaker 02

  Like, the best thing that startups can do, building around the sort of foundation model lab ecosystem, this is not just OpenAI, it's also all the others. It's just prove that you're a really good engineering team, because if you're good enough, they'll buy you and, and, you know, pull you into their, their orbit. Like, money is free to these people. Like, it, it doesn't actually matter. Are you choosing to work on interesting problems and do you ship fast enough that you're making an impact? And I think 11 Labs has definitely shown that they, they can do that. And hopefully, you know, enough of the, the, the teams that, that I'm seeing can do that, but not everyone will make it, right? Like, there, there's definitely going to be collateral damage. OpenAI, you know, has such a huge footprint that it cannot help, but realign people. Like, when a lot, a lot of what LangChain and Barrios used to do was

• 1:10:07Speaker 02

  G-Son output validation, right? And like, now that it's completely useless because of OpenAI. And and they just have to roll with it. And I think everyone's completely fine. We all understand we're building on top of shaky foundations and it's kind of moving, but that's what you get for being first. You know, in terms of the the multimodality stuff that you talked about, I do think like that will be a focus for Dev Day. I don't know if there'll be, they'll have the sort of APIs ready by then, but we'll be able to see what you can build with it um in in the in the talk that Simon and Logan will be giving. But this comes back to runes post on text being the ultimate universal interface. Um I do think like text is sort of the king modality.

• 1:10:44Speaker 02

  Yes, like whatever species synthesis is being done, whatever sort of um text to image or image to text stuff that they're doing, like everything just kind of just passes through text and being uh working on text is like the central pipeline. Um I do think it has some links to it for the time being. Very happy to to eat my words. Ultimately, everything is just tokens within a token space. And text has is no particular domain of it uh in there, except that we have much more of an understanding of what high quality data looks like in text than in everything else.

• 1:11:14Speaker 01

  Yeah, it does seem like, if I had to guess, I think things are gonna get a lot easier for the AI engineer over the next couple of months with just some really integrated kind of high quality things, you know, such that kind of end of this year. You could probably feed in multiple modalities. You could kind of have them manage your custom data set powering retrieval, but you know, powered by them. And then they can kind of stream out to you text, yes, of course, but audio on top of that, perhaps even, you know, images integrated into the responses. And that is going to be quite a uh a difference and and quite a simplification.

• 1:11:55Speaker 01

  We've got GPT four fine tuning as well, which is like another huge one because to the degree that you have a use case that you can't get to work well yet with any current model, you know, this could be really the thing that that allows it to uh to become possible. Things that OpenAI might watch that like wouldn't compete with anybody in the ecosystem, but which might actually just be, you know, a win for everyone. One concept that I had there is login with OpenAI. And I think this is like potentially really good for a lot of developers.

• 1:12:37Speaker 01

  If you are struggling with this tension between I want to show off how sweet my product is. But to do that, I'm like taking a non trivial hit in aggregate in terms of token cost. At Waymark, for example, my company, we estimate it's about 15 cents per user. That tries the free product in hard cost to us. Some of that is OpenAI language models, some of it is image understanding. We have some self hosted stuff on Amazon. We try to process all their images that we possibly can to create the best, you know, profile we can of the user, et cetera, et cetera. I'd rather that the user could kind of pay that free cost, you know, instead of us having to pay it, but obviously they would need some mechanism for that.

• 1:13:22Speaker 01

  Do you like that idea? And how would you riff on that idea? Do you see that as as a plausible thing that they might introduce?

• 1:13:30Speaker 02

  Absolutely. I think it makes sense. I just don't know if it's meets their bar for interesting work. I think login with Google is effectively this equivalent thing for login with OpenAI. And but also like implementing SSL provider, I think it's relatively commoditized to begin with, then gets very complex over time as you're starting to manage non trivial amounts of user data, which maybe OpenAI doesn't want to get into business of, right? It's more just about choice or alignment with the mission. And I don't know if that helps them build AGI. I think it's a nice quality of life thing for them. Maybe they can get more information as people sort of, you know, use you sort of universal login with OpenAI.

• 1:14:12Speaker 02

  I do think that if they do build this, then they have effectively decided to become the AI cloud. I think they maybe own AI.com. Don't quote me on that. It might be them or Elon. This would make them the fourth major public cloud, which is very exciting, but also.

• 1:14:28Speaker 02

  Maybe not like quite a research organization. And again, I still don't know if they want to do that. I do think it's like kind of a choice of paths and they're fortunate enough that they have that luxury and capability. There's, there's many things that they could point their engineers at, and I don't know if like building SSL for AI is like one of them. Could you just kind of offload it to Octo, off serial, whatever? And someone else could do that as well. So I will say one thing which I think I like as an idea, which is sort of context that kind of lives with you, right? And so I think you maybe mentioned this a little bit. That does seem to help, but probably actually a character he added does that, which, which has by far the longest history with working with this. Maybe not so much an OpenAI maxi in this regard.

• 1:15:13Speaker 02

  I tend to describe myself as quite an OpenAI maxi with regards to L M progress. But with regards to like products, I feel like character has just kind of earned its stripes in trying to be a chatbot much more than OpenAI has, has done. Even though they haven't had, you know, sort of state of the art new models to ship.

• 1:15:30Speaker 01

  I don't necessarily want to be an OpenAI maxi, but it's hard not to be.

• 1:15:37Speaker 02

  I write a monthly recap, right? And then my OpenAI news section takes two, two screens. And then my other frontier model, you know, that, you know, on the frontier model, model forum, I have, I have other frontier model updates. And it's like five bullet points. Like it's, it's not a competition, like OpenAI is just running away with it so hard. Yeah, I mean, it's just being, being objective makes you an OpenAI maxi.

• 1:15:59Speaker 01

  Competition right now is just not that deep, right? I mean, you know, we said just before we started recording, we got to give love to Claude two. I would say Anthropic has definitely shown that they can create really high quality models that are not like qualitatively outclassed by OpenAI's. Even though they do fall, you know, a little bit short on like the M M L U benchmark and whatnot. But it's like definitely a very good experience to use and something that I do in fact use, especially when I need long context or whatever. But they kind of have a strategy of like not leading the market in product releases as far as I know. And so there really aren't that many other competitors out there right now that could, could bring a similarly robust and just useful.

• 1:16:44Speaker 01

  Model forward. I mean, Google is not there. They are maybe going to get there in the not too distant future, but as of now, you know, the the models are just not as good. And, you know, who else is there, right? So.

• 1:16:57Speaker 02

  So I haven't done enough, spent enough time on Palm two to actually definitively make that statements. I want to believe in Google. I like I've met people there. They're very smart. They have all the resources in the world. They just don't have the institutional capacity to the shifts of being interesting for people, unfortunately, so far.

• 1:17:16Speaker 01

  I think the research is not the limiting factor there. If I had to guess right now, it's certainly not the compute either. So and the fact that they didn't participate in the last Anthropic fundraise and and allowed Amazon to kind of set that.

• 1:17:32Speaker 02

  Amazon just bid way higher than anyone else was willing to bid. Anthropic is probably worth like 10 billion now or something. And Google, I don't think they will. They probably want to invest in house. I mean, I've I've talked with people involved in in the the previous rounds and they said it was much more hands off. So like Google in no way owns that relationship as as strongly as Microsoft and OpenAI were.

• 1:17:52Speaker 01

  My biggest update from that was just that Gemini must be on path to work pretty well, because if it weren't, then they would be bidding, you know, as much or more than Amazon, right? Like, it seemed like they must have had some, you know, inside track based on the early relationship, even if it was fairly, you know, just kind of friendly arms length, whatever. If they didn't have confidence in their upcoming releases, then hard to imagine how they'd let that one go.

• 1:18:19Speaker 02

  Yeah, who knows? Who knows? Battle of the Titans is they hopefully the the tell all books that someone will write in 10 years. Maybe Walter Isaacson's fill around.

• 1:18:28Speaker 01

  People are wearing their rewind pendants, so they're capturing everything.

• 1:18:31Speaker 02

  One of the themes that I featured for my monthly recap this yesterday was the AI core crux is what I was saying, or like AI pendants, which I I saw a demo from Avi Schiffman of TAB yesterday. I've also talked with a number, a number of people also working on kind of similar projects. I would be willing to wear one of those. So I have an aura on me right now. And this is kind of logging my my heartbeat and whatever, but I'd love to log all my conversations and be able to talk with it. These things are coming. And yes, there will be multimodal, but also like I, I think all of us having an effective digital twin that we can talk to and, and use, use as at least just a note taking thing, if not just exposed to the wider world. I'm very excited by it and it will be built by

• 1:19:14Speaker 02

  the AI engineer. So yeah, I will feature like a probably a, an entire stage next, next quarter on, I mean, uh, you know, the next time I do my summit on, on these things because I think in terms of like the B two C frontier, if we're half a form factor that is beyond the phone, it's going to be the humane clip or the AI pendant or the watch or whatever it is that's just always on you.

• 1:19:37Speaker 01

  Yeah, that definitely seems like it's coming before too long as well. So last thing I wanted to ask you about, and I've been kind of asking everybody about this, listeners to the show will have heard me pitch it before. So I'll try to do it super briefly. Concept is the AI bundle. And I'll give you just kind of brief inspiration from the consumer side and from the app developer side. Consumer side, you know, pretty obvious, right? I've got chat GPT pro, I've got Claude pro, I've got Copilot, I've got Repli Ghostwriter, I've got perplexity pro, I've got, you know, other stuff that rewind and, you know, all these things, right? And my AI, my monthly AI bill is like getting up to the point where it's at or north of what

• 1:20:16Speaker 01

  my cable bill used to be. It notably doesn't cover hundreds of other apps that I like, might like to use, but you know, they're all like another 20 bucks a month or whatever. And I'm like, man, do I, how many of these like $20 a month things do I want to buy? Right? So that's the consumer side. On the application developer side, just, you know, representing myself as Waymark, we have this, you know, easy to use video creator. It used to be DIY. Now it's done for you with AI. You tell us what kind of video you want, tell us who you are, we make the video, you just get to sit back and watch it. It's awesome. But we do have this kind of weird tension where, as I said, it costs us 15 cents for every user. We don't want to put it behind a paywall because we want to show it off. You know,

• 1:21:00Speaker 01

  people are most likely to buy if they can try it. But then the flip side of that is like, you know, we have the kind of classic, whatever, $30 a month plan, and we need one person to buy for every 200 free trials just to break even on our tokens, right? Before we cover any other cost or make any money whatsoever. So that's a tough situation because it puts us in this weird thing where we're like, when do we gate it? How do we gate it? And then we also see, of course, the behavior that SaaS app developers hate to see, which is where people, you know, do the thing they want to do and then immediately cancel. And there's been, you know, increasingly a lot of talk about this, like churn is pretty high. Everybody's getting a lot of new customers. Interest is really high, but churn is also

• 1:21:41Speaker 01

  pretty high. On both ends of this, it seems to me like there could be real value in some sort of bundle. And what I want, I think as a consumer and potentially also as an application developer, is something where it's like, hey, pay 100 bucks a month and get 1,000 AI apps. And those would be like your flagship apps, like maybe your JetGPT Pro, and then maybe, you know, a bunch of long tail apps, like a Waymark, right? What I figure is, if I'm Waymark, which I am, and I'm one of 1,000 apps, and I'm like, okay, look, I know that, you know, there's a lot of things going on. Most people don't need us that often. You

• 1:22:24Speaker 01

  know, we've got some power users, but we also got a lot of people that just, hey, just one off here and there, right? And they buy and they immediately cancel. And it's not because they didn't like the service. It's just like, I don't want to pay for this every month. So if it was $100 bundle and we were one of 1,000 apps, let's just say we got the average, which is 10 cents a month per, you know, app subscriber, kind of in the same way that like ESPN Classic, you know, gets a very small fraction of my cable bundle. I think we would do that, you know, and I think we would be, we could still kind of upsell the power users into higher tiers or whatever. But, you know, for every million, if it was, if that was the economics, every million subscribers that the bundle had would represent $1 million

• 1:23:06Speaker 01

  in annual revenue to an app like Waymark, even just getting, you know, 10 cents per month per user. Can you see that happening? On the supply and the demand side, it seems like there would be a lot of utility there. Obviously it takes some doing to pull that trick off. But do you see that being something that could make sense? Or if not, like where would it break down?

• 1:23:30Speaker 02

  The closest equivalent would be set app. Are you familiar with set app?

• 1:23:33Speaker 01

  No, I don't think so.

• 1:23:35Speaker 02

  So there are existing software bundle type things out there. And they do bundle subscriptions. And I do think that they provide some value for a portion of the population. But I don't know how big this is. People are pursuing the sort of bundle economics play in traditional SaaS. And maybe you can have an AI bundle SaaS play. I would say some of the core questions would be, do you get paid regardless of whether or not people use you? And I think that would very much affects whether or not the the extra million comes to you or not. I don't know if you have a quick answer to that one.

• 1:24:08Speaker 01

  Yeah, you know, I mean, I think it could be managed in any number of ways. Obviously, for whoever is managing the bundle, you'd want to have some rules or framework in place to keep it somewhat sane. You know, just using the cable bundle as kind of the jumping off point. There's certain channels I never watch, you know, and there's and other people don't watch the channels that I watch. And my understanding is they all get a per subscriber share regardless of, you know, whether or not I tune in. But then when it comes up for renegotiation, then there's like different levels, of course, depending on like how popular you are. I imagine something like that, perhaps like managed even purely algorithmically could work. I think you'd have to have some kind of baseline.

• 1:24:50Speaker 01

  If you're in, you're going to get something from this to make it appealing to the app developers. And I definitely would see it kind of where the actual money flows, I would see shifting around over time with shifting usage patterns.

• 1:25:03Speaker 02

  Yeah, exactly. So I would say like there, there's difficulties there. There's difficulties with regards to the differences between like what a bundle of a newsletter or a bundle of content would would look like. I would say that the people who have probably thought about this the most is Nathan Pashaw, who has now started Lex Page as as as an independent startup. And Ben Thompson of Stratechery, who's written a lot about the media and cable bundle. So they would be much more clear thinkers on this than I am. And I do like the sort of magic of bundle math, right? They say you get more choice, you pay one fee. And then the people also have, the suppliers also have more predictable income.

• 1:25:43Speaker 02

  And the share and access to income without, and sort of pooling that distribution in terms of getting people to pay and then getting to people to understand what they are and try them out. I will say like for SaaS, the economic difference between SaaS and content businesses is that SaaS people typically really, really want to own their customer base. When I want to change pricing, like, do I have to get your approval as Mr. Bundle Guy? And I'm not, you know, I'm not gonna be happy if you push back on something. And so there, I think there's a lot of these internal policies that happen when it comes to B two B. And then like pile that on top of, you know, I'm very immersed in the B two B or enterprise infrastructure environments in San Francisco.

• 1:26:27Speaker 02

  Like I, like that's where my background is coming in terms of developer tools. And all of them are not going after the ten, twenty dollar a month subscription, right? They want to go after the ten K, one hundred K, a million dollar a year contracts that they want to get from other businesses. So effectively, this is a rounding error for them.

• 1:26:49Speaker 01

  Yeah, I would not expect like your human loop kind of infrastructure players probably to get into something like this. Yeah, it would have to be kind of the, the way marks.

• 1:26:59Speaker 02

  Consumer B two C, yeah.

• 1:27:01Speaker 01

  Yeah, all these, you know, there's a million of these, like, I recently had one of the founders of Gamma on the show and, you know, just, it's like, I don't make that many slides, but when I do, it's cool, right? But I feel bad if I'm just using their free version and, you know, not paying them anything, but I'm also like. I don't necessarily have, you know, hundreds of dollars to drop on every app that I'm interested in trying on an annual basis. So, you know, I either kind of sign up and cancel, or I just don't sign up. It's probably not going to happen, but I kind of, uh, something about this feels like it should happen to me. So I kind of can't, uh, let go of it just yet.

• 1:27:34Speaker 02

  Well, the, the real question is, what is the minimal viable version of this that you can test to, to find out, right? Like, and that's something that one of my mentors, um, has always given me as, as sort of just life and career advice, which is whenever you feel like you have this idea, or whenever you're stuck with a fork in the road, what is like the minimal viable step to gain more information for you to be sure? Because I think you could sit there and like, you know, have this like, it could work, it could not work, I don't know. Like, the only way is to take steps. And so like, I don't know how the size is down because like, there is a sort of appeal to scale of these bundles, but, uh, maybe like get five things together and then see, see if that works.

• 1:28:13Speaker 01

  Yeah, I think one of the, the big challenges with this is it does feel to me like you would need anchor products in the bundle to make it compelling. If somebody said, hey, you know, for 10 bucks even, you know, you can have 300 apps you've never heard of, then I'd be like, eh, I'm not sure I need that. But for a hundred bucks, if I could have, you know, ChatGPT Pro and Claude Pro and, you know, Perplexity Pro and a bunch of other things that I've never heard of, then I'm like, yeah, I might go for that because those three, I really, you know, I already know I want. And then, you know, whatever else, I'll kind of discover along the way. Anything else you want to talk about today before we break?

• 1:28:54Speaker 02

  That's a real pleasure. Um, I have just been, uh, enjoying the podcast. Uh, I would say just keep up what you're doing. There's always room for these sort of long form deep dives into AI. And, um, I, you know, I love the passion that you're taking to approach the space. I'd love to have you on at my next, uh, conference.

• 1:29:10Speaker 01

  Well, thank you very much. Uh, obviously the feeling is mutual. Why don't you, um, give us the details of the summit and, uh, what people can do if they still want to get a ticket and get there?

• 1:29:21Speaker 02

  Yeah, I mean, tickets are completely sold out, but you can get an online ticket where it's basically the livestream plus the Slack community discussion at AI dot engineer. I really love these short domains, by the way. So just like, yes, engineers at TLD. We bought AI dot engineer. So we have AI engineer summit, user dashboard. The podcast is latent dot space.

• 1:29:41Speaker 01

  Cool. Well, this has been a lot of fun. Swyx, thank you for being part of the cognitive revolution.

• 1:29:46Speaker 02

  Thanks for having me.