AI Engineers, Pendants, and Competition Between OpenAI and Developers with Swyx of Latent Space

In this episode, Nathan sits down with Swyx of Latent Space to chat about AI engineers and tools to check out, competitive dynamics between OpenAI, other foundation model providers, and developers, and if Swyx would wear an AI Horcrux. We also learn that Nathan lived in the same dorm as Mark Zuckerberg back in his college days (and was a late adopter to Facebook…). If you're looking for an ERP platform, check out our sponsor, NetSuite: http://netsuite.com/cognitive Subscribe to the show Substack to join future conversations and submit questions to upcoming guests! https://cognitiverevolution.substack.com/ We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. SPONSORS: Shopify | NetSuite | Omneky Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive NetSuite has 25 years of providing financial software for all your business needs. More than 36,000 businesses have already upgraded to NetSuite by Oracle, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you're looking for an ERP platform ✅ head to NetSuite: http://netsuite.com/cognitive and download your own customized KPI checklist. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. TIMESTAMPS: (00:00) Episode Preview (00:00:49) AI Nathan’s intro (00:03:14) What is an AI engineer? (00:05:56) What backgrounds do AI engineers typically have? (00:15:51) Sponsors: Netsuite | Omneky (00:17:13) Swyx’s Discord AI project (00:20:41) Key tools for AI engineers (00:23:42) HumanLoop, Guardrails, Langchain (00:27:01) Criteria for identifying capable AI engineers when hiring (00:30:59) Skepticism around AI being a fad and doubts about contributing to AI (00:34:03) AI Engineer Conference speaker lineup (00:41:14) AI agents and two years to AGI (00:46:04) Expectations and disagreement around what AI agent capabilities will work soon (00:50:12) Swyx’s OpenAI thesis (00:53:03) AI safety considerations and the role of AI engineers (00:56:24) Disagreement on whether AI will soon be able to generate code pull requests (01:01:07) AI helping non-technical people to code (01:01:49) Multi-modal Chat-GPT and the future implications (01:03:33) Nathan living in the same dorm as Mark Zuckerberg (01:04:44) Competitive dynamics between OpenAI and other AI model developers (01:05:39) Play.ht vs ElevenLabs (01:09:20) The tension between platforms and developers building on top of them (01:11:40) The best thing startups can do to compete with foundation model providers (01:16:26) User identity/authentication services like Login with OpenAI (01:19:20) Google vs the other live players (01:20:46) AI Horcruxes / Pendants (01:22:05) The concept of an AI app bundle for consumers and developers

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The AI core crux or like AI pundits, I would be willing to wear one of those. So I have an aura on me right now. I'd love to log all my conversations and be able to talk with it. These things are coming. And yes, they will be multimodal. All of us having an effective digital twin that we can talk to and use as at least just a note taking thing, if not just exposed to the wider world. I'm very excited by it and it will be built by the AI engineer.

• 0:24Nathan Labenz?

  Hello and welcome to the cognitive revolution where we interview visionary researchers, entrepreneurs and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life and society in the coming years. I'm Nathan Labenz, joined by my co-host Erik Torenberg.

• 0:47Speaker 02

  Hello and welcome back to the cognitive revolution. This is Nathan's AI voice cloned with Eleven Labs. The real Nathan is traveling for a wedding today, but we are both very excited to share this conversation with Swyx. Swyx is a renaissance man in the AI space. He is a builder, an educator, a connector, and an all-around very nice, encouraging and collaborative person. We cross-posted one of his latent space podcast episodes with Linus Lee of Notion AI a couple of months back, and I was excited to finally have a proper one-on-one conversation. We talked about Swyx's thought leadership, which has helped define the emerging role of the AI engineer, and also the AI engineer summit that he has organized for October 8th through 10th.

• 1:31Speaker 02

  Tickets are sold out, but you can still join online to hear from an impressive lineup of speakers, including a number of past cognitive revolution guests. We also covered about the state of AI agents, whether code PR generators will soon begin to work, the future of the app layer, the possibility that OpenAI and other leading model developers may come to compete with app developers as they add more tools to their platforms and modalities to their APIs, and the possibility of an AI application bundle. As always, if you're finding value in the show, I invite you to leave us a review on the platform of your choice. Now, I hope you enjoy this conversation with Swyx.

• 2:09Nathan Labenz?

  Swyx, welcome to the cognitive revolution.

• 2:12Speaker 00

  Thanks. I've been a longtime listener and very excited to be a first-time caller.

• 2:17Nathan Labenz?

  Well, thank you. Glad to have you here. And I'm also a big fan of your work with the Late in Space Podcast, the newsletter, and also looking forward to what you guys are putting together with the AI Engineer Summit, which is coming up in just a couple of days. So I'm excited to get into all of that with you.

• 2:35Speaker 00

  Yeah, happy to dive into that. We did a cross post, I think, a few months ago. And I really liked your deep dive into the Tiny Stories stuff. And that's the one that we featured on our feed. And so I feel like you go, you have the room to go much more in depth than us. So I really appreciate the work that you're doing, going like these like two hour things with researchers. It's really impressive.

• 2:56Nathan Labenz?

  Thank you very much. I really appreciate it. I guess for starters, I thought we'd kind of organize this by taking a little bit of like a broad view survey of your work over the last year or so. As far as I know, you've coined this term AI engineer. And so I guess I wanted to start off by just kind of asking you like, what is an AI engineer? I'm fascinated in general by these like new AI jobs, right? We've got the prompt engineer and a kind of a few different things have been put forward. Seems like the AI engineer, though, might have more staying power than like the prompt engineer. So how do you think about that new emerging role?

• 3:31Speaker 00

  Yeah, I definitely think of prompt engineering as like so 2022 and AI engineering as so 2023. And I feel like this is a controversial take a little bit because everyone should be able to use AI. There is no restriction on, you know, who does and does not use AI. But I do think that people who are choosing to specialize in the AI stack probably deserve a full time role that describes what they do. And out of all the possible names that people have proposed, like cognitive engineer, LLM engineer, probably the one that is going to win is AI engineer. And so I'm not so much pointing it as observing that this is a trend that's happening and putting all my chips on red, as they say.

• 4:16Speaker 00

  So the AI engineer is a software engineer specializing in AI and the emerging AI stack. An ML engineer is not an ML researcher, both of which are much more established roles and much more on the sort of research oriented and ML ops side of the fence. It is everything to do with what happens after you have a model in production and maybe with a little bit of fine tuning, which is, as everybody knows, just extra training on top of the vast amount of pre-training that's already been done. And I think it's basically An emerging category for a few reasons. It's more or less just demand and supply. And you know, I started life as a finance and economics guy. I was a trader and hedge fund guy for quite a few years before I was a developer.

• 4:59Speaker 00

  And I just think it's just pure demand and supply. You know, there's maybe like 5,000 good LLM engineers in the world, LLM researchers in the world. And you cannot hire them like as the average company, average startup, whatever. There's just, there's no way you ever actually you able to build this talent in-house. For better or worse, these models are now available as APIs or as open source models. There will be a corresponding rise in demand to for engineers who are capable of putting them to use, even though they don't necessarily train them as a on the foundation model research basis. I do think that there will be a rise in this category of the AI engineer. There's definitely a rise in startups serving that category. And so I pivoted in space, the newsletter, the podcast, and now the conference,

• 5:41Speaker 00

  all towards serving this persona, which I identify to be one of them.

• 5:45Nathan Labenz?

  Yeah, I think I would count myself among the AI engineers as well. I probably, I guess I probably come to it from a somewhat non-standard background, but maybe you can tell me like, where do you see the AI engineers coming from? Are they mostly software engineers who've kind of just taken an interest in AI and gone down the rabbit hole, so to speak? Or are there other backgrounds that you see as well?

• 6:08Speaker 00

  Yeah, they're going to be mostly software engineers taking interest and getting deeper in AI. In the post that I wrote on the space, the key visual to have in mind is this kind of left to right spectrum of research constrained work versus product and customer constrained work. And or basically, how much, how close are you to machine learning research? Or how close are you to applications of that research? And so on the far left is the research scientists and ML researchers. A little bit further along is the ML engineers, data scientists as well. I would class within that category. After you've sort of put those models into production, then you get the AI engineers, which are the emerging category. And then finally, the full stack generalist engineers,

• 6:51Speaker 00

  who are working on just like the last mile of UI and UX to deliver to AI products to people. And so I think just like this emerging category of engineers, it's much more people. On the right side of the spectrum, the software engineer is moving left into AI. So they're learning this, they're suddenly learning what tokenizers are and what embeddings are, what vector databases are, why prompting chain of thought makes sense rather than zero-shot generation of stuff. And then it is much more likely to be people from the right moving left than people from the left moving right, meaning the ML engineers and data centers moving right into applications, even though you do get those. So for example, Raz Haib, my most recent guest from Human Loop, has a PhD in probabilistic

• 7:35Speaker 00

  programming with DPO networks, but he's working on an LLM off solution just because he sees that a lot more people can be served that way. So there are people with all sorts of backgrounds, but I do think it's primarily software engineers.

• 7:47Nathan Labenz?

  We've had a somewhat of a parallel trajectory in the podcasting game over the last few months. Raz was also an early guest because I've been a customer of Human Loop since early this year and definitely have got a lot of value from the platform. Now starting to use it also to, I think, really getting to the part of the vision that they probably had in mind early on, which is the really accessible fine-tuning loop that they've enabled. And we just did an episode kind of going down that rabbit hole a little bit, talking about like, I think my number one insight there was using GPT-4 reasoning as part of the fine-tuning dataset for 3.5 as a way to really, in my experience, dramatically improve the results.

• 8:32Nathan Labenz?

  And that was a, something that Human Loop made much, much easier than it otherwise would have been. It's funny because I had already been a customer for months, but I hadn't used all the latest stuff. And then I was like, kind of thinking to myself, like, how am I going to kind of code up my own loop to do this? And then thought, well, I should check Human Loop and see what all the latest features are. And sure enough, they had done a really nice job of anticipating the need and really streamlining that process. So what do you think are the key skills for the AI engineer? If I were to come to you and say, okay, I've got a background in software development, you know, I've played around with ChatGPT and, you know, almost everybody's like at least used Copilot or something like that at this point.

• 9:15Nathan Labenz?

  Where do I go from there to make myself employable as an AI engineer? What are the employers, the app developers, most need?

• 9:25Speaker 00

  Yeah, totally. I think that's where most people start, which is use the off-the-shelf models that are that have the most adoption and that is going to be Copilot and ChatGPT. I do think I have been mapping this out. So if you go to the about page on Lit and Space, we also have an emerging email course that we call Lit and Space University to trigger the Louisiana State University fans.

• 9:48Speaker 00

  And and it's kind of mapping out the curriculum of what I expect to be the baseline competencies of an AI engineer, like just write the job title, like what what do you expect people to do? And I think I want to ground this in a very practical manner because it's a demand and supply issue. Companies are trying to hire these people and people are trying to learn to be useful to companies so that they can be part of the AI movement, but then also apply their their engineering knowledge towards building useful and interesting things. And so I'll just kind of list through some of them, but we can go into more details as needed. The high level themes are AI UX, AI coding tools, LLM tooling, AI infra and

• 10:29Speaker 00

  infra's hardware, and that includes fine tuning as well. And then finally, the most speculative would be AI agents, which everyone has been talking about, but doesn't have that much practical use. So I think there's definitely a trade off between what people actually use at work versus what people just like to talk about on Twitter and star on GitHub and promise AGI without actually doing it.

• 10:51Speaker 00

  So maybe that might be a hot take there. In my course, so we have a seven day free sort of email course type of thing. And that's meant to be like, I'm never going to make money from that. That's just meant to be like, like, start here if you want to learn AI engineering, you're a software engineer, you've never dealt with any of these APIs. So day one would be just try the GPT API for the first time. And just a lot of a lot of software engineers just haven't bothered. And I think you'll be surprised at how much controls, how much how many options there are available, thinking about running into the context limit for the first time, which you know, ChatGPT and Copilot can hide away for you. And just like, get familiar with all those basics. I think it's very important. Day two will be prompts, tooling and memory. And that's where you get into like the vector databases and the line chains and all that. And those are just frameworks that

• 11:35Speaker 00

  Will hopefully help with the developments and and I do encourage people to kind of build some extractions of these themselves before using something off the shelf, you know, so you get a better understanding. Day three is code generation, day four is image generation, day five is speech to text, day six, fine tuning and running open source models, and then day seven is building an agent. And I think once you've finished that little sampler course or like a tour, you have the base capabilities that I expect every AI engineer should have such that whenever any PM or any CEO comes to you with like, hey, I have an AI products that I would like to build, is this possible? You can figure out if you can build it, or you can just tell them like, hey, it's not possible for X, Y, and Z reasons. You know, that that way that you'll be useful as an AI engineer.

• 12:16Nathan Labenz?

  So that's really interesting because this is like, you know, the old XKCD, somebody's wrong on the internet, right? I almost have my own version of that with people posting G P T four can't do X things. When in fact, like it definitely can, and they're either in many cases, like using the wrong model or prompting it wrong or doing like a bunch of things that are wrong. So I actually find that people often come to wrong conclusions about what AI can and can't do today. Obviously, it's a fast moving frontier. So like your value of knowledge decays pretty quickly. I mean, seven days definitely seems like enough, especially if you really go for it to, and this is like a remarkable thing, right?

• 12:59Nathan Labenz?

  The part of what I love about the AI space in general is just that you can kind of jump right into the frontier. I think that's just extremely cool and super fun, and I would think most technologists would agree with that. So I do think like a seven day, you know, intensive is a pretty reasonable amount of time to like get mostly up to speed. I guess if there was one thing I would wonder about, it would be

• 13:23Nathan Labenz?

  would folks after that kind of intensive have a well-developed sense for what really is or isn't possible and like? How would you coach people maybe on like continuing to refine their sense of what is and is impossible? Because I see way too many people just giving up too soon.

• 13:40Speaker 00

  Yeah, hmm, this is interesting. So first of all, I actually wouldn't describe the course as intensive. It's just an email course. It's meant to take an hour a day. Uh, it's probably, and then we leave breadcrumbs to explore with a lot more details and suggestions for side projects and stuff. I think that if you want to be up to speed, there's a vast gulf between, hey, cover the fundamentals and be, be cutting edge. Because the, the people who are cutting edge are the people who are hiding in discords and on Twitter and talking about very nichey, jargony stuff that you won't even see for a few months out. So I don't really know if like, that's a realistic goal for most people. I think most people want to make sure they cover the fundamentals and then be able to build most projects that you've seen out there that make money.

• 14:23Speaker 00

  And ultimately, I, I think that's what most people want. Um, to be cutting edge means you have to go down a lot of, uh, rabbit holes. And, uh, I'm not exactly sure if I would recommend that for most people because that is a full-time job in and of itself. Which is why, by the way, I, I call it an AI gender because I, you know, I, I do think that this deserves a separate category or job title because it's, this is a full-time job, keeping up on, on things. But I do have a recommendation, which is I have a list of Twitter people to follow. Um, I have a list of Discord communities that I watch and obviously a list of podcasts and newsletters, which I recommend, uh, which you're definitely on. Um, so all of those are on my GitHub.

• 15:03Speaker 00

  Uh, I have an AI notes GitHub where I, where I link to all these lists of Twitter, YouTube, podcasts and newsletter people. Um, by the way, we're also running a survey, uh, of, uh, who people listen to. So a little bit of a market share competition going on. Uh, and if you, if you want to Google state of AI engineering survey, you can get a sense of who people are listening to. Um, and, and I think that's important to understand, like, like what drives people's attention towards, uh, individual projects. Uh, that is something that I should keep note off as well.

• 15:34Nathan Labenz?

  Yeah, interesting. I'll have to, I'll definitely subscribe to your Twitter list, which I did not know that you had out there. But I do actually get most of my stuff, I think. First from Twitter still, and then definitely have like a ton of discords that I've joined over time as well. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 15:54Speaker 00

  I might check in with you on this. So I have this little project where it basically scans discords on a daily basis and then summarizes them into an email. And I'm wondering if I should just release that as like a thing that people can subscribe for. Because I think it'd be kind of popular, but also very noisy. Because people discuss all sorts of things in Discord. And like, it might not actually make sense.

• 16:16Nathan Labenz?

  Well, I mean, that's kind of the AI challenge in general, right? Is like, how reliable can we make these things? I definitely think if you could get it to work well, it would be valuable. I've got to be in, I don't know, 50 different discords at this point. And it is super noisy in there. And I honestly kind of, I join them, I'll often kind of scan around, see what's happening, you know, see what discussion is there. And then often I don't really come back very much. So, good God, the number of Discord notifications have become unmanageable. So that just kind of gets tuned out. So I actually wonder if there would be a way to intercept the notifications, or kind of use that as a particular signal

• 17:01Nathan Labenz?

  as opposed to like, you know, having to kind of grab and summarize everything. I do think if it worked well, and it could kind of create an auto GPT style digest of the news from all these different projects of interest, I definitely think that would be really interesting.

• 17:16Speaker 00

  I might release that as a side project. I don't know how popular it would be, but I think it definitely solves a pain point for myself. It's funny, because I was, one of our unreleased episodes, we just recorded, this is Jeremy Howard from fast.ai. And in his recent post, he had a post recently on sort of one-shot learning, or like sudden drops in the loss curves, trying to figure out like the origins of that. And he mentioned the alignment lab Discord. And I always just find these discords of very elite LLM people. Just to emerge from nowhere, like news research is another one. I'll just throw these names out there. For people who want to find it, they're all on my list, so you can go check it out. But like, how do you find them as they're emerging? Because like, you know, most people found out about Eluther AI after

• 18:01Speaker 00

  Eluther was forming and the core community was already established. And now it's kind of, you know, people have departed Eluther into what I think calling this Eluther mafia recently. Eluther is still going, obviously. And I just didn't find like, you want to join these communities when they're early and everyone's still trying to figure it out and building something interesting. And Jeremy said, like, most of this stuff is also happening in private channels, not public channels. So it's just an extra wall garden inside of a wall garden. And just joining Discord isn't enough. You have to invest enough to get access into the private channels, is what I'm saying.

• 18:35Nathan Labenz?

  I need to get myself invited to some more private channels.

• 18:38Speaker 00

  For what it's worth, the bar for the in-space Discord is you show up, you introduce yourself, you're invited in. So.

• 18:44Nathan Labenz?

  Well, I appreciate your inclusive approach. A few more just kind of general state of the field questions, then I want to kind of shift toward the event that you have coming up soon and talk a little bit about kind of where things are going as well. The tooling for AI engineers, if I understood you correctly earlier, it seems like the pattern is mostly identifying the best available tools and then figuring out how to make those work together. As opposed to, you know, obviously you kind of distinguish this from like

• 19:18Nathan Labenz?

  deep ML research, but even from like self-hosting, it seems like most of these things are kind of services that people are figuring out how to put in concert and not too often spinning up their own services at this point. Is that accurate?

• 19:34Speaker 00

  Sure, even though they absolutely can, right? Like, this is the power of an engineer that you can decide to build versus buy at every point in time. And that's always a fun topic of conversation whenever you need to build versus buy.

• 19:45Nathan Labenz?

  When you think about that build versus buy, like, that's such a common debate, obviously, and it's like a tricky one for a lot of folks because the tension that I experience is like, I want to be early to market, if not first to market, right? Like at my company, Waymark, it's like, I want to be the, we have this video maker. I want to be the first video maker that does like A, B, and C, X, Y, and Z for you. And to do that, you know, you kind of have to. Build more than you would have to build if you just waited a little while because, you know, if you have a little patience, then sure enough, you know, the market kind of provides solutions for a lot of these things that you need. So I think it's obviously super contextual, but I wonder how if you have like any sort of

• 20:29Nathan Labenz?

  framework or kind of general, you know, high level view of how people should be thinking about build versus buy, especially given how quickly things are coming online for us to be able to buy.

• 20:41Speaker 00

  I generally have a pretty vendor friendly version of this because I have been a developer relations person for five years and but I also very sympathetic to the choose boring technology, try to limit your dependencies as much as possible mindset. And so basically, I typically what I often say is buy first and then the moment you understand your domain well enough, be in a position to be able to build yourself. And this way you get the benefits of people setting you on the right path with best practices that they've learned from everyone else. But you're ready at this point to understand that this field is so immature, you may have

• 21:23Speaker 00

  to rewrite, you may have to rip out something that you started not liking. So this was the topic of a lot of discussion this summer with Langchain, where a lot of people did had this exact journey, right? Like they started out building out some Langchain and they ran into a lot of problems that maybe Langchain wasn't well designed for, and then they ripped out Langchain and they said Langchain is crap. And I think that's kind of unfair to Langchain just because they're evolving as well. I mean, the company is less than a year old, you know? So I think it's fine, like everyone, like if you're, if you chose to be in this field, you are choosing to be on the cutting edge and sometimes the cutting edge cuts you. And that's what they do. That's the nature of these things. I would basically buy off the shelf first and understand what people are building out there because you plug into existing communities of people who have encountered these problems

• 22:06Speaker 00

  for far longer and have dealt with them and thought with them much deeper than you have. And then anything that you don't understand or don't still, you know, don't appreciate after working with them for a few weeks, then you can rip them out and build your own, right? But like I think at this point, you know, we have on our show notes like human loop, guardrails and Langchain. These are fairly well established frameworks among the community that you should at least understand as table stakes for what AI is doing is today, just because there's been a few hundred thousand people ahead of you mapping all these things out.

• 22:39Nathan Labenz?

  What other tools would you put on that list? I mean, we talked a little bit about human loop as kind of, you know, just to set the level there, a, at least the way I think about it, first is kind of a playground in which to develop prompts. So my workflow is, I go there and I workshop a prompt until I get it working reasonably well. Then the reason I do it there as opposed to anywhere else is because I can hit save and then immediately I have an API that I can call that basically puts all the prompt management stuff on the human loop platform so that all I have to do as a developer from outside is just provide a couple of variables, you know, one or more, whatever I set up. And that makes it super

• 23:21Nathan Labenz?

  easy to develop against. Then I can also update my prompt without having to do code changes, which is quite nice. And they log everything for me and allow me to, you know, kind of come in later and post-process it, evaluate it, export certain subsets of data, which I might use to, you know, power fine tuning based on like the most successful data points, whatever. That's really cool. We've had Shreya from Guardrails on the show a while back as well. You want to give your take on Guardrails and how that's used?

• 23:53Speaker 00

  Yeah, Guardrails is, I would say, in the output validation side of the fence. And, you know, as the name implies, it puts sort of safety rails on what generative AI does, specifically generated text. And I would say it is competing somewhat with the other orchestration frameworks because everyone, there's only one room for the sort of LLM interface layer and everyone's fighting for it. Human loop wants to be in that interface so that it can track everything. And so do all the other LLM ops and prompt ops companies. Is what they what they'd be calling in my podcast within Rasa called it the huge foundation model of companies, but they all effectively have the same thing, right? Like we'll track

• 24:35Speaker 00

  inversion, manage your prompts, and then we'll we'll eval them and then we'll like, you know, we'll track your costs too and track your latency and whatever, right? They all have the same roadmap because it's fundamentally an ops product. And Garvials and Langchain, slightly different, they're much more in the asset application framework side of the fence with an open source first and monetization second. And Garvials would do things like validate that SQL generated is correct if you're if you're trying to use AI to generate SQL. They used to have sort of output validation. Langchain also has this, by the way, output validation for valid JSON if you're if you're sort of outputting valid JSON. And that went away when obviously OpenAI came out with its own version of that. And so I think the roadmaps

• 25:15Speaker 00

  with these sort of orchestration or application LM specific frameworks will evolve. But Garvials is much more focused on safety and production readiness, let's say, and Langchain much more on the orchestration, even though they collide on some features because all of them are built on each other's features.

• 25:33Nathan Labenz?

  Yeah, that's super interesting. Let's come back to that in a second and get a little bit more into the competitive dynamics. But before we do, and this, you know, starts to give you an opportunity to talk about the upcoming AI engineer summit as well. But, you know, that and and more broadly, how would you suggest that people go about looking for AI engineers today? So, you know, whether you're whatever kind of company you you might be, right? Software application company, or, you know, I would say honestly, any company that has kind of a lot of operational overhead could probably make great use of an AI engineer, even if they're not putting a, you know, a public facing or customer facing application

• 26:15Nathan Labenz?

  out there necessarily. What do you look for if you're hiring that skill set? You know, it's so new, people are like lost, they, you know, people are kind of just looking for somebody to tell them what to do. But obviously, that creates a lot of opportunities for them to hire somebody that doesn't really know, you know, what to do. So if you don't have the skill set yourself, like, where do you go to look for it? How do you evaluate it? And of course, you know, the the upcoming summit, you know, could be one of Of those places.

• 26:39Speaker 00

  I honestly, I really should set up a job board. I've been chatting with a couple of companies on like, hey, I need someone to just set up a job board. You know, I don't even have to make money on these things. I just want to make sure that people who are looking for others, whether it's, you know, you're hiring or you're wanting to be hired. You have a commonplace to match. Right now, it's on Twitter, you know, anyone like actively talking in shipping projects is is fair game. Within Discord, it's like in the latent space Discord, we have a hiring channel and people can, people post jobs there. And I know people have gotten hired off of there as well. I think just like regular channels and communities posting in like the monthly sort of who is hiring job posting boards on Hacker News. That's, that's typically the way I would recommend these things.

• 27:22Speaker 00

  And obviously, uh, coming to AI engineering conferences and talking with people who are also attending. That that tends to be a very high signal filter for uh, who's very, who's very engaged. And I would say like, you know, especially if you yourself are not an AI engineer and you want to hire someone who is an AI engineer. The kind of core competencies that we listed out earlier in this podcast, you know, the uh, that we have on the about page that we have on the latent space university curriculum. Is what I would expect a bare minimum for people to understand um, for working as an AI engineer. And I don't think it stops there. Because I do think that a core requirement that you cannot really test for, you just have to sort of observe in an AI engineer is that they're kind of entrepreneurial.

• 28:04Speaker 00

  They, they're kind of, they're comfortable with things that are not that well defined. Because prompt engineering is not that well defined. Because the, the model landscape shifts every single month with the release of a new model or whatever. They have to be able to be on the ball and proactive and not just, hey, we are, you know, I'm a, I'm a sort of data scientist and I, and now I do L L Ms now. And, and, and now I'm an AI engineer. And I don't think that that is the kind of person that will put your business. At the cutting edge of what you can do with generative AI, they have to be a little bit entrepreneurial. They have to proactively come to you with ideas instead of saying, hey, well, you know,

• 28:47Speaker 00

  throw an L M layer on top of the existing app, which is totally fine. But like, I do think that especially for me, and especially like the people hiring that I talk to, they want someone who is a little bit entrepreneurial. So people who can just kind of like ship their own projects and get attention for them or like use new techniques and put them in practice. And I do have some in the Rise of the AI Engineer blog post. There's some sort of role models that I do highlight. You know, I think the entire Versal team has been doing an awesome job of showing what it is like to be entrepreneurial, right? Even though they're not a foundation model team, they can put these things to good use and in a way that is appealing to enough people that they get millions of users on their free projects.

• 29:31Speaker 00

  And I think that is something that every company should have. And that is very much the ethos of the kind of AI engineer that I want to encourage.

• 29:40Nathan Labenz?

  Why are not all software engineers jumping at the chance to be AI engineers? It seems like there's something weird going on there. If somebody says to me today, oh, these AIs, they'll never be really useful. They're not that useful. There's no good use cases, whatever. The first thing I would say is code, okay? Like you can, you know, question anything, but you really cannot question the utility of a copilot. And you definitely cannot question the utility of coding with G P T four. It is, you know, for me, I would say a multiple speed up safely. And I maybe am not like the very best software developer in the absence of G P T four. But like, you know, I've got plenty of experience and definitely can make stuff work. Given how.

• 30:26Nathan Labenz?

  Powerful it is for developers in their existing workflows and you know and the fact that it's like kind of been integrated into their tools before most other toolsets. What am I missing? Why is it not the case that like everybody is kind of gravitating this way?

• 30:44Speaker 00

  Yeah, I do think there's some baseline skepticism about whether or not this is a fad. And a lot of people have been burned by effectively crypto. It's a very fair thing to have some skepticism about anything new. It's a very fair thing to have some self doubt over your credentials. Like, do you need a PhD to make progress in this field? And very much like why I'm trying to promote the AI engineer is to encourage the idea that no, you actually don't need credentials to make progress in this field because everyone is effectively uncredited when transformers themselves are six years old. And when GPT-3 itself is three years old, then you know, many of the techniques and the companies that we talk about in this space are all less than one year old and all making tremendous progress in AI. And by the way, you know, one of the biggest

• 31:27Speaker 00

  promoters of the AI engineer concept is Andre Karpathy, who's who very kindly supported the the concept with the idea that there should be more AI engineers than ML engineers, and they'll be very successful without trading anything. So I think there's there's some mix of skepticism about AI being a fad. And then there's some mix of self skepticism about whether they can contribute in AI. And then there's always like fundamental misalignments or some fundamental doubts about the stochastic parrots argument, right? Whether or not just multiplying a bunches, a bunch of matrices can actually approach anything regarding simulated intelligence, which we can always talk about in sort of like those kind of dorm room hallway style conversations, like what is consciousness? What is, what is intelligence?

• 32:09Speaker 00

  But I mean, you and I know, like especially we're quoting with with AI, like these things are actual productivity enhancers. And they've gone from single line autocomplete to function autocomplete to code based, entire code based generation. They can help you as tools of thought, you know, with a human in the loop, or they can function without humans in the loop, all of which we're going to see as part of the conference, right? And that leads us, the more and more you have this sort of AI in the driving seat, that leads you more and more towards autonomous agents. There's a wide spectrum of AI as sort of autocomplete, all the way to AI as autonomous agents. And I do think you just kind of depict your line of like where you think usefulness currently is and and understand that that will probably move over time.

• 32:49Speaker 00

  We have about six orders of magnitude more improvements in terms of scaling according to Nat Friedman until the end of the decade. So take whatever we have today and project forward.

• 33:00Nathan Labenz?

  Yeah, it's gonna get wild. It's kind of pretty hard to imagine what pops out the other end of a million fold more compute than went into GPT four. So tell me about the speaker lineup. I went through, there's twenty eight accounted speakers shown on the website. I was proud to have had seven on the podcast as former guest, which was pretty cool. And it seems like you're basically kind of creating a lineup that covers kind of all the inputs, basically, that an AI engineer would need, right? From foundation models to these kind of frameworks, to the quality control measures. And then there's like a number of folks from agent startups as well. Run that down and particularly on agents, I'd love to hear your perspective.

• 33:45Nathan Labenz?

  I wonder, is that something that you see as being like the next thing that's going to come on for the AI engineer to tap into? Or is that kind of a different sort of thing where it's like those more are the AI engineers that are building the coolest new stuff on those lower levels?

• 34:00Speaker 00

  I basically pulled together from my network all the top speakers that I thought would building interesting things for AI engineers. And specifically for more more technical audience, because there's other conferences happening all, you know, all this fall in San Francisco. But they're all very high in general and they spend a lot of time talking about policy, safety, regulation and copyright and all those things. But for builders, I think there wasn't a builder specific conference until mine came along. And then obviously OpenAI had to top it with Dev Day, which we can also talk about, which I think you and I are also going, which is I'm very excited. We should do a live podcast on Dev Day, right?

• 34:37Speaker 00

  Yeah, we have people from AutoGPT is our presenting sponsor with OpenAI, with Microsoft speaking, Notion speaking, Amazon speaking and GitHub. Like all these are top names, I think, in the AI engineering field. But I also wanted to balance it out with names that you've never heard of, like Mithun Hansar from from the sort of Rust O L M community is speaking. And we but we also, you know, have like the first. I I think one of the world's first demos of Adept, one of the world's first demos of new computer. And just trying to get a mix of like projects that you've never seen before. So like Lindy is a very, uh, very, very hyped agent project that they've never done a public talk here. And so, so I think we're, we're trying to be the stage where people kind of launch these new features,

• 35:20Speaker 00

  new products, projects for the first time to provoke some thought. And then balance it out with people who talk more about sort of active production issues. So I think one perception of AI is it's all greenfield. It's all, you know, twenty something year olds building toy projects is not actually in serious work. And so, so I want to balance it out with Eugene from Amazon, who is running Amazon Books and with, with language models, you know, and, and Abi Arian, who's who just finished the O'Reilly book on L M ops in production. People like that who are actually implementing AI in like large scale production systems. Hex and Prefect are also speaking with us on, on how to pivot an existing non-AI company into an AI company. And I think both of them have done fantastic job of that.

• 36:05Speaker 00

  So I'm kind of curating this list. I do have very large gaps that I'm very conscious of, right? So I don't have image generation people and I don't have that much infrastructure of people. These tens and replicates of the world, and I would like to feature them next year. I just had a limited schedule of a two day single track conference. The, the underlying thing about behind this conference is I'm, I'm trying to basically create the ICML for engineers. So ICML, the International Conference of Machine Learning is like the ultimate, like, if you go to one conference a year, that's the conference you go to if you're a machine learning researcher. There's no equivalent for the engineer. And so what we're trying to do is like try to offer that. And so this thing was going to grow over time. I, I kind of see this as like a ten year commitment towards building the ultimate survey of the field

• 36:50Speaker 00

  and, and, and any given point in time for engineers.

• 36:53Nathan Labenz?

  I'm jealous that you have a demo from Flo and Lindy among a bunch of other cool stuff as well. I had him on as an early guest and continue to follow their, you know, progress from little hints they give out in the public. But I still have not been able to get into that thing even as a, an alpha user. So be very, very curious to see what they're about to show off.

• 37:14Speaker 00

  I, I'm very excited too. And honestly, you know, I, I think you, you're doing, you're doing a great job with, with your reports as well. And I'd love to have you as a, as a speaker in. In the next iteration, but yeah, I maybe I'll talk, I'll spend a bit of time talking about agents, right? Which is a very, very hot topic. So I would say, you know, maybe the surprise now is that AutoGPT is obviously now a company. Being one of the fastest going open source projects ever.

• 37:39Speaker 00

  I think, I think there's a ton of interest in, in what they're doing and Torin will be speaking. On our stage, I think for his first time ever as a conference speaker. So I think that'll be super exciting. I do think that there's a range of agent type projects, right? So the open source agents definitely tend to be sort of single use. I would say like their goal is really tries to optimize for the, what's the one most impressive thing that you can do? In a single, you know, short video recording or Twitter screenshot. And I do think like that, that really is pushing the boundaries in terms of like what is possible. And then obviously being open source, people can actually go through the source code and, and copy ideas off of that. And I think a lot of people have done that with both AutoGPT and Baby AI.

• 38:22Speaker 00

  And I think the, the closed source agent type companies like the Adets, like the, like the Lindys. Like the new computers and, and you know, the others that are the people working on here. They tend to work on very much more mundane, but daily use type of use cases. Because they're trying to work towards like, what would you subscribe for like a $20 a month, $100 a month subscription? I think both approaches are valid and I do think that you need to have representation for, from both. The amount of human intervention is something that people are trying to get a grip of. Because the way that AutoGPT does it is they basically ask you for a confirmation step before they do anything. And like, that's fine, but that's not autonomy. That's just like, you know, assisted prompting, whatever.

• 39:07Speaker 00

  What you want is to fire and forget. Like that, that is where ultimately these things have to go. You know, whereas in San Francisco, cruises and Waymos are now common modes of transportation. And literally I just get in a car, I never talk to anyone and it just brings me to a destination. That's what I want. And we might be permanently for a while, at least five to 10 years away from full self-driving in agents. Which is where self-driving was like 10 years ago. You know, so I do think like this is one of those the most speculative areas. But I have seen some of these demos live, you know, because I'm friends with most of the speakers that they're featuring. And I will say like they are useful today, so they may look trivial now, but there's active research going on towards making them more and more substantial.

• 39:53Nathan Labenz?

  I'd love to hear more about specifically like what use you have been able to get out of any of the agents. I've kind of played around with everything I've been able to get my hands on as well. And I recently called them broadly as a class like still just for fun.

• 40:09Speaker 00

  Yes.

• 40:10Nathan Labenz?

  Although I don't think it will stay that way. In fact, if anything, I would say maybe I would put my timeline to AI agents, you know, kind of crossing whatever chasm they need to cross. Probably short, if I understood you correctly, probably shorter than, you know, multiple years. I think I'd probably put my money more on like, I guess, kind of the anthropic timeline, if you will. You know, Dario has said in a couple of interviews that, you know, they're gonna.

• 40:38Speaker 00

  Two years to AGI, come on.

• 40:39Nathan Labenz?

  Yeah, something along those lines. I mean, I can't confidently rule anything out at this point. I'm not confident that that will happen. It does seem like probably the biggest question in the whole space right now is like, does the current paradigm get to the ability to do sort of insightful work of the sort that currently humans are kind of the only things that are able to do? You could come short of that, though, and still have like very useful agents, right? That can like decompose, you know, somewhat complicated tasks and reliably execute on them. I mean, there's, you know, a lot of different places on this spectrum of AI progress we could zoom in on.

• 41:20Nathan Labenz?

  But I'd love to hear your, your take on both, why you think AGI is farther off than that? And, you know, then coming in a little bit toward AI agents, or take them whatever you want, but also on the AI agent side. I would be thinking like a year from now, they probably will be working well and like reasonably commonplace. But it sounds like you're maybe not expecting it to happen that fast.

• 41:43Speaker 00

  I think this will happen in gradations. And this is one, one maybe difference between our two podcasts. I tend to not discuss AGI. I tend to not discuss timelines, even though obviously there's some implicit assumption of them in everything that we do. Just because like it's so hard to predict the future and like it's not falsifiable in any way. So it's just, it's just a fun dinner topic conversation. I, so I will say like, there are some categories which I'm more interested in than others, right? Like, so the original founding prompt of AutoGPT was I want to increase my network. That kind of category of agents, not interested in, like that's, that's too general for me. That's, that's too AGI bro sciency. But like, uh, they're very limited scoped agents, which I think are useful today. I, I have a piece on agents, which was relatively popular that I wrote in April.

• 42:29Speaker 00

  And I, and I said, actually, the most useful agents that I use today is one that has no LLMs in it at all. And that's the Savvy Cal or Callan Lee agents.

• 42:39Speaker 00

  Right? Because I, I, I send you a link and then like you schedule a meeting with me on, at your own convenience. And if you need to reschedule or cancel, uh, it just happens on your side of the fence. And it just kind of happens autonomously without me. And that is kind of the experience I just want for all my agents. And the fact that we don't have that with LM-enabled agents is a problem. And we, we're, we're gonna sort of slowly emerge to get there. The second form of agents, which I think have kind of already proven, uh, proven themselves relatively successful is code interpreter or advanced data analysis, as it is now known, because it can generate code, run that code, and then use the output of the code to decide whether or not it needs to fix that code or to stop.

• 43:21Speaker 00

  And that is the beginnings of a loop that, uh, is basically required for agents to have some level of autonomy in decision-making. And, um, if you sort of break that, break that down even further, you need, uh, some ability for planning and prioritization. You need a broader set of tools. You need memory to go with it and then you need ways to interact with the outside world, whether it's sort of generating text or manipulating some kind of UI. I think those are sort of like the agents research. For those who are interested, definitely read Lilian Wang from OpenAI. Her blog on agents, I think, is one of the most comprehensive survey overviews of the research in agents to date. So I think some categories of those will emerge, right?

• 44:02Speaker 00

  So one of the people that I forgot to mention, Deddy from Codium AI, that's an Israeli company that raised an $11 million seed for building coding agents. They only focus on test generation as an agent. To have an agent sort of independently running around generating tests in your codebase and then for you to extract, reject them. I think that's a relatively small problem that I, yeah, I'd be perfectly happy to agree with you. It will probably be useful and commonplace in a year. But things where, which require a lot more self-driving and a lot more, have a lot more degrees of freedom to fail. Those things, I think, will be waiting for them for quite a while. My last observation is, I think you'll be surprised what Linde and NoShin have to show. I can't say more than that.

• 44:45Nathan Labenz?

  Yeah, I can't wait. I feel like you have more insider information than I do and yet I want to take the bull side of a bet perhaps. Like, we need to maybe, you know, think it through to really refine the decision criteria. But I'm trying to zero in on what you think would fail that I think would work at a certain point in the future.

• 45:07Speaker 00

  Yeah. What do you think I would think would fail? And then maybe we can kind of discuss there. Because I'm positive, I just, I just, I'm much more interested in the engineering challenges than timelines.

• 45:17Nathan Labenz?

  I do think people go well past the point of usefulness on their timeline discussions and their, you know, P dooms, if you will. Whenever anybody asks me about my P doom, I always say somewhere between 5 and 95%. And my quick, you know, add-on to that is like, and I'm not sure it's that much worth trying to narrow it down further because 5% is enough to be very concerned about it in my mind. And like 5%, you know, if it is 95%, then the 5% chance of, you know, surviving is like worth fighting for. So kind of anything in there, you know, to me is like likely enough to be a problem.

• 45:55Speaker 00

  And yeah, 5% makes you a doomer, basically.

• 45:58Nathan Labenz?

  Well, the real doomers would go much higher. I don't know that they would have me in their camp as a bona fide doomer with a 5% number. Because you're multiplying by infinity, right? So 5% rounds to 1. Well, it certainly does make it for me like the issue of our time. Again, I don't try to zero in on a specific number much more than that. And I'd say same thing for timelines. Like, it seems like things could get really crazy in the next two to three years. It also seems still pretty plausible that we just kind of hit some sort of top out where it's like, hey, this paradigm kind of closes in on like expert performance on a lot of things,

• 46:40Nathan Labenz?

  but never really achieves that sort of eureka insight capability. And we kind of stay there for a while until something else happens. I definitely think both of those are true, or not true, but like plausibly true. Going back to the question of like, you know, where our expectations might differ. I mean, I guess the stuff that Lindy has shown would be like the kind of thing that I do expect to work, you know, in the not too distant future. And to, you know, describe those a little bit, it's kind of like text to automation is a lot of what the demos are. Like Flo will post something where he'll have like a pretty simple prompt. That's like, hey, every time somebody emails me, you know, from this domain,

• 47:22Nathan Labenz?

  check this other thing, you know, draft me a response, put that in a calendar invite, you know, send it here, whatever, right? And he kind of just says all this stuff. And then the platform, you know, just judging from the screenshot that he posts, interprets that, sets up this kind of automation workflow. And in theory, then it's like set up, ready to go, almost like you have a Zapier Zap that you've just conjured, you know, with like two sentences. Some of them that he's shown have been reasonably complicated. And, you know, I assume that it doesn't always work, or he'd probably have launched it by now. But I guess I think that that probably will start to work pretty well

• 48:06Nathan Labenz?

  in the not that distant future. And I guess maybe to refine it a little bit more, it seems like once you can fine tune GPT-4.

• 48:15Nathan Labenz?

  Probably should be able to make a lot of those things happen, right? If you can just kind of document the reasoning, the breakdown, the planning. I mean, GPT four is currently doing everything. If you could really just zero it in on like the decomposition and scaffolding of kind of mundane, even if like somewhat complicated tasks, feels like that would be enough to me to get it to work pretty well. I don't know. What do you think about that?

• 48:41Speaker 00

  I think it could work well. We just don't really know. We don't really have benchmarks for planning and prioritization yet. And I I've had actually debates with OpenAI people who are researching this actively. Part of my GPT four point five thesis is that long inference is kind of the next frontier of OpenAI. Like, what can you do if you gave it a year to inference instead of a few nanoseconds or milliseconds? Anyway, so I would say that you're exactly right in your your mental model of where what Lindy does and how that interacts or overlaps with Zapier. And I do think that that is very useful. It's it's still not like that impressive, I guess, maybe.

• 49:22Speaker 00

  But to everyone who's ever had an executive assistant or, you know, wished for an executive assistant, like this is going to approach what you would want as a as a sort of always on personal assistant in your life. And I do think that that will unlock, you know, a level of productivity and quality of life, to be honest, that we just never seen before. And the fact that we can pay, you know, dollars for this, I think is is useful. For what it's worth, I think Lindy's working out like it's not just reliability, but also like economics and security as well. And I do think that one of the pitches of why AI engineering is a thing is because we, the programmers, are going to be the people who put Shag off in a box, is what I is what I say. Right? Like the like.

• 50:07Speaker 00

  Us fine tuning things down to smaller models, to domain-specific models, is the safe approach. Being able to compose them into usable software systems to put humans first, instead of creating sort of potentially ruinous AGI. Um, I do think that that is a movement that I can stand strongly behind and um is is like, nobody's against it. Effectively, like, we all, we all want this to happen. It cannot happen quickly enough. And when it happens, there's not, there's no safety concerns, uh, except to the point where uh, you let agents loose on the internet with, with uh, with no permissioning. Um, and so far, I think everyone, including Flo, uh, who, who recently had a, a nice, uh, safety discussion turn. Um, I think everyone's being like very responsible about it.

• 50:52Speaker 00

  Like, if you interviewed Connor Lee, Connor's like one of the most like safety-minded people, like he split from Uber there because they, they weren't safe enough. And he has like this very strong criticism of the other GPT team. Uh, and I think he's just never met them. Like, all of them are very concerned about safety. And, and I think, uh, it is the engineers who will put Shogun in the box. That's, that's my bottom line. Um, and, and understanding how to, how to wield these tools for human benefit and not human ruin. I think, uh, it, it, like, the, the responsibility lies very heavily on the engineers.

• 51:25Nathan Labenz?

  I think I agree with everything you're saying there. I often describe myself as a adoption accelerationist and a hyper scaling pauser. In other words, like, let's get the benefits that we can from our current systems. Or, you know, I think we're gonna go a little bit farther than current, and that's probably okay, you know? But I'm very nervous about what happens if you put a million times more compute into a model than was put into GPT-4. Cause I just feel like we have zero ability to, to know. Reminds me a little bit too of Eric Drexler's comprehensive AI services vision.

• 52:03Nathan Labenz?

  You know, just the idea that if you have narrow, perhaps superhuman, but still just fundamentally narrow systems doing all kinds of jobs. In a way that can kind of become a buffer for the world against more generalist systems. And, you know, as such, like, it's, seems really good to try to create these narrow systems that hopefully work really well. Is there something that, going back to like, what do we expect to work and not work? What do you expect to not work? Is there something that you would say like, yeah, two, three years from now, I still think, yeah, we're definitely not going to have an AI that you can get to do X?

• 52:38Speaker 00

  Oh, God. That's so.

• 52:40Nathan Labenz?

  I have a tough time answering this one, so.

• 52:42Speaker 00

  Yeah, yeah, putting upper bounds on these things. Well, no, one of the reasons I pivoted is very, very hard into this thing is I have to sort of updating algorithm. Whenever I've been wrong, I ask what else I could be wrong about. And then I progressively update. It's kind of my sort of Adam optimizer version of momentum based learning. But so effectively, I, so I think I'm very skeptical about code agents that do PRs. So there's like sweep dev and then there's a, there's a, there's a bunch of other agent type companies that will promise, you know, file an issue will turn into a PR, right? And it always, always works on like very, very simple sort of copy change demos. And it never works on more complicated things.

• 53:24Speaker 00

  And I do think basically like the, the problem of software architecture and translating user requirements into code. I would, I would be happy saying like, that's a five years out thing and like it's not an immediate solve thing. I mean, I still think it's a very worthy problem because AI engineers, instead of AI, like, yeah, engineers drop title, but like AI, like primarily AI engineer. I do think that's a very useful goal and worthwhile goal because, you know, engineers are expensive. And if you can pay an AI bot, that's a hundred times cheaper than a regular human engineer, but is maybe 10% of the capabilities, then you'll probably use it for quite a few situations. I just think like, yeah, the, the whole promise of like issue to PR

• 54:05Speaker 00

  is probably going to be overhyped for a long time, just because the complexity of what software is. You know, I'm very happy to eat my words because I, and there are quite a few friends pursuing this, this goal. And more power to them, right? Like all, all human progress is, is based on unreasonable people.

• 54:23Speaker 00

  I think Meta GPT, I don't know if you've covered that paper recently. I think Microsoft also had a similar paper come out where effectively you have AIs play different roles in a sort of agile scrum process in working towards a common outcome in software. Has shown better human eval results than GPT four. But again, like you're fundamentally not describing the full complexity of what software systems want to optimize for. And I don't think we have nearly enough training data to do that. Like it's very much in the world of intelligent design rather than evolution.

• 54:58Nathan Labenz?

  I think we can maybe refine this a little bit more offline, but I think I would take the other side of that bet. I mean, there's, you know, how big of a pull request and, you know, how big of a code base and I think obviously for it to be interesting, you'd have to have enough tokens that it would be like beyond what a Claude two can currently handle. But even so, you know, it does feel to me like I would say in two to three years, it does feel like I would expect us to get there. And still on the basis of like, maybe there's no like eureka moments. I had an episode not long ago with the Med Palm M authors from Google, where it's like, they're really closing in on expert level

• 55:41Nathan Labenz?

  medical question answering, the ability to take a X-ray or another scan and provide like a radiology report at very close to a human level. Forty percent of the time their Med Palm M X-ray report was preferred to the actual radiologist. So it's still under fifty, but it's like, damn, that's getting really close, you know? So if we can get that far already on problems that hard, I do feel like in a, you know, in a two-year timeframe, some pretty complicated. Kind of prompt to PR type things should also be possible. So we can maybe firm that up and put ourselves on record for a little friendly wager.

• 56:23Speaker 00

  Look, there's, there's, there's a, there's a bit of undefined thing here, which is like, how much do you have to specify things in order for a decent PR to come out? And will, will that prompting effectively approach a new programming language? Right? Like, you know, English is the new hottest new programming language. But like to spell things out in some level of detail to which the, the, the L M's will get it. Like, are you, are you, are you basically just, just programming, but in a different, higher level assertion that is kind of like a domain specific language for L M's? So yeah, that, that is open for debate. And I do think like some of the approaches, I think I've seen G P T engineer kind of approach that with their sort of pipeline YAML thing. Does that count? I don't know, right? Like, is that's kind of pseudo prompting, that's pseudo programming already.

• 57:06Speaker 00

  In my mind, I want a non-technical person to just like manage a software project by themselves. Right? Just filing issues. I would, I would say that's my bar for like issue to PR as well. Non-technical person. But if you need a technical person to like come up with like some interesting YAML file and like prompts in specific ways, then it's not, your hand is too much on the, the, the, the weighing scale at that point.

• 57:28Nathan Labenz?

  Yeah, okay, I think that makes sense.

• 57:30Speaker 00

  I, I have one more thing, one more topic to discuss, which is probably going to be one of the most talked about talks that is coming up, which is OpenAI just confirmed their talk with me. And we're going to have the first public Dolly three and G P T for vision talk. I, I'm pretty excited about that because that, that is the signal of the finally, we're moving into multimodality for everything. It's going to be a world, world, worlds. And I don't really know how to handle it because most of the people I have on, they're all in the tech domain and we're going to be very off balance when, when new modalities come out.

• 58:04Nathan Labenz?

  Yeah, I got a little glimpse of that last week in trying to add a feature to a React app. And it was a fascinating experience. I have never, prior to this, you know, session last week, programmed in React at all. I've done JavaScript before, but it's been a while since I've really been into it. And so, you know, I, my first question was, can you explain to me the general structure of a React app? And then it did, and then it kind of, you know, I, there were like a couple of file types in my repo that I asked it about, and it explained to me, you know, what those are and said, oh, well, that means you've got these libraries in place, and that's what their purpose is. And then I asked it

• 58:45Nathan Labenz?

  if it could give me a command that I could use to print out the tree structure of all my files, so I could paste that back in, so it could see the structure of my project, so it helped me do that. And then, you know, basically just kind of worked my way through to a working module. And along the way, because the new image, I didn't really have to do this, but along the way, took a couple of screenshots, fed those in as well. I was like, man, this is really amazing. You know, it's kind of understanding like what the defect is in the screenshot and, you know, modifying the code based on, in part, on the screenshot. It definitely is still at the point where like, if you looked through this transcript, you'd be like, Nathan, you qualify as a technical person here. This is not, you know, at the level that you're describing. But you can see, you kind of squint at it

• 59:30Nathan Labenz?

  and see your way there with these screenshots where like, I want it to be doing something else, and here's what it is doing, you know, and if it can start to deduce or infer, you know, the, the right fixes based on just like, the visuals of the app not working, then it does seem like, you know, you're not too far away from a non-technical person, maybe not as efficiently, but still kind of being able to like, get, you know, something along the lines of what they want, you know, which today is just like unthinkable. So.

• 1:00:00Speaker 00

  For, for those interested, I think McKay Wrigley has been making a lot of us jealous with the GPT-4 vision access, with some examples of what he can do from like, you know, sort of the vision to code to modality. It's not just vision, right? It's also voice. And part of my update that I, that I did for September, and this is something that Ben Thompson of the Strict Techery Podcast, I think he's, he's now talking on the Sharp Tech Podcast. He had, he had, so he had a private demo of both modalities, and I've seen both in person as well. And I kind of agree with this point, which is vision is more of a Impressive individually, but voice is the thing you're gonna use every day, all the time. It's kind of an interesting inversion of expectations, because I would count vision as my most important sense out of the six senses or five senses.

• 1:00:45Speaker 00

  But voice, the ability to hear with, it's hands-free and it's just always on and understanding what you want. I think that's kind of a thing that I see a lot of people here in San Francisco building with. And so my perception, I used to dismiss voice. I was like, oh, like AutoGPT has like an eleven labs, that's just for fun. Like, who wants an agent talking back when I can just kind of read the text by myself way faster? But the fact that it's hands-free, the fact that it's just always on, and the fact that you can imitate some of human expressions, I think it's gonna be remarkable. And I don't know if you've seen, have you seen the Russian language teaching demo of GPT-4 voice mode or ChatGPT voice mode?

• 1:01:25Nathan Labenz?

  No, I haven't.

• 1:01:25Speaker 00

  So Greg Brockman tweeted this, I mean, maybe like two, three days ago. Duolingo should not exist anymore. This thing is gonna teach you languages on a personalized basis, much better than any other app in existence. Any human teacher, just because it's way more patient and way more forgiving, probably can understand, you know, multiple languages way better than you can. So I think it's gonna be a very, very interesting time for understanding all these new values, because I'm sure they're working on more that they haven't told us.

• 1:01:54Nathan Labenz?

  Yeah, so you're kind of the, you know, Duolingo shouldn't exist anymore. That is kind of what I wanted to take your temperature on next, because I feel like in some sense I've seen like a version of this movie before, where, you know, going back to early Facebook days, for example, I was, I just happened to be in the same dorm as Zuckerberg in college, and I actually thought it seemed pretty dumb at first. I was like, I'm gonna go online and put a picture of myself. So I was a late adopter, actually, of Facebook on campus.

• 1:02:26Speaker 00

  Fun fact, Matt Walsh from Sixty, he used to be Zuckerberg's CS50 professor. I don't know if he's ever told you that.

• 1:02:32Nathan Labenz?

  That did not come up, actually. That's fascinating.

• 1:02:35Speaker 00

  So in the social network, like the class that Jesse Eisenberg walks in on and then like solves the question and he walks out, that was him, that was Matt Walsh?

• 1:02:44Nathan Labenz?

  Okay. Small worlds. Yeah, that's really funny. You know, Facebook initially had this like, come one, come all, build on our platform. You know, you get access to everybody's friends. Everything's better with friends. It's all going to be social. And then it kind of became clear over time that like, actually, we're going to build the most high value use cases natively on the platform. And, you know, the app developers can kind of like, get a little bit of stuff around the edges. But today, basically, you can like, log into Facebook and that's really it, right? There's not like, a lot of utility in the Facebook platform. I'm wondering if we're starting to hit a moment where OpenAI is going to go a similar

• 1:03:27Nathan Labenz?

  direction and start to kind of eat these adjacent use cases that are currently other companies. And so like, companies we've talked about, right? Like Human Loop. I love the product. We're both fans, but it sure seems like from an OpenAI standpoint, you know, they just kind of released the like, fine tuning jobs. Yeah. It was like, well, what are you doing there? That's starting to look a little like Human Loop, right? And then, so there, I mean, there's a ton of stuff that they could do there and it makes total sense. And it's like, not, they wouldn't obviously be doing it because they're hostile to anybody else in the ecosystem, but just because they're asking their users, you know, what do we, what do you need? And they're just executing on that. And these are things that people need.

• 1:04:10Nathan Labenz?

  And then I'm kind of like, the same thing might start to happen with, for example, text to speech. You know, we've got some really good companies that have created amazing stuff. I've cloned my voice with both PlayHT and Eleven Labs. They both sound awesome. Do you have a favorite? I don't know if you're willing to go on the record. Yeah, I think it does depend on use case. I would say Eleven Labs is probably the favorite for ease of use right now. But if I was doing something more artistic or kind of creative or wanted more emotion. Yeah, Play seems to have a lot of diversity. Yeah, the, the creative direction that you can give it is a little bit more open. So I think, you know, if you're analyzing this from kind of a, who's OpenAI posing

• 1:04:52Nathan Labenz?

  the biggest problem for, in that analysis, it would be Eleven Labs probably because OpenAI presumably is going to be more reluctant to like, Create the voice that's gonna, you know, feed into your horror movie or your video, you know, your shooter video game or whatever, where you're gonna want kind of a wide range of emotions, at least in the short term. But it seems like quite clear that they're headed for a, I would, I would imagine, tell me if you think this is off, but it seems like, you know, when I think about this dev day coming up in a month, I'm like, well, geez, they just launched voice in the app, they just launched vision in the app, you know, they just launched this preview of fine tuning, they've got vector database built in natively to ChatGPT Enterprise.

• 1:05:37Nathan Labenz?

  Seems like all these things are coming to the API. Sorry, I haven't heard about the vector database. What details are there? I don't have a lot of details, but they have said publicly that, you know, you're gonna be able to connect ChatGPT to your information, right? As an enterprise customer. Basically, what I understand that that translates into is you can connect your Google Drive or your Notion or your Dropbox or whatever to their system. I understand they're gonna support a number of them out of the box, you know, kind of with their own connectors, and then they, you know, presumably will have some way for people to connect their own, you know, random data sources into the system as well.

• 1:06:20Nathan Labenz?

  And, you know, as a ChatGPT Enterprise user, then you can like query all your company's Google Drive assets as part of the ChatGPT Enterprise experience. I don't know exactly where that is in terms of whether it's launched or not, but they've sort of said it's coming, and that much, I think, you know, is not like super secret. Beyond that, I don't have a lot of details. But I guess I just feel like all this stuff is coming to the API, right? And it seems like they're gonna do this to serve the AI engineer, because, you know, the AI engineer doesn't really want to have to go piece five different services together. Like, you know, today, if I want to create a voice assistant, I've got to like transcribe,

• 1:07:03Nathan Labenz?

  send something into the language model, get text back, convert that to audio. And, you know, what would be a lot better is if I could just send audio into OpenAI and get audio streaming. Back at me, right? And similarly, like today, and I've done this and you've done it too, right? We've got to go select a vector database provider and figure out how we want to chunk our stuff and how it gets loaded in there. And then what's our query strategy? And you know, that is its own art, you know, to figure out like, how do we generate the query for the vector database to get the right stuff? Because often like the question the user asks and the actual contents that you're trying to, you know, find similarity with, those maybe not, don't line up so well. So there's like, whatever, all these kinds of different

• 1:07:46Nathan Labenz?

  tricks and techniques. And again, I, what I really just kind of want is like, one place to be like, here's my docs, here's my question, I'm putting it in as an image and an audio file. And by the way, I want my response back as streaming audio and just have kind of OpenAI handle everything, right? Like, does that seem right? It seems like everything's, they're going to kind of just eat all these sort of things around them. And the AI engineer benefits. But a lot of these companies, you know, I wonder like, for a lot of the speakers that you have, as much as they have led, you know, how do they end up not having kind of been the, you know, community R and D department for OpenAI?

• 1:08:31Speaker 00

  Look, every large enough platform eventually comes into this tension with their developers. And OpenAI is just coming into that right now. Apple has this long history of Sherlocking, which is an official term, I think, where, you know, some, some very successful community app gets killed effectively by an official Apple version. And I do think like, you know, OpenAI has promised to not compete with, with its own developers. They have said that, funny enough, in the deleted transcript of the interview that Sam did with Raza, which I, which is still publicly available. They promised to not compete with developers, but I don't think that promise is ironclad. I don't think they want to stand behind that.

• 1:09:17Nathan Labenz?

  That reminds me of the dumb and dumber scene, not to call anybody dumb, but the, you know, these are just as good as money. These are IOUs. And it's like, yeah, that was easy to say at that roundtable, but because they're at a thousand people now and, you know, they've, they've got product managers.

• 1:09:33Speaker 00

  Six hundred.

• 1:09:34Nathan Labenz?

  Six hundred? Oh, okay.

• 1:09:35Speaker 00

  Yeah, I mean, I happen to have very life stats, so I know. I would say that they always want to focus their people on the bigger things, right? Where, you know, there was this recent discussion about the OpenAI phone. This is what I've been calling it with Jony Ive. Like, working on a phone, that's big and ambitious, you know? Working on, you know, A/B testing, fine tuning infrastructure for the enterprise, not that ambitious. Leave that to the lesser engineers in the community to pursue and, you know, I think, go for the big things and obviously try to solve problems that everybody has. So I would say like, I think, yeah, OpenAI does, does exist in the tension with the developers that go on top of it. Probably some of the companies will be Sherlock's. I don't think 11 Labs

• 1:10:20Speaker 00

  would be a threat. Like, the best thing that startups can do, building around the sort of foundation model lab ecosystem, this is not just OpenAI, it's also all the others. It's just prove that you're a really good engineering team. Because if you're good enough, they'll buy you and, you know, pull you into their, their orbit. Like, money is free to these people. Like, it doesn't actually matter. Are you choosing to work on interesting problems? And do you ship fast enough that you're making an impact? And I think 11 Labs has definitely shown that they can do that. And hopefully, you know, enough of the teams that I'm seeing can do that, but not everyone will make it, right? Like, there's definitely going to be collateral damage. OpenAI, you know, has such a huge footprint that it cannot help

• 1:11:01Speaker 00

  but realign people. Like, when a lot of what LangChain and Barrios used to do was GSON output validation, right? And like, now that it's completely useless because of OpenAI. And they just have to roll with it. And I think everyone's completely fine. We all understand we're building on top of shaky foundations and it's kind of moving, but that's what you get for being first. You know, in terms of the multimodality stuff that you talked about, I do think like that will be a focus for Dev Day. I don't know if they'll be, they'll have the sort of APIs ready by then, but we'll be able to see what you can build with it in the talk that Simon and Logan will be giving. But this comes back to Rune's post on text being the ultimate universal interface. I do think like text is sort of the king modality. Yes, like.

• 1:11:44Speaker 00

  Whatever species synthesis is being done, whatever sort of text to image or image to text stuff that they're doing, like everything just kind of just passes through text and being working on text is like the central pipeline. I do think it has some links to it for the time being. Very happy to eat my words. Ultimately, everything is just tokens within a token space. And text has is no particular domain of it in there, except that we have much more of an understanding of what high quality data looks like in text than in everything else.

• 1:12:13Nathan Labenz?

  Yeah, it does seem like if I had to guess, I think things are going to get a lot easier for the AI engineer over the next couple of months with just some really integrated kind of high quality things, you know, such that kind of end of this year, you could probably feed in multiple modalities. You could kind of have them manage your custom data set powering retrieval, but you know, powered by them. And then they can kind of stream out to you. Text, yes, of course, but audio on top of that, perhaps even, you know, images integrated into the responses. And that is going to be quite a difference and quite a simplification.

• 1:12:54Nathan Labenz?

  We've got GPT four fine tuning as well, which is like another huge one because to the degree that you have a use case that you can't get to work well yet with any current model, you know, this could be really the thing that that allows it to to become possible. Things that OpenAI might watch that like, wouldn't compete with anybody in the ecosystem, but which might actually just be, you know, a win for everyone. One concept that I had there is login with OpenAI. And I think this is like, potentially really good for a lot

• 1:13:34Nathan Labenz?

  of developers. If you are struggling with this tension between I want to show off how sweet my product is. But to do that, I'm like taking a non-trivial hit in aggregate in terms of token cost. At Waymark, for example, my company, we estimate it's about 15 cents per user that tries the free product in hard cost to us. Some of that is OpenAI language models, some of it is image understanding. We have some self-hosted stuff on Amazon. We try to process all their images that we possibly can to create the best profile we can of the user, et cetera, et cetera. I'd rather that the user could kind of pay that free cost, you know, instead of us having to pay it. But obviously, they would need some

• 1:14:19Nathan Labenz?

  mechanism for that. Do you like that idea? And how would you riff on that idea? Do you see that as a plausible thing that they might introduce?

• 1:14:29Speaker 00

  Absolutely. I think it makes sense. I just don't know if it's meets their bar for interesting work. I think logging with Google is effectively this equivalent thing for logging with OpenAI. And but also like implementing SSL provider, I think it's relatively commoditized to begin with, then gets very complex over time as you're starting to manage non-trivial amounts of user data, which maybe OpenAI doesn't want to get into business of, right? It's more just about choice or alignment with the mission. And I don't know if that helps them build AGI. I think it's a nice quality of life thing for them. Maybe they can get more information as people sort of, you know, use you sort of universal logging with OpenAI. I do think that if they do build this, then they have effectively

• 1:15:14Speaker 00

  decided to become the AI cloud. I think they maybe own AI.com. Don't quote me on that. It might be them or Elon. This would make them the fourth major public cloud, which is very exciting, but also maybe not like quite a research organization. And again, I still don't know if they want to do that. I do think it's like kind of a choice of paths and they're fortunate enough that they have that luxury and capability. There's many things that they could point their engineers at, and I don't know if like building SSL for AI is like one of them. Could you just kind of offload it to Octo, OffZero, whatever? And someone else could do that as well. So I will say one thing, which I think I like as an idea, which is sort of context that kind of lives with you, right? And so I think

• 1:15:56Speaker 00

  you maybe mentioned this a little bit, that does seem to help. But probably actually a character he added does that, which which has by far the the longest history with working with this. Maybe not so much an OpenAI maxi in this regard. I I tend to describe myself as quite an OpenAI maxi with regards to L M progress. But with regards to like products, I I feel like character has just kind of earned its stripes in in trying to be a chatbot much more than OpenAI has has done, even though they haven't had, you know, sort of state of the art new models to to ship. I don't necessarily want to be in OpenAI maxi, but it's hard not to be. I write a monthly recap, right? And then my OpenAI news section takes two two screens. And then my

• 1:16:40Speaker 00

  other frontier model, you know, the you know, on the frontier model model forum, I have I have other frontier model updates. And it's like five bullet points. Like, it's it's not a competition. Like, OpenAI is just running away with it so hard. Yeah, I mean, it's just being being objective makes you an OpenAI maxi.

• 1:16:57Nathan Labenz?

  Competition right now is just not that deep, right? I mean, you know, we said just before we started recording, we got to give love to Claude 2. I would say Anthropic has definitely shown that they can create really high quality models that are not like qualitatively outclassed by OpenAI's, even though they do fall, you know, a little bit short on like the MMLU benchmark and whatnot. But it's like, definitely a very good experience to use and something that I do, in fact, use, especially when I need long context or whatever. But they kind of have a strategy of like not leading the market in product releases as far as I know. And so there really aren't that many other competitors out there right

• 1:17:37Nathan Labenz?

  now that could could bring a similarly robust and just useful model forward. I mean, Google is not there. They are maybe going to get there in the not too distant future. But as of now, you know, the the models are just not as good. And, you know, who else is there, right? So.

• 1:17:56Speaker 00

  Done enough, spent enough time on Palm two to actually definitively make that statements. I want to believe in Google. I like, I've met people there. They're very smart. They have all the resources in the world. They just don't have the institutional capacity to leadership, something interesting for people, unfortunately, so far.

• 1:18:15Nathan Labenz?

  I think the research is not the limiting factor there. If I had to guess right now, it's certainly not the compute either. So, and the fact that they didn't participate in the last Anthropic fundraiser and allowed Amazon to kind of set that.

• 1:18:30Speaker 00

  Amazon just bid way higher than anyone else was willing to bid. Anthropic is probably worth like 10 billion now, something. And Google, I don't think, they probably want to invest in-house. I mean, I've talked with people involved in the previous rounds and they said it was much more hands off. So like, Google in no way owns that relationship as strongly as Microsoft and OpenAI were.

• 1:18:50Nathan Labenz?

  My biggest update from that was just that Gemini must be on path to work pretty well, because if it weren't, then they would be bidding, you know, as much or more than Amazon, right? Like, it seemed like they must have had some, you know, inside track based on their earlier relationship, even if it was fairly, you know, just kind of friendly, arms length, whatever. If they didn't have confidence in their upcoming releases, then hard to imagine how they'd let that one go.

• 1:19:17Speaker 00

  Yeah, who knows? Who knows? Battle of the Titans is they, hopefully the tell-all books that someone will write in 10 years. Maybe Walter Isaacson's fill arounds.

• 1:19:26Nathan Labenz?

  People are wearing their rewind pendants, so they're capturing everything.

• 1:19:30Speaker 00

  One of the themes that I featured for my monthly recap yesterday was the AI core crux is what I was saying, or like AI pendants, which I saw a demo from Avi Schiffman of TAB yesterday. I've also talked with a number, a number of people also working on kind of similar projects. I would be willing to wear one of those. So I have an aura on me right now, and it's just kind of logging my heartbeat and whatever. But I'd love to log all my conversations and be able to talk with it. These things are coming. And yes, they will be multimodal, but also like, I think all of us having an effective digital twin that we can talk to and use as at least just a note-taking thing, if not just exposed to the wider world. I'm very excited by, and it will be built by the AI engineer.

• 1:20:14Speaker 00

  So yeah, I will feature like a probably a an entire stage next next quarter on I mean, you know, the next time I do my summit on on these things because I think in terms of like the BTC frontier, if we're half a form factor that is beyond the phone, it's going to be the humane clip or the AI pendant or the watch or whatever it is that's just always on you.

• 1:20:35Nathan Labenz?

  Yeah, that definitely seems like it's coming before too long as well. So last thing I wanted to ask you about, and I've been kind of asking everybody about this, listeners to the show will have heard me pitch it before, so I'll try to do it super briefly. Concept is the AI bundle. And I'll give you just kind of brief inspiration from the consumer side and from the app developer side. Consumer side, you know, pretty obvious, right? I've got chat GPT Pro, I've got Claude Pro, I've got Copilot, I've got Repli Ghostwriter, I've got Perplexity Pro, I've got, you know, other stuff that rewind and, you know, all these things, right? And my AI, my monthly AI bill is like getting up to the point where it's at or north of what my cable

• 1:21:15Nathan Labenz?

  bill used to be. It notably doesn't cover hundreds of other apps that I like might like to use, but you know, they're all like another 20 bucks a month or whatever. And I'm like, man, do I, how many of these like $20 a month things do I want to buy? Right? So that's the consumer side. On the application developer side, just, you know, representing myself as Waymark, we have this, you know, easy to use video creator. It used to be DIY. Now it's done for you with AI. You tell us what kind of video you want, tell us who you are, we make the video, you just get to sit back and watch it. It's awesome. But we do have this kind of weird tension where, as I said, it costs us 15 cents for every user. We don't want to put it behind a paywall because

• 1:21:57Nathan Labenz?

  we want to show it off. You know, people are most likely to buy if they can try it. But then the flip side of that is like, you know, we have the kind of classic, whatever, $30 a month plan, and we need

• 1:22:09Nathan Labenz?

  One person to buy for every 200 free trials, just to break even on our tokens, right, before we cover any other cost or make any money whatsoever. So that's a tough situation because it puts us in this weird thing where we're like, when do we gate it? How do we gate it? And then we also see, of course, the behavior that SaaS app developers hate to see, which is where people, you know, do the thing they want to do and then immediately cancel. And there's been, you know, increasingly a lot of talk about this, like churn is pretty high. Everybody's getting a lot of new customers. Interest is really high, but churn is also pretty high. On both ends of this, it seems to me like there could be real value in some sort of bundle. And what I want, I think as a consumer and potentially also as an

• 1:22:51Nathan Labenz?

  application developer, is something where it's like, hey, pay 100 bucks a month and get 1,000 AI apps. And those would be like your flagship apps, like maybe your GPT Pro, and then maybe, you know, a bunch of long tail apps like a Waymark, right? What I figure is, if I'm Waymark, which I am, and I'm one of 1,000 apps, and I'm like, okay, look, I know that, you know, there's a lot of things going on. Most, most people don't need us that often. You know, we've got some power users, but we also got a lot of people that just, hey, just one off here and there, right? And they buy and they immediately cancel. And it's not because they didn't like the service. It's just like, I don't want to pay for this every month. So if it was $100 bundle and we were one of 1,000 apps, let's just say we got the

• 1:23:36Nathan Labenz?

  average, which is 10 cents a month per, you know, app subscriber, kind of in the same way that like ESPN Classic, you know, gets a very small fraction of my cable bundle. I think we would do that, you know, and I think we would be, we could still kind of upsell the power users into higher tiers or whatever. But, you know, for every million, if it was, if that was the economics, every million subscribers that the bundle had would represent a million dollars in annual revenue to an app like Waymark, even just getting, you know, 10 cents per month per user. Can you see that happening? On the supply and the demand side, it seems like there would be a lot of utility there.

• 1:24:19Nathan Labenz?

  Obviously, it takes some doing to pull that trick off. But do you see that being something that could make sense? Or if not, like, where would it break down?

• 1:24:28Speaker 00

  The closest equivalent would be set app. Are you familiar with set app?

• 1:24:32Nathan Labenz?

  No, I don't think so.

• 1:24:33Speaker 00

  So there are existing software bundle type things out there. And they do bundle subscriptions. And I do think that they provide some value for a portion of the population. But I don't know how big this is. People are pursuing the sort of bundle economics play in traditional SaaS. And maybe you can have an AI bundle SaaS play. I would say some of the core questions would be, do you get paid regardless of whether or not people use you? And I think that would very much affects whether or not the the extra million comes to you or not. I don't know if you have a quick answer to that one.

• 1:25:07Nathan Labenz?

  Yeah, you know, I mean, I think it could be managed in any number of ways. Obviously, for whoever is managing the bundle, you'd want to have some rules or framework in place to keep it somewhat sane. You know, just using the cable bundle as kind of the jumping off point. There's certain channels I never watch, you know, there's and other people don't watch the channels that I watch. And my understanding is they all get a per subscriber share, regardless of, you know, whether or not I tune in. But then when it comes up for renegotiation, then there's like different levels, of course, depending on like how popular you are. I imagine something like that, perhaps like managed even purely algorithmically could work. I think you'd have to have some kind of baseline.

• 1:25:48Nathan Labenz?

  If you're in, you're going to get something from this to make it appealing to the app developers. And I definitely would see it kind of where the actual money flows, I would see shifting around over time with shifting usage patterns.

• 1:26:02Speaker 00

  Yeah, exactly. So I would say like there, there's difficulties there. There's difficulties with regards to the differences between like what a bundle of a newsletter or a bundle of content would would look like. I would say that the people who have probably thought about this the most is Nathan Pichard, who has now started Lex Page as as as an independent startup. And Ben Thompson of Stratechery, who's written a lot about the media and cable bundle. So they would be much more clear thinkers on this than I am. And I do like the sort of magical bundle math, right? They say you get more choice, you pay one fee. And then the people also have, the suppliers also have more predictable income.

• 1:26:41Speaker 00

  And the share and access to income without, and sort of pooling that distribution in terms of getting people to pay and then getting to people to understand what they are and try them out. I will say like for SaaS, the economic difference between SaaS and content businesses is that SaaS people typically really, really want to own their customer base. When I want to change pricing, like, do I have to get your approval as Mr. Bundle Guy? And I'm not, you know, I'm not really gonna be happy if you push back on something. And so there, I think there's a lot of these internal politics that happen when it comes to B two B. And then like pile that on top of, you know, I'm very immersed in the B two B or enterprise infrastructure environments in San Francisco.

• 1:27:25Speaker 00

  Like I, like, that's where my background is coming in terms of developer tools. And all of them are not going after the ten, twenty dollar a month subscription, right? They want to go after the ten K, one hundred K, a million dollar a year contracts that they want to get from other businesses. So effectively, this is a rounding error for them.

• 1:27:47Nathan Labenz?

  Yeah, I would not expect like your human loop kind of infrastructure players probably to get into something like this. Yeah, it would have to be kind of the, the waymarks.

• 1:27:57Speaker 00

  Consumer B two C, yeah.

• 1:27:59Nathan Labenz?

  Yeah, all these, you know, there's a million of these, like, I recently had one of the founders of Gamma on the show and, you know, just, it's like, I don't make that many slides, but when I do, it's cool, right? But I feel bad if I'm just using their free version and, you know, not paying them anything, but I'm also like. I don't necessarily have, you know, hundreds of dollars to drop on every app that I'm interested in trying on an annual basis. So, you know, I either kind of sign up and cancel, or I just don't sign up. It's probably not going to happen, but I kind of, uh, something about this feels like it should happen to me. So I kind of can't, uh, let go of it just yet.

• 1:28:32Speaker 00

  Well, the, the real question is, what is the minimal viable version of this that you can test to, to find out, right? Like, and that's something that one of my mentors, um, has always given me as, as sort of just life and career advice, which is whenever you feel like you have this idea, or whenever you're stuck with a fork in the road, what is like the minimal viable step to gain more information for you to be sure? Because I think you could sit there and, you know, have this like, it could work, it could not work, I don't know. Like, the only way is to take steps. And so like, I don't know how the size is down because like, there is a sort of appeal to scale of these bundles, but, uh, maybe like get five things together and then see, see if that works.

• 1:29:11Nathan Labenz?

  Yeah, I think one of the, the big challenges with this is it does feel to me like you would need anchor products in the bundle to make it compelling. If somebody said, hey, you know, for 10 bucks even, you know, you can have 300 apps you've never heard of, then I'd be like, eh, I'm not sure I need that. But for a hundred bucks, if I could have, you know, ChatGPT Pro and Claude Pro and, you know, Perplexity Pro and a bunch of other things that I've never heard of, then I'm like, yeah, I might go for that because those three, I really, you know, I already know I want. And then, you know, whatever else, I'll kind of discover along the way. Anything else you want to talk about today before we break?

• 1:29:52Speaker 00

  That's a real pleasure. Um, I have just been, uh, enjoying the podcast. Uh, I would say just keep up what you're doing. There's always room for these sort of long form deep dives into AI. And, um, I, you know, I love the passion that you're taking to approach the space. I'd love to have you on at my next, uh, conference.

• 1:30:08Nathan Labenz?

  Well, thank you very much. Uh, obviously the feeling is mutual. Why don't you, um, give us the details of the summit and, uh, what people can do if they still want to get a ticket and get there?

• 1:30:19Speaker 00

  Yeah, I mean, tickets are completely sold out, but you can get an online ticket where it's basically the livestream plus the Slack community discussion at ai-engineer. I really love these short domains, by the way. So just like, yes, engineers are TLDS. We bought ai-engineer. So we have ai-engineer summit, user dashboard. The podcast is latent.space.

• 1:30:39Nathan Labenz?

  Cool. Well, this has been a lot of fun. Swyx, thank you for being part of the cognitive revolution.

• 1:30:44Speaker 00

  Thanks for having me.

• 1:30:46Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.