Ep 26: Replit Founder Amjad Masad on the 1000x Engineer, ChatBots are Overhyped and Why We Don’t Really Have True Open-Source Models

Replit raised nearly $100M at $1.2B valuation last April and powers over 20M developers. On this week’s episode of Unsupervised Learning, we sat down with Replit Founder and CEO Amjad Masad to talk about the future of software development, how Replit is empowering young users, how Replit developed its own models, and the data advantage Replit has. Amjad also shared his takes on why he’s bullish on agents, where the value in AI will most likely accrue and why open-source models might not be truly open today.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  This is unsupervised learning, Red Points AI podcast. I'm Jacob Ephron and I'm joined by my co-host Pat Chase. And today we had an awesome episode with Amjad Masad. Amjad started his career as one of the early hires at Code Academy and then founded Replit, a really interesting company focused on getting the next billion developers into the space. Replit is at the forefront of using AI in a ton of interesting ways. They've raised over $100 million at an over billion dollar valuation. And Amjad hit on a bunch of things, including the future of agents. He's actually quite bullish. What it will mean to be a software engineer, given all the progress in AI coding models. We hit on why Replit actually built their own model, spent $100,000 to build a really massive model. And we also hit on why open source models aren't actually that open today. I think folks are really going to enjoy this. Definitely stay tuned for Pat and my debrief. And without further ado, here's Amjad.

• 0:42Jacob Effron?

  So to start, I mean, you're obviously at the forefront of a lot of these AI coding solutions and where the space is headed. I'm curious, you know, what advice you'd have for someone that's starting learning coding today? Is it something they should or shouldn't do? Yeah, how would you think about that?

• 0:57Speaker 01

  I actually, my answer to that would be the similar pre-LM would be similar to that. I always believed that, you know, the way to learn coding is to make things, which is nothing groundbreaking as a conclusion, but, you know, the sort of academic way of going to university and studying the basics. All of that's like, there are certain type of person that learns that way. But actually, I don't think most people learn that way. Most people will have some kind of goal. And in the way towards achieving that goal, they happen to learn things the long way in order to achieve that goal. Which, by the way, is, you know, is the, is almost the definition of intelligence, right? And you have a goal and you kind of learn to get to that goal.

• 1:40Speaker 01

  Obviously, our education system inverts that, where you go in this very undirected way of learning random facts that maybe someday would apply, what actually turns out 90% of the things you learned, you never apply. So my, my view has always been the reason coding has been more niche than it should be. I mean, what is it like, less than 0.5% of the global population, you know, has any exposure to any kind of coding. You would expect it to be a lot more, given how prevalent computers are and how powerful coding is and the ROI you get from coding, whether

• 2:23Speaker 01

  Pursuing as a career, but also as a tool. It's incredibly enriching to your life. And so there's this paradox, why isn't more more popular? And, you know, I worked at, actually, my first, kind of my first job out of college. I was founding engineer at Code Academy. And and Code Academy's whole thing was actually the best way to learn how to code. It's so like actually make things and and do it by learn by doing. L L M's just takes that to to to this extreme conclusion because actually you can you can get something running in like five minutes contact with an editor that is L L M powered.

• 3:09Speaker 01

  Right? And and that

• 3:13Speaker 01

  that makes it so that you're you're not immediately faced with all sorts of I T drudgery and tasks of installing things and figuring things out and trying to get something running. If you go to Replit, you could start by prompting. Or you could start by forking an existing template, a website, personal website, and then prompting the A I to do a basic edit. And then you get that dopamine hit of running it for the first time and seeing your results.

• 3:42Speaker 01

  And then you want to, you know, do another thing. And then you have an idea for a bigger project and and it's all messy, right? It's like you start a template, you fork a template, you try something, you hit an error, you hit something, you throw that out. You go to GitHub, you, you know, grab some code, you go to Stack Overflow, you talk to an L L M, you talk to multiple L L M's. And I think that's how it should be. And so I think really L L M takes this idea of learning by doing to to its conclusion.

• 4:09Jacob Effron?

  Do you think that the skill set of like a of a software engineer then needs to look different? Like if you were, you know, advising someone who's in middle school right now and thinking about like a long term career in in in software? Like is there, you know, maybe relative to prior generation, something they should prioritize more in the in the way they're learning?

• 4:26Speaker 01

  So, you know, my prediction that I made, you know, a few years ago now, a couple of years ago, when it became obvious the AI is going to change programming, was that sort of software is going to bifurcate. And it was already on that trend where you had the front-end engineer and the back-end engineer. But I think that gap will widen even more and you'll have sort of the product engineer, the product creator, you can be an entrepreneur, you can give a crap about coding, you just want to make something, you want to get customers, you want to get users. And that will look very different than how we thought about software engineering. I think majority of the time, you're prompting, you're iterating on your prompts.

• 5:09Speaker 01

  You're obviously still debugging code for a period of time. At some point, maybe that also would get automated away. But I think the traditional way of thinking about software engineering, and you know, the analogy of engineering is that you're thinking about components, you're thinking about systems, you're thinking about, and that sort of bifurcates. So I think it depends on what you want to do. Do you want to be a product creator? Or do you want to go the traditional path, software engineering? Which I don't think it's going to change all that much. It will obviously still change with AI. But you know, if you want to build cloud infrastructure or data pipelines or backend systems, I don't think that's going to change all that much.

• 5:53Speaker 01

  But the act of creating software for a customer, for end-user consumption, I think that's the thing that's going to change rapidly. So I think the root question here is like, which one do you want to go? So I would say if you want to go the traditional software engineering path, probably getting a computer science degree and doing all of that is still relevant. But if you just want to be a builder, you can just start with that.

• 6:17Speaker 02

  You touched a little bit on the Replit's AI products, and there's a bunch and tons of ways that you're using Replit. And a bunch of our listeners are super familiar with the product as well. But maybe just, could you give a quick overview of some of the ways that Replit is using AI with things like Ghostwriter and what those are?

• 6:34Speaker 01

  Yeah, so Ghostwriter, we actually renamed it. So Ghostwriter was our AI product. And actually, it's kind of interesting to kind of talk about the reasoning of why we renamed it. It's just Replit AI now, or just think about it as Replit. The analogy I give is, you know, when GUIs first came on the scene, they were actually an add-on. So you bought DOS and then you paid another hundred dollars to Microsoft to get the Windows add-on. And you guys are probably younger than me, but like then you would like launch Windows from DOS, right? And like, your main entry point to the OS

• 7:15Speaker 01

  is still DOS. But then if you want GUI every now and then, you get that. So right now, we're in this era of the co-pilots, right? You buy a piece of software and then Notion or Excel, whatever, and then you pay an additional fee and then you get a AI-powered software. My view is that that's a transitionary period. Mm-hmm. And companies counting on that add-on revenue should like really be worried. Because I think that we're gonna start to rethink software from the perspective of LLMs and what we can do with LLMs. So Replit has always been about, you know, long-term thinking. And so we went through this period where it's an add-on, we call it a ghostwriter. But

• 8:00Speaker 01

  then it was like, okay, I know this is where the future is headed. So we actually embedded it in the product. And we tried to make it so that every interaction you have with the product is AI-powered. And we made it a part of the free plan as well. So literally the first line of code you type at Replit, you get an AI suggestion. And then, you know, our pro plan comes with bigger models and better models and better services and things like that. But still, one of the reasons is structural. Like, inside the company, I want the designers not to think about, you know, every workflow is like, this is the workflow without AI, this is the workflow with AI. No, think about it. Start from AI.

• 8:44Speaker 01

  And so the way the product works today is that, you know, when you're typing code, we give you code suggestions. So this is like Copilot. Actually Gmail started with that. You had the ghost text, if you remember that. I don't think they even used LLMs at that time, but it was quite powerful. And Copilot kind of copied that UX pattern and now it's become very popular. So we do that. We trained our own model to do that. It's really fast and really powerful. And then now there are all these other, so this is, you know, I like to call it the the push model where the user is passively getting AI suggestions.

• 9:26Speaker 01

  Now, the sort of pull model, when the user wants to do a bigger task, wants to generate an entire file, it can do right click and generate. And you can type up a prompt, you can spend your time kind of crafting that prompt and you can get an entire file generated based on the context and everything. That's another one. And then there's sort of sprinkles of AI workflows in every aspect of the product. For example, get an error. If you get an error in the console, we show you an AI debug button. You can click that. It opens the console. It opens the AI chat pre-prompted with the error and whatever relevant context and kind of tells you about that.

• 10:07Speaker 01

  And and there's probably another like, you know, five to ten features kind of like that, but this is the general landscape of how the product work.

• 10:18Jacob Effron?

  It's a really interesting point that it sounds like basically, you know, in thinking about doing a copilot, you know, add-on setup features versus just embedding in the product. It's almost like you didn't want to continue supporting a product that didn't have AI in it. It's like, you know, for all your users, you want your entire team just focused on that, like AI embedded experience.

• 10:33Speaker 01

  Yes.

• 10:34Jacob Effron?

  I'd be curious, like, I feel like you have a great vantage point into this, like, where are we today in what, you know, these AI models can and can't do in the coding world? I'm sure you guys are always pushing the limits. So how would you characterize kind of like what's possible, what's not? And how much does that change with like, you know, when we move from to GPT four or something like with each model release, does it feel like that dramatically changes, changes a bit? Would love your thoughts on that.

• 10:55Speaker 01

  So in the GPT two, starting with the GPT two era and GPT three, there was a mystical component of LLM almost. Nobody knew how really they work. There was this sense of, oh, it generalizes after distribution, maybe it's AGI, maybe we achieve AGI. And it was actually a very interesting, fun period because you had people approaching GPT three like anthropologists, like trying to study it. You had these blog posts kind of trying to.

• 11:21Jacob Effron?

  What's going on inside the box?

• 11:22Speaker 01

  Yeah. And it's super interesting, right? And and I think, you know, my view has sort of evolved as I understood LLMs better. In my view on LLMs now, sort of in a very reductive way, it's a function of data. I mean, the most productive way you can think about it, and others have made that point, it's the compression of the data. That doesn't mean it's not powerful, but you know, the real power of Alums is the interpolation of different distributions of data. So, you know, writing a rap song in the style of Shakespeare, right? That's a sort of a fun example of interpolating. Pat and I love doing that. Yeah. It's super funny.

• 12:06Speaker 01

  That's what everyone spent last year doing in ChatGPT. Jacob still listens to these in his free time. Yeah, you know what you should do? Pretty good. You should do that and then now plug it into Suno. Did you know that? We plug Suno on this podcast. Suno is very cool. Yes. And so you start chaining these things and now you have a complete sort of AI-generated piece of art, which is, which is amazing. And piss off the He-Man these people a lot. It's like, you nerds, you think you're doing art? It's not art. Yeah, so it's a new programming paradigm, and I think Karpati, I think called it software 2.0 at some point, which is you're programming with data. And so to understand the capabilities of these models, you have to understand what

• 12:50Speaker 01

  kind of data you're feeding them. And then what kind of fine tuning or post-training mechanisms you're using, whether it's, you know, sort of instruction fine tuning, it's RLHF. Now there's like direct, what's it called? DPO, direct optimization, I think. There's so many new of these post-training. And one way to think of the post-training is you have, you know, what we're calling foundation model, which is like a vast collection of different capabilities and personalities and all sorts of weird things. In, in post-training, if I tuning, you're sort of, you know, you're.

• 13:31Speaker 01

  You're trying to make it better at some downstream tasks. That's that's very specific. And and so the question is like, where is it headed? It's it's a question of where the data sources are coming from? And how much scaling and how much compute can we put onto it? There's a bit of a bearish answer on on programming, because I would argue we've sort of run out of open source tokens.

• 14:04Speaker 02

  Yeah.

• 14:04Speaker 01

  Like, I think if you, you know, even even our small 3B model, we trained on permissive data, but we trained all GitHub repositories, permissive data and and our own Repli user code. And if you think about something like GPT-4, it's actually trained on all code data on the internet. Plus, paying tens, I don't know, hundreds of millions of dollars to companies like Scail and contractors to annotate coding data. So now we're in this paradigm of paying people to generate data.

• 14:40Speaker 02

  Yeah.

• 14:41Speaker 01

  Now, Repli has the advantage of actually having a large user base of actually people actually generate unique, unique tokens. So, you know, but but that doesn't mean where the capabilities will stop here. I think we'll figure things out, and maybe it's just a question of paying a lot of money. But also, there's there's a way in which just the, you know, the coding ability could improve by function of scale, by function of even getting as much data as possible, even if it's not coding. For example, scientific data has shown, has been shown to improve code, even legal text.

• 15:23Speaker 01

  So we can find these coding-adjacent reasoning things that will actually go back and improve coding. So I would assume we have, we have another two to three years of increased capabilities of coding. I don't know what kind of products they will unlock, and that's the hard thing about working in the space. I could sort of start to imagine what it could look like. I actually gave a TED Talk about where I think this space had it in three to five years.

• 15:47Speaker 02

  On the data front, if LLMs are, and these generative models that are coding, if they're as good as the data that you feed them. I guess what makes the best data? Is it size? Is it, you know, curating it? Or you guys, obviously, have a bunch of proprietary data, but like who? Or how do you think about what the ideal data set would be for a code generation model?

• 16:08Speaker 01

  So size matters a lot. Like, you want to be able to, size and compute are very related, right? Because the more tokens you have, the more compute you can spend training this token. The more diversity of tokens, the more fresh the tokens are, the better. We've also shown with a 3B model that you can do multiple epochs on the same data. If it's high quality, you can continue to get lower loss and better improvement in performance and accuracy over that. Now, on the quality of data, I mean, there are some basic things. For example, if you train on modified JavaScript, you're really messing up your model.

• 16:51Speaker 01

  It's learning all sorts of wrong things, and nobody codes like that. You want to train on data generated by the best programmers? Because one way you're thinking about GPT is they are emulation machines. They emulate the, actually, the process that humans have of, you know, because they're trying to guess the next token, they are, you know, behavior cloning almost of humans. So the higher quality the human thinking is, the higher quality the model is. So there's all sorts of heuristics you could use. If you take, you know, public repositories like on GitHub,

• 17:32Speaker 01

  you can even do some kind of mapping of users and figure out who are the best programmers. One thing we found is, actually, one under, you know, a lot of people when they're training code models, they think, okay, we want to get the, you know, the highest, you know, code with the high stars. Those tend to be the libraries, which they're good, they're high-quality tokens. But actually, the thing that there's more poverty of is the usage of the libraries. It's the downstream applications. GitHub is full of high-quality. Infrastructure code, but it's not full of high-quality application code. And this is where we think Replit sort of shines, because a lot of people are writing applications on Replit.

• 18:16Speaker 02

  On the the application side, at the beginning, you were talking about, you know, there would be this bifurcation of engineers into people building apps and then maybe more of the the infrastructure folks. How do you think AI will benefit the different groups, whether that's, you know, an engineer who's just starting versus intermediate versus advanced? Like, are they all going to use AI in the same way? Or do you think that it will disproportionately benefit, you know, certain types of groups? Maybe it's the product engineers or or someone else?

• 18:43Speaker 01

  My hunch initially was the return on investment for for beginner now went way up, right? Like I said, you can, with minimal amount of code understanding, we have people on Replit, we have this 100 days of code course that we provide for free. We have people go from that, starting a company, to making hundreds and thousands of R R doing Microsoft applications in a matter of months, which is

• 19:06Speaker 02

  That's incredible.

• 19:07Speaker 01

  There's never been a kind of ROI on learning how to code like the period we are in right now. So I think this one group has obviously benefited a lot.

• 19:20Speaker 01

  I think that I think that, you know, there's been a study, I think it wasn't coding, it was some consulting group.

• 19:28Speaker 02

  BCG consultants, yeah, that's really interesting.

• 19:30Speaker 01

  Yeah, so it it showed that it benefited beginners more than than advanced people.

• 19:35Speaker 02

  Yeah, I think it was actually, it was like the worst consultants, right? Or like the lower-rated ones or whatnot.

• 19:39Speaker 01

  Those are the ones that got

• 19:40Speaker 02

  Yeah, that got the most boost from it.

• 19:42Speaker 01

  I mean, that sort of intuitively makes sense. I do wonder, you know, there's not a study been made where you condition on training people to use AI as as best you can, right? Because we all know there's a way to leverage ChatGPT, there's a prompt techniques, there's a chain of thought, there's all these ways in which you can get better at prompting LMS. And so I think if you start to condition on that, then I think it will like, it will tilt over to to the more advanced people. Because because not only you have the skills to debug and and write the software, but you also have the skills to use LMS better and seek different LMS and different diversity

• 20:27Speaker 01

  of LMS, like. You know, I often will like, and this for my personal use outside of coding. You know, I'll go to like, you know, chat G P. I wouldn't get an answer. I would open the perplexity playground and go to like mix stroll.

• 20:42Jacob Effron?

  Yeah.

• 20:42Speaker 01

  And might get an answer from that. And so that being good at that sort of seeking the answer for L M, I think will start to make a difference.

• 20:51Jacob Effron?

  I think your point's really interesting because it seems like there's, you know, there's a set of people that know how to use these applications in super sophisticated ways. And maybe if you haven't had exposure to that yet, you have some not great experience and give up. I imagine that manifests itself a lot in your own product experience, right? And I'm sure you guys have thought about a lot about like, how do you? You know, there's probably some set of people that use Replit A I in a way that you're like, oh, I just wish everyone had exposure to that. How have you thought about like the user education side and like getting the 95% of people up to what the top 5% are using it?

• 21:19Speaker 01

  You know, in retrospect, what I'm gonna say is not so surprising, but it was surprising for us.

• 21:26Speaker 01

  I think, you know, younger people, kids are, you know, they are much better at adapting and starting to build a mental model for what the L M is able to do. And be able to naturally start to prompt it. And naturally, Replit has, you know, a large base of young people. And so we found that when we started rolling it out for free and it's just suggestions started to appear, a lot of people just like rolled with it. Like they didn't even blink, a lot of younger people, a lot of more established people kind of were a little jarred by it. You know, we had some teachers that rely on Replit to teach programming, reach out to us and say, you have destroyed my classroom.

• 22:12Speaker 01

  So our response to them is like, look, you can turn it off. You can tell your students to turn it off. But we think, you know, this moment is sort of like the calculator. You know, there's a lot of teachers, if you go look back at history, that were really banned the calculator in class and all of that. But it's a tool. It's gonna get widespread usage, so might as well try to learn and be ahead of the curve. And we started seeing some teachers that within two or three days actually find that their students are learning better. And their students are better than them at figuring out what these models are capable of. And the reason it's not so surprising in retrospect because you see it in a lot of places

• 22:52Speaker 01

  where younger generations are able to adapt to new innovation that just this is a biological fact of the plasticity of the brain, right? And so it's just, it's true in this case as well.

• 23:04Speaker 02

  How have you guys thought about structuring your teams around AI? We had Linus on the podcast and he talked about how they were doing it notion, which it seemed like over time was moving more towards horizontal. We have kind of AI in all of our products. And so they structured their team that way. We've also had a bunch of other folks that say, hey, you know, we want to embed them within the product because it's so core to this vertical or this specific product. Like, how have you guys thought about structuring the AI team?

• 23:30Speaker 01

  I've always been about horizontal and I think it's just the infrastructure in Jeremy. I'm just generally infrastructure guy. So it's like you always want to build platforms. And I think it just makes, you know, business sense because again, if you absolutely convince yourself that it's going to touch every aspect of software, which I think it is. And by the way, in this separate discussion, I think actually will like, I'm surprised how slow it is to, you know, people saying, oh, it's like moving really fast. It was like, I actually would have guessed that would be further ahead instead of.

• 24:07Speaker 02

  What did you think would have happened by now?

• 24:09Speaker 01

  I would have think every piece of technology that I have would, you know, be proficient in natural language. Like, why can't I talk to Siri yet and like give it complex instructions? You know, I don't know. And, you know, it's just the nature of corporates being slow.

• 24:28Speaker 02

  What do you think about the speed with which it's affected coding? I feel like that's been really rapid in some respects, but then also maybe even slower than now that machines can write code, like why isn't everyone using them? Or I guess to some extent they are.

• 24:42Speaker 01

  Yeah, yeah, the way corporate spread is definitely like surprising, especially how fast corporates were open to the idea, especially given the legal risk and things like that. So I think that's one area, but you know, but it's sort of lame in a way because of course we're going to adopt it like we built this shit like.

• 25:06Speaker 01

  But what we want to do is like have the like everyday, you know, person improve lives and improve economic conditions. And, you know, we're, we're in this world of, um, you know, uh, slow growth and, and it's causing all sorts of problems in the world. And I think AI will help us kind of kickstart another sort of S curve of, of growth. And I think it's imperative that it moves faster, in my opinion. And, and, and, you know, there, there are forces that are working against that culturally and legally and in all sorts of ways that are even within these companies. I mean, we saw the OpenAI drama, right? There are elements within the big AI labs that are trying to slow things down, which

• 25:50Speaker 01

  I think is, is ill-advised. So, uh, you know, um, you know, I, I would have expected the everyday consumer life to have changed meaningfully. I would have expected the everyday professional life, the consumer life to have changed meaning, meaningfully. I mean, I still, I still, you know, Gmail should be able to create calendar invites based on my emails. It, it doesn't. Why not? Right? I mean, it's just basic things. I feel like it should have rolled out faster.

• 26:21Jacob Effron?

  Totally. I mean, one thing I definitely want to make sure we hit on is I feel like early on you were, I, I think one of the first ones we publicly heard about that was, you know, you're like, we're going to build our own model. And I think you were like the poster child for Mosaic as one of their case studies. And, um, I'd love to hear a bit more about that decision. Cause I think, you know, a lot of folks we talked to say, well, first we like to experiment with like the latest OpenAI models, figure out what's possible, you know, make sure we build something our users want. And then maybe, I feel like a lot of people are in the wait and see mode of, yeah, maybe we'll build our own model or maybe we'll use open source long-term. But for now, you know, we're, we're not there. And it feels like very quickly you guys moved to, no, this is something we really want to have in-house.

• 26:57Speaker 01

  So I think it's totally rational, uh, what people are, are saying, especially the speed at which the, um, the models are getting more affordable. I mean, GPT 3.5 just, uh, got halved in price, I think, recently, right? It was, you know, 10x, uh, you know, just six months ago and it'll continue. So I think it's totally rational for a lot of these companies to not have, not want to train these models. The reason it was crucial for Replit to do that is because actually none of the commercial models, um, have the characteristics that we want, which is low latency.

• 27:31Speaker 01

  The way Copilot was able to get to market is because of their deep partnership with OpenAI, in which they were able to take, you know, have custom models and have a custom partnership where they can hit these latency constraints. And by the way, Copilot has gotten way slower recently. I think they're just using now Stock GPT three point five or something. But but for us, latency was very important. Cost was was was was very important. I always had this vision that it needs to be part of the free experience. Yeah. Because again, I wanted to structure the company in a way that we're just AI native.

• 28:12Speaker 01

  And when we ran the numbers, it just never made sense to be a customer of these commercial models. And then when we used open source to prototype our first, you know, Ghost Rider release, you know, we used Salesforce three billion parameter model. And I think one of the things we were early on is that actually small models are capable. And we did a lot of work to to productionize it. And then I was like, OK, small models are capable. Small models are affordable to train and deploy. Therefore, it made sense for us to spend a hundred K. I mean, really, we spent a hundred K, you know, training our three

• 28:56Speaker 01

  B models. It's not a huge capital expenditure. And I think we're early to start developing that talent internally. And we have a the AI team at Replit is still small, it's like seven, eight people. Wow. But, you know, Replit is all about, you know, being lean and mean. We're a hundred plus people total. So so for all these reasons, it made sense for us to do that. But by the way, we are now customers of commercial models for a lot of these other use cases that I talked about. Doesn't make sense for us to go create.

• 29:36Speaker 01

  A general purpose chat model for some of the chat features that we have. So I would say always start from the problem, from the customer pain point, explore potential solutions, run the numbers. And if it makes sense to make the investment internally, but there's some strategic considerations. Like if you consider yourself to be an AI company, I mean, Arvin is a great example of that at perplexity. Perplexity is by and large a user of commercial models. You know, the reductive way of thinking about GPT wrapper, whatever, which I disagree with these characterizations. I think they're dumb. The.

• 30:16Jacob Effron?

  Arvin has some hot takes.

• 30:20Speaker 01

  But but they're building a very competent AI team. You know, they're using perplexity playground as a way to build and battle test that team. Cause they know they're going to be able to deploy different AI technologies in the future. So strategic, there's a strategic angle that you need to to look at as well.

• 30:38Jacob Effron?

  Totally. So I guess like gazing forward, you know, three, five years. Do you think there's going to be a lot of companies like Replit that decide to build their own model? Or will most people be using commercial APIs or maybe doing light fine tunes of open source models? Like, how many companies are going to spend the hundred K to kind of train, train something large?

• 30:55Speaker 01

  That's a good question. I mean, it's hard because, you know, I recently

• 31:01Speaker 01

  made this tweet that went viral and generated a lot of discussion, which I said, you know, we don't really have true open source models. And the analogy I made is that

• 31:15Speaker 01

  you can't reproduce the open source models and therefore it's not really open source, right? So imagine if Linux, imagine if you can only use the Linux binary. Would you consider that open source? No. Imagine if you get the Linux source code, but the compiler for it is not available. Would you consider that open source? No. This is where we are right now. I mean, I, you know, have nothing but respect for Meta and Mistral, all these companies. I'm friends with all of them. But if you're using open source models, you're dependent on the goodwill of Zuck to continue to to to push out, you know, Lama two, three, four, five, right? So in in the in the absence of that, I think a lot of companies are gonna be saying,

• 32:00Speaker 01

  do we really want to? You know, you know, bet our business on the fact that, you know, Zuck will always be in a good mood.

• 32:13Jacob Effron?

  Or that his shareholders will always let him spend $10 billion on a, on compute.

• 32:17Speaker 01

  Although, you know, Zuck is in, in like the dual class.

• 32:21Jacob Effron?

  Yes, exactly.

• 32:23Jacob Effron?

  That's fair.

• 32:24Speaker 01

  So, so I think the reason I'm having trouble answering this is because I would have said like maybe a year ago, of course, open source. Of course, a lot of companies will just do the fine tuning of open source and there's an ecosystem. But the more I thought about it, you know, I, I think strategically, you don't wanna, you don't wanna depend on the current version of open source. Long term, I think you wanna treat them sort of like commercial models. You wanna experiment with them, you wanna prototype with them, maybe you wanna deploy in, in production. But you wanna have a path towards some sustainable way of using them and depending on them. Without having, you know, this external dependency that you don't have any control over.

• 33:08Speaker 01

  So I would say TBD, I think the way things look right now, I think the commercial side is actually ahead. And I think most people will depend on the commercial side. If the OpenAI fine tuning business goes well as well, and they have this custom model business now.

• 33:26Jacob Effron?

  There's a few million dollars now.

• 33:27Speaker 01

  Just a few million bucks. I think Anthropic's doing a lot of that, they have solutions engineering and, and things like. The AI companies are looking more like traditional enterprise companies. They, they have ways of, of implementing these solutions for, for customers and there's a services aspect of it. And so I guess I, I don't think I have a great answer for you, but, but you know, that, that the landscape is not set yet. But, but so far, I would love to see an open source project that is actually truly open source. That improves over time, that allows. Contributions back and that has that open source flywheel.

• 34:08Jacob Effron?

  It's a really interesting point to your earlier, you know, statement that like these models are basically just reflections of the data that is put into them. If you don't know the underlying data that trains these models, how does your own AI team iterate and kind of improve them? And so it seems that, you know, obviously a really powerful part about having your own model now is you know exactly the levers you can pull on the data side and how that ends up impacting the the underlying model.

• 34:27Speaker 01

  Yeah, you're you're programming with data and therefore your the data is the source code in that in that analogy. And so if you don't know what's the source code, I think Karpathy was sort of early to that, but he talked about this idea of building essentially backdoors into these models and backdoors that are not activated. Actually, going back to the 70s, there's this paper by, I think, Ken Thompson, one of the inventors of the C programming language, was called Trusting Trust. And he comes up with this very clever system where you know how the C compilers written in C and, you know, in in sort of software

• 35:09Speaker 01

  engineering terms, it's bootstrapped. You kind of write the first compiler in assembly or assembler and then you you get the C language into a place where you can write the compiler in it and then and then you use the previous compiler to compile the the new compiler in C and then the result is a new compiler that's fully bootstrapped, right? Which is kind of an insane concept. But he shows that you can put in a backdoor in a compiler and after it's bootstrapped, there's no way to recover the source code. And therefore, there's a there's a backdoor built into it that that can evade inspection. And so the LLM training can suffer a similar thing.

• 35:54Jacob Effron?

  Right.

• 35:55Speaker 01

  And and you can hide this thing that's only activated after a certain thing is uttered, after some certain command is given to it. And so, you know, if you don't have clarity on the training process and the data that went into that, there's a huge security risk.

• 36:12Jacob Effron?

  Kind of sounds analogous to the sci fi stuff we were talking about before before we started recording.

• 36:16Speaker 01

  Yeah, yeah. I mean, I think the world's gonna get way weirder. I mean, it's already getting pretty strange. I mean, I now have a background thread whenever I see any type of media. So I'm sure you too as well. Any media I see is like, there's something that's check. Is this AI? Check the hand, check the, you know. So I think it's just gonna get weirder and weirder. Actually, there's a, there is a, I think a sort of a French intellectual tradition,

• 36:58Speaker 01

  most famously, how do I know how to pronounce it? Paul de Roiard, I think. His name is, and he coined this term hyperreality. And they were talking about sort of the end state of this, you know, postmodern world that we live in is that it would be incredibly hard to know what's fiction and what's reality. And in a lot of ways, we would be living in a sort of virtual reality. And so I feel like we're running towards that world as fast as we can.

• 37:29Speaker 02

  I was tricked by a video very recently of, I think it was Kim Kardashian and Taylor Swift teaching calculus.

• 37:36Speaker 01

  That sounds amazing.

• 37:39Speaker 02

  It is incredible. I forget exactly which account this was, but for the first three seconds of this, I was like, wait, how did they get Kim Kardashian? I thought it was something where they had done this partnership and she was teaching them about derivatives.

• 37:51Speaker 01

  That would be great, actually.

• 37:52Speaker 02

  I thought it was like a really, like an educational thing, which, you know, maybe judge me for that. But then over time, I was like, oh, this is actually fake. But to your point, it's that period where you know is getting pushed out farther and farther, and you now have this question of, wait, is this actually real or did they generate this? And I think there's gonna be a lot more of that, so.

• 38:11Speaker 01

  Yeah, and I, you know, the pessimistic thing is that no one is really building the technology to counteract that. And maybe there's no money in it. That's why the market is not really. But like, why don't I have a Chrome extension now that I can like, tell me which media is is fake and which media is is.

• 38:29Jacob Effron?

  Yeah.

• 38:30Speaker 01

  You know, real.

• 38:31Jacob Effron?

  Yeah. You know, they target everyone with what they think is going to resonate with the most. So you were targeted with derivatives is a thing says, yeah.

• 38:38Speaker 02

  Yeah, it's good content.

• 38:39Jacob Effron?

  It's impressive. I don't think I'd be targeted with that. That's probably more says more about me.

• 38:43Speaker 02

  You're targeted with the Shakespeare poems.

• 38:45Jacob Effron?

  Yeah, Shakespeare rap. Exactly, is what I would be targeted with. You kind of alluded to this earlier, but like, I think one thing that's interesting, it sounds like, especially because you're offering these AI features in your free product, is just the pricing, like the cost of inference for these models and how that impacts your decisions.

• 38:59Speaker 01

  Yeah.

• 38:59Jacob Effron?

  Wondering like, you know, we were talking about earlier, the pricing of this stuff keeps coming down. Is there a set of, you know, features that you're like, oh, we would love to do this, you know, this often or something, but today it's kind of cost prohibitive, you know? If if our friends keep slashing prices every three months, you know, maybe in a year it's something we'll do.

• 39:15Speaker 01

  Yeah, I think the agent model is interesting. The way I think started with the AutoGPT wave, I think, where you're cursively call the model and Langchain sort of allows for that as well. And being able to run it in the background, being able to like, give a prompt. And okay, you know, there's a high likelihood of failure, but I don't care. It's going to run in the background. And, you know, a prompt of like, refactor this and run the tests and make sure they're successful. I mean, these things, especially with GPT four, get really expensive really quickly. And most consumers, most developers would not want to pay for that.

• 39:56Speaker 01

  So background agentic workflows, which by the way, I don't think they work all that well, but you can sort of throw a computer at them and try to make them work. But again, it gets too expensive. So I think we haven't really spent a lot of time there. We have some research projects about how we can train more effective agentic models. The thing is, GPT four is agentic almost accidentally. And not entirely accidentally, I think OpenAI programmed it in some way to be, you know, respond better to like chain of thoughts and things like that, but.

• 40:34Speaker 01

  But the fact that LLMs have any agentic capabilities is actually somewhat surprised me because I thought that the way to get agents is to do what's called action transformers, which is probability distribution not over utterances, it's over actions. And as far as I know, I think Tesla self-driving, the new thing is like an action transformer. But there hasn't been a lot of progress there.

• 41:15Speaker 01

  Adept released a demo like maybe two years ago now, Act One, where they trained. I think people are calling them LAMs, Large Action Models. There was the recent RABBITS kind of presentation where they talked about LAMs. But again, this hasn't really affected people's everyday lives yet. But maybe that's the next chat GPT moment, when agents actually start to work.

• 41:41Speaker 02

  On the pricing side for your users, you talked about wanting to have AI as part of the free tier, but then also maybe more advanced features. How do you think about pricing over time as you continue to build more and more AI features and build that into the product?

• 41:56Speaker 01

  You want to, again, if you're starting from the assumption that AI is going to be table stakes, then the way to think about pricing is from a value-based perspective as opposed to like cost plus, and this is what AI is costing us. You want to try to project forward some amount of the

• 42:26Speaker 01

  How how much cheaper these uh models are getting to run uh the inference stack getting more efficient um and companies frankly, I mean, there was an information article about Anthropic's margins not being super great, which makes sense, right?

• 42:45Jacob Effron?

  They were still pretty good, though, right? It was like fifty five to sixty. I thought when I clicked it, I thought it was going to be like five or something.

• 42:51Speaker 01

  Yeah, okay, all right. But but you know, there's some amount of, you know, in the like Uber twenty thirteen period of, you know, VC subsidized rides, where like there's some amount of VC subsidized um.

• 43:07Speaker 02

  Models and training and tokens, yeah.

• 43:10Speaker 01

  So uh but but you know, you want to project that forward a little bit uh and then, you know, try to reason about, you know, what kind of value you're providing to your customers, thinking about the competitive landscape as well, but I really wouldn't start from the perspective like this is what AI is costing me.

• 43:25Speaker 02

  Yeah, I think that makes sense, and I think that's something that we're seeing more and more companies are saying, hey, the product's not going to make any sense without AI, so the pricing, the price is the price for the product, and then we're going to build in in AI.

• 43:37Speaker 01

  By the way, one one sort of bet I made uh, you know, a couple of years ago was like usage-based pricing is going to get um a lot more uh prevalent, because even if you have a tiered pricing, you can have um you can have users that are that that are costing you a ton, like real power users that are spending a lot on on on these models and um and and it's true of the enterprise as well, like you can have a distribution where where it just uh you know, wouldn't make sense to have a SaaS, a pure subscription

• 44:20Speaker 01

  SaaS sort of model, and and we're seeing that, we're seeing a lot of usage-based um. Implementations of AI, Replit has, you know, a bundle that gives you a ton of compute, ton of AI, ton of storage, all of that. But then you can get overages on all these things as well. So I think there's some of that that's happening. I actually invested in a company called OR, but that does user-based pricing as a service. And they're doing super well. So I think that's one area, I think, that's, you know, based on what I said before. Yes, you want to think about this value-based thing, but there's going to be a way, you need a way to scale with usage.

• 45:04Speaker 02

  Yeah, I would imagine that's super important, especially for engineers that are going to use the models in a bunch of different ways, right? If you are working at an agency and maybe you're cranking out a web app every day, like you could be generating tons and tons of code, which could be quite expensive for whatever platform you're doing.

• 45:20Speaker 01

  I think it's going to get really expensive when these models are in the C I C D pipelines.

• 45:24Speaker 02

  Yeah, exactly. They run every time you build something.

• 45:27Speaker 01

  Yeah, GitHub Actions with L M's. Oh, my God.

• 45:30Speaker 02

  Yeah, that's a good point. Definitely gets more expensive when they run when you're sleeping or something like that.

• 45:36Speaker 01

  Which they will. Yeah. I'd be, again, this is one area I think that I would have expected now we would have some kind of L M's doing, doing things in the background.

• 45:46Jacob Effron?

  I mean, you alluded to it earlier about like, you know, the kind of future being this trend around like agentic work. You know, one, I think would be helpful to just contextualize kind of where we are with that today and the timeline in which you see that changing. And then another question I have, just hearing you talk about it is like, how do you think about whether like you should be doing research around that at Replit versus like, let me, you know, wait and just see what like the major labs are able to come up with or the adepts of the world?

• 46:10Speaker 01

  Yeah, so I do think that agents is the next, the next big. You know, a lot of people thought multimodal is going to be the next big thing, which I think multimodal is, is great. It's important. You know, maybe there's another hype moment that we'll see this year when everything is truly multimodal, but I don't think it's as big of a step. I don't think it's another chat G P T moment. It's just an incremental improvement that's pretty profound, but it's still incremental. And it depends on a lot of things getting better. It depends on pricing getting better, on latency getting better. You know, that Google demo that turned out to be somewhat staged, which I honestly don't fault them for it. I think they had a.

• 46:56Speaker 01

  And we're also partners, so I'm not supposed to say bad things about Google. But.

• 47:00Jacob Effron?

  It was a cool day.

• 47:01Speaker 01

  Yeah, yeah, yeah, I love Google. But the, you know, they had the demo, but it was obviously not feasible today. And so multimodal needs a lot of things to happen. So I think agentic workflows and background sort of agents doing things on your behalf. Some version of it will start, I think, happening this year. In terms of, OK, I really want to be part of this agent. I have use cases for agents. I want to start a startup around agents. What do I do? Do I just wait for the, you know, AI gods to kind of drop some agent APIs on us? Or do I go and try to do something?

• 47:44Speaker 01

  I'm of the mind that entrepreneurs need to be be able to kind of walk through walls. Right? You need to be able to get a product to market with whatever you have to do within your means. Right? So if you if you really want to make a big bet on an agent, you could probably try to make something worth GPT four. It could be very expensive. It could be more of a prototype demo, but you can start learning. And then if it just makes sense for you to go and try to fine tune and try to experiment and try to do more researchy things. That also, I think, is a is a relevant path. For more established companies, I think the, you know,

• 48:26Speaker 01

  let's say, more established companies with not a lot of money to spend or not a lot of technical talent or AI talent, probably wait and see.

• 48:35Jacob Effron?

  Yeah.

• 48:35Speaker 01

  So it really depends on the situation.

• 48:37Jacob Effron?

  Is there like a particular milestone you're looking for, like in the agent space? Like, you know, you're like, I really, I want to see the day that like it can build this basic app. Or, you know, is there something you point to internally that you're that you're gunning for?

• 48:48Speaker 01

  Being able, and this is like more of a soft kind of milestone, but like being able to give a model a.

• 48:57Speaker 01

  Just a bulleted list of actions without insane amount of chain of thought and recursive debugging and all of that and for it to follow it and not go off the rails and not start talking to itself or all that stuff that's happening today. I think that's the first sign that, okay, you know, we're closer. The ability, especially for the commercial models, to have a sort of a non-hacky way to be able to call in a dependable way, sort of like function calls. And function call is one of those things that it works in like 90% of the cases, but there are like 10% of the cases that are really catastrophic failures and it does like really

• 49:38Speaker 01

  random things. And you can't depend on it in a context of doing financial workflows or legal workflows or things like that. So I would say it's more a liability question. And, you know, there's, there's a way to look at these metrics. There are some of these, you know, tasks, task to pull request startups like sweep.dev. Sweep claims or they say that their data shows that 30% of PRs are getting accepted, which is already, I assume it's like very simple tasks. But, you know, if we get to a point where like it's 80 or

• 50:19Speaker 01

  90%, I think this is where you feel, okay, it's, it's getting better.

• 50:24Jacob Effron?

  Totally. And you kind of alluded to it, but I'm, you know, I'm curious with this broader AI coding market feels, there's lots of startups going after it in different ways. How do you kind of think this evolves? Like, is there just like a company, maybe Replit, that wins the entire space? Is there room for like multiple companies and like this future world? Where do you kind of see this market as a whole going?

• 50:41Speaker 01

  I mean, honestly, the default assumption, and this is sort of a bear case on Replit, is that Microsoft wins it all, right?

• 50:47Jacob Effron?

  That's true, like most things in AI, I feel like.

• 50:51Speaker 01

  Yeah, I mean, it's it's a little depressing, right? But I mean, if you look at the Slack Teams kind of things, if you look at that best history, you know, Microsoft has this install base, this enterprise base, this really brutal, really awesome sales team, and they have this commercial mindset, and they have a great leader. And so the default thing is Microsoft wins it all. Now, I think the optimistic thing is there's a new crop of companies that specialize in, you know, different parts of the stack, of different aspects of the coding workflows.

• 51:34Speaker 01

  You know, one criticism criticism of Copilot is that it's sort of locked into your editor. It is not, it doesn't have context over your other repos, over who's coding with you, over even your Git history, over your Jira tickets. Versus if you take a holistic approach, kind of what we're doing at Replit, where, you know, we're trying to kind of provide a cloud development environment with AI sitting on top of the entire stack. I think you'll be able to build much more ambitious AI products, including these agentic workflows. But I think there are going to be companies that are going to be like the best at generating tests.

• 52:18Speaker 01

  Right? There's like Codium, I think there's two Codiums now. One of the Codiums, I mean, that is like they are like, we're just gonna like be the best at generating tests, right? And I think those companies will do well. But the question is like, is there is there going to be a new platform? Or is it just a Microsoft sort of GitHub platform? Is GitLab gonna be an important, you know, part of this equation with their DevOps platform? And so again, the default pessimistic answer is is the existing players were just like, win it all. But what we're playing for is that we're going to be able to invent.

• 53:00Speaker 01

  Invent and reinvent how software development is made.

• 53:04Speaker 02

  What do you think of the the pure code generation startups? There's been a bunch of them. We're fortunate to be investors in Poolside. Are those but you you guys obviously have your own models like is that? Would you want to integrate with those folks or kind of build your own models? There's kind of a bunch of different ways that it could evolve.

• 53:23Speaker 01

  Yeah, so so I think what Jason's doing is fascinating.

• 53:31Speaker 01

  And and, you know, I've talked to him and ESO about it. And I think I think it's a bet worth making is like, can you build like the best large code model?

• 53:44Speaker 01

  Again, there's like the default pessimistic sort of point of view, which, by the way, I don't. I'm not like naturally pessimistic, but I'll try to be rational. The default view is that OpenAI like GPT five is going to be so much better. It's going to do do another leapfrog in coding ability that the economics of something like Poolside would not make sense. Like, oh, we're going to spend $100 million on training, but we're going to match GPT four. And by the time we match GPT four, GPT five has come out. By the way, Code Llama has now matched GPT four, I think they claim. We'll see.

• 54:20Speaker 02

  We're excited that news came out right as you were going to have you on. I'm actually curious. What did you what did you think of the Code Llama release?

• 54:25Speaker 01

  I haven't played with it yet. I tend to like try to sit with them and have vibes.

• 54:32Speaker 02

  It's all about vibes with the models.

• 54:34Speaker 01

  Yeah, I think I coined that and now it's become a kind of a term of art. But yeah, just like getting getting the vibes of it. Look, from my Twitter feed, it it's mixed. Some people are saying it's obviously very competent at coding, but a lot of people are saying that it still falls short of GPT four. It's GPT four level on the benchmarks does not mean GPT four level on the vibes, right? So it looks like the vibes are a little off.

• 55:01Speaker 02

  Vibes are a little off.

• 55:02Speaker 01

  But some people are saying it's a good base for fine tuning, which I think what you would want out of an open source.

• 55:07Speaker 02

  Yeah.

• 55:08Speaker 01

  Model. But, you know, insofar that like Poolside can find a.

• 55:18Speaker 01

  Code specific optimization or direction to take these models in, it could be, it could be great. For us, how we think about it, yeah, we're willing to work with, with anyone, really. Like, we are, the way we see it is, we are a applied AI company. We'll do research, we'll do training if we need to.

• 55:39Speaker 02

  Yeah.

• 55:40Speaker 01

  But the, the most important thing is how can we deliver value to our customers and users?

• 55:45Speaker 02

  Yeah, that makes a lot of sense. Yeah, the code llama I thought was, was interesting because it seems like the kind of open source code generation models were behind L L M's and

• 55:55Speaker 01

  Yes.

• 55:55Speaker 02

  and the image generation companies. And that seems like the first release where open source is now maybe on par with the proprietary models, although the vibes may be off.

• 56:05Jacob Effron?

  We need to set up a vibes benchmark.

• 56:07Speaker 02

  Yeah.

• 56:08Jacob Effron?

  Our contribution to the space.

• 56:09Speaker 01

  There are a bunch of interesting startups doing, doing benchmarking and evals. And I think this is super, super important.

• 56:15Speaker 02

  Yeah, we've, we've seen a ton of startups doing that. And it was actually, I think Logan from OpenAI, he said that evaluation would be the biggest trend of this coming year.

• 56:26Speaker 01

  Interesting.

• 56:27Speaker 02

  Yeah, or one of those. Hopefully I'm not misquoting him, but.

• 56:30Jacob Effron?

  He was definitely into it.

• 56:31Speaker 02

  Yeah.

• 56:32Speaker 01

  That seems kind of boring, though.

• 56:33Speaker 02

  Yeah.

• 56:33Jacob Effron?

  If evaluation is the biggest thing.

• 56:35Speaker 01

  I think he also, I think he also said multimodal.

• 56:37Speaker 02

  That and multimodal, multimodal. I think, yeah.

• 56:39Speaker 01

  He was a little more bearish on agents. Yeah, okay. Interesting. Well, you know, that, you know, he probably have some insight knowledge if he's bearish on agents. That's, that's not great. But maybe that's great for all the other agent startups and L L M's.

• 56:52Speaker 02

  Should we move to quick fire?

• 56:54Jacob Effron?

  Yeah, sounds great. We always like to end our interviews with kind of quick fire round where we ask you some, some quick questions and get your hot takes. And so maybe to start, we'll just be curious, one thing that's overhyped and one thing that's underhyped in the AI world today?

• 57:07Speaker 01

  Chatbots, I think we've arrived at the overhype scenario. I think there's, like, there are things that should not be chatbot. Underhyped, I think, using L L M's as part of everyday systems, meaning as part of the call chain of, of software, as part of backend systems and things like that. I haven't seen ways of integrating L L M's in this capacity, capacity I think could be, could be big.

• 57:38Speaker 02

  What's been the biggest surprise in building LLM features at Replit?

• 57:42Speaker 01

  I mean, how much latency matters. I mean, you sort of kind of intuitively understand that, but like, it really changes how you think about the feature. If you have to wait two, three seconds for a response, that changes the UX entirely from being able to do it in 300 milliseconds.

• 58:00Speaker 02

  Yeah, I love that. It kind of reminds me of the the search world where people are like, it's better to be fast and wrong than slow and right. And interestingly, in LLMs and generative AI, it seems like we've kind of taken the slow and right for some workflows.

• 58:15Speaker 01

  Yes, that's right.

• 58:17Jacob Effron?

  I mean, obviously, a lot of the stuff you've shipped has worked quite well. I'm wondering, is there an example of something you thought would work and just like fell flat with your users, like I said?

• 58:24Speaker 01

  I mean, there are things that we failed to

• 58:29Speaker 01

  to expose, like inline actions. Like inline actions, and I think Cursor is pretty good at this, where you're able to do very contextual actions inside the IDE. But we haven't exposed it very well. Most recently, we started exposing it. Initially, no one was using it. But we started prompting users through all sorts of UI patterns to because the cool thing about contextual actions that are better than chat window is because you can derive a lot of information from where the cursor is sitting.

• 59:04Speaker 01

  And so initially, that was sort of a flop, but now we're seeing it growing as a use case.

• 59:10Speaker 02

  Which AI startup outside of your space are you most excited about right now?

• 59:14Speaker 01

  I mean, it's kind of a boring answer, but like OpenAI is so interesting because of just how ambitious they are. Like, you know, they're doing all sorts of seemingly random things, right? Like, they're doing a partnership with

• 59:29Speaker 01

  Arizona University or something like that. I think that's real, right? They're doing education stuff. They're doing, I think I've heard robotics. They're doing self-driving. They're doing partnerships and investments and things like that. And I think, I think just something I admire about Sam Altman is just how ambitious he is. And he has a sort of, you know, ability to be able to spin a whole lot of plates, which is pretty impressive. So I think OpenAI will continue to be a very impressive company. And I think we're just going to continue to be surprised by the kind of things they'll be able to deliver directly and through their, their partners.

• 1:00:08Speaker 01

  On the, let me give you more of a startup perplexity. I think there was like a recent hype wave around perplexity, which is good. But I don't think people understand the kind of engineering that's happening behind the scenes to deliver these results. I mean, there was a ton of competition. They zoomed ahead of all of, everyone there. So I'm very bullish because of how they're building the company and the technical competency there will continue to deliver value.

• 1:00:37Jacob Effron?

  I guess if, you know, if your average startup today has like, you know, ex-engineers, 10 years from now, like what percent of that do you think, you know, a series B company will have?

• 1:00:45Speaker 01

  Wow, that's, that's a great question. I mean, again, it depends on adoption. Like, again, a lot of companies are just lazy. They're not really pushing for, you know, how much they can reconfigure their teams, given LLMs. I do think over five years, you could get, you could get what we're getting done now with 10th of the year. So it's not maybe a crazy estimate, but I think over 10 years, you know, I coined this term 1,000x developer, you know, two, three years ago. But over 10 years, I think there's 1,000x component to this where you see companies really, really shrink in size.

• 1:01:24Jacob Effron?

  Do you think we'll still continue to see the number of software engineers grow?

• 1:01:28Speaker 01

  Yeah, but again, I, we would start to call it something else. Like, I think the product engineer that we call engineer now, I don't know what we would call them. I mean, you know, we don't call TikTok, you know, people like, like your coworker, a movie star, right?

• 1:01:47Jacob Effron?

  Yeah, sorry, Rashad.

• 1:01:48Speaker 01

  We should probably call them movie stars. Yeah, you don't call Logan Paul like a movie star. You know, we call him a creator. You know, we came up with a new name. It's fundamentally the same thing. They get on the front of the screen and they do stuff, right? So I think that software creator of the future.

• 1:02:05Speaker 01

  Yeah, well, could either grow, but it, you know, we might call it something else. Software creator, maybe.

• 1:02:10Speaker 02

  I love that. I feel like that's the perfect note to end on. I mean, I guess I want to make sure we give the, the mic to you. I feel like there's tons of folks that will want to pull on different threads in our conversation. Where's the best place for them to go to learn more about you, about the AI work you're doing at Replit?

• 1:02:22Speaker 01

  So we have, we have, we have a blog, um, which I think has a lot of technical things, both on AI and, and, uh, different product launches. So blog.replit.com.

• 1:02:36Speaker 01

  I have a blog that's kind of, you know, every now and then I'll, I'll say something on it. Amasad.me and on Twitter, Amasad and basically everywhere else on the, on the web. Amasad. But yeah, the best way to kind of, you know, get a sense of what we're working on is to use the product. And, and again, uh, you can get a feeling of AI at Replit by just going to a replit.com and trying to make something.

• 1:03:04Jacob Effron?

  I love that. Uh, well, thanks, but this has been an awesome conversation.

• 1:03:07Speaker 01

  My pleasure. This is really fun.

• 1:03:08Speaker 02

  Yeah. Thank you so much.

• 1:03:09Jacob Effron?

  This was, this was fantastic. That was a fun one.

• 1:03:12Speaker 02

  Super fun. Yeah, he's a, he's a cool dude.

• 1:03:15Jacob Effron?

  Like super interesting. Um, I feel like the level of ambition to, uh, you know, I think he said it best. He's like an entrepreneur should be able to run through a wall. And so, you know, maybe some of these large companies will wait for other folks to make progress in agents. But like, we should be at the, the cutting edge of that.

• 1:03:29Speaker 02

  Yeah, I thought it was super cool what he was saying about the businesses that people are quickly able to build with using LLMs to kind of build these bootstrapped. And, uh, you know, I think he called them micro SaaS startups. There was a few people that were making, you know, hundreds of thousands of dollars in ARR in a couple of months. And I think that's something that, that we're going to continue to see for, uh, you know, maybe companies that aren't venture scale. They have, you know, Doesn't make sense for them to re-venture back, but they can still be really good businesses and grow and build them quite efficiently.

• 1:04:00Jacob Effron?

  Yeah, I mean, it's a really interesting question of what it means for our business. Obviously, like, the kind of employee needs to build large startup outcomes are a big reason why venture funds can deploy large amounts of money. And, you know, if he, if what he says is true in 10 years, he'll have these thousand X improvements in what you know, the engineering side. It's just interesting to think about the implications for, you know, is there just a lot less businesses that actually require, like, large venture checks?

• 1:04:25Speaker 02

  Well, I think it could be really good for some companies because what you were able to build, you now can build with one-tenth the cost.

• 1:04:34Jacob Effron?

  Yeah, this is the classic, you're on the early-stage teams.

• 1:04:36Speaker 02

  I know, optimistic, you know?

• 1:04:38Jacob Effron?

  But also, like, you know, you get that first round in, and then maybe they don't raise again. On the growth side, you know, it could be more challenging to raise some of our larger checks.

• 1:04:45Speaker 02

  Yeah, I mean, it's, yeah, that makes sense. I also thought it was hilarious how he was talking about model training and saying that training on code is really good, but there's also these other data sources that you can train on that can improve performance. And he was like, oh, you can train on legal documents. And then I was thinking about, think about how bummed the engineers are that wrote all of this code, and then now their code is being improved by training on legal documents. Like, isn't that just, I feel like those engineers are probably so bummed.

• 1:05:12Jacob Effron?

  Yeah, exactly. It's like, you know, we can get all the data from the top engineer out there, but it turns out it's way, it's way more additive to find other sources as well.

• 1:05:21Speaker 02

  The lawyers have their revenge.

• 1:05:23Jacob Effron?

  Exactly. But it is funny because I feel like he said that right after he was saying, you know, look, anthropologists were, like, fascinated by these models originally, and it turns out like they're not actually that, that complex, they're just spitting out data. But like, that is one of the crazy aspects of these coding models, that it's like actually throwing a bunch of non-code-related data at them makes them much better.

• 1:05:40Speaker 02

  Yeah, I, yeah, it's fascinating. It's like, then there are some interesting questions of what is the model actually doing? Like, it's clearly learning some, in some latent space, it's gathering knowledge that that helps it to understand the world based on the data that we fed it. And it is, it is cool that that improves code performance.

• 1:05:58Jacob Effron?

  I also love the bullishness on agents. I feel like a lot of other guests we've had have been like,

• 1:06:02Speaker 02

  Yeah.

• 1:06:02Jacob Effron?

  Ah, you know, there's just, there's just so much that can go wrong, it's not something that's going to happen anytime soon. And I think, you know, you get the sense that someone just super hard-charging and inspiring like him might, you know, might be one of the first to deploy these, these, these agents.

• 1:06:16Speaker 02

  Yeah, I thought his take on agents Being something that could run in the background when you're not thinking about it. And how we're not there yet, but we could be heading in that direction. I thought that was that was really interesting because you could imagine going to sleep and then your AI has fixed a bunch of bugs or done a bunch of things in your repo. So it's it's ready for you to start coding again in the morning. And I think that that's kind of a cool future to imagine.

• 1:06:40Jacob Effron?

  I also thought it was really interesting, like his whole take on how they built their own models. I mean, obviously, I feel like they were one of the poster children of, you know, I think Mosaic featured them very prominently is like, you know, the non-foundation model companies that were still, you know, throwing massive amounts of data to train their models. One, I thought it was interesting, you know, cost 100K, right? It's like, it's both expensive, but not that expensive, especially.

• 1:06:59Speaker 02

  It's less than you would think. And I think that was part of the Mosaic's post with them was talking about, you know, the economics of it and.

• 1:07:06Jacob Effron?

  Totally, because if it's going to impact your inference run every, you know, in their case, if they're giving out for in the free tier as well. Probably pretty good ROI if you with the with with with much lower inference costs. And then I think, you know, he kept hammering on latency, which makes total sense. You know, we've we've heard, you know, this idea of like, you're either in the flow of creativity or you're not. And the second a model takes a little bit longer, even like, you know, to the concept of of, you know, close to a second, it's like it gets you out of that that kind of flow state.

• 1:07:32Speaker 02

  Yeah, it seemed like they had had a lot of success around with this kind of model size around 3 billion, which I would think a lot of people would think of as small. But to your point, then it's cheaper to run, it's faster for the users. And if you're just doing code generation, then, you know, it can be good, quite good at a smaller set of tasks.

• 1:07:53Jacob Effron?

  I also loved like his, you know, his take at the end that like, you know, Microsoft might just want this whole thing. It feels like that that could between Microsoft and OpenAI and the partnership. It's like they do cover a pretty broad surface area of stuff.

• 1:08:05Speaker 02

  Yeah, I thought that was a very authentic take and and kind of something that, you know, I would imagine every founder thinks about that's building in this space is is competing with Microsoft to some extent. But yeah, I'm I'm. As you would say overly optimistic as an early-stage investor that there'll be a lot of people to win besides Microsoft.

• 1:08:27Jacob Effron?

  Totally. And I feel like for anyone with like, you know, a bingo board of stuff that we normally do in these episodes, I was pleased that they could knock off the Taylor Swift reference the second you made it in the middle there.

• 1:08:37Speaker 02

  Yeah, yeah. I was worried about weaving that in, but we got it in.

• 1:08:40Jacob Effron?

  Yeah, I mean, you know, halfway through, there wasn't a natural place to put it. And I'm not sure where you put it was necessarily natural, but it worked.

• 1:08:46Speaker 02

  Yeah, I probably could have had a better transition, but yeah, it was.

• 1:08:50Jacob Effron?

  I do love the idea of you being targeted with this Taylor Swift, Kim Kardashian calculus video.

• 1:08:55Speaker 02

  I don't spend much time on social media, but it's mostly kind of parent-esque content. Some surfing and then I guess Kim Kardashian teaching calculus. So, yeah. The last thing I think we should cover is just Code Llama, you know, announced last week and went viral. As I was talking about in the episode, I think it's really cool because code generation is kind of in the open source has, I think, been behind LLMs and image generation. And this seems like the first open code generation model that's really high quality. I guess the vibes may be off, time will tell, but I thought that was cool and it was interesting to get Amjad's take on it.

• 1:09:37Jacob Effron?

  No, I mean, I think to your point, it feels like the, in every, you know, type of model, the open source options seem to quickly catch up and, you know, it feels like that will just continue to be the case. And it's really, it's really cool to see the, just how good that model is, at least on the traditional benchmarks. A lot of fun doing another great episode and for folks that enjoyed this, definitely like, subscribe. We'll, we'll continue putting these out. We'll look forward to more Taylor Swift references and AI content here next week.

• 1:10:03Speaker 02

  Thanks so much for joining us.