E51: Replit's VP of AI, Michele Catasta, on Artificial Developer Intelligence

In this episode, Nathan sits down with Replit’s VP of AI, Michele Catasta. Replit is building what CEO Amjad Masad calls "the perfect substrate for AGI." In this discussion, Michele and Nathan discuss Replit's state of AI development report, advantages when it comes to AI development, and the company's custom models. If you're looking for an ERP platform, check out our sponsor, NetSuite: http://netsuite.com/cognitive We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. TIMESTAMPS: (00:00) Episode Preview (00:00:57) Introduction (04:44) What is artificial developer intelligence? (15:05) Sponsors: Netsuite | Omneky (16:58) Michele's background at Google & decision to join Replit (19:16) Startups vs incumbents (24:42) Whether Replit identifies as an e/acc company (26:37) Staying apolitical on AI while being responsible (30:36) The rise of LangChain and polarized reactions (35:14) Estimates on developer productivity gains from AI assistance (2x-10x) (38:35) Democratizing software development through easy customization (44:02) AI generating disposable single-use software (51:33) Optimism about humanity's ability to handle transformative AI (55:01) The need for nuanced AI safety discussions (56:14) Replit's data advantage from user code execution (01:04:31) Replit's approach to training custom AI models (01:08:51) The value of both open source and commercial models (01:13:18) Michele’s highlights from being a Google researcher (01:15:47) World knowledge needs in AI development (01:20:37) Replit’s approach to AI safety (01:24:28) The advantage of having a commercial model (01:25:54) The costs of serving AI features to millions of users (01:28:49) Modeling cost per user with AI workloads (01:30:50) Pushing AI inference to the edge (01:32:18) Ghostwriter integrating more deeply into Replit's IDE (01:33:37) Replit as a potential "substrate for AGI" LINKS: https://replit.com/ Replit's State of AI Development Report: https://blog.replit.com/ai-on-replit X: @pirroh (Michele) @labenz (Nathan) @eriktorenberg (Erik) @cogrev_podcast SPONSORS: Shopify | NetSuite | Omneky - Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive - NetSuite has 25 years of providing financial software for all your business needs. More than 36,000 businesses have already upgraded to NetSuite by Oracle, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you’re looking for an ERP platform, head to NetSuite: http://netsuite.com/cognitive and download your own customized KPI checklist. - Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. MUSIC CREDIT: MusicLM

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Because I would have spent more time learning interesting things about coding rather than setting up my coding environment and debugging very stupid syntactic mistakes. These are things that are honestly not intellectually stimulating about coding. And we're trying to get rid of them as soon as possible so that you can really focus on the creative process of writing software. Building large software still requires a lot of human effort. But allowing a lot of people to be creative will also expand dramatically the pool of good ideas that we explore as mankind.

• 0:34Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to Replit Week on The Cognitive Revolution. For context on this conversation, if you haven't already, I definitely recommend checking out our most recent episode with Replit product designer Tyler Angert. I spent 10 full minutes at the top of that episode explaining why I believe that Replit could become one of the most important companies in the world,

• 1:19Nathan Labenz?

  why I already consider it to be one of only 15 to 20 live players in the AI game globally today, and why, if I had to pick a single platform where the human-AI collaboration economy is most likely to take shape, I'd pick Replit. I also posted a version of that essay on Twitter, and we'll put a link in the show notes. Today, we're going even deeper on the future of AI at Replit with special guest Michael Catasta, Replit's new VP of AI. Michael recently joined the company from Google and made an immediate splash by publishing a new AI manifesto that outlines Replit's plan to support the next billion software developers. They plan to do this by enabling a seamless

• 2:01Nathan Labenz?

  human-computer symbiosis, creating an artificial developer intelligence that will create AI coding agents capable of developing complex software, plus a complementary economy that rewards human tool creators and hires humans for advanced problem-solving when needed. Macaulay even proposes a new meaning for REPL. In place of the original "read, evaluate, and print" loop, which describes a traditional interactive computing environment, Macaulay now unpacks the acronym to mean "reflect, evaluate, percolate, and learn." These much higher-order concepts are now the core feedback loop at the heart of the company's artificial development intelligence efforts. This is not just one of the biggest AI visions

• 2:45Nathan Labenz?

  you'll hear from any company in the world today, but one of the most well-specified, coherent, and, given REPL's track record for execution, one of the most credible as well. While it might sound like AGI, that is Artificial General Intelligence, Macaulay believes that REPL's focus on code, which, unlike natural human language, has been designed over decades to avoid ambiguity and is also now supported by a wide range of quality assurance tools, will improve reliability enough to allow REPL to unlock huge value without needing to confront the most perplexing challenges related to potentially smarter-than-human intelligence. Now, to be honest, I'm not entirely convinced that this AGI-ADI distinction

• 3:28Nathan Labenz?

  will prove as meaningful as Macaulay and team believe. And I will definitely be watching closely for new features designed to protect the next billion developers. After all, I've learned from experience training executive assistants in the art of AI task automation at Athena that the next billion developers will inevitably be very AI-dependent.

• 3:49Nathan Labenz?

  Still, I am convinced that Replit is a phenomenal platform for developers, both seasoned and new, and definitely a company to watch as the power of their product and the influence of their platform seem almost certain to continue to grow. As always, if you're finding value in the show, we would appreciate it if you'd share it with friends and post a review to Apple Podcasts or Spotify. And I also really appreciate your feedback, including guest and topic suggestions, which you can send either by email at [email protected] or via Twitter DM, where I am at Labenz and my DMs are open. I get a handful of very nice notes every week and they really do influence the direction of the show, so please reach out. I would love to hear from you.

• 4:33Nathan Labenz?

  Now, I hope you enjoy this fascinating conversation with Replit's new VP of AI, Michele Catasta. Michele Catasta, welcome to The Cognitive Revolution.

• 4:43Speaker 00

  Thank you, Nate, and thanks for inviting me. I'm excited to talk about Replit and AI today.

• 4:47Nathan Labenz?

  Yeah, me too. Regular listeners to the show will know that I'm a big fan of the Replit platform, which honestly even predates the current AI push, but that has also been a pretty exciting thing to see. I want to start with just a couple of real big questions and then we'll kind of come back and ask some smaller, more detailed ones as well. But for starters, you guys have put forward this ambition, I think one of the most ambitious, that you call ADI, Artificial Developer Intelligence. So what is that?

• 5:21Speaker 00

  So that's our, I would say, midterm roadmap. Hopefully it's not going to take us decades to accomplish that. And the idea is we try to identify an attainable goal in terms of how we need to evolve our current AI Replits to make developers much more effective. So the reason why we made a play on the name and made it different compared to AGI is I don't want to go on record ever trying to predict the date where AGI is going to happen. I think everyone agrees largely that it will happen, but we're not interested in figuring out in the next 20 years. What we care about is what can we deliver in the next few years to developers?

• 6:04Speaker 00

  I think L L M's have built an amazing foundation to help developers be more productive. GitHub Copilot was a pioneer of that. And we have GoSwriter and Rapid, which is helping you both to complete code. And we also have like a chat feature that helps you as a pair programmer while you're developing. I think this is all great and we're getting a lot of amazing feedback from our users. But what we describe in that manifesto is how can we bring that, you know, a notch up to the next level? And the idea is we will love to really empower the next billion software developers in the world. People that don't have any computer science background, they're just amateurs.

• 6:45Speaker 00

  They land on Rapid and they wanna go from an A I to a prototype in the shortest amount possible of time. We need an A I to make that happen and we need a much more powerful A I compared to what we have today. L L M's generate an amazing amount of semi-coherent code. Usually it's aligned with what user wants. At times it has some mistakes, it has some issues. I think that's not a blocker for people that are really experienced developers such as us. So I, I always know how to find my way out from a bug that, you know, G P T or Palmer or the models are giving me. This is not the same for a person who has never written a lot of code. For A I is our vision of how can we make sure that through several different iterations?

• 7:27Speaker 00

  And, and I'm gonna go a bit more in detail of what that means. Eventually you reconverge towards the right solution that the user was asking for. The way in which we envision that is there will always gonna be a fundamental generative component to that, which could be a much more powerful L L M compared to what we have today. Likely the field has been evolving at such a fast pace that I can only imagine what we're gonna have like, say, two years from now. So definitely they're gonna be more powerful, more independent. But Rapid has a key advantage of offering also an execution environment. And we've been spending years building all that kind of pattern. So now you can think of our feedback loop where an A I generates code.

• 8:12Speaker 00

  Then we execute it. We also include back the feedback from the user. We include also all the errors that came from the execution. And then we learn from these initial steps. We feed it back to an AI model, generate the code again until we converge to something that actually runs and it is aligned to what the user wants. So this is our ambitious goal, as we defined it before. Hopefully it's not too many years away. I have a feeling that maybe in a couple of years we can crack something pretty powerful. And we have been working heads down towards this goal since I joined Repli.

• 8:48Nathan Labenz?

  It's amazing how far you already are. I've been a user of the platform myself and I bring a certain amount of coding knowledge to it, but there's much more that I don't know than that I do, of course, and that's really true for everyone. So even for somebody who knows their way around, the built-in autocomplete and chat tools are really useful. But I've been kind of pursuing this myself. Again, listeners will know I'm advising a company called Athena, which is in the executive assistant space. And we're seeing that, in line with your vision, right, about the next billion developers,

• 9:30Nathan Labenz?

  we're kind of seeing that software is becoming much more permeable to non-software developers, much more accessible. And we're starting to use Replit and starting to teach what is ultimately coding, or at least it's like software development. But honestly, to people that have never even coded and don't know even the basics of coding, and we skip the traditional stuff these days. We skip the for loops and the syntax and all that kind of stuff and we just jump right into, here's this platform, you can start to talk to it and try to get it to do what you want to do. We actually do often have them go to GPT-4, which is something that's pretty interesting

• 10:12Nathan Labenz?

  because for the top level request, obviously GPT-4 is still the boss model and they need all the help they can get in many cases. So to get that first skeleton, we do often bounce out of the platform and go to GPT-4. But then once you're in it, you start to iterate, especially more locally, a lot of the built-in tools really work well. And it's been amazing to see how quickly people can go from, as you said, wandering in, not even expecting that they were going to be asked to code in this job, to being able to actually manipulate applications and get somewhere. Like, it's crazy. It's only 2023 and this is starting to...

• 10:53Nathan Labenz?

  Starting to happen.

• 10:54Speaker 00

  I agree. I mean, it has been an exciting ride. I would have never expected things to move this fast since we saw the first generative models based on transformers. I I could see like the light at the end of the tunnel. I felt it's it's about to happen. It's bound to happen, but who knew that the timeline would have been so compressed? And and now, as a matter of fact, as you're saying, we see a lot of people that have never approached software in their life being very productive. And for instance, at Rapid, we host monthly hackathons, and the vast majority of them are revolving around the AI topics. And a lot of people that come there are not software engineers by trade. We get amazing projects bid by PMs who spend most of their time communicating with software engineers.

• 11:40Speaker 00

  And as a matter of fact, they're extremely good at prompting models. Because what do they do for a living? They put in English requirements and descriptions of how software should work. So if you put them in front of GPT-4, they do an amazing job at describing exactly what they want. They get the output and Rapid helps them to stitch pieces together, helps them to get a quick feedback loop in case there is a mistake in the in the code. And in turn, you can also have a model helping you to debug the code. And we have a debugger, Fishfingers Writer, that does exactly that. When you execute your REPL and you hit a problem, you generate an exception. Then immediately we try to tell you that's how you should fix it.

• 12:21Speaker 00

  And we are seeing, you know, PMs winning hackathons just at their first experience writing code. And I I think this is only gonna be even more pervasive in the engineer future. And yeah, I feel Rapid is really enabling this wave, you know, more than perhaps any other company out there.

• 12:39Nathan Labenz?

  Yeah, Rapid has really done a lot of work in terms of teaching people to code in the traditional way. How close do you think you guys are now to a mode where you would say, "Forget that and let's go this PM route and then you can kind of fill in those, you know, lower level details later."

• 12:57Speaker 00

  My dream is that we're gonna go in that direction more and more. Of course, there are obstacles to that. Any AI is expensive, not only to train, but especially to serve when you have a lot of users. And I think right now we're, we just went past the 24 million plus users at this point. So it's not an easy platform to deal with. And whenever you're giving access to powerful models, you need to think also you need economics. That being said, the trend in AI and especially in Netherlands is that of a better economies of scale, pretty much showing up every quarter. So I do see a future where at least some basic AI features will be accessible to any user. And that will radically change the way in which people write software today.

• 13:41Speaker 00

  And I, that's a welcome change, to be fair. Like I, if I teleported myself back in time when I started to write code, I was like very young. It was an extremely frustrating experience. I, I don't regret going through that. It was a lot of fun in my sight. But if I had wrapped it when I started, I would have loaded much more. And I think there would be much more advanced today because I would have spent more time learning interesting things about coding rather than setting up my coding environment and debugging very stupid, you know, syntactic mistakes that I was doing because I didn't notice that it includes a parent or, you know, I forgot a quote here and there. These are things that are honestly not intellectually stimulating about coding.

• 14:24Speaker 00

  And, you know, we're trying to get rid of them as soon as possible so that you can really focus on the creative process of writing software. That is here to stay for a while longer. Again, I'm not gonna make timeline estimates of when people are gonna stop to write software completely. I do think that the creative process still has a place for a while longer. But no one loves to deal with low level, tiny mistakes in code. And yeah, and I'm, I'm happy to see that a new generation is writing, is learning to write code in a completely different way compared to how I did it.

• 15:02Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 15:05Nathan Labenz?

  It really is something mind blowing to see.

• 15:08Speaker 00

  And by the way, it's the same event for for experts because I've seen like, you know, there are some notable tweets. I think Andre Karpathy is saying, I can't imagine myself writing code without GitHub Copilot. I would easily waste like 50% of my productivity. And that's coming from a person who has been writing software for like as many years as I've been doing. He's a very prolific, you know, researcher. They will not, you know, get rid of the new AI features in writing code. And I think soon this would be agreement across every person who writes software.

• 15:39Nathan Labenz?

  Yeah, totally. So you came, just for a little personal context on you, recently from Google to Replit. And Google and Replit also have a significant partnership, which I think is multifaceted in terms of like investment and cloud and now even some product connectivity. Were you involved in that, in the creation of that partnership? Or were you just kind of separately attracted to Replit?

• 16:03Speaker 00

  I mean, I was I was involved to to an extent. I was there more like as a researcher, as middle management. So I'm definitely not the person who calls the shots about making such a, you know, being partnership happen. I was an internal champion for that. I was a big fan of Replit, you know, even before I started to work there. And I was giving demos of ARM to Amjad and his team back in the days because I was one of the contributors to the code skills of ARM, which we called ARM coder back in the days. So I think that that was my first interaction with the Replit team more officially. And then I stayed in touch with them. I unofficially advised them for a while. And then I became an advisor

• 16:45Speaker 00

  and then all the way to join them full time. So it was a slow cook. I love to believe that I played a role to make that happen. And, you know, after I left, I'm glad that the partnership actually happened. And it's a great partner to have by our side. Replit is a very resource intensive company to run because we give compute to millions of users. So it's something that we couldn't do on our own for sure.

• 17:12Nathan Labenz?

  Yeah, that's interesting. That might be worth getting a little bit more into the weeds of. Before getting there, you know, it's, in some sense, you've just come from maybe the most formidable AI company in the world, certainly with, you know, probably more. Top notch researchers than anyone else, probably more publications than anyone else. Certainly, you know, more resources, more compute. And you've gone to, I think, probably the smallest company that I have on my live players list at basically 100 people at Replit, right? So.

• 17:47Speaker 00

  We're getting close to that, yes. So it is that small.

• 17:51Nathan Labenz?

  Close to, but still under 100, yeah. So how do you think about that trade off? And kind of what should we be watching for? I mean, typically, these new technology waves are, you know, startup favoring. But I've been pretty commonly of the opinion, you know, as I look at different markets, that I think the incumbents are going to do really well in this case. And you're going from, you know, one form of strength to another. So how do you think about the the trade offs in scale and resources?

• 18:17Speaker 00

  Yeah, so of course, there is a trade off. Some things that I'm currently doing at Replit will be maybe much easier if I attempted to do them at Google, just by the sheer amount of resource available in that company. At the same time, there is a, there is such a strong benefit in being lean and having less process and being able to move fast and iterate faster and don't fixate perhaps on mistakes because they are considered as sunk costs. Replit, you know, we can decide this direction didn't pay off. Let's do exactly the opposite. I, I love it. It's a breath of fresh air. I don't think I belong long term into big tech. I'm very grateful at the experience that I had. I met a ton of amazing people.

• 19:01Speaker 00

  As you said, the density of AI talent at Google is incredible. But then at the same time, I, I feel, you know, at home being at Replit because we can, you know, we have a very narrow focus and we're very much obsessed to give the best possible AI to our, to our developers. And it doesn't mean that we necessarily have to build a. A team exactly as the one in Google Brain has. We can be complementary and the partnership in a sense helps us to make that happen. Because you know, we get access to models and we get access, maybe you know, we can have discussions and we can help each other in the process. And I think there is space for everyone, there is space for incumbents and I'm glad that they are there because you know, it would be

• 19:47Speaker 00

  short sighted to claim that OpenAI and Google have been changing the AI landscape thanks to the APIs that they offer. But then at the same time, there is also space for startups that are capable of building new generations of products that an incumbent wouldn't be making a bet on. So I like to use this example with office productivity. I wasn't surprised to see that Microsoft and Google immediately integrated GenAI in their office productivity. And I wouldn't be going against that kind of product as a startup. If the product look exactly the same, as in I wouldn't try to crawl in Google Docs with GenAI because an incumbent will do that much better. Now, if you come up with an idea of a completely different way to edit slides or documents,

• 20:32Speaker 00

  which has GenAI as a first-class citizen, that I would tell you, that's a very good company. That's something that will have its own space in the in the market. And you know, we rapidly kind of play the same game. We're building something that doesn't exist. We know that a lot of users love that and you know, we're gonna keep building that with the help of the incumbents as well.

• 20:52Nathan Labenz?

  Yeah, there's even a frame there where I would say in some ways, while it's small, you know, in terms of headcount. In some ways, Replit kind of is an incumbent. And by that, I mean, you know, again, going back to teaching these EAs how to code without really teaching them how to code. When I was introducing Replit to them, I was kind of like, it's gonna be hard for you to understand how much better this is than the old flow of like having a local development environment and then having to, you know, keep your settings in sync and your Python, you know, environments all whatever. It's just a total cluster. Obviously, everybody who's experienced it kind of knows that. But if they haven't experienced it, I was like, to you, it's just gonna feel like

• 21:35Nathan Labenz?

  this is how it always naturally would have been and you'll never know the difference. So. I think that, you know, in many ways, like the strength of the platform itself and just the, you know, all the provisioning and all of the infrastructure and all of the seamlessness that, you know, has obviously been created over time is, puts Replit in kind of this unique position of being both small and in a meaningful way, having like an existing technology mode that's super complementary to now an AI layer put on top of it. So I guess, I don't mean to pitch the business too hard, but as you can tell, I'm definitely a fan of the, of the product.

• 22:12Speaker 00

  I appreciate that, you know, I'm the same, otherwise I wouldn't be where I am today. But I agree with you, like, that's, that's one reason that drove my choice, because I think that we have such a strong advantage in what has been before I even joined the company, that, you know, I think it's an amazing playground where to apply AI and where to evolve AI. That being said, I think as a company, we always love to keep a low profile in a sense and not become complacent. So as much as I would love to believe, oh, we built something that is unique and is gonna help us grow over time, I love the fact that we still all feel the urgency of becoming better and growing and making our users happy.

• 22:55Speaker 00

  And it still feels as a small startup to an extent. Like, I've been a co-founder in teams, much smaller teams, and I feel the same urgency right now in a company that by some silicon valley standard, you could consider it as a company is successful, but no one inside feels, oh, we made it. We all talk about, oh, we need another three years to do this and five years to do that. And I love this kind of long-term goals that we have in mind to grow and become better.

• 23:24Nathan Labenz?

  So that maybe speaks to kind of the company's identity, and I wanted to ask a little bit about this because people refer to Replit in, in ways, you know, like, in my case, on my list of live players, it's also appeared frequently on lists of like most EAC companies. So do you guys like identify as an EAC company? What does that mean to you? Do you want to wear that? Label?

• 23:49Speaker 00

  I don't know much of these things, to be fair. I'm more of a spectator on Twitter rather than anything else. I, of course, you know, we're a tech company. We are focused on AI. We, we love progress and, you know, we are providing a novel way of writing code to users. So by definition, we're a company driven by a will of contributing to the technological progress and, and we strongly believe that giving development capabilities to the next billion of people is gonna make the world better. Now, I don't think any of us likes labels, both as individuals and as a company. As a matter of fact, I think we have a, we have a webpage for like the new hires where

• 24:30Speaker 00

  we say we are apolitical, like we don't take a stance. And I think we, we love, we love exceptional talent. We love to work with people that are respectful and at the same time, we respect any point of view. Um, so sure, like I'm, I'm sure that I'm, I'm just interacting with some yack people on Twitter, but you know, for, for those of you who follow him, I think he's a, myself have been knowing him for more than a year and now, I wouldn't be able to give him a label. Uh, like it's, it's such a unique person in his own beliefs and in his own rational way of, of seeing the world that I, I just find him to be an inspirational leader in the company rather than a person that, you know, belongs to a certain, uh, clique of people on Twitter.

• 25:13Nathan Labenz?

  I wonder how that kind of apolitical, I mean, there's a lot of different takes on this, right? We've gone through waves over the last few years of like highly political, even activist discourse at a lot of companies, including big tech companies. And I'm sure you, you know, were there for some of that. And then there's been this kind of movement away from that and saying like, okay, we're a startup, we want to focus on one thing and that's our mission and we don't want to be political. And it sounds like for most things, that's probably kind of a pretty simple, you know, relatively straightforward policy for a company like Replant. But then when it comes to AI, it's a kind of tricky again because it is both core to the mission and it's like inevitably kind of becoming political, but also as people

• 25:58Nathan Labenz?

  that are advancing the technology, like you obviously have real responsibility, you know, in the ways that you do that. And it would be easy for that to kind of bleed into the, the increasingly political. You know, discourse around AI. So how do you guys navigate that? Like, what is the internal AI discourse at Replit like?

• 26:20Speaker 00

  Yeah, I think that political and responsible are two not necessarily like power tracks. And in a sense, I'm not a big fan of the fact that, you know, politics bleed into AI. Conversely, I would love us as a committee to be very focused on, you know, building responsible AI systems. I think that the fact that we partner, for example, with Google and that we work with OpenAI and like we are surrounded by companies that I do believe are putting a certain amount of effort in that. No company out there is perfect. I think there is no playbook on how to do responsible AI today. And it doesn't mean that the research is working on that, not delivering. It's more about, it's a new field.

• 27:04Speaker 00

  It's gonna take us years to figure out how to do it correctly. If you think about social media, it has been around for 15 plus years at this point. I will argue that we haven't figured out still how to regulate that correctly and how to do it responsibly. So AI is much newer. AI is arguably way more powerful, especially in potential. So it's gonna take us as mankind a while to figure out how to do it correctly. Now, as Replit, I think we are in a privileged position where, for instance, we don't generate text. We rather generate code. So some of the shortcomings about, I don't know, having a model online that generates fake news. Maybe they don't apply directly to us.

• 27:48Speaker 00

  We are not in the realm of like image generation and all the issues in our regard, you know, using stock images that are copyrighted. We don't do face detection. So I would say we dodged some of the hottest short-term issues about AI. I don't think, we're not part of the discourse about AI because I think. Is not in our radar at the moment like we have way more grounded to planet earth technical challenges to work on, you know, before we think of that. And again, like I'm glad that there are other institutes and other companies thinking about it. So the the the internal discourse that we have now about AI is mostly

• 28:27Speaker 00

  how do we make sure that, you know, our users are aware of what we do. And that's the reason why, for example, we even release our L L M is open source and we use a open source data set. So we try to be as transparent as possible when it comes to our AI features. By all means, I'm not claiming that everything we're doing is perfect. But, you know, by means of being as transparent as we can, we get a lot of feedback and we can correct course easily. So it's better than working in a vacuum at least, you know, like we know and we get to hear what people believe we do wrong and we try to fix that. So I think, yeah, responsibility is important to us. The political discourse, I don't think we even have the cycles to spend time on that right now.

• 29:11Speaker 00

  And we have, we have to put so much work on growing that and that would be, I think, a waste of time in the short term.

• 29:17Nathan Labenz?

  You guys recently put out this state of AI development report and one of the things that had been jumped out at me was the rise of Langchain, which we've talked about many times, you know, many different episodes. It is, I've, I've used it like, you know, moderately, but not like super intensively. Because the biggest projects I've done actually got underway before Langchain. At the same time, it's taken a lot of heat recently online for being, I'm not sure, like people suddenly like for a second, it seemed like it had flipped and it's not cool anymore. And I'm not entirely sure why. So I'd love to hear your comments on kind of Langchain in particular and that tool ecosystem in general and kind of how you would guide, you know, developers.

• 30:01Nathan Labenz?

  Towards or perhaps away from certain tools.

• 30:05Speaker 00

  So first of all, I wouldn't guide anyone away from LangChain. I think it's such a fundamental tool in the AI space right now that at the very least is interesting to follow. And I'm grateful, you know, Harrison and the whole team using it. I find it amazing that it's basically collecting most of the research papers that come out. They also have an implementation of LangChain within 24 hours. Uh, so it's a, it's a treasure trove regardless of how much impact it has in production. Now that being said, I think LangChain is going through the same life cycle of frameworks in general. I, I remember back in the days, you know, when we had the first powerful web frameworks such as Django and Ruby on Rails.

• 30:50Speaker 00

  Uh, they were getting exactly the same heat in the sense that when you force or when you recommend a developer to work in a specific way. Uh, some people will love that kind of mindset and will be compatible with that. And so people will find it like completely uh counterintuitive or they will hate the fact that debugging LangChain is of course more um challenging compared to debugging your own code. Uh, so it always comes with, with pros and cons and that's the reason why even as of today. If you wanna write a basic web application, you probably have a choice of at least like the top 10 frameworks out there. And each one of them is good and each one of them is powerful. So LangChain is getting beat because it's probably the first one covering the space.

• 31:36Speaker 00

  I wouldn't be surprised to see a few more popping up in the, you know, in the next months. And I think that the code base, no matter how intricate it could be today. If first of all, it can always improve. And again, we've seen this in other frameworks that. Maybe Tensorflow went through something similar, although Tensorflow 2 wasn't exactly such a story. But you know, even in deep learning, we have seen Labrys coming out in version one, collating all the feedback from users and then the next big release is much better because the developers learn the best practices and the anti-powers. So let's, let's give some time to Arizona and the team to, you know, maybe make a next little chain. But I think this is the nature of AI development today.

• 32:22Speaker 00

  It's early, it's scrappy and it's gonna be headache inducing. But that's also the reason why it's fun.

• 32:29Nathan Labenz?

  I think it's a really great answer, great perspective. And I definitely would encourage everyone to follow the project, at least as you said, they are incredible. You might even call it Replit Speed, you know, they, they might coin their own Langchain Speed. I've seen not once, not a few times, but a lot of times, as you noted too, where they very, very quickly implement some new scheme that somebody published, you know, in a research paper. And that's incredibly useful from a developer standpoint.

• 32:59Speaker 00

  Yeah, that's making AI better as a field, if you think about it, because whenever a paper comes out, there is always some doubts about, you know, how easy it is to reproduce. Are those results real or not? And the fact that the implementation, either sometimes the authors themselves building Langchain so that it's easy for people to reproduce, or someone in the community is gonna take care of that like within a day. That allows us to find out, is that paper actually real? Or are those claims completely false? And you know, that helps us to make progress as a field. So even if it doesn't have an impact in production, it's so valuable on the research side. That again, I'm glad that we have so many people working in AI today compared to even six months ago.

• 33:43Nathan Labenz?

  It's powerful. That, that also just makes sense to me as to why I wasn't bothered by any of it, because I generally am fine to follow the framework. So, uh, you know, I, I think I'm in the target, I'm in the framework target market, I guess. You had said, you know, with all these tools and especially all the AI tools, obviously, you know, the goal here is higher productivity, greater, you know, accessibility, more developers. As I understand your earlier comments on the ADI notion, it's like, we're not looking to replace developers. We're just looking them to give them, you know, a next generation set of tools that. Will make them much more productive. Do you have a sense for how much more productive we're talking about here? Like, are we talking 2X more productive? Are we talking 10X more productive?

• 34:27Speaker 00

  I would say the 2X has been already kind of accomplished by code completion. I think in the process of at least writing codes, Copilot has published some metrics where they make developers up to 55% faster. We have something comparable. We didn't rapid, we didn't publish the metrics yet, but we see high acceptance rates from our model. So I would say the 2X threshold has been kind of reached. And also, informally, I've seen several people tweeting about the fact that they would be wasting half of the time if they were not using modern code completion. I think 10X is attainable, achievable, like in the next year or so.

• 35:11Speaker 00

  And that's by means of adding better ways of debugging code, more agentic behavior. That's another maybe direction that we can go more in that later. But this idea that you don't want to use an AI exclusively to generate blocks of code. You want an AI that comes up, first of all, maybe with a plan of what you need, like in terms of architectural design. And then that can go ahead and be the scaffolding of a project and then gives you the basic implementation of everything that you need. And then as a developer, you just put the finishing touches, you put some glue code, you make sure that a few bugs are done. That I see easily giving you like a 10X improvement.

• 35:55Speaker 00

  I think I'm just having a few inspirational podcasts where you talk about the 1000X developer. So that's our NoStar, which I think is going to take us a while. And the idea there is the moment you have a model that can not only generate what you need, but can also orchestrate tools that are really available. And this is something that I explained in the EDI manifesto, talking about at the beginning. Then all in a sudden, you have Like a group of developers that work for you. You can think of yourself as a tech lead. And then you have like basically the A D I below is a set of interns and like junior developers that can get tasks done immediately.

• 36:36Speaker 00

  So if you know that you need a certain pipeline, you need to fetch data from somewhere, you don't have to write the code any longer. Maybe you have a tool in Rapid that does that. And the A D I knows, oh, I should be invoking that service, getting back the results, processing them in this way, and then bubble them up to what the developers are writing. So that's the future that I believe will start to give orders of magnitude improvements. But yeah, I think ten X is something that we should be seeing relatively quickly. The A D I agents are going to become more powerful and more independent, which is far from being an easy task.

• 37:14Nathan Labenz?

  I'm feeling a little cognitive dissonance around kind of another billion developers, but also, you know, ten to even a thousand times more productivity. Because I'm kind of thinking, what, it's hard for me to even fathom like how much software gets built, you know, or sort of what we would be all building in that scenario, you know, if we're, if we add another billion developers and make developers just say just 10 times more, right, then we have like the equivalent of, you know, at least 10 billion current developers. And that means we have more than one current developer productivity per person on earth. Do we even have, like, is there demand for all that software?

• 37:57Nathan Labenz?

  Like, what do you think, taking the, you know, taking the other perspective, like not thinking so much about the developer experience for a second, but just kind of, what does software even look like if there is more than one current human developer productivity per person available to be building stuff?

• 38:15Speaker 00

  Right, no, that's a great question. I think it's hard to make a, to make a good prediction about that, you know, but what I can tell is. Every person who today is using any kind of technology is spending quite a lot of time on tedious tasks. And that's the reason why, for example, like you have shortcuts on iOS that allows you like to chain in a row like a few actions. That's why like a lot of people do macros or functions on Excel spreadsheets. And that's why there are variable products today. So you see these hints of programming languages and automation bubbling up in every widespread application. And the reason being, everyone who's a bit tech savvy will love to automate even further their life.

• 39:01Speaker 00

  So to me, having the next billion software developer doesn't mean that, you know, 90% of them will be writing the new line chain or the new Google or whatever it's gonna be. It's more about can we make programming so pervasive that it becomes like a useful skill for everyone? And I think there is a future for that. Like I've seen people building like WhatsApp bots during some of our hacktions, you know, for very specific needs, like, you know, things that were ordering pizzas for a group of friends, organizing a party. So rather than having to download applications and all the startups build that kind of niche product that is never gonna have a powerful business model,

• 39:44Speaker 00

  you're gonna build yourself, you know, the few niche things that you care about, you know, for yourself, for your family and for your friends. So that's the one, that's the horizontal growth that I'm talking about. So the one billion software developers. Now, on the other dimension, that'd be vertical growth, how much a software developer can, you know, can become more productive. I think that multiplier, like especially 1000x, doesn't apply to the amateur person who's automating their life. It applies more like to the John Karmax of the world or the Jeff Bezos of the world, like people that are extremely skilled software developers. They manage very large teams in their lives and all in a sudden, they are not gonna need anymore a full org below them.

• 40:28Speaker 00

  You know, like I think Karma wasn't matter for a while and Jeff Bean's probably managing. As thousands of people reporting directly, directly over to him, imagine replacing that or part of what they do on a daily basis with an AI and with an agent and with an AI. That's the account of thousand X that we think of. And the goal is not to displace other developers. It's actually to make even more people capable of building big and great companies. So we will love to see fewer humongous companies and way more mid-sized startups that can tackle daily challenging problems. If you think about it, that's going to be a big accelerating factor for mankind.

• 41:10Speaker 00

  I don't want to go back to the EAC, but that's the acceleration part that I'm excited about. And that's the reason why I've been doing research in this field and I want to build products in this field. Because it can really have an amazing net benefits for mankind if we crack this problem correctly.

• 41:24Nathan Labenz?

  Yeah, I certainly see the billion-fold opportunity in kind of last mile customization and integration of different things. That's honestly, I think, a big part of what the EAs at Athena will end up doing with a platform like Replit. Is they're going to kind of say, and I'm developing all these mantras. One of them is copy and customize. And Replit obviously is, you know, perfect platform for that. Where, you know, the goal is to find a good starting point template, fork it, make it your own. And a lot of times those things where you're making it your own are not conceptually complicated.

• 42:09Nathan Labenz?

  It's just like, I use this messaging app and I want it to trigger, you know, a certain thing when a certain thing happens or whatever. Or I use this task tracker or I use, you know, this database or spreadsheet, you know, whatever. Right. There's all these kind of, everybody has their own kit. And there's not really a great way to bring those things together in some universal sense. Although Zapier certainly, you know, and others kind of try. But even then, you got to like make Zapier work. So that part definitely makes a lot of sense to me. I guess I'm reminded of, we had Flo Cribello on, actually at one point, I think on the same episode as I'm shot earlier, and then also he did his own. And he talks a lot about this like single-use software paradigm where he holds up an aluminum can and he's like,

• 42:54Nathan Labenz?

  When Napoleon was alive, you know, he was the only guy that could drink out of aluminum because it was so special and now we just throw it away. Is that kind of how you're in, am I in line with your vision here of just software kind of just spinning up in little throwaway moments all around us?

• 43:10Speaker 00

  Absolutely. Yeah, that's exactly what we want. And in a sense also, we want to empower the creative process of software, not the excruciating aspects of developing complex software, because that for a while longer, it will be still the responsibility of larger companies. It will require process, it will require management, it will require PMs, DPMs and so forth. So building large software still requires a lot of human effort. But allowing a lot of people to be creative, we also expand dramatically the pool of good ideas that we explore as, you know, as mankind. So I think, you know, the analogy that you were bringing before from a judge and drinking from a little can makes a lot of sense.

• 43:55Nathan Labenz?

  So as I hear you describe that future, I guess it seems to me like there's some sort of implicit assumption where, for context, I kind of would describe the state of play today as AI has become better than the average person at a random task. And certainly that includes coding, right? AI compared to person off the street, AI is going to dominate in code. And AI is closing in on expert performance where there's a pretty well-established protocol of what you're supposed to do. So things like answering medical questions, when the answer is pretty well established

• 44:38Nathan Labenz?

  and you as a doctor are supposed to know, then the AI is also getting like really good, closing in on doctor level for just like answering the question in the right way. And then there's this kind of third tier of Like the real human genius or creativity of coming up with the spark of a new idea, whether that's a scientific hypothesis or an application concept or whatever. That basically, I don't see AI doing anywhere as of now. And it seems like your model underli-- you know, your expectations underlying a lot of what you're saying are like, it levels out before we get there. Am I interpreting that correctly? It seems like you're-- we already can do the basic stuff.

• 45:22Nathan Labenz?

  You want to enable like closer to expert so that, you know, obviously it can be more and more useful, but you're just not expecting that third tier of breakthrough where AI's like actually drive creative process.

• 45:35Speaker 00

  Correct. That's exactly why we called it ADI and not AGI. The reason is the creative process you you talk about for me is still exclusively in the realm of human intellect. And being able of having an AI that designs and thinks of the next line chain, for example, that to me means that we are capable of building an AI that is as smart as a human or even smarter. I don't care about when that happens. If it happens one year from today, of course, it's gonna change drastically the way in which rapid looks like, but it will be accessible to everyone most likely. So it will not disrupt only rapid, it will disrupt every single company in the world.

• 46:18Speaker 00

  It will be like a disruptive factor for the whole mankind. But whatever is the timeline for AGI, there is this interim solution where, as you said, we level out, we help as many developers as possible to gain, to be more efficient on basic skills, but then the creative process is still, you know, something that only humans can do. And one example I always use when people ask me, shall I drop out from my computer science degree because maybe there's not gonna be a job for me in the future? The answer I give is-- We still need people coming up with the design of the next library and the next programming language. So if you think about it, what they've been doing in the last sixty plus

• 47:02Speaker 00

  years in computer science is literally thinking of new abstraction layers. We were working with punch cards and then assembly and then C and then we started to do operating systems and kernels and libraries and web frameworks. So every time there is a new paradigm shift, that has been humans working together, learning from the anti-patterns and the pain points of how we're running software today and then coming up with the next we do that. And AI can learn how to do things as we write code today. It can't think of how we're gonna be writing that in two years. So that is still something the humans have to work on. And it's great to me that if we

• 47:45Speaker 00

  reduce the amount of tedious tasks that we have to be working as developers, then everyone can focus on how can we make things better compared to how they are today. So this is the dream that we have, you know, that's the reason why the AI for code should exist, is to make sure that we can put our brain cycles on something more, you know, more interesting than just writing code as we do today.

• 48:10Nathan Labenz?

  Just to make sure I understand your outlook. It sounds like you're not, by any means, like ruling out that there could be this next advance such that, oh my god, now AI can drive creative process. You're not ruling that out. It doesn't even sound like you're saying it's particularly unlikely, you know, as you said, hey, maybe it could happen next year. But it's basically just like, nobody has any idea what to do about that. So we just kind of plan for a more normal scenario because that's all we really can plan for. I mean, normal can still be pretty weird, but it's like, near-human level developer would be transformative in and of itself. But that's kind of the most transformative future you can plan for. And so that's what you focus on.

• 48:54Speaker 00

  Correct. Like AI has a history of going through like hype cycles and winters. And I think like in the early 60s, Marvin Minsky wanted to hire a summer intern for them to solve completely vision. We're talking about six plus years ago, and that problem was considered to be trivial. And then it took us literally 60 years to have like models that are powerful enough to match the vision that Minsky had. I, it would be, I would say, ill-advised for me as a person leading the IT team at RAPID to say, oh, let's just work on AGI because it has to happen in the next two years. And that's the only reasonable path for us to make. If I have to think about the probability distribution of events,

• 49:38Speaker 00

  all my probability mass is on AGI can happen and we can build it and we can bring benefit to our users. And if in the low light view that AGI happens before us, we're gonna adapt, we're gonna be there, like we're gonna keep our eyes open. We're deeply entrenched in the AI community regardless. So we're gonna be hearing about it because I think people are not gonna be talking about anything else across the world, to be honest, when it happens. But in the meantime, it's much more realistic to say, I'm glad that people are working on more long-term problems. And in the meantime, let's focus on something that I feel we can actually do.

• 50:14Nathan Labenz?

  Where I come out on that, just put my cards on the table, is like, I don't think we should necessarily continue hyperscaling orders of magnitude beyond what we have. Because I can't rule out that it might, you know, that a sufficiently powerful system might pop out of there to be like truly very, very disruptive. And I would rather give us a little more time to You know, get used to what we have and implemented and, you know, enjoy our AI coding assistance. Before we, you know, kind of say, hey, what happens if we scaled up another thousand X, you know, more training compute?

• 50:55Nathan Labenz?

  Where do you come out on that? Like, what? I mean, there's multiple different questions, but before we even get anything like regulation, whatever, just like, what would you advise if you were saying, you know, hey, Sam Altman, like, do it or don't do it, you know? Do you feel like if they do that, we are at some risk of crazy disruption? Or you feel like it's not, it's, it's more, that's more just kind of speculative than anything?

• 51:17Speaker 00

  First of all, given that I'm not a person who studied these topics in depth, my, my answer, you know, should not have a lot of weight. So I will just express what is my opinion, but you know, I trust much more people who have been studying this for several years. I'm, I'm far from being well-informed person or like responsible AI and, and all the different issues related to that. That being said, I think that no matter what is our opinion, the scaling will happen. I think, you know, there have been letters signed by several influential people and the progress hasn't slowed down at all. We know that even startups at this point have enough capital to build exceedingly large H100 clusters.

• 52:00Speaker 00

  Like Inflection is one of them. They just announced, you know, their interest is gonna be. So I think it's pointless to talk about how can we stop that? It's more important to pour our energies on, okay, once anything that happens, how can we be ready? How can we make sure that nothing goes in the worst possible way? I might be naive, but I tend to be macro optimistic about mankind. In the sense that eventually, no matter how many, you know, fault players are there and no matter how weird behaviors we experience in our history timeline, eventually it seems that as mankind will always find a way to survive and thrive.

• 52:40Speaker 00

  And so I do hope that, you know, even when a GI happens, we're gonna be driven by, by common sense. I think some of the doomsday scenarios that we have been, you know, hearing through mass media and Twitter are possibly just detrimental. I don't think it's worth to talk about them. They just. Give even more fear to people and we we came out from like two plus years of pandemic fear and now we just reached to the AGI fear. It's not helping anyone to be fair to to be that that pessimistic. I think it you know it it's important to think of ways to mitigate that in case it happens, but I wouldn't go around and get on the news and talk about

• 53:21Speaker 00

  nuclear bombing data centers. That doesn't seem to be like a healthy way to start a good discourse.

• 53:29Nathan Labenz?

  Yeah, and I do agree that that was probably unfortunately phrased at the at the best, even though I'm fairly sympathetic to some of the the argument, but when you get into airstrikes territory, it does doesn't do much for discourse, I'm afraid.

• 53:45Speaker 00

  Exactly. And you know, I think I'm glad that there is discourse going on regardless. Like I I I've seen people, you know, on both sides starting to have debates and, you know, maybe softening their points of view from from both hands. And I think that's the kind of discourse that is gonna lead to to progress and, you know, to us being ready as mankind for that event to happen. I think any extreme point of view, I think historically it never helped. So I I wouldn't overindex on the extremes for sure.

• 54:16Nathan Labenz?

  Okay, cool. Well, let's return then to some nearer term concerns. I wanted to just give you a chance to kind of talk about, in a sense, like what, you know, has you excited about Replit's future, but also there's this constant debate around, you know, who has moats in the AI space? What do moats look like? I just spent three minutes thinking, like, what are some moats that I see in the case of Replit? One we already talked about is just having the incredible infrastructure that you already have. And, you know, that even extends to like multiplayer mode. When I tell the assistants, by the way, you can jump onto my thing right now and start helping me code in the environment that I'm in. Here's the link. You know, again, I don't know if they appreciate how much that they're not

• 54:59Nathan Labenz?

  even as mind blown as they should be. You know, because to them, it just feels like, okay, that's something I can do. But yeah, historically, you know, people who've done this for longer would be mind blown. User feedback seems like another significant one. Like, nobody is getting, I would only identify two companies, Microsoft and Replit, that are getting the kind of inline, you know, frequency and volume of user feedback.

• 55:23Nathan Labenz?

  That you guys are getting today?

• 55:25Speaker 00

  Let me correct you there. I think we get even more than Microsoft, we have way more telemetry. Like VS Code is not logging at the level that we do, of course, because we are running in the cloud and everything is running on our backend, while VS Code is most of a time-based experience. So I would say there are two companies that have the data. One is Google, because they run their own internal ID that looks very similar to Rapid as a concept. And there are some papers that explain how the architecture looks like. And the other one is Rapid, but with a much larger user base. So that is a dataset that I'm extremely proud of, and I can't wait to show models trained on the data that we collect.

• 56:08Nathan Labenz?

  So a third one I was gonna say is community. And then within that one, maybe I'll ask a question around like, is Microsoft not logging? Like what co-pilots are completed? It's not for technical reasons, right? It must be more a matter of like licensing or positioning or sort of how they're relating to the customer?

• 56:28Speaker 00

  They do log, of course, you know, co-pilot interactions and that's how they can track so many interesting metrics in their blog posts that they publish. My point was more the fine-grained interaction that we capture, such as literally keystrokes. Like we have the complete edit history of what to do in the editor. So not at the level of commits, but at the level of keystrokes. And that is something that VS Code, to my understanding, unless like you install additional extensions, Microsoft is not snooping the code that you're writing on VS Code. Because that would be a PR nightmare for them in case it happened. But we do, because we offer features such as history and rolling back.

• 57:13Speaker 00

  And we have master player support, so we need to make sure that we can reconcile edits coming from different users. And that's how we, you know, we have to maintain all the additional data pipeline. So it's very unique data that one day we plan to pour into an AI model and give it back to the users.

• 57:32Nathan Labenz?

  Yeah, that history feature is pretty cool unto itself. It's even really more than like a Google Docs. It's kind of a playable history of everything that you've done in the environment and

• 57:45Nathan Labenz?

  It's another one of those things that feels like, man, why wasn't it always that way? It's quite a few of those that Replit has managed to create. But so going back to the community then, you mentioned kind of, you know, it could be a PR nightmare for Microsoft if they're like logging all your keystrokes. I mean, obviously they have enterprise customers, Replit's coming at it from a totally different user base where, you know, people sort of maybe more assume that like they're using a web app and that data is going to get logged. But how do you think about the

• 58:16Nathan Labenz?

  like the relationship between the platform and the community, obviously of the marketplace for bounties as well? And that, you know, could even be said to be like the perfect input output pair for a random user doesn't know much, you know, asks for this and this is the code that they got. So I see these kind of, you know, compounding dataset advantages. But how do you think about yourselves relating to the developer community? What rights should developers have? This could be, you know, specific to Replit or in general. There's, you guys are kind of, I'm sure, training on open source software and there's lawsuits around that. But then there's also like, how should I think about if I'm trying to create something original of value on Replit? Do I have an opt out? Should I have an opt out? You know, my stuff is going into the next generation of model.

• 59:01Nathan Labenz?

  Like, does that, should I feel like that's taking from me in some way? Or like, what do you think the terms of that engagement and relationship should be?

• 59:09Speaker 00

  Totally. So we were working on an opt out feature. It's going to be in our documentation very soon. I would send a tweet when it's up and running. You're totally right that we are training only on permissively licensed code. So, you know, we didn't want to make the original mistake that Copilot did. I think it's a, it's a gray area, like from a legal standpoint, especially if you are in the US, that there is, you know, you can follow the fair use. And that will, in theory, allow us to keep training models. And perhaps those lawsuits are not going to go much further because of that. In practice, I think it's way more important for us to be correct towards our users. So we're going to be listening to them.

• 59:54Speaker 00

  I think until now, we haven't received any pushback. And again, with the opt out feature, we think that we're going to be making it right for some people that want to make sure that their code doesn't end up in the models. By the way, we only train on code that is public for Replit. So if your Replit is private, we don't even touch it. And the rationale there is if you are keeping something private, you know, then you might have good reasons why. And then, of course, we don't want our models to be trained on that. Now, I don't think that's the end of it all. I think, you know, we're moving to ways of attributing contributions to users that are more advanced. So, and I, I think you've seen bug, for example, that, you know, they give you links to the

• 1:00:39Speaker 00

  snippets of the tab code that contributed to what bar is generated. So we're, we're, we're gonna be moving to that model in the future. And I think the, the best way for us is really to follow the discourse and see what's gonna become the industry standard. We wanna be like a beacon in terms of how we treat our users, our developers. Uh, so we, we're gonna try to adopt those features as soon as possible. And then if the lawsuits go in such a way that we change our mind, then, you know, we're ready to adapt. Um, but as of now, I think, you know, we're trying to follow what's considered a good practice.

• 1:01:21Speaker 00

  That, that being said, you know, if you want my, my personal opinion, I do think that the landscape will change as it's changing for image generation. And I think like stable diffusion was like the inciting factor to start the discourse. And, you know, maybe there would, of course, there would be lawsuits and there would be like even more debates. And perhaps in a few years from today, we're gonna be finding what is the right balance between giving attribution and still training such models. Um, I would love to see a model where if training data created by user contributes to a lot of code completions or generations, then we find a way to share part of the revenue with that user. It's far from easy to be built.

• 1:02:05Speaker 00

  I think it requires like a lot of infrastructure, both on the AI side and on payment side. But that's, you know, one idea that I was riffing with with on Jod back in the days. And, you know, perhaps when they were be able to to make that happen.

• 1:02:18Nathan Labenz?

  I don't know if that would qualify as an interpretability complete problem, but it's close, probably, right?

• 1:02:24Speaker 00

  It is close, yes, because like a lot of common code patterns will appear pretty much everywhere. So what do you do in that case, you know? How can you attribute it to the first person who came up with that? Or is it too trivial that it should not even be attributed to anyone? So it's it's a very hard problem to to solve. And what we do is we we keep an open mind and we read papers, we discuss with the community. And I I think we all have to adapt, in a sense, like companies have to adapt. Developers will adapt as well because, you know, there is progress going on. It was the same, in a sense, what happened with the open source movement, like back in the Linux days, was also another disruptive event. And some people loved it, some people hated it.

• 1:03:08Speaker 00

  So I'm sure this will happen here with AI for developers.

• 1:03:12Nathan Labenz?

  So one thing that you guys have obviously also made some news on, but which is not apparently something you think of as a competitive advantage, is training custom models. Or maybe you could correct me, but you're training them, you're open sourcing them. It doesn't seem like you view the models as any sort of secret sauce or kind of, you know, core defensible IP. So with that in mind, I guess I'd love to hear a little bit about how you guys think about training them in the first place. Like, why do it? You know, why not just use somebody else's? If it's not something you, you know, feel the need to ultimately own? Is it a data set issue? Is it, you know, just the desire to show off, which honestly, you know, might be good enough reason?

• 1:03:57Nathan Labenz?

  Yeah, tell me about why create an open source these these original models.

• 1:04:02Speaker 00

  Of course, you know, showing off or more like proving that, you know, we're capable of doing that and we have those skills in house definitely helps because, you know, it helps us to talk with a lot of people in the AI community who then ended up maybe joining us or deciding not to to help us in the future. So it's important, you know, for a company not to just talk about AI, but rather to prove that they can do it. So that was a way for us to, you know, let us stand out. That being said, Rapid is a Great success story built predominantly on open source software. And we love open source at Rapid. The buzz major, the teams we use and the projects we rely on. We either donate back to them or we do like a lot of upstream contributions.

• 1:04:47Speaker 00

  So Rapid wouldn't exist if open source wasn't such a thriving community. So we felt the moment we decided to work on our models. We almost felt compelled to give back. And it came almost as a no brainer discussion. I mean, I wouldn't say that it took us half a day to make the decision. Like, of course, we talk about it a bit internally. But at the end of the day, there was like a complete agreement that it was the right move for us to make. Now, I don't think that means that we don't see any competitive advantage in the models. As a matter of fact, we released the baseline model, which is trained on an open source dataset. Now, we were grateful for the big code project to have open source in the stack,

• 1:05:30Speaker 00

  which is a permissively licensed dataset extracted from GitHub. But the model that we today use in production is a combination of that dataset. And then it's a further pre-training done on Rapid data, uh, which is only accessible to us, of course. Like, we're not gonna be releasing, uh, our user datasets, uh, our user code bases. And that model comes with a pretty substantial performance improvement compared to the base one. You know, depending on the language, we have been seeing all the way even to up to 50% improvement compared to the base model. So that's how the way, you know, we kind of reconciled, uh, the two aspects. Um, on one hand, it's great to give back to the open source community.

• 1:06:11Speaker 00

  And we got a lot of cool projects built on that, honestly, more than we ever expected. And then at the same time, you know, we built our own custom model that is in production since early May, if I recall correctly. And it's been doing great. And yeah, I remember there was another sub-question that you asked, which is, why did you even do it? Why don't you use third-party models? We had in mind a very specific trade-off, as in, we had a fixed point in terms of latency. We wanted the model, you know, say, um, most of the responses, we want to give them back in two hundred, two hundred fifty milliseconds. That's the threshold that appears to be basically instantaneous for a human.

• 1:06:52Speaker 00

  So we had to work with a certain model size, given the GPUs that we had available a couple of months ago. And we decided, okay, given that we want roughly a three billion parameters model, can we squeeze the best possible performance out of a small model? Uh, so that we give a different experience compared to Copilot. You use Copilot, so you know that usually the latency there is at least one second. And our users love the fact that maybe we generate shorter completions, but they're much faster. So we wanted to build the best possible model for that use case. It wasn't available open source, and you know, we, we took the challenge and we made it happen.

• 1:07:30Nathan Labenz?

  How much of a hassle is that? I know that you have worked for at least some of these projects with Mosaic, ML, another recent, uh, guest. How much of a lift, you know, and, and if you are a, uh, like, I think a lot of people listening here, you know, probably work at companies that are like, trying to figure out what they're supposed to do. And I think for most of them, certainly pre-training their own custom model is not going to be the thing. Fine-tuning is more in play for most, but even honestly, for a lot, they don't even need that. And then, you know, in some cases, they just need to like, set up a vector database and, you know, and get a chat running against it. But how, it seems like it's getting pretty accessible to do these pre-trained things if you have the data and the need.

• 1:08:14Nathan Labenz?

  What was your experience like working with Mosaic to actually go through this whole process?

• 1:08:20Speaker 00

  Yeah, Mosaic was a great partner for us. I don't think we would have been there to work on that project with such a short timeline if we didn't have them, you know, uh, covering our backs and giving us access to their infrastructure. So the, um, the, the truth is, training another L M today, uh, still requires quite a lot of engineering lift, uh, when it comes like. Orchestrating several different nodes and GPUs, writing the right training framework to build your model, doing a lot of ablation studies to find out the right architecture, the right hyperparameters and so forth. So in the last few months, it's true that we've been seeing quite a lot, a lot happening

• 1:09:05Speaker 00

  in the open source world, but as you said, the vast majority of progress we see in open source is about fine tuning, fine tuning and instructing. And all of those tasks can usually be accomplished with a couple of consumer-grade GPUs. Way fewer companies and people are training from scratch models. So to attempt that task with a very small team, in my case, we were basically two people and a half working on this for a sprint of 10 days before we released it and maybe a couple of weeks before doing ablation studies. That happened only because we had access to Mosaic infrastructure and also quite a bit of back and forth with the engineers and the researchers there who helped us a fair

• 1:09:50Speaker 00

  share during that process.

• 1:09:52Nathan Labenz?

  So your job was largely to find what good looks like and assemble the data and they kind of handle babysitting all the GPUs, orchestrating obviously, and also that kind of hyperparameter expertise?

• 1:10:09Speaker 00

  So yeah, largely we work on, on our hand, we work on data collection, curation and a lot of ablation studies with smaller model sizes. And then Mosaic app does, you know, with their software framework that orchestrates multiple nodes. We mostly babysat the run ourselves in the sense, you know, we were in front of Waze and Bases 24/7, seeing if something was going wrong. But anyway, when, when we hit a snag or when there maybe were some harder failures than, you know, Mosaic people were on call pretty much as us like 24/7, it was like a pretty, pretty amazing team effort that we did together. And yeah, we shared a lot of notes, for example, on, you know, what was the best

• 1:10:54Speaker 00

  learning schedule that you use for this model size. And we made different choices, for example, compared to the models that they use open source. So depending on how much you want to personalize the model, you can go all the way to like a wide globe service where you put data in a bucket and you ask Mosaic or other similar companies to do the training for you. But then you get something not customized to your needs. For Rapid, we train our own vocabulary. We had a different architecture. We, you know, we, we, we come up with a different even training schedule in terms of like how many times we repeated the data, how many epochs we did on the code data set that we used. So it was kind of an uncharted territory to an extent. And that's why it was a lot

• 1:11:36Speaker 00

  of fun, you know, I felt I was doing research and product at the same time, you know, and that that was a fun process.

• 1:11:43Nathan Labenz?

  Yeah, I love that about AI right now, that the, the blurring of research and development and productization is, is super fun for me.

• 1:11:53Speaker 00

  Yeah, people ask me, do you miss doing the research at Google? And I said, you know, I'm, I'm maybe I'm not doing that full time, but I still do a lot of cool stuff.

• 1:12:00Nathan Labenz?

  Any highlights that you would give? I mean, it's, it's interesting right off the bat that you trained your own vocabulary. That essentially means you have your own tokenization scheme, right? So we, we just did an episode with a, a woman named Lily Yu from Facebook, who was author on the Megabyte paper where they're kind of trying to get rid of tokenization altogether.

• 1:12:21Speaker 00

  Yeah, I saw some of the highlights of that talk. It was amazing. Yeah.

• 1:12:24Nathan Labenz?

  I don't think the transformer is the end of history, you know, as it currently stands. That was definitely a big takeaway. But so any, any kind of notable details that you would want to highlight there? And, you know, any big differences between your tokenization scheme and kind of what people are used to, or any architectural decisions that you think people would find particularly interesting?

• 1:12:42Speaker 00

  Two key insights. The first one is we, we went with a smaller vocabulary, which by the way, is the same size as the first Lama release. Honestly, I have to read in detail the Lama two paper. I, I would, I would be surprised if they went for a larger vocabulary. But anyway, a smaller one comes with the price of adverse compression in exchange of better inference speed. And again, given that our model is optimized for Being in production rather than benchmarks, we made that design choice of going with a smaller vocabulary.

• 1:13:15Nathan Labenz?

  Can I just ask how big the vocabulary is?

• 1:13:18Speaker 00

  Oh, sure. It's 32K.

• 1:13:20Nathan Labenz?

  So solid gold magic carp is right out.

• 1:13:23Speaker 00

  Exactly. Yeah. I think GPT three, for a while, I did like a 50K vocabulary. GPT Neox, you know, there's a rounded ballpark, the 3.5 Turbo and four, I think they all have 100K vocabulary. So that's why we start to say that a token is usually around four characters. With the kind of vocabulary size that we use for the rapid model is roughly three characters of token. So it's a lot of worse compression in exchange of faster inference. But also, we specialize our vocabulary on code. We train it on a exclusively code dataset. And that means that specifically on code, it achieves a better compression rate.

• 1:14:05Speaker 00

  So what you what you will miss as a whole, if you use that vocabulary for a standard natural language dataset. In reality, you gain that back when you work exclusively with code. And again, we could make all these choices because we knew exactly that the model was meant to do one thing and one take only. So we're not building a generic LLM. We're building one specialized to code completion.

• 1:14:28Nathan Labenz?

  How do you think about kind of the world knowledge that underlies a lot of programming tasks? The one thing that I've tried, for example, is write me a tic tac toe game in, you know, JavaScript or whatever, right? If I go to, I mean, obviously that assumes or requires that you know tic tac toe, wouldn't know what it is and what the rules are and, you know, have that kind of. Maybe there's enough tic tac toe demos out there that that kind of finds its way into a coding set, but you can imagine a lot of things like that where there's general knowledge assumption in the program, you know, specification that a code only dataset probably doesn't have because that stuff doesn't make it all the way into

• 1:15:13Nathan Labenz?

  the code itself, I guess, in a lot of cases. How do you think about, I mean, that that starts to go from the like ADI to AGI kind of a little bit as well, but. How do you think about the world knowledge that you do or don't want to bake in?

• 1:15:26Speaker 00

  So I think small models showcase a certain degree of our knowledge. Out of curiosity, we ran basic natural language benchmarks on the rapid model. And it turns out that it was on some of them even competitive with Llama 7B, which is a much bigger model, especially the self-contained benchmarks that didn't require a lot of external knowledge. They were more based on basic reasoning skills. And the reason is, we know for a fact that when you train a model on code, the reasoning performance improves quite dramatically. So far from being a model that doesn't know how to do anything except code. But of course, it lacks several skills compared to much larger models.

• 1:16:11Speaker 00

  And we're very much aware of that. We never claimed the opposite. It was more like, out of curiosity, we tried to test, are code models completely clueless about anything else? And the truth is, no, they can do some basic tasks. And that, you know, there are, I think I'm just trying some like tieral mind prompts that, you know, were became common with GPT-3. And it was doing a pretty decent job also down. So scaling perhaps is one of the ingredients required to do much better at world knowledge. And I would say also retrieval augmented generations seems to be making the difference there. Because no matter what is the information cut off of a model, you would always be lagging behind compared to the present.

• 1:16:56Speaker 00

  So it's important to have a way to inject information in the prompts to make sure that the model knows as much as possible. But yeah, I think scaling seems to be the main ingredient to make that happen today and to have new skills merging. And that model size, we know for a fact that a lot of skills are going to be lacking. But again, that's the beauty of specializing a model. You know the trade-offs you're making and you know what you're getting in exchange. That's We need it also to show that not necessarily all the effort should be in the mega models. We're glad that people do that because we use them, but there is also space for smaller ones optimized to specific use case.

• 1:17:36Nathan Labenz?

  And you will presumably have like a whole suite over time of, you know, various sizes for.

• 1:17:42Speaker 00

  They're coming. Yeah, we're killing them. Yeah.

• 1:17:45Nathan Labenz?

  Watch this space. So just a couple of final topics that I wanted to touch on. One is kind of returning to this notion of safety and now I'm approaching this in a more kind of mundane way. You know, safety for the user, safety for the platform, safety for, you know, other users. I know that you guys are making progress. I'm, you know, sometimes amateurs, sometimes, you know, official red team enthusiast. And so I've tried asking just about every code gen AI to write me a denial of service script. Results vary, even on Replit, results have varied. In May, when I did that, it just did it.

• 1:18:27Nathan Labenz?

  You know, no hesitation wrote me a denial of service script and I was like, hmm, this doesn't seem great. I tried that again this week and it no longer does it. It now refuses to write a denial of service. I haven't gone, you know, all the way into can I jailbreak that or not? But I'd love to hear kind of how you guys are working on that dimension because clearly you are. Do you have a, you know, a standard process that is specifically dedicated to, you know, avoiding, you know, harmful code? Is there a sort of standard battery of tests that you measure these sorts of things against? I'm sure you have standard tests for performance, but like, is there a kind of safety specialist, you know, wing of that?

• 1:19:13Nathan Labenz?

  Do you have a red team process? What's the, what does that aspect of the development look like?

• 1:19:20Speaker 00

  Yeah, we still don't have a rep team process by means of how small is our team, unfortunately. So like we, we need to make some, some choices regarding that. That being said, we, we rely mostly on, on third-party providers to, to make this happen. Because I, I believe, you know, that they're gonna have more cycles and more resources to do a better job than we, we currently doing on our own. So that's the reason why you see progress being made. And that's the same kind of progress you see being made today between G P T three point five March versus the June release. And you know, you have been seeing a lot of rumors on Twitter, I'm sure, about how the models have been further robotomized. And then OpenAI replying that this is not the case.

• 1:20:03Speaker 00

  And I think there is this interesting distinction between the skills of a model and the behavior of a model. So the underlying skills and capabilities are still the same because the base model is exactly the same. What changes is how the model behaves given a prompt of the user. And while back in March, for example, they were a bit more casual replying to you and giving you like a DDoS script. Now in June, the behaviors are different because the R L H F or the, you know, in the case of Anthropic, the, you know, the responsible AI project they use has been evolving over time. And I think every company is doing that. It's hard to come up with the correct solution to anything.

• 1:20:47Speaker 00

  I don't think anyone sees R L H F as the silver bullet that is gonna solve the issue as a whole. But you know, at least you can see progress being made. More interesting now, Llama two has been released and it's heavily R L H F, as you might have seen yourself, you know, from the demos online. I believe a lot will be also discovered there. As in, there will be behaviors that are aligned with how we expect the models to, to behave. And then there will be behaviors that are completely disaligned and people will find ways to jailbreak it. And people will find ways to prompt it in such a way that you will, you know, reply horrible things. That's the nature of the beast. And it's, it's still hard to tame in a perfect way.

• 1:21:31Speaker 00

  But I think, you know, we're all working towards making that better and safer.

• 1:21:37Nathan Labenz?

  So first of all, when you want to start that red team part of the company, give me a call.

• 1:21:43Speaker 00

  We'll do it. Perfect.

• 1:21:45Nathan Labenz?

  If I understand correctly, like you had a custom model in May and you have another seemingly updated custom model now? Or are you, have you added a sort of intercept layer? Because what I was assuming had happened was you'd updated the model, added some more refusal training, and now it's refusing. But maybe it's different than that.

• 1:22:04Speaker 00

  So our custom model right now is mostly used on code completion. I believe what you were using was because Writer Chat, which is using third-party models right now and some additional prompting and retrieval that we do on our hands. So we try to make prompting on our sign more robust. But there are also underlying behaviors of the models that we use that are changing. And you know, we work with Google, we work with OpenAI. So like depending on which side of the API testing you're on and you know, the kind of changes that we allow that week, you might see as well different behaviors. So that is definitely not going to be stable in the near future.

• 1:22:46Speaker 00

  Because we, we don't like to rest, you know, we will, we always try to find ways to make it better. But yeah, it's a fundamentally hard problem. So like I, I asked our users to be patient about the progress there because it's something that no one has cracked in the community yet. So likely Rapid is not going to be the first one to crack that specific problem.

• 1:23:08Nathan Labenz?

  For all the complaining that goes on about the models refusing to do stuff and being kind of chiding and annoying. I think that's actually a huge advantage for the OpenAI's and Anthropics of the world that specialize in it. Because relative to like having to manage that myself, if I'm going to go use Llama or whatever and you know, fine tune it to my ends. I want to kind of ride in their wake, you know, of all the work that they're doing. And it's really interesting that you guys are also doing that, right? I mean, probably a lot of people should update their thinking on how excited they should be to go adopt the latest open source model, based on the fact that

• 1:23:53Nathan Labenz?

  Replit is still using commercial models in their, you know, in their production product.

• 1:23:59Speaker 00

  Yeah, I think there is space for like similar to before we were saying, there should be space for incumbents and startups. I believe there is space also for open source model and models exclusively offered by companies. And, you know, each one of them would find their best application area. I don't believe that we should all be using the big buzz model GPT-4 everywhere. Not only because it's so traditionally expensive today, but also because it doesn't have, it doesn't solve every single use case out there in the best possible way. So that's, that's why I'm excited to see progress being made both by open source and by OpenAI and by Google and so forth.

• 1:24:36Nathan Labenz?

  You mentioned the economics a couple of times.

• 1:24:39Nathan Labenz?

  Can you give a rough picture of kind of what users cost you? Or how you think about, you know, what users cost you? I've priced this out a bunch of different ways in, you know, even in looking at like, what is the wattage of an A100 and an H100? And, you know, what does that cost, you know, given my kilowatt hour price and that kind of stuff? But I think everybody kind of ends up having a little bit of a different cost profile that seems to relate mostly to the workload and kind of how, you know, a lot of idiosyncrasies of the workload. At the scale you're operating at, I imagine you have a pretty good sense of like what the workload is going to look like hour by hour, one day to next. Maybe that's optimistic of me.

• 1:25:23Nathan Labenz?

  But how do you guys think about, and you can share, you know, actual numbers if you, if you want to. But how do you think about kind of modeling the cost for a user in this going from just, you know, not just containers, but going from containers to now adding on models as well?

• 1:25:38Speaker 00

  Let's put it this way, the instead of AI report that you saw, you know, published in our blog like a few days ago, that kind of growth.

• 1:25:49Speaker 00

  We're also experiencing that in our AI features. We have more users that want to use API. They, the moment they find it useful, they use it way more extensively than we expected. And then literally goes rather becomes their favorite pair programmer, helping them to be as productive as possible. Uh, that comes with a price because of course, especially when you use third party models, you're not being charged by how long you keep a server up and running, you're charged by traffic. Uh, so I would say if I gave you a detailed answer today, I might have to deprecate it in two weeks. Because that's, that's a constantly moving target. So we, we try to adapt to that.

• 1:26:30Speaker 00

  Uh, we try not to, to be like financially responsible, but not to obsess too much about costs. Because the, the landscape is changing constantly and you know, you're seeing your stuff. OpenAI is lashing the, the API costs uh by like two orders of magnitude in the last, you know, eighteen months. Uh, so I expect those economies of scale to improve over time. For now, it's more let's learn to be AI products useful to our rapid users. And then we try not to bankrupt and we know we, we have good margins yet. And then the moment they become even more widespread than they are right now. And it makes sense to have our own models rolled out everywhere for economies of scale.

• 1:27:16Speaker 00

  We're gonna work on that. But uh, it's, it's interesting to be in a rocket ride that goes so fast, but it comes with some additional headaches of sizing the, the load and the usage correctly.

• 1:27:28Nathan Labenz?

  Yeah. So does that kind of cash out? I think this is where most people end up being. If somebody's paying, then it's all good. You don't really have to worry about it. You're gonna make money. You might, you'll make more on some and less on others, but.

• 1:27:40Speaker 00

  I'm not, I'm not sure if I agree with that. Like it, it depends on what you're exposing. If you give them, uh, say GPT four and they are power users, you can quickly start to lose mine on that specific user, uh, just after a few days. And that's, that's the same reason why like even OpenAI, when you're a chat GPT pro subscriber, they limit the amount of GPT four messages you can send. I think they just bumped it up double this morning, if I remember correctly. But you know, we are still in this realm of scarcity and the scarcity applies to any user even outside of OpenAI. So it's. If not that we will to come up with a proper pricing when there is powerful AI in the world.

• 1:28:23Nathan Labenz?

  Yeah, interesting. So you can lose money on a small percentage of paying customers. Presumably, the average is still healthy. And then the real challenge that you kind of alluded to earlier is, how in the hell might we extend AI features to, you know, 20 million plus non-paying customers?

• 1:28:40Speaker 00

  Correct. So we're gonna push that problem gradually. I think we might be starting from code completion, which is easier. We own the models and we learn how to deploy very efficiently. So I see a near-term future where we can make that happen. Now putting something at the level of GPT-4 Gemini in front of users, every users for free, that that's gonna take a while, I'm quite sure. No matter how fast is the progress on the like inference frameworks, and even if NVIDIA lashes the cost of H100s by 90%, who knows if this guy ever gonna happen, it's gonna take a while until we're there. But models

• 1:29:24Speaker 00

  might become much smaller and cheaper to serve if someone comes with the next amazing architecture. So who knows?

• 1:29:31Nathan Labenz?

  Yeah. Do you think of pushing inference to the edge in the near term? Like, could you get your code complete to, I don't know, if you can get through the browser all the way to, you know, the the Apple silicon, for example, presumably in a mobile app you could. But is that a is that a direction you're thinking of pushing?

• 1:29:53Speaker 00

  We've been brainstorming exactly about that. Maybe not in the way you described it, but we are thinking of ways of either doing that on the edge or doing that in the context of a repo. So with the with the limited amount of resources assigned to every container, can we also run a smaller model that gives you like a basic code completion? So these are all directions we're exploring, especially because you might have seen that there are quite a few local plugins with our rapid code V1.3B model, where people are doing a local copilot, basically. They take a VS code extension, they take G G M L to serve the model on their MacBooks, and they get pretty amazing speed of completion, like

• 1:30:37Speaker 00

  on par with what we have on on the website. And they run their own version, their private version of Copilot. So. We know it's already happening with powerful laptop. So imagine that we should be able to make that happen in the in the near future with even more less powerful hardware.

• 1:30:56Nathan Labenz?

  On the near term roadmap for Replit, anything you want to kind of tease that you think people will be excited about? You mentioned more agentic assistance, for example?

• 1:31:05Speaker 00

  Yes.

• 1:31:07Speaker 00

  Gozrider is going to quite a lot of work under the hood. We are very excited. We started a brand new internal effort to make it not only more powerful, but also way more integrated with the idea. So you will literally start to see Gozrider everywhere across Replit, helping you on several different tasks. So that's the first teaser that I would love to give. And the second one, as I as I already teased before, we aren't done with the first L L M that we release. We we plan to be on a semi rigorous schedule of releasing new open source models, more powerful ones.

• 1:31:51Speaker 00

  We we got so much benefit out of releasing that and so much community gathering around us and also giving us back help. And like instructing data, as for example, we have been seeing quite a bit of them being contributed. That we want to keep working in that way for for a while longer. So these are the the two main teasers I want to share in this podcast.

• 1:32:16Nathan Labenz?

  Zooming out to this concept of live players that I've mentioned a couple of times. I'm Jad, CEO of Replit, is notorious in my mind. I don't know how many people pay attention to this stuff like I do, probably not many. But he's tweeted a couple of times, something along the lines of, Replit is the perfect substrate for A G I. I kind of have to squint at that to try to figure out what it might mean. But I do, you know, if I think about like, what would like a self propagating A I system need to? You know, what what kind of in what kind of environment would a self propagating A I system thrive? I think about something like Replit as being, you know, a pretty natural growth medium, perhaps.

• 1:33:00Nathan Labenz?

  Is that something that you've kind of talked to him and said, hey, maybe let's not tweet about that anymore? Or do you have a sense for like, is there a vision there for what it would mean for Rapid to actually be the substrate for AGI?

• 1:33:12Speaker 00

  I mean, Anjadeep and I definitely chat about this pretty much every time we meet and we have a chance. So I think it's more about talking about the future of the company and where the field is going, rather than about what we should be doing on the short term. That being said, I don't know exactly, first of all, what I didn't mind when it is because Anjadeep is a very prolific user.

• 1:33:39Nathan Labenz?

  Yeah, you can't track every tweet, that's for sure.

• 1:33:41Speaker 00

  Yeah, yeah, I get his notifications, so I try to read everything, but I can't pick his brain. That being said, I agree with the specific tweets and I will give you my interpretation. The reason why, you know, I would say a lot of the community today believes that LLMs are not the only way towards AGI is that no matter how good is the award model that they create internally, it's still very limited. There are sometimes it lacks common sense, it lacks like an understanding, like it doesn't do well spatial reasoning and many other shortcomings. So there are a lot of research labs and a lot of visitors working on that on a full time. And I'm sure we're gonna be seeing a lot of progress.

• 1:34:24Speaker 00

  Now, that being said, Rapid can become a substrate because if you think about yourself as a developer, you don't write everything from scratch. You use resources that are available out there, you use cloud computes, you use services, you use APIs, you use tools. The way I see an AGI is something that behaves exactly the way we do as developers. He knows what to use, he knows what to orchestrate. That's why I explained in the Rapid A manifesto, when I described the AI, like an AI that not only is capable of building things on the ground up, but is rather also very skilled at picking and choosing what to use. So Rapid becomes a substrate even in the future when we have an AGI because

• 1:35:09Speaker 00

  Regardless, it doesn't make sense to generate anything from scratch. It makes sense to know what to use. So that's the reason why I believe approaches like AutoGPT or BabyAGI are not have have not, you know, materialized AGI yet. I I don't wanna comment on how powerful or limited they are. I think I'm always glad when people do experiments because we learn out of them. But like, there seems to be a clear bottom like there because GPT four can't materialize intelligence out of nowhere, you know, it can't materialize tools and software and cloud compute. Uh, so Rapid becomes that, I believe, you know, once you have an AGI, it understands, oh, I can invoke this API from OpenAI. I can call this tool that is being deployed on Rapid. I can get CPU power there.

• 1:35:54Speaker 00

  I put everything together and I accomplish a certain task. So that's that's the way I interpret it to it. And I I do believe that I'm judds align on on this, but next time we have a chat, I will let you know.

• 1:36:05Nathan Labenz?

  As you survey this scene, I think obviously you're somebody with Google and Replit on your resume now who has demonstrated good taste in AI companies. Who else do you see as super influential, possibly? Yeah, and you know, in a position to really shape the future. I'm kind of struck by the fact that I have this list of, you know, ten, twelve companies that I'm like, these companies seem to be the ones that actually might, you know, come up with something that really changes the world. And outside of that, I don't see too many, but you know, is there any that you would add to my list?

• 1:36:46Nathan Labenz?

  Anybody you think that I should be watching that I'm not, you know, maybe not currently watching?

• 1:36:51Speaker 00

  I don't know if perplexity AI in that list, um, if you don't recommend to add it. I I love what they're doing a lot. They seem to be like an early stage rapid in terms of uh, velocity of delivering new features. Apart from the fact that, of course, you know, the team is amazing and have a lot of experience, but they're attacking such a ambitious. Problem in a field that of course as a large player that has been dominating for like 20 plus years. So but I admire their boldness in going in that direction. I think they're executing very well. And I'm very curious to see where they're gonna be going. So that's a that's a company that I keep on my radar. And sometimes I chat with Arwin, their CEO, to get some

• 1:37:35Speaker 00

  inspiration. Yeah, that's the first one that came to my mind. I mean, I talked with a lot of founders, I don't wanna, I might, you know, mention another 10 of them that I also like, you know, but I I feel some like connection, the perplexity story because I see them growing like at the same speed that I did. And that's inspiring. This has been amazing, Nathan. Great questions, you know, thanks for for making this happen. I think we've been talking for one hour and four today, didn't even realize. But yeah, it was a pleasure to be a guest in your podcast.

• 1:38:03Nathan Labenz?

  Well, thank you very much for making it happen. We appreciate all the time. I'm a huge fan. Michele Catasta, thank you for being part of the cognitive revolution.