How Replit’s AI Tools are Changing Software Development with Co-founder and CEO Amjad Masad

Replit’s develop-to-deploy platform and new AI tool, Ghostwriter, are breaking down the barriers to entry for beginner programmers. Replit’s CEO, co-founder, and head engineer Amjad Masad joins hosts Sarah Guo and Elad Gil to discuss how AI can change software engineering, the infrastructure we still need, open source foundation models, and what to expect from AI agents. Before co-founding Replit, Amjad Masad worked at Facebook as a software engineer, where he worked on infrastructure tooling. He was a founding engineer at CodeAcademy. Throughout his career, Masad has been an advocate for open-source software. Show Links: Amjad Masad - CEO & Co-founder of Replit | LinkedIn Replit Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @amasad Show Links: Amjad Masad - CEO & Co-founder of Replit | LinkedIn Replit Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @amasad Show Notes: 0:03:55 - Impact of AI on Code Generation 0:11:09 - Breaking Down Barriers to Entry in Development with Replit 0:14:35 - The Impact of Open Source Models, Meta/Llama 0:20:32 - Bounties, Agents who Make Money 0:24:26 - The Missing Data Spec-to-Code 0:32:29 - Building the Future of AI, Money as a Programmable Primitive

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Sarah Guo?

  This week on No Priors, we explore Replit and its new AI tool Ghostwriter. Replit now has more than 22 million users and recently started a partnership with Google. With Replit, anyone from beginners to experts can quickly build and release fully functional live apps in seconds now with the help of AI. I'm Jad Masad, CEO, co-founder and head engineer at Replit, joins a lot of me this week on No Priors. Welcome.

• 0:29Speaker 01

  Thank you.

• 0:29Sarah Guo?

  I'm Jad. Just to start, you were working as an engineer at Facebook. What made you decide to start Replit?

• 0:35Speaker 01

  My journey with Replit actually starts way back in college. I was really interested in programming languages and I found it really difficult to just like keep setting up the development environment. I didn't have a laptop at the time. And so I was like, you know, why can't I open a browser window and start coding? Turns out nobody has done that. And I was, I was kind of naive and, and thought I could probably try to do it. I got a prototype up pretty quickly, but my friends loved it in college. Everyone started using it. It was just like a simple webpage with a text box and, and a button that says run, which is still kind of in a lot of ways what Replit is.

• 1:13Speaker 01

  But then that started me in a like multi-year journey to actually build the first kind of in-browser sandbox. And at the time, we had a bit of a breakthrough. We were the first to compile Python and Ruby and other languages to JavaScript. I open sourced that project. That got me a job at Code Academy. I was founding engineer there. And after that, I went to Facebook and I was one of the founding engineers and React Native. We like made React and JAST, Babel and a bunch of tools that JavaScript developers use today. Our goal was to make mobile development as fun and easy and fast as web development, which I think we succeeded at.

• 1:57Speaker 01

  You know, towards the end of my time at Facebook, I just wanted to see what to do next. And kind of looked around and, you know, whatever happened to this online coding space? And there was a bunch of online editors, but none of them really felt like web native. Like you can just share URL and someone can hop in and code with you. Kind of like how Figma or Google Docs work. And we decided to start in 2016.

• 2:24Sarah Guo?

  You have made a huge bet on AI as a company. Can you just talk about Ghost Rider and how it came about and the investment in this area?

• 2:34Speaker 01

  Yeah, throughout my career, working on code that handles code, right? Whether it's at Codecademy for teaching programming, whether it's my own project, whether it's React and building the runtime around React Native. I always felt like our tools that were handling code, whether it's compiling it, parsing it, modifying it, all that stuff. We're very rigid and very laborious. In a lot of ways, you're building a classical intelligence system, very algorithmic, and a lot of heuristics and all of that. And I always thought that you can probably apply machine learning to it. And I started reading around whether anyone had done it.

• 3:15Speaker 01

  There was this seminal paper in 2012 called "On the Naturalness of Software", and basically a bunch of researchers tried to apply NLP to code. And what they found is that actually code can be modeled like any language. That's why they call it naturalness, because hey, code is repetitive. Like, language, you can do things like Ingram. Basic Ingram model can actually start to generate code that is compilable. That had a huge impact on me. And every year or two while starting Replit, I would go look at the state of the art on ML on code, and nothing ever really worked all that well up until GPT-2. And you can take GPT-2 and fine tune it and try to make it write code, and it was kind of okay.

• 4:00Speaker 01

  But obviously, GPT-3 was like, okay, it's here. And that's when we started building what became Ghost Rider. Initially, it was a bunch of tools sprinkled all over the Replit IDE, explain code, generate code here. And then we wrote our autocomplete product, which is like Copilot. As you're coding, it gives you suggestions. And then we added the chat product, and that all became Ghost Rider, our AI speed.

• 4:29Sarah Guo?

  It's amazing that this idea of type ahead code autocomplete with local context has become as widely adopted as it has. Like your tool and Copilot are sort of a handful of tools that people like very much accept have changed workflow. Maybe if we just project out a little bit, where do you think that the like next big leaps in productivity in software development come from in terms of AI? What else is gonna happen?

• 4:55Speaker 01

  It's an amazing tool, but I think it's still very primitive. And we haven't fully explored the full capabilities of even GPT, but also transformer models applied to code in general. There's a lot to do. I think there's a lot to build on the sort of the layer just around the models. So they're basically how to give models better context, how to give them tools, the ability for the model to actually be able to go out and read a file, install a package, evaluate code, the ability for these models to be more agentic and be able to kind of write entire features by themselves. And so that's all the work just around the model itself with the model in the center.

• 5:38Speaker 01

  And then there's a lot of work on the models themselves that haven't been, you know, fully explored. The way we train models today is we just give it a large corpus of of of code. But you can imagine different ways of training models. And we've played around with different techniques at at Replid. There are ways to make the models, give models more intuition about how code execution happens as opposed to just static code. Because you're training them on static code, they understand the structure and syntax of code, but they don't really understand the semantics of code. So you can imagine a way to train models where you're not only feeding them code, but you're also feeding them the results of the evaluation of that code.

• 6:20Speaker 01

  You can imagine ways of training models not just to be good at code, but also be good at the editor, the debugger, everything that comes around code itself. And so I feel like we're in the very early beginnings of it. I think for professional programmers, perhaps it's like a marginal improvement. And like it, some people claim it's a huge margin based on our research of how people are use our products. We think it's 30 to 50% more productivity on coding itself. Obviously, there's a lot of things other than coding, such as debugging, deployment. Meetings, being able to translate specs into into code and everything that we do just around

• 7:02Speaker 01

  the act of coding itself, AI hasn't touched yet, and I think it will touch every part of the software development lifecycle. I think actually the people that are getting the most amount of value today are tend to be the beginners, tend to be the people who don't have the skills typically to like start a startup. And we're seeing a lot of those on Replit just today. We've profiled a an entrepreneur that has a quarter million ARR startup in a few months. And they haven't, you know, earlier this year, they didn't know how to code. And I think this is where most of the impact is right now. But I think if you project forward, those people will also get a lot more tools and get to get more productive, gonna hit superhuman productivity.

• 7:45Speaker 01

  But the professional programmers, this is where actually you start to have that two X, five X, ten X multiple in productivity.

• 7:51Speaker 02

  What what do you think is a timeline on that? Do you think that's a year away, three years away, five years away? I'm just sort of curious about when we both see really massive productivity gains. And then also when we move towards this more agentic world that you mentioned where, you know, you have AI go out and do things on your behalf in terms of generating a set of features or grabbing a packet from a package from somewhere or doing other things for you.

• 8:13Speaker 01

  My feeling is this is within reach, even in the current capabilities. Like if you hold constant the current capabilities, this is when reach within reach in the order of, you know, six to eighteen months. We're in the demo era of GPT models. Right now, we're doing the hard work of building infrastructure on Replit. Like we are building this service that sits inside the container that embodies the AI in the development environment such that it can do things like go to the internet, like install a package, like read a file, write a file, start another project, get cloned, do all of that stuff. That's a lot of work and that's not just one model, it's probably multiple models. And so within the current capabilities, there's a lot more to do on the infrastructure side.

• 8:58Speaker 01

  So I think we're within months of actually having.

• 9:03Speaker 01

  Really cool, basic agentic experiences where you can give a high-level task to a model or to an AI, such as build a login page and for it to do a decent enough job to give you like a starting point for that. Now, if you try to project like the actual advancement in AI itself, increased context size, increased performance, better prices, potentially like even new net new advancements or new ways of training code models, i- it's it becomes really hard to predict. Like, I think software could fundamentally change in the next four, three to five years where it looks totally different to someone sitting, sitting today and looking at it into the future.

• 9:46Speaker 01

  Maybe it looks something like programmers tend to be more like engineering managers, like even, even ICs, you're just like working with employees. Uh, and those employees are mainly AI and you're, most of the time you're reviewing code and giving instructions and you're not actually like spending 50% of your time typing. I- it's hard to know, but I'm confident to say within the current capabilities, there's still a lot to do over the next six to 18 months.

• 10:12Speaker 02

  And then I guess one other thing that you have as a real advantage for your product is just you have such user liquidity, right? So you have, I think it's on the order of 22 million users or something in that, that range. And that's obviously can create an enormous feedback loop for training your own models, for reinforcement learning, et cetera. There's a paper that recently came out from Google comparing a very specific type of reinforcement learning through human feedback versus AI feedback and it roughly netted out, right? You could start to use AI for certain tasks to effectively train itself. Like, how do you think about that in the context of code and sort of where you're heading?

• 10:45Speaker 01

  I think it's easy to overstate data modes. You know, you need to be measured about that. I think the real advantage of Replit is actually just the platform, just the end-to-end platform. Cause I think, you know, you can be, you can try to build these code models, but without having the platform to be able to apply it on, to be able to get feedback from, i- it gets very difficult. But also, even if you just have a code editor or editor extension, that doesn't give you all the feedback that you would want, because you would want feedback from deployments, you would want feedback from execution of the, of the code and not just user feedback. I think user feedback, you're right, for our LHF-type things.

• 11:31Speaker 01

  I don't think that's the real advantage. The real advantage is the end-to-end journey from the first line of code to the deployment, to getting crashes in production, to making edits, to all of that feedback cycle, which typically in most cases is divided between GitHub and VS Code and AWS and all of that. And the real magic of Repl is putting all of that together. And this is, I think, where we're going to get the kind of richness of training data that allows you to train over a large action space. And that's, that's really exciting.

• 12:06Sarah Guo?

  Can you talk about the decision to train models versus use existing APIs? Like, was it just a latency thing? How did that happen?

• 12:14Speaker 01

  So when Microsoft did it, like, they took GPT-3, they distilled it down, and they hosted it on their own infrastructure. They did a bunch of caching, a lot of things like that. If you're someone who's just using the OpenAI API, you couldn't do any of that, and it was very expensive. It was very slow.

• 12:34Speaker 01

  And even now, you know, there's no completion models. Now, all the models are chat models. You know, they're not going to be releasing any completion model going forward. And so if you wanted to build a completion-based product, it's actually fairly difficult to do it using commercial APIs. And we felt like we can, we can make a model that's both cheap and fast and good enough for that autocomplete use case. And we found that the three billion parameter size is kind of the nice sweet spot where you get enough raw IQ from it. Because like one billion felt kind of too dumb in a lot of ways. And it was going to be cheap enough and fast enough to host and be able to do a fast inference from.

• 13:16Speaker 01

  And it was a time after, you know, the Chinchilla paper first came out and Llama had been announced, and the idea of just training them longer had just been in the air. And, and we're like, okay, what if we, we apply all these open source tokens on a smaller model? And then we applied some more tokens from Replits data that gave us a 50% improvement over the model that we open sourced. So we open sourced the base model because we think that's sort of the ethical thing to do because we trained on open source data and we want to give it back to the world. And it was state of the art model was better than most open source code models at the time, even things 20x its size. But but the need for

• 13:58Speaker 01

  it came from a product need, like we needed to build that product. I think a lot of founders now see it as a point of pride or status to train a model. And the moment we put out the three billion parameter model, like three or four copycats came out of after that. And I I'm not a fan of what's happening in in the open source world where everyone's competing over these silly benchmarks. I think it's important to start from a product, start from a user and customer need, and then go from there. Right now, if we could do what what we can do with Llama, we'll just use Llama. If we have to train a model, we'll train a model. And that's how we always approach building this company is start from

• 14:39Speaker 01

  the customer and the use case, and then solve any technical challenge that you you get along the way.

• 14:46Speaker 02

  One of the interesting debates I I've been hearing about Llama recently is how long will Facebook continue to sponsor them as open source models? And, you know, so far, it seems like Llama 2, for example, is the one that that a lot of people are viewing as the most performant, with Llama 3 perhaps being a big step forward in terms of catching up relative to GPT-4 and sort of other closed source models. What do you think happens if if Facebook stops open sourcing? Do you think somebody else steps into the void? Do you think everybody's kind of on their own? Like, I'm a little bit curious how you think about the open source world relative to models, given how important Llama has become so rapidly?

• 15:23Speaker 01

  So when I came to the realization early 22, that we can't build the product that we want to build using commercial APIs, I was kind of depressed for a moment. It was like, oh, you know, we didn't have, you know, a lot of funding or whatever. It's like, okay, you know, Microsoft's going to be the only one's going to be able to build these kind of products. And I got a meeting with Zuck, actually, and I pitched Zuck. On this idea, and I remember around the time he did the Open Compute Project. So what Open Compute was, like, Google and AWS are going to keep all their data center secrets because it's a, it's a company advantage. But Facebook did not sell compute, did not sell cloud. And there's a lot of other companies that did not sell cloud.

• 16:07Speaker 01

  Cloud was a complement for them, was not a competitive advantage. And so they pooled their resources together and they worked on this Open Compute project to be able to optimize computers for data centers. And it was a huge success. And what I told Zuck at the time was like, hey, why didn't you do that for LLMs? Obviously, open source AI is going to be great for you because you can build better products. It's a complement. And so it's sort of the traditional, like, the classical, like, commodizer, complement business strategy. And at the time, he just kind of nodded his head and I don't know if it was a novel idea, maybe he was already thinking about it. But that's basically what he's talking about.

• 16:48Speaker 01

  I, I think in one of, um, one of his interviews, he talked about Open Compute as an analogy for what they're doing with LLMs. So I think it makes a lot of sense for them to do that. What I'm actually surprised by is how little industry participation they've gotten. And I think the difference between LLMs and Open Compute is that the AI safety angle makes, makes it a little toxic for a lot of companies to touch. Like, they wouldn't want to be associated with something that spews something that, that is toxic or they don't want to touch. So that's another wrinkle that I didn't appreciate. And it takes a certain amount of guts to release an open source language model, especially

• 17:30Speaker 01

  with political heat that Meta is getting from that. Now, what happens if they stop? Uh, yeah, I think, I think it would be bad. I think it would be bad for the ecosystem. Will there be another player that will emerge? And the, the problem a lot is that, as you know, there isn't a lot of guts in the industry. That's our problem. It's like, and Zuck has balls, right?

• 17:50Speaker 02

  Yeah, no, I think it's been super impressive what Meta has done with Llama, and I think it's really helped. Push forward the entire industry in an amazing way. And so really kudos to the company for doing it. I think beyond guts, which I agree with, there's also the cash side of it. I actually think the money exists in the ecosystem to pay for open source and to help train models. I think the talent base is much harder. And the magic of what I think Facebook did with Llama is they had an amazing talent base and then the capital to train something and then the guts to release it. And if you look at every other open source software wave, almost every major one had a big corporate sponsor, right? Linux was sponsored by IBM to the tune of like a billion dollars a year for some period of time. WebKit was obviously sponsored by Google and Apple.

• 18:33Speaker 02

  Like, there's lots of the really big things that were important in terms of fundamental pieces were sponsored by a company at one point or another. And I think in the absence of Meta providing not only capital and guts, but also talent, there may end up being a gap. But that's one of the reasons I was asking the question is they have that really amazing intersection of guts, money and people.

• 18:55Speaker 01

  Right.

• 18:56Speaker 02

  And I, you know, it's unclear to me what the replacement there is.

• 18:59Speaker 01

  Yeah, it's also unclear, like you said, which other companies would be able to participate in that. And I'm sure they tried to do outreach and they tried to work with other companies. I think on a long enough timescale, obviously, there's, there's going to be open source models that are as good as commercial models. But the current landscape makes it hard to, to kind of imagine how it would play out. And also, there's this whole question of like, if the architecture gets more and more complex, it actually makes it harder and harder to open source. Right? Like, there are tools and there are services. Tools especially are easy to open source because you can, it's a binary, you can run.

• 19:44Speaker 01

  And basically, that's what we have today. You know, you can just run, it's like a binary blob, right? But if there's like mixture of experts and there are components, stateful components for running this stuff, then it becomes exceedingly harder to create an open source alternative.

• 19:58Sarah Guo?

  I'm curious if we can talk about bounties for a second?

• 20:02Speaker 01

  Sure.

• 20:02Sarah Guo?

  Because you're somebody who like clearly believes in agentic development. And then you stuck a marketplace in your product that allows developers to collect dollars for code that they've produced. Do you think we end up doing that with development agents or how does that play out?

• 20:17Speaker 01

  So the idea behind bounties is that in roughly, we have all this talented people across the world.

• 20:25Speaker 01

  That are learning how to code and that are actually like becoming great programmers that kind of no one knows about. And I felt like it was almost our duty to, you know, make it known that there's so much talent around the world that you can be kind of leverage. And a lot of these people don't have any resumes, don't have a lot of skills to be able to get, go get jobs themselves. So part of it was this community building activity. Can we get all these people money? And like we have people in Africa, India, Thailand, Serbia, all sorts of countries now that are like making money on bounties. The other thing was my view at the time, or my thesis that I wanted to test was

• 21:10Speaker 01

  it's gonna be a lot easier and it's gonna be a lot cheaper to be able to get basic software tasks done. Because AI, like I said, will have in the short term, the most impact on beginners. So beginners are actually getting a two, three, 10x multiplication on their productivity. You learn a little bit of code and now you can actually build a product that makes 100K ARR. And so the idea was that, okay, how do we leverage this, this opportunity? How do we monetize it? And so, okay, human plus AI, the human who knows a little bit of code plus AI could actually be a really great developer. And so, yeah, turns out I was right. You can go to Replit and get a, get a startup, prototype for a startup or an MVP for a startup

• 21:55Speaker 01

  for as little as like $50 or $100 in some cases. My, my hope as well for Replit in, in general is that there is some kind of currency or way to transact built into the, the platform. And that was upstream of the idea of bounties. The bounties was, was kind of the first step into that direction. My view is that there's something broken about open source where the currency in open source is sort of stars, right? Like how many stars you get? And it's sort of a fake currency and it creates all sorts of weird side effects.

• 22:34Speaker 01

  We talked about how open source tend to be, the most impactful open source projects tend to be corporate sponsored, right? But that doesn't really have to be the case. I think the software economy is missing some native currency for exchange of value.

• 22:51Speaker 01

  And open source is good and great. We want more of it. But also, we want it to be sustainable and we wanted the people to be rewarded who are contributing to it. And then there are all sorts of things you can do with a platform that has concept of money and transactions built into that. So if your development environment and your deployment environment has a currency built into it, what kind of collaboration and ways of doing things doesn't unlock? One idea is that you sort of publish a module or publish a component or publish a service to the network. And I, and you kind of price things per function.

• 23:34Speaker 01

  And I can sort of call these functions and be able to have a wallet in my application, be able to kind of spend money on the different functions and different databases, different services that I want to to compose, to create an application that someone can consume. And AI was, was an interesting sort of addition, additional component to this because I think agents in the future will also be able to, to use money to, to do transactions, do things in the world, right? So right now, our AIs, again, we're using them as these co-pilots, right? They're basically something that's helping you do some kind of action or task.

• 24:14Speaker 01

  They're predicting the next step. They're literally doing the, the, you know, auto-aggressive action. But what, you know, what if we get to this place where we actually have these agents that can act semi-autonomously? What would you want them to do? Obviously, we want them to interact with a wider, wider internet so that they're coding, they're evaluating code. But you also would want them to be able to spend money and make money. And I, in the future, I could see people creating bounty hunters that are fully automated. So can you create an AI that can go earn money for you while you sleep? And I think that that would be a really cool, cool idea. And, and finally, I think on the data side of things,

• 24:58Speaker 01

  one thing that I thought was missing from the data landscape around code is complete project spec or product spec to actual end product. And, and so we have this data now where you have a literal, like, complete end-to-end project spec and the actual code accompanied with it.

• 25:18Sarah Guo?

  Yeah, we host this college hacker group for my fund for this commit hackathon. And it is very cool seeing how sure some of the like 19 year olds are that AI is going to change development such that they don't really need to like learn to code in the same way before, right? Like, and if you ask them like, well, do you need to know architecture? And they're like, nah, like, you know, it's more prompt guidance. Like, do you like, who's going to debug? And they're like, oh, another agent, right? And these are people who, you know, 20 year old MIT kids are like designing and building their own GPUs. They're very interested in computing, but the orientation toward, I would rather orchestrate

• 25:58Sarah Guo?

  some of Amjad's like bounty hunter type developer bots is very strong. So I'm excited to see if that happens. I think there'll be some of your first users.

• 26:08Speaker 01

  I think embedded in their answer is this attitude of like, if it becomes a problem, I will learn it, right? If the AI can't do it, I will learn it. And I think that's the right attitude for hackers is that you learn just in time to be able to do that task.

• 26:24Sarah Guo?

  Yeah, absolutely. I'm not going to like move myself up the MIT computer science curriculum hierarchy. I'm going to go figure out what I want to build and then kind of work backwards and look, I can jump 20 steps forward because of Ghostwriter now. It's a very cool setup.

• 26:38Speaker 02

  One of the things that you mentioned was, you know, different aspects of design, payment, different activities, et cetera. Do you think from a payment perspective, anything needs to change in terms of financial system? Is it crypto? Is it just centralized payments? And it's just the standard old rails?

• 26:55Speaker 01

  You know, I'm very pragmatic about like technology, like if Stripe gets us there, Stripe gets us there. If it is Bitcoin, then it's Bitcoin. I think it's a combination of both. I don't view them as that separate. Actually, like one interesting historical tidbit is that Stripe, Square, and Bitcoin started the same year.

• 27:18Speaker 02

  Satoshi was behind all three, it turns out.

• 27:20Speaker 01

  Right, maybe. It's actually Jack.

• 27:23Speaker 02

  Yeah, it's Jack and Patrick working together.

• 27:28Speaker 01

  But you know, i- i- it was obviously in the air, this idea of like, the internet is missing this core feature. Money is a coordinating mechanism, right? Like, there's so many cool things you can do with money as a primitive, not only for people to be able to kind of buy and transact and value, but you know, all the things around the people in the Ethereum community that sort of invented with staking. And when I was designing bounties, I really didn't want it to be this like rigid thing where people apply and you accept and whatever. What if you could like stake your currency and say, I'm gonna do this task, and if I

• 28:10Speaker 01

  don't do it, you can take a hundred dollars from me. I'm gonna do it in five hours. And for every extra hour, you can slash my funds by X percentage or what have you. And so you have this market dynamic where people can kind of bet on how confident they are that they're gonna be able to do the task. And then the coordination happens sort of semi automatically, because like the biggest bet, you know, takes that task and so on and so forth. So money is, could be a programmable primitive. And I think that's true. And it's really hard to argue with. You know, I think there are a lot of drawbacks to the current

• 28:50Speaker 01

  financial system and all the problems that that come with it. I still believe, I think Bitcoin is gonna be long term, the TCP IP of money, but there's gonna be a lot of services that could also be Stripe and others and Square that is built on that like, on that sort of rails.

• 29:10Speaker 02

  Cool. I think we covered a ton of ground. Thanks so much for joining us today. It was a really great conversation.

• 29:15Speaker 01

  Yeah, likewise. Yeah, thank you for having me.

• 29:19Sarah Guo?

  Find us on Twitter at NoPriorsPod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.