Building toward a bright post-AGI future with Eric Steinberger from Magic.dev

Today on No Priors, Sarah Guo and Elad Gil are joined by Eric Steinberger, the co-founder and CEO of Magic.dev. His team is developing a software engineer co-pilot that will act more like a colleague than a tool. They discussed what makes Magic stand out from the crowd of AI co-pilots, the evaluation bar for a truly great AI assistant, and their predictions on what a post-AGI world could look like if the transition is managed with care. Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @EricSteinb Show Notes: (0:00) Introduction (0:45) Eric’s journey to founding Magic.dev (4:01) Long context windows for more accurate outcomes (10:53) Building a path toward AGI (15:18) Defining what is enough compute for AGI (17:34) Achieving Magic’s final UX (20:03) What makes a good AI assistant (22:09) Hiring at Magic (27:10) Impact of AGI (32:44) Eric’s north star for Magic (36:09) How Magic will interact in other tools

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Speaker 00

  So, welcome to No Priors. Today, we're talking with Eric Steinberger, the co-founder and CEO of Magic. They're developing a software engineer copilot that will act more like a colleague than a tool. And Eric has a really fascinating background between work at Meta on different types of games, running climate science, which was a nonprofit focused on the climate world, and now, of course, developing an incredibly interesting AI model and system. So, welcome to No Priors today, Eric.

• 0:30Speaker 01

  Thank you so much for having me. It's great.

• 0:32Speaker 00

  So, you have a super eclectic background.

• 0:36Speaker 00

  Could you tell us a little bit more about what you worked on in the early days, how that evolved into working on AI and sort of the path you've taken?

• 0:43Speaker 01

  Yeah, yeah, thank you. So, I guess when I was 14, I just had my midlife crisis and thought I had to do something important with my life. Spent a year trying to look at everything. It was pretty stupid. And basically, I looked at things like string theory and like all the things a 14-year-old would look at. I mean, like, okay, what can I spend my life on? And eventually, my mom bought me a book on AI. And I didn't read it, I'm sorry, but it was like the idea was sufficient. So, I was like, okay, this could do anything. And so, you should just do that, and then it does everything. Then, it seemed plausible that you'd need to do reinforcement learning. So, I didn't know how to code at the time. Then, I learned to code over a couple of years. This was sort of in high school times. And then, it seemed plausible that you'd need to do reinforcement learning,

• 1:28Speaker 01

  because otherwise, you'd not be unbound it. So, I sort of just started working on RL, played around with things for a bit.

• 1:36Speaker 01

  And eventually, I reached out to someone at DeepMind to basically do this. I was like pitching this like multi-page email. I was like, could you like do like a mini PhD thing where I'm a complete newbie, but if you can bash me, just please bash me like every two weeks and like tell me how to be a good researcher? And so, eventually, I got like reasonable and then did some actual research work and worked with, you know, a few other people, including Noam Brown, who was at Meta on developing new RL algorithms to be more sample-efficient and just generally better and faster, or whatever was the goal at the time, to solve whatever environments we were interested in at the time. So yeah, that's how I got into it. I have no background in language models when we started

• 2:17Speaker 01

  magic at all. It just seemed, I just was like totally not on my radar. I was like, oh, wait a second. Like if you take this and this and put it together, like, like maybe this works. And so, so then, then I sort of, it felt like this huge relief of uncertain, like uncertainty relief of like where AGI would come from. Cause you just put those two things together and then, and then it will work was the sort of hope. But yeah, yeah, my original background is in RL and trying to come up with algorithms that sort of, yeah, just like have better structures to be more self efficient and faster or better conversions.

• 2:49Speaker 00

  And a lot of emphasis on magic is for a twofold. On the one hand, you're doing a large scale custom model, specifically in part focused on code, and then you're also building sort of at the product suite that can really help address coding and working on the software development side. How did you decide to start magic and why, why focus on that versus other aspects of AI?

• 3:12Speaker 01

  It sort of came from a place of working backwards from AGL. If you, your end goal is to have a system that can do everything. You can reduce that to building a system that can build that system. And so that minimal system is a system that writes code and comes up with ideas and can validate those by writing code and running experiments, which is still like in the same order of complexity as the full thing. But at least we don't have to train Zora and like, you know, we don't have to think about 10 billion other use cases that everyone building general domain products has to think about. We only have to think about code. So it's a lot simpler in all aspects, except compute and slightly simpler on the aspect and slightly cheaper on the aspect of compute. I think it's not a lot cheaper. I probably overestimated how much cheaper it would get

• 3:54Speaker 01

  on the compute side at the beginning, but the other things are simpler, I think.

• 3:58Speaker 00

  And you've taken a slightly different architectural approach, right? Than what a lot of people are doing in terms of just going with a. Traditional transformer model, is there anything you could talk about there? And you know, you also had a very early emphasis on long context window. And I think you were the first model that was publicly announced at like 5 million tokens. And I think that was like a year ago or something, right?

• 4:16Speaker 01

  Yeah, that was a year ago.

• 4:18Speaker 00

  So I was just curious like how you chose a specific architecture that you decided to focus on? And how you decided to focus on context windows before most people thought it was a thing? And you know, I think you've been very sort of pioneering on a lot of these areas.

• 4:30Speaker 01

  Thank you. Yeah, it seems important for models to have the ability to learn from long histories of their own and their collaborators' actions, as well as take into account a large amount of fast-changing data. And so if you imagine having 10,000 employees or you know, everyone on earth having their own model and wanting to feed in all their data, you can now fine tune everyone's model, maybe do some Laura tricks. But in practice, context just works better. And that's like in context, learning is the magical part that came out of transformers. Like this is, this is what makes them great.

• 5:10Speaker 01

  I think of that as some sort of as an online optimizer in a sense that instead of compressing a set of data, you're trying to learn an optimizer. So the perspective we take on models is instead of, this is what one of my colleagues put it this way. I find it very fitting, so I'm stealing his quote here, is instead of bringing the data to the compute, we're bringing the compute to the data. So you have a set of stuff and our model acts on that stuff rather than having like a giant model that you know, you have to sort of work around. So the whole system is designed for this. So, so yeah, like a year ago, we announced 5 million. And by the time this is out, we may have announced a larger number.

• 5:56Speaker 01

  The main reason being that you would want to deploy these things. For very long horizon trajectories and you want them to spend a lot of time thinking and you want the model to remember all of that. And you can't really do that by fine tuning because you'd have to fine tune every like whatever many thousand tokens your context window is long.

• 6:13Sarah Guo?

  I don't know if people would think of this as a trick, Eric, but like, you know, can you give us some intuition for why, you know, quality of output would be better than a retrieval based system?

• 6:22Speaker 01

  Yeah, I mean, so you could comment this from two perspectives. I can explain it mechanistically and I can, the other cheap way is to just point that Richard Sutton's bitter lesson. Retrieval selects a subset of data for one completion. Our model sees all the data all the time. Clearly, a subset of data for the whole completion is a subset of all the data all the time. So if retrieval was optimal, our system could learn it.

• 6:46Speaker 01

  And it just turns out that it's not optimal. That's the sort of mechanistic, I guess, explanation or logical explanation as to why long context will be better. You could make arguments around the quality of long context if it weren't sufficient, if it wasn't sufficiently high quality, maybe having a short context window and pulling in some data. So you obviously have to evaluate this, like, you know, but in principle, yeah, on the assumption of Richard Sutton's bitter lesson, you would want the thing that can learn your heuristic rather than the heuristic.

• 7:13Sarah Guo?

  The, the other area of expansive exploration right now amongst researchers for better AI code generation tends to be test time search. So, you know, more compute at inference time, you know, speaking of a, speaking of a mutual friend in Noam, like, how do you think about this?

• 7:31Speaker 01

  Well, so you can think of model performance as some function of training compute times some function of inference time compute. Now, those are specific functions that are just scaling all things that you can like model, but the, the general way to think about it is some function and some function. And then you would want to estimate how much inference you're going to do and how much, what your total budget is, and then you would want to create the optimal trade off in your allocation of money. You also want to consider the distribution of outputs. There'll be users who will want to spend less money and users who will want to spend more money. And this is likely going to follow some sort of very sort of spiky distribution where there'll be like four users spending a million dollars and, you know, four billion users spending 10 cents and. A curve in between. It seems strictly beneficial to be able to provide that choice.

• 8:16Speaker 01

  So instead of training the, putting all the compute into training and having that $1 million inference performance be purely from the training compute, which is just hilariously inefficient. You can, you can allow the user to choose their, I think so. Or rather, you can just deploy multiple things. The reason I can talk about this now is because everyone like is getting, like everyone gets this. But, but basically you clearly want to be able to regulate the amount of compute you save. So now it turns out this is actually not trivial. Like, like doing this is hard, like finding like the right algorithms to do it. That being said, people have done it in RL for a decade. And so to those, you know, I don't, I don't want to name people. I don't know him obviously as one of them, but there are others at other, other labs as well.

• 8:58Speaker 01

  Like there's a set of people to whom this is like the opposite of a surprise, but it's still not trivial because it's the general domain. There is no game.

• 9:06Speaker 00

  Yeah. The analogy I've heard is sometimes when you're asked a question, you kind of pause and think about it. And that's almost like your inference time compute. So you're basically investing some sort of resources to actually consider the problem at hand versus just spot react to it.

• 9:22Speaker 01

  Correct. And there are, there are things you have to learn during training. Like if you are asked to write a piece of code and a, and you've never learned coding, you can spend, the inference time compute you have to spend is ridiculous. Like, like you're going to have to at inference time, learn programming, which, which maybe I actually think this is possible. But I, I, it is also crazy. And the, this is like, clearly this is shared among like everyone who like posts a query. So, so it's stupid not to bake this into training. Even the best mathematicians in the world for the frontier of mathematics require a long time to solve the problem. And so, so, so like, I would love to have a Terrence Tao in my computer, but I would then still need to run Terrence Tao for a year of, of human thinking time. And, and like Terrence Tao will not just token by token spit out a proof or a ream on or whatever.

• 10:08Speaker 01

  You know, and so, so I think to achieve things like that through pure training compute deployments, inference and compute work would be to drastically fail simply because it is a shortcut. Like you can also train this model for a quadrillion dollars. Maybe you can't actually, because there's no data, but say you could.

• 10:28Speaker 01

  Well, what if I just need a billion? Like, you get the idea. So I think it's like this fundamental trade off that you want to be able to bake into. The humans can do this exactly as you say. And this applies to all parts of the workflow that you're asking them to do.

• 10:44Speaker 00

  And I guess if the goal of the company is eventually to build AGI, how does that impact your choices from a design perspective relative to some of these trade offs? Or is it more, we're going to iterate till we have a system that's very good at writing code? And then it bootstraps its own next version? Or how do you think about your roadmap relative to AGI itself?

• 11:01Speaker 01

  Yeah, that's a great question. So I remember one of our first conversations, we were talking about sort of the step function relevance of safety risks. Let's say, right, where like, there's a lot of stuff people are panicking about that really doesn't matter in the short term for like the grand scheme of society. And like, some people will get pissed when me saying this, but it's just what I actually believe. I just don't think this is like, the current complaints are like, similar to what we have seen in like all other technologies. And like totally resolvable and, but, but then there comes like the evolutionary one. And I was like, okay, shit, like humans succeeded in the world because we're smarter than chimps and we're smarter than bunnies. And, uh, you know, I had, and I get bunnies do not rule the world. Uh, and, um, that would be cute probably, but also, um, it's, it probably wouldn't be as nice for us.

• 11:44Speaker 01

  And like, here we are, turning ourselves into bunnies and apes and creating this thing that, you know, we're all thinking is going to be way smarter than we are. This is insane. And, um, so the reason I think like the sort of recursive approach that you're mentioning here and anything at. And then obviously you know, this is what, you know, this is, this is what we're founded to do. Is exactly that. I think the only way to, to, to sort of reasonably approach this is to iteratively ask your model to solve

• 12:14Speaker 01

  alignment and safety at that stage, not, not, you know, surely you can also ask it to solve your product level problems. But like that, that's, that's nice, but that's not, that's not the fundamental objective. The fundamental objective is to iterate towards AGI with a safety boundary. And there's just no knob like this. There is just no knob like this in the human world. Like you, you can say like, oh, I'm going to like spend X percent of my resources on this, but that doesn't indicate an outcome. Like you, you can actually control it. But, but if it's like compute, you can, you can somewhat do it. So, so I think it's just a, the most promising path we have, both on the progress towards better models, because you can, if you just have like a hundred thousand brilliant researchers in your computer. That is better than having a hundred, especially if you can connect all their brains.

• 12:58Speaker 01

  And then if you can say like, okay, we think this is safe to do, like, let's go do it. And then you have the next thing and then you're like, okay, like, maybe we should use that thing to do some more safety research. You can bootstrap and so, so just very pragmatically, even if there was goodwill, I just don't know that we have the mechanisms to prioritize safety without automation. So, so yeah, that's the primary goal. Now, the good thing is, as we pursue this, if we're successful, if we're not successful, then I'm deeply sorry to all of our investors. And including you. If we do succeed, this will, especially if we are first or early to succeed, drop a beautifully profitable apple of a tree

• 13:40Speaker 01

  that that happens to automate a good chunk of what we call work today, which is another thing that I think is like a responsibility of our company to do is just say, look, if if this type of thing works, it's not really an assistant. It's, it's, it's just like, you just put it there and like you talk to it and it's like a colleague and that's great. Like, I think the economy does the same thing, same outputs with less input and or even more output with less input. This is fantastic for the world if we if we just do it well, like, like capitalism and competition can be great and like progress. The entire history of progress comes from this, like, we'd all be farmers if we weren't in favor of automation. It's, it's like, it's scary, I get it. And, and, you know, if you don't think about it all, like, for the whole way, like, it's very scary, especially if you're affected and you're not in one of the lucky seats I'm in, for example. But, but, but ultimately it's good.

• 14:25Speaker 01

  And so, so, so, yeah, I just, I just basically, like, when we started, tried my best to think through this and, and this seemed like the most productive path forward. And, and again, I think it just like we're lucky in to be in a position where we both get to pursue an incredibly valuable product that that's a new generation of thing that just doesn't really exist yet. It's, it just, there is no, like, thing that does your work for you. I think ChatGPT was one of these, you know, monumental moments of a new type of thing. But you just have the first AI assistant, this is like mind blowing to all of humanity. And I think this will happen once more and then maybe once more when you like can just like query the solution to Reman. And then that's probably it, at least in this domain.

• 15:08Speaker 01

  So I hope we can be a part of the second one, maybe the third one and on the product side. But we, we chose the domain we chose because of what I, what I said previously.

• 15:17Sarah Guo?

  Can you, let's talk about those two pieces separately. Like you said, you think it'd be slightly cheaper than a more generic AGI effort. Like the last conversation, we really had one on one, was about how much compute you might need.

• 15:29Speaker 01

  Yeah, I was wrong.

• 15:31Sarah Guo?

  So like, I guess I contrast this effort to like a more generic effort because clearly the large labs, they're using a great deal of human generated coding data. They care about the use case. So like, you know, what makes the efforts different?

• 15:43Speaker 01

  I think a lot of it is direct competition. Now, I think that like, there are totally things I could say here that sound like entirely believable as like major differentiators, but I think fundamentally they are, and I'm going to talk about them. But I just also want to highlight that I think at the core,

• 16:03Speaker 01

  the wider world has understood that code is very helpful and there are ways to deploy compute to improve coding performance. And so therefore, because a lot of compute is deployed in this domain, the need for compute is, is large. That said, also in parallel with the release of this, we're going to be announcing what is going to be one of the largest clusters to, to ever be built. So, so we are correcting for that. I was wrong when we initially talked. You just definitely need a lot of compute. Now there's still, I think, a notion of enough, but it might be a lot.

• 16:42Speaker 01

  There's like this interesting, right? Because there is enough compute for search, for Google search. I don't think if you, if you took GB, if you do Gemini 1.5 Pro and you put it into Google and you like did all the, you know, fine tuning properly and then you swap it with Gemini 4, I don't think anyone would notice unless like you go like prove Riemann, right? But, but like it's, it's good. Like 99% of users' use cases will never notice the difference. So like this does the job. If like AI overviews works now. And I, I think this is going to like similarly be the case for some, for, for each thing. Like there's going to be a model that can prove Riemann. You can make it a hundred times smarter.

• 17:19Speaker 01

  You have your proof, right? So for each thing, there's going to be improvements after a certain amount of compute. And I think I underestimated that number. And I underestimated how much others would focus on code. So anyway, just to clear up, I think our 101, you were right.

• 17:33Speaker 00

  So I guess there's the model side of it. And then there's sort of the productization of that model or the ways people access or interact with that. Is there anything you can share at this point regarding those types of things?

• 17:43Speaker 01

  As we have built the UX, each iteration of the, like, in X UX internally,

• 17:50Speaker 01

  we thought about launching it. We were at this interesting stage of it feels like an uncanny valley where, like, well, you know, clearly you can see signs of life. First of all, I mean, completion is the trivial one, right? Like, like, we decided not to launch completions because it's just obviously going to get killed by the next thing. And then, but okay, it's going to take us a few months to get a prototype of the next thing. We got a prototype of the next thing. And then, when you're looking at this, it was like,

• 18:17Speaker 01

  you can see signs of life, but, you know, like, you guys tried it when you decided to invest. So it's, you know, it's like, would you be using this to like write your, no, not yet. But yeah, we can train the next model and then like that model can do it. But then that model like can also do all these other things that like would go into final shape. So you just enter this like stupid recursive loop until the point we got, we got to this, we get to the point where we're like, okay, like, what's the final UX? Let's just like, let's just make sure this never happens again. And so, so we're trying to like meet the bar of the, that UX now,

• 18:46Speaker 01

  which I hope we will sooner rather than later. The closer you get to it, the sort of dumber it feels to launch the thing before it, because you're going to replace it in a few months. So if it's good enough for that, like, you know, how hard can it be to add these last little few things? So I do think there is a difference between, like, you can launch an extremely capable assistant before you launch full automation. That's fine. But launching a sort of mediocrely capable assistant, like we might do it. We have a deadline internally by which if we don't have the, like, you know, nobody has this right now. Like, there is no amazing, can do everything. It just like feels like a true genius colleague on your team. But, and if we don't hit it by that time, we'll launch the other thing. But I would prefer hitting it. It's just the honest, you know, the reality is.

• 19:31Speaker 01

  Things are hard. Things are, some things, some projects are going great and some are delayed and some are just, you know, there are a hundred fires all the time. This is just how every hard engineering project goes. Everyone who's listening to this and has ever worked on an engineering project, project was like, this is how it goes. I think we, we, you know, there are a lot of things we learned along the way. And there's nothing we're stuck on. It's just things are, you know, God, there's this thing we didn't think about. Okay, let's fix it. And so, so I feel very optimistic.

• 20:03Sarah Guo?

  What makes a assistant like more mediocre versus amazing? Is it reliability? Okay.

• 20:10Speaker 01

  Like, do you just trust it? My engineers, when I have one of the right, right piece of code and another one review it, like, looking at it, this, why, why would I look at it? It went through these two guys, you know? So, so like, what's the point? And

• 20:25Sarah Guo?

  So is the eval bar like, I'm not going to do code review?

• 20:28Speaker 01

  For the, so that's like the fully automation thing, right? And then you can launch something where you're like, I'm going to do code review, but it's not frustrating. If you have to do code review and it's like really taxing and you have to fix half of the problems, like half of the PRs or whatever. So I think the bar for this product is just high. It's not that like, we are like so ambitious and, you know, I think that too, but I, I just genuinely think that there is a gigantic market that gets unlocked in a step function moment where users decide that they're no longer, they're no longer going to use a VS code to write code and send it to their colleagues. They're going to use magic or whoever ends up doing this well first to, to write their

• 21:11Speaker 01

  code for them and then briefly look at it and correct every now and then what, what has been done and then eventually not, right? But that is a step function moment, I think. It's not like you're not going to use this for like 5% of your tasks. You're going to use this for 90% of your tasks or zero.

• 21:25Sarah Guo?

  You don't believe that this is something you can cut by use case, right? We'll be trustworthy on some set of things.

• 21:32Speaker 01

  No, I think if you can. The leap to doing all use cases is small. Like, you can build a UI builder, and then it's like a normal UI builder, or you can build a true great UI builder driven by AI with some added features for that vertical. But then you can do the same thing for all the other verticals, just add the features, you know? And so maybe your product team needs to do one by one. That's feasible, like that, that's totally imaginable. And maybe your go to market needs to be one by one. But the model, I don't think so.

• 21:59Speaker 00

  I guess, to your point on having a very high trust team, how did you think about the team that you assemble?

• 22:05Speaker 01

  So now this is really easy, because we've raised an unbelievable amount of money from great people, and we've got things to show that even risk averse people who don't think from first principles can understand that this makes sense, or who need that initial like seed of trust. But I did find it very hard to recruit when we got started, to be honest, because, you know, when Dario Amode goes out and starts a company, this is trivial.

• 22:34Speaker 01

  Hi, I made GBD3. And when like, you know, there are others like this, it's easy to establish that, that trust in the outcome. So the strategy we adopted at the start was to hire, actually, kind of people that like you do once you're referencing, people who might be, we have one guy who was just like the press at Amazon, working on like Alexa AI, and he just hated it. And he was like, so great, like, he just knew everything, every single paper. I was, he was like, he is rag. Like, I would go like, you know, let's talk about this. And he would just know everything. And let's talk about this, and he would know

• 23:16Speaker 01

  everything. And let's talk about this, and he would know everything. And, you know, like, he studied engineering in college, and then just got into AI. He's, he's one of the most brilliant people I get to work with. And so we hired him, and like, he did a bunch of stuff, like, he invented a new sharding dimension, like, for total training. And this is just like a random, you know, you, you have to like, do, you have to really pay attention to identify these people. But then there is an amount of drive and loyalty that you just don't get if you poach, like, the obvious guy, right?

• 23:51Speaker 01

  And so that's the type of person we have. And we have a decent number of them now. We've gotten really good at identifying them. I would like to have like roughly four times as many if we could. But that said, like, again, I think with the series of announcements that's, you know, going out in the in the batch that this podcast is going live, that again will get, you know, easier and better. But I love our culture. It's just. Everyone cares about the mission deeply. You know, what I said earlier about why we do what we do, that, you know, safety bounded AGI recursion. Like, yeah, this takes a lot of brain power and understanding of the world to comprehend that this is the right thing to do.

• 24:32Speaker 01

  And and or a lot of trust to trust in organization with doing that in the first place. And everyone cares deeply about this, but, you know, we don't. We don't do it for like the, you know, years like your marketing sign or whatever, you know, we don't have to. That's it's just who we are. At the same time, everyone is deeply productive. They all, you know, when there's one of our primary, I go over our core engineers who he writes like the inference. And I guess one of the two people writing the inference engine and some stuff and kernels. And when we need join, I was like, why do you want to join? What do you want to do? And this was before we raised this giant stack of cash that's getting announced now. It's a it was like the tiny amount still compared to other labs, or I guess compared to any lab by a large margin.

• 25:15Speaker 01

  And he was just like, well, you know, he saw this as an opportunity to. He wanted to be one of the best or he said he wanted to try to be the best kernel engineer outside of Nvidia. And he didn't say this in an arrogant way. He just said like, this is a ton of work. I've done this for the last few years and I just need to be in an environment where I'm sufficiently challenged to do this. And he's been grinding every day and and just having like that level of ambition, but not with like the typical San Francisco. You know, but but it's it's it's like quiet with humility. Drive, you just come into the office every day. You're not like working until 3am because you prove that you look like this is our culture. You do it sometimes because you're just so obsessed and you try to be healthy. You do work all the time because you just care so much.

• 25:58Speaker 01

  But we're not buying the IP by poaching someone from like a lab who tells us how they train GPT. Never done this, will not do it. We just do our thing. We have our plan and, you know, we have brilliant people who I'm delighted have trusted us and spent their energy and their best years on our company.

• 26:17Speaker 00

  What does AGI look like?

• 26:19Speaker 01

  I think it just talks to it and it does everything. And it asks two questions. That's important.

• 26:24Speaker 00

  And do you think the existence of that, I guess one could argue it can increase GDP, but it may also decrease a lot of sort of human-driven activities.

• 26:37Speaker 00

  How do you think about the eventual implications or impact of AGI?

• 26:40Speaker 01

  This is going to take longer. I'm going to try to compress it, but it's actually quite complicated. One of the biggest problems with this question is that everyone tries to simplify it

• 26:51Speaker 01

  by picking one side of the argument. For example, and this is just one example, centralizing power is terrible, so therefore you should open source everything. If I stop now, this is reasonable, right? And please don't make a quote out of this because I don't believe this. And then you could say, well, I should not give these, I should not, I should learn how to do PR. I should not say these sentences. Anyway, you can keep this in. So there's one way to say this. And then the other way to say this is, well, this is like nukes. We all fear this existential risk thing, like maybe we care less about the, you know, we, we, or we care, it's not that we care less, it's just that we think these intermediate problems are completely solvable. But, but you can't open source how to build a nuke. This is just terrible. So, so the problem is that both of these things are true.

• 27:35Speaker 01

  And the problem is that there are 10 questions like this, and both of the answers are true in all of them.

• 27:41Speaker 01

  And so what you get is people arguing on X, claiming one side and ignoring the other. And so, so I think there are like 10,000 possible futures. And which one we end up with depends entirely on which answer we choose or whether we find the sensible middle ground and we, we manage to have a rational debate.

• 28:02Speaker 01

  The reality is, I really truly believe that capitalism and competition are the only chance we have to provide a, an optimizer that is capable of getting us to the right place. I think we need the right guardrails to do that. Not stupid guardrails. I'm not saying any guardrails. I'm saying the right guardrails. And then by the end of it, all work on a computer at least, and probably like robot factory stuff. I know less about that. I haven't run the cost structure. But probably that too. I just don't know the cost structure. We'll be automated. And humans will do other things. I don't think they'll do, I don't think we'll be required to do work for financial gain, but we will probably be able to. Property will be a thing. I think if you own apartments, like that won't go

• 28:46Speaker 01

  away. If, there's Etsy. Etsy is a, Etsy is a great proof of concept for what happens after AGI. This is completely useless. Like you could just buy the made in China product. It looks the same, but it's not made by that human, you know. So that's a thing, I think. Like that will grow really big. Whoever owns Etsy, I don't know, but you will get rich. Josh, are you listening? So, so just do your thing and hold through. If you don't get automated, maybe you do actually, but your company doesn't. The, the, so that might be one way this could go. I think games will be huge. People who are competitive, like all three of us, I would guess, will be deeply frustrated by the fact that they can no longer fulfill their desires

• 29:29Speaker 01

  for competitive interaction through work. I, look, I care much more about the positive outcome of what I do than I care about winning personally. But this is a hell of a lot of fun. Like, I love what I do every day. I get to build AGI. I mean, holy shit. And like, I mean, isn't it, it's great to compete against other competent people. Like this is, if it wasn't in such a serious environment, you know, I would just be enjoying it. Now I have to be conditionally enjoying it. But enjoying it. And it's gonna go away. So weirdly, I think like we're the ones who are like aren't the most on like a meaning level. Because, because like, we won't be able to contribute to society as much. Because, because the work I feel like, at least I can speak for myself, like I'm doing, I feel tremendously is fulfilling and meaningful, and that's going to be deleted. As much as

• 30:13Speaker 01

  it sucks to say and hear this, I just, it is going to be deleted. And then, and then my ability to be competitive is going to be deleted. Because, you know, Deep Loop beats everyone in chess and like. Hopefully, magic or some AI system will beat everyone at coding. And then like, what am I, you know, and then like, there'll be some CEO system. And then like, the responsible decision will be to, you know, like, have that thing be the CEO. And then that will happen at some point. You can debate how long it takes, and that doesn't really matter for the argument, to be honest.

• 30:38Speaker 00

  Definitely feels like 10 to 20% of society will be deeply frustrated in a post AGI world. And there may be, you know, 70% that's indifferent or happy. And then maybe another 50%, whatever it is, and then the rest will be very excited and thrive.

• 30:56Sarah Guo?

  And There's a, there's a book that is like worth reading, skimming from Ryan Event all the way back in 2016, called The Wealth of Humans, and explores the question, like, it's not like really focused on AGI, but it explores the question of, in a time of abundance, where does your identity come from? And how do we keep people like happy and productive in that society? I think it's really interesting because it goes beyond like a surface level question of just like, oh, like, if we can make UBI work, like, are we all okay? And the answer is no, right? Like, I wasn't here for UBI to begin with, or any sort of income.

• 31:25Speaker 00

  Well, you can actually argue that abundance has created more issues in society than one would expect. If you just look at fragility and other issues and, you know, what people consider actual problems in the world versus real problems and, you know, one could argue that's an outgrowth of abundance and relative peace that we've seen for 30 years. And so it's an interesting question to ask, and the limit, what does that look like?

• 31:47Sarah Guo?

  Still on this abundance train, but yes, it's an interesting question.

• 31:52Sarah Guo?

  I have one more for you, Eric. So you mentioned the Reiman hypothesis a few times. Like, what is the thing that you really want to try in terms of new knowledge? That you hope magic will be able to answer, right? Because you could, you could say Reiman, you could say Navier-Stokes, you could say P versus M. There's a bunch of like interesting problems in math, or maybe climate, or whatever else.

• 32:11Speaker 01

  Yeah.

• 32:12Sarah Guo?

  But if you're truly ambitious, like, there's got to be a question.

• 32:14Speaker 01

  Look, my honest reply is that I think all of these questions are going to get answered. And my North Star, at least personally, and I think most, this is true for most of the company, at least, is that

• 32:26Speaker 01

  I just want the world to be in a good place in 30 years. And after all this is done and thus said, and this is the past, and we talk about it the way we talk about mobile phones.

• 32:38Speaker 01

  I just want the world to be in a good place. This is the largest transition we have ever faced and work will get automated and this is crazy. I will have to find new ways of finding meaning and that's crazy.

• 32:51Speaker 01

  The economy will be like, I don't even know. Governments are gonna have to figure that out. But I'm curious about a number of things, but it's just not the North Star. It's a nice side effect that I think is just, I mean, in a way it is the North Star, right? Like, what are we building? We're building this like automation engine that can answer a lot of questions. But in a way, I think this is just gonna happen. So you don't need to try. The thing you need to try is to make it go well. And if you make it happen and go well, all these questions are just gonna get answered. Like, this is a side effect. Is it, if someone's gonna ask like, oh, what's remote? I, I'm gonna look it up. I'm gonna try to understand the proof. I'm gonna fail because I'm not smart enough, but I'm gonna try. Like, it was really interesting. I'm gonna spend like a few weeks on it and, you know, it's gonna be fun. But, but that's just not, that's just not my North Star at least.

• 33:34Speaker 01

  I just want everything to be fine in 30 years. If it is, the world will be amazing. And like, because all the ways in which it could not be amazing are like terrible. So, so if we simply keep it not terrible, I think it will be amazing. Because I can't come up with like a mediocre AGI future. It doesn't exist. Like, I don't really like get all the stuff we need and we find a great way to like lymph together and not use it as a weapon. And, and, and then, and then like everyone has all the thing, nobody is starving and like we all have like infinite computer stuff and like, like it's, it's a, we find new meaning and like all this is like, and we're not dead. Like, that's a good start.

• 34:16Speaker 01

  Like, the, the, um, that world is amazing. And so all the failure modes are terrible. So, so really, I'm sorry, it's just the only thing I can think about is like the, the bimodal nature of this distribution that we're rushing into. And really what's happening is that this like smooth distribution of like this cloud of uncertainty is like slowly collapsing in, in like this bimodal thing. And everyone is sort of gonna progressively understand this more and

• 34:44Speaker 01

  And we just, as sitting in a chair that I'm sitting in, I just don't feel like I have the right to think about anything else. Like, like, this is, this is just the responsibility. And like, people will look back in history and like, all these questions are answered, and that's amazing. But, but like, great that like this stuff didn't go wrong, you know?

• 35:00Sarah Guo?

  Distribution skew is right.

• 35:01Speaker 01

  I'm sorry, this is not the answer, but.

• 35:03Sarah Guo?

  No, no, it's okay. The distribution skew is right. It's gonna be, it's gonna be good. Like, I think an interesting product, like user experience question is in, you know, in an era of coworkers, rather than, let's say, completion, copilot type products, like, how do you think about magic interacting? Magic is, you know, first and foremost, a model company. But how do you think about it interacting with all of the other tools and interfaces that developers use today? Like, the IDE and whatever, like, does it matter?

• 35:32Speaker 01

  Great question. Very good question. I changed my mind on this like four times. Again, I also just like to be like, nobody has a clue. Everyone's like, try things. So it really just matters what the market wants in terms of product. We'll just do whatever the market wants. Our current state of belief is that you want the system to behave like an employee. So it uses the toolset that you give to your staff members in an interface that is either the same or specifically crafted for AI to be better than a human could use a tool. For example, Graffan, a log ingestion, I'm sure we can come up with better ways for AI to use this than humans are using it. So maybe that, I anticipate that companies will integrate into magic and maybe others. Hopefully, I'm guessing there'll be competition. What's this is a thing?

• 36:18Speaker 01

  But it'll be, I think it'll be such that systems will be, AI systems will be on a level with the human, and tools will be below it. That we will not view this as a one-to-one integration, that we will view it as all these tools are being adapted for AI, the way websites had optimizations made for Google search crawlers, crawling. I think there'll be a tool optimizations made for AI. And for those that don't have it, the models will just use it natively. And the agent, the model, is the main thing that matters. And everything else will get sold for you by other companies. And it's the same way how like all, you know, this army of wrapper companies is gonna get swallowed

• 37:00Speaker 01

  by AGI companies doing their own agent stuff. The same thing is gonna happen there. Like, if you build your own tools, it's just gonna get swallowed by simply the model learning to adopt everyone. This is just. I just like to think in the endpoint. So I think the endpoint is the model uses things the way a human does, and then maybe also more.

• 37:19Speaker 00

  Oh, well, Eric, thanks so much for the very wide-ranging and interesting conversation. Thanks for joining us on No Priors.

• 37:23Speaker 01

  Thank you. And yeah, first and foremost, thank you again for supporting Magic and thank you for giving me the opportunity to be here.

• 37:30Sarah Guo?

  Great to see you. Find us on Twitter at NoPriorsPod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.