Ep 97: Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It

Jev creator and TypeSafe AI co-founder, Diogo Almeida, joins Jacob Effron to talk about why AI still isn't automating the easy work it's been smart enough to do for years, and why he built a new class of model around reliability. Diogo coauthored OpenAI's InstructGPT paper and spent two years in stealth building Jev, a "system one" model that returns structured decisions instead of text, which launched in September. Diogo and Jacob discuss what the viral launch was really like, why he thought Jev would take a week and it took years, why almost every LLM call should be meant for a machine, and why developers should push model providers harder on jaggedness. They also cover the tyranny of the KV cache in coding agents, why new model shapes are cheap and reliability is the hard part, his estimate that coding agents deliver a 20 to 40% speedup, why he sees financial engineering as the labs' underrated strength, and why he thinks we're two gigantic breakthroughs from true AI agency.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  As you think about Anthropic, is that something you're optimistic on is like, hey, that's that's undervalued or something? You're like, oh, I don't know what's going to happen to that.

• 0:06Speaker 01

  I get the sense that a lot of the expectations baked in are ones that are not just product level, but based on like misunderstandings of the future of AI and what it can do. So my guess is that because some people might be valuing like building God is basically infinite, there will be some amount of miscalibration there. And I think that, you know, like the let them cure cancer is going to be like Marie Antoinette's, like let them eat cake.

• 0:37Jacob Effron?

  Jeb took the AI world by storm about a month ago, and I've been wanting to talk to Jogo, their CEO of TypeSafe for quite some time. So it was really fun to have him on the podcast. I think maybe even better with with a month to reflect on everything that's happened. Because we talked about a ton of different things. It was really awesome to get Jogo's perspective on, you know, the shapes of problems that he still thinks need to be solved by AI models, how he thinks about pushing Jeb models to be even better. It was also fun to get his thoughts on Anthropic's upcoming IPO, AI safety, a bunch of other topics. I think folks will really enjoy hearing from him. Without further ado, here's Jogo.

• 1:13Jacob Effron?

  Well, I've really been looking forward to this. I think ever since your like viral launch, I felt like the second that happened, I was like, I have to have you on the podcast. And so appreciably making this happen. I think actually in many ways even more interesting that we're some weeks removed from it. Given that like the early results were not what you expected and maybe not as as as strong as you were hoping, like what gave you that kind of early confidence in this research direction of RLCD? I mean, I feel like you kind of alluded to it in the in the GPT three, you know, example. Like, it seems like there has to be a leap of faith that any, you know, great research effort undergoes. And what was it about this approach that just, you know, in the early days, when you didn't have the signals early on, you know, you were still kind of confident in?

• 1:53Speaker 01

  Oh, man, oh, man, oh, man. This is like, I'm getting a little emotional right now.

• 1:59Speaker 01

  I have a fun little side topic that gets into this, if that's okay. My co-founder was debating on whether or not I was mission driven at OpenAI. I would have described myself in hindsight as one of the lowest percentile mission driven, because I think that the OpenAI mission is not really, it's kind of intentionally left as vague. So many people's missions can fit into it, but like the grand mission was never something I super believed in, even though I truly want AI to be extremely, extremely transformative. So I'm still trying to figure out, like, am I, was I mission driven then or not? Because I'm really mission driven now. But the answer was, you know, just a lot of work trying to square like AI being super incredibly awesome,

• 2:42Speaker 01

  which I think it is, and AI being really not that useful in every like productivity stat around. And this is like the kind of thing that like most people like just focus on one or the other, and they never have like a bridge answer or like they never have a sensible bridge answer. I think distribution is like a nonsensical answer here, because so many companies have gone all in on that and then gotten burned. It's it's almost certainly not a distribution. And to me, it was like believing that like, hey, AI is clearly smart. And also that the future is clearly going to be machine native. The question that always motivated me, and like I used to ask this question even to early people joining or even to investors in our early, you know, like our early rounds,

• 3:24Speaker 01

  or even like the pre-seed round, which I don't even know if that counts, like taking a safe. I used to ask this question of like, okay, like you have, imagine LLMs are an API, you know, like a like a sub module that is called by software, which even in the most string model lands, they still need to be called by software, by something like a harness that harness can be a while loop. But what percentage of those calls going into the LLMs are actually meant for human consumptions? So for consumption, so things like style matter and, you know, prose and all of that versus for computer consumption, which is going to be very, very different. You know, like, I don't think a computer really ever wants a string, except when that string happens to be an extreme cardinality enough, which is a very valid use case.

• 4:08Speaker 01

  But it's very rare that that is the thing that computers want because they operate on bits. And when I just ask myself or anyone this question, even the most string pilled of people will give many nines of like percent for the machine native component. It's just a question of when that will happen. And to me, it was always like, okay, what does that thing look like? Let's work backwards from that cognitive core and make it work. And I will say Jeb is not yet there. There's still so much more work to do to make it so much better. And that's that's what really drives us all.

• 4:40Jacob Effron?

  Is Jeb not yet there because, I mean, you know, today, the shape of of like, you know, calls and tokens actually is more for humans than it is for for computers themselves? You know, how much of it is just you're also waiting for the use cases themselves to evolve? And how much of it is, you know, areas you want to push on the model?

• 4:57Speaker 01

  I think it's a bit of both. It's like this interactive dance. And this is also why we released like these dreamer ready features, like the things that we are,

• 5:07Speaker 01

  you know, like wouldn't recommend for the most stable of use cases, because we really want to be honest about our models' properties. So we've released a lot of these things, because the reality is that the people who best know how much reliability, you know, robustness, like intelligence is needed for their downstream application is not us, it's the developers themselves, because developers are the expert at this, and they will know way better than anyone in our team. We have smart people, but they will know better what reliability profile they need, as long as we give them the right knobs, which string models don't tend to do. So there's like an entire spectrum of this. I think that they need to co-evolve together. The reason why we launched was not like, okay, this is like a good moment in AI. Everyone told us not to launch then, just FYI. It was that we felt like, hey, this is getting to be something we're really proud of.

• 5:52Speaker 01

  Like, it's kind of like impressively smart at a lot of things if you prompt it right. And we are ready to like keep pushing that limits, that those limits with the rest of the field to help us co-evolve use cases, figure out weaknesses, deal with jaggedness. Like, there's so much we do to focus on jaggedness and insufficiency and intelligence that having people play with it would be so helpful.

• 6:13Jacob Effron?

  Why did folks tell you not to watch?

• 6:15Speaker 01

  We launched around the time of the pacing the frontier discussion. And people were like, hey, like, it's kind of like a bad vibe to like launch a capabilities product during this. Like, everyone's like really doom and gloom about AI. And it's a kind of a bad vibe, and like, I don't know if you guys should do this. And we're just like, that discussion never really made sense in the first place. Or it is at best incomplete, in my opinion.

• 6:38Jacob Effron?

  You talk about kind of, you know, wanting to push in so many directions. I mean, I think one thing, you know, that you guys do is you're always kind of transparent about the things the models can't do, you know, the areas where they're not as strong. How much of that is like the roadmap for what you're pushing on? Like, you know, how much of it is is other areas? Like, you know, how do you think about like where you where you want the company to push in the next month or two?

• 6:58Speaker 01

  Month or two, oh, that's a, that's a.

• 7:00Jacob Effron?

  I mean, the AI world, I don't know if I can ask, I don't know if I can ask longer than that, right?

• 7:04Speaker 01

  I wouldn't say jaggedness is purely like our roadmap. There are some things that models truly are weak at. And like, this is where like that taste stuff comes in, right? Like what I was talking about, like the reality of what you can extract from these models. The reason we call them system one models, and among other things, and system one, actually, people did not want us to name that it either, because system one thinking tends to be a bad thing. Like, people use that as kind of like an insult, right?

• 7:29Jacob Effron?

  Yeah, it's like the the part that just naturally happened from evolution, you know, from back in the day before.

• 7:33Speaker 01

  Exactly, instinctual, which makes mistakes, etc. Exactly. But and like, people often like say that that is the mistake making side. And I believe that the shape that actually AI gets compressed to happens to be like very system one, and therefore the honest thing to do is to communicate its weaknesses. Because I think that on its own can be incredibly, incredibly valuable for software. And most of what people need actually is system one stuff. You know, like, why can we solve millennium price problems in math, but not automate anything touching accounting? Like, that is, that should be really weird to a lot of people. And like, I think that that is like still our North Star, even if people get distracted by like cost, speed.

• 8:16Speaker 01

  You know, like, oh, it's it's good for like really fast decisions or whatever else. Like, I see these as like distractions compared to what is really needed for AI to meet its potential. And that is my my North Star.

• 8:29Jacob Effron?

  And so obviously, you put the models out there and you have a lot of people using them in different ways. How much does like the the real world usage of the models come and feedback into the roadmap versus just like the stuff you were you all planning to do?

• 8:39Speaker 01

  I would say it's almost none, real world feedback. I think people should be complaining to us so much more. I think that like this, the field is still in a Stockholm syndrome of like, oh, yeah, the model providers just do stuff in their one track path. And we happen to be like, the victim is the wrong word. But like, you know, like in, what's that movie, WALL-E? You know, like, they're like, they're just like the people in the chairs that are just getting fed the AI. And I think this is like the state of things right now, where like people plan around AI changing, but they don't have a sense of agency of where to take the field. And I think that this is not the way it should go. That's actually really disempowering and kind of lame, in my opinion.

• 9:21Speaker 01

  And also, I think that there's so much more we could do collaboratively as a field. And I think that the field could like push us a lot more to say like, hey, we want to be doing these things. We think it is within the realm of capabilities. Let's figure out how to do it together, but not just putting stuff into the model, but figure out how to prompt them, figure out how to like unearth capabilities. Let's work together for these sci fi use cases, because I think that is the way the future is going to go. And I would want so much more of that. My my joke internally is that the number one way to get me to respond to something is to talk about model jaggedness, because you have like very high up people say like, hey, can we meet with Yogo? And I'm just like, no, can you meet with my CTO or GTM head instead?

• 10:04Speaker 01

  But if they complain about model intelligence, I am like on that and like debugging that myself, for better or for worse, because that, I think, is the thing that will really move the needle for us long term. So right now, we mostly have to go with our old roadmap, because we are not.

• 10:20Speaker 01

  I think people are still in that mode of like, yes, let's just do whatever the model providers let us do, instead of like, let's collaborate on this like insane future together. And actually, there's another aspect of that. I think people have a deep distrust of model providers too, because they train on their data. And, you know, like, this whole like discussion of like, can GPT wrappers even exist? I think is very disempowering too, because the whole application ends up becoming a chatbot. So the moat ends up becoming the equivalent of a system message that you send every time versus like the moat and the intelligence and the behavior and the specialization actually existing in software, where we just make little micro decisions for you, and you do all of the sick logic.

• 10:59Jacob Effron?

  Yeah.

• 11:00Speaker 01

  So it, there's so much to do, there's so much to do.

• 11:04Jacob Effron?

  No, I think your point's really interesting, because I feel like when Jeb came out, there became maybe like a common set of patterns or rules that folks were like, oh, yeah, like this is a really good use case for Jeb, and you should, you know, try it out in this thing or that thing. And everyone tried it out and, you know, kind of got that sense of patterns. You know, I almost feel like people are trying to get their own mental models of like, okay, I use like, you know, Opus five five for like this kind of thing, and I use Jeb for like this kind of thing. And. I feel like your point's well, you know, your point is like, this is a dynamic evolving thing that you could, there's not strict boundaries, I guess, between those two types of model categories. But like, what's your?

• 11:38Speaker 01

  Ish. Ish. I actually think that what you're describing is a little bit sad to me. And I understand that like, we're playing a role in this. I do not think that that is the most developer centric worldview. Like, for people to start building like theories of mind of the sub models they call and have their systems locked into that is, I, in my opinion, a really, really bad thing. I think the North Star for intelligence is gonna be a lot like SQL. Like, we wanna be so reliable that it's actually provider agnostic. We obviously want the best models, but like, we don't want it to be locked into like, things that only work with Jev.

• 12:14Jacob Effron?

  So what is your, you know, mental model for how this all evolves in terms of, you know, this like, thing people are doing today of like, you know, hey, you know, Jev works really well for these things. Maybe like, there's other sets of models that have been on, like, to your point, RLVR works really well for code and math, other, you know, we'll see what else it ends up working really well for. Like, does, well, I had to interrogate that gesture. What was that about?

• 12:36Speaker 01

  You wanna, you wanna interrogate it right now?

• 12:39Jacob Effron?

  Yeah.

• 12:39Speaker 01

  Well, really well is quite relative. I, as far as I can tell, even the best companies still require humans in the loop for all of this stuff, despite superhuman results in some axes. And that still, like, people should have an answer to why that is the case. Like, if they are so unbelievably good at software, like, why do normal software engineers have anything to do? Not that I want them automated, but like, they clearly aren't. And there's clearly something missing there. And I think, thinking of like, intelligence as this single axis that is measured by whatever the benchmark of the day is, is like, totally wrong. And there's like, obviously nuance missing there, because the people

• 13:20Speaker 01

  who go all in on not understanding that nuance get burned super hard. So, I think that RLVR is incredible, but it is incredible at human in the loop stuff still. And I think that wisdom comes from understanding reality where it's at, knowing its weaknesses. And like, this is, we are obviously use agents a lot, but we really value understanding its limitations. You know, I just wouldn't say that it's like, oh, super great, and it just solves everything. Personally, maybe I'm using the wrong models.

• 13:48Jacob Effron?

  I think a lot of our listeners are trying to figure out, you know, like, a simple mental model maybe of Jev, which I know you disagree with, is like, you know, hey, there are some set of tasks or some set of things I'm trying to do that will make sense for this. There'll be some set of things that make sense for other things. And I'm sure 95% of our listeners right now are trying to just like, adjust their own mental model of like, okay, what are my like, Jev use cases? When do I use Jev?

• 14:09Speaker 01

  That's a very pragmatic take for the short term, and I'm in favor of it. This is like, where it's about giving engineers power. Because, like, we shouldn't be overly opinionated about what they use it on. And if they need to do things to make it work, abso fucking lutely, hell yeah. Like, if you need to do like, the worst dirty attacks to provide value in the world, like, you know, like, we are not gonna stop you. Like, that is awesome. And also, if you have to do too many hacks, that's still our fault. You know, like, I had a deal where you wouldn't need to do that. And the more it becomes close to like, semantic understanding, the more beautiful the code is, the more reusable it is, the more you can plug it in into places that you wouldn't expect it to. It's kind of like, the Unix philosophy of like, doing like, incredibly small, useful things, but because people have a mental model of it, they can compose it together into big ones.

• 14:54Speaker 01

  I feel like this is great. And also, like, we should not be like, recreating red Xs. We should be recreating like, beautiful primitives. Not that I hate red Xs, I just don't like reading them. You know, and like, you know, like, it's like, it's like more, what's the word? Is it write-only code type stuff? And like, we want to be beautiful, composable. We want to like, build on layers, like, LMs have failed to do so. And we want stuff to be more like software. You know, I think I have an okay mental model of software engineers at worst. Like, it's kind of weird that people even want to update to the latest and greatest models. I hate updating my dependencies. Like, updating dependencies is like, the worst thing ever for production software. It's fun for personal use. Like, I used to

• 15:36Speaker 01

  use Arch Linux, I get it. And also, like, why would I want to bear the burden of that? And the only reason people keep doing this treadmill of like, maybe the next model will fix my problems, is because I don't think they solve those problems so well, that they can run in the background, fuck off for literally ever, and maybe, you know, revisit it every like, couple of months to see like, is there a cheaper or better alternative available? But it works fine right now. I don't, I don't know of anything like that. And that is, it's a sad state of the world. Like, it's very counter to how software engineering tends to work.

• 16:07Jacob Effron?

  I guess in your vision of the world, like, do LMs have a place in the world in like, 10 years?

• 16:10Speaker 01

  Super duper. Absolutely. I, I, I hope if that they do not have a place, then something strictly better exists. I'm not opposed to LMs. I'm not actually opposed to either chatbots or coding agents. Completely different things. And I, I really don't want to go to a world where they don't exist. The thing is, I think people overuse them for a lot of tasks. And I actually hope the market for that increases too, as people get better at it. And I'm, I'm like, basically certain that there's tons of use cases for it at basically every single intelligence cutoff that has existed, maybe since GPT-4. Like, all of these models had beautiful, awesome use cases that maybe would get Pareto dominated by another model, but like, they should be able to run in production. And

• 16:55Speaker 01

  it takes a lot of nuance and understanding. And it's like, it's like, really not built for engineers. So, I love LMs. I hope that I, I can continue to like, chatbot anything that I have like, I need like, ex, pseudo-expert level answers to that don't need to be correct. That is the use case in my mind for chatbots. And I love being able to like, spin up, like, throwaway software instantly when I want custom dashboards and stuff. Because I think it's the best use case for coding agents. And that should stay for the rest of time. I don't think people should be doing that. And we should keep on building our toolbox, so to speak. Like, it's, I think it's just awesome.

• 17:31Jacob Effron?

  Is your intuition that the toolbox will include like, many other shapes of models in the future with, you know, other RL acronyms? Or if we're talking 10, 15 years from now, like, are there, you know, are there a bunch of different like, tools in this toolbox? Or is it kind of like, your approach and the LM approach cover most of the use cases?

• 17:48Speaker 01

  I'm gonna end up saying something spicy here, but, phew. I would say when I first, when we first launched this, I was really excited for novel shapes to emerge. Really, really excited. And I still think that that will be the case. I think new shapes are freaking awesome. We plan to keep shipping them. I'm a little bit worried that, again, what we talked about, maybe this was like, before we got on the record, that ML people especially have like, kind of like, lost the actual message I was trying to say. And I'm worried that people are now overfocused on like, the shapes of tasks, rather than the reliability. The shape of task is incredibly, you know, like, it's super easy to do that. When I

• 18:32Speaker 01

  thought this would take a week, I thought it would take a week to make the shape of the task, which it did. And this is what's happening now with other people making similar models with similar shapes. But the shape really doesn't matter compared to actually it being useful. And that is the thing I think people really, really need to focus on. So, there will 100% be way more shapes than these two. Even if no one else does it, we will do them. I think people are making too many new shapes right now and not caring enough about, like, the utility of it. And I'm worried that this will like, actually burn the goodwill that people, people don't like being overpromised, underdelivered to. And like, these things are like, pure overpromise, underdeliver. And I think that that's like, not good for the world. You know, like, if people have like, creative playful energy,

• 19:14Speaker 01

  we don't want them to waste it on stuff that clearly doesn't work. As far as I can tell, people dislike that. And even if they disliked it, I hate it. Like, I, oh man. And I think there's going to be a balance. There will definitely be a lot more future shapes to be had. And I just really, really hope that people start giving like, unbelievable levels of fucks for utility, usefulness, reliability. What would a developer want in this situation? Rather than ML people who I think focus on like, I want to flag plant. You know, like, I want to be the one who names the thing that gets cited. I want to like, get revenue in the short term, because people will play around with it. And I want to do the same

• 19:56Speaker 01

  thing that every other AI company does. I really hope we don't do this. But overpromise, underdeliver in order to get attention, because it feels like a lot of the AI economy-ish is really driven by that attention. And that is so, so bad for developers. It's like, actually, the absolute worst for the developers who do not have like, the time to test these things themselves. Because they just get, they're the ones who get swept up and all this. And this is also why I'm anti-b benchmarks. It's a, it's a- It's so bad for those people, you know?

• 20:26Jacob Effron?

  You have to play around with some of this stuff yourself to have any sense of how good it actually is.

• 20:30Speaker 01

  Yes, and like that is tragic, you know, like it makes sense. Like, I actually think that as far as I can tell, people have stopped caring about like benchmarks for our class of model. Maybe I'm like in the wrong niche for this. But my sense, and there could be survivorship bias, is that people already don't care about new releases here because they've all been lied to so many times. And this is good for us as a company, but bad for the field. Because actually, if someone makes a genuinely good thing, there should be a way to acknowledge it and shout it out and say like, hey, this is awesome. But like in the world where everyone optimizes for benchmarks and demos and things that don't require reliability,

• 21:13Speaker 01

  it's very hard to communicate that, other than this very manual process of evaluating models themselves, which is actually super duper hard. Because benchmarks also don't capture that.

• 21:24Jacob Effron?

  One thing I'm struck by, you know, is obviously this idea of like these shapes of problems, right? And chatbots are very good for some shape of a problem you're trying to solve. Coding agents, like you said, very good for some shape of a problem. Obviously, folks have found JAV really helpful in different shapes of problems. Are there other problems out there that you're still like, God, there's just nothing out there today that's good?

• 21:42Speaker 01

  So many.

• 21:43Jacob Effron?

  What are like some that come to mind that like of these future areas?

• 21:47Speaker 02

  Hi, I'm Anne Maris and I'm the producer of Unsupervised Learning. Thank you so much for listening to the show. I love working on this show and I have the challenge of hitting 35,000 subscribers by the end of the year. If I don't hit that, I'll be fired, so please subscribe. Thank you, bye.

• 22:03Jacob Effron?

  What are like some that come to mind that like of these future areas?

• 22:06Speaker 01

  What I realized is that a lot of people are still stuck in like fairly basic patterns. And we have like a gigantic job to both push the frontiers of model intelligence reliability, all of these things that we really care about, as well as education to push the limits of software. And it like, there's like a thing in like the intersection of education and pushing the frontier forward, which is like this collaboration between research and product, which is, I think, where all gigantic like ML wins come from. And this is, I think, is like kind of an unexplored place. Like, if we had more help, we could do so much more here to like encourage people in the world or even fund the kinds of things that we think should exist and really allow people to push their front,

• 22:49Speaker 01

  the frontiers of what software can possibly do. So I'll give a couple of examples of this. And, you know, this is going to be a little bit of a spoiler for a launch, a release we will have sometime in the future. It is that we want to like fund these things and there's so many cool shapes of stuff missing. Like, one of the things people know I'm passionate about is coding agents that are free from the tyranny of the KV cache. I think that so much coding agent is built around the KV cache. And the KV cache is freaking awesome, too. We, I don't want to like toss it away, but I want to be aware of it and design around it and make different sets of trade offs to make things that were like pure sci fi possible. Another thing that I think would be so unbelievably cool is, um,

• 23:35Speaker 01

  do you know what dependent typing is? Dependent typing is a lot about like adding constraints to your programs that are like higher level than purely the type system. You know, like you can make sure that your function returns an int, but dependent typing would allow you to do things like it is an int greater than zero. And then now downstream or downstream applications can use those properties to have their own sets of guarantees, et cetera, et cetera, et cetera, to get higher and higher levels of composability with your software. Because they're more legible and they don't require as much like unit testing edge cases and also as much human understanding to understand the whole system. And one of the things I think would be so freaking cool would be a dependently, a semantic dependent typing,

• 24:17Speaker 01

  where instead of like having to represent your outputs as pure types like int, float, you could actually say specific properties of that in natural language, because there's so many properties you could care about in software and have a type system that uses JAV like models verify that everything works together. I think that that would be so unbelievably cool. Maybe it doesn't matter in a world where everyone uses coding agents. But I think that that like, like, like we should do stuff like that just because it's awesome. And that's what early internet energy is about like to me. So, security is a huge thing as well. Like, as these models get cheaper, faster, all over the place, where can we add extra levels of security? Kind of to avoid, if you guys saw the news about like the Cohere Qwen 3.8 Next Flash,

• 25:02Speaker 01

  debatably makes a sketchy call to like phone home, it's debatable. I'm not going to talk about whether or not that's legit or not, because I haven't read enough about it. But why can't we just have checks on top of every single tool call that actually are semantic to your organization, rather than whatever Anthropic thinks should go into their auto router? You know, like maybe in this part of the codebase, they shouldn't be making any calls to this other part of the codebase at all. And how do you add these checks? You know, you, you need like extremely fast and cheap security all over the place, which could be super awesome. You know, I'm not saying everyone should build it right now, but I think that we should collaborate with, you know, the making, pushing capabilities and software in order for like thinking.

• 25:43Jacob Effron?

  It's so interesting, because a lot of those problems, like the problem you're just talking about now, like you could imagine, you know, approaching from the perspective of a model company or like there's all these, you know, enterprise agent security companies that are popping up, right? That aren't model builders themselves, but they're like, yeah, let's put in some like, you know, some actual like deterministic software to monitor these things. And it's interesting to like, think about you approaching a similar set of problems, but more from like a model perspective. Versus obviously, there's plenty of other folks that are just trying to build software that you can put around in LLM.

• 26:11Speaker 01

  So my philosophy is that we should enable those software players to win.

• 26:17Jacob Effron?

  And just give them the tools to build the products.

• 26:19Speaker 01

  Exactly. But ideally, they also tell us that we don't want their data. But they should tell us like, hey, you have jaggedness on this particular axis that makes our thing less reliable. Let's work together. And actually, they could just tell us like, we are not going to tell you what we were going at all. Here's the rough shape of this task, fix it. And I would just be like, hell yeah, thank you so much for telling us this. Hopefully, it solves your problem and a whole bunch of other people's problems. I hope you produce tons of value in the world. Like, I hope you make like a decent amount of money. And I hope that this enables other people to build all sorts of other things, because now this jaggedness thing is healed. And that is the relationship I hope to have. And this is why I'm like so, so jazzed about software.

• 27:00Jacob Effron?

  No, I love it. You said something there that I wanted to come back to, because it was just a great phase. I think you said that the tyranny of the KV cache in coding agents. Maybe explain for our listeners a little bit about like some of the limitations in coding agents, you know, due to the KV cache and like how you think about that problem space.

• 27:14Speaker 01

  I would say almost all design decisions around the UX of coding agents, like the stuff that actually functionally changes stuff, is built around the KV cache. And this is natural, because like the biggest wins in ML are always like very hardware aware. This is how things have gotten so efficient recently, or pseudo recently, since GPT-4 days. GPT-4 is very expensive. Like, you just need, really need to design around this, which is great, because it's efficient, but it's also quite limited. It, this is why like routing doesn't tend to work. Routing absolutely works if you're making the equivalent of like, what we call instruction following calls. Like, you know, task in, task out, you know, fuck the cache, throw it away. Routing is amazing for that.

• 27:59Speaker 01

  Actually, you can do way more advanced things than just routing in that case. There's so many advanced patterns you can do, when that is your assumption. Like, you could actually try it with a cheap model, use an extremely cheap judge. And if it's efficient, keep it. If not, try a larger model or, you know, try multiple times and pick the best. Like, there's so many things you can do. But when you have like this stateful KV cache, you are like really constrained to it being kind of append only. And you have this complicated trade off, kind of like an iPython notebook, where you can never delete the memory, because it's like a really weird, you know, agglomerated state. And it turns out that it's very hard to engineer around this, because the models are trained with it. So if you actually start doing fancy stateful operations with a model having free reign over that state,

• 28:42Speaker 01

  they'll end up looping and stuff like that, because they're so tuned to having that KV cache and so efficient to have that KV cache. So this prevents you from doing like routing effectively. It prevents subagents from being really effective, because now you have this weird trade off of like, what state, what context do I pass to my subagent? Because this requires intelligence in order to know what to pass in. If you pass it all in, this is a huge input token cost instead of a cache read cost. And when your subagent makes an output, how do you know what to pass in? And like what subset is relevant down the line? And the safest thing to do tends to be to append a lot of stuff. And now this is the equivalent of like, from a state management perspective, imagine your state is like global variables. It's passing one giant dictionary around forever until the models break

• 29:25Speaker 01

  and they start getting dumber over time, because context brought is a real thing. And this is totally cool, just to be clear. I don't believe that there's like anything super tragic about this on its own, because it's very useful. I think what's tragic is that's the really only, the only way to do it. And I think there's so many more advanced patterns that you could do if you could assume that I could just put intelligence all over these operations. And that is the kind of thing to me that I think from a disengineering, architecture, design, beauty, power perspective, is so limiting. And that is what that tyranny is, because the people who have tried to do those cool algorithms have mostly failed, because they've tried to use LLMs for it.

• 30:05Speaker 01

  And LLMs are not optimized for, you know, routing logic within the software or making discreet little decisions. They're optimized for making human pleasing outputs that are not super reliable, running as a sub process. And this then makes people believe this doesn't work. And actually, a lot of what our original shape looks like is a, it's not too dissimilar from LangChain. Like, there's nothing wrong with LangChain's interface, right? Like, someone even say that it was like maybe perfect for classifiers on top of LLMs. I think, I can't remember back in the day, but I'm pretty darn sure it's going to be pretty darn good if someone made like a, like a LangChain compatible wrapper on top of our model.

• 30:46Speaker 01

  The problem is, people didn't like it, because it made models somewhat dumber, because the models are not optimized for that type of pattern. And this is also a form of tyranny, where everyone just thinks Anthropic and OpenAI or whoever else is like number three right now, is like doing, doing their best and giving us the best intelligence possible, when the reality is there's so many other intelligences possible. And like, there, there's trade offs there. You know, like there's truly a lot of trade offs that are made in this design that people can feel, but cannot really measure, because it's really hard to do so.

• 31:21Jacob Effron?

  Let's take an example of the shape of what a coding agent could look like, I guess, with like a different intelligence design.

• 31:27Speaker 01

  One feature that I think would be really sick is what if you could have like multiple agents sharing the same context? And you're like truly like read writing from the same context. That sounds really complicated, like there's no way that works right now because of the KV cache. But like, nothing stops that from happening. And actually, a very common pattern people have with agents these days is like, they're just cooking and crazy stuff. They don't know what's going on. And after it does the thing you asked it to do, people are like, make a visualizer or a dashboard or a learning tool. And it's kind of crazy that that has to be like an extension of the context that has all of the trial and error in it. Or starting from scratch, looking at a Git commit. Why can't and rereading everything from scratch? Why can't it share the state?

• 32:09Speaker 01

  And why can't you have like an agent here that's actually basically read only for the code base? But allows you to write the visualizer such that as the agent is cooking, you actually get live visualizations of maybe, you know, using like a slightly dumber model, like Sonnet instead of Opus, to write out what Opus is doing. So you kind of get a sense of what's going on instead of having to read like the, you know, the gibbeting updates or whatever else. I think that would be cool. Another example here is compaction. Compaction is makes a lot of sense for a write once KV cache that is append only. Cause like, oh, it's full. We need to restart it. But compaction is a form of compression and compression is just not totally solvable.

• 32:53Speaker 01

  Like, it's very hard to compress without priors for compression. But what happens if you compress based on what problem you were solving before? And now you want to actually ask a problem that is different from that one? And it would use the information that was compressed away. Like, like, like that agent is like Delta, very much losing hand. But what if how the state gets compressed was actually semantic? And it could detect, hey, I'm doing a new task. I should actually think about what the context should be for this new task using an ROI calculation, which should be in code. You know, the code should compute, you know, expected value of like cache reads and cache writes and everything. And recalculate perhaps a new KV cache for this other task and maybe even keep the other one hot. I don't really know. This sounds like cool sci fi. One more, one more, then I will stop on this topic.

• 33:38Speaker 01

  But I, I just so excited about this. One thing that I think would be really cool is, hey, I keep cooking on the same code base. And I keep reading the same files and having to re-understand my system's abstractions over and over and over again. You know, there's a partial solution to this, which is like, maybe the agent should write markdown files. But then how do you keep it in sync, blah, blah, blah? You know, it's a kind of a pain, you know? But what if you could look at the history of old models, but like, not as like a, like, I feel like right now it's like a little bit of a hack, like, oh, it's a tool that you can use sometimes if the model deems it as necessary. But what if like, by its nature, one of the first steps you do is even a extremely cheap map reduce over all history to find the relevant stuff here?

• 34:21Speaker 01

  And what if that history could also be shared with your coworkers who are using the same code base? So you can actually have insights from what they're working on. Like, these are very doable software problems, right? The, the problem is that it's like very unfriendly with a KV cache when all the intelligence needs to be append only. And there's, there's so many other cool things.

• 34:40Jacob Effron?

  I love it. The creative juices flowing. I, I guess, you know, another thing I'd be curious about is, I feel like you, you all obviously have released some, some, you know, stuff around images, right? And like, and you know, you know, obviously multimodal, I imagine the focus. I think most of our listeners probably think of Jev in the context of like, you know, uh, these, these, uh, decision models or type one things. And, and kind of in classic like LLM replacement. Maybe talk a little bit about like the multimodal world, like how you see the shape of problems there, how like Jev models fit into those.

• 35:11Speaker 01

  So there's a couple of like branches to go off of here. I will say, branch one is that being aware of the reality of AI is so unbelievably important. And by its nature, just like long context tends to degrade intelligence, no matter the model. Multi-modality tends to degrade intelligence too. And then this is like well-known fact by like anyone in ML circles. And actually, every modality has different rates at which it is degraded. And this is very measurable stuff, except when you put them into chatbots, because chatbots are optimized free in preference. It becomes a very, very fuzzy thing on whether or not it's better or not. It's very measurable that multimodal models will be less reliable than the text models. And my philosophy has always been automate the easy work before the hard work.

• 35:55Speaker 01

  Like, will Jev models be used to automate software engineering? Maybe one day, but software engineering, maybe I'm arrogant, seems a lot harder than like really easy tasks that LMS fail at. And in order to like be calibrated to the world, we should study why they fail at those easy tasks. And like really honestly, like go in-depth into studying them, not to cheat them, which I think is like quite common. But like actually like dig in and figure out what is missing and why. And sometimes it's not the model, sometimes it's the prompt, sometimes it's every other stuff. You know, like it's a, it's a collaboration between software and model. But it's through that that I think we like learn the truth. I have like stories about how we got to even be anti-benchmaxing more.

• 36:35Speaker 01

  But this is a thing that LMS try to avoid because they will look bad on benchmarks, as far as I can tell. And so that is one aspect of multimodality. We have to take into account text tasks will be automated before the image tasks. Thing two is we invested really heavily on text tasks and very much like, we've done like a tiny amount of work in comparison to the image tasks. Like, I am like kind of jazzed that our model does as well as it does. I, I think that that's great. Like, as far as I can tell, our lead in images is not as large as our lead in text, but we do have an overall lead in images. Is my vibe, could be totally wrong, but that, that's kind of what I've sensed.

• 37:16Speaker 01

  And I would actually describe the image model as completely insufficient. Like, I would be truly embarrassed to say that this is prod-ready because there's so much jaggedness left in it. That I'm, I'm jazzed if people find stuff to actually deploy. But I will say that this does not have like my seal of approval yet. We need to do so many rounds. Because like, it did like, like, it's, it's not quite as bad as like other people just shipping a shape of something. But it's, it's like, in the grand scheme of it, like image, our image model is like maybe 10% of the way that our like text model is.

• 37:51Jacob Effron?

  As you move from like 10% by your characterization to like 90, 95%, what are the first kinds of things that you're excited about Jev models doing on the image side?

• 38:00Speaker 01

  On Image Land, I have a soft spot for video games and computer use. Video games, because I have like a video game background. I, and, oh man, I, I, confession, I've also like played around with botting them, not for competitive reasons, but just because it's cool to automate stuff. And I think like being able to play around with that and like, I, I feel like software is a little bit like warping technology to your will. And being able to like expand what you can warp to your will is just very cool in general. So I think that is really cool in video games. Computer use, I think, is going to be super duper valuable, unbelievably valuable. I, I think it's, it's like, especially combined with like lots of other stuff.

• 38:42Speaker 01

  I think that AI should be so much better for interfaces. It's actually kind of confusing to me how it is so bad. You know, like, why, there's a lot of people who don't like using computers and they still have to, even if they're like old and not experts. Despite that, despite AI being probably better than an assistant that they could hire to do a lot of computer stuff for them. And I would love for that to be reliable enough that people could like trust it as an interface. And, you know, I, I guess an additional thing that is like, I have a soft spot for, and this is why I'm wearing my like unslopable shirt, is I would love ways to just stop slop. For me personally, it might be bad for the internet, like, like the internet is funded by ads. So like, maybe it's a good thing, I don't know.

• 39:27Speaker 01

  But I would love if I could have some, or someone would make me like a slop stopper where I just don't need to see it anymore. That'd be lovely.

• 39:36Jacob Effron?

  But one thing I've been curious about from the outside is like the extent to which, you know, uh, the advances that you had at Jev were enabled by like all the advances we've seen in, in, you know, AI in the last two, three years? Or could you have just done this five years ago? Like, obviously you have a different approach that, that is like in some ways orthogonal to the rest of the field. Like, couldn't you have done this five years ago? Or how much of what you built like does leverage some of the stuff that's been done, you know, on the LM side?

• 40:01Speaker 01

  Five years ago, it would have been harder, but doable.

• 40:07Speaker 01

  Back when I was more naive, oh man, I sound like an old dude. When I was more naive and I thought that this would take a week of work. What convinced me, I love writing documents. I love, like, I think it's a great way to think. I wrote a document at OpenAI that was titled, "We could have had AGI a year ago", at least by OpenAI's definition of AGI, which I don't think that people really listen to the letter of the definition. But I think it's actually a really great letter. I, I think that models from the, starting from GPT 3.5 era, possibly even GPT 3 era, could have met the definition of AGI at the time, which is still the same, like doing most of the world's economically valuable work by volume. It doesn't say by volume, but I assume the spirit of that is by volume.

• 40:53Speaker 01

  Because the models are so unbelievably smart. Like, GPT 3.5 level models surpassed median level intelligence at all sorts of things. And like, shouldn't that be used for tons of automation, at least for the easy work? And so, yes, I think that something like this would have been harder, but it could have been done five years ago. Absolutely. And, you know, like on the people who are like obsessed with like, for example, like the, the benchmarks of the day. And they see like what tasks AI is doing. These tests are like really hard. They're like really, really, really hard. GPQA was saturated 2020.

• 41:31Speaker 01

  Three-ish, maybe not totally saturated, but it was like surpassing PhD student level at 2023, I believe. Google-proof question answering for like PhD-level intelligence at these extremely hard multiple-choice tasks. It was passing that level 2023. Like, how is it that that level of intelligence was not being put to work? That is super tragic. So I think we've had a lot of like raw material for intelligence for a very long time, and we're gonna see a whole lot of craziness.

• 42:01Jacob Effron?

  Yeah. I mean, speaking of that craziness, I think part of, I mentioned there's a lot of noise, you know, I feel there's been a lot of noise in the ecosystem, and I bet many of our listeners will be familiar with, you know, after you guys came with these, these models, you had OpenAI and CloudFlare and Microsoft, and everyone kind of released these, these models. I, I know you'll, you'll say that the benchmarks aren't, aren't a good way to test them. They obviously, you know, I've, I've tried to show that, that they're near equivalent. I guess, how do you think about, like, you obviously have now invited a bunch of competition into the space. How do you think about like what, you know, keeps Jev? You know, I guess everyone likes to ask about moats, so I guess I'll ask it on, on behalf of our listeners. Like, how do you think about that going forward for the business?

• 42:38Speaker 01

  Oh, boy. It feels like other people are digging our moat for us, which is kind of not how I thought it would go, and I think that that's kind of bad for the field. I think that releasing models that are like kind of more benchmark-optimized than real-world-optimized actually burns that goodwill for people to try things out, which is the only thing that matters. I, I truly think that this is not great for the field, and my hope is that just giving unbelievable fucks about reliability is going to be the North Star. I think that will be immensely, immensely valuable for the world. There's just so much more to do there that I think that it's a little bit early to worry

• 43:23Speaker 01

  about competition. It, it's kind of like OpenAI worrying about a moat at, at GPT-3.5. Like, who cares? Like, GPT-3.5's market is nothing, or it was nothing, right, in comparison? It's still, I think, going to be nothing in the grand scheme of things.

• 43:39Jacob Effron?

  Yeah.

• 43:40Speaker 01

  Like, we truly believe in Jevon's paradox, obviously, hence the name Jev. Like, Jevon's paradox, like, are we seeing rebound? You know, more demand because it's so cheap? Absolutely. I believe every good sees rebound. Are we seeing backfire? Like, the total market size increases? This is, this is when Jevon's paradox holds? Absolutely not. Like, it, it, like, like, a key part of it has to be, like, wow, this is so good and so cheap, I'm gonna be throwing it all over the place because it makes value in all these places, and my budget for AI will actually increase.

• 44:11Jacob Effron?

  I mean, speaking of investors, I guess you, you, you obviously recently raised a, a pretty big round. How did you even think about the amount of capital that made sense to raise for, for this business? I'm sure people were trying to throw money at you, so.

• 44:23Speaker 01

  Yes.

• 44:24Jacob Effron?

  You know, how, how did you even, you know, think through that?

• 44:26Speaker 01

  I will say that people probably don't believe it, but like, we can run our service profitably, and I hope, I hope that we never have to do the thing of like temporarily subsidizing to get lock-in. I, I don't want to like ax it out entirely, because like I'm a pragmatic person, but like I really hope that doesn't have to be the case. I hope that, I, I, I find that to be like a very shadowy, sneaky way to get people to use your service. So I, and I hope to be the kind of person who's like, you know, always collaborating in like these prisoner's dilemmas until, you know, until forced otherwise. So we actually did not need the money.

• 45:09Speaker 01

  And our end goal instead was, hey, we want actually like the most valuable people on our side to help us build this business, because we have no fucking clue how to do this. So we really opted for like, who are the best people in AI who will help us? You can see who we got on our board.

• 45:26Jacob Effron?

  Well, I always like to end with a quick-fire round where I just stuff some overly broad questions into the end that I haven't had a, a chance to ask elsewhere. I'm curious, you know, over the last month, you know, or last six weeks of having this product out there, like, what's one thing you've changed your mind on in AI in the last like six weeks, having just seen how?

• 45:43Speaker 01

  Whoa, that's such a cool question. Whoa. Whoa. That's a quick-fire question? Holy smokes. I will say that we were really worried if our philosophy about caring about like the actual reliability of intelligence was a mistake. Like, we had people in the team say, like, hey, should we just release a benchmark version and just hype it up and like do one of those early releases that like obviously is not great, but just to like test out if the market exists? I'm so glad we didn't do that. And I, I say seven weeks because like I, I don't remember when launch was, but like seven weeks I think was before launch. We really did not know this. We, it was like a big leap of faith for us, and like we were just really, man, like, I

• 46:27Speaker 01

  don't think we could be like proud of ourselves if we did the same over-promise, under-deliver. And like, I think that that paid off super-duper well. I, actually, I don't know if it paid off. Maybe the product is sold itself, and maybe it would have done even better if we like really optimized for, for attention. But that was like awesome, and I actually feel like now that my brand has been solidified as our brand, I hopefully never have to do that, like even close to that again, and that debate no longer exists for us. I hope. I hope, I don't know, I truly, truly hope, because I have like a hard time, like even giving like unnuanced, slightly incorrect answers, because I think that that's just a disservice to anyone listening to me.

• 47:12Speaker 01

  I would rather just like not answer the question than do the standard like CEO compression of the truth that airs on the side of generous. So I hope that I will never really have to do that. But like, something I've learned as well is that this reliability saga was sped-run so fast for our class of models. I feel like, I, I could be wrong, but my temperature check of the environment now is people have already gotten like extreme fatigue of like the benchmarks and not actually useful stuff way faster than I would have thought. Like, I, I would have like thought that this would be a concern, based on what I see, because people keep falling for benchmarks every time a new model's released, right?

• 47:57Speaker 01

  People are like, whoa, cool, let's talk about it. And then it, it disappoints, except when it's like an OpenAI or Anthropic model, apparently. And that just happened, I, I could be wrong, and like maybe we'll, we'll see. I think it just happened a lot faster than I would have thought, because our model is so benchmarkable, such that people could like blast out so many benchmark-crushing things so fast.

• 48:18Jacob Effron?

  I feel like this next one is gonna be very anticlimactic in comparison, but I am, as a podcast host, I'm contractually obligated to ask you, you know, like, we're, we're an AI podcast. The other big AI story right now is obviously Anthropic, you know, going public and folks kind of thinking through that business. And I'm struck even in the course of this conversation, like, I actually don't know how you're gonna answer this question, because on the one hand, I feel like you're very disappointed in like so many of the, the lack of capabilities that LLMs have, and maybe that they're like almost a hammer trying to like, you know, go on every nail of, of like different sorts of use cases. On the other hand, like you have been complimentary of like, there are things that coding agents are really good for. And so I guess as you think about Anthropic as like, you know, one and a half, two-trillion-dollar company, like, is that something you're optimistic on? Is like, hey, that's, that's undervalued, or something you're like, ooh, I don't know what's gonna happen to that?

• 49:03Speaker 01

  I think coding agents are massively valuable. It's unclear to me how much of the actual use is truly valuable, like it being like net positive is different than it being worth all of the money put in. That is probably a product question for them to figure out. If they actually are truth-seeking, they probably have a better answer to this. Like, did some people like just token max too hard? Or are they actually getting the relative value that spending that would make a difference on? So I don't think I'm informed enough for that. I have my guesses on what percentage gain coding agents are for various levels of projects and skills of engineer. And I wouldn't say that software is anywhere close to solved, but that doesn't mean that they're not valuable.

• 49:47Speaker 01

  I just think that there needs to be like a pragmatic take here. I think that it's possible that the coding, I could see the coding agent market going either way. Like, I could imagine it going very up and very down, very up in the sense that as people learn to actually get more value, can people use it more? I believe so. I could imagine it going down because a lot of people have like overdone it, because like there's so much FOMO in the field. And actually, my sense of the state of AI is not that it's been like an upward exponential, but a lot of like hills that keep going up, down, up, down, up, down, and then eventually having like a little bit more at the tro, trough, trough of disillusionment? I actually don't know how that word's pronounced.

• 50:25Jacob Effron?

  The trough, yeah.

• 50:26Speaker 01

  Yeah. And then eventually it becomes like more stable usage, and I think that, you know, like the, the flatlining revenue might be indicative of that people are like realizing that maybe they spent too much. I think, so from a product perspective, I think that there will be better experts than me at that. I actually get the sense that the greatest strength of the labs is not models or product or distribution, but financial engineering. This is the thing that everyone who's like very veteran at the labs always talks about, where my understanding is they structure these deals in incredibly beneficial ways to not fuck themselves over, but fuck other people over who are taking the risk and trying

• 51:09Speaker 01

  to like latch on to the rocket ship and are willing to like accept whatever terms that would.

• 51:15Jacob Effron?

  On the compute side?

• 51:17Speaker 01

  Yes, on all sorts of sides, but especially compute. I think that financial engineering, as far as I've heard, is their greatest strength. And I have no idea how much of a rat's nest that is, and how vulnerable they would be to any which direction of the market going. You know, like, a bubble can pop just because expectations are not met, not because it's not truly a transformative technology. So I'm a little, I try not to, I try to be calibrated in my predictions. I try not to be overly confident when I have no privy information, and I, I would, I'm, I'm a more conservative person.

• 51:59Speaker 01

  I, I get the sense that a lot of the expectations baked in are ones that are not just product-level, but based on like misunderstandings of the future of AI and what it can do. So my guess is that because some people might be valuing like building God is basically infinite, there will be some amount of miscalibration there. And I think that, you know, like the let them cure cancer is going to be like Marie, Marie Antoinette's like, let them eat cake.

• 52:28Jacob Effron?

  That is a, that is a good line. Yeah, I mean, obviously.

• 52:31Speaker 01

  Oh, maybe I, maybe it did end spicy then.

• 52:33Jacob Effron?

  That, that, that was, that was a great line. Um, I feel like the, yeah, I mean, obviously so many people are underwriting, you know, the move into from coding agents to general knowledge work to bio to these other spaces. And obviously like not the perfect use cases for RLVR by any means.

• 52:46Speaker 01

  I still get the sense that the, like, the nuanced way to understand agents perhaps, and maybe all of AI right now, is what decisions can you trust them with? And in terms of like stakes for the company and like, can you like, can you get like value without too much of the downside? And coding is like quite unique in this, or like math is actually like, like super duper unique, where like there's no downside other than the compute you spend, right? Like math models do not cost companies anything, except if they're like paying people to validate their proofs, which it's unclear if they even pay them for that.

• 53:25Jacob Effron?

  Yeah.

• 53:25Speaker 01

  To me, maybe someone knows. But code also has like somewhat more kept downside because you have like a get, right? And you have humans in the loop and like there's ways to protect you from unwanted side effects, which is the main downside of powerful tools. And powerful tools are necessary to do knowledge work because knowledge work often involves like side effects in the world. And talking a lot like I give you a science right now. My sense is that for a lot of other fields, they don't have anywhere close to the same levels of like safeguards and ability to revert and like tracking and all of that. And that is why, to my knowledge, no one has automated like a drive through yet, even though people have tried really, really hard.

• 54:07Speaker 01

  And the stakes of a drive through are a lot lower than the stakes of many other forms of knowledge work. So I think that the, the truth will be like, there'll be like nuance and skill into it. And the people who will be the big winners are not the ones who just YOLO all in, but like understand reality and like deploy them, um, you know, in, in the, in, in line with the realities of technology. So I am, I, and I'm not, given my state of AI, I feel like most people are very FOMO driven and they do not have a great understanding of the state of AI. And they will probably err on the over optimistic, which will get them burned with many side effects,

• 54:47Speaker 01

  or like the over pessimistic, which is business as usual, which is how most businesses have been for this AI boom.

• 54:53Jacob Effron?

  Well, and one thing you said there, like, you know, you, I'm curious, you said you have an estimate for how much these coding agents actually improve engineering productivity. What, what is that estimate?

• 55:04Speaker 01

  Okay, I, I will say I have like error bars on this because I think that there's like long-term effects, like, you know, skill degradation and tech debt and stuff like this. So my estimates are in the 20 to 40% speed up range. But again, it really depends on the use case. And it also depends on like the skill of the user and if you can like, you know, because the side effects are the bad thing, if you can isolate the side effects in a way that those are very legible. So depending on the use case, for some, it's kind of like an infinite percent speed up. You know, you would do things you would never, you can do things you can never do otherwise. For others, they are like just so bad that they're as bad as like coding the thing by hand. I would guess on average, it's like a decent but small boost.

• 55:49Speaker 01

  And I, I believe that is the consensus that I've seen in the field right now.

• 55:54Jacob Effron?

  Yeah. Another question, I'm not totally sure how you, how you answer, like, um, how can, you know, there's obviously so much discourse around AI safety these days. Like, how concerned are you about, uh, a lot of these, you know, this AI safety conversation?

• 56:06Speaker 01

  I think AI safety is not one monolith and there's like a ton of different slices of it. I think that the ASI discussion is actually extremely different from the bio risk discussion, which is very different from like the cyber risk discussion, which is different than like trust and, trust and safety.

• 56:25Jacob Effron?

  Maybe like ASI versus like bad actors, you know, using bio and then cyber and whatnot.

• 56:30Speaker 01

  I get the sense that most people concerned with ASI are a little bit divorced from the reality of the technology. And this has been the case since day one. You know, like you, I, I feel like you need to like actually build a reliable system with AI to have a voice in the room that something is going to be like so reliably, um, maybe not evil, but misaligned that it can destroy the world. But I, I think we're pretty far from stuff like that. Additionally, there's actually a bunch of nuance towards like, you know, ASI, RSI, and like whether or not the models have a sense of agency on their own. Like paperclip maximizer is actually very different than bad actor programs,

• 57:13Speaker 01

  super employee to destroy the world type things. And like, actually, I think that those end up becoming quite different in threat model. I think that we're breakthroughs away from true agency. As much as people like to like read these chains of thought and like try to ascribe real agency to it. I, I would guess that we are something like two breakthroughs of like gigantic magnitude away from this. I, I think even one breakthrough would not be sufficient is my guess, but maybe one gigantic one could be. And I think the threat model of like humans with super tools who want to do damage is very different than the, the ASI RSI thing. Uh, I don't, uh, I think a lot of people who also believe in RSI,

• 57:57Speaker 01

  both have not studied the history of AI very much and also like have no appreciation of Andal's law. Where like, like, you know, you know, like if you assume that intelligence is the, is the spectrum and it's the only one that matters and you have to map things into that spectrum and you can also like map humans and chimpanzees, horrible argument in my opinion. Then it does look bad, but the reality is that it doesn't work that way. And like the nuance of like the many different types of intelligence and how to do them reliably really matters a lot in this whole analogy. Could the field of AI development become a little bit faster? Yes, but people have been trying this for a long time and it usually doesn't work out.

• 58:36Speaker 01

  And it, I don't think that people would count like a 10% speed up on AI research as RSI, if it doesn't actually increase over time. So I, I, I'm very open to being convinced by evidence, but I, I think that it's more of like a philosophical problem rather than a technological one. The kinds of people who like read, uh, like a Bostrum's Super Intelligence and treat it like reality over like a very interesting sci-fi book makes, makes me kind of worried.

• 59:09Jacob Effron?

  Yeah. Um, well, this has been a fascinating conversation. I feel like you covered a ton of things. I want to make sure to leave the last word to you, like anything you want to point our listeners to. I mean, I'm sure they're all familiar with, uh, and certainly by the end of this podcast, certainly hope they are. But like anything we haven't discussed or I don't know, the floor is yours.

• 59:24Speaker 01

  There, there's so much stuff to have happened in software plus AI. I think it'll be so fucking incredible. I think that we need to start untraining ourselves of this lack of agency that LLMs have kind of taught us about AI and start really building again. Like, like there's, you, you, you know, like there's so many possibilities that are going to be opened up and we should get back to playing rather than being like these victims of stuff. And I will do whatever we can to make, you know, playable AI. But it's, I, I really want, oh man, I have like so much to say here. I, I feel like I, I will be doing whatever I can for the field

• 1:00:09Speaker 01

  to like have this opportunity to just rethink software, rethink interfaces, rethink everything. But it's going to require like real builders to figure out how to do that. And I think that together we can make AI this really awesome force for good in creation and just all around awesomeness, you know, like, and, and sometimes even like really weird, wacky stuff. I, I feel like that, that, that's where like technology really shines.

• 1:00:33Jacob Effron?

  I'm Jacob Effron and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project in addition to my day job as an investor at Redpoint. But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so please consider doing that and thank you so much for your support and listening. We'll see you next episode.