Eiso Kant (CTO poolside) - Superhuman Coding Is Coming!

Eiso Kant, CTO of poolside AI, discusses the company's approach to building frontier AI foundation models, particularly focused on software development. Their unique strategy is reinforcement learning from code execution feedback which is an important axis for scaling AI capabilities beyond just increasing model size or data volume. Kant predicts human-level AI in knowledge work could be achieved within 18-36 months, outlining poolside's vision to dramatically increase software development productivity and accessibility. SPONSOR MESSAGES: *** Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich. Goto https://tufalabs.ai/ *** Eiso Kant: https://x.com/eisokant https://poolside.ai/ TRANSCRIPT: https://www.dropbox.com/scl/fi/szepl6taqziyqie9wgmk9/poolside.pdf?rlkey=iqar7dcwshyrpeoz0xa76k422&dl=0 TOC: 1. Foundation Models and AI Strategy [00:00:00] 1.1 Foundation Models and Timeline Predictions for AI Development [00:02:55] 1.2 Poolside AI's Corporate History and Strategic Vision [00:06:48] 1.3 Foundation Models vs Enterprise Customization Trade-offs 2. Reinforcement Learning and Model Economics [00:15:42] 2.1 Reinforcement Learning and Code Execution Feedback Approaches [00:22:06] 2.2 Model Economics and Experimental Optimization 3. Enterprise AI Implementation [00:25:20] 3.1 Poolside's Enterprise Deployment Strategy and Infrastructure [00:26:00] 3.2 Enterprise-First Business Model and Market Focus [00:27:05] 3.3 Foundation Models and AGI Development Approach [00:29:24] 3.4 DeepSeek Case Study and Infrastructure Requirements 4. LLM Architecture and Performance [00:30:15] 4.1 Distributed Training and Hardware Architecture Optimization [00:33:01] 4.2 Model Scaling Strategies and Chinchilla Optimality Trade-offs [00:36:04] 4.3 Emergent Reasoning and Model Architecture Comparisons [00:43:26] 4.4 Balancing Creativity and Determinism in AI Models [00:50:01] 4.5 AI-Assisted Software Development Evolution 5. AI Systems Engineering and Scalability [00:58:31] 5.1 Enterprise AI Productivity and Implementation Challenges [00:58:40] 5.2 Low-Code Solutions and Enterprise Hiring Trends [01:01:25] 5.3 Distributed Systems and Engineering Complexity [01:01:50] 5.4 GenAI Architecture and Scalability Patterns [01:01:55] 5.5 Scaling Limitations and Architectural Patterns in AI Code Generation 6. AI Safety and Future Capabilities [01:06:23] 6.1 Semantic Understanding and Language Model Reasoning Approaches [01:12:42] 6.2 Model Interpretability and Safety Considerations in AI Systems [01:16:27] 6.3 AI vs Human Capabilities in Software Development [01:33:45] 6.4 Enterprise Deployment and Security Architecture CORE REFS (see shownotes for URLs/more refs): [00:15:45] Research demonstrating how training on model-generated content leads to distribution collapse in AI models, Ilia Shumailov et al. (Key finding on synthetic data risk) [00:20:05] Foundational paper introducing Word2Vec for computing word vector representations, Tomas Mikolov et al. (Seminal NLP technique) [00:22:15] OpenAI O3 model's breakthrough performance on ARC Prize Challenge, OpenAI (Significant AI reasoning benchmark achievement) [00:22:40] Seminal paper proposing a formal definition of intelligence as skill-acquisition efficiency, François Chollet (Influential AI definition/philosophy) [00:30:30] Technical documentation of DeepSeek's V3 model architecture and capabilities, DeepSeek AI (Details on a major new model) [00:34:30] Foundational paper establishing optimal scaling laws for LLM training, Jordan Hoffmann et al. (Key paper on LLM scaling) [00:45:45] Seminal essay arguing that scaling computation consistently trumps human-engineered solutions in AI, Richard S. Sutton (Influential "Bitter Lesson" perspective)

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  You will never hear me argue against scale. The narrative at the time in the world was we're just going to scale up, you know, the next GPT model, make it 10x larger. This was such a strong narrative that frankly, many people at the time in the world, you know, believed this. We didn't.

• 0:17Speaker 00

  There was a missing axis of scaling that wasn't being discussed. And it's frankly, why we started this company. And it was the axis of scaling for the use of reinforcement learning. Scaling of next token prediction is the equivalent of imitation learning. Scaling of reinforcement learning is the equivalent of trial and error learning. The reason we built this company is because we saw a future that I personally think now is 18 to 36 months away, where human level intelligence across the vast majority of knowledge work is achieved. You don't get to do that unless you build from the ground up. You don't fine tune your way to AGI.

• 0:49Tim Scarfe?

  I've just had the most amazing conversation with Izo Kant. He is the co-founder and CTO of Poolside AI. Now they are building frontier language models. They're one of about seven or eight companies in the world who have the technical chops to build foundation models from scratch. They have a really cool solution for doing generative AI coding. Honestly, it's now possible to write software about 10 times faster than we did before. Now, what these guys have done is they use reinforcement learning from code execution feedback, which means they're going one step further in the stack to align the language models that they build to the code and the software that you are writing. There's so much low hanging fruit in this space at the moment. When are we going to have code solutions that can watch your screen?

• 1:34Tim Scarfe?

  That are multimodal, that help you collaborate better with developers? Izo has a very interesting story to tell about all of this. We also had some pretty cool galaxy brain conversations about how to train foundation models, about test time computation and thinking and reasoning. I think there's a lot for, you know, quite a few people to get their teeth into in this conversation. MLST is sponsored by Tufer AI Labs. Now they are the DeepSeek based in Switzerland. They have an amazing team. You've seen many of the folks on the team. They acquired Minds AI, of course. They did a lot of great work on Arc. They're now working on O1 star models and reasoning and thinking and test. Time computation. The reason you want to work for them is you get loads of autonomy. You get visibility. You can publish your research. And also they are hiring as well as ML engineers.

• 2:19Tim Scarfe?

  They're hiring a chief scientist. They really, really want to find the best possible person for this role. And they're prepared to pay top dollar as a joining bonus. So if you're interested in working for them as an ML engineer or their chief scientist, get in touch with Benjamin Cruzia. Go to two for labs dot AI and see what happens. Izo, it's an honour to have you on MLST. Thank you so much for joining us today.

• 2:42Speaker 00

  No, thank you so much for having me. I appreciate it.

• 2:44Tim Scarfe?

  Can you tell us a little bit about yourself and Poolside?

• 2:47Speaker 00

  Personally, I'm a computer geek. I started programming when I was quite young. In 2016, found myself building what I believe to be the world's first company that are focused on making AI capable of writing code. So I met my co-founder as well, actually. It's a longer story. And in April 23, we founded Poolside. And Poolside was really founded on our view that the world was going to achieve human level intelligence in AI. And we took our own point of view on how to get there. And that's been the fundamental start of us since now almost two years.

• 3:23Tim Scarfe?

  And your co-founder is Jason.

• 3:24Speaker 00

  Correct, yeah. Jason and I met because in 2017, he was the CTO of GitHub. And I'm sure we've ever said this publicly, or at least not on camera, is he actually made an acquisition offer for that company I was building called Sourced. And we back in the day had the world's first models. They were able to do code completion and things like this working. And I turned down the acquisition offer, but nonetheless, we became really good friends.

• 3:48Tim Scarfe?

  So tell us a little bit more about Poolside. What's the main goal?

• 3:51Speaker 00

  Poolside's main goal comes down to wanting to build a world where we have human level intelligence that we can scale up on compute. We think it is essentially going to have two ways of having impact. One is the more that we can make capable intelligence scalable, we can start driving the cost of goods and services down to zero. On the other hand, there's this entire frontier of technology and scientific progress that's ahead of us. And by definition, it's infinite, right? We will always continue to find more. And so being able to pull that closer in time has always been our mission. But we took a slightly different path than others. We took the path on focusing on making AI incredibly capable of building software.

• 4:33Speaker 00

  And we laid out this three-step plan kind of on our website on day zero. It's still there in the bottom of the footer if you click on vision. And it said, "Make AI capable to assist developers in building software." Step one. Step two, allow anyone in the world to build software. And step three, generalize it to all other fields and domains.

• 4:51Tim Scarfe?

  So there's a kind of winner-takes-all dynamics in the space at the moment. There are amazing frontier models out there. I mean, I've been playing with, you know, Sonnet 3.7 thinking and whatnot. And there's always the question of differentiation. I mean, Anthropic, they released this Claude C L I thing. And you can just stick it on your repo and my God, it's really, really good. So how can you differentiate on top of that?

• 5:16Speaker 00

  So I think all of us at the frontier are constantly competing with each other for model capabilities. And I think in the fullness of time on capabilities like software development, we might all end up even in the same place. But if you look at right now where the world is at, it's a very small number of companies that are actually competing there. I would say that we have kind of the old guard like Google. We have kind of the first generation of AI companies, OpenAI and Anthropic. And you kind of have the second generation, XAI, Poolside, Mistral. We all were founded around this, you know, April, May 23 date. And all of us, I think, are in that same race. Now, we've decided to not focus on making our models

• 6:00Speaker 00

  generally available for every possible use case, but really make them available for software development. And this allows us a set of liberties in terms of where we focus on and areas we don't focus on. But don't be mistaken, like the work that we do to build really capable foundation models still lends itself to building really, really capable models across the board. Because software development is not about writing code. You need to understand the world. You need to be able to do multi-step complex reasoning. You need to be able to plan across long objectives. And so I was super excited what Anthropic did with Claude 3.7. I think it's an amazing model, the Sonnet model. And of course, it's for us to make sure that we don't surpass that. And so we're constantly in this race with each other. But we take certain views on our research and our approaches

• 6:44Speaker 00

  that I think will allow us to accelerate over time faster towards those goals.

• 6:48Tim Scarfe?

  So I'm trying to understand this because there seems to be a dichotomy between having really general foundation models that can do lots and lots of things versus the story from many folks in the space who are saying, "We need customization, we need personalization, we need on-prem deployments and so on." How do those two worlds come together?

• 7:05Speaker 00

  So I think it's a spectrum. I think absolutely in the first part of your training of your models, you want to embed as much diversity and knowledge of the world as possible. Software development is not about writing code. It's about being able to interact with the real world and turn that into a digital form. So having that general part down is incredibly critical. I might not care as much about how humorous my model is and how good it can handle nuances of comedy, but I absolutely care about the knowledge that it has in many different domains. What ends up happening, though, is that all of us have a fixed parameter space. At the end of the day, there's a cost of inference, so there's only so many parameters that I can load up and then actually run. And by having a fixed parameter space, it means that you have to choose what you want to do with those parameters.

• 7:50Speaker 00

  And so we try to shift the distribution of our model capabilities very much towards software development in terms of the capabilities that they have, but also meaning that we're willing to trade off, you know, not maybe being as good as, you know, writing a creative bedtime story or, you know, writing comedy or areas that probably more you would find in a consumer AI from other people. But you mentioned the second part, which is this notion of customization and where do you deploy? I think this really comes down to how do you view where models are going to be in the coming years? And so I think all of us at the frontier, it's our responsibility to build the world's most capable models that can interact across all of science and technology and knowledge work.

• 8:34Speaker 00

  Even if we focus on their abilities to build software, we still care about all of that. And over time, in our step three, we want to branch off to opening up to those other areas as well. But if you look at what's going to happen in the future, there's a big question. There's a question of do we have some all-powerful model that is static? Right? It's one model and we all use it to do economically valuable work. Or do we have all-powerful models that are able to actually become versions of themselves, deployed in environments, learning from the data in that environment? So the question is, you know, do we have a software developer that can write all software? Or do we have one that's deployed inside a banking environment and has truly access to all the information and learned over time from it?

• 9:18Speaker 00

  So it's a little bit, do we anthropomorphize it? Is it going to be like a human, right? I can be a very capable software developer, but you deploy me at, you know, a bank, all of a sudden, I'm going to have to learn everything over time from that bank and that embedded knowledge? Or am I something generalized that gets, you know, applied to it? I think the honest answer is that we don't know yet in this space. What I do know is today, when the models are not yet at human level capabilities and are not even superhuman level capabilities are there yet, it's very valuable giving the model access to as much data and as much context and as much ability to learn in an environment. And so we just look for the shortest path towards doing that. And the shortest path with enterprises towards doing that is by willing to deploy the model,

• 10:02Speaker 00

  the context intelligence layer, and the applications behind their firewall close to the data. It's a tactical in-time decision. I think over time, the form factor of that might change, but it's something that we've seen resonate really well with the customers we have.

• 10:16Tim Scarfe?

  It's interesting because I'm trying to tease out your view on scaling, so that there are many folks who think we should scale the models up. GPT 4.5 just came out. It's quite interesting that there was, you know, Gwern and Kapathi, you know, were on Twitter, and they were basically saying, well, you know, high vibes people, that means like smart people, they can see that this is a step up and it's doing well in very nuanced things, but the benchmarks aren't capturing it. We need better benchmarks. But undeniably, at the moment, there is a gap in capability, right? So we need to have customization and like, you know, thinking onsite, surface contact with domain verticals in order to do well. But I think you're saying that you can imagine a world in the future where we could bring all of this data back into a huge foundation model and it would work just as well.

• 11:00Speaker 00

  You will never hear me argue against scale. Scaling of compute and scaling of data is critical for us to close the gap between where models are today and where we believe they can be, right? Human level intelligence and even beyond. But that doesn't necessarily mean that the axis of scaling today are the same axis of scaling that they were when people thought they were two years ago. Right, so when we started this company, the narrative at the time in the world was we're just going to scale up, you know, the next GPT model, make it 10x larger and provide it with more web data and we're going to have this, you know, AGI like moment, this human level intelligence. And this was such a strong narrative that frankly, many people at the time in the world, you know, believed this. We didn't.

• 11:45Speaker 00

  And it's not because we don't agree with skill. I think skill massively matters. But our view was that there was a missing axis of scaling that wasn't being discussed. And it's frankly, why we started this company. And it was the axis of scaling for the use of reinforcement learning. You mentioned Krapati. I liked how he said it the other day, like scaling of next token prediction is the equivalent of imitation learning. Scaling of reinforcement learning is the equivalent of trial and error learning. And while there's probably some nuance to that, I think it is a right way of thinking about it.

• 12:12Tim Scarfe?

  Yeah, I read this amazing book by Max Bennett called A Brief History of Intelligence. And he was basically saying, when you look at the animal kingdom and humans in particular, you see this axis of simulation. Right, so it's the ability to imagine things or, you know, imagine experience which you haven't directly had access to. Right, and that just creates this explosion. And language, of course, is an even more sophisticated invention because it allows you to sort of memetically share those simulations that you didn't actually have with other people. So with reinforcement learning, you can actually try things and you can accumulate knowledge without needing to have direct physical experience.

• 12:54Speaker 00

  So I think it depends on where you apply reinforcement learning here. Right, so I tend to agree with a lot what you said, which is that at the end of the day, what we do in thought, and I think it differs by the way, the personal experience, the personal experience. We spoke about this earlier. My thought is entirely language based. It's a constant internal monologue that is so language based that there's no visual representations or abstract concepts. It's so for me, I feel quite akin to language models. To put it kind of like finally said, because I see how they think and how they reason, I can relate to it. But the reason I mention this to your point is that language is a way, I don't think it's the only way, but it's a way that we can explore different possible, you know, chains of thought, different possible thinkings.

• 13:40Speaker 00

  And I agree, a lot of how my mind works, and I think how many is, is that you're looking at an objective and you're thinking through the different possible chains of thinking that can get you to that objective. If that's about writing a piece of code, or if that's about something far more long, you know, range. And there are several things that we do to ensure that that objective is correct. One is that we try to keep it consistent with what we know are representations of knowledge that we build upon. So if I am, you know, reasoning or thinking through a math problem, I'm constantly consistency checking against the knowledge representations that I've previously learned, right? The axioms that have to be true in math, or physics, or any other domain. But then there are certain things that, no matter how much I try to keep true against,

• 14:24Speaker 00

  I actually need to do the work, I need to get the real world feedback. So the, call it maybe slightly flawed example here, but I think it's a useful one, is if I want to learn Go or chess. I can read a whole bunch of textbooks. I can play out chess games in my head to some extent. But at the end of the day, I am still a snake eating itself in terms of my own synthetic data, right? Like, and so coming into an environment where I play against someone else, computer or person, doesn't really matter, and I learn from my mistakes. Hey, my reasoning chains that I explored led to me losing this game.

• 15:06Speaker 00

  That's actually where I think it's really, really valuable to have some form of external feedback. And that external feedback in the case of chess or Go is deterministic. And sometimes that external feedback can be human feedback, but it's very hard to skill. And it's also not necessarily always the right type of feedback. And so that's where I think reinforcement learning can, can live on that boundary. Where when we have the ability to form a reward that pushes us more correct, or at least in the right direction of correctness, we can then Improve the next set of thinking and thoughts that are those next thinking change that you're talking about.

• 15:42Tim Scarfe?

  Yeah, there was a great nature paper talking about model collapse with Ilya Shormanov. We were dropping the interview of Ilya actually next week. But in a way, it reminds me of some of these AGI doom type discussions, right? Because when you talk about omniscience, I think that's not a scientific discussion. I think that in the real world, we need to push molecules around. It's the same thing with with software, right? The reason why these gen AI software coding things are so are so powerful is because they can actually test the software that they generate. That that's really, really important, getting getting that signal from the real world.

• 16:14Speaker 00

  So we're very known for our work on reinforcement learning from code execution feedback. I started this work when we were building our first company in the space in in 2016. It's kind of a big part of what we've spoken about that Poolside uses. And this is really the notion that if you have a very large diverse environment, which in our case we do, we have close to a million repositories that are fully containarized with their test suite and many millions or tens of millions of revisions, that we can say, hey, at this commit hash in this repository, I want to change this code. And then I want to execute it and see what comes back. If that's either running tests against it or just a compiler or interpreter or even synthetic tests.

• 16:56Speaker 00

  And what this allows us to do is it allows us to have a very, very large environment because a million repositories represent huge diversity of types of problems. If that's in cryptography or a web app or, you know, or database kernel.

• 17:10Speaker 00

  And it allows us to then design tasks for the models to do where they can explore possible solutions and the thinking that leads to those solutions. And then learn from when they're right or when they're wrong. Or at least, I always try to be careful saying right or wrong, more correct than less wrong. Right? Because that's essentially what reinforcement learning is. You're trying to push the model in a direction that the next time around, when you're sampling your thoughts and you're sampling your solutions, you're slightly better. And you do that enough times in training, you can start getting to a place where you get very good. But if you do this on a very narrow task, you get this notion of model collapse, right? Or overfitting, you get to a place where it's Like, OK, the model can only do this now. And then it's no longer a useful, generalized intelligence.

• 17:52Speaker 00

  Right? So coding kind of sits in that spectrum of like, it's deterministic enough, but has enough diversity that even when you're overfitting a little bit towards it, you're still making it very much a generalized intelligence. You're not making it a task-specific thing that can only play the game of Go.

• 18:09Tim Scarfe?

  I'm fascinated by this idea of a diverse set of possible minds or different intelligences. I think that you can create intelligences through a variety of representations and as many degrees of freedom. You made an interesting comment, actually, that you're not very visual. You know, you think in this analytical language space. I'm very visual. I can imagine sounds and so on in my mind. And the way we write software is actually very diverse. You know, like design pattern books and so on. Those are different analogies, different abstractions. You know, even Einstein, when he thought about relativity, he was thinking about ripples and waves and so on. And with software, you know, there's the syntax. There's the actual sort of like, you know, how we write the code. There's the semantics, what it means. And there's the behavior space, how we actually test it.

• 18:54Tim Scarfe?

  And you're kind of talking to a form of AI where we're learning like a hierarchy of representations and it can kind of flit between them dynamically.

• 19:02Speaker 00

  It comes down to how we view these models, right? And I want to really caveat this with that we don't have a good grounded scientific theory yet or really like robust interpretability. That what I'm about to say is anything but my best guesstimate or opinion at this point. But I think probably most people in the space will agree on the following. Is that what's happening in these models is that we're learning extremely high dimensionality representations. And some of those representations that we're learning represent, you know, the ability to use language. So these are representations that are massively interconnected with everything else. Some of the representations that we're learning are very specific pieces of knowledge.

• 19:47Speaker 00

  Right? So we had a bit of a conversation about this earlier. But if I take a piece of knowledge, like when FDR was born, that sits in a high dimensionality space. Probably sits close to other things. Related to US presidents and such, if you go back to the early days of machine learning, we think word to vec and like stuff like this, like I think those are still useful mental models to have. But back then, we were talking about embeddings and representations that represented words or a bag of words or knowledge. Nowadays, what we have with the type of models that we've been able to build is that we have representations of things that are far more generalized and far useful. The ability to use language, the ability to start doing reasoning. And I mentioned this because I think in the first wave of how we've trained these models in the last years,

• 20:31Speaker 00

  of just scaling up next token prediction on more and more data on larger models, we were improving the representations that represented the most overly represented things in the data. Language, knowledge, but we weren't yet able to start really improving the representations of complex reasoning, of multi-step processes, of the things that are required for building complex software, the things that are required for like figuring out new scientific like breakthroughs and theories. And now we have access with reinforcement learning to really improve those. But at the end of the day, if we had an infinite amount of data in the world that represented all of our thought process and all of the feedback we got,

• 21:15Speaker 00

  then it wouldn't really matter what we use. We could just learn it with next token prediction, right? We wouldn't necessarily need to use reinforcement learning. These are all just ways for us to improve the data and hence improve the intelligence. And we're trying to do that in the most compute efficient manner. So my team gets very tired of me saying this. They always say all the work we do is either improving compute efficiency on training or inference, or it's improving the data and hence the intelligence of the model. And everything you do, you can put in one of those buckets. And I know it's an oversimplification and there's always a little corner case here and there. But when we come up with the crazy new architecture for linear attention, which is something we've put a lot of work into, to me, that's just improving compute efficiency of inference. Right? If we figure out like a really amazing way to scale up reinforcement learning

• 22:00Speaker 00

  and that's actually just improving the data. And so to me, there are just the two facets of of model building.

• 22:06Tim Scarfe?

  Yeah, I think the the economics of of models is really important, actually, because even now, when when OpenAI finally released O3, there's a new version of the Arc Challenge that that came out, and they were spending $2.5,000 per task, but they could solve it, they could get superhuman performance. So now it's simply a matter of computation. But there's still a Pareto frontier, right? You know, we've got like the Gemini model, it's it's very, very cheap. You can just sample it, you know, maybe 50,000 times, and you can still get the answer. So so then we get to this definition of of intelligence and AGI. We've been talking about that a lot. Francois Charlet said it's basically your your reasoning or your knowledge acquisition efficiency. So you know, how quickly do you take new points in experience space and turn them into new skill programs? And you must be wrestling this Pareto this Pareto frontier, right?

• 22:50Tim Scarfe?

  So you you're you're figuring out what's the appropriate size of model? What's the appropriate architecture? And what's the trade off between how fat the model is and how much knowledge acquisition and reasoning I do in the situation?

• 23:01Speaker 00

  I think at the end of the day, all of us at the frontier right now are taking advantage of as much computational resources as we can get. And I think it can't be ignored. Like, I think if you are in the race for frontier model capabilities, your amount of computational resources that you can direct towards training is absolutely critical. But the wrestling part comes to where do you apply it? Do you apply it to make the model larger in a parameter space? Do you apply it to make to massive amounts of synthetic data generation? Do you apply it to the scaling of reinforcement learning and sampling more? And and all of this is essentially an equation that has an optimal, right, for every single one of these things. And and the way that we operate,

• 23:45Speaker 00

  and frankly, I would say most frontier labs operate, is that we try to run experimentation in each of these areas and several others to find that optimal. So to give you a bit of a sense, our team in January ran over 4,000 experimental runs. These happened across architecture, across data ablations, weight mixes, reinforcement learning, number of samples, all of these different variables. And what we're fundamentally trying to do with all of this experimentation is trying to understand what is that optimal balance between those things? And you said something really important, at the end of the day, cost matters. So the overarching kind of objective function of all of this is the maximum amount of intelligence that you can create within a certain amount of time and budget

• 24:28Speaker 00

  that you can then serve for a certain price to the end consumer. And the training and inference part are really critical. And so in our domain, because we focus on software development capabilities, we're in a quite valuable domain, right? It's economically valuable. I think it is a lot tougher if you're trying to serve both that domain and you're trying to serve the free users who want to write bedtime stories, right? Because they have different economically like values associated with them.

• 24:54Tim Scarfe?

  From an architectural complexity and customization point of view, you know, the enterprise, they want to design their own architecture. They want to have clear security boundaries. So you know, the software engineers in the finance department, you've got the software engineers in the legal department, and they want to create their own trade-off along those boundaries that we were just talking about. So does that make it more complex that rather than designing one recipe for everyone, you're doing a lot of bespoke stuff?

• 25:20Speaker 00

  So I think there is the building of the foundation model, and then there's building of everything around it, right? All of the software that allows you to then deploy it in different environments, if that is workstation, server on-prem, if that's in the private cloud environments, VPC, if that's pure cloud, anyone can access. And so we made a decision early on from day zero of the company to say, we are going to do everything possible to become the trusted partners of enterprises. And it came from quite a kind of simple analytical process. We said, well, where does the majority of economically valuable software development work in the world sit? And it sits in software development in enterprises. Right? Seventy percent of all dollars in software development gets spent in enterprises.

• 26:05Speaker 00

  But these are also, like you said, very complex environments with lots of security concerns and boundaries. And so we took a, call it again, a simple point of view. It said, well, where do they want us to be? Where are we seeing that we can actually be that the customers ask for? And what we heard over and over again is bring the model to the data, not the data to the model. And so we decided to build accordingly. And so today we deploy our full stack, model and applications all the way on top in these private environments. That requires a lot of work, a lot of engineering, a lot of work that we do to streamline this so that it can scale.

• 26:41Speaker 00

  I personally think in the fullness of time, everyone will end up on the cloud. But today, if you are tactically look at, you know, the global 2000 enterprises. A lot of them are still very happy that we're able to deploy in environments that other people aren't.

• 26:53Tim Scarfe?

  So you've gone for a kind of Tesla-esque strategy in a sense that you're controlling the entire stack. You're building, I mean, you're one of not very many companies that have the skills and expertise to build the foundation models. But I guess the question is that there are so many folks that are just focusing down the stack. So you could add so much value doing the reinforcement learning from code execution feedback, doing the whole kind of like, you know, architecture piece. Why did you decide to go the full hog and build the foundation models as well?

• 27:19Speaker 00

  I think it started where we had this conversation today. Like, how do we start this conversation? We started around like, what is Poolside looking to achieve? And the reason we built this company is because we saw a future that I personally think now is 18 to 36 months away. Where human level intelligence across the vast majority of knowledge work is achieved. And if you hold that point of view, two years ago when we started that we were, and at the time, the timelines were not that concrete for us. We would say five to 10 years, maybe 15. We, we knew that the world was going to get to a place where we would be able to replicate our intelligence and even go beyond. And when that's the point of view you hold, the question is, well, what do you need to be one of the companies who can help bring that into the world?

• 28:02Speaker 00

  And if you remember, we had our own point of view on how to do so in research and execution. And you don't get to do that unless you build from the ground up. You don't fine tune your way to AGI. We put that on the website, on that same page we spoke about earlier that was there on day zero. We had this list on there that is essentially, it was called Strong Beliefs Weekly Help in the Face of Empirical Data on Our Research. And one of the things was you need to build your foundation models from the ground up to be able to get to these things. You can't fine tune your way to success. We said all data over time becomes synthetic. Reinforcement learning is key to be able to to scale up capabilities. So it really came from that. But what we wanted to achieve and what we believe mattered from a research perspective.

• 28:45Speaker 00

  And I think that has really played out in our favor so far. And I think it will be very unlikely that we see anybody in the world get to human level capabilities by post-training the latest open source model. And I would even question, and I'm a huge open source advocate. My first company in the space, I built fully open source. But I would even question if we are going to have at some point truly open source AGI. If there's going to be room for that in the world. If we are continuing down a path where the capital required to build this is so incredibly large.

• 29:24Tim Scarfe?

  How hard is it to build a foundation model? I mean, just to give you a couple of examples, you know, DeepSeek. What's cool about them is they've made a lot of their training methods and optimizations public. They've got some great papers out there, new sparse attention papers, really, really cool. And I'm guessing that as you go up in scale, it gets harder and harder to train these models. But it's so difficult for people like me to know. I mean, is it just a software engineering challenge? You know, how hard is it?

• 29:51Speaker 00

  So I'd love to break that into kind of two parts. One is actually common a little bit on DeepSeek. Because I think DeepSeek is a is a great example of this second generation companies, us and.

• 30:03Speaker 00

  XAI and DeepSeek, we've taken a different approach. So DeepSeek, as far as I know right now, is about 200 people, researchers and engineers. They've got, you know, billion dollars plus of infrastructure there. And they've got two years of incredible work already behind them. And we know it because they've been publishing their work. All of us in the field, they were a known entity. We've been following the papers and have a lot of respect for what they've done. Because if you look at that last paper that they put out, the 47-page technical report on V3. In this space, we all know if you do all of that work, you get a really good model. And so I will even, you know, say to controversy of some that like, I don't think they stole data. I don't think they did anything nefarious. I think they actually just did great work.

• 30:48Speaker 00

  And we've got a two-year track record of research papers to actually follow to see that. Now, there's questions that we need to have about, do we want, you know, capable AI, you know, deployed in the world in the West where we might not share the same values or same, you know, principles as, you know, the CCP, you know, has. But that's a completely separate discussion. But the other notion is that process of two years of building, what we have gone through, what XAI has gone through, what DeepSeek has gone through. It's, yes, of course, the models get larger and the engineering gets more complex and the research. But at the same time, I think it's building up compounded advantages over time. You know, I think talent is so critical in our space. We have this incredible team.

• 31:31Speaker 00

  And if I look at what we have learned together as we've grown over the last two years, there's an immense amount of value in that. Now, of course, that needs to go hand in hand with actually improving your data constantly. Right? Like every month you can look back and you say your data on which your training is better. It's cleaner. It's better representative of what you're trying to achieve. It's improving your actual distributed training stack, right? Either by making it more compute efficient, which is a big part of it, but also allowing it to work at certain skills that before it wouldn't work out. Right? This is all of the work that you've seen on the different types of parallelism and that you get as you scale up into larger and larger clusters. There's work that you do that really is specific to how the chips underneath change.

• 32:14Speaker 00

  Right, so we've got the the Hopper series. But now if you look at what's coming out with the GB 300s, all of a sudden we have 72 chips with NVL connection. That changes the architecture that you want to run if you're training on that. So people often take the view of like the the architecture comes and then you map it to the hardware. It's actually the opposite. You look at the hardware and you determine what the optimal model architecture is for training and inference. And you're so you're constantly going back. But over time, these all just become compounded things. Our environment of code execution went from 1,000 repos to 10,000 to 100,000, now close to a million, and it's going to keep growing. So some of it's engineering, but some of it is also just the implicit knowledge and experience that you gain in your organization

• 32:56Speaker 00

  that builds these kind of modes for others to not be able to compete overnight with.

• 33:02Tim Scarfe?

  Is it fair to say, though, if every time you 10x your scale, what you already know isn't enough to get you there? You have to sort of not go back to the drawing board. You have to spend a lot of capital trying a whole bunch of different things.

• 33:13Speaker 00

  I think scale needs to be can can use a little definition. Because a lot of people have talked about scale as just scaling up model size. Right, I mean, that's you mentioned GPT four and a half, which I think OpenAI says, you know, 10x more compute, which probably means, you know, maybe five times larger and X amount of times of data or like whatever that combination is. Until, you know, recently, the world assumed there were two scaling axes. Size of model, size of data. I think reinforcement learning is truly a third scaling axis right now. And of course, that still is a a proxy for data. Don't get me wrong, but I think it's important to call out as a separate scaling axis. Because this changes the dynamic in terms of what skill means.

• 33:58Speaker 00

  Maybe you are not taking the model to, you know, 10 trillion parameters. I'm throwing out a number here. You're not taking the model that large anymore because you're finding that you can scale more efficiently by scaling up the, you know, reinforcement learning side. A good example of what we saw in the last couple of years was the the Lama models, where you saw at some point they went from, you know, 2 trillion tokens to 15 trillion tokens. By the way, all of us in the space have been doing this for some time. Because and that would be referred to at the time as overtraining. Because it was not Chinchilla optimal. But Chinchilla optimal never took into account that these models actually have an inference cost. So there are places where on the theoretical, yes, a certain model size with a certain amount

• 34:41Speaker 00

  of data with a certain reinforcement learning is the theoretical compute optimal way of training a model. But actually, if you have a constraint of running this in the real world, I need to serve this to a customer and it can cost me only so many dollars per million, you know, tokens to serve or a number of requests, then that constraint changes where I might spend my skill. And the reason I mention this is that I might say, okay, I might train it for much longer. But it doesn't actually necessarily introduce a lot more engineering complexity, right? But if I'm trying to go 10x larger, then it introduces engineering complexity. But it also depends on what hardware I do it. If I'm scaling up like Elon on, I think it's 632K, you know, H100, H200 clusters that

• 35:23Speaker 00

  he has interlinked together, then all of a sudden, that's a very different way of scaling than if I take the equivalent flops on 100K, you know, GB200s or 300s that are outcoming. So don't get me wrong, there's always next engineering. To us, it doesn't really feel like 10x from one to another. But there are major changes that we make as we find having more access of scaling. Where I think we've been very well positioned is that two years ago, we started already deeply the company around large language modeling and reinforcement learning. So this wasn't a new thing for us. So we've been building up incrementally over that. I think other companies all of a sudden had to bring this out from almost nowhere. And I'm pretty sure that felt like a 10x engineering project.

• 36:04Tim Scarfe?

  Can you help the audience understand a little bit more about thinking? So, you know, the R1 came out and you, you know, it's the same with Sonnet 3.7 and O3 and Gemini flash thinking and so on. And the end user perspective is you see these thinking tokens and the language model is kind of doing a form of self-prompt augmentation. So there's, you know, there's like the tier zero of chain of thought and scratchpad. And now the things are prompting themselves. And one way of thinking about it is, you know, we train them with reinforcement learning and it's kind of like it's imbuing this process in. But the interesting thing is that you can just take 100,000 thinking traces, you can fine tune just a normal base language model. And then just through sheer, you know, dint of interpolation, you know, you get a lot of the performance.

• 36:49Tim Scarfe?

  So you can buy performance, there's a kind of sigmoid relationship.

• 36:52Speaker 00

  I see where you're going.

• 36:53Tim Scarfe?

  But what's really going on there? Is there something special about the reinforcement learning in of itself? Or should we just think about it as a form of data augmentation?

• 37:01Speaker 00

  I have opinions here. Some of them, I think, already backed up with papers that are out there and things. Others, not yet. And we don't publish. So I'm saying this because I want to reference things that are that are out there publicly to back up some of the argumentation. Look, at the end of the day, we are updating a model, right? Based on, you know, a loss.

• 37:31Speaker 00

  Or some, you know, function that we are applying there. And so, yes, in the in the truest of definition, all of it is just data. Right, all of it is just data. Like, say, if you had infinite data, you had infinite reasoning traces for everything else. You could learn it through next token prediction and it would probably be an incredible model and reach human level capabilities. Right, so. But it is very clear that what you are doing today by taking 100,000 reasoning traces versus the equivalent compute spend on reinforcement learning to get there. That reinforcement learning outperforms the SFT side. And and I think that it's just because there is more signal in terms of what you can provide in terms of reward.

• 38:16Speaker 00

  And then you can just by providing only like the sample of data. This is, I think, it's a trade off, right? And again, it always comes down to like the data and the compute efficiency. And so I think if you had an extremely large amount of data, there is a path there. What I think is happening right now in the world where it's like, OK, we're seeing massive generalization from 100,000 reasoning thinking examples. I don't buy it. I don't think that's actually what I'm seeing in the models. I think even current benchmarks are

• 38:49Speaker 00

  are able to show that. But it's quite often when you look at these things, you see like, oh, it went up on this math benchmark a lot. And then the reasoning traces are all very specific and quite closely linked to like what's happening in that benchmark. So at some point, yeah, you might grok something a little bit more generalized, but you need a lot of data for it. So I think that the scalable ways of improving models are more around applying reinforcement learning where possible. That being said, there might be compute trade offs where at some point you say, oh, I want to use some of that SFT data to either bootstrap versus having it, you know, learn from scratch. I think there are places that can be done there.

• 39:31Speaker 00

  But you mentioned R1, the most exciting thing that I think was finally published in our space, because you know, none of us in the West really publish anymore, was the zero work. And the fact that you could see a model develop its thinking capabilities in coherent language without having been provided, you know, sufficient samples of what thinking looks like. This should be the thing that blows all of our minds. It shouldn't be the headline around the $5 million. No, the headline should have been, models are able to develop human-like thought in language that is leading to better, you know, outcomes in like things that are objectively measurable, like math encoding capabilities.

• 40:10Speaker 00

  That's the exciting part, without having actually been like aligned towards it.

• 40:14Tim Scarfe?

  Yeah, that paper blew my mind. I'm not sure if the zero was, it's not like AlphaGo Zero where it's, you know, there's no human seeding. I think there was still a little bit of human seeding, but it was mostly self play. But it was incredible, right? Because it learned these emergent behaviors, like it would say wait, and it would do stop, and it would do reflection, it would do reasoning. And many of these seemed almost like there's, there is a natural way of doing reasoning. It was very like human aligned.

• 40:37Speaker 00

  Well, I think, look, we can't forget that the basis of training data is still the internet, right? It's still the web. And so when people go and say, oh, like, I just want to see it like perfectly learned from nothing. It's like, well, you want all of evolution to happen like overnight. Like, we still have, we're teaching these models based in our own image, in our own data and what we have. Otherwise, frankly, also wouldn't be very useful, right? We want them to act in our environments. And so, but the fact that before

• 41:05Speaker 00

  If you take a pre-trained, you know, base model

• 41:10Speaker 00

  and you see the difference between that pre-trained base model versus one that's had actually, you know, reinforcement learning applied for developing its thinking, you can see that the capability of those thoughts, like you said, the self-reflection, all of these things, is improved. Right? And I think we often use the word emergence, but I think it's a spectrum. And we see things, you know, improve and all of a sudden, we now have a lever that we can pull. And we've had it for some time where we can improve the thinking capabilities of these models. And by improving the thinking capabilities of the models, to our point earlier, there's a smaller space of solutions you have to explore to get to the correct thing. And the more we can make that space, you know, more and more correct

• 41:53Speaker 00

  in the areas we care about, mathematics, software development, like scientific theory, all of these kind of areas, the more useful and valuable these models come. And by the way, humans are exactly the same way. Right? Like, I have this massive, you know, set of learning behind me that makes me useful in a software development environment. But if tomorrow you drop me into quantum physics, I'm probably not that useful.

• 42:15Tim Scarfe?

  Yeah, even then, though, there's this kind of analogical relation between flexible forms of thinking. And I guess I use the word emergence because it's just surprising. It's a surprising arrival of a capability which gives a significant uplift. And you see interesting dynamics as well. So you SFT these thinking traces, because I kind of think of the thinking traces as being a form of flexible thinking, that they give you more degrees of freedom to kind of operate in a particular situation of intelligence. And also, there's an interesting relationship between how fat and how thin the base model is. So with O3, it's a very thin model. And they found that it was easier to scale, you know, a computer inference time.

• 42:59Tim Scarfe?

  But they hit the edge of the sigmoid faster, whereas Sonnet 3.7 is a much fatter base and it's harder to scale, but they've actually got more headroom left, you know, if they continue to scale. And there's also an interesting thing that when you fine-tune the models, they fine-tuned like a Llama 1 billion model. And because the Llama model was kind of so thin, it's almost like the thinking traces couldn't take root. You needed to have like a base level of intelligence in the model that you're fine-tuning on to in order for them to work.

• 43:26Speaker 00

  What is thinking in this concept, right? Thinking is able to explore the possible space of a solution, right? Because right now, The reason we call this reasoning models and often not thinking models. The reasoning really is, to some extent, objective oriented. It needs to be able to achieve something to be actually be reasoning and then follow a certain step process. And the more complex the objective, right? It's still very clear you need a better understanding of the world. You need better understanding, you know, manipulation of language. And so thought in that sense is still something that I think is constrained by the model size. Absolutely. Right? And it always was, like even before reasoning models, it already was. But now we have an access to use what's already there and really improve it.

• 44:11Speaker 00

  Another way of thinking about this is if I sampled a model a thousand times with enough diversity introduced, either through temperature or prompt or whatever I was using. And I could find the correct solution in there, like the correct thinking lead it to correct solution. It means that somewhere there in the model, it's already, it's already there. We just haven't found the perfect latent space for it. The best is, of course, if I just only do this with temperature, right? If I, if I start allowing for more creativity of different sets of probabilities. Because what you fundamentally want is to be able to reward the model in such a way that for things that have deterministic correctness, at temperature zero, it will get it. Right? And frankly, even hopefully at a temperature point seven, it will, it will still get there. Right?

• 44:55Speaker 00

  It will kind of expand open its, its cone of possible options. And then it will collapse as it gets closer to the solution, to the correct one. And when you take a very small model, you realize that that opening cone is very wide. But you can't actually collapse into the right solution, right? For many things. But if it is somewhere there already in that cone, then you can try to push it towards it. Right? And that's where, again, name of your podcast, right? Like the machine learning side still kind of comes true. Like you can still overfit a small model to a task, but you can't get, you know, to actually generalize enough. And so I often think about this as compression. Models are just a massive amount of data compressed into a certain space. The compression is too small, it becomes, you lose way too much.

• 45:39Speaker 00

  And if the compression becomes lossless, like so large compared to the data, then you're not really learning anything. So I also don't hold the view that there's this like infinitely large model world. Like at some point, it's going to make more sense to say for this amount of compute, this amount of data that we're applying to teaching a model, this is the optimal size and then we're going to want to maybe paralyze them horizontally to try to achieve objectives. The bitter lesson, like learning and search.

• 46:08Tim Scarfe?

  I interviewed the winners of the ARC challenge and they said that, you know, language models, we sample them greedily, which means, you know, we take the next token, the next token. And natural language is kind of messy, so there are many, many degrees of freedom. And they were kind of speaking to the fundamental trade-off between, you know, creative thinking and reasoning. So with the ARC challenge, there is just one solution. Well, you know, not one, but there's a very sparse space of solutions. And they actually came up with a depth-first search kind of sampling type strategy. And there were some folks at DeepMind I spoke to and they were saying, you know, when you do reasoning, you actually want the softmax to be very precise because you want the thing, right? You want the thing. But sometimes you actually want to have like creativity and diversity. So how do you square that circle? How do you have both?

• 46:52Speaker 00

  Well, I think there are problems where you need the creativity and diversity to get to that determinist, like to that final solution, right? I think that's the notion of it. Like if there is a single algorithm that you have to run over and over again, like addition or multiplication or something, then it's a very narrow cone of things that you want. But if you're trying to find, you mentioned the general relativity of Einstein, right? So for Einstein to discover general relativity at the moment in time and the axioms he had around him of truth, it required quite a wide cone of diversity of creativity, right? To be able to explore different ideas and then of course make sure they're consistent with what's already true in the world, to make sure to know which ones are correct and which ones are wrong. But if you would have kept a very like small like diversity, you know, like things,

• 47:36Speaker 00

  you would have probably never discovered it, right? It's the quote of it's a fine line between genius and madness, I think kind of applies like to models in this case as well. So but at the end of the day, if you have infinite creativity, drag the temperature up of a model like, you know, it'll it'll collapse into garbage, right? It won't be something that that that makes sense. So it's our job when we train models and build them that for the things we care about, the things we define as valuable intelligence, that we make that trade off like perfectly. Because that's actually what we're doing. When we're using reinforcement learning to teach these models, we are doing exactly that. We are saying, hey, when you're sampling in this part of the cone

• 48:20Speaker 00

  of possible, you know, thinking traces, you're directionally correct. When you're sampling in this part, you're entirely wrong, right? And so if if you start going down the path of trying to apply calculus to a problem that doesn't require calculus, you want that entire set of thinking traces that stem from that to no longer be something a model does when it encounters a similar situation in the future. And so it's that's, I think, is is the trade off what we're constantly doing. And I think human intelligence is exactly the same thing. You see this if you take someone who is very new in a field, very young. So I also think most interesting scientific breakthroughs come from people in their twenties and like, you know, because they're no longer there's not a constrained cone yet. They're willing to explore lots of different ideas. It's why you see people who are great at lots of different fields,

• 49:05Speaker 00

  like a Feynman, being able to come up with very creative ideas. But also, they might at times, you know, go way too far off and don't get to the right objective. So I think intelligence is always that trade off. I don't think we'll ever get it perfectly right. But we can keep getting it more and more efficient. And over time, we might be willing to say, hey, for known compute budget work, which I think is the vast majority of knowledge work, like in the real world, accountant, a software developer, we kind of know the intelligence budget. That's where you want that cone to be as perfectly narrow so that it can, you know, maximize for the economically valuable work. But when we come to unknown compute budget challenges in the world, solving cancer, the next breakthrough in material sciences, here we might say, you know what? I want that exploration much wider

• 49:50Speaker 00

  and I'm willing to pay money for it to be much wider. That we can explore more ideas, right? It's coming back to AlphaGo. It's how many moves do you want to explore next? How much depth and breadth do you want to have?

• 50:01Tim Scarfe?

  Yeah, I love that. I mean, the space of intelligence is very gnarly. And we work as a collective intelligence. There's a great book by Kenneth Stanley called Why Greatness Cannot Be Planned. And he basically said that monotonic, objective optimization is the dumbest thing you could possibly do. So, you know, what we actually do is through serendipity and what's interesting, our nose for what's interesting. We collect diverse stepping stones and many of them lead to greatness. But of course, in an LLM context, it's about just sampling and actually taking in those diverse perspectives. But I want to talk about software a little bit. So, Paulside, your company, your product is intelligence. But initially, you've been very focused on software engineering in particular. I can speak to just my own personal experience that Gen AI software has revolutionized the way I write code.

• 50:46Tim Scarfe?

  I'm now writing software in, let's say, a month or two that would have taken me years to do before. It's absolutely incredible. But what is your main objective here? And how do you see the sort of the software engineering trends changing?

• 51:00Speaker 00

  The main objective is to achieve human-level capabilities and go beyond. And that means that in a world today where we have maybe 100-plus million people who are building software, we want to bring that from 100 million-plus people to anyone is able to build software. And the core and the sets of people who today are at the frontier of what they're able to do in software to be 10x or 100x more productive. And I think this just comes because what is software? Software is a lever that we have on the world to be able to bring productivity, right? It's a lever to abundance. It's a lever to reducing the cost of things. And so to me, jumping really hard on the end of that lever, right? Putting the biggest weight on it is putting the biggest, most capable intelligence on it.

• 51:46Speaker 00

  Because that allows all of us together with AI to pull that lever and to drive the cost of things down. And that's always been the thinking behind this. Now, I think it's important to not just talk about the future. Because otherwise, you get the founder of a frontier AI company just saying AGI, AGI, AGI. And I'm sure you've had plenty of those podcasts already. I think it's important to also know what can you do today? And today is in the intersection between model capabilities and limitations.

• 52:16Speaker 00

  A world that's developer-led, AI-assisted. So the AI capabilities and limitations and the human. And how do you find the perfect intersection by adding product on top of that? Right? How do you create a product that allows people the max personal lever that they have for their own productivity? To, like you said, do what you do, which before would have taken you years, you can do now in months. And that has a lot to do with the model, but also has a lot to do with the user experience. It has a lot to do, how can you bring the right context to the model? Right? How do you make it easier for it to find the information to be able to give you the correct answers? Some of that is external from the web, some of that is within your code base, some of that's within your knowledge bases. And so building really powerful assistants. Today, we do that in editors, we do that in web, we have soon CLI coming.

• 53:01Speaker 00

  I think is really critical. But this is like a symbiotic relationship, like as your model gets better, you can do more on the product. The product form factor changes constantly as the model gets better. You've already seen that it went from code completion to chat to now increasingly more agentic. And I think in the future, increasingly autonomous. And so you just have to kind of be on that frontier constantly playing with all of those things.

• 53:24Tim Scarfe?

  So one of the things I've noticed, and again, the reason why I'm so excited in particular. I think it's really good for founders like me. I've got a very small team and I can just rapidly iterate. And I guess my process of coding has become more like I'm a reviewer. Right? So I get the language model to generate a bunch of code. I do a bunch of tests and increasingly, rather than writing the code, I'm just reviewing it. And I'm saying that looks good to me, looks good to me. Sometimes I'm going backwards, I'm going forwards. And the thing that I want to understand is how does it work when you have teams of people? Because the way we write software is we have a mental model. We create these abstractions and we have some idea of how the software should be crafted. And we share that with our friends.

• 54:09Tim Scarfe?

  And now we're generating code almost quicker than our ability to review it. So how does that scale in teams?

• 54:16Speaker 00

  In the world today, that's developer-led, AI-assisted. The question as always comes is like, you know, what requires essentially knowledge sharing? What requires review and what doesn't? When you make a one-line changer to the documentation, it doesn't require review. When you make a massive refactoring of the entire codebase that impacts every developer, you're going to want to knowledge-share that with the entire team and maybe get input. And I think AI is no different of you scaling up your team, right? Or if you're scaling up AI in this case. So it always sits in that boundary of where is it important to make that knowledge-share happen? And where is it not? And so code review has often been seen as this like process you have to do to catch bugs or to do X, Y, Z.

• 54:59Speaker 00

  I always think code review has truly been around knowledge-sharing first and foremost, and then occasional where needed, the ability to get input from other people because it touches surface areas that impact others or that you might not know or you weren't the best person on. And so when all of a sudden you're now producing, you know, 10x the code or you're moving a lot faster, and it sounds like in the domain you're working in, it already feels like AI is, you know, a very valuable partner, like almost like an entomorphized like intern. Right in other domains, it's not there yet. It should be seen as adding someone to the team. And I think increasingly, that's the relationship that we're going to have with AI, where it is like adding people to our team, except instead of adding humans, we're actually adding, you know, AI agents.

• 55:43Tim Scarfe?

  So one way of thinking about this is, you know, we've got this fairly linear software development lifecycle. We have business analysis, and then we do story points. And then, you know, we write some code, and then we do some tests. And then, you know, we approve all of those, you know, with release control and so on. And one way of thinking about this is it's about control. And it's about aligning the code that we write with our business objectives. That's why we have all of these different gates and approvals and so on. So what does that mean? When we have increasing autonomy and when, you know, in the code crafting process itself, we can just do so many things, does this traditional software engineering lifecycle give us bottlenecks?

• 56:24Speaker 00

  I think a lot of the software development lifecycle collapses into the models over time. And so, but I think it depends again on the environment. If you're, you know, writing code for nuclear missiles, Probably want a lot of those steps of that linear process to to still exist, even if AI is the one acting in it, right? Because you care about a certain number of nines of reliability. In that case, probably, hopefully, not even nines, just hundred percent reliability, which you all know is impossible in software, right? But we want many, so you're willing to invest in that. In other places, software becomes more ephemeral. Maybe you write something that works for a week as a tool or does a set of tasks. And so there, the S D L C doesn't really matter. And so on those two extreme ends of a spectrum,

• 57:07Speaker 00

  and you're starting to realize about myself, I use spectrums a lot. It's just the way my brain works, is as AI gets more capable, we're going to probably have a distribution of software that tilts far more to the left in terms of volume of software. It's closer to not requiring those stringent processes than to the rest. But the world's global banking infrastructure is not going away, right? And there we want a certain set of checks and balances in place. The question just becomes, is it is, is it AI acting across all of those checks of balances? And is at some point it becomes so reliable that I can start removing a lot of them? Because if I have a software developer, a human software developer, who never, you know, makes a mistake, never writes a bug, his C I tests always hundred percent pass for the last five years. At some point, I might say, you know what?

• 57:52Speaker 00

  Give the three hours of C I back to the human, you know? Let them just move faster. Now, again, that's a theoretical example, because in the real world, that's not perfectly like this. But as we get more and more above human level capabilities, at some point, we might just say, you know what? That's fine. Maybe even unit tests fall away. Maybe even, you know, like C I, like all of this starts becoming less and less. So I think it's useful to think about things in terms of the limits, not because we're at the limits tomorrow, but because it allows us to kind of show where we're heading towards. And so I think, yes, in the fullness of time, a lot of the software development lifecycle collapses into the model and and doesn't require many of these checks, but not everywhere and not all at once.

• 58:31Tim Scarfe?

  So thinking about the sort of the dynamics of software engineering in large enterprise, you know, a lot of companies in the FTSE 500, they frankly can't hire really talented software engineers. There was always this problem that they would do low code and no code. They would, you know, build things on Microsoft Power Platform and so on. And now we're in the era of Gen AI coding. And now almost anyone can write amazing software, almost discardable, software applications to do whatever they want to do. So do you think that we're going to see more folks writing code? And how will that change over time?

• 59:02Speaker 00

  I'll have to say, and look, that's maybe because I spend so much time with enterprises. I think we find incredible software engineers everywhere, first and foremost. But I hear you in terms of what you say is that, you know, what you get paid to be a staff engineer at Google is not the same what you get paid to be a staff engineer at a bank. Right? And so there are some distinctions here and there. But overall, I think there's great developers everywhere. What we're seeing right now is that AI is, while it might be for you in your field or for greenfield projects, already at the place where it feels like that massive, you know, unlock. I still think right now in most enterprise environments, it's feeling like a 20 or 30% productivity gain. And it's a productivity gain for some people by 2X, because they're doing, you know, unit test automation.

• 59:47Speaker 00

  And all of a sudden now it is way faster and maybe even becomes 3 or 4X. And in other places where people are working with, you know, a company-specific programming language in a specific domain, where the models are really not that good yet. And so I think it's worth acknowledging that model capabilities, mapping over to the real world in enterprises, is not a one-size-fits-all. And what I do see is that people are excited about being able to do more. If that's being an existing developer who can now, you know, take the boring parts of your job and actually automate it, or can build a lot more software faster. Or the fact where, yeah, I now see a product manager going, "Oh, I can actually build this prototype myself and show it, you know, to the managers, and then see if we want to go build it out like at a better scale."

• 1:00:29Speaker 00

  So I do hold the view that more and more people will be able and will want to build software. But the want is an important part. You know, because we're software developers ourselves, we often assume that everybody wants to build software if they're able to. And the reality is not everybody wants to build software, even if they tomorrow can. But the product manager who's always wanted to build his ideas quicker, absolutely. Right? That maybe one in five business person who, you know, has this idea that he's been trying to rally a team around but couldn't now can. But it is not universal that everybody in the world will be building software because it's still something that requires you to want to do it.

• 1:01:06Tim Scarfe?

  Yeah, I mean, I guess I agree that clearly the lift is, you know, the nought to one.

• 1:01:13Speaker 00

  Yeah.

• 1:01:13Tim Scarfe?

  You can now build individual applications just in seconds and it's incredible.

• 1:01:18Speaker 00

  Absolutely.

• 1:01:19Tim Scarfe?

  But I still think that there's a much bigger lift and the only thing blocking us is a lack of imagination. So for example, the reason why Google engineers are paid so well is they're building scalable distributed systems that millions of people use. And, you know, the multi-agent fault resilient systems.

• 1:01:35Speaker 00

  Yeah.

• 1:01:36Tim Scarfe?

  You can still build systems like that. And I guess the question is, some of this is an education thing, right? There are, there are good ways and bad ways of using gen AI code. A good way of doing it is understanding that there is a complexity ceiling. And if you build a monolithic application and you just keep, you know, building on top, on top, on top, it's going to collapse. But you can, it's almost like it's teasing you to design modular, almost serverless type, you know, multi-agent type systems. And, and the L L M's can handle that, you know, to a much higher complexity ceiling.

• 1:02:07Speaker 00

  I think you're right in mapping the current limitations of models to a really good way of getting around it, which is building small modular things that, you know, the models are able to essentially understand easily and work within. And having those kind of architectures where there's some separation of concerns help in that world. But it's the models of today. And I think this is an important part to always come back to, that I don't think there is a universal limitation to the type of software that can be built by models when we talk over the next three to five years. I think, but you're absolutely right in saying that if today I try to build a massive monolith application and just have the model vibe code the hell out of it, at some point, the whole thing collapses in on itself. By the way, I don't know about you, but there have been times in my career early on where

• 1:02:52Speaker 00

  I did exactly the same thing. I would build something and build something and build something, and at some point I'd be like, oh my god, what monster did I write here in code? And then it pushed me to refactor it, make it more modular, make it better, right? And so, we're seeing limitations in the models today, while they're still far from our capabilities, that also we can reflect on having seen in ourselves at different moments as well.

• 1:03:12Tim Scarfe?

  Yeah, and that's really interesting, because it's like, the language models, they kind of scale with quadratic complexity, and software scales exponentially of complexity. So it allows you to build software which is two orders of magnitude more complex than you would otherwise have built, but you still hit the complexity ceiling just, you know, very, very quickly.

• 1:03:28Speaker 00

  It's an interesting way of looking at it. I haven't given that much thought, but I'll have to do so.

• 1:03:32Tim Scarfe?

  The other thing which I think is really interesting is, at the moment, you know, when people do gen AI coding, they're generating software and they're doing unit tests and so on, but we still have this fairly linear mode of software engineering, which means, you know, we have release control, we put stuff into production. Now we're starting to see the advent of like MCP service and so on, which means that during the development process itself, the intelligent system can actually talk to your database. It can say, well, what's the schema on the live database? Talk to my actor system, how many actors are in play at the moment? You know, do I need to repair this actor? So now increasingly, there's an operational layer to the software process.

• 1:04:10Speaker 00

  I'd love to anthropomorphize this for a second, right? What we do as developers is, we'll open up the database console and we'll check the schema. We'll go talk to someone on Slack or in person, and we'll gather information, and I'll pull up the documentation. And so I think that some of the protocols that the world is creating are a way to make that with current model capabilities easy for the model to do. I think, again, if we play this out over the next couple of years, I'm not sure if it's a protocol, right? I'm not sure if it's just a computer use agent that's been doing this, or the model is just writing code to hit the API, you know, of Jira, or hitting the, you know, like just directly connecting to the database and executing a SQL command to get the schema. So I think there's things that we build today

• 1:04:53Speaker 00

  that matter for the limitations that the models have. And they matter, and they're useful. The, I think the interesting question that stems from this, at some point, when we talk about a large multi-agent system, just like we talk about a large company with lots of developers collaborating, how do we make that collaboration efficient and work well? Right? Is this a world where we have a thousand instantiations of our model, you know, each essentially acting upon, you know, in an organized collective, like a company? Is that hierarchical in nature, just like we are in our organizations? Because there's things that we can't do. I can't access the thinking traces and solutions

• 1:05:36Speaker 00

  of my, you know, 500 peers. It's not, can't be stored in some central database. But it's something agents can do. And so all of a sudden, I think we, while we're building towards human-level intelligence, there's already things that models today can do that we can't do by the nature of how we operate. The paralysation of being able to run through an entire codebase and summarize it file by file, over a thousand files. I can do it, but it wouldn't take, it took a very long time and not be so efficient. The length of context windows, I don't know about you, but I can't hold a million tokens in my context window and retrieve perfectly across it. At some point, we're gonna have to get away from this realm of like anthropomorphizing it and starting to say, okay, these are certain things that models can do and different. And that, to me, comes to these protocols. Is, right now, we need them.

• 1:06:21Speaker 00

  I'm not sure if we need them in 18 months.

• 1:06:23Tim Scarfe?

  You were speaking to something interesting there, which is that there's a kind of semantics gap in code. I mean, there was always a famous adage that, you know, many companies, they didn't want to release their codes on GitHub because it's so valuable. It's actually not that valuable because the semantics, the intentionality, the motivation behind the code is not in the code. You know, it's the same thing with language, you know, that there's, there's a lot of missing information that is, that is not in the data. But we can capture this information because you've now got an organization of developers. They're building code, right, using these tools. And the entire thought process is in language. So you could capture that into a kind of semantical database. You can rag into it. And now you've just got all of this kind of meaningful, motivated information,

• 1:07:07Tim Scarfe?

  which means the language model won't always be making the same mistake because it knows, oh, the reason why we did this is because, is because of that.

• 1:07:13Speaker 00

  There's lots of really cool stuff, right? We've never, like you said, we've never had the ability to trace the thinking of humans and store it. And as these models are being deployed increasingly more as assistants and future agents and even autonomous agents, we can now all of a sudden access that. The question is, will we access it or will the models access it themselves? And, and so I, I always try to be careful with going too many steps into the future because at some point you're, you know, you start collapsing into like sci-fi. But I don't think we are that far out from this specific scenario. There's quite a few people who won't agree with me on this, but I think it's really important that as long as we can, we should keep models thinking and reasoning in language.

• 1:07:56Speaker 00

  There's incredible work that can and probably even more compute efficiently be done with latent space style thinking. There was a great meta paper about the latent space reasoning tokens. There's been some other approaches that I've seen that I, that I like a lot. We saw a, a language-based diffusion model coming out like, you know, earlier this week. I really like the guys behind it. And so I think there's lots of architectures that will work. Actually, my team kind of gets tired of me saying that probably every architecture can work in the world. It's just a question of how compute efficient it is. So I no doubt the diffusion language models were going to work. It's just, is it the compute efficient thing for the type of capabilities and tasks that we care about? Now, the reason I mention language being important is that as models get more capable,

• 1:08:38Speaker 00

  having the ability to see the reasoning and thinking traces that they have, and like you said, reference them from the past that leads to what led to certain decisions, I think will become massively valuable, both from an interpretability perspective, from just the ability to build upon work that was done previously by other agents. And I think there is a discussion to be had on if it is going to be useful from a safety and alignment perspective as well.

• 1:09:04Tim Scarfe?

  This diffusion-based language model is a great example. I've been very excited about this for years, but they never worked particularly well. So this one that just came out that he spoke about, that had roughly 10x efficiency advantage. So you know, like with the auto-aggressive language model, you have to actually token, token, token, token. This thing, they only ran diffusion about, you know, five or six times. They got the same results. Diffusion's really good from a code point of view because, you know, just like with vision diffusion, you can actually edit things. So you can say, okay, well, I want to hold this code fixed, but I want to edit this bit in the middle. Why is everyone not doing this?

• 1:09:38Speaker 00

  There are so many architectures that can work. And I think diffusion is one other field of it. But we've done a lot of work in our industry and in each frontier company to make the ones we have really efficient. And so to shift over from one to another, you have to have the efficiency gains. You have to be willing to take the time that it takes to actually go after a new architecture. All of the experimentation you've done until that moment, right? Does it still hold true in the same way or not? And so, you know, we made a big bet a little over a year ago on linear attention, RNN-inspired attention, and we've had models in production since about fall of last year already with linear attention.

• 1:10:23Speaker 00

  And so we would ask, like, why is not everybody doing this, right? Why, like, this makes so much sense, or it's like gains, et cetera. And part of that is also just because all of us end up investing heavily in a certain area, and then we just scale it up further. And making that shift needs to be really, really valuable to do so. And so I need to dig further into the diffusion language models in terms of how much of that efficiency, you know, shows up compared to some of the things that we're doing or others. We're unlikely to go down this path ourselves

• 1:10:58Speaker 00

  because of different things that we've done in other architectures that we maybe just don't disclose yet. But I think it's exciting, and I think we, there's a lot of room left for architecture research. I think diffusion language models, the tip of the iceberg, I think a lot more things can be done. But also, they need to be skilled up. So some of the coolest stuff that we've seen in open source, you know, or others, comes at this seven billion parameter, if at all, at seven billion. And then the question becomes, well, how does it, you know, operate at seventy billion? How does it operate if we want to try to make an MOE equivalent of, you know, of a diffusion language model? Like, and so at some point, it's like what works at small scale very likely can work at large scale, but does it stay the same efficiency? Like, can you get those same gains?

• 1:11:41Tim Scarfe?

  Yeah, that's a very good question, because they built a mid-sized model, and it was commensurate in performance to the other frontier mid-sized models. The other thing I loved about it is you can actually run, you can do an unbounded amount of computation. So you can actually just continue to do diffusion, and from a test time computation point of view, it's actually very, very flexible. I love that. But just coming back to the software engineering thing. So I think what we're seeing, though, when we do increasing AI dev is a lack of autonomy on the part of the humans and a lack of legibility, right? So we're now building increasingly inscrutable software. Now, you know, let's make no bones about this. No one understands, no one at Google understands. How the how the software code, but you know, so we shouldn't be hyperbolic about it.

• 1:12:22Tim Scarfe?

  But we could, as we were just saying, we could design an information architecture, which mediates a cognitive interface, which means at least at some level of abstraction, we understand what the thing's doing. We're putting guardrails and so on. But we are describing a future, though, where we're just building these inscrutable monsters. What does that look like?

• 1:12:42Speaker 00

  In terms of to your first point of of models writing code, I think code is that higher level abstraction already, right? We can all go in and try to understand code and maybe we add some print statements and we spend time. And it takes, look, it takes cognitive load, but code is deterministic. It gets interpreted by a compiler. And all of us as humans, if we are willing to invest the time and effort, can understand it. It's just isn't worth it, right? So are we OK with building big software code bases that we can fully understand? And what we historically already have been with human intelligence. Like you said, to your example of the Google code base, not a single person understands it anymore. And we're totally fine with that because it does what it's supposed to do. And when we want to actually introspect a part of it, because there was a bug and it went wrong, or because we deeply care about,

• 1:13:27Speaker 00

  you know, the recommendation algorithm that X released, you know, open source, like, we can choose to spend time there. But just like with human built software, we don't always spend time going back. Look at the amount of legacy code bases that no one has looked at for years, but they perfectly run. So I think that's just a choice. To the point of model, not the code base, with the model being the quote unquote, you know, monster, like the interpretability, there I think it's going to become increasingly important for good interpretability work. And I think there are question marks if at the limit of model capabilities, interpretability will be as useful as it is at a, call it, you know, less, like, smaller skill.

• 1:14:11Speaker 00

  What I mean by that is I'm not sure if we will ever be able to truly understand the reasoning and thinking process that happens inside a model's neural net, just like we can in our own biology.

• 1:14:22Speaker 00

  And so, but I'm super excited about some of the interpretability work that Chris Oles team has released and has been public about. I think that's an incredible direction. I highly want to encourage everybody to be doing, you know, this type of work and try and understand what's happening in models. I think that as we, if we keep models, reasoning and thinking and language, that adds a layer of interpretability. It doesn't mean that's exactly what's happening underneath in the model. You could imagine a model that develops perfectly fine sounding reasoning and thinking chain, but actually it's trying to optimize for a different objective. And that's where I think safety and alignment and interpretability kind of meet in the middle and really matter.

• 1:15:07Speaker 00

  But I like to think that with good work on interpretability of what's happening at the activation and weight level of the model, combined with keeping models reasoning and thinking and language, and trying to understand if those two things stay consistent, we can actually do a really good job at alignment and at safety.

• 1:15:26Tim Scarfe?

  Yeah, the interesting thing is because the language models are trained on all the colloquial human code, they produce locally interpretable code. It's a little bit like, you know, language models, they are actually surprisingly aligned. Like when I'm using Open Interpreter on my CLI, if I tell it to delete all the files on my file system, it will say no. And I've built an LLM application and if I tell it to change your name to someone else, it will say, no, actually, you know, it's ISO, this is the name, I'm not going to change the name. But then there's this kind of like global illegibility. But there's also the thing that we're generating code, but also there'll be a mixture of code and models. And some of the inscrutability will just come from this thing is like a living thing.

• 1:16:08Tim Scarfe?

  You know, like there was that Black Friday incident where, you know, we're doing high frequency trading and automated trading, and it works really well until it doesn't, and then you get this cascade effect. And the locus of control is now in the algorithms and the models and the machines. So we could easily build these very complex systems that seem to work until they don't.

• 1:16:27Speaker 00

  So I would take everything you just said, and if we replace models with humans, it would still hold true as well. Right? I think that when we talk about it, there's this massive, complex financial infrastructure code base that's been built by all of these humans, where not a single person, you know, knows it anymore. And our locus of control is not on anyone anymore, and then an unexpected event happens.

• 1:16:49Speaker 00

  I think we often take what can go wrong and put it on the models now, but it's already what the world is like. And so I think the best job that we can do is to make these models more capable than us. Make them highly capable in the ability to write good code that covers the edge cases. That's not lazy to write the tests, right? That's another thing. Let's talk very honestly about ourselves. There's things we enjoy doing and there's things we don't. How much of the world's code doesn't have proper test coverage? How much of the world's critical code doesn't? Right now, all of a sudden, if I can decide to deploy dollars to compute, to intelligence, to shore up the test coverage of critical code in the world of financial infrastructure, to shore up the security of, I think it's pretty much commonly accepted

• 1:17:34Speaker 00

  that the most of the world's critical infrastructure runs on really insecure code, right? The power grid and others. If now I can say, okay, we're willing to now invest, could be from a company, could be from a public sector, a billion dollars in making the code of our, you know, electrical power grid more secure. But now I don't have to bring humans together, but I can do this with AI. And I know it's not going to be lazy about writing the tests or other things. I think that's exciting, right? And we get to focus on the things of exploring the frontier of science together with AI and the things we want to spend our time on. We have a shortage in the world of people to write incredible software and code. And that just shows on the amount of legacy systems we still haven't upgraded yet. And so I think this is a way of overcoming that, not necessarily.

• 1:18:18Speaker 00

  And of course, just like with humans, it adds another area of fault. But I'm not sure if, uh, which one of us in the fullness of time would you rather have write code for a critical system? Like I often say privately is like, I want the nurse to be a human, but I want the surgeon one day to be a robot.

• 1:18:38Tim Scarfe?

  Just looking at the evolution of AI software, I'm very excited about meta programming. So we can actually have the system repair itself and generate its own code, you know, in response to failures. But then the next stage of evolution is, why do we need code at all, right? Why don't we just, I mean, it's so-called transduction, where we don't even have an intermediate code, you know, step at all. We just get the model just to do the thing, right? And the whole thing is adaptive. Are you excited about that?

• 1:19:04Speaker 00

  My opinion over the years since 2016 has changed on this. I used to be of like, Karpathi's quote in his, I think it was his 2016 blog article. Software 2.0. Software 2.0, right, where it's like at some limit, you know, everything just becomes a model. I think about it a little bit different today, no longer that extreme. I think that there are parts of the world's infrastructure that we want to have in something that's interpretable. And code is interpretable, can be traced, can be tested, can be understood. A human or an AI can go in and understand it completely. And so our financial payment infrastructure in the world probably want that to be in code. Our, you know, electrical grid probably wanted to be in code.

• 1:19:47Speaker 00

  What's operating, I'm seeing the railroad out here, you know, the switches between trains, probably in code. But do I care about other pieces of software if it's, you know, just a neural net simulating the whole thing behind the scenes?

• 1:20:00Speaker 00

  Now, I think also want to come back to what code is, right? Code is operating in most cases of most software, not everything, on a CPU. Right? And, and we have done an incredible job of optimizing the hell out of that evolution of hardware to be able to run deterministic code that, you know, can serve us to be valuable. There's lots of places where for a long time, the models might not be even compute efficient enough. It might just be too expensive to have all of software in the world collapse into a model call. Right? But you're not right that if we get to a point of model capabilities that are so good, so trustworthy, and we can treat them as doing deterministic things.

• 1:20:43Speaker 00

  Right? I know that every single time I ask X, it is so aligned that it's going to do Y. Because that's essentially what code for a CPU is, right? Like, at the end of the day, we are asking for something to be truly deterministic. Then maybe there is a, as those cost, you know, profiles change, more and more will move to the models.

• 1:21:02Speaker 00

  But I, I still, and maybe, look, maybe I'm holding on to an old idea. But I still like to think that it will be cheaper for a capable model to build and maintain the Uber Eats app and have it run on CPU infrastructure as maintainable code to update it and change it than it will be to simulate the entire thing as a neural net, right? And so I think for a long time, code in the real world will continue to exist.

• 1:21:28Tim Scarfe?

  The amazing thing, though, about these AI models is that in so many ways, they're just smarter than us. So they can, they can basically write or learn functions that we can't write code to do, which is, which is incredible.

• 1:21:40Speaker 00

  100% for NPR Pro. There's so many places where this is going to be incredible.

• 1:21:44Tim Scarfe?

  Absolutely. So maybe it is a spectrum. And I mean, I was, I was having a bit of a galaxy brain thought. I was driving to, you know, to pick Marcus up, our creative director this morning, and Google Maps, it was taking me this weird way. And I was thinking to myself, what's it doing? Like, is this thing being a utilitarian? Is it actually like optimizing for the, you know, to reduce the average route time? And it's taking me down a bad way because it doesn't care about me. And I don't know. And, and in the future, these, these AI systems, they, this is kind of what I mean by the loss of control. Like, it might be doing some weird galaxy brain thing, and it might not necessarily be what's right for me. And, and maybe I want to know about that.

• 1:22:23Speaker 00

  I think today, there's a lot of machine learning systems that already we don't have any good interpretability on. If that's in fraud detection, if that's in actually in, in the algorithm that's mapping you to your location, right? I'm not sure what's behind the scenes of, of Google Maps, but it could very well already be a neural net that is very task specific. I wouldn't be surprised, or at least a part of it, right, likely is, that's learned on a lot of data of, you know, the London traffic patterns and whatnot. So I think we've already faced that in machine learning and will continue to face it. It's, there's parts in the world that we want things to be deterministic, in my view, and we're going to want to keep them. And there's other parts where you're absolutely right, right? I want the most optimal route planned from somebody who understands me, not just the general London traffic, right?

• 1:23:05Speaker 00

  Maybe it's the fact that I prefer to take, you know, trains that are less busy and, and than ones that are more busy and I'm okay to get to a certain time. I think as we add intelligence, generalized intelligence, right? So, human-like intelligence to the world. We've got a lever that we can make continue to more compute efficient. That we might want to use it. I might want to use, you know, AGI to help me figure out my route. Right? Because at some point, it's going to be so cheap that it's kind of worth it to do. So, I do something today with the models. It's going to sound very silly. I travel a lot for work. And every time I, before I fly or I land somewhere, I tell it my schedule. I tell it what I ate. And I say, what's my best plan for dealing with my jet lag as I'm coming in this schedule where? And then I follow it.

• 1:23:46Speaker 00

  It will tell me, you know, protein meal with little carbs and sleep for 90 minutes here. And I will just follow the models. Uh, now, look, our model wasn't trained for this specifically, but it's a pretty good generalized model. And so far, it's worked pretty well for me. But I use it for that specific reason because all of a sudden, intelligence is now cheap enough for me to actually get it to do this. Now, you could write an algorithm. You could write code that takes all of these factors into account and calls the API location of the weather and things and like. Or you could just trust the model. And there's, I think these two things will, will always exist. But the code is going to be written by the model in the future. And it's choosing to do things in code because it's just more efficient and more deterministic. Or we want it to stay deterministic, like payment infrastructure.

• 1:24:29Tim Scarfe?

  Do you think we might lose something when, when we do this? I watched a great YouTube video last night. And it was, it was this guy saying, um, in the, in the days of analog recording, you know, we used to have to record things to tape. We could only do one take and there was lots of noise and so on. But it created this kind of serendipity that, you know, like, we, we can't just delete it and start again. We have to do it again. We have to talk about it and, you know, like, even this filming day to day, it's a very creative, serendipitous process. And do you think by kind of like offloading so much of the intelligent thought process to machines that we might lose something?

• 1:24:59Speaker 00

  It is worth asking the question. Do we think in the same ways today? Have, has our own thought process evolved in the same ways today as it did 100 years ago, 500 years ago, 2000 years ago? And I think there are things that, as we have learned, more representations of knowledge, of thought, how as we started getting augmented to tools, like, you know, I'm still of the age where I remember, you know, kind of pre-Google, I was young, but a pre-Google and post-Google world, right? And now, all of a sudden, I'm in a world where I'm willing to, all of a sudden, start, you know, no longer remembering certain facts. Actually, there's very few specific, like, you know, knowledge or history facts that I still remember. Well, probably, if you and I

• 1:25:43Speaker 00

  were born, you know, 150 years ago, remembering the things that we had learned from reading books and other things would be critical to us, right? And so that's changed. And it's changed because we now have a tool that we can use instead. So I might not need to know anymore the theory behind jet lag, right, and the biological process, because I trust something else to make a decision. But I think we've always had versions of that. If that was in the past, trusting, you know, the local, you know, wise person on X, Y, or Z, or the fact that we go to a doctor, like, to trust. So we'll change. Will our own thought process evolve? Like, if you give enough time over evolution of this, you know, will we become smarter? Will we become less smarter?

• 1:26:25Speaker 00

  History hasn't shown us, even though people like to say it in the moment, we've become dumber because of technology, I actually think it's made us more enlightened, it's made us more, you know, ability to explore lots more ideas, do a lot more, like, technology progress has continued to be exponential. And I think it's exciting that it continues to stay exponential. So maybe I'm an optimist, but I do think there's a fair argument to be made if you're a parent, right? And you've got a young child, and you essentially say, like, I still want you to study this, I want you to learn these things, just develop your own thinking. It's kind of a TikTok debate, right? Do you want to give TikTok the entire day to your kid, you know, and have it spend eight hours a day just on the phone? I'm not sure if that's, you know, how you want your children to grow up and develop their intelligence.

• 1:27:09Tim Scarfe?

  Would you encourage your kids today to learn how to code?

• 1:27:12Speaker 00

  Yes.

• 1:27:14Tim Scarfe?

  Go on.

• 1:27:15Speaker 00

  Because just like you would encourage your children to learn history, to understand mathematics, to learn how to program and write code. I think you want all of these things because at the end of the day, we still need to train our own intelligence as well. And so just because we're now getting increasingly capable of training models to reach our level of intelligence one day and even beyond, that doesn't mean that our experience of life should be one without us actually training our own intelligence. And I think coding and building software is a great way of training our intelligence. I today in my job don't get to write a lot of code anymore. It's kind of just the nature of being co-founder and where my role is. But everything that I learned from having written a lot of code in my life

• 1:27:59Speaker 00

  has helped me develop thinking process and understanding that I like to think make me more capable of doing what I do. So I think just developing intelligence in general is a good thing. And I think coding is a good tool for that.

• 1:28:11Tim Scarfe?

  Can we just speak about the role of multimodality? So at the moment, we're primarily talking about, you know, text in this generative AI setting. I can imagine a future where it seems like low hanging fruit, where we just record the screen and it can perhaps in an observational way, it can see the application, maybe in the future, it can interact with the application, maybe in a gentle way. Is that coming?

• 1:28:34Speaker 00

  Oh, 100%. Like look, multimodality from both visual language modeling to computer use to the work that you see that's happening in video and building world models in the world, all of that is coming. I think the question always comes down to the companies that are working on it need to work on it in terms of what their objectives are. So when I talk about right now building the world's most capable AI for software development and us being on the trajectory towards that, there are things that we care about that are massively represented in text, right, characters, language, code, etc. But also things that are massively represented in visual understanding of what's on a screen. Because applications are on screens, right, to put it. And so building the models, understanding to be able to know what's on a screen,

• 1:29:18Speaker 00

  how to interact with it, to have computer use like capabilities so that agents can go and open up the Amazon Web Service console and click around and find the data that they need, I think is critical. I think when you focus on software development capabilities, a lot can be done with text modality. With text modality, you're not going to get the model to create beautiful UIs. It's just you can't get, you know, that iteration on UI requires visual like, you know, modeling and having it as a modality. But where maybe a computer use agent is useful to go to the Amazon website and login and find the IAM role, I can also have a model understand how to do that by making an API call.

• 1:29:59Speaker 00

  And so I think we put ourselves with a bunch of guardrails by focusing on software development capabilities that allow us to focus on modalities that are slightly more compute efficient than video or image generation, and hence use our compute more efficiently to gaining to getting towards our goal. But if I was trying to build full self-driving, right, my modality would not be text.

• 1:30:21Tim Scarfe?

  I'm quite excited about a future, you know, there's like the genie paper from DeepMind, for example, and, you know, so they're talking about being able to generate video which is interactive in real time. Could we ever in the future have software which is generative? So the user interface is generative, because we all think about things differently, right? The user interface for you, you know, the optimal user interface might be different for me.

• 1:30:44Speaker 00

  I don't see a reason why that world can't exist. I think the question is where do we want it to exist? I want my Uber Eats app to look the same way every time I open it, because I build up my own, I train my own model to be able to make sure that, you know, I can find, you know, where the food is that I want to order. But there are places where I want a UI to be dynamic based on maybe the data that's behind something. I maybe want it to be more tailored towards me. But I venture to say that the vast majority of people who consume software in the world, us as humans, actually don't want our UIs changing massively on a day-to-day basis. We want a set of consistency in our UIs. I think that's just human behavior, but I don't think there's any technical reason why that can't happen.

• 1:31:28Tim Scarfe?

  Interesting, interesting. What's your relationship with the cloud providers?

• 1:31:32Speaker 00

  Publicly, we have announced in December that we have what's called a first-party relationship with AWS, Amazon Web Services. And it's quite a unique one. What it allows us is that when an enterprise customer is buying poolside, they can buy it as if it's under Amazon's paper, meaning that they're the seller of record. And so this opens up a path for large, complex enterprises to contract poolside and bring our product in as if they're adding on another Amazon service. So this hugely reduces the time that it takes to get started with us. And it also allows them to fully burn down their spend commits. Enterprises have large spend commits with Amazon. And so this is a way that we have done this for several reasons.

• 1:32:16Speaker 00

  One is Amazon's distribution in enterprises is massive. They're the largest service area of any cloud provider in the world. And if you think about our business, we are in a capabilities race. We've spoken a lot about that today, but we're also in a go-to-market race. Our ability to land customers and grow revenue allows us to invest massive amounts of capital into more compute and into more talent to be able to scale up like our model capabilities. And so these are very symbiotic and hence the relationship with Amazon, which has been amazingly exciting. We've also done a lot of work with them, not just on the go-to-market side, but on their silicon. We've done a lot of work around training them one and training them two. And so we have a fully dedicated team towards it. And we're pretty excited about what they're building there.

• 1:32:58Tim Scarfe?

  Yeah, it's great for startups as well because you get a bunch of free credits with Amazon, so presumably you can just use the free credits for your service.

• 1:33:05Speaker 00

  I think we're at a scale of compute where credits is not what makes the difference anymore, to be very honest. But for our customers and others, it's absolutely... Look, it's part of it, no doubt. I think when you're... Today, we're very much focused on enterprises and who we work with. And what we see for them is where you're deployed makes a big difference for their security profile. So we deploy inside Amazon Web Services, VPCs, in that private account. And the model waits and the full stack live there. And the combination of that really allows enterprises, they get comfortable with the model is going to their data instead of the other way around.

• 1:33:45Tim Scarfe?

  Interesting. And just from a sort of getting started point of view, so you know, folks want to have dedicated hardware, do you tend to find that your customers have a centralized model where they have one shared implementation? Or is it more complex than that?

• 1:33:58Speaker 00

  It really depends on the enterprise. A lot what we find is that inside enterprises, there are security boundaries that mean that they need multiple instantiations of the model. So if they're fine tuning a model on their proprietary SDK that they have internally, but that only covers, you know, one business unit, and that's not, that model is not allowed to be shared from regulatory compliance reasons with another part of like the business. You might find organizations that want to spin up many versions of poolside models that get fine tuned in many different environments towards certain use cases. From where we sit, we see all types of complexity, from on-prem to VPC to different model instantiations, different security boundaries, different onion layers of access rights. And we've built for all of that.

• 1:34:41Speaker 00

  Like it's, we talked a lot about the model today, but the sheer level of engineering we've had to do to be able to build, to succeed in defense, in government, in financial services, has been actually quite a lot.

• 1:34:51Tim Scarfe?

  And finally, do you have a forward engineering team just to help folks get up and running with the infra and the architecture?

• 1:34:56Speaker 00

  Absolutely, yeah. So we have the poolside solutions architects that are like fully able and willing to spend a lot of time with our customers. What we have increasingly done is that we've increasingly built towards a managed install approach. So that if an enterprise is going to give us temporary access rights, very, very limited access rights, we can spin up the entire infrastructure of poolside in their account. So what before might take, you know, several days of effort can now be done actually in under 40 minutes by just providing a single IAM role. But there's enterprises, there's always going to be something that you are going to discover along the way of a massive deployment, a firewall that sits on the network somewhere that needs a certain setting tweaked. And that's of course where often our solutions architects are very helpful.

• 1:35:40Speaker 00

  But also just in terms of helping them think through, you know, as we're making these models their models, as we're fine tuning them on their data and making them more capable in their environment, helping them think through about what data, you know, which groups of engineers to deploy what versions of models with, how to measure the impact, because we offer quite a lot of metrics that are available to the customer to see how the impact of the model is towards acceptance rates, towards how many changes are reviewed, but not, you know, actually not applied, how many lines of code are actually applied. And so we do a lot of that work and we try to help our customers think through it. Our commitment is to become the trusted partner of enterprises that as intelligence gets more capable, they'll want to scale it up with us.

• 1:36:21Tim Scarfe?

  Izzo, it's been an absolute honor. Thank you so much for joining us today.

• 1:36:24Speaker 00

  No, thank you so much. It was a pleasure.