Why Your GPUs are underutilised for AI - CentML CEO Explains

Prof. Gennady Pekhimenko (CEO of CentML, UofT) joins us in this *sponsored episode* to dive deep into AI system optimization and enterprise implementation. From NVIDIA's technical leadership model to the rise of open-source AI, Pekhimenko shares insights on bridging the gap between academic research and industrial applications. Learn about "dark silicon," GPU utilization challenges in ML workloads, and how modern enterprises can optimize their AI infrastructure. The conversation explores why some companies achieve only 10% GPU efficiency and practical solutions for improving AI system performance. A must-watch for anyone interested in the technical foundations of enterprise AI and hardware optimization. CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Cheaper, faster, no commitments, pay as you go, scale massively, simple to setup. Check it out! https://centml.ai/pricing/ SPONSOR MESSAGES: MLST is also sponsored by Tufa AI Labs - https://tufalabs.ai/ They are hiring cracked ML engineers/researchers to work on ARC and build AGI! SHOWNOTES (diarised transcript, TOC, references, summary, best quotes etc) https://www.dropbox.com/scl/fi/w9kbpso7fawtm286kkp6j/Gennady.pdf?rlkey=aqjqmncx3kjnatk2il1gbgknk&st=2a9mccj8&dl=0 TOC: 1. AI Strategy and Leadership [00:00:00] 1.1 Technical Leadership and Corporate Structure [00:09:55] 1.2 Open Source vs Proprietary AI Models [00:16:04] 1.3 Hardware and System Architecture Challenges [00:23:37] 1.4 Enterprise AI Implementation and Optimization [00:35:30] 1.5 AI Reasoning Capabilities and Limitations 2. AI System Development [00:38:45] 2.1 Computational and Cognitive Limitations of AI Systems [00:42:40] 2.2 Human-LLM Communication Adaptation and Patterns [00:46:18] 2.3 AI-Assisted Software Development Challenges [00:47:55] 2.4 Future of Software Engineering Careers in AI Era [00:49:49] 2.5 Enterprise AI Adoption Challenges and Implementation 3. ML Infrastructure Optimization [00:54:41] 3.1 MLOps Evolution and Platform Centralization [00:55:43] 3.2 Hardware Optimization and Performance Constraints [01:05:24] 3.3 ML Compiler Optimization and Python Performance [01:15:57] 3.4 Enterprise ML Deployment and Cloud Provider Partnerships 4. Distributed AI Architecture [01:27:05] 4.1 Multi-Cloud ML Infrastructure and Optimization [01:29:45] 4.2 AI Agent Systems and Production Readiness [01:32:00] 4.3 RAG Implementation and Fine-Tuning Considerations [01:33:45] 4.4 Distributed AI Systems Architecture and Ray Framework 5. AI Industry Standards and Research [01:37:55] 5.1 Origins and Evolution of MLPerf Benchmarking [01:43:15] 5.2 MLPerf Methodology and Industry Impact [01:50:17] 5.3 Academic Research vs Industry Implementation in AI [01:58:59] 5.4 AI Research History and Safety Concerns

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Tim Scarfe?

  Let's say if AWS gave you $5 billion to be exclusive on their platform, would you be interested?

• 0:10Speaker 01

  I think a few people I really respect were quite technical, like Bill Gates or Mark Zuckerberg. They don't necessarily have fancy degrees, but they just see the opportunity drop out and went for it. I think that one of the success of N V is actually in their engineering structure. There is way less bureaucracy in between the layers, and it means they're still operating closer to their startups. So for the company at that scale, the way they organize, it's very impressive to me. I use G P T two myself using Python interface, and I've seen how cool those models long time ago. So essentially, whatever systems were built using A I under the hood has to have that interface. It's very easy for people to manipulate.

• 0:51Speaker 01

  What is also challenging here is that when we talk about power limitation and dark silicon, if you actually make everything to run, you might actually overheat this thing because it won't be able to dissipate the heat as well. But that's where the lot of opportunities there, untapped potential that we at Center Mal in our company actually use for the benefit of our users. Center Mal is the company that focuses on optimizing machine learning workflows, both training and inference. Our focus is to make things easy to use, cheap and efficient for you.

• 1:23Tim Scarfe?

  You cited, you know, Zuck and Bill Gates. You could argue that they've become less technical over time, and is that because they are communicating with stakeholders?

• 1:33Speaker 01

  Wow, great question. So Center Mal is the company that focuses on optimizing machine learning workflows, both training and inference. Our focus is to make things easy to use, cheap and efficient for you when you try to build a new model, fine tune existing foundational models or deploy the models of your choice at scale.

• 1:56Tim Scarfe?

  Do you struggle being, I don't know how to say this, but you're by far the most technical C E O I've ever seen in my life. Is that something that, do you struggle with that? Because you're the guy who's normally, you know, down in the details, you want to know how everything works. And now you're kind of like trying to coordinate everyone together.

• 2:19Speaker 01

  So far so good, I actually think that mostly give me an advantage if anything, right? It's like I can save time of my CTO in many meetings because like if they just need to understand what we're doing in general, I can handle most of those questions, right? So they can focus on actually building things. Uh, I think I partially as a, you know, as a life of a professor, you need to learn some of the business aspects of fundraising and like, like, you know, built in teams and all of that. I've been to industry, so I think it helps me to see it kind of in both camps, right? And I do like uh that in conversation with both customers and investors, I can handle the technicality as usually as much as I can go, right? So I can handle pretty much any technical question.

• 3:03Speaker 01

  So I like it, but I can also cover everything business related. So I like they had a good enough understanding. I'm not a replacement for the tech team, but I understand uh what's doable, what's not. I had a good grasp of the uh what our team can build.

• 3:20Tim Scarfe?

  In an ideal world, do you, if it were possible, do you think every tech CEO should be as technical as you?

• 3:27Speaker 01

  Uh, that's too much to ask, but I think a few other people I really respect were quite technical, right? Like just like look at like Bill Gates or Mark Zuckerberg and uh um you know, like quite a few others that were successful were actually quite technical. They don't have had fancy degrees, but they just see the opportunity drop out and went for an opportunity, right? So, but I think they're very technical and they do understand that space. And I actually think it's very important for the success of the company in this space to be technical, like, like look at Apple. You know, CEO like, like at the beginning, it was all very technical, right, really? And they do understand the space and the customers more or less equally, right? So it is important.

• 4:10Speaker 01

  It's not mandatory, like, like you can do the same with a very strong CTO, VP of Engineering, but it's a good feature to have if you can have it.

• 4:20Tim Scarfe?

  What are the forces that erode that over time? Because you cited, you know, Zuck and Bill Gates. You could argue that they've become less technical over time. And is that because they are communicating with stakeholders that have a different kind of horizon of understanding? You know, is that a pernicious inevitable thing? Or do you think you can kind of hold on to that very high resolution view on things?

• 4:43Speaker 01

  It's almost inevitable. I see it with B as well. So the minute you start to write code daily, you start to get a little bit disconnected with what's possible, what not. Not to the level you're completely lost, but it's never the same as what the engineers have, right? But like you can have both zoom in and zoom out at the same time, right? Like you need to have some, you know, trade off. And so I think it's inevitable. Even in my academic career, I at some point I had to admit, I can't look at every line of code that my students are doing, right? I can't code with them. So I need to trust what they do. It's the same as with any management role in general. So I think it's a normal path and the bigger the company grows, the more and more it is. I believe when I was young, we were like learning about how Bill Gates, for example, managed the engineering teams.

• 5:28Speaker 01

  And at the beginning, they, you know, testing was literally, he was looking over every single line of code that goes into the product. Obviously that doesn't scale, right? So you need to know how to trust others, how to build the team that scales and still successful and delivers things. But you've been able to understand in a reasonable scale, what they're doing, still very valuable.

• 5:51Tim Scarfe?

  Do you see that kind of organisation building as a, as an engineering problem?

• 5:56Speaker 01

  Oh, very good question. I actually think, yes, it is. It's a very interesting problem in general. I had a lot of thoughts about it. Usually right now, they're not my main focus because I had other problems to solve. But like I'm, I really admire, for example, what Jensen did at Nvidia. Like, like, look, he's managing one of the flattest structures, uh, engineering wise and then general in the company. And when I met with him and like asked about how do we manage this, it's like, oh, like I've had like 50 direct reports or something like that, right? It's, it's a very different scales. Like it's very hard to manage. Ten people, like fifteen people is more uh more or less the limit, right? And you know, he manages that many. So it's a in very interesting scaling problem. You definitely cannot do like one on one meetings.

• 6:39Speaker 01

  You need to figure out how to meet with the teams instead. But people are doing that and they're very successful. And I think like one of the success of Nvidia is actually in their engineering structure. Like in general, uh company structure, they actually uh very flat, right? And it means that a lot of people know what others are doing. There is way less bureaucracy in between the layers. And it means they're still operating closer to their uh startups than a lot of other big companies. So for the company at that scale, the way they organize, it's very impressive to me. So yes, I would like to laugh, not necessarily have fifty direct reports, but I do want to have more flat structures. Very important to be efficient. I see how extra layers easily can make things complicated.

• 7:24Tim Scarfe?

  Are you, I don't know whether it's fair to say this, but if Nvidia has that kind of structure, it's it's a bit more centralized. And we'll talk about this a bit more later, but do you think that there's something good about having centralization?

• 7:41Speaker 01

  Very definitely good thing. That's a very old debate about being fully free and decentralized, right? Like like like some company, I don't know, I think Google comes to mind as an example of a more decentralized structure or something more centralized. I think centralized is very efficient where you're on a mission to deliver something like usually startups are more this uh centralized, right? You had a very limited amount of resources vision and the main advantage versus others is the speed, right? You can go much faster, much more efficient than others. So this is why the decision has to make the right uh the right way and you don't have the chances for mistakes, right? So in a way, for some critical features, that's really a requirement, right? Again, long term is just very hard to maintain it, right?

• 8:25Speaker 01

  I don't think others didn't want it. It's just very hard to keep to leave at that pace and like of kudos to Jensen to be able to keep doing this over so many years, right? It's really taken a toll on you, I'm pretty sure, yeah.

• 8:40Tim Scarfe?

  Cool. Well, in which case, we should slowly kick off the interview. So I want to introduce the audience to Gennady. As I was just saying, probably the most technical CEO I've ever spoken to. I think it's remarkable, actually, especially when you when you when you look at the research that he's done. But Gennady is the co-founder and CEO of CentML, an associate professor at the University of Toronto. I love that university. I was there recently interviewing a couple of guys. And you're a faculty member at the Vector Institute, research co-chair at ML Commons, a founding member of ML Perth, and you've also been a researcher at Microsoft. You've got a PhD from Carnegie Mellon University. You've had internships at Nvidia at Microsoft Research, and you're at IBM.

• 9:23Tim Scarfe?

  And I mean, basically, you've done a lot of stuff, which is which is really cool. Welcome to MLST. Thank you so much for coming.

• 9:30Speaker 01

  Yeah, well, thanks a lot for inviting me. Really excited to be here and to talk with you.

• 9:34Tim Scarfe?

  Very cool. Okay, so why don't we let's start with the low hanging fruit. How do you think open source models compared to proprietary ones?

• 9:45Speaker 01

  Well, it's a very good question, and definitely there are a lot of different opinions here. What I can say from my experience and my perspective is open source models are really improving over time, like they're really getting closer and closer. There are a lot of different benchmarks, to the extent we can benchmark things and assess things, they are closing the gap over time, right? And I actually think it's good for the society and the IT space in general, like it's nice that the models are accessible to a wider community of developers and people in general, that they can use it and build on top of that, right? Why this happens, I think it's fundamentally when it was clear that AI is so important

• 10:30Speaker 01

  to the world and the value generates are all in front of us. There are a lot of people that got interested. And obviously, well, so many people got interested, they start to be creative and add value. It's very hard to be just this one company that is go ahead of everyone and release it to the rest of the world, and the rest of the world learns from you. And doesn't catch up like like no matter how smart are you people, the rest of the world is so much larger. And people seen this through many other software product, like the minute you make it available to the world. It is really important, the world will catch up quickly to you. And I think that's what happened right now. Yes, there are a lot of great proprietary models right now. But models like open source model like Llama and Mistral also are very high quality.

• 11:15Tim Scarfe?

  I mean, what what kind of advances do you see in this space? I mean, I'm I've been reading about, you know, there was that paper talking about R N N's are coming back. There was that Min L S T M and and so on, and we've had Mamba and state space models. I mean, what do you think we're going to see any big shifts in in this space?

• 11:34Speaker 01

  Um, the shifts are always possible, but I had multiple conversations with many prominent people in the AI space and the pure AI field. I don't see any obvious replacement for like attention based blocks that use in transformers, right? So I think a lot of focus right now is actually in building on top of that architecture, right? So and there's always a chicken and a problem this, you know, this basic concept. Ended up being really well capturing the different properties of the data we're dealing with. If it's good enough and if hardware and the software evolves around it, it becomes like almost like a dependency that it will sustain being around and like it's not going to easily disappear.

• 12:18Speaker 01

  There are several different types of models that people suggest in building startups, but I didn't see any of them as a clear replacement. Whenever they really need to compete in the benchmark, they really usually use some form of attention mechanism anyway, right? And there is still plenty of room to improve around attention and transformer based models, so I'm not particularly worried about that we're stagnating or anything. I'm more excited about what people are going to build on top of it because I was this at this event last called Intelligent Applications, where we actually as a company won one of the Rising Stars Awards. And it was very clear, people excited and see beyond the models. There were several fireside chat there.

• 12:59Speaker 01

  Saying like it's not just models, it's about building those sophisticated systems on top of the models that had multiple different components, that is very critical. So it's like a core foundational models are still there, they're not going to disappear, they're important. But people are looking, what do we do next on top of it? So like say one year ago, it was fine tuning, people were excited about it, then it was ragged. Now people talk a lot about agents and compound systems like that, the bricks talk about it. This is very exciting things and I expect that will keep moving. So people build not just model, the new model, people build applications, something that the world can use right around it. And I think that's what's even more exciting than building a pure new model right now.

• 13:42Tim Scarfe?

  Yeah, Francois Chalet's term is this progressive disclosure of of complexity and and I love that. But of course, you're a guy that's at the bare metal. So you know, you're you're building compilers, you're building kernels, you're you know, you're working with the hardware, you know exactly how the hardware works at the very, very lowest level. But you're still talking to this sandwich that we're building. So we're going to have abstractions on top of abstractions, on on top of abstractions to build these systems.

• 14:10Speaker 01

  Yeah, you're absolutely correct. We had and my team and I'm personally had a lot of background in computer architecture, systems and building many different layers of the software stack. But I think this is our expert, that's what we're good at. But what the world consume is the actual applications, right? I think that's ultimately where the value is coming from. People were excited about the models, right? And some pieces because that layer never existed before. Out of a sudden, we get another knob to play with. And obviously, there was a lot of excitement. We spent several years building those models. But then we reached the situation where it's okay, it seems like to find something that scales well. So let's kind of start to build on top of it, right? And I actually think whatever we're building that sandwich, how you call it,

• 14:54Speaker 01

  would be valuable, no matter what the model is. Like tomorrow, someone would replace like attention with another model, I don't know, like, you know, space, you know, like some of the space model that people discussing right now, like SSAMs and others. Say this will become a new thing. But that would be just a block in replacement, right? You would just replace one model in the sandwich with another block in the sandwich. Yes, it would change hardware utilization, but mostly that would be visible by us hardware people, compiler people. The rest of the world don't even need to worry about it, right? They would just get a better, high-quality responses, right? So it's very important to me that we don't stick to the model alone in terms of what we contribute, that we contribute to the rest of the stack. People still can evolve new model, build new things.

• 15:37Speaker 01

  But if the stack is mature enough that people generate value, this is much better, right? Because if we only all invest in building better and better foundational models, at the end of the day, there's no use case that people benefit from it, right? We're just going to burn a lot of resources and not having a really tangible outcome, right? So I'm not worried about the new model will come up. I think the system-wise, we're going to be much better, right? Ready than we were at the beginning.

• 16:04Tim Scarfe?

  How much coupling is there in the stack? You know, we like to think that we're building these little decoupled atoms of computation and, you know, we can just kind of build on top of it and build on top of it. But there must be some very complex behavior between, you know, what emerges and what properties the overall system has as a function of swapping out those components.

• 16:26Speaker 01

  Well, that question has many different layers. At a very low level, our agents, like what's running under the hood are the kernels, right? Like, and that's running on the actual hardware. So the entities there are the chips, and this chip communicated with some formal communication medium. So it can be, you know, either internal interconnect, like something like NVLink, right? Or it can be something that connects multiple nodes. It can be Ethernet or like Mellanox cards or something like proprietary optical interconnect that, you know, Google is using right now.

• 17:01Speaker 01

  At the end of the day, at that very level, those entities is like it's a hardware business, right? We program them in a certain way and their interactions are very predictable, right? They know they behave in the way we program them to behave. There is certain level of complexity with them. There are not a lot of uncertainty though, but those are very repetitive. Like if you set up a training workload or an inference workload, like even though you run billions of examples, all the rounds are more or less identical. There's a very little fluctuation. There are certain things like frequency that can fluctuate, but almost all the other things are fixed. So here, after we program the chip using either manual effort like or the compiler, things are predictable.

• 17:43Speaker 01

  But it's very exciting to think about what people can do with that hardware and the software stack. That is well less predictable. So the compound system or the general systems built out of agents and other, you know, players and models can be very unpredictable and very exciting. So this is a very open field. And I'm not going to pretend to be an expert there, but I think it's a very, very promising field, where it's even hard to imagine where the limits are. With the hardware, we know that there is a certain limit and what we can achieve with that technology. After the chip is fully utilized, that you're at max, right? Like, and even if you had eighty percent, you know, you cannot go jump way more far. But it's how the hardware is used, the sky is the limit. There's no limit.

• 18:26Speaker 01

  So we just need to make sure that people's creativity is not limited by the complexity of that stack. Just make sure that even kids can experiment with it. So essentially, like, I think many people said it, it was the beauty of ChatGPT. For example, for many people, they're never aware of the technology. Like, I use GPT two myself using Python interface, and I've seen how cool those models long time ago, right? Not necessarily as good as GPT three, but very, like, very powerful, right? But for the rest of the world, that was all mind blowing, because the interface was a common line, similar like in Google search, right? So essentially, whatever systems were built using AI under the hood has to have that interface. It's very easy for people to manipulate.

• 19:10Speaker 01

  And it doesn't have to be just a Google search, like it's our voice. It's like us manipulating with the rest of the world, potential augment in reality. All of that can be really nice inputs, right? And this is where I think this becomes very open-ended and very unpredictable on what humans can build with it.

• 19:28Tim Scarfe?

  Just a quick point on the predictability, you know, down at the hardware level. I interviewed someone from Co here for AI ICML and they've got a paper out saying that there are differences when you run models on different hardware. I don't know whether you've seen that and apparently the differences can be significant, especially on the long tail, you know, for things like fairness. Have you seen anything like that?

• 19:51Speaker 01

  So you mean that the results are different depending on which hardware you run it on? Yeah.

• 19:57Tim Scarfe?

  Yeah, the same model on different hardware.

• 20:00Speaker 01

  There are two things here that a lot of people don't realize, let me clarify it. First of all, the way we run training right now, even on a GPU, forget about different hardware, just like canonical GPU, it's quite non deterministic, right? There is a component there that would, if you run the experiments multiple times, it would ended up being different. What people don't understand and why that is fundamental to the way we manipulate was floating point numbers, right? So it's like every time you run, if you add up the same numbers in a different order, that the result would round it and be indifferent. And stochasticity of the neural networks would make it go one way or another way very easily because it's very multi dimensional problem. So it's like you're running the same experiment on the same hardware, it can go different.

• 20:44Speaker 01

  Between different hardware is even more differences because they would easily use very different precisions. So I don't think that, for example, we run experiments, my both research group and the company, we operate on Nvidia GPUs, AMD GPUs, Google TPUs, Amazon Trainium chips. Fundamentally, you can get convergence on the models very similar in all these models, right? But you need to be careful with the precision and everything, right? So I don't think the hardware is guilty of anything. Hardware does what you build it for, right? And it usually has certain level of precision. But if you pick a chip hardware with low precision and not very careful about the convergence, you might have very different results. I don't think the hardware fundamentally changing the way the computation is done.

• 21:29Speaker 01

  Yes, computation is not fully deterministic, right? So because of that, you cannot expect a completely identical results. Like if you ask the same question to chat GPT, it's not going to ever respond the same way, even if it uses the same chip.

• 21:45Tim Scarfe?

  Yeah, it's really interesting because you know when we build normal computer applications using C++ or whatever, we say to ourselves, oh, you know, we can verify it, we can test it, it's deterministic and so on. And that may or may not be the case. But now we're building the next generation of applications, we're building AI applications and they have increasingly sophisticated behaviour and we grow them, we don't design them and we're starting to build agent-based systems and so on. I mean, how does that look to you?

• 22:15Speaker 01

  It's a very interesting problem and challenge at the same time. Think about the very critical areas like automotive, for example. They prefer to have proofs for every little piece they build into the car. And now you give them a piece that says, oh, it might be non-deterministic, right? It might change this behaviour. So it is challenging and why it's like using AI in some of these areas is very difficult, like self-driving, for example, right? So again, we need to find the way where we keep those errors within boundaries. Or, for example, we make sure that we use a precision that deterministic or we build, we limit some performance characteristic of the chip, but making sure the threads run on the same order. So everything is in the same order and you use int eight, like integer precision instead

• 23:00Speaker 01

  of floating point. And then things become out of sudden deterministic, right? You need to make certain changes, but you lose a lot of performance if that happens. So in some critical areas, that might be the outcome. In other cases, I think we just make, to make theory around neural networks in general more robust, so that we're more susceptible to noise. And there were several really great papers I've seen in the community where people try to stabilize the training process and the model and be less sensitive to noise. So that's in general, like there are several problems that need to solve. So reliability of those results, reproducibility, and you know, all of that is really, really critical.

• 23:37Tim Scarfe?

  Just coming back to the open source thing quickly before we move on. And what do you think are the sort of like the implications for using these models like Lama and Mistral, et cetera?

• 23:49Speaker 01

  I do see a huge value for them, and not only because it's very valuable for me and a few other companies like mine that actually using them. And I think it's very valuable for our customers. Why I see so? So a lot of their enterprises, I think they are critical for AI to actually generate significant value for the world. So again, when OpenAI or Google benefits and invest into, you know, a new big, nice, shiny toy like AI, they can burn billions of dollars and there's no outcome. I think for me, unless there is a benefit for this second wave of adopters, like Fortune 500 and companies, for example, that's not really a success yet.

• 24:35Speaker 01

  Those companies obviously don't have as much expertise in AI, like as this company I mentioned was the first wave. So they probably, most of them, not gonna just build their own foundational models. And they also had a lot of their own sensitive data. So if you combine those two things, you see that they either have a choice, either give the sensitive data to someone else, like another hyperscaler or a startup like an OpenAI, or you actually leverage the open source model and go the path that require way less expertise, like fine tuning or rack with their own data. This way, they would generate way more IP, right? And I think more and more companies right now go into that direction.

• 25:18Speaker 01

  So that's why having models like Lama and Mistral, not only to exist, but also had performance on par with like models like GPT-4, right? Is very critical because people can't build their specialized models on top of it, right? And not depending on any particular company invent this technology. They can build their own IP and they feel much better about because they control it, it's internal, right? So they don't feel their business is fully dependent on that, they had to give customers. Data to something that's very like, you know, uh, less reliable. So that is making easier for the enterprises to adopt this technology. And it's very important because they would see all this risk before it took them a long time to uh adopt like using cloud.

• 26:03Speaker 01

  And not all the data is still in the cloud, as you probably know, right? And again, the same would be with AI, like people want to be sure that we can benefit from that technology ourselves. And we can build an IP on top of it. So I think having open source model is super critical for that. So model like Llama, Mixtral and others and Falcon are very, very critical for this trend to exist and actually the people to benefit from it.

• 26:30Tim Scarfe?

  But what do you think the performance gap is? It's quite interesting that OpenAI, they've shifted the narrative from we're building AGI systems that can reason and do anything. You know, on on the new version, they they've they've released a fine tuning API. And they're they're kind of hinting that actually, if you give us a thousand examples of like some labeled imagery, it'll do better. And of course, it's the same for Llama, right? You can just fine tune it. You can just give it a thousand examples of of your thing. And and I guess by increasing the specificity, you can make it for that application every bit, maybe even more reliable than GPT four O would have been.

• 27:10Speaker 01

  Yeah, so it's like obviously like everyone's gonna adjust to this reality and they're opening your adjusting as well. It's ultimately boils down to whether you are comfortable giving this data to them. So uh visual data points, it might be sensitive data, not sensitive data, it depends on what the model is. If the people know that exists the open source solution that's reliable and verified by the community, that that can actually be used instead. If the quality is comparable, you would just probably go with it, right? Why take in for an extra risk, right? At least that would be my choice. Uh it would be a different story if there would be a huge performance quality uh gap there.

• 27:48Speaker 01

  But I think this gap, as I said before, is shrinking every single time. Right now, it's even hard to say there's like a significant gap between like uh like some benchmarks. Llama three was as good or even better than G P T four. Obviously, they also keep improving, but Llamas also keep building new model. What I see in the last couple of years that that gap is always shrinking, right? I don't think this gap is growing, this gap between open source model is shrinking, right? And there is enough players here that could keep it exciting.

• 28:16Tim Scarfe?

  What's your take on O one preview? And I don't know if you have any interesting intuition on how it's done, but you know, people are talking about it being process supervision or something like that. And do you think something like that could be done on a model like Lama?

• 28:32Speaker 01

  It's a very good question. I think it's definitely a step forward. It's a very nice model. Like I play a little bit with it as an amateur, as a user, right? Without going deeper. It's interesting that some things it's like it's taken many, many seconds to think and give you a very basic answer that like G P T four would give you much better, right? Yeah, so I think that's quite funny. But and there are some other logical problems. It's very interesting like on how it can do it, right? So again, to me, it's a step forward, but it's very hard to say how much of a gap it was the rest of the world is there.

• 29:08Tim Scarfe?

  You know, like there were, there were these scaling laws and you can introduce what, what those are. But they, they seem to suggest that there's almost no ceiling. We can just scale the data, we can scale the training, we can scale the model size. And, and now they're saying we can just scale the inference compute and, and you know, there's almost no limit to where it can go. I mean, what's your take on that?

• 29:27Speaker 01

  Well, I think part of that is sad and done more for investors than for engineers. And scientists will understand that all this exponential loss has limits, right? And you're gonna hit them relatively fast usually. So I think reality is that training scalability laws start to diminish in a way, right? We're limited by the amount of the data, people already start to use synthetic data. We don't have sometimes enough compute and we understand that a lot of the benefits of adding extra data, if it's not high quality data, become very diminishing or even hurts, right? So it's like the, the big is not always bad. I think I've seen one recent benchmark, like literally today in the morning.

• 30:07Speaker 01

  Well, Llama thirteen billion model, the older version, has a worse performance than Llama one billion model, right? So it means like that it's not just about the scale that matters, the quality of the data, cleaning, like, like, like, how you train and post-training process matters a lot. Same for inference, I don't think it's an infinite scaling going on here, right? It's ultimately, uh, yes, you can scale the model further, keep getting improved the latency, but the question, do you really need it? Like, I've seen some of this benchmark saying, I can speed up, you know, the inference for Llama model by another like fifty X.

• 30:47Speaker 01

  But when I look at the baseline, it's already at the human, you know, tectonics faster than we can ever read, right? Like, what's the benefit of that, right? So it's very important to put things always into perspective. What's the benefit of that? For me, it's way more impressive if you not just improve the raw latency, right, on inference, but you could actually do it the most cost-efficient way. Because at the end of the day, that's a very, very important factor into adoption of the models. So it's not a matter of like how far we can scale the inference and the models and how big a models we can still run. It's a matter of like whether those models actually do the job well and whether you can get that in a cost-efficient way, because that's what the customers want. And then at the end of the day, they're not going to run the largest model available.

• 31:32Speaker 01

  They're going to run the best, the smallest and the most efficient model that would do the job for them, right? And that's what's the most important one. And I think that's not covered as much because that doesn't sound as sexy as like talking about the biggest possible models. But at the end of the day, this is what the customer wants, like they just want to make sure that they get the best quality uh, for the, the minimum amount of money.

• 31:58Tim Scarfe?

  Yeah, I mean, I was watching Lex Friedman interview the, the cursor team the other day. And it's really interesting hearing about how they designed a whole bunch of the models. So there's like, there's a tab model and there's a, there's an apply model. And naively using it, I thought, oh, they've just used like a big model and they're just kind of generating the entire code file. And for that reason, I thought, yeah, it'd be great. If we have more TPS because I can just generate my code faster. But apparently to make it work reliably, they've had they've had to do all these ridiculous optimizations. They're using like speculative decoding and they're like partitioning the code file up into all of these different tiles. And they're running it in parallel. And even in speculative decoding, you're like running it in parallel with it with a smaller model. Because they say code is lower entropy or whatever. So it's just an example that, you know, we have we have this notion in our mind that, oh, yeah, we'll just have a big model.

• 32:42Tim Scarfe?

  And it will just do all of the things. But actually, a lot of the innovation is when you take a specific task and you just optimize the hell out of it.

• 32:51Speaker 01

  Yeah, yeah, it's a lot of is really exactly what you said. It's because ultimately, this is where the value is coming from. The value coming is not from, you know, randomly get one metric selected and benchmark it to death. It's actually working under multiple constraints. This is actually much harder. Because if you just had one metric, it's much easier to optimize than you say. Oh, you need to be fast enough and has and good throughput and the cheapest possible and low power and low emissions. And like all this constraint exists, right? And ultimately, when all these things go to scale, those these constraints will be there, right? We already see the power limitations, limitations. People would have to obey those limits, right?

• 33:33Speaker 01

  So you actually need to build a system optimizations, knowing that that constraints are coming, right? And it's like, in a lot of reality cases, it's not like, oh, what's the best I can do on the best, the most expensive available cheap? The customer would say, I have some compute on prem. I would have some compute in the cloud. And I have all these different workloads given me by my data scientists and in product teams. What's the best way for me to run them? Whatever is critical to run on prem, to run on prem. Whatever can write in the cloud is got overflow to the cloud. And whatever is free to go anywhere would go there. Like how you orchestrate all of that and how you deal with like, that's complexity you need to solve. And this go way beyond just optimizing a single model for single metric, right?

• 34:17Speaker 01

  And this is what we are building in the company. Like we are trying to get people the help of addressing multiple different complexity of metrics. This is what a typical. Enterprise or mature company would work rather than like someone that just planned with one model.

• 34:33Tim Scarfe?

  So I interviewed Sabaro Kambahati and he was like one of the biggest L L M skeptics. He had all of these papers out, you know, testing self verification and a chain of thought and so on. And he was using this blocks world problem, which is a planning problem. And he found that when you do mystery blocks world, so you change the tokens, it's the same problem, but you just randomly shuffle the tokens. Or you put like random descriptions in the tokens. All of a sudden, the models don't work because he was saying they're basically kind of, you know, approximate information retrieval systems. But on O one, he seemed like a changed man. I was surprised. He was, he was kind of saying, oh, look, they, they now have non trivial performance. They're solving blocks world problems, sometimes many steps ahead.

• 35:15Tim Scarfe?

  And I've tried it on some problems that are NP hard and it's still, you know, for a very small size of the, of the problem, it still did something. So it's doing something like what, what is reasoning and what is it actually doing?

• 35:30Speaker 01

  I think it's a complicated problem in general. I think there is definitely was one thing that was missing from a classical L L M's to me. Essentially, there was this problem of what you can memorize from the world. Like L L M is like a great filter that memorize a lot of things from the natural language. But to me, it never actually could reason in any complicated way. Like the good examples I like, I've seen in some of the presentation was like it cannot do like three digit multiplication well. Like good luck doing it with L L M, right? It would memorize a lot of corner cases, but it cannot do the proper multi step reasoning. And that was missing. And there were people that always tried to build the AI around the reasoning aspect.

• 36:12Speaker 01

  But it felt like there were two separate camps that don't really talk too much to each other. But right now it starts to come together nicely. And I think that's probably a good step in the right direction here with this O one model, right? There where yes, it's, it's for some very basic things, it's taken too long of thinking. But at the same time, it actually starts to look more human in a way that they actually start to do some basic reasoning. Right, which which is good, right? It's not just memorizing things and just filter it. Because even our brain, it's not everything is so sophisticated, or some all the parts of the brain, similar to like animals, we're just doing like face, uh, you know, object detection, right?

• 36:52Speaker 01

  And that has to be, you don't need any sophisticated reasoning there, like you need to do it very fast or it won't survive. Right? That that's critical. And there's a parts of the brain responsible for that. But there are other parts of the brain that develop much later in evolution that actually can do basic math and things like that, right? So that one would take much more process. And we see like how we teach our kids, like if I have small kids at home like I do. Like you would realize that it's like they would learn object detection very early in their life and they're very good at it. And they don't need one billion examples. You just show them a couple of examples and they can get it, right? That's another, you know, direction for improvement for for, uh, you know, uh, for those models. And but like teaching them math takes many, many years, right? It's it requires way more examples.

• 37:37Speaker 01

  Such that it can solve any analytical problem and it's very like long term process. So I think it's the case here, we need to develop E I that would do this. I think in addition to this reasoning capabilities, what's missing to me in modern E I is connecting it to the rest of the world. I think that what slowed down the training process right now, it doesn't have the context. It doesn't understand what the world is like. And because of that, it can do very basic mistakes and all these hallucinations could have been avoided. If there, uh, the model would have ability to have a sensors in the world and collect the inputs and adjust accordingly, right? It doesn't have that experience and because of that, a lot of things look like stupid or dumb like still to this day.

• 38:19Speaker 01

  I think that's another missing point. So like reasoning capabilities definitely, but also ability to connect with the physical world that we as humans have. As we add more sensors and ability to sense the world in the process and having a notion of the world somehow as a part of the model uh, and physics of it, it would help tremendously to improve those models quality.

• 38:41Tim Scarfe?

  Yeah, I completely agree with you on the importance of embodiment. I think it comes down to the semantics gap. So clearly, we have access to richer semantics because there's a huge causal chain of things that happen outside of us and that helps us understand and actually cognize. But the other thing is that there's a real problem with ambiguity that I see in these language models, which is that even with O1, sometimes it can get the right answer, but it's simply because it hasn't understood you because you get better at prompting over time, of course, so you learn to kind of specify, oh, the tiles can only move this way and they can only move between adjacents and you kind of get better at it.

• 39:23Tim Scarfe?

  But we have this remarkable ability to deal with ambiguity. It's a real hallmark of human cognition and I'm not really seeing that yet because I think when we do overcome this kind of semantics gap and ambiguity gap, I think potentially the models could be more autonomous. But the other thing I wanted to comment on is as well as this kind of semantics gap, I think that there might be a computational gap. And my co-host Keith Dugger is always at pains to point out that these things are not Turing machines, you know, like there's a kind of difference in kind with a Turing machine, which means there is a space of computation that they can't learn with stochastic gradient descent, much less perform. I mean, do you think there's some weight to that?

• 40:05Tim Scarfe?

  Or are you one of these people who just said, well, in practice, it doesn't matter because we can just build systems which are Turing complete?

• 40:13Speaker 01

  I think probably more on the second camp, I don't see the fundamental limits on what cannot be learned, right? I don't think we had a computer inside of our brains, we just had a very good adjustable system, right? And I don't think there is anything that we have cannot be mimicked to a reasonable extent. It's very complicated, it's going to take, you know, many years. I don't think we're going to have the AGI like next year, but I think it fundamentally can be learned, can be improved, right? And we can see on how we interact with the world, how we learn as humans, and a lot of that can improve the models we build as well. Plus, they have also their own capabilities that we don't have, like they can remember everything they've seen, right?

• 40:56Speaker 01

  They don't forget things as easily as we do, right? We had a great way to deal with complexity of the world. Well, we'll filter things greatly, but because of that, we'll lose some context. But um you know, machines want, right? So that's the advantage, right? It's like some precision. So there's pros and cons here. I think we fundamentally have the ability to compute everything we need. I don't think there is a like a yes, maybe certain things with stochastic gradient descent are not perfect. We're gonna do certain things that can be done better for me. As a computer architect, I'm less concerned about what is computed, but how is it computed? I think we are ridiculously inefficient when I think about how much power our brain consumes. This is how much power training process consumes and more in data centers, right?

• 41:41Speaker 01

  That we're starting to talk about nuclear reactors back in, you know, in the US and things like there's not enough power and things like that. I mean, it it kind of tells you, well, we go quite aggressive in one direction and we probably want to do some adjustment, right? So it means there is like, we need to learn how the things like uh has to be done less aggressive on crunching numbers without thinking about what you're doing. You need to be more selective and it has many different dimensions. It's the data, it's what exactly you compute. It's like not doing the same compute too many times, like, like, for example, right now, like every single foundational models keep starting with more or less similar data sets and everything they get from the internet, right? And that's quite redundant, I would say, like doing this again and again and again.

• 42:25Speaker 01

  So hopefully you can do this in a more iterative fashion. So the future models are not built from scratch, but actually built from a corpus of the prior knowledge, right? You just additive to this, right? So I think that would be really, really important, yeah.

• 42:40Tim Scarfe?

  Yeah, I've got so many things in my mind, just to close the gap on what we were saying before. You know, there's this ambiguity thing and at the beginning of this interview, in a way, it it it's a bit nerve wracking because it could go in any direction. And just like with the language model, when you have a long conversation with a language model, the the mutual understanding increases because it's almost like the the entropy goes down, you're you're you're on a well trodden path. And it's a similar thing with with our discussion now.

• 43:03Speaker 01

  That model also train you, like you train the models, or they train us to ask the questions in the way they understand. Like, my keys very quickly realize if you don't ask questions to G P T in the right way, you don't get the high quality answer. But if you learn the protocol and how to communicate with this thing, you get what you want very quickly, right? So that's very impressive. Humans also adopt like to the behavior of the machines. Yeah.

• 43:27Tim Scarfe?

  Yeah, exactly. So it's it's all about knowing what question to ask.

• 43:31Speaker 01

  Yeah.

• 43:31Tim Scarfe?

  And it's so common in in software engineering. So because these cursor guys, they made the observation, I don't know whether this is true or not, but they said that code was lower entropy. And you know, it kind of, I have noticed this to a certain extent, you put a load of code in there. And straight away, it kind of like snaps into some kind of understanding. But still, if you ask it something dumb. Like if a if a level seven engineer from Facebook went in there and they would know what they were building, that they would be using the right kind of abstractions, they would be kind of designing things correctly. Um. It's a mistake to think that anyone can go in there and just write some software. Because it just gives you back exactly what you ask for, and that's quite often the wrong thing.

• 44:11Speaker 01

  Yeah, it's I don't think like, I think it's misconception when people said they would replace human like like, but I think that is like. We're quite far in my opinion from that, right? Like it's it's still an assistant, a very high quality assistant. So you really need to, like, you're still a designer, you're the one know that what's missing. But for example, you don't need to memorize this particular algorithm to do this, like, you know, like the best sort or the best walk pass through or the best like. Uh, graph traversal, right? You just don't need to memorize it from a textbook, right? You just know roughly should it look like. And you know rather the complex and then you just ask that question and it would give you the code. So instead of you figuring out how to write it and debug it, right? Uh, that somebody would give you those like very basic concept.

• 44:55Speaker 01

  And a lot of things when we manipulate data as a software engineers, we're doing repetitive things again and again. And this thing learn from us doing this and what's the best way to do it. So you don't need to, you just need to know that you need to do this task, but you don't need uh be the best of doing this model anymore. There were thousands of people that did it better than you and it's already part of the corpus and it's gonna give you. So you're gonna start building off to a very good, reliable pieces. But without the design at the top, it's not gonna be able to be like, good luck asking it to build a new operating system, right? Just good luck, right? This is just like, it's not about just adding a few for loops here and there. It's actually quite sophisticated piece of stuff.

• 45:38Speaker 01

  But yes, if you want to build a very basic, you know, operation with a web page, right? You don't need to learn about that standard APIs. It all will be there because it's very standard. So it will save you a lot of time of learning those things, right? Yeah. So obviously create some danger if that, if there's something wrong and fundamentally buggy come into its code base. It's gonna now suggest it to everyone. It's gonna be do everywhere, right? Uh, but like it's, it's the, the problem with like any library, the library has the same problem, like it's, like everyone would be susceptible to it. So I, I still think about all this coding things as not being replacement, but being just very solid assistance uh uh in the very best case.

• 46:18Tim Scarfe?

  Yeah, there's, this thing as well that right now coding is, you know, gen AI coding is still human parasitic. So one of the problems the cursor guys were saying is the verification problem. So you're, you're telling it to do something, you ask it the right question. And now we need innovative ways of just diffing many, many files. Because it might have generated four hundred lines of code and the Git experience, you know, like on GitHub, we've got this multi-file diff experience that hasn't changed in like fifteen years. And now we might have layers of L L Ms that are kind of helping us focus on the bits that change. So we're becoming verifiers, but still though, what we don't want is a system that just tears up the code base every single time. You know, John, John checks some changes in and now there's these, there's changes across fifty files.

• 47:03Tim Scarfe?

  And that creates this understanding debt because now all of the other developers need to figure out what's going. So there is something to be said for having a very modular architecture, which is understood by everyone. And then we, even though we might be introducing inefficiencies, we make changes in a targeted way because we don't wanna minimize the damage, if that makes sense.

• 47:20Speaker 01

  Yeah, you're absolutely correct. The design is like, it's not just an engineering, it's an art in a way. Like designing a complicated system that functions and functions efficiently and scales is an art, right? Like you need to do it. And I don't think we're at the stage where we can delegate that responsibility. So we delegate some more mundane work, you know, to people. There are definitely some risks associated with it, we start to touch with you. Like, I think I see it this way. But then, for many, many years, we rely on the people going through the ranks in the software development. Like, software engineers, junior ones, they don't go into design of the whole system, right? They learn from the basic tasks.

• 48:02Speaker 01

  Right now, if we start to, some of that functionality would be replaced by assistants, right, of different sorts. We need to make sure we have a good, basic learning curve for them. Like, if we want to raise a new, you know, generation of architects, like the people who build the system and raise from L three all the way to L eight, L seven or L eight, right? We just want to make sure that have ability to grow, right? In the new environment where a lot of the basic skills are replaced by automatic tools. I think it's doable, we just need to find the way to do it, the whole process needs to change. I think even the interview process has to change. Right now, we were testing people a lot on memorizing some basic concept. Right now, why would even need it, right?

• 48:45Speaker 01

  Like, you need to test more for design skills, like on how you use these components, rather than memorizing the best algorithm to do it.

• 48:54Tim Scarfe?

  Yes, and also surveillance. I was saying to you last time we spoke that I'm a huge fan of the actor pattern, and you have to become an expert in surveillance. So all of the actors are logging their activities, and they all happen asynchronously in different, you know, in a different order. And, you know, we need to build the next generation of platforms just to figure out what the hell our systems are doing. And, but it's also more flexible, right? Because we can do simulations and counterfactual analysis and god knows what. But we are moving away from this old school of software engineering. And that's kind of what I wanted to talk to you about, because enterprise adoption, right? At the moment, you know, the enterprise, they've been building software in a certain way for a very long time. There have been ways to make it more accessible, like low-code and no-code platforms and so on.

• 49:39Tim Scarfe?

  They're starting to adopt AI. What are you kind of seeing? You know, what's the state of play with enterprises adopting AI?

• 49:49Speaker 01

  Yeah, there are several things that we see as a company here, and again, that helps me to have a company behind me to have a little bit of experience here. They're definitely like majority of the companies we talk to, right? And the enterprise segment, they're on board with G N A I. They see the potential of the value for them. They very frequently have the budgets already allocated to start deploying the technology internally. Where things different become complicated is what exactly that the first use case been. They also sometimes don't have the right conceptual understanding, what's doable, what's not with that technology, right?

• 50:30Speaker 01

  Because usually the way it looks like like someone in the company would say, okay, let's start to use G N A I internally, decision is made. Then they start to get some people from the internal organization, say data scientist, that would help to adopt the technology. This data scientist is excited about it, but they never actually built sophisticated system and plus this is a new tool. So there's all this complexity. We never built complicated systems ourselves before. We were scientists, right? More in A I space, out of sun, we need to do it, and this whole thing itself is very complicated. And the system is very complicated built out of it. So it's very hard and also you need to find that killer app within your org to justify your existence.

• 51:10Speaker 01

  So it's very easy to come say, oh, give me five million dollars, I'll train another G P T for company X. And we had our own model and our own A P. That's great, but do you really build anything helpful for the company? That model probably going to be much worse than Llama, right, that you could have used, right? And then instead you should probably do fine tuning. And then if you do fine tuning, who's going to prepare the data, select what's useful? Select sensitive versus non-sensitive data, like all of that has to be solved. So there is a lot of challenges in how the technology would have have to be adopted. The good thing, it seems like pretty much everyone is on board that it has to be used right now, right? People do see the value. It doesn't look like it's just, oh, there's a hype there and we don't know what to do. Like people had a general feeling that it would help them certain different use cases.

• 51:56Speaker 01

  And then you go to different segments of enterprises. Potential sectors, insurance, automotive, like they all see that there is a value for them, that technology. But what's complicated is what the exact first use case it should be. And also, after you narrow that down, how to go and implement it and develop at scale. Because they never had an org typically that would be able to do this efficiently. Again, remember, they are not were, uh, you know, AI first companies. So for them, it's a new toy and they had some piece of an expertise, but not the rest. So company like ours, Sentinel, is actually trying to close one of those big gaps for them. It's like how to build the system from our data. Essentially, we had foundational models, we had our data.

• 52:40Speaker 01

  Who's gonna help us to milk the data and get the IP and get these models deployed scale for us? Such is also gonna be cost efficient, right? Also, what we see is that the people that use it once successfully, I feel like they would never go back. They might criticize the AI for being unreliable and blah, blah, blah, blah, that. But if, for example, you use ten million dollars before to find broad in your organization, like say on their bank statements, and now you use a model to do this, and that model costs you two million dollars. You're probably gonna uh keep complaining to me as a company that's responsible for optimization, why it's so expensive, two million dollars. But you're never gonna go back to ten million dollars expenses with humans ever again, right?

• 53:25Speaker 01

  You're gonna try to optimize what you have, but you already know you make that step. There is no step back, like you're always gonna use some AI now, some form of automation, because you're never gonna go back to doing this with humans, right? Because it has its own downsides, you start to forget when you use AI. And you say, oh, yeah, hallucinate. Well, the problem is humans also make up things and also hallucinate sometimes and also like make mistakes, right? So, yes, there are challenges, uh, but we need to admit what the challenges is. We need to build a good roadmaps for enterprises to adopt the AI, right? And I think the good thing that helps right now is that there are general consensus across many of these organizations that you need to be part of it, the use case.

• 54:08Speaker 01

  This is naturally coming as you look carefully. It's not, you're not randomly pick it, you talk across the team, but you see where your data is, where your uh your company unique proposition is, and you try to adjust it and help AI to enforce it, right? Almost all conversation we have right now with enterprises, like the first few conversations around, do you really want to do it? They're on board, right? That's actually relatively smooth. It's the challenges, what exactly that would have as the first step? Because they never did it before.

• 54:41Tim Scarfe?

  One of the things I've found is when you have a a great software engineering culture, there's there's less of a problem. So, you know, we've been on this journey from the the individual lone wolf data scientist. And then we start thinking about ML DevOps and we start thinking about joined up approaches for data architecture and data engineering and so on. And it all sounds great in principle. And then we start to build templatized ML DevOps framework. And we say, guys, everyone, you need to use this framework. You know, we've got a team building it. And if you will use this template, then everything will be good. But of course, that doesn't happen because people tend to build applications just for them. So it's a real big step, isn't it? Moving towards this centralized way of thinking and having functions and platforms.

• 55:24Tim Scarfe?

  So for example, if I'm doing pricing, I might have a team and it's their job to maintain this pricing platform and it has a fixed API, it has a standard interface and and they and they just kind of, they make it really optimal and it's and it's kind of centrally managed. That that seems quite a difficult step.

• 55:43Speaker 01

  Yeah, so it's like, I think when you talk about this managed platform, like like like a lot of people support right now, I think it's just a step, that's not a final solution. I I don't think that would be, maybe I'm wrong, but I don't see a lot of people that just wants to use the vanilla llama for everything. I just like feed people, this is just a building block of the system, right? And I think those system would have many of those building blocks with a lot of fine tuned specialized system to do what's needed. You probably need guardrails around things in each of these components, you like, you need a lot of things. So it's a way more complicated than just say, let's take a model here and then go run it and deploy like somewhere, right?

• 56:24Speaker 01

  So in reality, I think what you need to do moving forward is you need to build an infrastructure that can. Run reliably those complicated system, right? So essentially, the system would be built out of multiple, say, agents, for example, and multiple different models with all the different constraints, potentially connected to different databases of knowledge, right, back to databases, or something like that that people can utilize. So you need to build very different UIs for those system, where people can really like plug and play components, play with them, test different things, test, like essentially you need to build like, you're not testing one model on one, again, another model, you're testing two systems against each other.

• 57:06Speaker 01

  So you want to put like two complicated stuff, one against each other and see how they perform. And the scenario is also very complicated. Remember, they're interactive. Things can be non deterministic. Based on one input, you can conclude that one system is better than another. Like you need to get a very sophisticated environment tests on how those things behave. It's very easy to test, oh, it's like, like how much faster it is and how much cheaper it is on a one metric. Remember, like multidimensional problems are way more complicated, right? So you need to make sure that environment is captured properly, right? And you can, you know, evaluate what's good for you, right? I think that's a very exciting thing to work on, and that's, I think, that the community would move over time.

• 57:48Tim Scarfe?

  Last time we spoke, you you used the term dark silicon, which I thought was was very interesting. What did you mean by that?

• 57:54Speaker 01

  Oh, it's a relatively simple concept, if you think about it. So imagine that you had a limited budget of the power that can you supply to to a chip, right? But you have the ability to put way more transistors on that chip. So essentially, you have to live in the world where, at any point of time, you cannot power all of it. You need to power only some of it. So you have to decide what to power, what not to, right? For a long, very long time, people were saying that we're never going to build anything like that. We're always gonna like, everything that CPU was having, we're always building it, right? But then at some point, you know, the end of, you know, Moore's Law and several other

• 58:39Speaker 01

  limiting factors lead us to a situation where Then our scaling, especially from coming back to computer architecture terms, we're not able to keep their power constant as we uh improve their, you know, their dimensionality of the transistors. Why is that a problem? It means I still, as a hardware company, want to sell big and bigger chips with more and more transistors, but I had not enough power to supply them. So either I had to make them bigger and bigger and bigger physically, or if I limited by their uh physical dimensions of the chip and speed of flight and interconnects, I had to limit what I power. So essentially it has many different forms, like that limitation of dark silicon. Either you're gonna reduce the frequency on how you run things.

• 59:22Speaker 01

  So instead of fully turning off some parts of the chip, you run it at like half the frequency, right? Or uh essentially we aren't utilizing the resources we have. I remember from uh my computer architecture class back at CMU back then, we're already talking about the phones and they being very power limited. And for example, it was very common on the phone where you had certain memory bandwidth available, but you can only use it for a very short period of time because otherwise you get power off, right? Like or or heat. So there was a hardware there, but you can fully use it. So it means you need to become much smarter in how to use this hardware. That problem didn't disappear. The chips right now are extremely heterogeneous. Let's take, for example, Nvidia chips, right?

• 1:00:06Speaker 01

  Like uh that is quite uh interesting thing to look inside of it. With all the public information, we already know it has many different precision support, many different specialized pieces of it, right? Um so for example, even if you think about a E I alone, you have F P thirty two precision, F P sixteen precision, right? That's combined together so that two F P sixteen can generate output for one F P thirty two. Then there are brain float precision, there is integer precision, and there is a special dedicated using uh units called tensor cores that don't use a vector instruction, they use like direct matrix multiplications in them. All of that is an independent units.

• 1:00:48Speaker 01

  Uh most of the time, your software is not written to use all of it. What does that mean in practice? It means one way or another, a lot of these resources can be easily wasted on your chip right now. Whether you power them more off or reduce the frequency, that's the the choice that the computer architect who builds the chip has to make. It's a very tricky problem because the power becomes a limiting factor for many of these components, right? So, yes, it's a very interesting problem that anyone who builds a modern chip has to deal with and prioritize one thing or another, well, power it on, and powering on different parts of the chip.

• 1:01:25Tim Scarfe?

  Part of the problem here is that people are unaware of how much compute we're wasting. Why is that the case?

• 1:01:34Speaker 01

  Um, the first time in my life I faced it when I I just finished my PhD and I was at Microsoft Research, right? And back there, I've seen a lot of people were excited to train a large model. We are talking about two thousand sixteen, two thousand seventeen, quite some time ago. Uh, where people were using Nvidia GPUs, but very old one, like like Maxwell generation before P one hundred. P one hundred was just built back then. And essentially, there was a lineup of people waiting for the chips. And I was like, hmm, interesting. And I I'm an optimization guy. I was very interested in how well it's running because like out of a sudden, there's all these frameworks like, you know, TensorFlow, C N T K back then that people were using. I was curious how well it's running.

• 1:02:17Speaker 01

  I got a permission and access and I profiled some of those workflows that people were running. I was mind blown that utilization is easily can be like ten percent. And I came back to the people like, do you understand when you run those workload? Utilization of these chips could easily be only one tenth of its potential. And they were like confused by what I'm saying because like for M L people, that was a ridiculous claim. It's like, what do you mean? Like GPU is running, it's busy, how it can be like not utilized? So this is the way of understanding and a level of abstraction, how people from M L community typically think about compute versus other computer architect system people think about compute. So to this date, it's actually very difficult to get ninety nine percent utilization of the chip.

• 1:03:02Speaker 01

  Usually when you see that, it means like some one of these metrics is lying to you. It's just telling you some part of the story of what is possible, right? And that's a huge opportunity there to actually use all that, you know, compute that is there. And part of that, there are many different sources of the waste. One of the sources of the waste is very basic. GPU is not even running all the time because you had a model that technically sits on the CPU and move the kernels on the GPU, right? And when you do this move, there are periods where not in the GPU yet. So you want to make it that's properly pipeline and you're always on the GPU running something, but it's not always the case.

• 1:03:44Speaker 01

  Then another problem when you're running, you're not being using all the potential of the compute available on the right? Because it has so much compute and the models are all flexible and different. Kernels are not always perfectly mapped to what you need. And that means there's a lot of waste there. So there are many different sources of waste. And even if you solve that, it still might be possible that you're only using one, like it might be your workload is just using one precision at a time or one compute unit of one type. But Nvidia might fully power the whole chip to run at full, like, like everything else. And you're just wasting that compute, not running anything through it, right? Yeah. So that's very important thing. And remember, what is also challenging here is that when we talk about power limitation and dark silicon,

• 1:04:29Speaker 01

  if you actually make everything to run, you might actually overheat this thing because it won't be able to dissipate the heat as well. So you need to be very careful not overdo it and do it in a smart way, right? So it's a very interesting problem to solve, but that's where the lot of opportunities there. Untapped potential that we at CentML in our company actually benefit, uh, use for the benefit of a user. One example I really like, uh, to mention, like we get like one paper recently accepted around that that will be out in a few months. Essentially, you can run your training and inference at the same time on the same chip. And the, the trick around it is just using all that gaps, uh, that's wasted and also use that different precisions that are otherwise not utilized.

• 1:05:12Speaker 01

  You essentially can run training and inference back to back at almost for free, right? It's possible, right? If you orchestrate the whole system properly around it. So it, it's an opportunity there. It's all this untapped potential that can be utilized.

• 1:05:24Tim Scarfe?

  Yeah, that's amazing. That's amazing. I mean, it's tempting to think that, you know, we have these very high-level abstractions, like even Python, for example. I mean, of course, a lot of deep learning doesn't run on Python, but as you know, Python is hideously inefficient. But by the same token, it's democratized computation, right?

• 1:05:41Speaker 01

  Yeah.

• 1:05:41Tim Scarfe?

  So many people can write Python code. But what you're talking about is a little bit more nuanced than that. I think you're saying that when we actually start to think of this systemically, when we have a big spooler and we've got all of these different workloads in this fabric, we can start to do optimizations that wouldn't have been possible if we were thinking of them individually.

• 1:06:02Speaker 01

  Yeah, absolutely correct. And I'm glad you also brought up higher up the stack, because I tend to, because we talk about dark silicon, I started to focus more on about the low-level stuff, but there's huge waste is possible if you're doing things right at the top level. I remember when David Patterson, one of our scientific advisors, received his Turing Award. In his Turing Award talk, he showed the examples of matrix multiplication done in Python versus done in properly optimizing C++ and then optimizing all the way to the silicon. There were like three or four orders of magnitude there, the differences, right? So there's a, Python is very inefficient, but Python is very easy to use and develop things. It requires way less skills. So something that PyTorch and the PyTonic ecosystem was very efficient

• 1:06:47Speaker 01

  is that they hide all their things that require sophisticated computation from their running on the Python itself. Like you use that interface, but under the hood it's running C++ libraries and you don't need to worry about all that complexity is hidden from you. That is where the beauty of it is. Because ease of use is also very critical. Because writing everything in C++, it's not the way to go if you want to make that technology adopted by much wider audience. And even some of the inventions of our group and our company were around automated the compiler development. Like, for example, for many years at Nvidia, people write very carefully crafted manual kernels to get the top performance. And that's considered the best way to go.

• 1:07:32Speaker 01

  But say several years ago, a lot of us realized that's not a sustainable way with AI. There's so many kernels and so many targeting hardwares and all of that is very diverse. So we start to advocate, you know what? You actually do this more automatically using machine learning compilers instead. So in that case, You still need to give an input on what kernel you want, but use a Python interface to give an input. But the all sophisticated CUDA code, under the hood of PTX code even, is generated automatically, right? And it's feasible and people are benefit from it right now. And I think people at companies like Google built XLA compiler that also does, you know, some of those things quite efficiently, like in a more automatic way.

• 1:08:15Tim Scarfe?

  Now, this is one of the products that you have, actually. So you can plug this thing into your PyTorch code. It's just another few lines that you need to add and you can compile the model. So you can say, what optimizations do I want to apply? And it's going to make the model run significantly faster. Can you talk us through that?

• 1:08:34Speaker 01

  Yeah, so it's like, again, a couple of years ago, where PyTorch ecosystem starts to evolve and starts to be more open to other people to contribute in the development. There was a possibility to use something called PyTorch compile, right? So essentially, it allows to take your model as an input and use your own backend to optimize things. So our research group at University of Toronto and then our company supported a new machine learning compiler and a YAR called Hydad that allows you, pretty much within a single line of the PyTorch code, to optimize it significantly. And since then, a lot of people start to go in this direction. Like one very successful project in this space is Triton compiler as well.

• 1:09:19Speaker 01

  So again, people realize that keep writing kernels manually is not the way to go and you need a more automated way. I just want to remember, to tell you though, that this is just only a piece of the puzzle. This is where I'm talking about the kernel level optimization, not the whole model optimization. So then, generative AI workload complicate this process even further. Why? Because they typically require something called dynamic shape and compilers are not very good and optimizing something that has random boundaries. It's thinking about Properly optimizing the loop that has unknown boundaries of the loop. That's very difficult, right? So essentially, that's what make the life of compiler optimizers difficult.

• 1:10:01Speaker 01

  But it's still a solvable problem and we also solve it in our company on how to run dynamic shapes workloads very efficiently, right? But that's at the low level, right? So why is it important? It's because that unlocks the ability of other people to contribute to the success of this PyTorch or any PyTonic based ecosystem, right? And I think people realized that it's the way to go and a lot of people contribute in the space. I think it's like right now, it's very heavily supported, including Nvidia, very heavily supports building this open source community around like machine learning compilers.

• 1:10:36Tim Scarfe?

  You know, like with with C++ we we have, you know, we have a handful of compilers. And if I'm compiling something on on macOS, I use the same compiler every single time. But I'm I'm getting the read that for machine learning workloads, it's far more heterogeneous. So there are specific optimizations for specific jobs. Can you explain that to me?

• 1:10:57Speaker 01

  Yeah, it's a very important nuanced thing. Like, for example, even like, first of all, training versus influence are very different workflows. Like, people say like, isn't that very similar? Like, you know, like doing a lot of matrix multiplications, all of that. And the answer is yes or no. There are common kernels, but a lot of things that are very different between the two and the scale and all of it. Then it's also very important on and more nuanced on how you are optimized for particular target. For example, your target might be latency, your target might be throughput, your target might be cost, or more commonly a combination of that. Then you had a bunch of tricks at your disposal that you need kind of balance. And none of that is usually just make everything better.

• 1:11:40Speaker 01

  It's one thing sacrifices something for something. Like, for example, you mentioned today speculative decoding, very cool trick, but there's nothing for free. So speculative decoding means you do something speculatively somewhere. It means you would have to burn that compute that you can use for something else instead. Why would it make sense? Well, if you want to cut the latency as one of the metric, so you want to speculate on the outcome, and you can maybe have something intermediate values ready for you to use, so you optimize it. But it means you need to burn some compute somewhere, and it might be actually burned for no reason if you actually run, right? So there is a trade off here. So when you improve the latency, you would make your throughput worse.

• 1:12:21Speaker 01

  That would make the cost of the process uh uh higher, right? So there is no like pre-launch in many of these optimizations. Another common example, people like to reduce the memory footprint of their model because memory bandwidth and memory capacity is one of the bigger limiting factors on running these models on Nvidia, uh on existing chips, Nvidia, AMD, or TPU chips. Then, there are several different optimizations we can do to reduce the footprint, but none of them is completely free. So for example, you can recompute the values you otherwise need to store. But recompute means doing compute multiple times, the same compute. So you waste compute in quarters of memory. Why would you ever do it? Sometimes, in some chips, you had more memory and less compute,

• 1:13:06Speaker 01

  and that's a good, or the other way around, and that's a good trade off. Or you might actually do compression, right, of certain parameters. Compression, lossless or lossy, introduce other trade offs. Quality of the model changes. And also extra compute associating with compression, decompression. But then it can potentially save you parameters. Each of these cases, and there are many, many, tens of those different optimizations. And this is why it makes it hard. Like, for every model and for every use case, the optimal Pareto frontier might change dramatically, and you need to find the way to, uh, what would be the best set of techniques to do it. And it's very hard to do it fully automatically for every new model, for every new workload, for every new chip that you have at your disposal.

• 1:13:50Speaker 01

  And this is some of the inventions that we push through in both research community and as a part of the company. We say that has to be done more automatic, right? Because a lot of early solutions were really hardcoded to run on a given model on a given hardware. In reality, it can be done More automatic fashion. So it means that the customer brings a new model quickly. You can do it quite well. For example, I'll give you one example that I think highlighted what we were very proud of. Last year, when Lama two went out, a lot of famous companies were like, oh, we need now like optimize our models to run it efficiently. And it took many, many weeks for some of them to do it. It took us less than a day to make like Lama two to run an over software stack. Because everything is very automated process.

• 1:14:35Speaker 01

  Again, this year, when Lama three was released, four hundred five billion model. We did a live hackathon. We didn't have partnerships, unfortunately, with Facebook and Meta on that. So we didn't have early access to the model, but within one hour it was working for us, right? Why? Because everything is automated, like we anticipate certain characteristic of the model. We know how big it can be and we were prepared certain trade offs. Then we found the optimal mappings and then we just, you know, make it available to people pretty much immediately. And that's very important when there's diversity of models and diversity of the workloads is changing. Another very important practice specifically about G N E I is that even if the model is fixed, the input itself can be dynamic.

• 1:15:19Speaker 01

  You change the sequence length, all the other different hyper parameters, there are many of them. That changes what's the best way to map on the hardware. So I like to show in my lectures that for a basic attention block, depending on which parameters, like how big your sequence length and the window and like another the parameters. You might actually want to fuse these kernels instead of these kernels and slice it this way or other way. It's a very complicated problem even for humans to find it. Obviously, you don't want to do this manually every single time where someone change the input of your model, right? So this is something that a compiler has to do to make it efficient.

• 1:15:57Tim Scarfe?

  Yeah, you've articulated that really, really nicely. So what you're saying is that this is very situated. We're in a non-stationary environment. There are many moving parts. So, you know, we might have different batch sizes, different input sizes and so on. And also we might be in the context of a system. So we might have this big cluster that's doing all these different jobs. And that actually gives us a whole margin for optimization where we can select the correct optimizations and the correct kernels to use and so on.

• 1:16:24Speaker 01

  You're absolutely correct. So there is like all that complexity in there. And this is why it's when we talk about enterprises early in this talk, remember it's like it's just hard for them to do it because like if you do it manually, you need like very, very, you know, sophisticated engineers that know how to do it. It's expensive for them to hire these guys, right? And it actually not required because like it can be replaced by the software, right? This is where all that complexity comes from. It's like the workload itself is so dynamic, so complex. And as you said, the system itself has this all these external components that are moving. So you want to make sure that this whole thing interplays really well. So you optimize for not a particular benchmark, not a particular model, you optimize for this use case for that customer in that environment they're running right now, right?

• 1:17:08Speaker 01

  And this is why it has to be more dynamic. And this is why you need to build the system stack that it would be able to adapt with that complexity.

• 1:17:17Tim Scarfe?

  Yeah, it's a bit like the no free lunch theorem applied to machine learning optimization.

• 1:17:22Speaker 01

  I agree. I agree. Yep.

• 1:17:23Tim Scarfe?

  We we were saying that we now have this new paradigm of model development where we have this big compute fabric, we can do a whole bunch of optimizations against that. I mean, who are your customers and how is this working?

• 1:17:37Speaker 01

  Yeah, so like, um, so several of our customers, like enterprise uh customers, like, like, for example, Deloitte is one of them. And like, it's like one thing that important at this stage right now is that, in my opinion, you really need to build this relationship together with this customer. It's like, for them, they never actually had G N E I in production before ever, right? That's the first experiment for them. There are certain expectations, there are certain like knowledge in the team and you want to make sure that we augment their team knowledge with everything we had external. They they best know their workloads, right? They best know their data. And they had an understanding roughly what they want to build.

• 1:18:17Speaker 01

  We had a very good understanding how to build the system and we have how some understanding what data they might have and we are trying to build that bridge. So if you say me where the challenge is, is how long that bridge takes right now, like building that bridge is an enterprise customer because a few of these enterprise customers, they're more like Partners to you, like you spend a lot of time, you nurture this relationship, they might become your channels. And we're building multiple of those. And right now, we also had another customer called Redapt, which is one of their biggest Dell hardware resellers. And again, they had the hardware they built and they want to make their hardware L L M appliances. And they had accelerators like Nvidia GPUs there, but not necessarily H one hundreds or A one hundreds, like smaller GPUs.

• 1:19:00Speaker 01

  And they want like magically that these boxes out of a sudden starts to run like all their L L M's that people are using H one hundreds for. So someone had to find all that new transformation and balance them out. So it worked for them, right? And it's not a trivial thing because like these boxes not only had smaller GPUs, they also don't have fast interconnect like NVLink. It means you need to find how to communicate with less data, making sure that all of these smaller chips that are cheaper and smaller, but also less compute and less memory are busy and doing useful work. And then you get the output and the speed that you actually good enough for the real use cases.

• 1:19:44Speaker 01

  So we get a lot of this chat with you data interfaces for the companies like that. Another similar use case for us was a company called Arcatis, right? It's an engineering company. It also had multiple, you know, boxes with GPUs and want to make them use in L L M and RAG scenarios. And that's very common. We also had multiple different ongoing partnerships with like, you know, big vendors like G C P and A W S, where we help them better utilize the compute, unlocking their own customers as well. So this is all ongoing and ultimately it's right now when we talk about enterprises is the time that takes to go from research to the real, real adoption at scale that's important.

• 1:20:28Speaker 01

  Because there is a wrong perception sometimes by data scientists is if I Found some really cool models somewhere online and I'm running on this few inputs and the quality looks cool. I'm ready, I can go and deploy it. It turns out, no, that the tool you use to deploy it might work on A100, but it won't work on like, on large scale cluster, like operated like that. And you actually didn't optimize it by running other workloads and didn't realize the way you run, can only run one input at a time because you exercise all the memory right now. And if you run one at a time, the price running it is like enormous and it kills the whole point of the workload or the use case to begin with.

• 1:21:10Speaker 01

  So you need to educate the people, say like, no, you need to be careful about that. You like, you tell us what the constraints are, we help us to do you, uh, to do it for you. We try to automate this process as possible. For us as a company, it's very important not to become another service company, right? That's I'm trying to avoid. We do this initially in some of this engagement, but for scalability, we try to automate this process from each of these new engagements, right, to make sure that's becomes very repetitive. So we had several other Gen AI startup companies like Echo AI, for example, and a lot of other conversations where that learning process with the bigger enterprises help to fully automate the process. So when they arrive, we give them all the software stack and they can use it out of

• 1:21:52Speaker 01

  the box with no any special features needed. So that's another interesting thing. There's different segments. Some require some customization, but some are like fully automatic. And where it's fully automatic, it's cheap. It's cheap for them to use us and also cheap for us because it's very easy for us to onboard customers like that. It's because it's fully automatic, right? We can support a lot of that. Why is it important? It allows us to keep very low prices for them, uh, for, for, you know, for our software stack because we don't need to use engineering time extra for that, just a basic support. So it also means that they can be successful because they are, you know, software stack building on top of the models, not that expensive for them anymore.

• 1:22:32Tim Scarfe?

  Yeah, I'd, I agree. It's, it's good to, um, avoid the consultancy trap, but of course, you know, you guys want to build this massive platform. You want to scale up and, and there's always this tension between, um, you know, kind of focusing too much on, on individual installations. But what is your relationship with the, um, cloud providers?

• 1:22:50Speaker 01

  Yeah, we, as I mentioned, we had a very good partnership with several major cloud providers. So like, uh, obviously not a blades of GCP and AWS. We're official, uh, you know, officially partners with each of them. We're on their marketplace and we had several strategic engagements with them directly with their engineering team. And, um, you know, uh, also with, uh, with some of their customers as well directly. Then we also, uh, had different stage of engagements with other cloud providers. You can think about all the, the usual suspect in the space. I'm not going to say until we publicly announce it, but there are a lot of those conversations, a lot of them are looking at the software stack built on top of their hardware.

• 1:23:31Speaker 01

  A lot of them had very good expertise in how to build, uh, you know, cloud service, like how to provide that bare metal access to their hardware. But they want distinguishing characteristic, they don't just want to be bare metal supplier. Right. It's very competitive market right now with margins for their hardware going down and down. So they want to have a software stack that would attract customers and keep them on the platform. Right. So that's a very important thing right now. And we help them to get good use to bring the customers and connect them with their level of expertise. Cause majority of them don't, are not like bare metal experts to use, you know, their hardware directly. And you also try to help them utilize their hardware better.

• 1:24:14Speaker 01

  So they, they would understand there is like, you can co-run things together and you can do this and that, and you can save a lot from your compute that you didn't even realize is possible.

• 1:24:23Tim Scarfe?

  Interesting, but you don't, you don't feel that you're competing with them in any, in any way. I mean, uh, I, I guess this comes to, to the moat question, but let's say if, if, um, I don't know, if AWS gave you five million, uh, you know, five billion dollars to be exclusive on their platform, would you, would you be interested?

• 1:24:43Speaker 01

  Well, five billion dollars is a lot of money, right? Like, it's like, it all depends on the stage, right?

• 1:24:50Speaker 01

  If the deal like that will come, we'll talk. But like, honestly speaking, we always try to build something that would be not overly specialized for any cloud provider or any particular hardware even, right? Like, we always wanted to have a platform where all cloud providers are equally represented and all the different hardware equally represented. And I think it's very important for us overall is because at the end of the day, we want to get the best performance for the lowest possible cost. And we want a place where we had an honest opinion and that would optimize my workload for my, for my model and for my use case and map it the best possible way, right?

• 1:25:30Speaker 01

  And I think for that, it is important to be in good terms with all the key cloud providers and all the key hardware vendors. This is why we had a very good, you know, partnership relationship with Nvidia and then Nvidia was our investors in our previous round next year. And then, but we're also like working with Google had TPUs, we're talking to AWS has Tranium chip, we talk to other vendors, like we are trying to be equally friends with all these companies because I think they're all important. And people don't want, I think in most cases, one dominant monopoly of some sort. They prefer to have a choice of multiple things that makes the market also more healthy, right? So I think we are trying to help them to find the best chips, optimize it with whatever

• 1:26:15Speaker 01

  constraint they have. And they had any concern, they have all their Nvidia GPUs and they had their new advanced AMD GPUs. We try to make sure that they can leverage everything that they have at their disposal.

• 1:26:26Tim Scarfe?

  Interesting, because there's always the question of like being in a sovereign cloud or being kind of like, you know, embedded in one cloud. But I suppose you could go one level of abstraction up because it's great that you have this agnosticism to hardware. So you have this incredibly intelligent system that can optimize for the hardware that is running on the cluster. But could you in principle do that at the cloud level as well? So you could do cost optimization by setting up clusters, you know, this bit on AWS and this bit on Google or something like that?

• 1:26:55Speaker 01

  It is technically possible. We already do like multi-cluster things right now, like within one vendor, but it can be done across vendor. It's in principle can be done. The problem is the communication between them and how costly it is. Like inbound traffic and egress, you know, that can be expensive, right? Uh, but technically it can be done. What uh you uh at the beginning you raised the important question like how you integrate with all of these different infrastructures. And I want to say that there is some level of complexity, but not dramatic, because ultimately we pick a level of obstruction at the level of a container. Right? So all of them support containers, one form of another. That's a very basic concept. So we make sure that whatever we deploy is just based on the concept of container and we can build on top of that.

• 1:27:39Speaker 01

  That means that we can easily go between GCP and AWS. And then all the other guys that are bare metal, they're just need to support, I say, basic Kubernetes on top of their infra and that's good enough for us to build on top of them. So it's really a few specialized cloud providers that is like a hyperscalers. And everything else from our low-level perspective, others are more or less identical. They're all like bare metal access and we can manage, you know, them quite well.

• 1:28:07Tim Scarfe?

  Yeah, I think part of the thing as well is you need to have this pragmatism because you need the models to be close to where the data is. So for example, I think you're built into Snowflake. And from uh from a database query, you can actually like, you know, call into the cluster and you can say do some sentiment analysis or something like that. But in order for these um architectures to work efficiently, you actually need to kind of respect the the the data architecture boundaries as well.

• 1:28:34Speaker 01

  Yeah, you're absolutely correct. And there is like you brought up this case like, yes, we partner with Snowflake as well. And like we are available in their marketplace, so we can accelerate the models that the customer wants. Why we would do it? Because there are people that really like Snowflake infrastructure. Their data is already there and they would prefer to do things closer to where the data resides, right? And not moving it necessarily to the cloud, like a different cloud provider. So we provide them that flexibility where they provide them the ability to run whatever model and whatever.

• 1:29:07Speaker 01

  Applications built on top of the standard ML models where the data resides for them. So that's where it's convenient for them.

• 1:29:13Tim Scarfe?

  Cool. So moving over to the agents discussion. Now, I'm very excited about agents. I think they, I mean, don't get me wrong. I'm not excited in a kind of, you know, it's AGI and the singularity kind of way. I'm just excited in the way of, I love building, you know, distributed asynchronous systems. I think it's very exciting. I think it's, it's really cool to think about these little like asynchronous units of computation at the application level. But of course, you know, using a shared computation fabric, which, which you guys are doing. So how do you think this paradigm of, of kind of software engineering is, is going to change things?

• 1:29:50Speaker 01

  Yeah, I'm definitely with you and like, I also like to build distributed systems, asynchronous one. It's very exciting that it's opened up. Again, going back to the event I attended last week, Intelligent Applications, it was a very nice event organized by Madrona, AWS and Microsoft. And it was clear, a lot of people, they were running around saying like, how about agents? What are you guys doing? What are you doing in this space? Like, everyone is excited about it, similar how they were excited about RAG, like we say, six months ago, right? But it's like, you want to find this killer app where it's used. So the problem with this, it seems like it's. One challenge is that it's not used as heavily in production as we would hope to. So it's a little bit early. Another thing, people, when they hear a nice buzzword, start to use it for everything.

• 1:30:33Speaker 01

  So whenever they say an agent, it becomes like, almost like a synonym of any application. So it's not very helpful, like you need to understand what kind of intelligence you encapsulate and what you mean by an agent in general. But I do think this is the direction we should go. Like, I'm not as excited if all the future, we're going to use GNIAs by running individual models here and there. And they still talk, if you have human in the loop, it's not very efficient. Like, I want the system to be able to communicate and exclude the human in the loop. And human being a designer at the top and overseeing the process, but not overlapping into every single conversation. It's the same as managing a company. Like, if I, as a CEO, have to be in every single meeting just to orchestrate the communication with everyone, that's like a failed company.

• 1:31:16Speaker 01

  Very quick. It's like I rely on them to have the proper meetings protocol without me and being efficient. I don't need to be there and not be in a bottleneck. It's the same here. It's very interesting how to build this sophisticated system out of those agentic systems, right? It's exciting. It's still a lot to be proven on how it will be used in real examples. Remember, people talk about fine tuning and fine tuning was used to a certain extent, but it didn't solve all the problems. It turns out that fine tuning for these models is also not very cheap, right? You still need hundreds of GPUs in many cases to do fine tuning and you need an expertise to do it. So a lot of people say like, hmm, even fine tuning might be too expensive for me right right now.

• 1:31:57Speaker 01

  Then RAG came up and it helps with certain things, but you realize that RAG without fine tuning is not as good as RAG with fine tuning. So like, like people look at, it it helps you with certain things like guardrails, but not with everything. But we anyway seen the real use cases where people benefit from it, but not of that concept alone is the answer. It's just all pieces of the puzzle that can benefit depending on the use case. But then ultimately with all this concept, you're building something more exciting and that become agents. I think everyone starts to talk about the agentic system. I lost track how many startups are around claiming that and fundraised recently around that space. But again, I want to see a customer tell me that I'm actually building it then I will be excited. Remember, we talk about enterprise customers.

• 1:32:41Speaker 01

  This guy is always going to have certain lag of adopting things. So I don't see them like adopting agents yet, but it takes some time. And I think we as a company, Sentimental, need to be ready when that comes. And that helps with us interacting with these GNI startups because they are the driver of that novelty of that technology. But ultimately their revenue and the future of that technology and the use comes from enterprise adoption, in my opinion, and cloud providers adoption. So it's start to be used there. Right now, I think a lot of that is just an experimentation phase. And I talked to a few companies, they're doing agents and they say, guys, like, do you need our help?

• 1:33:22Speaker 01

  Do you need optimization? Like it's like a running scale. They're like, oh no, everything is like, like few requests here and there. And we are calling chat GPT under the hood, like or GPT, like, you know, open AI API here. It's like, oh, obviously it's not at the scale where like we can help, for example, right? I think generally it's not, it's like they're just proving that something can be done. It's an important step, but it's not the scale step yet.

• 1:33:45Tim Scarfe?

  But still, I'm a big fan of your methodology, which is that, let's say, I've built this big distributed asynchronous system with the actor pattern and I'm looking at the logs. There's a logging actor and you can start to see patterns which can be optimized, which you wouldn't have been able to do before. When building these AI applications as well, what I've noticed is that part of the blocker is the complexity grows very quickly if you try to do it monolithically. Even if like with a RAG system, if you're building an information retrieval system, typically you're kind of like sticking together, you know, query auto-completion systems and then like, you know, re-rank systems and you're talking to all these different heterogeneous data sources. And, you know, the footprint of the thing is very complex.

• 1:34:29Tim Scarfe?

  It's very difficult to monitor it and deploy it and so on. And the reason why people don't use, let's say, the actor system is because I guess, you know, from zero to one, it's actually more complicated. And, you know, when you look at Twitter and LinkedIn and all these big globally distributed systems, they're already doing it. So it's almost like we need to say to people, okay, you need to start by building these distributed systems. I know there's a bit of a learning curve, it's going to take you a while to get there, but you need to do that. Clearly though, we need to come up with a new abstraction. I don't know whether we could have a new type of programming language or some new way of building these systems. I mean, what might that look like?

• 1:35:08Speaker 01

  Yeah, so I think there are several different directions we can go with that. And I've seen some attempts to solve the problem. There are definitely people at Berkeley I've seen working in the space, like an data brig's working on compound system that I mentioned before. It's very promising. I've talked to a few of my graduate students also as one of the next interesting big problem to work on. So I think we would need, not necessarily a new programming model, but we need a new level of abstractions to work with the systems. And you're absolutely correct. We not only work, we need to identify things. We need to be able to monitor and debug them. Like, remember, as with all of the system, like, you're never smart enough.

• 1:35:50Speaker 01

  If you build it, you know, the system as good as you can, you're never like smart enough. To debug it, right? So that that's a conventional problem. It's like we don't only, it's not only a matter of building it. You're gonna build it with issues, that's for sure. You need to be capable of and then monitor and debug it afterwards. Everything that is asynchronous distributed is always difficult to debug, right? Making it deterministic, it's a known problem fundamentally from the auto per- optimization efforts we had for many years. So essentially, we need to make sure that we build that as a part of the system that they are scalable, monitorable, profileable. And we can actually can debug them with reasonable pace. All of that is still an open problem, right?

• 1:36:35Speaker 01

  I think some of the things we learn from building classical distributed system would help. But ultimately, there are gonna be new uh new problems there. First of all, I believe those systems are gonna be very, very heterogeneous. So I believe there's gonna consist of running things on GPUs, GPUs, other accelerators like TPUs and other components. Because it might turn out each piece would run in different environment, right? That particular model might reside on this cloud provider and things like that. And you need to orchestrate all of that. So it would be like building software that has to be orchestrated across the globe type of thing. Potentially running in different cloud environments. You also want to make sure you build smart version of migration, right?

• 1:37:19Speaker 01

  People work on that for a while on CPU world, but now you would need to do this at, you know, in AI. Uh space, so you need to make sure you had checkpointing mechanism, different, like a lot of things are needed. Make sure to manage that complexity. It's a very open, interesting problem. It's very interesting to build products like that. It's also very interesting for any future, you know, graduate students looking for a pro- uh projects to work on. That's a very promising area, in my opinion, right? A lot of very exciting things will come up. And I'm only touching the system level aspect of it. I'm sure there's a lot of excitement on the AI space on what can be built with things like that.

• 1:37:55Tim Scarfe?

  Yeah, indeed. And of course, there's no such thing as a free lunch. There are always bottlenecks. But I love that book, The Mythical Man Month. And it was saying, even if you have a perfectly partitionable task, you know, you still have this kind of curve, right? You know, at some point, you can just add more and more developers to the problem and it doesn't go any faster. And I'm sure there would be similar bottlenecks. But it's still cool, though, just to have a whole bunch of engineers that can work on different units of the system. And they can work independently and they can deploy independently. And that's very exciting. But I wanted to move on to the ML perf and ML commons. Now, you're a founding member of ML perf and the research co-chair at ML commons. Now, this is all about benchmarking in AI. Can you tell us about that?

• 1:38:39Speaker 01

  Yeah, so what was the motivation to, first of all, build ML perf before it was all created? So my own journey was like that. So I was, as I said, at some point here in 2016, I was at Microsoft Research, right? And I want to understand the problem of utilization, how well these workloads are running. Because I had a theory that the guys running those, a lot of those experiments at scale, never actually built sophisticated systems themselves. So they never looked at that. And it turns to be true. And then the second problem back around 2016, 17 was almost everyone claimed back then they are thousandx better than everyone else. And when that happens, you know, someone was cheating and majority cases, everyone was cheating.

• 1:39:22Speaker 01

  And they were cheating because they're using different data set and non-traditional methodology. A lot of things you can cheat around the system. And we've seen this before, the world where was this explosion of GPUs being built in 70s and 80s. At some point, got a benchmark called spec that helped to build it. And there were people that clearly build it and have this experience. So I thought, it seems like that benchmark doesn't exist. So when I started my academic journey at University of Toronto, my first four graduate students all work on building benchmarks for ML essentially. I said, look, I'm a systems guy. I want to optimize it. I can't optimize it until I know what's happening. I need to know what the things we can do right now and what's problem, what's not the problem.

• 1:40:07Speaker 01

  There are models, there are different data set. I don't understand. Everyone claim ridiculous things online. It cannot be true. So we start to work on that and we release the first benchmarks you called TBD. So it was training benchmarks for D N N's back then. But also it was called TBD because the models were changing so quickly. Every time we finish an iteration, there's a new model there that has to be excluded. So it was always to be done. So we called it TBD. I think it's still like the website is still alive, probably somewhere. So we make this TBD suit available to the public. And in parallel to that, I was talking to my friends at Google about potential collaboration on different research topic.

• 1:40:49Speaker 01

  And I mentioned somewhere among other cool things I'm doing like, oh, I'm doing this benchmarking for M L. And then one of the guys there, Cliff Yan, who is one of the architects of T P U said like, that's very, very relevant. There is something happens between us, Biden and Stanford that's not announced that we're talking about. But like, you seem to be a good fit. Like, do you want to partner? You want to do it on your own? I said like, oh, I was almost open to partner. I didn't know you guys are doing this. So this is how I got into this thing. And then after a few months of emails, there was a first meeting at Stanford, right? Back then it was pre-COVID, like we're talking about 2017, 2018. Everything was in person, so I fly in there very regularly.

• 1:41:31Speaker 01

  And then we just meet with a small group where I try to identify like what that benchmark should be. It's very different workload from the traditional C P U's. It's very different from anything else, the world benchmark before the scale is very different as well. And there's training, there's inference, there's so many different use cases. There's like a large scale, small scale, there are so many categories, very hard to pick one particular thing to measure. So we sit down and start to build working groups, start to build methodologies. Several of your early benchmarks in M L P R were built by my graduate students, right? We contribute one of the first speech benchmark from the base on the deep speech. And a lot of that was done at the beginning by the academic community, where people from Berkeley, like people like Dave Partison were involved.

• 1:42:15Speaker 01

  The people from all the industrial companies you can think of gradually start to join. So it started by about maybe fifteen, twenty people in the room. Now I think I'm all common, I don't know, thousands and thousands of people, many of the work groups, like, like, it's just an avalanche of people. Uh, rather it was half and half at the beginning, academics versus industry, but obviously when it starts to get traction, like it starts to be over dominated by industry. Now it's a huge organization, right? That does a lot of good things to people. It provides that proper measurement. Uh, you know, uh, state to say where you are. You cannot come right now and say I'm thousand X better than Nvidia or Google. If you claim that, go come to ML perf.

• 1:42:55Speaker 01

  Uh, and benchmark and ML commerce will release the results you really want thousand X. And usually the answer, no, you're like cheating somewhere, right? So right now those claims uh gradually disappear and essentially uh uh. People are not claiming this anymore, like people know what reality looks like.

• 1:43:15Tim Scarfe?

  We've got this big problem, don't we, in in benchmarks that they don't represent real world usage of the thing. I mean, it's been spoken about Kaggle. It's been spoken about MMLU and L N Sis Arena and so on. And, you know, how do you, and of course, you know, benchmarks get good hearted as well, which is that when they become uh a measure, they cease to be a good measure. How how do we actually make these things faithful?

• 1:43:42Speaker 01

  It's a very hard problem that people try to solve for decades. Um, the question is, um, can you make them perfect? Or you just want to get them reasonably good enough for the purpose they are reasonable. So essentially, uh, when we build ML prof, we know we cannot make it perfect. It's too complicated of a problem. But we try to make it sure reasonably fair because the rest. Uh, alternative was a wild, wild west that exists online back then, where everyone claims that we're better than everyone. And potential customers were all got lost on what can be done. So essentially what uh.

• 1:44:17Speaker 01

  You know, there are all these benchmarking crimes that you can do. And it's the choice of the company that present their benchmarks and how honest they want to be. So we create the rules that the playing field is reasonably honest. Like, for example, for training, very early in the day, we realized that some company has way more resources than others. So they can win in convergence, just because they had better hyperparameter tuning. And they had, they can spend, you know, millions of dollars in advance, find the right hyperparameters so they converge faster. So in order to make things more equal, we say like, look, if someone is doing that, others can steal your hyperparameters in the intermediate round of results submission.

• 1:44:58Speaker 01

  So if you ended up using this, if, say, you are like a big company, a hyperscaler or Nvidia, and if you do that, it's okay, it's totally legal based on the rules, but others can use the same hyperparameters that you do. So there, we try to equalize the playing field as much as we can do. Obviously, the big companies like Nvidia and Google still have way more engineers to do the manual tuning. But at the same time, it's important to make their field as equal as possible for people. This is why we're, we'll think we're very successful in MLPerf. You have big companies submit their results and small entities and even academics can submit some of the results, right? Again, it's very far from being a perfect, but it tried to keep up with what's going. For example, this MLPerf has Llama seventy billion parameters for inference.

• 1:45:44Speaker 01

  That's not the best that Llama has taught for, but the next MLPerf, say, in the February would have Llama three with four hundred five billion parameters, right? As an inference benchmark. So the goal for MLPerf was never range their state of the art faster than the available best model, but something that being perceived by the community as a stable status quo, like they're really, really good model. So it means that by that time, there might be another Llama around, but we still benchmark the chips and the software stuff, stuff that's something that is really reasonable, right? So no miracle there, we don't solve all the benchmarking problems. You can still overfeed things, like you can still put, say, hundred developers and manually tune to get just. Great results on these models.

• 1:46:28Speaker 01

  And then a new customer come to you as a very different model and they get very poor performance. This is on you as a company, right? Like we cannot stop that from happening. But we create an equal field for people to demonstrate result on a representative set of the models. And that ability to do performance cheating is getting harder and harder because there's so many models across different benchmarks. Good luck doing this manual, you need like hundreds of engineers to do it.

• 1:46:51Tim Scarfe?

  Yes, I mean, you spoke about, you know, reproducibility, leveling the playing field. The good heart thing we kind of spoke about earlier before, you know, because you could optimize for accuracy. You know, we tend to optimize for headline metrics and then maybe we we lose something on on performance. But have you noticed that the diversity of submissions has actually kind of influenced the evolution of the benchmark?

• 1:47:13Speaker 01

  It's definitely the case, like I'm obviously like running a company for more than two years. So I'm not as closely working with them as I used to be, but I clearly see the communities growing. And it creates a lot of different camps there that people doing algorithm, that people even start to do. You know, data benchmarking specifically for ML, people are doing storage benchmarking, like all the different aspects of the system. So and they're separating different working groups. That's how you manage the complexity of so many people involved. And I was involved in several different groups at different stages and obviously was helping with research and promoting ML perf on the academic stage. And I think it becomes like, if you ask me whether I'm more perfect and I'm more successful, to me it is, and I'll tell you why.

• 1:47:57Speaker 01

  Like every time right now you see the announcement from Nvidia, from Jensen about like they're new cheap. It's like what the numbers they're citing, that's what they're citing. They're not citing any special numbers that they got Nvidia, they're citing their ML perf results, right? And there is a good reasons for that, right? Because that is not just measured by Nvidia, that was validated by their peers and competitors. It's not perfect, right? But it's still someone else go and look at your numbers and they can reproduce and run what you're doing, right? To an extent, so they can see that you legitimately got these numbers and they can see where it's coming from, right? So yes, and all the tricks become public and actually push the science further because like all of that is visible for the rest of the world. It's very important.

• 1:48:40Speaker 01

  It also make a fact on the fate of the company. I remember there was one example where. Uh. One of the about hardware startups uh decided to participate. And they had a very good chip, but it turns out to be not as good as some others. So they bring up their solution and the results were not great. And the investors then said, oh, we're not believing that anymore. They took the money away, right, gradually, right? So it it is a powerful tool, right? And it's some more honest tool because like before they were telling investors that they're better than everyone else. But now they came and measured and they've seen they're not. They must still have a good system, but they will say two X worth and like say something like Google TPU. And because of that, they just like it.

• 1:49:24Speaker 01

  It's not something that that was proceeding as active. So that's very important because otherwise it's very hard to validate all these claims, how good performance you have. And and like what kind of a system you have. I think after that, the number of uh BS online about the performance reduced dramatically, right? Because they, if you claim like that, people said, go to ML Perf, validate your numbers, right? And you have, like, if it's so good, why don't submit it? Why don't make it visible? And even before people were open sourcing things, but when there is open source things, but no proper methodology, it doesn't matter. Like, how many different rest net results we had was different data set that claimed ridiculous things around the performance? And there was always some methodological. Here, everything is under control, right?

• 1:50:06Speaker 01

  So like we limit the amount of variables that you can cheat around. So we get you all the optimal parameters that you can pick, but you can cheat against others with those parameters.

• 1:50:17Tim Scarfe?

  So I wondered what your thoughts are on the the state of industry research. Now, um, we were talking about this last time that, you know, uh, Noam Brown tweeted. And I I think he got into a bit of a um, a bit of a spat with Jan LeCoon. But he was kind of saying, look, you know, we've actually got millions or even, you know, maybe more than that, millions of users using our stuff. And as as an academic, you've just got to convince an area chair and like, you know, two two reviewers. But how do you kind of contrast the the kind of industrial research that you're doing versus academia?

• 1:50:48Speaker 01

  Yeah, and it's like all my life I like to sit on the both sides of the fence. I'm very glad that my family also supported me with that because it's actually taken a toll on you. So there is a reason for that. So there is a benefit of being in academia because you work on clean, nice ideas without any industrial biases and the immediate deliverables, right? Remember in industry all at the end of the day, we are for profit. We need to generate revenue. A lot of decisions have to be based around that. And it's good and bad at the same time. In academia, we're solving problems, right? And we are usually very honest or we're supposed to be very honest in how we do it. And yes, you can criticize the academic system, how you select the papers, but that depends on the community.

• 1:51:31Speaker 01

  I actually belong to the systems community, like in some of our top conferences. In order to get in, you need to get five to nine reviewers to accept your papers, right? So it's actually not that random as it might sound, right? And the bar to get in the top conference is actually quite high. It's very different from a mal community that had tens of thousands of submissions and thousands of accepted papers. But that's part of the difference between different communities. So I think it's unfair to say that everyone has necessarily low bar of accepting the papers. But it's very important because the innovation comes from them. Remember, it's not really the industry that invented a lot of these advanced things.

• 1:52:10Speaker 01

  Like deep learning, like, you know, a lot of those things were invented actually academically, academically, because academic thought, it has the potential, they believe in it and build it. And they didn't require tons of resources to make it work at the beginning. The industry is very good at making it at scale. So when people realize what the deep learning potential is, Microsoft Research, Google, OpenAI as an example, starts to scale the technology. So like Google, for example, is behind Attention Models, right? Invention and BERT. And then OpenAI took, you know, a lot of that and start to build further and further and scale further, right? So that is a good thing that industry can do.

• 1:52:54Speaker 01

  So the industry was always important because I Believe in academia, there is this fallacy, you can go into solving an irrelevant problem. You just become, it's like you become like a pure mathematician, you just pick up a random problem that looks fancy and you just solve it and publish a paper. I never liked myself, like that's something always bothered about me. I always want to be close to industry because these guys have the right problem to solve. This is why always in my career, always at least one hand was uh one leg, whatever the right way to say it was in industry, right? Uh because I want to understand what the real problems exist there, right? And that's the way to contribute. And then there's evolution in some of those things that in industry usually have the right problems,

• 1:53:37Speaker 01

  but they not always have the time and engineering focus to solve the right problem. They solve what's needed for their customer or for the next deadline. And this is why for many years I was always taking the problems from industry and solving them academically, right? And then I had to move uh now for uh do the focus slightly differently because I realized the solution I came up to the problem that exists in industry, I cannot scale to the enterprise quality in academia. That's the limitation of academia. Like I can hire twenty graduate students maybe, but I cannot hire fifty. And they're not going to generate the code that's needed. They had a different uh uh goal. They had to publish their papers to get their GPs and master theses. Rather than actually uh building an artifacts, right?

• 1:54:22Speaker 01

  We do a little bit of artifacts, but so it's a very tight interplay. So I reached the stage in my career that like, well, I built something really interesting, done it now become practical and I realized that if I don't do it, like others like kind of ignoring me, they're thinking it's not important. So I think I'd better build the company and do it the right way, right? And remember that was all the pre-JGPT. So it's very nice interplay between the two fields, right? And I think both need each other. So it's not like, oh, research lab, like people between research labs very frequently move, right? And I always uh like to think that we need to have also one of these best leading labs in the world, like, like, you know, like Bell Labs was uh at the time and AT&T lab, but all these labs were like.

• 1:55:08Speaker 01

  A place where the brightest, you know, graduate PhDs want to go and develop new stuff. It's always nice to make sure that we have something like that and Microsoft Research was like that for a while. Like I want us to have something like that moving forward as well. I think I would hope that someone would pick it up. I think Matter did it to a certain extent with AI, right? And Yann LeCun pushing a lot of really cool stuff there, like in AI space. But I think I would love to see this more beyond AI as well, like in other, in other areas as well. I think that's a very important interplay in between these two areas. And I think I spent all my life, you know, being either on one side or the other, but always staying in connections with the other side.

• 1:55:49Tim Scarfe?

  And what do you think about the trade off with exploration and exploitation? I mean, I always feel that there's a bit of a basin of attraction, even, you know, Sarah Hooker called it the hardware lottery. And I suppose that's kind of what we're talking about now, that so much of the deep learning revolution was just kind of influenced by the investments in research and hardware and so on. And you can just build this whole directed acyclic graph and we can do a bunch of graduate student descent on top of that. But, you know, do you think that just having a few random people working on completely crazy ideas might, might lead to something?

• 1:56:22Speaker 01

  You need to select those crazy people carefully. I think you need to have some of that. Remember, like, there was a reason why Jeff Pinton, who actually won a Nobel Prize today, right? Like, you know, a great achievement for everyone, like, you know, who knows Jeff and Jeff obviously personally. But like, again, talking about that, like, for example, some point, like in US, they just didn't believe in this whole field of AI, right? And I'm glad there was a country like Canada that actually was willing to invest their people with this crazy ideas that that is feasible. And see the potential in that and believe long enough until they can reach the stage that actually practical. And how to for a sudden and change the world, right? So you need people like that. The fundamental research has to be there.

• 1:57:08Speaker 01

  But you need to also see where it's like, there is a chance for this is just like a crazy person, like you can't just give everyone. Like, there should be a selection process and there is like, like you select. The people that have the chance to change the world one way or another and give them resources to make this happen. Whether there's like grants exist, this is why industry also give money into research grants and give us the chance to innovate, right? I received some of those awards from like Google, Facebook and you know VMware and others. It was also very helpful to, you know, advance your research in the areas that otherwise you can't afford because you had the budget, right? But in terms of exploitation exploration, I think it's a very good problem in general. And I think there is even resemblance about what we have in the space right now.

• 1:57:52Speaker 01

  I actually think about spending money on training is exploration. And using inference and deploying exploitation. And for a long time, I was very puzzled that we were mostly doing exploration without no exploitation. I was like, like at some point that has to change, right? People has to start to actually deploy those models. And I'm glad to see that starts to happen first, like, like at a reasonable scale. I started like last year, but also very significantly this year, the clear trends I see in Nvidia reports and others that. You know, people want to deploy more and we reached the stage where, yes, the training is. Pre-training is probably only can be done by a dozen of companies in the world.

• 1:58:35Speaker 01

  And fine tuning maybe by a few hundred. And the rest of the world is going to use like the fine tune models and, uh, you know, rack and like building on top of that. And it doesn't limit the creativity, right? It's still actually a good thing for the world. But I think like someone needs to say like, like enough is enough. We don't need to like, you know, retrain. Uh, ten times like the same thing again and again, and things will be done in a more incremental way.

• 1:58:59Tim Scarfe?

  Imagine going back in time to the nineteen eighties and I think was, um, Hinton was in Toronto even back then, right? I'm not sure when he moved over there, but imagine saying to him that all of this stuff you're working on. Um, he would have been kind of intrigued that we're having the opposite conversation. We're talking about graduate student descent inevitably on deep neural networks. And, you know, imagine telling him he was going to win the Nobel Prize. That would have been amazing.

• 1:59:22Speaker 01

  Yeah, especially if you tell him you're gonna win a Nobel Prize in Physics, he probably would be puzzled, right? Yeah, but like, it's like, yeah. But yeah, it's it's it's an interesting, like, he was always a strong believer as far as I know in what he built, like, it was evident from his whole career. But the the scale of it, I feel like it's even impressed him, right? Like, it's it's just and Vera, and I think he had a, I mean, like, the last time I talked to him on one of the flights from Toronto to San Francisco, like, he was had very serious concern right now where it can goes with these eight years. Because it seems like those models are improving at a very rapid pace. And we need to also be very careful about where they can bring us. So he had a very valid concern about how powerful it is, right?

• 2:00:04Speaker 01

  So it's a little bit like the story, you know, Oppenheimer was like in a clear, you know, bomb, right? An invention and then not being able to control it, right? So it's again with the AI right now, right now, it's in the hands of the world, right? You contribute, but you have no control. You can't say that's a concern, but you need to convince others you don't have a control over the technology. And it would never become what it is if you don't make it available to the people. But I think that has to happen one way or another, like, it it's just, like, it might have been happen later, but it has to be invented. It was asking for it, like, like, we need to learn, like, we were looking for a way to learn differently, right? And that was imperative to have something like that.

• 2:00:47Tim Scarfe?

  Yeah, it's interesting, the point about Hinton's fears. In a way, no one disagrees. I think everyone agrees that this is transformative technology and the world will never be the same again. I think people on the left think that it it's kind of changing, you know, it might be it's reducing our our agency or it's kind of dramatically changing society. And of course, some people think that it might recursively self-improve and become super intelligence or something like that. But everyone seems to agree that this is a very dramatically transformative technology.

• 2:01:18Speaker 01

  Yeah, it's hard to argue about that. It's like, it's like for a long time, I was asking, I was asked like those questions like, do you believe it's, if it's just hype, is it another bit mining or is there a real substance there? And so like, look like everything. As a scientist, I know there is a real substance behind it. Yes, hype is always on top of it, but there is a real substance. And now, yes, I agree with you that it's really transformative. We are never going to be the same, right? You can't just forget about what we learned or we're able to do with this technology. Tell it to all the students that use ChatGPT to pass their exams, that it doesn't exist anymore. They need to can keep memorizing things that, like they're not going to go back, right? That let's forget about it. So essentially, the technology is there. We just need to understand how to operate with it, right?

• 2:02:00Speaker 01

  I, I don't think it would be fair to say that humanity, humanity never had anything to disrupt. We had industrial revolution before, right? And people were worried about the machines. I also like those pictures from eighties, where there's all these high school teachers demonstrating in front of White House with calculators. And they're arguing that they, uh, that they shouldn't store, uh, they shouldn't allow calculators in high schools, because kids would stop, uh, stop know how to learn and count, and they're not going to be as intelligent as they used to be. That was like, that's a reality, that it was like a perception that that's a danger. And right now, we smile at it, but like, you know, we, we just adopt it like that.

• 2:02:40Speaker 01

  And we know how to use it, like when they teach kids right now, that calculators use at the proper time, at the proper age, right? But we don't try to outcompete the calculators. So I think that would be the situation for a long time with the AI as well. Uh, I don't think personally that we are that close to superintelligence yet, right? Yes, it's exciting technology to play with, but there's a lot of missing gaps. It's hard to say how long is until we get any form of real intelligence from it. But it's already very capable. I'm saying that that's going to be a gradual process. I don't feel like we're gonna wake up one morning, it's gonna be Terminator scenario and everything is captured and we, out of a sudden, that's, uh, intelligence get outside of the control.

• 2:03:21Speaker 01

  You have to build this intelligent process, like we are, uh, in the early innings, in my opinion, of building it.

• 2:03:27Tim Scarfe?

  It's interesting how our perceptions changed over time because it was a similar thing with chess. I think in the in the late nineteen seventies, it was thought that if we could build a chess computer that could be, you know, the best human, then that thing would be generally intelligent. And of course, with experience, we've just changed the way we think about things. But just final point on this, you know, like the the physics thing, because you can think of deep learning as being a form of physics in the same. You know, what is physics? It's modeling the world that we live in. But Noam Chomsky said that he he thought that these language models are not a theory of linguistics because they don't, you know, a theory should explain. It should carve the world up by the joints and actually tell you something about what it's not as well as what it is.

• 2:04:10Tim Scarfe?

  So what do you think about this as as a form of science? Do you think it's legitimate?

• 2:04:15Speaker 01

  I think it's legitimate. And in general, I like when the awards like that are given to interdisciplinary stuff in general, that in the intersection of multiple different things, like say physics, biology, computer science, and this particular examples. Because I think those are where the biggest breakthrough happens, not in one narrow field, but actually across different areas. It requires much bigger vision. I'm really excited about the whole field that that's got a recognition. Award is an award, there is definitely, you know, credit comes with this. But what I think is it is definitely an effect that we're all excited about this technology right now. And people want to recognize it because it's so transformative, right?

• 2:04:55Speaker 01

  And again, people are inventing a pure elements of, say, biology and physics deserves it as much as the people that inventing their methodologies or measuring tools or even modeling simulation tools. They are like as important, like, for example, when we think about the brain, it's important to study the brain from the physical perspective, but there's only that much we can do. But so is modeling, because the modeling allows us to do things we can never can do in the physical world. So it's important, especially if it's proven to be so disruptive, right? It's not become just like a tiny experiment. It's actually changed the world in a way that is hard to imagine before. Like, you ask someone ten years ago how disruptive AI would be in our life, it would be very hard to imagine, right?

• 2:05:39Tim Scarfe?

  Do you think that, let's say Nvidia, for example, do you do you think that their stock price is going to continue to go up? Do you think that our compute demands and GPU demands are just going to continue to explode? Or do you do you think it will level off?

• 2:05:52Speaker 01

  Wow, great question. So, uh, well, the short, honest answer, like, for example, I have some stock myself and I I still do own Nvidia stock, so I still believe that they will grow. I think that's the most honest answer. Like, I can come up with any explanation, but that's the reality, what I'm doing myself. How far it can go is a hard question, right? Like, but I I I do think they deserve to be where they are right now because they believe in this technology early enough. So they got lucky in a way that, like, the chips they had at the time was, ended up being a good fit, but a lot of people lose those opportunities and they didn't. They

• 2:06:33Speaker 01

  understand and capitalize and do a huge investment into software ecosystem beyond just building chip, which is a great vision by the company and their founders on like what's really needed. And I think they're great in benefits of that. How far they can go with the panel, like, like, a lot of that things in where the stock is, it depends on people's expectations. Sometimes they show phenomenal numbers and people's like, oh, my God, they didn't grow by another 10x, so they're like not doing well. And it's like ridiculous, like their their revenue and everything was growing by huge numbers and the stock next day can go down. I look at that, it's like, that's ridiculous. But there is like a lot of that gambling that goes into stock shares, but I think as a company, they had a chance to become, um, uh, like, like, continue being

• 2:07:16Speaker 01

  a transformative company in the space. But I also think they're not going to be the only hardware providers moving forward. There will be other people with their hardware as we see right now. And because of that, uh, the world can utilize all of them. But I would expect for the next, say, safely to say, I would say, three years, right? I can't see that anyone would, uh, buy person video at the top, right? I think they're going to be a dominant player in the space for the years to come. And definitely for the next few years, I don't see anyone who are going to be easily top them. Like, I don't see this coming. I don't know what that means for the shares and the expectation in the stock, but I think they're going to be there, one of the top companies for their, for some time now, for a good reason.

• 2:08:00Tim Scarfe?

  Awesome. Ganadi, I thoroughly enjoyed this conversation. Thank you so much.

• 2:08:04Speaker 01

  Me too.

• 2:08:04Tim Scarfe?

  Folks at home, if you want to work for Ganadi, please reach out to centml. I'm sure you're hiring good people, right?

• 2:08:13Speaker 01

  Always growing, always hiring, so please reach out.

• 2:08:16Tim Scarfe?

  Thank you so much. It's been a pleasure.

• 2:08:18Speaker 01

  Yeah, it's been a pleasure for me as well. Thank you.