E47: GPUMaxing with Dr. Ronen Dar of Run:ai

Nathan Labenz sits down with Dr. Ronen Dar, CTO and co-founder of Run:ai, an Israel-based company that helps enterprises train and deploy AI models by optimizing GPU usage. The discussion covers how chip makers can meet the soaring demands, geopolitical fears, to the best practices companies can secure compute capacity. If you’re looking for an ERP platform, check out our sponsor, NetSuite: http://netsuite.com/cognitive We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. RECOMMENDED PODCAST: The HR industry is at a crossroads. What will it take to construct the next generation of incredible businesses – and where can people leaders have the most business impact? Hosts Nolan Church and Kelli Dragovich have been through it all, the highs and the lows – IPOs, layoffs, executive turnover, board meetings, culture changes, and more. With a lineup of industry vets and experts, Nolan and Kelli break down the nitty-gritty details, trade offs, and dynamics of constructing high performing companies. Through unfiltered conversations that can only happen between seasoned practitioners, Kelli and Nolan dive deep into the kind of leadership-level strategy that often happens behind closed doors. Check out the first episode with the architect of Netflix’s culture deck Patty McCord. https://link.chtbl.com/hrheretics RECOMMENDATION: The AI Scouting Report Playlist Parts 1-3: https://www.youtube.com/watch?v=0hvtiVQ_LqQ&list=PLVfJCYRuaJIXooK_KWju5djdVmEpH81ee TIMESTAMPS: (00:00) Episode Preview (00:51) Introduction to Dr. Ronen Dar (03:30) Run:ai's technology and what differentiates it from other solutions (06:00) Today's market compared to when Dr. Ronen started five years ago (13:40)Run:ai on market competitors like mosaicML (14:55) Sponsors: NetSuite | Omneky (22:00) The process and best practices by which companies secure compute capacity (25:00) Dr. Ronen explains the GPU shortage (31:50) GPU solutions (36:00) Relative pricing across major providers (41:00) What other chip makers are going to be relevant? (49:00) Global outlook for chip production (52:45) Worldview around the US-China AI race (58:00) Can controls on hardware actually control access to AI? LINKS: https://www.run.ai/ SOCIAL MEDIA: @labenz (Nathan) @ronen_dar @runailabs (Run:ai) @cogrev_podcast SPONSORS: Shopify | NetSuite | Omneky Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive -NetSuite provides financial software for all your business needs. More than thirty-six thousand companies have already upgraded to NetSuite, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you’re looking for an ERP platform -> NetSuite: http://netsuite.com/cognitive and defer payments of a FULL NetSuite implementation for six months. -Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that *actually work* customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Then we saw how difficult it is to train models on a lot of GPUs doing distributed computing. So our goal, it's still our goal, was to simplify that, to simplify the way better scientists can train big models. Right now, you have these open source models, you can train them on your data. You don't need 10 to 15 GPUs to do that, right? Open source lab, it's free for for research and commercial users, if they say. That's really, really exciting.

• 0:35Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, I'm joined by Dr. Ronen Dar, CTO and co-founder of Run AI, an Israel-based company that helps enterprises train and deploy AI models by optimizing their GPU usage. With companies around the world racing to adopt next-generation AI systems,

• 1:18Nathan Labenz?

  we first discuss Run AI's technology and what differentiates it from other solutions in the market, before then getting Ronen's perspective on today's market for AI chips. From the ongoing shortage, which he expects to go on for a while at least, to the process and best practices by which companies secure their compute capacity, to the relative prices across major providers, the prospects for chipmakers other than Nvidia to meet the soaring demand, the geopolitics of chip production, including the view from Israel on the US-China rivalry, and finally, the prospects for so-called compute governance to effectively control the pace of AI development, whether in China or anywhere else around the world. As I mentioned on our July 4th episode, one goal we have for the second half of the year

• 2:03Nathan Labenz?

  is to speak to more researchers, builders, and entrepreneurs who are working on AI outside of the United States. And this conversation with Ronan, who combines deep technical expertise with a global strategic outlook, was a great first step in that direction. If you have any other suggestions, please do let us know. You can always email us at [email protected] or DM me on Twitter, or should I say X.com, where I am at Labenz. Finally, for now, I encourage everyone who hasn't already to circle back to my AI scouting report, which is posted exclusively on our YouTube channel. It really is the best way that I know to get up to speed on the fundamentals of AI progress and the most important recent development trends.

• 2:49Nathan Labenz?

  And it's been gratifying to see that the YouTube audience seems to agree. With that, I hope you enjoyed this enlightening conversation about the physical substrate powering the AIs of the future with Dr. Ronan Dar of Run AI. Ronan Dar of Run AI, welcome to the cognitive revolution.

• 3:08Speaker 00

  Hey, Ronan, good to be here.

• 3:10Nathan Labenz?

  So I'm excited to talk to you about all things AI workloads, GPUs orchestrating them, and kind of the future of where we're going as really a global AI user base and market over the next year or two. For context, you are the CTO and co-founder of this company based in Israel, Run AI. So for starters, tell us about the company and what sort of AI workloads you're orchestrating for folks.

• 3:39Speaker 00

  So we run AI. As you said, we're based in Israel, but we have people all around the globe. And we have offices in New York as well and more in California as well. And as you said, I'm not in California. And so I spend a lot of my time here in the US. And okay, so we run AI. And we started in early 2018. We're starting to run since then. So we're an AI infrastructure software company. And we help companies and to train and deploy AI models. We're doing a lot of uh stuff orchestrating, I work on orchestrating GPUs.

• 4:18Speaker 00

  And I think um we're bringing in a lot of technology very close to the GPUs, to the GPU level. And and to the cloud, the edit more, over edit is more, we have technologies to up like uh and optimizing the usage of GPUs and optimizing how and workloads are being orchestrated, scheduled in the in cloud and environments. And we're bringing a lot of tools or M and Ts to easily deploy, train more models. And we're working on big enterprises right now, the rubber solution is quite horizontal, so we're working with big enterprises, again, industry with Sony's mobile customers, we're working with the finance industry, yeah, piano and then on the password.

• 5:04Speaker 00

  So we're working with a research institute, it's all MIT, you can yeah, on it all daily, he's on top of around here. And we're arguing, we're seeing this year exciting, exciting things that are happening in our space we see the last six, seven years. And that magic stuff happening right now, so it's very fun to be the place that would be we are right now.

• 5:27Nathan Labenz?

  Maybe give us a little bit more of the history, because it's funny, you know, it's only been five years, right, that you've, since you founded the company. But obviously, a lot has changed. The types of models that were available to run and the kind of scale of training runs that people were attempting in twenty eighteen is obviously, you know, much smaller than what we have today. So what was kind of the, you know, the market that you anticipated going after that inspired you to get involved in starting a company? And then how has that, you know, how is the actual market as it's shaped up, deviated from your expectations?

• 6:04Speaker 00

  Yeah, that's not a great question. So in fact, right, we started in twenty eighteen. And when we started, it was all about computer vision applications, right? So I think the big breakthrough in the industry for machine learning and deep learning happened around twenty twelve, twenty thirteen with the ImageNet competition and researchers from Royal University training AlexNet and doing this big breakthrough in how machine learning models, deep learning models can get insights from images and videos. And back then, I think AlexNet was trained on two GPUs. That was the big breakthrough back then in twenty twelve.

• 6:45Speaker 00

  Two GPUs, right, to train the state of the art models from scratch. And then we saw a, an interesting trend of people training bigger models with more automators using more data and, and using more GPUs. And so a few years later, twenty fifteen, ResNet came out and ResNet was already trained on hundreds of GPUs. And so we saw this trend of computer vision models being scaled from a few GPUs to train the state of the art models to hundreds of GPUs, so one hundred X increase in the requirements

• 7:29Speaker 00

  to train state of the art models back there. So we started, I mean, back then we saw how difficult it is to train models on a lot of GPUs doing distributed computing. So our goal, it's still our goal, was to simplify us, to simplify the way data scientists can train big models and just to allow them to train huge models very easily, just one click. On a distributed computer. So we started with that. And then we saw that there are huge problems on distributed training, but around also how GPUs are being utilized.

• 8:11Speaker 00

  How GPUs are being out distributed and how ML is again access to GPUs. So we saw a lot of inefficiency there, a lot of complexities on just getting access to GPUs. And we solved those problems as well. We bought, as I said before, we built a lot of technology around that aspect of just getting access to GPUs and utilizing them efficiently. And I think we saw in twenty seventeen, twenty eighteen with dual breakthrough in natural language processing. So we saw again the same trend that we saw in computer vision. More people training state of the art models, NLP models, nine hundred models in twenty eighteen on hundreds of GPUs.

• 8:56Speaker 00

  So I think GPT one was training, I think around hundreds, using hundreds of models.

• 9:02Speaker 00

  This day, GPT four, just according to rumors, right? Now GPT four is closed source, but according to rumors, GPT four was trained on more than twenty thousand GPUs. So in four, five years timeframe, we went from training state of the art models on hundreds of GPUs to tens of thousands of GPUs. So another one hundred things that they're computing requirements for training state of the art models. Models become bigger, more data, more computers. So we see this trend once again, and we see now a lot of companies seeing what can be done with generative AI and language model.

• 9:46Speaker 00

  And they are actually to, yeah, actually to, to, to bring that, those capabilities into their organizations. For sure, AI is going to transform industries. And and people see that and I think people need to take action and and and bring AI into their organizations, how their business, how their business, how their industry is going to be, but it's all about AI. And they can work on that. So we are helping our customers to do that. Now it's about a large language models, about learning AI models, how to train them. Yet other people are still training your computer vision and models and models that are much smaller than those language models. But those those models we staged, we're wrapping with those models what?

• 10:28Nathan Labenz?

  So where do you play in the overall stack or architecture of all this kind of stuff? And I mean that maybe in a couple of different ways, like conceptually, there is obviously a significant divide between training and inference. And then also there's kind of, you know, vertical integration, right? You partner with Nvidia. I wasn't quite able to figure out, like, are your customers primarily like managing their own physical computing resources and you're a software layer that that complements that? Or do you have, you know, deployments across all the, you know, the major public clouds? You you know, describe kind of where, how far the the tentacles reach.

• 11:09Speaker 00

  So we can run our platform, whether the the customer wants their GPUs to run. So we have customers run the whole platform on premises. And we support many of that solutions. We had a lot of, and it depends custom, we are, as we know, connection to the format. And we have a lot of customers in the cloud and managing small clusters of GPUs with tons of GPUs. And we have customers running thousands of GPUs in just one cluster, managing all AWS. For our way, we are also customers in Azure and Google.

• 11:52Speaker 00

  So we're we're the so-called hybrid solution. So with run AI, you can get just one platform which with which you can run workloads on premises in the cloud, in any cloud. And it's all all the same interface for the same, with the same tools. And so where we're in the stack, we're we're running with Kubernetes, with cloud edits. So we can run on any Kubernetes. It can be Kubernetes, around any managed managed solution of a cloud provider. So we read it all and shift. And then it uses a man who thinks they can use our tools to train their models or deploy their models.

• 12:37Speaker 00

  Right in person, it's more training any friend and and we're really taking an integrable. And the approach, so we can integrate with any actual with any tool that runs on Kubernetes. So we we working with a lot of tools in the M N ecosystem. And certain tracking tools, added to the description, workflow description tools. So being open, I think it's really important because the field moves so fast and new tools are being created every day. So that was one of our, you know, that was one of our weight law, right? We want to be as open as possible.

• 13:18Nathan Labenz?

  A lot of that sounds, you know, for we just had another episode with a couple of guests from Mosaic M L. And, you know, I've I've made a couple of jokes about the cognitive revolution bump that obviously, you know, led them to a good outcome very shortly thereafter. I don't want to get overly bogged down in kind of comparing, you know, one company against another. But how would you kind of compare and contrast the business that you've built versus how you kind of see Mosaic? Is it just the sort of thing where business is growing, you know, the industry is growing so fast that you guys don't even worry about competition? Or do you see like that there are kind of head to heads and you have like meaningfully, you know, different positioning in the market?

• 14:00Speaker 00

  Oh, so Mosaic, first of all, they started after us or they started like two and a half years ago. And we started five years ago. Mosaic is really have clear focus on generative and that language models. We're more general than that. But I think that the main thing is that we we.

• 14:22Speaker 00

  We came and we built our platform very much from the bottom up. So we came from the GPU itself on the hardware itself. And we saw the issues and problems we utilize on GPUs. So when we built our platform, we really had this very well. The software stack, the transport out of GPUs, and we started limitations of those software libraries. So I think so.

• 14:52Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 14:55Speaker 00

  And we built two main components, I think. So one of the components, you know, I see the technology. So we built that, one thing that we built is GPU optimization. So we made what we call like the API or the virtualization, CUDA virtualization. So we sit at the CUDA level and we intercept CUDA calls and we control access to GPUs. And we do that for enabling better access to GPUs. So with that layer, workloads can share a single GPU. So we have this feature called GPU virtualization. So we can virtualize one GPU and can be shared.

• 15:37Speaker 00

  And we know how to swap memory between CPU and GPU. So people can all provision GPUs to run media models or more models on the same GPU. So essentially, we bring out those capabilities and to utilizing GPUs much better. The second component is more scheduling and orchestration. So we saw that Kubernetes is the state of the art infrastructure and management today with clouds. But Kubernetes was built for running microservices on commodity CPU. Whereas AI workloads are totally different. AI workloads are really using Texas or there are on GPUs.

• 16:19Speaker 00

  There's a lot of experimentation in terms of how data scientists, machine engineers, many of their workloads.

• 16:27Speaker 00

  And we saw that there are a lot of scheduling capabilities missing on Kubernetes that are actually available in other states, other states like high-performance computing, third-hPC, four um scheduling capabilities on the Hadoop ecosystem. Yarn scheduler is an amazing scheduler that was good, like ten years ago. It brought a lot of scheduling capabilities that were really needed around Spark workers. So we brought two different concepts on those work, on the HPC and Yarn. And we brought them into the Kubernetes world. So we're bringing a lot of batch scheduling capabilities, ability to preempt walkers, to queue workloads. And and and and to gag scheduling, so a lot of scheduling.

• 17:12Speaker 00

  And at the end, what it allows teams to do, it allows them to, first of all, get a guaranteed quota, a way to get access to their GPUs. But they are not limited to that guaranteed quota. They can always go, well, I'll use more GPUs, run more workloads. There are available GPUs in their clusters. There are idle GPUs. We allow other people in the in the in the organization to use those GPUs. So essentially, those scheduling capabilities are bringing, again, a lot of efficiencies into GPU clusters. A much more availability of GPUs.

• 17:51Speaker 00

  So suddenly, MFTs becomes much easier for them to get access to more GPUs, there is a on demand. So interesting that efficiency of GPUs are being utilized. And we're casing the availability of GPUs. And availability of GPUs is a serious aspect, a serious problem. And so I think that is really unique for us. The fact that we bring in a lot of efficiency and a lot of increasing availability of GPUs. I don't think that anyone else in the world is doing that. So bringing that, we are also doing a lot of simplicity. So we're helping MFTs to easily train them, or as an easily deployed model. So we're investing heavily, we're invested heavily, still invested in providing tools

• 18:35Speaker 00

  and ways to abstract the complexity with all with just running workers, just training, training models, deploying them. Okay, advantages. Without the right infrastructure solution, without the right platform, they they would spend like ninety percent of their time on infrastructure, on on setting up libraries, setting up drives, setting up a lot of stuff. And for them to scale or scale out the experiment is really difficult. They are running just one experiment and they want to run it instead of one one GPU on a lot of GPU. Tens of GPU starts, particularly really difficult. Or or if they want to run a lot of experiments in parallel, so scaling out, out those experiments, that's also really difficult.

• 19:19Speaker 00

  So we bring a lot of tools to allow researchers and air practitioners to iterate that. Run, scale up, scale up the experiments. So for us, I think it's about simplicity and about bringing a lot of efficiency. And then the efficiency, I think no one else provides the value that we we we bring. We have customers that are that had four X, need a modern back to achieve their GPU generation, increase GPU availability. And so together with the simplicity, I think we're bringing a unique also to enterprises under work.

• 19:56Nathan Labenz?

  Yeah, okay, cool. That's very helpful and it is certainly in the context of the shortage that and I want to kind of, you know, turn in a second toward just kind of the macro context that's driving, you know, all of the the concern and focus on utilization, right? These things are not they're not cheap by default and they're getting bid up at the current moment. So that puts you in a in a position to, you know, be even more valuable. Am I understanding correctly that like this sort of assumes a dedicated capacity? Because when you say like it could be more efficient, that could mean, you know, if you're buying like on demand access, that you buy less.

• 20:36Nathan Labenz?

  But as you talked about it, it sounded more like there's an assumption of certain, you know, either physical infrastructure that a that a client has, or at least some sort of contractual commitment that now they have a certain amount of capacity. Let's make the most of it and not have our data scientists, you know, waiting around when they could be getting to work.

• 20:57Speaker 00

  So that's another good question. So for what we see, that big enterprises, big organizations, they usually secure access to GPUs.

• 21:09Speaker 00

  So just getting access to GPUs to a lot of GPUs, that could be really difficult. If you go to your AWS account manager and you ask them to increase the limit of your GPUs, GPU instances. So you can try and do that. And if you want to increase your limits, you know, going on L GPUs to one other GPUs for that, that's a big difference. You know, you'd get a lot of questions while trying to do that. So, so I think just securing access to GPUs and having them available for you, that's a, that's a big problem. So it makes enterprises right now and buy and reserve those GPUs.

• 21:52Speaker 00

  So we see a lot of big enterprises using reserve and instances of GPUs, huge clusters of GPUs. That's for enterprises, that's what we see. Sure, reserve is something that is being done, but it comes to GPUs. Now, when it comes to smaller companies and for them, on-demand access to GPUs is totally red. Right, yeah, many startups won't want to do a big investment upfront and buy GPUs. So many of them are using on-demand, right? Or maybe it's a combination of reserve and on-demand. But then in that case, we have also a lot of startup companies work with us and our customers. And that, I tell you, what we get is clusters that are elastic.

• 22:34Speaker 00

  So those clusters, you call or sheen, maybe they have like multiple clusters in different regions and just deciding on where to run the workflow, the meat on, on, on which region and how to get five, whether the GPUs in the right regions run the workflow, that can be a challenge. So we're helping also customers just get access to clusters that are, it might be added in different regions and they might be a dynamic cluster, but you can spin up or down. So for sure, it's about reserve and about on-demand instances as well.

• 23:10Nathan Labenz?

  Yeah, so let's talk a little bit more about access and kind of the nature of the market. And I don't have a lot of experience in this for context, and I imagine most of our listeners don't, right? If you're, unless you're, you know, one of a, a small pocket of people at a company that's going to make this kind of capital investment or commitment, you know, you're probably like me and you are mostly paying the hourly rate for GPUs, you know, as you go. And that's worked fine for me. I haven't done anything at huge, huge scale. But obviously, the biggest companies in the world, you know, are kind of waking up to the fact that there is this shortage. Maybe for starters, can you just tell me, like, how do you understand the shortage?

• 23:50Nathan Labenz?

  How bad is the shortage? Like, are we are we now in a period of like eternal GPU shortage, like until further notice? And what do what do people actually go through to buy? Like, in Googling this, it's not super or using even perplexity to search for information about this. It's not super obvious. Like, you can go get one here on eBay and then there's like, you know, an H one hundred on Amazon in a random spot. But it's not really obvious, like, how the the big buyers interact with Nvidia in the first place to even get, you know, bulk orders. So maybe you could demystify some of that for us a little bit.

• 24:27Speaker 00

  First of all, I think the GPU shortage is relevant to whoever buys GPUs, you know, for their own premises environment, right? Or for, you know, for companies, people trying to get access to the high end, the most advanced, the newest GPUs in the cloud. So also getting access to those newest GPUs, that can be a challenge if you need a lot of GPUs. I think what we have seen in the last six months is that the demand for GPUs increased amazingly fast. It grew in a way that Nvidia didn't anticipate.

• 25:07Speaker 00

  OpenAI came out with ChatGPT and they showed the data world what can be done with the other engineering activity there, right? And a lot of companies, including Microsoft and all the cloud providers, and including their own mask with his new company, he went and bought ten thousand GPUs just a few months after ChatGPT went out. So a lot of companies and all the cloud providers went to buy more GPUs from Nvidia. So that created a big jump in the demand for GPUs. With hardware compared to software, it's much more difficult and much more complex

• 25:50Speaker 00

  to supply the demand. We have unexpected changes in the demand. So that's what happened. And for Nvidia, they need to change the the way and the pace of which they manufacture those GPUs. So we're speaking about real machines that are actually manufacturing those GPUs. To manufacture more GPUs, you need more machines. And that takes time to build, to to set up those machines and let them operate. So supplying that demand and bridging on that gap between the demand and the supply when it comes to hardware, that takes some time. And I'm sure Nvidia are working on it.

• 26:33Speaker 00

  And I'm sure they are increasing the pace in which we'll manufacture their GPUs. Um, I think a GPU shortage would go away with good time. Right? I don't know how much time it would take your shortage to go away. But a lot of companies are still now waiting for their GPUs, for the GPUs that they order for. And each one of them, so each one of them is the newest GPU, right? People can buy. And a lot of olders are waiting in line. So that's the GPU shortage. I think it's really interesting because what we see, we spoke about the train of using more and more compute to train AI models.

• 27:17Speaker 00

  Yeah, I think for the first time, we're seeing also a huge increase in the demand for GPU power for inference, for running those models in production. So L N M's, generative AI, are so huge that usually they don't fit into the memory of one GPU. So if you want to run state of the art models with hundreds of billions of parameters, typically you're on a few GPUs, about like four GPUs, eight GPUs, maybe more than that. But I think now we're seeing a panic, increase in the computer requirements for inference work.

• 27:59Speaker 00

  And that's significant, that's significant for every company that is actually providing running models like OpenAI and like GitHub with their copilot, where it's not sure that other scale. So Microsoft between OpenAI and GitHub with their copilot, they need to manage a lot of GPUs. So for reference, for sure, but contributed to the demand for GPUs was very significant. And I think what we've seen is that the pace in which AI application and AI innovation is happening is much faster than the pace in which GPU or the hardware

• 28:43Speaker 00

  is being more pretty progress. They like to do that to more slow in that really to like to other other issues as well. But but AI, the AI space, AI innovation moves at the next, like, very exceptionally fast pace, much faster than the capability of your hardware. So with that, when that happens, you're getting more and more demand for the demand for hardware. And then you're getting these problems of demand goes too fast, the supply doesn't able to to catch up. And so I think that it's I my prediction is that this is

• 29:26Speaker 00

  not the last time that we are seeing GPU shortage. I don't know when will happen, the the next GPU shortage, if or when, I think it will happen again. But now it's like a temporary thing, it would be closed. But I think that it would happen, GPU shortage. It has major consequences. Yeah, I think it has major consequences on the industry, on how companies are operating on the cost of AI. So a lot of interesting aspects to relate to that.

• 29:56Nathan Labenz?

  So how does it actually work today if you want to go by start your own little cluster, right? If you are, let's say you're in, I'm not sure what, you know, a relevant threshold would be. Obviously, as you said, you know, leading clusters now getting into the tens of thousands inflection, you know, AI just made this headline with a huge raise, I believe with Nvidia as a backer. And, you know, a lot of that's going to get plowed right back into the chips. So, I mean, at the highest level, there's clearly some like Strategic deal making going on where, you know, Nvidia is kind of investing in the, you know, the people that it's going to be then willing to ship the most chips to.

• 30:38Nathan Labenz?

  But how does it go if I, if I just want like a hundred H100s or a thousand? Do I contract directly with Nvidia? Is there like a secondary market that tends to kind of, you know, are there like futures contracts? That, that people sort of trade in and then ultimately, because pricing seems fairly dynamic, right? Like I'm seeing headlines that are like H100s now cost $40,000. Presumably that's not, you know, Nvidia changing the, the, you know, I couldn't even find a price on their website. So like, yeah, how does that actually work in, in practice to, to go through this buying process?

• 31:13Speaker 00

  There are a few decisions that one needs to make. And one of the first decisions is around where to invest, like is to invest in that on previous environment. You just buy GPUs and then operate those GPUs by yourself. Or whether to go with a cloud solution. And what I'm saying, buying GPUs and operating an on-premise environment, that is certainly on-premise. But it can be also a co-location or someone is managing your, your GPUs in some co-location. But still it's, it's not managed by a cloud provider. It's managed by a co-location provider. So co-location or previous, that's one thing. Cloud is another.

• 31:56Speaker 00

  Deciding there, there are some trade up costs and there are these few, you know, interesting, interesting trade up there. But that's the first decision. So if you go and you, you choose going for cloud, then you need to choose the, the cloud provider. You can actually, you could choose multiple cloud providers. It's not necessarily just be AWS, Google or Azure. Right now we have amazing security cloud providers, smaller cloud providers like Lambda, Abs, that COVID. We're working with both of them and both of them have amazing.

• 32:37Speaker 00

  The GPUs and then call access to GPU. So you can go with that as well. So that's one thing on premises. That's totally different story than you need to to decide on from who to why the GPU is can be for video or some table. Be from instead of another story. And then it's about choosing the right GPU type, right? Whether you want to go to the highest and you know the newest GPUs cage one hundred. Or you're saying, okay, I'm good enough with the with secure access to the one hundred or one hundred. That would be good enough for me and the cost will be reasonable. So I think there are some trade offs around that.

• 33:19Speaker 00

  There are some trade offs of not just relating to GPU, there are some trade offs relating to storage. Where are you storing your data? How how performance is it? And questions around networking. If you're running a small scale experiments or maybe networking is not that important for you. Second, three or or even anything, those are interconnects are connecting with GPU. So maybe those are not that important for you. But if you do train with models on multiple GPU or multiple machines. Then networking can be actually crucial for your performance. Then you need to get some decisions there as well.

• 33:59Speaker 00

  So so it's it's it's those decisions combined cost, performance, needs of users. So it's really, yeah, so we're seeing customers doing just building huge on premises and still securing access to GPUs in the cloud. And we see customers doing just just now secure access to GPUs in different clouds. But the world right now, I think everyone is seeing me what the cloud, I would tell. There is is it is the optional. It's actually the. And cause you don't want to just look at yourself to one provider. So we're seeing enterprises choosing

• 34:42Speaker 00

  more than two providers on premise.

• 34:46Nathan Labenz?

  Prices today. An A100, what I saw on Amazon, it was like $7,000. And H100, I'm seeing $40,000 in headlines, but I'm feeling like that is probably more a reseller price, not like what people are mostly paying. Could you give us general guidance on like what these things actually cost in the market today?

• 35:10Speaker 00

  If it's cloud prices, then they are automated, right? You know, cloud providers are publishing the prices. So it's A100 can cost, one machine can cost between $10 to $40 an hour. That's really expensive. H100 will be more expensive. They are still not available on AWS. I think they are available on Azure, for example. H100 are available on smaller cloud providers, like how we already offered H100. So you need to pay a lot for the newest GPUs. They will be more performant. So it's important to understand that it's not just about the absolute cost of the GPUs.

• 35:55Speaker 00

  It's also about the performance. Because if you GPU costs two times more, but your workloads are only four times faster on those GPUs. So yeah, it's actually the best solution for you, right? Both cheaper and faster. So H100 is according to Nvidia benchmarks, much better, much lower performance compared to A100 and previous GPUs. Usually GPUs become more and more performance, right? The actual numbers around the performance increase are really dependent on the workload themselves.

• 36:38Speaker 00

  But they also come up with higher costs. Now, when it comes to on-premise, if you go to Nvidia. GPU state of the art DJX machines and you can pay hundreds of thousands of dollars for one machine. It would be the very high end performance GPU machine, but you pay it hundreds of thousands of dollars.

• 37:04Nathan Labenz?

  One quick follow up on the on demand pricing because when I was researching for this, I found Lambda Labs seemed to have the lowest A100 price that I was able to find online, which was just over a dollar. I think it's a dollar 10 an hour and they quote AWS prices at being $4 an hour. Were you maybe citing something different there or is that accurate in your understanding?

• 37:29Speaker 00

  Okay, so I said between 30 to 40 dollars an hour for a whole GPU machine with eight A100 GPUs, eight packs. So that's AWS, right? Yeah. So that's one of the advantages of Lambda Labs. They are significant, significant D dollar prices. That's one of the advantages for sure.

• 37:53Nathan Labenz?

  How do you see it? Well, how is that sustainable? I mean, I understand there being a big delta between making an upfront investment in an on premise, physical hardware, even if you're co locating it, whatever, versus, you know, the flexibility of the cloud. But it's surprising to me that there would be a 4X difference, especially in the presence of companies like yours that are multi cloud and help you kind of, you know, optimize and shift around. How do you understand that 4X difference in on demand pricing?

• 38:25Speaker 00

  That's an easy question, you know, I don't know. I really don't know. For H, I agree. It's a, it's a lot. It's a big difference. We can just get a spot. But I think it's a, it's a, it's an interesting question to ask Lambda Labs.

• 38:42Nathan Labenz?

  Yeah, their H100 pricing is

• 38:46Nathan Labenz?

  Two dollars an hour, basically. So they're selling the H one hundred, renting the H one hundred at still half, though, of the AWS A one hundred price, which is definitely pretty confusing. Okay, well, yeah, I'll continue to try to figure that one out. So let's maybe move to, and this, I don't know if this will be kind of in scope or out of scope for your business. Because if I understand the technology, as you've described it, you're at the CUDA layer, which is the Nvidia proprietary software layer, which I would say many people, from what I understand, and I haven't explored other alternatives, but from what I understand, that is often said to be one of the huge value drivers, is that the software actually works.

• 39:31Nathan Labenz?

  And then you're kind of injecting another layer into that kind of strategic advantage that Nvidia has, and even adding more functionality. But I was kind of also curious, like, what other chips are relevant today? Or what kind of other chip efforts are relevant today? Obviously Nvidia has, you know, got huge demand and the stock price reflects that. And people are waiting for their things. Like, do you guys see yourselves trying to partner with other chipmakers? And, you know, whether you do or don't, I'd love to get your take on kind of what other chipmakers are going to be relevant over the next few years as AI just continues to presumably scale and scale and scale.

• 40:14Speaker 00

  As you said, we're sitting in different layers of the software stack. So we're sitting in the CUDA layer, we're sitting at the Kubernetes layer, sitting on top of Kubernetes there. And all the tools that are running. So we're sitting in different layers. And we're working very closely with our hardware players. And we're working very closely with Kubernetes for virus, you know, like, like, create a cloud for virus. And I think that market for AI chips is really interesting. There are few players, as I see. So Nvidia, of course, people say that Nvidia has eighty seven percent of the market.

• 41:00Speaker 00

  That's a, that's a big number. The players in the market, as I see, first of all, there are cloud providers. So AWS and Google, they have their own AI chips. And lots of them are building, working on those chips in Israel. So Israel engineers are amazing. They are really good with, with hardware and they are really good with software as well. And so I'm using and Google are building AI chips here in Israel. I think to us, man, the AI is also best in Israel heavily, and they bought Meta Knox.

• 41:44Speaker 00

  Or they're talking, not as a mutual account. So.

• 41:48Nathan Labenz?

  That's all fabulous design, right? There's no actual, are there fabs going in as, as well? Or I'm understanding this to be the design layer.

• 41:57Speaker 00

  No, that's fabulous, that's fabulous. That's, that's really interesting. Now, Intel is, now, Intel becomes like so many, Intel now. Now is, uh, Intel always had their own fabs for their own usage. And now they're going to the region where they offer their fabs to others. But that's a night, that's a, another interesting topic. Then, but in terms of the AI chips, all of those players are still, they are fabulous. AWS, they have already, uh, chips for trading, chips for reference, offering that cloud. Google is already for several years, I think more than five years already. Their GPUs is also doing that cloud. There are players like, uh, like AMD and Intel.

• 42:39Speaker 00

  So ideally is a strong competitor because their GPUs are used in a significant way in the gaming industry. So they, they are strong there and their GPUs are really good, the technology is really good. So they are a strong competitor. We're starting to see more and more of the usage. We have customers running workloads on AI. And Peter is also investing, Peter also investing in Israel with those chips. And there are startups, like there are startups, a few of them, that are also, also their own chips. Online, and that's interesting. And as you said, I think Nvidia now is controlling the market, it controls the market.

• 43:24Speaker 00

  Because of their software stack, a lot of because of the software ecosystem, so it's starting with a cool development. But it goes much beyond that, so we here is investing a lot, therefore it's just getting out a lot of software. Software library, software framework, software tools. And in the ecosystem, just doing even AI to enable more and more workloads to run on GPUs. But as you said, it's really easy to run workloads today on GPUs, on a GPU, sometimes it's more which controls on other chips. But I think with it will become more and more easier, easier, but. So that that's the ecosystem, and I think also Nvidia really, Nvidia is a great company.

• 44:05Speaker 00

  The GPUs and the technology that they are really, really advanced, and they are offering what the market will need. In like two years, three years, so they are already offering right now. So they came up recently with a great software offering, so that's a new offering. And they can really interesting technology there, and they are offering big GPUs with a lot of memory. Now we're already seeing what the market needs, and I think right now the market needs. Needs to run huge models with a lot of memory, and memory becomes at the bottom. And they have this offering already out. And that we go to, that we support those big models, I think Nvidia is the big player.

• 44:49Speaker 00

  And as time will go, we'll see a second player, which will be really interesting to see who will be that second player.

• 44:56Nathan Labenz?

  So it sounds like AMD, you probably put in the second position.

• 45:02Nathan Labenz?

  Right now, in terms of at least being a, you know, a proper rival to Nvidia. We've got the Amazon, the Google TPUs. Microsoft has recently announced that they're designing their own chip too. I don't know if I said Meta yet, but they've had their own chip and have one of the biggest clusters in the world as well. What about other companies like Samsung didn't come up there? Tesla, do you see those two as potentially big timers?

• 45:31Speaker 00

  Maybe, maybe. And Tesla, you forgot to mention Quaum as well. So Quaum is also a player that you don't underestimate. And then let's see like Quaum and Samsung, they have their own offering. They might have a niche to offer when it comes to inference and running inference and running models at the edge. That's a market that will continue to grow. And there are opportunities for sure for players that are traditionally good in those markets like Quaum and Samsung. So the edge and devices, but you know, the AI market is really different than than previous markets. First of all, because software is really important.

• 46:16Speaker 00

  It's not just hardware. And this is why hardware and software must have the software at a start on top of it. Second, it's security can make that. And and also the AI space is moving so fast. So you know that other manufacturers need to move fast as well. They need to, they need to provide more and more innovation in the faster pace. So so in this AI space, I think it's really different than compared to previous markets that we saw.

• 46:49Nathan Labenz?

  Are there any other kind of smaller or specialist companies that you would suggest to keep an eye on? Like one that comes to mind for me, because we had CEO Andrew Feldman on as a guest with Cerebras Systems. And they've made, as I'm sure you're aware, the biggest ever chip. You know, very different approach, obviously. Do you see them or other kind of smaller, you know, what had been more niche chip companies?

• 47:15Nathan Labenz?

  Being able to grow in a significant way?

• 47:17Speaker 00

  Yes, they have an amazing offering, actually, very unique, right? I haven't seen you just share. I haven't tried it.

• 47:25Speaker 00

  But but it's really unique and it's really differentiated. And you know, let's see, let's see, maybe they might have a little opportunity. I think the AR chip market is going to be huge. And even small players will get, you know, huge, can get huge revenues. So this market is going to be huge with no doubts. And so there are opportunities there with no doubts.

• 47:50Nathan Labenz?

  How about just the rest of the world? Like, what do you think is kind of the state of or the outlook maybe for Chinese companies? Any European companies? I mean, there's, you know, they kind of want to get your take at a higher level as well on just the geopolitics of all of this. You know, everybody's kind of waking up to the fact that this is some somewhat of a strategic resource, right? I mean, it's maybe not quite oil, but it's increasingly, I think, I think, kind of thought of as the next oil. Do you think countries, you know, like China or, you know, trading blocks like Europe can develop their own champions that can enter this top tier over the next few years?

• 48:34Nathan Labenz?

  Or is that just they're just too far behind and it's just still impossible?

• 48:38Speaker 00

  Listen, this is a great question. And and a couple of years ago, people spoke the opposite, right? People spoke about all the innovation coming out from China, right? People spoke about all the research and all the academic papers coming out from China. And people said, you know, what's going on with the US? You know, US is behind. That was the story like two, three years ago, right? The US is behind China and see what happened. I think what happened in the last six months or one year. That was amazing. It really showed the strength of the the silicon valley, I mean, all the all the, you know, fish. And the technology coming from California eh and with the opening AI coming out with ChatGPT.

• 49:22Speaker 00

  So I think the US really showed the strength in the last six to to one year. So we we show, you know, gee, it's moved so fast. I think China now is the arm. The geopolitical issues, right, that's that's the that's a really interesting eh topic of different angles, right? On where the the chief angle without with this chip war happening, right? Eh, it's really interesting chips are strategic, not just to come to cultures. And we're seeing this chip war retire, really interesting stuff eh around the AI. I think there is a ace right now, there is a race in the Western world, but there is a race retire as well.

• 50:07Speaker 00

  And it's the the doesn't think that it's just starting. If things are moving fast, so so rapidly, right? You could think, like, for example, on the OpenAI with their ChatGPT. A few months ago, I heard that, you know, everyone spoke about how good ChatGPT is compared to others. Right? That it's much, much better than anything else in the market. And I think now I'm I'm starting to to hear that, you know, I know that more than providers provide. Very good models make something is even better than than than ChatGPT. So I think things are moving so fast, technology gaps are being closed very fast.

• 50:52Speaker 00

  Right now, my my analysis is that China is behind the US, but but, you know. I don't know how fast that gap would be closed, or you think it will be closed. Like, I guess it will be close, but but things are moving so fast. So it is really difficult to predict, I think, right now in the AS, is what will happen in the next six months, one year.

• 51:12Nathan Labenz?

  Yeah, no, no, unfortunately, I totally agree. It's uh, like I always say, my crystal ball gets real foggy after about six months.

• 51:20Speaker 00

  Yeah.

• 51:20Nathan Labenz?

  I'm with you on that for sure. Obviously, I have a US-based perspective. I guess I'd love to kind of hear, do you think, and you spend time here and also in Israel. I'm interested in like, is there a difference in worldview around AI dynamics? You know, here, I think we honestly have a sort of, in my view, counterproductive framing of AI as this, you know, new front in the contest or rivalry between the US and China. And I'm not really comfortable with the idea that we are creating these sort of seemingly,

• 52:00Nathan Labenz?

  you know, pretty severe escalations along the lines of trying to cut China off from leading-edge chips. To me, that seems like it really increases the risk of other kinds of conflict because, you know, for example, if they can't buy any of the chips from Taiwan, you know, then maybe that makes the cost of disrupting the, you know, production in Taiwan much easier for them to bear. Maybe I'm simpleton on that, but how would you describe the kind of, you know, watching from maybe not a neutral country, but certainly a country that's, you know, far from both of the, you know, the two would-be hegemonic rivals? How does the rest of the world see this US-China dynamic?

• 52:45Speaker 00

  I think from my perspective, the geopolitical issues are really escalating in the last, you know, 10 years. I think the biggest challenges, it's China and the US. Now, Israel is a very small country, and we have a lot of political issues and a lot of challenges with our neighbors as well, right? So we're also really always in a need to protect ourselves against our other countries that are threatening us, you might say.

• 53:24Speaker 00

  So I think for us, Israel, we used to be in this state of mind that, you know, that there is danger, that there are countries that are eating good stuff, right?

• 53:41Speaker 00

  And so, yeah, it's a big question. It's a huge question. I think like I don't like to see things escalating and geopolitically wise. But it seems to have gone in that direction, right? But I'm optimistic in my nature, so I always hope the things will become better.

• 53:59Nathan Labenz?

  It's obviously impossible to characterize Israeli opinion briefly. I know there's a lot of disagreement and contention on everything, but you could either, you know, speak for yourself or try to characterize it however you would. But do you see, do Israelis see, you know, the view from this kind of middle part of the world? Does China look like an adversary from there? You know, I mean, obviously, there's a lot of kind of local dynamics with neighboring countries, which are much more historically rooted. But like, does China look like an adversary from Israel?

• 54:31Speaker 00

  From our perspective, Iran is the adversary. Iran is the number one adversary for Israel. And from our perspective, Iran is playing with China. They're on the same side, from our perspective. I think that's one of the strengths of Israel and partnership with Israel, the US. From our perspective, Iran is the number one adversary. So, you know, and from that perspective, I think the Israeli people are really seeing eye to eye with the American people. They don't like to see things escalating, but they also know that they see the danger.

• 55:13Speaker 00

  And so, I think for several decades, the collaboration between Israel and Iran is around those topics. But as I said, I'm optimistic, I always hoping for the best. AI, for sure, you know, going back to AI, AI for sure had an impact on that. Because now it's a race, it's a race, countries are being armed with AI technology. And AI technology is going to change the defense space. But no doubt, it's going to change the defense space. So for AI, chips are really important, that's critical. So chips are strategic, AI is strategic for countries.

• 55:56Speaker 00

  And you will see it, I think, the political state changes in the next decade, it's a lot of because of AI.

• 56:04Nathan Labenz?

  Do you see that this sort of supply control is likely to be viable for kind of controlling who can do what with AI? I mean, that could be in the context of China, right? We could say there's these export controls and they can only buy H eight hundreds instead of H one hundreds. And then I wonder like, is that really going to work, you know, or is there going to be sort of a, some way to get around that, either by like, you know, post manufacturing even, or, you know, maybe just very clever software work that could circumvent some of these imposed restrictions. And then I also think about that in the context of the, just AI safety, you know, dialogue and or discourse in general, people

• 56:49Nathan Labenz?

  are very kind of hopeful. And I'm not sure how realistic these hopes are that, you know, maybe we could have a sort of know your customer regime for GPUs and you can only buy so many before you got to have a license or a permit to do whatever. Do you feel like this is a sort of controllable enough resource that it's ultimately going to be fruitful to try to control the development of AI through controls on hardware?

• 57:18Speaker 00

  That's a great point, right? People are thinking about regulating AI. And so coming from the computer angle, that can be an interesting angle, just controlling access to your compute. So that's what we are right now with the, with the cheap war between the US and China, right? They are just controlling the access to the newest GPUs. So China is not allowed to get access to the newest GPUs according to the US regulation, right? So people are speaking about AI safety and about regulating the progress of AI, regulating

• 57:59Speaker 00

  that how AI is being used. And so can that be controlled with, with compute? I don't know. I don't know. Like, cause, cause, cause right now to, to train the issue go. I'll become a little bit technical, but if you train huge models and you try to be like, you know, to get to state of the art models, then you need a lot of, a lot of compute. Right now, you have these open source models, you can train them on your data. You don't need ten thousand GPUs to do that, right? Just yesterday, Meta, which I allow Twitter is to right now, they open source Llama two, right?

• 58:42Speaker 00

  Open source Llama two, it's free for, for, for research and commercial use, if they say. That's really, really exciting. And now if you have those very capable models, open source, and people can take that on that. You find those models you're using that, that they don't need a lot of compute. So then, right, so then controlling the access to compute becomes less, less the other amount, I guess. So it's interesting aspect, you know, regulating AI, I don't think it's an easy question. It's a little complex.

• 59:17Nathan Labenz?

  That's probably a good note for us to leave it on. I know that, you know, regardless of how all this develops and there is tremendous uncertainty. You and the team at Run AI will be helping people get the absolute most out of their investment in GPUs. So Dr. Ronan Dar, thank you for being part of the cognitive revolution.

• 59:38Speaker 00

  Thank you, Nate, and that was great fun. Thanks.