Ep 96: Nebius CTO on Racing the Hyperscalers Before AWS Wakes Up

Nebius CTO Danila Shtan joins Jacob Effron to talk about what it takes to run an AI cloud at the frontier, and why he thinks the hyperscalers haven’t won this market yet. Today Nebius is a Nasdaq-listed AI cloud with infrastructure deals with Microsoft and Meta. Danila and Jacob discuss why open-model inference is still far from optimized, why 95% of Nebius customers run modified models instead of vanilla ones, and why prefill/decode disaggregation is more talk than practice. They also cover what’s actually constraining the AI buildout, why chip scarcity has kept AWS and Google from winning AI cloud, whether we’re overbuilding infrastructure, and how Nebius runs a large engineering org with no limits on AI token usage.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  If Nvidia could have bought, like, could have produced like 10x the chips, this whole market would have been shaped differently. Because there is not enough capacity for their scale, this is a relatively small business for them. For the hyperscalers, it's not just a matter of like sales. It's also a matter of changing their infrastructure that was built at scale for other kinds of workflows. This is a massive investment on their part that needs to be justified by the size of a potential business. And it's not that there yet.

• 0:34Jacob Effron?

  The emergence of Nebius and the other hyperscaler competitors these past years has been a fascinating part of the AI story. It's something I've really been interested in. And so it was a lot of fun to have Nebius's CTO, Neela, on the podcast because we were able to dig into a bunch of interesting things about the business and how the space has unfolded. We talked about Nebius's really interesting origin story and the original relationship with Yandex and how the company came to be. We talked about what it's like to be inside one of these companies, how Nebius works with Nvidia, what it looks like to get their next generation of chips live. We talked about why the opportunity actually existed and why AWS, GCP, Azure can just capture this entire market. And we also talked about Neela's view on a bunch of new technologies like PISAG and Fringe, among other things. All in all, just awesome to get to talk with someone who's at the epicenter of the AI infrastructure market

• 1:19Jacob Effron?

  about his thoughts on everything about where the space is headed. I think folks will really enjoy it. Without further ado, here he is. I guess I'd love to talk just maybe about the broader AI cloud market today. And I think that there's a few maybe larger and prominent public, I guess, termed neo-clouds. And then it feels like there's a new one getting funded every day. I know you don't like that term. But maybe just talk a little bit about how you think about what differentiates. Like, some of our listeners are probably thinking like, you know, all that really matters is just access to GPUs. And like, wherever you can get them, you go. Maybe use that as a straw man and tell us why that's not the case.

• 1:54Speaker 00

  Well, I'm obviously biased because, like, first of all, I'm a software person. And our product and business was launched on the idea that software is the mode, right? So your ability to serve the customers that consume, I don't know, tens of thousands of GPUs and then one interconnected cluster with the same platform as you serve like a self-service customer that runs one GPU for half an hour was our like value prop from the very beginning. So we believe that this is extremely important, being able to build large scale, but have the possibility almost like sell it in retail, in a sense.

• 2:38Speaker 00

  This is what we think like definitely differentiates us from the market, from everyone on the market.

• 2:48Speaker 00

  Like a small remark, neo-clouds, hate that term like with a passion. Because like, it started as a distinction as in, these are kind of cloud companies that are doing this weird, strange new thing. But then everyone wants it in that market. So like, when we started, there were like, I don't know, five, seven companies that were in this area that were called neo-clouds. Now, I think it's closer to like 300 to 500.

• 3:16Jacob Effron?

  Yeah.

• 3:16Speaker 00

  Depending on what you even are ready to call a neo-cloud for the scale. And we really don't like this stretched umbrella term anymore. We are the AI cloud. We don't care. Like, let other guys be neo-clouds. We are the AI cloud, and this is our mental model. We are not competing with the neo-cloud markets. We kind of believe we are past that. Our next competition point is taking on AWS and Google Cloud. Yes, we are way smaller. Yes, our like revenue is like, what, I think like 1% of AWS nowadays. But we are running fast.

• 4:01Speaker 00

  We serve a very particular market. We have a bet that this is the market that is going to grow, that is going to explode. So like, people are concerned that like these investments are huge. Like, people put a lot into the infrastructure, et cetera, et cetera, et cetera. My view, if I compare what we see in the market with like what was there two years ago, we are barely scratching the surface. I think the overall demand in the next three years will increase 10-fold, 100-fold maybe. So like, we are like at the virtue of another explosive growth. Right now, we see these huge facilities, right?

• 4:45Speaker 00

  But a lot of those are privately owned for internal needs of certain companies, right? So Elon Musk is talking about his Colossus cluster and it's like, it's a man's scale, like really, it's like real solid, large-scale thing. But it's not for the broader market, right? It's for their internal needs, maybe for some partners that they decide to share. Yeah, Anthropic can open AI, amassing the like the capacity, but they do it for their own product. And this is like a wholly separate discussion. But these companies will essentially want to be everything. They will want to be the internet and contain everything and every product and every capacity angle within themselves.

• 5:29Speaker 00

  There is a question of Google, Meta, and Microsofts, who have a lot of capacity but are constantly juggling and struggling between like giving it out to the market for like to rent or giving it out to internal teams. Because all of these companies are extremely like machine learning, AI-enabled. They need a ton of capacity for internal usage, for runtime and for research. So they have to constantly juggle this. So our bet is that there is a niche for independent, as in not affiliated with a large company with huge internal consumption, player that will serve the markets of the meteor players

• 6:14Speaker 00

  who want to be the large companies.

• 6:16Jacob Effron?

  I thought it would just be interesting for our listeners to talk a little bit about the history of Nebius. I feel like it's a really interesting path that you had to end up here. And so maybe just talk a little bit about how the business came to be.

• 6:27Speaker 00

  I'll start with the like with the moment I've joined, right? So I am CTO of the company, but I've been with the company from the very beginning, like since before it was called Nebius. So the new company is established, I think during 2024 also. The deal was finalized. We finally announced our separation from Yandex. No connections to the Russian assets anymore, clean slate, et cetera, et cetera, et cetera.

• 6:57Speaker 00

  And that was it.

• 6:57Jacob Effron?

  You know, people have come to these AI cloud businesses in different ways. And obviously, you know, I thought the Yandex team was like legendarily really, really strong, you know, on the engineering side. You know, a lot of folks building AI clouds came from like crypto or energy or like not necessarily the same, you know, maybe technical heritage. To what extent, I guess, was the Yandex experience really relevant? And to what extent was it like? You know, it turns out, no one knows how to build a GPU cloud because it's all kind of new and, you know, everyone's learning on the fly.

• 7:27Speaker 00

  First, we had and still have the core team that supports the hardware side of business that came from Yandex, that had experience operating hardware, like infrastructure at scale, like at real scale. Hundreds of thousands, millions of servers, span over multiple facilities, et cetera, et cetera, et cetera. These people like were the core of our build outs, like initial build outs and expansion, and they're still with the company. So this definitely helped. I think like we would be way less successful if not for that core team. On the software side, it's interesting because it's a twofold. Like, because like with hardware, it's very straightforward. We needed people with experience.

• 8:12Speaker 00

  We had people with experience. They started building out from the get-go. On the software side, it was a weird one. First of all, we had to rewrite everything from scratch. So our previous experience was kind of like garbled. Second, we had to grow so fast that by, for example, by the end of 2024, when the new platform launched, like a significant portion of our engineering team, software engineering team, was not from Yandex anymore because we had to hire during like this period extensively. As a team, we were working on this cloud platform from like from the initial idea, right? So we kind of just went with it and doubled down on software.

• 8:57Speaker 00

  Like, we are not doing a managed bare metal. We are not doing a patchwork of third-party services. We want to build a holistic cloud experience, the holistic cloud platform.

• 9:08Speaker 00

  And we kind of doubled down on that even with the whole rewriting of the platform, like doing it from scratch, et cetera, et cetera, et cetera. And of course, like having a ton of talented engineers on the team obviously helped. Like, just for the scale of it, right? You've mentioned like people who like came from crypto or came from other avenues of life. They are mostly concentrated on hardware. Like, their idea is like, you need to buy GPUs, you need them to run, and then you'll find the buyer.

• 9:40Speaker 00

  In our case, for example, in 2024, when our hardware team that actually like procured GPUs, did the build outs, we're working on our own servers and chassis, writing firmware and stuff, building the network, that whole team was roughly like 40 to 50 people. The software organization was already 200, right? So we kind of can certainly think of ourselves as a software-first company. We are building the cloud on the solid foundation of hardware.

• 10:13Jacob Effron?

  Yeah.

• 10:14Speaker 00

  We're not a hardware-first company.

• 10:15Jacob Effron?

  Makes total sense. I guess I'd love to take our listeners like inside Nebius. And I thought there's so many interesting parts of the business. But maybe one place to start would just be, I imagine as you're like getting, you know, new Nvidia chips that come in, like there must be a bunch of stuff you have to do within the business to get this all up and running. So like, what does that look like from, you know, I don't know, you're talking to Nvidia folks and they're, you know, giving you early insight into into the next generation of chips to like, you know, actually getting these live for a set of customers?

• 10:43Speaker 00

  That is a mess. No, I mean, like in a good way, you know, like this whole, this whole market, this whole field is running at such a pace that like everyone is like trying to catch up, right? And Nvidia, despite being like in the lead of this, essentially leading and showing the direction for this market, is not the exception. So first Nvidia folks like way before show you the prototypes of what they want to build in the next like year, right? They kind of show you the schematics, like how it will be like optimized internally. Where's your like PCI connections? How would that look like?

• 11:25Speaker 00

  What is the like future platform would look like? Then sometime later, they come to you and said, oh, like we've optimized things. It's now going to be different. And you're like, all right. Then they come to you and like, oh, we have like the first lab samples. Do we want to have a look? Yes, of course we do want to have a look. Oh, by the way, it doesn't look like the stuff you've showed us two months ago. Oh, we've changed stuff because like we found like, because they are so compressed in time, same as they, everyone else, they are constantly iterating while these chips are being produced, right? So they've, it's not your like, you know, there is like an ocean in the consumer market. Like Apple iPhone that you see today was actually planned three years ago and final

• 12:10Speaker 00

  design more or less was settled like a year ago. And they take the final year to polish it to build this product. Like AI market is running at such a pace, like that whatever like they announce is it's indicative and then teams are constantly iterating.

• 12:28Jacob Effron?

  The architectures and the workflows are changing, right? So I'm sure they're iterating as well.

• 12:31Speaker 00

  Yes, like they're iterating, like going further and further and further. And Nvidia is a behemoth now, right? Whatever they do influences the whole market, supply chains, there's everything. So they have to like that. So at some point you get a sample and it's more or less like what you'll get in production. Then you start building, like you're testing everything. You like incorporate support for the new hardware, for the new restrictions or capabilities into the software platform. You figure out a way to isolate workloads from the chat. We are, we are cloud. We are not providing bare metal hosts, right? So we need to split, we need to make sure that our virtualization works, that our storage works, that our networking works.

• 13:16Speaker 00

  Sometimes we are on the bleeding edge of this. So we like, for some of the generations, we were probably the first team, like maybe one of the first teams that actually built like a cluster of a certain size. So we've hit some issues with like with the software that no one has seen before. So we had to report that back to Nvidia, like propose a patch, get there, like fix from it, make sure that it runs, et cetera, et cetera, et cetera. So like, we kind of feel like we are at the forefront of it. But all in all, it's a pretty normalized process. It may sound messy, right?

• 13:57Speaker 00

  There are like multiple iterations, you go back and forth, you're in constant contact with like Nvidia hardware folks, with Nvidia software folks. You're trying to solve the issues on your end at the same time incorporating into a more complex platform, thinking how can you expose that to customers? Because for example, working with like your usual black hole chip, like, you know, the B300s and then transferring that experience to GB300s, which are a whole different architecture in terms of isolation. You're talking racks rather than servers. You need to find a way how to do maintenance on the rack instead of the server, which you just switch off and you move the workloads to another one.

• 14:42Speaker 00

  Despite all of that, this kind of works all the time and every day you like push it a little bit further. And then in the end, yes, there is a product and the customers use it and that's it.

• 14:55Jacob Effron?

  One thing I think is would be so interesting to get your thoughts on is just the differences in, you know, kind of what end customers end up wanting. Like obviously it can be easy to think, okay, everyone just wants to, you know, come to you and get similar things. But I imagine you segment your customers or think about differences in what, you know, is important to different customers. How do you think about like the, the key differences in, in, in these customer desires today?

• 15:18Speaker 02

  Hi, I'm Anne Maris and I'm the producer of unsupervised learning. Thank you so much for listening to the show. And if you've enjoyed the conversation, please subscribe. Subscribing helps us grow, which helps us continue to lock down some amazing guests. Thank you so much for your listenership and see you next episode.

• 15:35Jacob Effron?

  How do you think about like the, the key differences in, in, in these customer desires today?

• 15:42Speaker 00

  It's not that different, to be honest, because like every customer is like its own unique flower, right? But at the same time, workloads in principle are not that different. It's either training and you kind of know how training looks like or inference and you kind of know how inference looks like. Circling back a couple of years ago, for example, we kind of underestimated the storage component. We thought GPUs are the most important thing and they are. But alongside the whole training angle, you see a massive demand for storage. And it's not for the like amount of like AGDs or VMGs that you can provide to the customer. It's about throughput. It's about latency.

• 16:27Speaker 00

  In my prior life, pre-AI workloads and infrastructure build outs, I was talking about gigabytes per second, if we were talking about network attached storage, right? It was like, it's a good like. Right now we have customers routinely kind of like hitting our like a common share, like one volume, right? Connected to a number of GPU servers with like, I don't know, five, six, 700 gigabytes per second of throughput. This is extreme. Like two years ago, I didn't even think about that. And now this is more or less like something we expect.

• 17:08Speaker 00

  At the same time, if you are targeting these numbers for the highest demand, like the, the most demanding customers, your baseline is so high, you kind of can serve anyone.

• 17:20Jacob Effron?

  Yeah.

• 17:20Speaker 00

  Right? So if you hit this like use case where a customer needs 700 gigabytes per second from a, like, I know, three petabytes of shared file storage connected to a thousand of servers or like 8,000 GPUs, you kind of know how to serve a massive cohort. So I'd say in terms of like performance or features per se, the customers are not that different. Then you meet another subset of like what the customers really want. Because there are definitely different customers in terms of not their scale, not their size, not their workloads, but rather how mature they are or what are their obligations like

• 18:07Speaker 00

  for their clients, right? Because we can talk about like a Neo Labs, like Neo Labs routinely have a seed rounds of in the billions nowadays, right? So most of that budget goes into commute. So you can have like a contract for a billion dollars with a newly founded Neo Lab who they don't care that much about anything but GPU, storage, performance, availability. This is it. They like oftentimes they don't even care about the location of those GPUs.

• 18:38Jacob Effron?

  Yeah.

• 18:38Speaker 00

  They don't care if that's US, Europe, like France or Iceland, like.

• 18:45Speaker 00

  Why bother? Like, internet connection is good, storage is there, GPUs are working, and we are talking about a billion-dollar contract, right? At the same time, we now see and like start onboarding enterprise customers. These are companies whose like the contract with us will be smaller, sometimes significantly smaller, but they are more established. They have customers and they have obligations for their customers. They have procedures. They need to make sure they're compliant to a number of regulations in like multiple countries.

• 19:24Jacob Effron?

  Yeah.

• 19:25Speaker 00

  So now we are getting into that whole angle of being the real cloud, you know? Like certifications, compliance, security posture checks. Let us look at your source code. How do you solve problem X? How do you like protect your customers from attack vector Y? Et cetera, et cetera, et cetera. And I'd say if you are talking about differentiated customers, this is the most differentiating thing as opposed to what you would expect from like the size or the performance.

• 19:57Jacob Effron?

  You know, obviously, I think a lot of people are thinking through this this trade-off of of like throughput latent or a trade-off of like latency and and kind of cost. And I imagine you have some customers that have workloads that are super latency sensitive and they'll, you know, do small batches or whatever is required to to get to that latency. And then you have other customers that are much less latency sensitive, maybe background agents or something. Is that all just, you know, determined on the software side? I guess if you have those different customers of like basically how, you know, the batch size that you're running these workloads in? Or do you actually construct the clusters in any different ways based on those?

• 20:30Speaker 00

  No, no, no, no, no. This was the premise from the very beginning. We are building the universal cloud platform and we need to make sure that our baseline is high enough to fit most of the workloads.

• 20:46Jacob Effron?

  Yeah.

• 20:47Speaker 00

  Then there is the SLA play, right? So customers come like, for example, for the managed inference part of the business, customers come with the different demands. And then you are in this kind of a benchmarks game, right? They want this particular footprint based on this particular traffic pattern, fit these SLAs in terms of latency or like latency is time to first token or the whole like request latency. And you start optimizing for that. But this is never done on the hardware level because hardware is good enough. This is rarely done on the lower infrastructure level because it's universal and again, good enough, no compromise for performance, et cetera, et cetera.

• 21:30Speaker 00

  And you're going further away or to the orchestration level, to the model level, the way further up the stack, all the way to how exactly does the request from the customer, like the endpoint, gets to the inference servers and what are your performance hits along the way?

• 21:50Jacob Effron?

  Yeah, that makes sense.

• 21:51Speaker 00

  So this is like 100% software.

• 21:53Jacob Effron?

  I guess a lot of people have talked about, you know, it feels like this this disag, you know, splitting pre-fill and decode is very in the air these days and a lot of people talking about that. Like, is that something you guys think about now or at least how do you how do you think about that kind of in the future?

• 22:05Speaker 00

  I think we are in the stage where everyone's talking about that and no one's doing that properly. So because like it is a complex problem to solve. And particularly in inference, you have a lot of other problems, like way lower hanging fruits that you can fix address and hit those SLAs. So we are quite cautious about that. We don't have it, for example, as a go-to solution for every customer, for every workload because like first it doesn't make sense, second, it's really hard to operate and work with, like provides proper SLAs around that is like proving quite difficult as of today.

• 22:50Speaker 00

  We have some promising results in our like prototype services, but not as of today. At the same time, we have some like disaggregated setups for the most demanding customers in production right now. It's just more, you know, white glove manual approach.

• 23:06Jacob Effron?

  Yeah.

• 23:07Speaker 00

  But especially for this newer architectures of like, like, you know, those gray black worlds, whole node that has really fast interconnect, a lot of challenges that you've had before with disaggregated steps are kind of solved if you're able to fit in the footprint of a rack. Because like this throughput is so much faster, like interconnect is so much faster, you kind of stop caring and that this is a way of just balancing the pre-fill and decode nodes to match the particular pattern that you see from the customer.

• 23:41Jacob Effron?

  What's your mental model for like the workloads that do fit in a rack?

• 23:43Speaker 00

  I don't know. It's a very case by case. Well, these are definitely significant workflows, like we are talking like thousands of requests per second, right? We are talking like billions of tokens that flow through. But at the same time, because we are the cloud, we can, for example, like share the rack between two workloads and not be that bothered about that, right?

• 24:05Jacob Effron?

  Yeah.

• 24:06Speaker 00

  So, for example, on the cloud side, we, for for this particular product, we only allow to consume it like at rack level. You cannot order two gray black worlds nodes. It doesn't make sense.

• 24:21Jacob Effron?

  Yeah.

• 24:22Speaker 00

  Right? Like you have to allocate the whole rack and then like structure workloads within it. But for example, when you're a client of such a solution, as we do with managed inference, for example, you can allocate workloads and you can find a way to do that. So in a sense, you don't need to fit anything in the rack. It's just, is there a workload that will benefit from this disbalance, like intentional disbalance of pre-fill and decode nodes?

• 24:52Jacob Effron?

  Yeah.

• 24:53Speaker 00

  And can you fit more than one into a single rack? And how do you solve the packing problem?

• 24:58Jacob Effron?

  Yeah, no, makes sense. You kind of mentioned there's like lower hanging fruit on the on the inference side, you know, than than doing that maybe. What what are like some of the lower hanging fruits today that you still think we're we're not as as an industry optimized on?

• 25:10Speaker 00

  Well, so I think as an industry, we didn't optimize shit. So like, pardon my French, but in a sense, I strongly believe that running inference, particularly in the like variety of open with open source-ish models that are available today, is not a solved problem. Like if you take something, just run with it as it's published, like initial support in some of the open source inference servers, run with the novel models, your results will probably be underwhelming. It's not that fast, it's not that cheap, especially with like current prices on the infrastructure. It's easier to go and use like, I don't know, Fable 5.1, Astro 6, et cetera, et cetera,

• 25:55Speaker 00

  et cetera. But because of the open architecture, because like of everything that we know as of today, you can optimize the way you process those requests, you can optimize the way the inference server is configured, you put a lot of effort in kernel optimizations, sometimes kernel over optimizations based like on the particular pattern that you see in the traffic, right? There is a lot of underexplored potential in model weights optimization. And I'm not just talking about like stuff like speculative decoding, for example, I'm talking about like model pruning, selective pruning, compression.

• 26:40Speaker 00

  Because I believe that, and I and we see that within our customer base, 95% of our customers do not run vanilla open source models. These are modified, fine-tuned, et cetera, et cetera, et cetera. It means that these are the models that start to address a particular use case that the customer has. Particular use case usually means limited domain, scoped domain. Scoped domain usually means that you need to retain like frontier intelligence and reasoning for a particular subset of tasks and you don't care about the rest.

• 27:21Speaker 00

  And now we are getting somewhere in terms of like, maybe you don't need such a large model anymore. Maybe you need just like a model that is selectively compressed in a way where it works for a particular task and it doesn't work as good for the rest of it. Now, of course, that doesn't work with like open-ended, you know, deep research or broad coding tasks anymore. But I strongly believe that for That for real AI adoption, when we are talking about serious enterprise business, when we are talking about like practically AI that actually solves problems,

• 28:02Speaker 00

  you are getting more and more into this specialized agents, limited domains, where you even stop caring about the costs. You care more about predictability of the outcome and you care about time as in like real seconds and minutes till you get that outcome.

• 28:20Jacob Effron?

  Yeah.

• 28:21Speaker 00

  And I think there is a lot of potential there and we still will still see a couple of nice breakthroughs there.

• 28:31Jacob Effron?

  There's a ton of work around inference optimization and a lot of that gets done in open source, right? Or like, you know, contributions that that share throughout the community. There's obviously a bunch of work on on like smaller purpose-built models and compaction and a bunch of these things across all these problems, which do impact, you know, how enterprises consume inference, which ones like Nebius should have a solution or offer a product around and which ones you're like, hey, it's great that that exists and will benefit from that, but we don't need to be at the forefront.

• 28:59Speaker 00

  We want to be at the forefront of everything.

• 29:03Speaker 00

  In a sense, these are additive and even maybe multiplicative, right? Like, if you have a right subset of things, optimizations that you introduce for a particular use case, it means that you can reach the targets easier, better, faster, in a more predictable fashion. So we invest in all avenues, like one of our acquisitions back in like the beginning of summer was targeting specifically that. Like, we basically bought the company for the talent in model optimization, model compression, speculative decoding, et cetera, et cetera, combined with existing talents. Like, we have like a team that is two X this size.

• 29:46Speaker 00

  And we like, we have serious manpower, like as in tens and tens of people who are working on these problems, like we have specialized groups for like kernel optimization, specialized groups for like model compression problems. Because this is like, I feel if you want to talk about like running inference as a service at a serious level, this is the only way you can do it nowadays.

• 30:11Jacob Effron?

  And I guess like, you know, maybe along those lines, you guys have done, you know, a few acquisitions and one of them that that caught a few people's eyes was, you know, Tavoli in the kind of like web search space. Maybe talk through a little bit like, you know, the rationale for that, how you think about that fitting into the broader platform?

• 30:27Speaker 00

  So first, we were very impressed by the product and their go-to-market motion. It was almost a year ago now, right? But we felt like we knew how to help them run even faster with our expertise and technology and low-level stuff. Because actually, running a web search engine is

• 30:50Speaker 00

  is not a very simple task, not neither from like the software development perspective nor from

• 30:56Jacob Effron?

  It's a lot, a lot of you used to run one.

• 30:59Speaker 00

  And some of us dabbled a little bit in our previous lives, yes. Right? So this was the premise, like, we have a great team that is running real fast in terms of like market penetration and in general go-to-market motion. And we know how to make it complementary with our expertise and the low-level stuff and running large-scale infrastructure and, for example, large-scale web index. It's not exactly like embedded into the rest of the offerings that we have now today. Like, we are exploring and we have something cooking for like for the managed inference combined with like agentic tools, because, for example, we also develop sandboxes nowadays, right?

• 31:42Speaker 00

  So this is another like building blocks for building one of one of the building blocks that people use to build their agents. We have several other products that are in the earlier stages of development in that regard, too.

• 31:55Jacob Effron?

  Yeah, I mean, it seems that there's obviously a set of tools that folks are going to use to build agents and build on open source. And it seems like you want to have a bunch of those tools to be able to offer to to your end customers and web search kind of being a key primitive for any any long-running agent will consume a decent amount of tokens on just pulling information in.

• 32:13Speaker 00

  So, yeah.

• 32:14Jacob Effron?

  I'd love to switch gears to just like the broader, you know, industry build-out and kind of the, you know, I think there's lots of questions folks are asking about that. I figured one place to start that actually might just be interesting for our listeners is to what extent are like the data centers that you work with similar? And, you know, like, there's a reference design and they all kind of like, you know, I don't know, to you, it doesn't really matter which one's which. And like, what are some of the differences in them that actually impact the business of, you know, running an AI cloud on on top of them?

• 32:41Speaker 00

  Well, to be frank with you, I have no idea what the walls of the facility are made of. So they're probably different, but that doesn't really help. But in all seriousness, what's within the walls is something that we fully control. We do not rent data centers when someone like have built the like the network on their own or brought some GPU, GPUs or servers, et cetera. If we don't own the facility and we rent it, for example, we rent, well, the building, electricity, but that's it. All the internal build-out is controlled by us.

• 33:25Speaker 00

  It's like, it's done according to our standards. So from the software side of things, there is literally no difference which data center we run in. Is it our own facility? Is it something that is built within a rented facility? Or, for example, we've recently announced this so-called asset-light model when a third-party brings capital in order to set up the facility and GPUs and we run our platform on top of that. It's going to be the same. These data centers are built according to our like fitted out according to our spec. Hardware is something that is familiar to us because like unknown hardware is quite

• 34:09Speaker 00

  hard to adopt and make sure that you provide proper SOAs.

• 34:15Speaker 00

  And that's it. So we don't compromise there. We never rent something from people and present it as our own cloud. If you launch a virtual machine in Nebius Cloud, you can be sure that it runs on our stack top to bottom.

• 34:29Jacob Effron?

  Yeah, makes makes sense. But I guess there's a lot of dialogue around the bottlenecks to the AI buildup today. And I think people talk about a bunch of different ones, you know, whether it's power or labor to actually make, you know, data centers, cooling, networking, you know, HPM, GPUs, I guess, as a overall, like, how do you think about what the what the key bottlenecks are today? Or if you were to kind of rank the across all of them, the few that feel most most poignant right now?

• 34:55Speaker 00

  I'm sorry to disappoint. There is one single bottleneck and that's your ability to execute. Because, yes, all of those are issues, like there are GPU shortage in the market, components shortage in the market. Nowadays, it's CPUs shortage in the market, like NVMe shortage in the market.

• 35:15Speaker 00

  There are like not enough facilities, not enough electricity, not enough qualified like workforce to help you build it out.

• 35:26Speaker 00

  Not enough like small components, like network transceivers, for example. What else is not in abundance? Oh, you know, like those large-scale, I'm not sure, like, I know the proper English terminology, but like, you know, those machines that actually convert electricity in place, like transformers.

• 35:49Jacob Effron?

  Transformers, yeah, yeah.

• 35:50Speaker 00

  Transformers, yeah, like, do you know the lead times on those these days? Ooh, years. Right? So like, everything is a problem. And I think the single bottleneck and at the same time, the single like mode that you can impose on that is like your ability to execute around all these problems and concerns, your ability to source, your ability to cut corners where appropriate, your ability to execute in general, your ability and flexibility to move stuff between facilities. Okay, we're going to move this one from here to there and then we're going to move it back kind of like, yes, we'll postpone this one, but this means we'll be in time with this one, this is important, et cetera, et cetera, et cetera. So I think that like,

• 36:36Speaker 00

  In this particular field, like an experienced project manager is probably in highest demand I've ever seen throughout my career.

• 36:46Jacob Effron?

  Yeah, that's I guess the bottleneck is really good project managers, which

• 36:50Speaker 00

  Yeah, exactly. Like your overall ability to like have this large context of like problems everywhere and mind and navigate that becomes a moat.

• 37:03Jacob Effron?

  Yeah, it's interesting because you obviously have, you know, the hardware itself changes, changing, you have a bunch of workloads changing, obviously, we have much more long running agents, we have more post-training going on, you know, people running open source models. How does that actually impact the way that, you know, the overall, like, I guess the overall build out of the clusters themselves, to what extent is it like, hey, you know, workload's a workload and, you know, we can, we can run it?

• 37:27Speaker 00

  No, the workload's a workload. Like, exactly like you said, it is slightly different, but in a sense, you need a high enough baseline to not be like constrained by that. And then the rest is just your like optimization problem, the way like you pack things, the way you split things, right? Like, you don't need whole nodes for some of the workloads, so you need to allocate workloads, you need to have the right primitives, you need to expose those through the platform, et cetera, et cetera, et cetera. But in a sense, like differentiated workloads almost never like impose any changes on, on, on how you build on the lowest level.

• 38:12Speaker 00

  Yes, of course, like nowadays, we're like this new architectures, like what, like, like,

• 38:21Speaker 00

  what is it called, Nvidia LPX, like the outcome of the Grok deal, right? This like adds to it. But in a sense, if you have experience running other generations, this is just another like point five generation, right? Yes, this will be a little bit novel and it's connected in a, in a weird, like compared to the previous one's way. And it kind of like, there are some restrictions when you can run and not run on that, but efficiency gains. And this is like just a like a niche local addition to the platform to what, to what

• 39:06Speaker 00

  you offer to the broader market. But in general, no, there is like way less custom, like custom solutions that you would expect from that. The workload is a workload.

• 39:17Jacob Effron?

  Why didn't the hyperscalers like move into this market faster or win this market, you know, more convincingly early on in this first chapter at least?

• 39:26Speaker 00

  First of all, like, sometimes there is an ocean that they didn't even try.

• 39:32Speaker 00

  There, there is a solid, like solid amount of business that hyperscalers get from out of this market, like huge deals, they are very competitive. Sometimes people say, oh, have you seen AWS prices? Well, these are public prices. I assure you, like for proper deals, they are doing proper prices and are very competitive and like, like they're very aware of the market.

• 39:56Speaker 00

  I think in a sense, like there are two, two angles to that. First, like what I've mentioned before, a lot of these companies need to juggle like their internal consumption and what they can put for like for the broad market. And there is like, we are in chip shortage world, right? So like, if Nvidia could have bought, like could have produced like 10x the chips, probably this, this, this whole market would have been shaped differently. Because they like Microsofts and Googles of the world would both buy capacity for themselves and for the broader market to operate. This was part like simply not enough capacity for their scale, right?

• 40:41Speaker 00

  Because there is not enough capacity for their scale, this is a relatively small business for them. AWS revenue is what, like 100 billion or something?

• 40:52Speaker 00

  I mean, like a business for like two, three, four billion, it's not insignificant by margin, but like compared to what they, their conservative large scale established business is, it's not that easy either.

• 41:08Jacob Effron?

  Yeah, but your point is basically like they're, you know, they get a chip allocation that is like, you know, not as large as they'd want, they have to use some of that, you know, a large amount of that for training themselves. There's only so big that that inference business or, you know, that the cloud business can be given all that. And yeah, it's interesting to figure out how to prioritize. Do you go after like the, you know, the few folks with large contracts or do you try and get this like mid-market enterprise, you know, up and coming segment? But I guess you're, you're kind of doomed neither way if you only have a certain number of chips to

• 41:38Speaker 00

  Yes, exactly. And then, and, and then at the same time, this becomes a relatively small like part of your revenue portfolio and you're like, do I even need to allocate resources to that? And that is especially a pressing question because like for the hyperscalers, it's not just a matter of like sales, it's also a matter of changing their infrastructure that was built at scale for other kinds of workloads. The networking, the like throughput performance, the storage performance, by the way, right? That I've mentioned earlier. So this like, this requires, if you want to have it universally like implemented on,

• 42:24Speaker 00

  through the whole stack, this is a massive investment on their part that needs to be justified by the size of a potential business. And it's not that there yet. I think if we start getting out of this like cheap shortage in like, I don't know, two, three, five years, maybe, we will definitely see this shift. And they will pay more attention because this will become like increasingly large chunk of business and they will start investing, they will like reorganize and redesign their like internals to be more accommodating to these demands, et cetera, et cetera, et cetera. And then with their whole sales power, they'll go and try to win this market.

• 43:07Speaker 00

  And the, the question is whether the novel players like, like us, for example, will be fast enough to grow and represent like significant competition when they get there.

• 43:21Jacob Effron?

  Yeah.

• 43:21Speaker 00

  I'm like, I'm 100% that sure that they will pay more attention in the future.

• 43:28Jacob Effron?

  Yeah.

• 43:28Speaker 00

  Our bet is that we'll be fast enough to be in the league with them and fight as equals rather than a small player trying to run in their trails.

• 43:39Jacob Effron?

  Makes sense. I mean, what do you think happens to like, you mentioned there's like these like, you know, 300 to 500, like, you know, Neo clouds, everyone trying to get into this market, like, as you kind of, you know, you mentioned, you're kind of thinking the chip shortage comes to an end at some point, like as you zoom five years, 10 years in the future, like, how many of these businesses are still around or like, what is the end state of, of this market?

• 43:58Speaker 00

  Listen, I, I, I think the whole like AI consumption is going to grow and it will be like universally used everywhere, like AI is everywhere, like right now, like web services are everywhere, AI services will be everywhere.

• 44:19Speaker 00

  So in a sense, like you have AWS, right? But you also have Hetzner, you also have Digital Ocean, you also have Linode, you also have like tons and tons and tons of local like hosting providers that are like giving you virtual machines, sometimes parametal, et cetera, et cetera, et cetera. So this market is going to be vast. Will all of the current like three to five hundred companies survive the next five years? I'm not sure. But I am pretty sure that there will be a place for the very long tail of providers down the line. Like some of this, the current players will consolidate, some of them will die off, new, new players will be launched, et cetera, et cetera, et cetera.

• 45:05Speaker 00

  It's a like, it's not a bad market right now, it's not a bad business to be in. So it will constantly attract capital and attention and investments, et cetera, et cetera, et cetera. And I trust that like in five years' time, there's still going to be like hundreds of players in this market who found the investment to like, and several hundred million dollars and they can offer something to someone in the market. Same as it is with like, with the, the small scale clouds like Digital Ocean these days, right? Like Digital Ocean did not die because AWS is growing.

• 45:42Jacob Effron?

  You're obviously doing a lot of things in-house like sandboxes and web search and, you know, helping people with their, their models over time. And, you know, I think one question a lot of people are asking is, you know, what parts of the AI infrastructure end up just being subsumed into these future AI hyperscalers and like live, you know, in just one combined offering? And then what are like the standalone pieces of AI infrastructure that will persist as companies?

• 46:06Speaker 00

  In a sense, I think it will like, it will kind of mimic what happened with like the legacy generation of clouds, previous generation of clouds, right? When your growth kind of dictates that you need to start addressing new and like new sectors, new markets, et cetera, because otherwise your growth is like bounded and you can't like, can't show your projections, can't justify your evaluation, et cetera, et cetera, et cetera. So there will be a natural push for the larger companies to start like increasing their portfolio, like trying to get into new products, into new services, et cetera, et cetera, et cetera.

• 46:52Speaker 00

  It doesn't mean that there will be a one size fits all solution. Because in a sense, if you're like, if you're concerned by that, you will like, we just need to stop doing whatever we are doing and just wait for, for like OpenAI to rule the world, Entropic to rule the world. Because these companies will eventually like, you do realize that OpenAI has models, runs inference, they have their own search that they, like they don't exactly expose it, but if you're building the agent on their platform, it has access to search.

• 47:29Speaker 00

  They have all kinds of like sandboxing, local agents, cloud agents, et cetera, et cetera, et cetera. They both, OpenAI and Entropic are now investing in forward deployed organizations and are gonna go and do the work that was traditionally like, I don't know, Deloitte's or like Accenture's piece of pie, right? So they will be everywhere. And I think this is a natural motion for the large company to start getting into a broader markets.

• 48:02Speaker 00

  But it doesn't mean they are going to be the only ones. So even nowadays, there is a place for independent object storage players. All of a sudden, right? Or independent compute players or independent small scale inference players for that matter. Independent database players like, like one wants like, it seems like Google has like Big BigQuery and whatever, all kinds of data stuff. But then Snowflake kind of like appears from out of nowhere. AWS has like all kinds of like managed databases, large scale databases like Aurora, Redshift, et cetera, et cetera, et cetera. But then MongoDB appears and it's pretty successful.

• 48:48Speaker 00

  I can't say it's like super large scale and probably by nowadays standards will never be because nowadays standards for a large companies like trillion plus. Right? But like two years ago, it was a very solid and very successful company or someone like ClickHouse appears where they are cloud offering, right? Also a massive managed database, an independent company that finds its niche and works it real well. So I think like larger company, larger clouds will grow. They will have like products in every sector and will be pretty successful at selling them those because of bundling, because of their sales power, et cetera, et cetera, et cetera. But there will always be a place for a niche players.

• 49:31Speaker 00

  And because the market is huge, niche doesn't mean small. Niche doesn't mean small scale or low revenue or low valuation. This can be solid companies.

• 49:43Jacob Effron?

  Yeah. But it's interesting, you mentioned like, you know, OpenAI, Anthropic having this kind of broad suite of offerings too. I mean, in many ways, they're obviously building their own AI clouds. And I thought one thing that was interesting recently was, you know, obviously limitation of those AI clouds versus others is the, you know, having just one model family, right? To be able to call from. But I thought it was really interesting that OpenAI announced that partnership with Base10. I'm curious what your take on was, was on that. I mean, is that them just trying to get into this general, you know, hyperscaler game across models?

• 50:10Speaker 00

  In a sense, we see a lot of push and concerns about people like claiming that OpenAI models or Anthropic models are too expensive. Everyone is puzzled or surprised by the bills they get, get to the end of the month. So this might be like an answer to that. Well, if you're like, oh, go try open, open models.

• 50:32Speaker 00

  At the same time, open models are still behind the closed source, closed models, right? However, good those models are, they're still lagging behind. And giving people an option to try like lower scale with the ability to switch back to the top frontier model might also be a good incentive. Like, anyway, I think like OpenAI is doing a very interesting move with that partnership. And I like, I like it really much because like

• 51:05Speaker 00

  this kind of shows flexibility, right? They're not set and stuff like they're exploring different options, different things that they can or cannot do, like looking for different options to like. But I, I, I think in a sense, this all plays into one single thing like OpenAI wants, same as Anthropic, they want to be one single like touch point, like they, they want to be one single thing for everything. Even that is like, okay, but sometimes I want to use open source models within my codex controlled environment, by the way.

• 51:45Speaker 00

  Right? So like, well, good for them. But if the company, the person, the user is still within your ecosystem, you still have a lot of options to steer them, to upsell them, to introduce them to new services, et cetera, et cetera, et cetera. So if that like alleviates one of the concerns, so be it. Why not?

• 52:07Jacob Effron?

  Well, I always like to end these interviews with some like overly broad questions that I stuff into the, to the very end. And so if you'll, if you'll humor me, I guess the first question is just, what's one thing that you've changed your mind on in AI in the last year?

• 52:19Speaker 00

  In the last year? A lot, a lot has happened in the last year. No, like the obvious answer would be software engineering. Like a lot of people forget, but a year ago, we lived in a totally different world.

• 52:33Jacob Effron?

  Yeah. It's amazing, like dog years in AI world.

• 52:36Speaker 00

  Way less capable, yes, way less capable in terms of the model capabilities and in terms of how good the modern harnesses and tools are. It's like the AI coding, because like a year ago, it was total bullshit. Right now, it's, it's a solid part of like your everyday tooling. But for example, if you're not talking about the industry trends or like talking about more like my personal thing, I've started to rely on more on the like AI enabled like local agents and harnesses to afford a part of my cognitive load, to be honest.

• 53:19Jacob Effron?

  I mean, obviously you run a really large technical organization and I think everyone's trying to figure out, you know, I mean, obviously the models only recently got really good on the coding side. And so everyone's trying to figure out what it means to like build teams in the, you know, given the new model capabilities and run an organization. I'm curious like what you, where you've settled and like, you know, you know, yeah, how you've thought about that?

• 53:39Speaker 00

  Cautiously, conservatively. I'm very concerned that modern models and the attractive user experience that they provide, like modern agents, modern models, et cetera, et cetera, et cetera.

• 53:57Speaker 00

  Is something that will lead us to

• 54:02Speaker 00

  two issues. You like, you see a slop like being revealed in like, in chats, in merge requests all over. And I like, I try to pace that and I try to like be very observant and vigilant about that. So for example, like we don't restrict talking usage, like people can use whatever they want. Our budget is quite significant. We don't impose like per person limitations or per department limitations. But they, at the same point, we try to like, look at the usage, figure out what it is

• 54:44Speaker 00

  used for, look at the most ineffective ways people use it.

• 54:50Speaker 00

  We like constructed a set of policies that doubles down on the idea that whatever you've done with the agent, it's still your result. It's your responsibility. It's your pull request. It's your code. It's you who is on call. It's you who's going to wake up during, in the middle of the night if something goes awry, et cetera, et cetera, et cetera. And if something was created by a completely autonomous agent, it is marked like that. Completely autonomous agents will not touch like the core services for a while because like this is like, this is like out of bounds.

• 55:29Speaker 00

  Like sometimes people are surprised that like I work in a company that's kind of participates in this whole AI revolution and I'm very conservative. But at the same time, like I've seen how this sausage is made and I like, I'll be cautious.

• 55:48Jacob Effron?

  What's the biggest mistake you feel like you've made in, in building Nebius so far? Obviously a lot's gone right, but I'm curious, you know, something that in retrospect you wish you guys had done differently.

• 55:56Speaker 00

  It might sound funny these days, but we underestimated the scale. Like, and it's not about the growth of the company. We were pretty optimistic from the very beginning, but I personally, that was like my strong bias for the first like significant part. I underestimated the scale that a single client operates at nowadays. Because like five years ago, if you're talking about a 30 megawatt data center, it was a serious facility. It hosted like hundreds of thousands of servers, like run

• 56:33Speaker 00

  glorious, gloriously large workloads. By now, 30 megawatts, sometimes it is not enough for a single client.

• 56:41Speaker 00

  Which I'm still getting used to, right? The storage I've mentioned, right? Like four years ago, when you were talking about network attached storage, you were talking about like hundreds of megabytes, maybe gigabytes per second of throughput. Right now, some of our clients routinely fetch, I don't know, 700s of gigabytes per second from a single like shared file system.

• 57:09Speaker 00

  I just like, I'm still like, I underestimated it in the beginning and I'm still getting used to this scale. Like it's like, I'm an infrastructure person. I've worked and have seen these numbers throughout my career multiple times. And all of a sudden you add a couple of zeros to the end and call it a new norm.

• 57:29Jacob Effron?

  Yeah.

• 57:30Speaker 00

  Still puzzling.

• 57:31Jacob Effron?

  It's crazy. And I guess lastly, like, there's obviously a lot of, you know, unsettled questions in the, in the AI field today that, you know, lots of people trying to figure out where, where the world goes. I'm curious, like, you know, what you're paying attention to. And I guess, you know, with a lens toward, you know, if it goes one way or another, it would really change kind of the strategy for, for Nebius as a whole. What are you kind of most paying attention to on, on like these, these unsettled questions of where the AI world's going?

• 57:57Speaker 00

  Well, I stopped caring about like the new models being larger or smarter. I just accepted that it's a given.

• 58:09Speaker 00

  I'm kind of trying to look out for like new novel architectures and approaches, but I don't see any as of today. I hope somewhere deep down in like a stealth lab, something new is cooking. But for now, I frankly don't see anything that like is a thing that I would be even like, would place even a small bet on. But I'm more and more interested in the whole outcome optimization part on the agent side, right? Because like we, we are used to talking about models and size of models and families of models, et cetera, et cetera, et cetera.

• 58:51Speaker 00

  But now that we've shifted to this multi-turn, long-running agents that we are talking not about requests, but rather outcomes, companies are talking about billing for per outcome, right? I'm very interested in how that part will shape up. And we have like a small research group internally within Nebius that's working on this problem, actually. How is like, how do you increase the chances of getting to that outcome? Not by increasing the model or like changing the model. That, that is of course part of the discussion. But I believe there is a lot of optimization on the part where you're, I introduce more sophisticated or maybe more simplified tools. You introduce the harnesses that start to evolve according to a particular task.

• 59:37Speaker 00

  And this kind of gets us back to like significant CPU consumption because like a lot of that, a lot of like modern AI workloads actually go back to becoming code, like it's qualified. It's like very formal. You see like quality gates, you see sophisticated tools, et cetera, et cetera, et cetera. And sometimes your agent works for minutes and minutes, minutes without even issuing a backend request to an LLM. Because the workloads start to like to sit on this part. And this is the part that really interests me because I think like a lot of future AI adoption lies in how reliable your agentic scenarios are and how reliably you reach those outcomes.

• 1:00:22Speaker 00

  And I think like the whole optimization loop outside the model is important part of that.

• 1:00:28Jacob Effron?

  Hundred percent. Well, this has been an awesome conversation. Really appreciate you coming on the show and yeah, thanks so much for, for coming on.

• 1:00:35Speaker 00

  Thank you for having me.

• 1:00:36Jacob Effron?

  I'm Jacob Effron and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project in addition to my day job as an investor at Redpoint. But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so please consider doing that and thank you so much for your support and listening. We'll see you next episode.