Speed will win the AI computing battle with Tuhin Srivastava from Baseten

At a time when users are being asked to wait unthinkable seconds for AI products to generate art and answers, speed is what will win the battle heating up in AI computing. At least according to today’s guest, Tuhin Srivastava, the CEO and co-founder of Baseten which gives customers scalable AI infrastructures starting with interference. In this episode of No Priors, Sarah, Elad, and Tuhin discuss why efficient code solutions are more desirable than no code, the most surprising use cases for Baseten, and why all of their jobs are very defensible from AI. Show Links: Baseten Benchmarking fast Mistral 7B inference Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @tuhinone Show Notes: (0:00) Introduction (1:19) Capabilities of efficient code enabled development (4:11) Difference in training inference workloads (6:12) AI product acceleration (8:48) Leading on inference benchmarks at Baseten (12:08) Optimizations for different types of models (16:11) Internal vs open source models (19:01) timeline for enterprise scale (21:53) Rethinking investment in compute spend (27:50) Defensibility in AI industries (31:30) Hardware and the chip shortage (35:47) Speed is the way to win in this industry (38:26) Wrap

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Sarah Guo?

  Hi listeners, welcome to another episode of No Priors. Today, Alad and I are catching up with Tuhin Srivastava, the CEO and co-founder of Base10, which gives teams fast, scalable AI infrastructure, starting with inference. They're one of the players at the center of the battle heating up around AI computing. Welcome, Tuhin.

• 0:22Speaker 01

  Hi, thanks for having me. Good to see you guys.

• 0:24Sarah Guo?

  Let's start at the beginning. For any listeners who don't know, what is Base10 and how'd you start working on it?

• 0:29Speaker 01

  Base10 is an infrastructure product that we provide fast, scalable AI infrastructure for engineering teams working with large models. Currently, we're focused on inference and we want to do a lot more after that. But for the past, say, four and a half years, actually, oh, that's a long time. For the last four and a half years, we've been cutting our teeth and trying to build this thing. I think it's been pretty rewarding over the last 12 months, seeing the market kind of show up and everyone get equally excited about AI infrastructure. We started this honestly because firstly, we thought ML was pretty cool. In 2019, we thought it was going somewhere and we wanted to build a picks and shovels business and kind of solve the problems that we were running into. I think the side note here is that I wanted to sell a company

• 1:14Speaker 01

  with my friends.

• 1:15Sarah Guo?

  You often say that Base10 isn't no-code, it's efficient code. Why does that difference matter?

• 1:20Speaker 01

  That wasn't always the case. I'd say there was times when we had elements which were definitely a bit no-codey. I think what we've learned over the last three or four years is code is just incredibly powerful and engineers want to write code. Even in its best form, you want to build really, really tight abstractions. But I think the ability to turn the knobs under the hood is very, very important. I think no-code kind of makes that a lot harder. I don't think it removes it, but it makes it a lot harder. So what we do is just build very strong intuitive abstractions that try to make the easy things super easy and that still make the hard things possible. So you can get a lot of value.

• 2:02Speaker 01

  Um. Really quickly, but I say unlike a lot of other infrastructure products that've been built over the last ten years, we're trying to solve against the graduation problem, which is that, you know, that we're able to support teams as they grow and scale.

• 2:15Sarah Guo?

  And just to sort of make it a little bit more visceral for our listeners, like what? What are the types of applications that run on Base Ten? Like, what's the scale of the platform? Do you have a favorite application?

• 2:25Speaker 01

  Everything from, you know, tiny side projects on weekends, all the way to companies that are pretty AI native, you know, we've we've supported foundation model companies. We work with companies like Descript where AI is very, very core to the product experience. We power a lot of AI features that Patreon has shipped. Um. But I'd say some of the more interesting use cases from our perspective, actually, on my perspective at least, are either, you know, the really small teams that we're giving a lot of leverage to, um, so that they can ship things very quickly. So a really good example of that might be a company like Plan AI, um, which is basically building an STK for call centers. Um, is how I describe it, but you know, they're able to ship models and, um, you know, co-locate workloads,

• 3:10Speaker 01

  um, so that they can get, you know, sub three hundred millisecond or sub two hundred millisecond, um, responses, um, without, you know, months and months of infrastructure effort. I think on the other hand, it's like, I'm, it's really exciting to see, um, you know, companies become AI-enabled, um, because that's where we see a lot of the value is going to be over the next decade is, you know, if I look at a company like Picnic Health, which has actually been around for a decade, um, and it's trying to do a very, very interesting thing with this corpus of data that they've gathered over the last ten years and like supporting those use cases. Um, I think their model is called Picnic GPT, which extracts information from medical records. And, you know, to me, that's, those are the really exciting use cases where, you know, you're getting, you're giving leverage to companies that are good at.

• 3:52Speaker 01

  Like the good of the domain that they are working in, their model might be proprietary, the data might be proprietary, but the infrastructure doesn't necessarily need to be proprietary. And we can give them, we can give them just an easy way to deploy that stuff without many, many people months.

• 4:09Sarah Guo?

  It's become like in, in, in vogue to compare the size of your GPU cluster. Like people are spending a lot of money on GPUs. We hear about 600,000 H100 equivalents and lots of venture rounds being raised often to train, you know, large models in some domain or another, or even, you know, more and more expensive post-training. Are, are training and inference workloads different?

• 4:31Speaker 01

  Yeah, I think so. I, I think they just have very different, almost like SLAs for the customer. Like, you know, the things that matter for inference are things like, you know, your, your cluster is somewhat co-located with where you're doing your work. Whereas, whereas training and stuff like that matters a bit less. It doesn't really matter where that training is happening, as long as all your GPUs are somewhat together. You know, even the GPU clusters themselves, like, you know, the, for training, networking is a very, very important piece to have networking on the racks themselves, where inference matters a little less because you're doing a little, a little bit more on individual GPUs and less so across GPUs.

• 5:13Speaker 01

  I think from a user perspective, there's a lot more workflow, I'd say, in inference that's repeated across customers as opposed to, you know, you guys work with a bunch of companies training models, you know, the state of the art really is, give me some SSH keys and let me go at it. Whereas with inference is definitely repeated workflow where people are trying to get similar things out of the inference infrastructure, whether that be, whether that be, you know, version management, whether that be the way they deploy it, hooking it up into CICD, you know, cold starts and so on and so forth. So I'd say it seems a bit more repeatable today, how the problem is being solved by customers. I'd say the hardware requirements are quite different.

• 5:56Speaker 01

  They're probably a little less to, to some degree. But I think, you know, resiliency and reliability matters a lot more. You know, you know, downtime is unacceptable from a perspective, not to get terminated all the time from a training perspective.

• 6:11Speaker 02

  You're quite early to this market, and I think you folks pioneered a lot of the sort of early ML. Sort of infrastructure for these sorts of use cases and applications. What has been the most surprising thing or what did you least expect relative to how things evolved?

• 6:25Speaker 01

  Yeah, I think I can answer that question to two different altitudes. I think you can answer that question from like a market perspective, which is that, you know, I think you have some old writing, which is, you know, markets, basically markets are all that matter. Um, I think we've felt that very viscerally. In some sense, which is that, you know, you can build all this cool stuff and then when markets show up, um, you feel that and that really pushes, you know, the customer forward. Um, and the needs for your product forward, I, I think. So that's one thing, which is like the acceleration we threw, we saw through the end of 2022. Uh, and in 2023, um, you know, definitely took us by surprise. Like I, I can, I can be really honest and say that from 2019 to 2022, it's, it's pretty quiet.

• 7:10Speaker 01

  You know, we had, we had happy customers, but you know, the demands weren't necessarily there. Um, I think from like a practitioner perspective, um, how fast some of these teams move has really shocked me, which is like, I, I think what is really clear in, um, AI and early stage AI and in general, and like, I think the enterprises are waking up to realize this right now. Is that speed is actually your number one advantage. Um, things are moving so fast that, you know, there is, um, if you're not, if you're not competing on speed, um, you're going to be left behind. Um, and so there's actually a lot of propensity to buy versus build. I'd say, or just like people, people are happy to, you know, um, to buy technology.

• 7:55Speaker 01

  Where I'd say in the past, people were pretty hesitant, um, to buy infrastructure. We took, we talked with companies all the time where, where we think that, oh, you know, they probably have something built out. Where, you know, they're a lot less sophisticated than you think and they're handling a lot more scale. So they need to be able to, um, have the infrastructure to support that. So I think that's, that's probably one of the larger things that we've been surprised by, which is, you know, how fast people need to move to be relevant. Um, I think the other thing, which is just like how GPU needs, um, have evolved. Um, going from, you know, at the end of 2022, most of our customers were using T4s and A10Gs. Um, you know, after that, it changed to A100s.

• 8:37Speaker 01

  Now it's gone to H100s, like the compute needs aren't necessarily going down. Um, they're only going up, especially as these services scale up.

• 8:44Sarah Guo?

  You guys were just highlighted for leading on the independent artificial analysis.ai benchmarks for highest throughput and lowest latency serving. Congrats. Can you give us some intuition for what is driving that? Like, you know, what makes inference hard or hard to run fast?

• 9:02Speaker 01

  Inference is, you know, I think there's like multiple things which are quite difficult about inference. You know, I think the there's a workflow headaches, which we talked about a bit and I could talk more about. There's like the scalability and reliability vulnerable. Actually, I think the stuff you're talking about is performance optimization, which is really like, how long does one generation take? I think, you know, there's a lot, lot of work and research that's being done to run generations as fast as possible to get the maximum throughput and a minimal latency. I think historically, historically, oh, I mean, it's funny when I say historically, because I mean over the last six months, but over the last six months, over the last six months, a lot of work has been done in the research community to get basically these things to move faster, stuff like speculative decoding,

• 9:48Speaker 01

  which came out, I think, can't really remember when it came out, sometime in the last six months. It started really being used. For us, you know, what it means is like how much, how much, how well you can use the GPU, how you can scale across multiple GPUs and how you can honestly like be really, really

• 10:10Speaker 01

  up to date with the latest things that are happening in open source and research. You know, we've partnered with a company with, with Nvidia, with a company. We've partnered with Nvidia and, and, and really like worked really closely with their LLM engine called TRT LLM. I mean, that's actually driven a lot of the performance gains that we've worked with and we, you know, we've contributed to that, we've forked that. But, you know, the, the hard thing there, a lot of the optimization you're doing is pretty low level. And there's no rule abstraction. So you either have to learn how to use open source very well, rewrite some of these kernels by yourself. You know, if you look at something like OpenAI, like what do people complain about all the time?

• 10:51Speaker 01

  Speed, um, and speed and like, that's probably, you know, one of the core performance advantages of open source is that you can get these smaller, smaller, um, smaller models to run faster. Um, and I think, you know, that will continue to be a massive focus for us going forward as well. Um, on, on the, um, on the benchmarks, I think, um, you know, it's pretty crazy how that's evolved as well. I think, you know, we've gone from like state of the art being ninety tokens a second, then, you know, it, it got over a hundred, now it's over two hundred, now we're talking to some people up to three hundred, four hundred. Um, and I, I think that's continued, that's going to continue to be, um, like a very, very important, um, place to innovate and we, we think over time.

• 11:35Speaker 01

  We, it will get somewhat commoditized, the performance, that especially, especially for language models. To be honest, I think, you know, more and more of that stuff should run locally, um, to, to some degree, I think, um, and, but, um, being on top of it and making sure that we're, we're kind of attached to the state of the art is, you know, a, i- if, if we're not existential risk to the business and so, you know, we ha- we have to do it.

• 12:00Speaker 02

  How much optimization have you been seeing for other types of models? So diffusion models, some of the language, the text as speech models, you know, other areas like that? I'm just sort of curious, there, there seems like there's different types of optimizations happening across different foundation model types as well. So I was curious, you know, what's, what's state of the art there and how you're thinking about it?

• 12:16Speaker 01

  I don't have the metrics like on hand, but we, we do, we're seeing, we're, we're seeing and, um, you know, also pushing limits there as well. So just yesterday, for example, um, you know, we were able to get Whisper running. So there's Whisper, there's faster Whisper and Whisper on TRT, which again is an Nvidia thing. I, I think what, what we are seeing is that there's more and more focus on bringing these experiences to real time as possible. Um, and so like, you know, one of our customers is a company, company called Gamma, AI powered storytelling, storytelling software. They use table diffusion image models to generate images, um, you know, getting that, not waiting four or five seconds there and still having high quality images.

• 12:56Speaker 01

  Is again core to their business and making it very, very fast and, um, and easy to use. And I think, um, you know, we, we are seeing like. Definitely from a customer requirement perspective, we're seeing that, I think, you know, have we been able to juice as much there? Not yet, but I think we're getting that.

• 13:16Sarah Guo?

  Speaking of the applications, uh, uh, you know, driven by these models still being generally startlingly slow, including the like really amazing capable ones from chat GPT to cognition to, um, you know, things like, like Pika and Mid Journey and, uh, like the, you know, in a, in a way that consumers have not seen in many years, we are waiting, you know, seconds for, um, for interactions. Is your view of that, like, it'll change because the models are getting smaller, it'll change because more, um, smaller models will get more powerful, people do distillation, people just get better at running these things, like, we'll get better hardware, what's the, oh, what's the path to like,

• 13:59Sarah Guo?

  not waiting 10 seconds a generation, two minutes a generation?

• 14:03Speaker 01

  You know, if you look at so many of the gains of running Mistral, let's just take that as an example. You know, the step function gains come from, you know, a few things, like the first one is running on H100 and not running on H100. So I think, you know, its hardware gets better and better, like, you get this almost like leg up. Um, and so as, as, you know, hopefully those prices go down and that gets more available, that'll be one thing. Hopefully the H200 comes up next and we do that. I think the next piece is around, um, software optimization, so stuff like continuous batching, um, and dynamic batching, um, and speculative decoding, which basically makes it, um, easier to either parallelize or batch process a bunch of things, or makes each individual generation faster

• 14:47Speaker 01

  or offsets it, offloads it to another model. There's lots of stuff in that place. I think these models also just kind of get smaller, to be honest. Like, I think that's like the The really, really powerful small model um that does one thing um that's um pretty exciting. Um I think is it um I think that's like stuff like Alama is very interesting. But I I think if you saw um SourceScript has this um where they they basically announced that Cody's now running locally for a lot of customers. I think, you know, that's a pretty actually exciting proposition. We have to figure out where we sit in a world like that. It's not necessarily, you know, amazing for cloud providers like ourselves, because, you know, we want everything to run on the cloud. But I think things do get smaller, things do get more efficient.

• 15:32Speaker 01

  And we have more distilled, more powerful model, sharper models to do. Yeah, it's it's it's the unbundling of models.

• 15:41Sarah Guo?

  How how and when do customers choose to deploy their own models or open source models or fine tuned open source models on their own infra versus use public model endpoints? Or like, what guidance would you give people?

• 15:52Speaker 01

  This is a general trend, right? Which is that you got to open now, you got to have a topic, you have an API that works. Really quickly, you're like, that's too slow, or that's too expensive. Or, you know, I I don't need something that powerful. So we often see like, if if customers have a lot of money, they end up going to pay for private deployments in Azure. They still have the stoniness issue, then they go to open source models. When you're starting out with open source models, you're gonna go to shared endpoint. There's lots of great shared endpoint providers. But there are things that that might matter to you, which um that shared endpoint providers can't give you. Firstly, you know, you want your own SLAs. So you don't want kind of this noisy neighbors problem that when my

• 16:34Speaker 01

  if me and Alad have two competing apps and a lot of apps gets slammed, I don't want my my app to slow down. My model calls still need to be fast. So in that case, you want might want might want dedicated compute.

• 16:47Speaker 01

  But there's also like data and privacy stuff. Maybe you want you don't want, you know, your data running on the same infrastructure, data going through the same infrastructure that other folks infrastructure is going to. And honestly, maybe even at some scale, it's cheaper to run it yourself. Um and they're the three things. I think then when you go to larger companies, for example, no brain and like shared endpoints don't really work. Um. For large companies is not gonna work. They're not definitely not gonna work for enterprises. And a lot of the times you're gonna want a um, you you might want Mistral seven B not coming off a shared endpoint provider. Maybe not even on a dedicated endpoint. You might want it self-hosted within your within your own AWS and GCP. Um. And that's kind of actually where we see a lot of the world going as well, which is that especially larger customers, they're gonna have actually pretty good compute deals.

• 17:32Speaker 01

  Um. And they're gonna have their own spent, um, you know, their own uh, cred system or spend commit with the marketplaces. Um. And actually running it on the infrastructure um, solves a lot of problems and has a massive cost advantage to them um, as well. And so I think there's like three stages to it, which is like you started sharing for an endpoint providers, you go to dedicated in the cloud. But I think for some customers, that's not enough either, and you wanna go um, into your own cloud.

• 17:58Speaker 02

  One prediction on the enterprise side would be that, you know, if ChatGPT only launched fifteen, sixteen months ago and GPT four just came out a year ago. Then most enterprises are still in a planning cycle and they haven't really adopted AI at any real scale. Which means for infrastructure providers like Base Ten, it's like a huge opportunity that's about to come, right? Already you're cresting this giant wave and the wave is about to get ten times bigger potentially. What what do you view as the timeline for really mass scale enterprise adoption of AI? And where do you think things will be in terms of order of magnitude usage overall a year from now, two years from now? I'm just sort of curious like what your view of that future is.

• 18:36Speaker 01

  I think I think it's a good question. I think we've been so wrong with time with time scales here. But I I do think like what we see it right now is when we go to talk to enterprises. Um, I'd say a lot of them, you know, like for hon- honestly, um, what we're seeing now is that like co-pilots, especially um, co-gen stuff. Um, that actually has already made its way into the enterprise. Like most enterprises we talk to, like when you say, when you ask them, how advanced are you on the AI strategy? They'll tell you, well, everyone uses Copilot. That's like their big first big for right. I think the next piece then is like using like OpenAI or or Anthropic um, in some in some way.

• 19:15Speaker 01

  Well, um, I think that is going on right now, and I think people are starting to experiment with that. I, I think like the fear, I would say like I, someone just telling me that I think it's Pfizer earmarked like tens of millions of dollars for ML investment or AI investment over the next twelve to eighteen months. That's kind of frightening to me in some degree. Like, I think it's great. It's great for us. You know, I, I, I love, I love to hear it as a business builder here. But it's also like, you know, to me, that means that the, the pressure is actually coming from the above and, um, and you know, that's kind of like the, I'd say the ML trap we, we fell into in twenty eighteen to twenty twenty, when CIOs were buying software and it wasn't really attached to real user value or product value.

• 19:57Speaker 01

  And so like, I would actually say that we're probably overestimating how big enterprise will get in the next twelve to eighteen months, but we're underestimating where it will be in, you know, three to five years from now. And I, I think ten X is like a pretty massive um, underinvestment like, you know, like, uh, uh, yeah, like I, I'd say, what we see, like, we're working with a customer right now that is foreign engineers and has, by the end of this year, will have, you know, mid hundreds of thousands of dollars annual spend. Um, you know, that's for one use case with sub thousand users um, for them. And that's, and they're already cash flow positive as a business, which is like, which is insane to me, um, in general.

• 20:41Speaker 01

  But like, I can't even imagine um, what these workloads are gonna look like um, when we get to the enterprise. When you start to think about, you know, take a cut like customer service and chatbots is probably like the number one place where people think about um, efficiency, um, the volume that some of these customers, um, you know, rather, my brother's the head of AI at Sunrun, which is like a public company, um, where that does solar panels. And I think that's like another really good example of a company where there's so much opportunity for, so much opportunity for AI just to eat away at processes. Um, and the volume is, you know, just so much higher than um, what we're thinking from like a traditional business perspective. With harder requirements, so which even drives spending.

• 21:24Sarah Guo?

  Yeah, I think one like reset in stance that people should have on on spending here is that traditionally, like, you know, people were looking at software companies, you got really concerned as a software investor if your cost of goods sold was affected by a lot of like data processing, basically. And so, you know, you have this like expectation that your average SaaS business at maturity might have 80% gross margins. And and I think like, you know, now people understand that the the training businesses have a big upfront

• 22:06Sarah Guo?

  capex investment that, you know, may or may not pay off. But I think one of the things that you're pointing out is that you can actually spend a lot on the inference and the core intelligence and that actually, you know, end up with a very valuable business on the other end with perhaps fewer people. And so I think people have talked about that shape of company, but they don't really think of it as a norm yet. And, you know, I at least in my portfolio, we are seeing more efficiency on headcount and like a lot more compute spend. And I know for some of the base 10 customers, that compute spend for inference is actually like the, you know, one of the largest items on the P and L.

• 22:47Speaker 01

  Oh, we were working with this customer. And we basically, yeah, and this was like a challenge for us. So we asked for an upfront like payment for the year of compute. And, you know, the CTO came back to us and said, hey, like, like, I appreciate what you're doing, but no, like, this is, you know, like, after payroll, this is our second biggest expense for the year. Like, we're not going to do that. And I think that's, you know, somewhat indicative of like, how much spend there is here. It's that, you know, and it's it's probably also somewhat indicative of how big, you know, I personally think the market can can be once we start seeing mass scale adoption.

• 23:34Speaker 01

  But I do think, you know, I think that's a good point, Sarah, which is, you know, I think it's somewhat of a reset in terms of, you know, I don't know if this looks like normal SaaS business. Actually, I know, sure, it does not. I mean, I think, you know, even like traditional multiples, it's really, really hard to think about. And, you know, what's crazy about it though is I think the most efficient businesses through markups and through software optimization can actually drive pretty healthy margins and still have these really aggressive consumption contracts. And I think that's, I think that's rare. I can't think of, I feel like you guys see more businesses than I do, I hope. And then, but I hope.

• 24:18Speaker 01

  But, but, but like you guys will be able to like chime in on that more. Is that, is that unique to this industry? Or where else have you seen that?

• 24:27Speaker 02

  It's been a while since I've seen so many companies ramping so quickly, like, and sometimes they were fake ramps. So like, you know, in the internet wave of the nineties, it was kind of startups selling to each other and kind of bootstrapping off of venture capital. And then there was giant telecom build outs on like a five year cycle that caused huge revenue uplift. And then suddenly there was a glut and things dropped dramatically. Here it feels like things are ramping really fast off of products that are a couple of months old, which sometimes suggests that there's not defensibility. And so then the question starts to become, okay, how do you build defensibility? And what does that mean? And how do things get commoditized? And do they, and you know,

• 25:09Speaker 02

  so there's a couple of different markets where suddenly you see three companies all go from zero to five or zero to 10 million of revenue in a year.

• 25:16Speaker 02

  And then you're like, okay, there's three of these companies and they all ramped at the same time, the same amount. And so there's enormous demand, but what does that mean in terms of, do they cannibalize each other? Can three more entrants come in and do the same thing? Like, what, what is the basis for competition in that market? And so I think there's a lot of that happening too, which is at least for me, pretty unexpected. And I think it's just because we have such a big technology capability shift. That suddenly you can do things you literally couldn't do a year ago, you know, it's kind of amazing.

• 25:44Speaker 01

  I think it's particularly exciting when you go and apply that to, I think, you know, you've cut your teeth on a bunch of different healthcare initiatives. Healthcare is like, you know, a really interesting place where like you look at, you know,

• 25:58Speaker 01

  like nuance, if you remember nuance technologies, like, you know, they had the struggle hold over this market for you. And honestly, it always looks like they were kind of struggling all along the way as well. And then, and then Whisper comes along and, you know, you see like that market now of like, you know, note taking for, for the medical thing, it's insane how fast it's going and like, there clearly is real value there. And then I think the question actually goes, maybe this does was like a SaaS business again, when you're like, okay, what's the workflow? And there's the power and the defensibility of the workflow you're powering.

• 26:31Speaker 02

  Yeah, it's a really good point. And then the other thing I think that people often forget is that many markets are not monopolies. Many of them are oligopolies, you know, that's payments with Stripe and AdGen and PayPal and all these things, right? And so it's also possible some of these market structures are oligopoly markets. And then it's possible it actually ends up being winner take all and there's some network effect or data factor. But if you look at some of these types of companies like healthcare, to your point is a great example where it's a deal driven, right? You have large deployments with big customers and you lock them in for multi-year deals. And if you're actually able to lock down customer bases and effectively you can fragment in an oligopoly market more easily than if you have renewals every year, right? So I think also part of it is just like, what's the contractual structure of a market?

• 27:12Speaker 02

  And people really don't talk about that kind of stuff, but I think it's really fascinating to think about through the lens of what actually is a sustainable business in each, each one of these categories.

• 27:22Speaker 01

  You know, beyond healthcare, it's like, where are these businesses going to get disrupted? I think like financial services is the obvious one and they're, funnily enough, I think they've been at the cusp of this stuff in the past. I don't actually think they're on the cusp of it. As, as much as, you know, you think like I, you know, if you think about. Um, a lot of the, the big data stuff like 10 years ago, um, you know, the hedge funds were all over that, right? They're like, hey, there's, there's alpha, there's alpha here. And, and, and I know that, um, you know, some of them are starting to look at large models and language models and, um, and whatnot, but I do feel like they're actually being a bit lagged, um, in terms of their adoption um, of, of these things. And it might actually be because they were so deep in the other sphere, in the, in like the old ML world, that's hard to kind of really quickly turn things around.

• 28:07Speaker 02

  It's just such a different capability set that I think like old school machine learning, or where you're just effectively doing regressions and just pulling out patterns and data. It's kind of different from some of the generative stuff in terms of what it does and what it can do for you. And one of the things I've been thinking about recently related to what you just said is, what are the companies that just don't care about this? And that might be a very good thing because they're defensible, right? In the era of AI eating everything, like what can't be eaten? And therefore, maybe those are really good things to get involved with or to work on because you're not threatened by a dozen different new startups.

• 28:40Speaker 01

  That becomes really hard with like, when, when you start to think about some of the demos we've seen over the last couple of weeks.

• 28:47Speaker 01

  I thought my job was safe until yesterday.

• 28:49Sarah Guo?

  Yes. One, one of my partners asked me how long I thought venture capital was gonna last in terms of like, you know, an agent-based automation taking over, um, because he was like all excited that he got out of software engineering at exactly the right time before his skills became useless. But I tried to give him, I tried to give him a real answer, which is I think on the early stage,

• 29:13Sarah Guo?

  at the early stages, a lot of the data doesn't exist, right? Like you'd have to capture real world data. You have increasingly meetings over Zoom, but, uh, you would want to capture a lot of information about people. Uh, um, so much of it is access and the information about who is like leaving and like a hundred X engineer and entrepreneurial and product oriented and it works with velocity, like a lot of that is not collected today, right? So you have this big inputs and it's, there's no digital trail for it. And so you have this big inputs problem. I think the decisioning, like if you think about, like, what is actually structurally predictable?

• 29:51Sarah Guo?

  Uh, maybe if you have all that data, the like people are the most identifiable piece. But the um, maybe you can, maybe you have a model that is doing continuous learning and can and learn like meta structures like a lot is talking about like, oh, this is a market that operates as an oligopoly where these are the core um, uh, core drivers of, you know, differentiation and these are the dimensions of competition and such. But I think that that feels quite hard when you're investing in a technology landscape that is always changing, right? Um, and so like, you're kind of always out of distribution and you don't have the data on the people. And like, I don't know how you make decisions on whether products are any good because you'd have to have all the customer point of view, or you'd have to

• 30:36Sarah Guo?

  have taste. Maybe models would have taste, so.

• 30:39Speaker 02

  I think Sarah is saying that her job is defensible, which I think is what everybody says. No, no, no, not my job.

• 30:45Sarah Guo?

  I am, I'm just saying, this morning, I gave it a good think and I was like, should we hire people to go work on this? And I was like, nah, that doesn't feel like a tenable problem this year, but I, I commit to if it is feasible, like, we're gonna be first. But I just, I feel like it'll be like, you know, another six months or so.

• 31:04Speaker 01

  So that means it'll be next month. Um, you give them all.

• 31:07Speaker 02

  Actually, in about 20 minutes, there's a company launching.

• 31:10Sarah Guo?

  Wait, I gotta ask you one more question because like, you, you know, you're working closely with Nvidia, you work with the hardware providers. Um, people are really interested in this topic now of like, I, I mean, generally, like, do you believe in hardware heterogeneity? Right? There are some strong opinions on this from, you know, Databricks and others here. Um, uh, and, um, do you like, you know, uh, do, do you still see the same supply demand, um, dynamics around GPU shortage, um, from your customers that you did maybe beginning of last year?

• 31:44Speaker 01

  I think the chips have just changed. So like, I think there is, like, be- before there was a shortage of, it just felt like of everything. Like, if you wanted anything except a T4, you could not get it. That was, that was hard. I think, you know, right now, um, there's, there's two things we're seeing is that one, like, most of the, like, it is now possible for us to acquire compute, for us to acquire compute pretty quickly. We have big spend though. And so that, that is, like, that everything should be kind of, Condition on that is that we are, you know, we're we're making long term commitments with providers. So we're we're able to, um, that gives us negotiating power. I think customers are still struggling with availability um for the most premium chips.

• 32:29Speaker 01

  And I and I think, you know, whether that's eight one hundred or eight one hundred, I think, even when there is availability. You're you're oftentimes looking for like three to six weeks of negotiating with cloud providers. Um, and then your rep calling in favors um in exchange for something or the other, like the amount of times that with the cloud provided with escalated conversations um. Just to get people moving faster is is unreal. And so I think customers are still running into it. Um, I do think it's getting better and I do think that it will, you know, it should seem like it it will go away. I think the heterogeneity argument around different things. I mean, it it's probably a good thing, right? Like, i- if there is more than one one provider of chips, uh, but that that being said.

• 33:13Speaker 01

  Um, I've I personally think that it's pretty overstated how easy it is to um to run. Something that looks like Kudo or Kudo in some form on an AMD chip. Um, seems seems like a challenge to me. I know a lot of people, there are, there are people who who believe that they've got it done. I think the amount of time we we spend debugging bad nodes we get um in a place where that you have a lot of information about. Um, existing infrastructure, um, that's that's challenging as it is. I can't imagine what that'll be on on these chips that are on test. And so like, I think over time, yes, I hope so. That'd be great. I think short term.

• 33:52Speaker 01

  Um, you know, it's really hard for me to see how we make investments beyond um Nvidia, espe- especially when there's a cus- where there's like. Customer a crunch on the other side from customers. We're like, hey, we need this now. And I don't thi- I don't think we want to, the other thing we don't want to do is that. You know, what base ten doesn't give you, it doesn't give you real access to GPU. It's conditioned on this inference problem today that, you know, like you. If I gave you a GPU to use on base ten, there's not that much you could do on it, except inference, just, just in terms of the access control. Um, that we give to you. Um, that being said, like, you know, our customers do end up, you know, fiddling with Nvidia drivers.

• 34:31Speaker 01

  They do end up, you know, um, like installing new versions of PyTorch and, um, and, you know, have like custom Docker images. Um, and I think running those things on things that aren't Nvidia, especially, you know, doing them in an abstracted way. Like, I think it'd be easier if I was building a service on top of these AMD chips and saying, take this service. But, you know, our customers do interface with those, with the GPUs in some way. And I think building an abstracted service where there's heterogeneity is like, that just sounds very, very challenging. Uh, I'm sure there are people much smarter than me who could figure it out. Uh, but I think for us, like, I think that would just add a lot of complexity and really slow down how fast we can move. But I hope, I think, you know, there, it is a good world when there is more than one option.

• 35:14Speaker 02

  How do you see customers thinking about build versus buy? And how do you think that's going to be evolving over time?

• 35:18Speaker 01

  Speed is the only thing that matters in this market. And, and, and what, what this means is that, you know, if, if you, if you spend time fiddling around with your infrastructure. Um, and your service goes down when you launch it, I think that actually hurts the end user experience a lot. Um, and it's something you just don't want to mess around with. And we see this from customers, you know, even customers like. Um, where AI is their core, core thing is that they are understanding that what, what they, what is proprietary to them is models data. And workflow, what is repeated for them is infrastructure.

• 35:56Speaker 01

  And I think like, um, the smarter, like, the amount of times that we've seen in the last 12 months that we're going to build this ourselves, which is very much like how infrastructure engineers were thinking. A decade ago, um, only to come back three months later. It's like, I have a, you know, Docker dumpster fire, um, somewhere. It's like, I, I, you know, we can't have like, it's our super qualifier. Like, have you built it yourself is how we know someone's going to be a great base in customer. Cause they, they empathize with the pain and they know that this is going to allow them to move a lot faster. We had, we had a company with a four person, um, AI infrastructure team.

• 36:34Speaker 01

  That been building this for two years migrate all their workloads over the base ten and thirty six hours. And you know, I think that is a pretty amazing case study for them, which is like, holy crap, we can now take these four engineers and focus on what is actually our competitive differentiated advantage. And the way we think about our business is not, you know, we don't need to scoop everything, something off every single customer either. Like, we offer options where you can run this in your own, in your own environment, and, you know, pay us a license fee. And, you know, like, I think there, it is very, very cost effective, the way that us and honestly, other providers are doing this. Like, I think it's crazy, to be honest, to try to build this yourself, especially at

• 37:15Speaker 01

  the scale that some of these customers are operating at. I was, I was looking at one of our, one like, the customers that we chatted with this morning, who was, who was tinkering around, tinkering around, and they said they were doing a billion tokens a day. This is a, you know, a, a six person chat, chatbot company that has a billion tokens a day going through them. Like, to, to build the infrastructure that supports that with the elasticity, reliability, and, and the performance, and then build the product experience around that, that's impossible for a six person team. And like, I think you should try to take away things that, you know, other people can do just as well, if not better. And that's my take. And I think that's kind of what I see the market coming around to as well, which is

• 37:59Speaker 01

  that speed is a competitive advantage. Let's, let's spend out, like, we can spend our way, we can buy that competitive advantage without a long build cycle.

• 38:08Sarah Guo?

  This is an awesome conversation. Thanks for doing it, Tuhin.

• 38:10Speaker 01

  Thanks for joining. Thanks, Sarah. Thanks a lot.

• 38:13Sarah Guo?

  Find us on Twitter at NoPriorsPod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.