Inferact: Building the Infrastructure That Runs Modern AI

Inferact is a new AI infrastructure company founded by the creators and core maintainers of vLLM. Its mission is to build a universal, open-source inference layer that makes large AI models faster, cheaper, and more reliable to run across any hardware, model architecture, or deployment environment. Together, they broke down how modern AI models are actually run in production, why “inference” has quietly become one of the hardest problems in AI infrastructure, and how the open-source project vLLM emerged to solve it. The conversation also looked at why the vLLM team started Inferact and their vision for a universal inference layer that can run any model, on any chip, efficiently.

• 0:00Speaker 00

  Our goal is to make VLM the world's inference engine, really push the capabilities on the open source front, and then build a universal inference layer. That means we'll have the runtime to power any new model on new hardware for new application, be able to tailor that to extreme efficiency and support all the AI workflow going forward. I fundamentally believe that open source, especially how VLM itself is structured, is critical to the AI infrastructure in the world. And what we want to do with this infra is to support, maintain, steward, and push forward the open source ecosystem. It is only that VLM, when VLM becomes a standard

• 0:45Speaker 00

  and VLM helps everybody to achieve what they need to do, then our company in this sense has the right meaning and to be able to support everybody around it.

• 1:01Speaker 01

  What if the hardest problem in artificial intelligence isn't training smarter models, but simply keeping them running? For most of the history of computing, once a system was built, the hard part was over. You wrote the program, pressed run, and the machine behaved predictably. Even early machine learning followed that pattern. Inputs were standardized, workloads were regular, the computer did its job and stopped. Large language models quietly broke that assumption. Every request is different. Prompts can be a sentence or an entire archive. Outputs can end instantly or stretch on indefinitely. Thousands of users can arrive at once, each making incompatible demands on the same hardware. And all of this has to happen in real time on GPUs that were never designed for this kind of unpredictability. Over the last few years,

• 1:46Speaker 01

  this problem has moved from obscure to essential. As models have grown larger, more diverse, and more deeply embedded into products, the challenge of running AI systems has started to rival the challenge of building them. That's where the tension lies. A public story of AI progress is about better models and bigger breakthroughs. But underneath it is a quieter systems problem. How do you schedule chaotic requests efficiently? How do you manage memory when you don't know when a conversation is actually finished? And what changes when AI systems stop behaving like single-turn tools and start acting like agents that think, pause, and interact with the world over time? This episode focuses on the hidden layer. We examine why inference, the act of running trained AI models, has become one of the most complex and important problems in modern computing,

• 2:31Speaker 01

  and why open source infrastructure is increasingly central to solving it. Matt Bornstein, general partner at Andreessen Horowitz, is joined by Simon Mo and Wu-Sa Kwan, co-founders of Inferact and creators of the open source inference engine, VLLM. This is a conversation about the infrastructure beneath AI and why it may matter more than the models themselves.

• 2:55Speaker 02

  We are here today with Simon Mo and Wu-Sa Kwan, lead contributors on the VLLM open source project and co-founders of Inferact, a new AI inference company. Super excited to have you guys on the show today. Thank you. Thank you so much for coming. We're going to talk a little bit about VLLM, the open source project. We're going to talk a lot about inference and what inference technology really is. And then we'll talk a little bit about Inferact, the new company. So to start, could you talk a little bit about where VLLM came from? What is it? How did you start it? And why is it such an exciting project?

• 3:25Speaker 00

  Thank you for having us. VLLM project started from, actually Wu-Sa's prototype project at UC Berkeley during his PhDs, and grow into today's open source project on GitHub for inference runtime for everybody. Maybe Wu-Sa can talk a little bit about the page attention paper?

• 3:42Speaker 03

  Oh, yeah. So basically, I think it kind of started in 2022 when Meta released the OPT model as open source. I'm not actually sure how many people actually remember the model nowadays, but it was kind of the, one of the first like open weight large language models to reproduce GPT-3. And our lab tried, created a demo service to run the model and to, you know, demonstrate it for the broader audience. And yeah, like it was working, but super slow. So I started a small side project to optimize that demo service. That was kind of at the beginning. And initially, I was thinking that it may only take like a couple of weeks to optimize the service end to end.

• 4:26Speaker 03

  But it turns out that it actually has a lot of open problems inside in it. Because this, you know, auto-regressive language model is pretty different. Actually, it was pretty different from other traditional, like... ML workloads. And it wasn't actually, it was kind of like a brand new, at least like outside these frontier labs. Back in the day, I started to work on it and it became a research project. And we wrote a paper and it even became a like a like open source project, pretty well defined, open source project. As yeah, more and more people got interested in it.

• 4:58Speaker 02

  So two thousand twenty two, this is pre G P T four, obviously.

• 5:02Speaker 03

  Yeah, pre chat G P T.

• 5:03Speaker 02

  Yeah, pre chat G P T.

• 5:04Speaker 03

  Yeah.

• 5:05Speaker 02

  And you're thinking like, oh, I'll just like work on this inference server. This should be a fairly straightforward problem. Like four years later, it actually you're like doing more work instead of less.

• 5:14Speaker 03

  Exactly, exactly. Yeah.

• 5:16Speaker 02

  Why did you think this is a meaningful problem to work on at the time? Because like, I would say most people in the world at that time saw G P T three as a curiosity in some sense. And O P T was kind of like a curiosity attached to a curiosity in a way. Like, what made you and your lab mates sort of excited to work on this back then?

• 5:32Speaker 03

  I think I also started from curiosity. I didn't really think it's the most important problem in the world back in the day. I just wanted to have a hands on experience on how this actually works. I mean, I think I also impressed by the size of the model. The O P T largest model has one seventy five billion parameters. And that was the largest model available. So it's kind of like a meaningful for me, like it's kind of pretty rewarding to work on such a large model.

• 5:56Speaker 02

  This reminds me of when like, when I was like growing up. You know, we would build like computers. That was like the cool thing to do. And each step change in like memory capacity was such a big, I was like, oh, my God, this one has four megabytes of RAM. Oh, my God, this one has five hundred twelve megabytes of RAM. Looking back, it's silly, but at the time that was actually like, maybe it's because we're like nerds. But like it gets, you get like emotionally excited about like the numbers getting bigger on these systems.

• 6:20Speaker 03

  Right, right, right. Yeah, I think that was like one of the main motivation, clearly.

• 6:24Speaker 02

  And so you started to say the sort of technical problem is different for autoregressive transformers compared to traditional machine learning. Do you mind explaining a little bit, you know, how that is? And even compare just to normal. Kind of computing workloads for, you know, listeners who, who, you know, or engineers may not be familiar with AI workload.

• 6:43Speaker 03

  So basically compared to the traditional workload, you know, the clear difference is definitely like GPUs, right? Now all the compute or kind of most of the computer happening on GPU. And we have to optimize for the, which like presumably have less memory than CPU, at least back in the day. Now like GPUs are much, has much larger memory, but typically it has much smaller memory than CPU, maybe still. And like, you know, like all the computations happens on GPU. So you have to write program in a different language, a different type of parallelism in mind. Yeah, so that's kind of like a fundamental difference from the traditional like compute happy workload versus like deep learning workload, I would say. And but within the deep learning workload,

• 7:26Speaker 03

  there's actually still a huge difference between the kind of traditional deep learning workload versus like large language model inference. So for traditional workload, I think the biggest kind of characteristic is that it is pretty like static. Which means, for example, like for image models like back in the day, like CNNs, like what people do is, you know, you may have a different, several images with different sizes. Then what we do is we like, we resize them or crop them into the same size. And then we batch them and then we put it to the model to run them, to run the inference at once. And this is basically, yeah, and that because of this resizing and cropping, like they all kind of, at the end, they're kind of compressing to the same size tensor.

• 8:12Speaker 03

  And that actually makes things much simpler for, you know, for the GPU to handle, right? All the shapes are pretty regular, static and it's kind of like well-defined.

• 8:22Speaker 03

  But for large language model, if you think about it, it's, they're pretty dynamic, you know? Your prompt can be either like hello, like a single word, or you can, your prompt can be a, like a bunch of like documents spanning like hundreds of pages. And this kind of like dynamism exists in, not inherently in the language model. And this makes things whole like, kind of into a different world. Like we have to handle this dynamism as a first-class citizen.

• 8:48Speaker 00

  And yeah, back in the day, that was not like, people didn't have a clear idea about how to handle it. And yeah, fortunately, we were the one of the first to, yeah, to solve, to see the problem.

• 9:00Speaker 02

  That's very interesting. So, so kind of regularizing a batch of inputs was, it sounds like one of the first problems you had to solve.

• 9:08Speaker 00

  It's actually more about Scheduling and memory management. Yeah, yeah. As well, yeah. We like to Yeah, so the problem we're solving before in all the serving system is about just what we call micro batching. To leverage first CPUs fundamental like vectorization in the early days before L M's. And then early GPU for vision, vision models like ResNet is all about micro batching. You put together four requests together that arrive around the same time. But the change in the L M world is you always have requests that continuously flowing and coming in. And then each request looks differently, you just cannot really normalize them. So that's why you have to have a notion of a step within the L M engine

• 9:50Speaker 00

  to process one token across all the requests at the same time. Regardless of each request having different kinds of input lens and output lens. Now, it's also non deterministic. The language model itself will decide when does this stop. Instead of in the traditional sense of other machine learning servings, it's very much like work like a clockwork. And here it's very stochastic, it's always flowing, it is always continuous. That's why scheduling is the first problem to solve. And then memory management, that's where page attention come about is a second problem to solve.

• 10:22Speaker 02

  So when did you get involved in the project, Simon?

• 10:25Speaker 00

  Well, I got involved as in around twenty twenty three. Uh, I first, Usuk issued a call in the Skylab Slack channel to say, hey, we need someone to work with us on this page attention paper and kernel. Actually, surprisingly, I was on spring break and I was like, look, someone else can do this. Let me just play with G P T for the entire week. So I would just ended up just playing with prompt engineering. So I actually didn't end up joining with Usuk.

• 10:50Speaker 02

  And so this is what a vacation looks like in Jan Stojka's lab, playing with models for a week.

• 10:54Speaker 00

  And he's playing with kernels. Yeah, exactly. So he's playing with kernels. I was trying to build more prompt engineering and explore it like different kind of agent, early agent workflow. And then over the summer, and especially this is when around August or September, and we really get to work together on, actually, this is where you come in. We get to work together on our very first VLM meetup, A C Q Z, and where I had the experience of managing open source project before, as well as deeply interested in actually building a serving platform and into a fully open source project. And this is where I start to get involved, write my first lines of code,

• 11:36Speaker 00

  and like sort of build out the C S system, build out the performance benchmarking systems, and then really much work with Wusuk ever since.

• 11:43Speaker 02

  I had forgotten about that. So this was the very first VLM meetup, right?

• 11:47Speaker 00

  Yeah.

• 11:47Speaker 02

  It was in this office.

• 11:49Speaker 00

  In this office, on this exact floor. I think we are previously anticipating just 10, like 10, 20, maybe 50 people showed up, and then the registration was like exactly over the anticipated capacity. People are extremely interested in this technology.

• 12:04Speaker 02

  I remember that very well because we run events here for ourselves, and it's always very hard to get people to show up. We're always scrambling. And instead, I got a call from our security team saying, "Too many people have been approved for this VLM meetup. We need to scale it back. This isn't safe." I'm like, "Oh, okay. Probably don't tell." I don't think we ever scaled it back, so don't tell the security.

• 12:24Speaker 00

  It was quite crowded. The P S I ran out like the first 10 minutes, so.

• 12:28Speaker 02

  But this is a big deal, right? Because this is not like a consumer app, right, that you were building. This is pulling from systems engineers, right, for the most part, who want to learn about how to serve L L Ms and contribute to it. And so it's actually a big deal to get, I think, so much interest from such a kind of narrow, sophisticated group of people who don't like meeting other humans in real life that often either, you know, at least speaking for myself. So can you talk a little bit more about the community behind VLM? Like, how big is it now? How did it come together? And like, how do you guys manage it as it's gotten big?

• 12:59Speaker 00

  Yeah, so in the beginning, of course, it's just a few grad students working on it. And then so, but over time, we start to having this very much an open minded, developing the open kind of mindset. So as of now, we're looking at fifty or more regular full-time contributor who open up GitHub every single day to work on V L M. We crossed two thousand contributor bars on GitHub, one of the fastest growing top open source project ranked by GitHub itself. And then this is really a diverse community. So there is folks like Usuk and I, also the team from UC Berkeley from grad student days, and as well as Meta and Red Hat pulling their way behind this open source project.

• 13:45Speaker 00

  And then as well as, of course, people who are not just, people who are making the model, Mistral and Kun team. And of course, like anyone who's making open way model are participating in our community. And then on the model side, Nvidia, AMD, Google, AWS, Intel, they're all having their own participation and be able to support the ecosystem. So everyone in V L, using V L, has the ability to choose among different silicones for accelerated computing.

• 14:13Speaker 02

  That's very interesting, though, which I think is a property that many successful open source projects have, which is that people aren't all contributing for the same reason, right? Some people, I'm sure, just love the technology, but it sounds like you're saying the model providers actually have incentives to contribute to the project because they want their models to run well. The silicon providers want it to run well on their silicon. The infra providers want to have first dibs on running it so they can sell infra, that kind of thing.

• 14:36Speaker 00

  Yeah, this is kind of a classic, we're solving the M times M problem. So that as a model provider, you're doing, you talk to everybody. And as a hardware provider, you can just go into this one system and then magically it'll work for all the models out there in the world. And then for applications who are using V L M as well as infrastructure building with V L M, like having a common ground where everybody can participate in and then innovate together is way easier and cheaper, in fact, in the end to deploy.

• 15:06Speaker 02

  What's your philosophy for managing a pool of contributors this large? Do you tell them what to do? Do they choose themselves? Like, how do you maintain high code quality?

• 15:15Speaker 00

  It's a constant sort of iteration, month over month, year after years. So for this, I have to go back to my previous open source project, which I was working on a project called Ray, and then later AnyScale, where we have this kind of, where I learned this, a community-driven approach in a way that have a clear requirements, have a clear roadmap, have a clear sort of milestone being set. So we kind of tried to borrow that, but also really study this really successful open source project out there. I went all the way back to Linux, and then to, and then study Kubernetes, study Postgres. How are these communities operating and together? So in Viam, we had kind of a special model that we do, like any normal engineering organization,

• 15:59Speaker 00

  set clear team scope, but also clear objective and result and milestones with different kind of technologies, technical features. We want to push forward and build. So this is where we have set forward our vision every quarter. And then, but also invite the community to contribute. So we're saying, great, we're working on these. We also need help on these items that we don't have anyone actively working on. If you are brand new and want to engage with us or engage with the community, here's what you can work on. And additionally, we keep an extremely open mind to all the GitHub pull requests that people just opened up. That we're seeing, oh, is this a good request? Is this a good feature? And then as well as a request for common processes. So kind of is a blend of all the lessons learned

• 16:44Speaker 00

  from the previously, from previously other open source projects. And then code quality-wise, code reviews, but also a lot of constant refactoring iterations.

• 16:54Speaker 03

  Yeah, yeah, I do a lot of refactoring, like every six months. And yeah, and actually, one thing to add is, you know, like we do in-person meetups, you know, like every two months. And we are kind of expanding to globally, actually, like sometimes in Europe, sometimes in some other places in Asia.

• 17:11Speaker 00

  Yeah, and yeah, like we actually from the first meetup in A C N Z, we learned that it's actually super, super useful to meet, you know, those like collaborators and users in person. And yeah, we are continuing doing that.

• 17:24Speaker 02

  It's funny, it's another one of these lessons that like, you know, Silicon Valley engineers, like we've gotten so kind of like, you know, high up the abstraction stack that we're like relearning, you know, lessons from a thousand years ago saying, oh, it turns out in-person communication is high bid and doesn't suffer from consistency problems. So, so around the time you guys did that first meetup, we also made grant funding to, to the project through, through the academic lab. I was, I think it was a small amount of money, but it was actually the very first open source grant that we made. So it's, so it's super, you know, just like fun and kind of gratifying for us to see like, the money was actually put to good use and the project grew massively. And then we even had a chance to invest in a related company later. However, I did hear a rumor that at the time that we made the grant funding,

• 18:08Speaker 02

  that you guys put a portion of the money into Nvidia stock. Can you confirm or deny?

• 18:13Speaker 00

  Neither did it.

• 18:14Speaker 02

  Not him. So someone else in the, in the recipient list. So, so you probably turned our tiny grant into 10 times as much money before, before.

• 18:22Speaker 00

  Oh, sort of, sort of the funding for V O M. A lot of these funding for V O M is that we set aside for a project development and sort of project development, testing and everything around operating this project. And once here, we're actually super grateful for the first grant is actually kicked off a culture. And nowadays you can get even a tradition for people really open up to sponsor open source projects in a quite significant way. Because running V O M, our C I B L, for example, is more than 100 K amounts. That's could be tiny for some folks and it's like over growing like over time. This is where at a burn of million dollar amounts and sorry, a year, a million dollar a year.

• 19:06Speaker 02

  For an academic project, it's actually very.

• 19:08Speaker 00

  Yeah, because we want to make sure every single commit is well tested. And then this is something that people can deploy at not thousands, but potentially millions of GPUs across the world in different environments. So we want to make sure it's well tested, it is reliable. And then. And this requirement, this infrastructure, or right now, all comes from contribution and sponsorship. And from everybody are chipping in to help on this project. And now, of course, we also run meetups and sometimes expenses associated with meetups are directly leveraging the sort of the grants that you all provided.

• 19:45Speaker 02

  Yeah, I mean, it makes sense, you know, for us and for other corporate sponsors of the LLM, you know, it benefits the whole ecosystem, right? So I think it makes a lot of sense. Let's talk more about the technical aspects of the problem, if that's okay with you guys. Do you mind to start just defining exactly what like an inference server and inference engine is?

• 20:03Speaker 00

  Sure. So an inference engine turns, it takes a already trained model. So this can be a very small model like Qwen-1B, it could be a very big model on DeepSeek or Kimi K2. Run it on a servered computing device. And its job is to fully utilize the computing device to be able to generate text and images and videos essentially. But this all got tokenized into individual tokens. So the goal of inference engine is to produce, the goal of inference engine is to run the model at highly efficient speed. To make sure that we can produce maximum outputs at the highest efficiency.

• 20:48Speaker 02

  And just from a high level, can you explain some of the architecture, how sort of a typical inference engine works? What are just the few most important components that people would be interested to learn more about?

• 20:58Speaker 00

  Maybe one goes through a life of a request, like if I say hello, what will happen to V R?

• 21:03Speaker 03

  Yeah, so basically there's a kind of a traditional API server, definitely, you know, guess the retest and maybe. And once the model generates output, it stream backs the tokens one by one. Yeah, so there's a definitely a traditional API server layer. And inside in it, we have kind of typically something called tokenizer, right? Like to transform this like inputs to like the tokens, the basically some integers, the list of integers that the language model can consume. And inside in it, we have basically an engine, what we call like engine. And that includes scheduler to, you know, to how to, which decides how to batch the request into incoming request. And we have a memory manager to manage something called KV cache,

• 21:47Speaker 03

  which is a kind of the core part of the transformer to for etalims. And we have a definitely have a some kind of worker, which is very generic term, but which basically actually initialize the model and run the model and get the output. And, you know, do all the like pre-processing for the input and, you know, post-processing for the model output. Yeah. So, yeah, yeah, that's basically, I mean, yeah, in a sense, it's not like a crazy new architecture, but each one basically highly optimized and specialized for this LM inference workflow.

• 22:20Speaker 02

  Do you think it's getting easier or harder over time, running, running inference?

• 22:24Speaker 03

  Yeah, definitely. I think it is definitely much, getting much more difficult over time. Like, actually, honestly, maybe one and a half years ago, I wasn't thinking like inference as a hard problem at all, to be very honest. But now things have changed, the trend has changed so far. So I think there are kind of three factors. One is scale, another is diversity, and the last one is kind of agents. So for scale, you know, like the models are definitely getting larger. And, you know, right now we have a Kimi K two with a like more than one hundred, more than a trillion parameters.

• 23:06Speaker 03

  Uh, but I think we believe we will see like multi-trillion parameter open source model this year. And I think that's still clear, clearly a trend that people will be training a larger model. And, you know, definitely it's much more challenging to deal with such a model compared to, you know, like the early days of etalims where we just only deal with like small llama models.

• 23:28Speaker 02

  And with larger models, presumably you need more nodes working concurrently. You need, you have more memory to manage that may or may not fit in each. You know, chips available memory. Can you describe some of the challenges from scale?

• 23:40Speaker 03

  Yeah, for these kind of large models, we definitely need to shard, you know, distribute the model into multiple like GPUs, multiple nodes. Right? And then, and yeah, then there's like definitely like a problem of how to shard, how to distribute this model, right? There are actually many dimensions we can use to shard the model, and they have a like different like trade offs. And yeah, trade offs, for example, in terms of how like how much communication we should pay to shard the model in this way. And also, there's a trade off in terms of like load balancing. Or if I shared this in this dimension, then how does the how significant is the load imbalance? So these all need to take into account for the final performance estimation and to get the best performance.

• 24:25Speaker 03

  And yeah, that's it's becoming more and more a bigger problem as the models get larger.

• 24:32Speaker 02

  And what about just cluster scale? I mean, I think, Simon, how many how many nodes is V L L M running on in any given time?

• 24:38Speaker 00

  Right now, we're looking at, this is through our sort of like a very small substance sample of our usage statistics that is used for us to figure out what feature to deprecate. Just literally from this one signal, we're looking at 400 K to 500 K GPUs, 24/7 running V L M. And this is quite a big scale, thinking about the global deployment of GPU footprints. And we definitely believe there's a lot more out there. And of course, this is a wide diversity of different kinds of GPUs, GPU architecture, as well as model architecture being deployed. We're not seeing like a one size fit all. People are using it for just one singular use case.

• 25:18Speaker 02

  I see. And and this is sort of your point. Your second point was like about diversity, sort of making it a harder problem over time.

• 25:24Speaker 03

  Yeah, the chip diversity, harder diversity is definitely one factor. And also, models are getting also diverse, you know? If you think about the like, for example, like for Nvidia, like a year ago, I think they only released a few series of open source models. But now they're releasing many open source models like every month in different domains, right? Someone on the video, someone on the robotics. Someone on the on the language. And yeah, this kind of like open sourcing trend is getting expanding. And that people are training many different kinds of models in many different domains. And releasing them like every month. So there's model diversity and even for just for text models.

• 26:05Speaker 03

  They're all transformers in that, but their detailed architecture is still are very diverse. And they're even, we see they're even diverging. Like say for like Deep six three point two was using sparse attention. Something called sparse attention. But say for Q one and Kimi, they're kind of exploring like linear attention, which is kind of different attention mechanism. And they have different ways to manage the memory. So yeah, this model architecture divergence is also getting uh getting more significant.

• 26:38Speaker 02

  And so is it up to you as you know, meaning BLM to implement all of these? Like all the you to, you know, implement sparse attention, for instance, so that it's available for the models to use?

• 26:47Speaker 03

  Yeah, definitely. We basically leverage open source community, definitely like we, you know, because we collaborate with these model vendors. Like we often get help from these model vendors. They basically provide some kernels or a list like reference implementations of, you know, of these new kind of like operations. And yeah, we like our job is often like basically leverage this collaboration and making more mature. And also available for more diverse environments.

• 27:17Speaker 02

  I remember early on in open source models, there was some standardization, like everyone was kind of using Llama. I think everyone's using sort of like the same tokenizer and the same like input format. And you know, and like end of stream token and stuff like that. Is that still the case or is it like, is it different for each provider now?

• 27:35Speaker 03

  It is, yeah, it diverged quite a bit over the last few years, maybe last couple of years. Yeah, one thing is that many, yeah, like the model architecture itself has changed a lot, you know, especially on the attention side. And also even for like input output processing, because like different labs have different kind of their own ways to form, you know, how to form the conversation and how to form the tool calls, for example, for their own models. So now like this has been diverging quite a bit. And now, yeah, this has been diverging quite a bit for the last couple of years.

• 28:07Speaker 02

  I see. Okay, so scale of models, diversity of models and hardware deployment scenarios. And then agents were the third thing you mentioned, sort of getting hard over.

• 28:15Speaker 03

  Yeah, yeah, you know, like for agents, we need a, definitely, we need a kind of different. I mean, beyond, just beyond the inference engine. We also need to set up the whole new, like, environment, actually, whole new, like, infrastructure to support all the tool callings and to support all the, yeah, like, multi-agent things. Yeah, like, that part becoming a, kind of a new, like, emerging challenge for inference as a whole.

• 28:40Speaker 02

  Do you think this means more, there will be more state managed in the inference layer over time?

• 28:45Speaker 00

  As before, the paradigm has been text in, text out, and then just single request, response, and then. But as we evolve into the year and the decade of agents, we're seeing multi-turn conversation turning into hundreds and thousands of turns. And then these terms also involves external tool use, like, interacting with Sandbox, performing web searches, running Python script or any programming languages, and be able to have this kind of long iterative process where LM is involved, but also external environment interaction is involved. And this really kicked off a huge wave of co-optimizing a agentic architecture with inference architecture.

• 29:31Speaker 00

  So just to give an example that when, because it, just to give an example, it is very important for VLM to understand whether or not the conversation is still happening. If the conversation is no longer happening, we can remove the KV cache. That is the persistence state associated with each text string, completion streams. But in agentic use cases, you actually don't know whether or not the agent will think it finishes. Or also, we, the interaction previously, and the interaction previously was just a human typing in the text box. But now it becomes external environment interaction. It could be one second just for a single script to finish.

• 30:13Speaker 00

  It could be 10 second for a search or like a complex analysis to finish. And then it could also be minutes, hours, if there's humans in the loop. Now. With that uncertainty, we actually don't even know when is a request going to come back. And then the uniformly of cache access pattern and eviction pattern got kind of, the patterns got pretty disrupted by the new paradigm.

• 30:36Speaker 02

  I see, I see. And so you have to be much smarter about how you manage the cache as one.

• 30:41Speaker 00

  As one example of that, yeah.

• 30:43Speaker 02

  Gotcha, gotcha. Which is one of the unsolvable problems in computer science.

• 30:47Speaker 00

  Cache invalidation.

• 30:48Speaker 02

  Yeah, so exactly. So I can see how that would get harder over time. I think I know the answer to this, but are you guys big believers in open source AI compared to closed source? And can you just explain like how you think about that?

• 31:00Speaker 00

  We're definitely big believers in open source. What we believe is diversity will triumph that sort of single of anything at all. So that means we believe in diversities in models, diversity in chip architecture. Fundamentally, this is because the world is complex. For any application, you're going to need to find and tailor the right sort of model architecture to the right chip architecture for the right exact use cases. And the best way to promote diversity and improve that is through open source. Because open source, everybody know where everybody else is up to and be able to make their opinion take based off the common ground.

• 31:41Speaker 00

  And finally, if you look at the history of computer science, operating system, cluster managers, databases, every single system field get better when they're starting to have a common standard. And everybody deviate a little bit, innovate on top of each other versus following a single line of trend that is proprietary and single source control.

• 32:04Speaker 02

  I see. That's very interesting. So you're almost saying OpenAI will tune their stack very tightly for their use case, which is ChatGPT or whatever other apps they're running. For an enterprise or another tech company, if I want that same level of tuning, I like can't just use off the shelf. Close source models, because I don't sort of have control the whole stack and like the different participants in the stack kind of aren't paying attention.

• 32:27Speaker 00

  Yeah, of course, one part is data, one part is the model architecture itself, which will impact the performance. And then ju- just on the model architecture itself, right? How smart do you want the model to be? Do you want the model to be, be able to handle millions of context, token context, or just sure the context is totally fine, right? And then you also need to specialize that model to your exact compute architecture. What chip are you using? For example, for Nvidia, the model you design for a H100 chip is very different from a B200 chip. And then it is very different for a GB200 MVL72 system. And then compared to, for example, the model architecture you designed for TPU, then again, that is also drastically different.

• 33:09Speaker 00

  And then using it for vision model, video generation, and for reasoning mask coding. In the end, we'll all look, kind of look at the vertical stack integration, and we're like, wow, they're so much different from each other.

• 33:22Speaker 02

  That makes sense. Can you just share any stories about live BLM deployments that, that you like, thought were particularly interesting or important?

• 33:31Speaker 00

  I have a few. One is, I think around 2024, we learned that Amazon is running VLM to power their Rufus assistant bot, which was like, really surprising to all of us, because one, as a point, like, of course, like, we believe VLM can be deployed at scale, but seeing this as a massive scale, like, kind of global e-commerce, deploying this as like, front page feature. That means when everybody, when they're opening Amazon app and clicking the bot suggestion, or even entering a search query, is going through VLM. And this is kind of the first sort of magical experience in a way. One of the first experiences was, wow, my purchase is going through VLM right now.

• 34:15Speaker 00

  It's kind of exciting, but also scary.

• 34:17Speaker 02

  You're like PhD students at the time.

• 34:19Speaker 00

  Yeah. And also, across not just Amazon, LinkedIn, and every major deployment of VLM, we're surprised to find out they're always the first adopter of cutting. edge features. So I've seen one of the example of deployment of VLM within character AI was when we first make the N grand speculation for a spec code available as just a single PR pull request in VLM, not even merged. And then while we're still iterating on that feature and I hear someone from character AI saying, oh, actually, we already wrote it out to hundreds of GPUs at scale. Given just your first iteration of this feature. So it's really much, everybody is staying on the cutting edge of VLM and we're quite excited about that. Yeah.

• 35:04Speaker 02

  Okay. Should we talk about the company then, Inferact? What is Inferact and why did you guys decide to start the company?

• 35:11Speaker 00

  So Inferact created by the creators and maintainers of the VLM project. Our goal is to make VLM the world's inference engine, really push the capabilities on the open source front. And then build a universal inference layer. That means we'll have the runtime to power any new model on new hardware for new application. Be able to tailor that to extreme efficiency and support all the AI workload going forward.

• 35:41Speaker 02

  And implicit in what you just said is that you're devoting a lot of resources, I think, to the open source project. Could you, I guess, is that right? And can you expand on that?

• 35:49Speaker 00

  Yeah, one thing we believe is I fundamentally believe that open source, especially how VLM itself is structured, is critical to the AI infrastructure. And what we want to do with this Inferact is to support, maintain, steward and push forward the open source ecosystem. It is only that VLM, when VLM becomes a standard and VLM help everybody to achieve what they need to do. Then our company in a sense have the right meaning and to be able to support everybody around it. So open source is definitely number one and in fact, sometimes the only priority of our company right now. Yeah.

• 36:30Speaker 02

  You're not supposed to tell your investors, by the way, that.

• 36:34Speaker 00

  We do believe that open source project is also kind of a secret weapon. In a sense that having this community all work together for this open source. We have the execution beyond any single entity can have. This is the thing we heard over and over again that people just tell us. We just cannot keep up with VLM. So that's why we're using VLM. We have our internal team. We have our internal fork. We have our internal inference engine. But open source moves so fast that the only way to stay ahead is adopting. And that's why we want to make it happen. And in fact, this is exactly why we're staying all in on open source.

• 37:13Speaker 02

  That's awesome. We mentioned Jan Stoika before, obviously, one of the founders of Databricks. He was your, I think, both of your PhD advisors at Berkeley. And he's going to be involved in Inferact, too. Can you talk about maybe a little bit how he's going to be involved in this company? And even more importantly, what have you guys learned from him as, you know, his students? And about startups and, you know, distributed systems and all this stuff?

• 37:34Speaker 00

  Sure, yeah. Yeah, you're exactly right. Jan is both of our co-advisors. I have actually worked with Jan since 2017, since I was an undergrad working on my first open source project for serving. And then worked with him at any scale for my second open source project for serving.

• 37:49Speaker 02

  You're just addicted to like Berkeley-based open source AI serving companies?

• 37:54Speaker 00

  So as this company and VLM, Jan is quite involved as, so as a company, he will be a co-founder. And then as an open source project, he has been advising this project for, since its inception. Jan knows open source project, academic project, industry research trend in and out. So from what we're working together on, Jan really helps us with both clearly understanding the, all the lessons learned about bringing open source through the final miles of adoption in companies, enterprises, as well as what is actually happening on the research world.

• 38:34Speaker 00

  A Sky Computing Lab over the last few years has produced amazing infrastructure and new research ideas. And Jan continued to explore a new frontier on that front. And then we're quite excited to hear that and also innovate on the open source together.

• 38:49Speaker 03

  Yeah, and he also helps recruiting a lot. And, you know, like, he is involved in all of our hiring process. He basically tells us, I mean, teaches us how to tell, you know, talents, where to find talents. These are all amazingly helpful.

• 39:04Speaker 02

  So on that topic, what are some of the big problems you need to solve now? And what type of people are you hiring to help you solve?

• 39:12Speaker 03

  Definitely, you know, the inference at scale is kind of the, one of the biggest challenge. I think in the field, not only for us, but in the field overall. So we are trying to hire more like a very experienced ML, ML infra engineers overall to make, you know, for, for example, you know, how we, what would be the best way to utilize the GB two hundred, GB three hundred, MBL seventy two rack entirely for the giant open source model. Still, I think it's a, it's an open problem. There are definitely some endeavors in academia and industry. But I think there are some like room for improvements. So yeah, that's some of our focus at the moment.

• 39:51Speaker 00

  Here's my pitch from a computer science point. Pretty rare if people ask me this question, that is,

• 39:59Speaker 00

  if you're working at a vertically integrated company that have an end product for, let's say, for chatbots, for assistant, you are working on the vertical slice of the problem. In Inferact, you will be working on a abstraction of horizontal layer. And this is similar to operating system, databases, and different kinds of abstraction that people have built over the years. Operating system, abstracted CPU and memory. Databases and file system, abstracted storage devices and networking. For accelerated computing, there's a brand new physical device that inference and VR abstracted a large part of it for inference specific work.

• 40:43Speaker 00

  Of course, it's training, but we are, our singular focus is on inference. And this necessitates.

• 40:52Speaker 00

  A layer, a software layer that abstracts away GPUs and accelerated computing device for models. And this is as important from my point of view as abstraction you need to build for OS for databases, which are both fields we're really passionate about when we're PhD students too. So, and that's why MO system is fundamentally a new system research and system deployment. So you, here at Infraic, will be working on this layer that is not a vertical slice, but a fundamental, about a fundamental runtime and impacting all the future generation of software that will run on accelerated computing device.

• 41:33Speaker 00

  And your work will stand from both working with different models, different, and then working with different applications, and as well as understanding the pros and cons of different chips, as well as their whole integrated data center systems, to be able to figure out, oh, actually, for these, we should build abstraction this way. And we'll constantly remove abstraction, break abstraction, and build it over and over again, just like how operating system got innovated over time, databases got innovated over time, with a new information we have at hand. So you'll come here to have the constant exercise of building an actual widely deployed

• 42:14Speaker 00

  production system that sort of, that will be at the frontier of this.

• 42:20Speaker 02

  And this is what you call universal inference layer?

• 42:22Speaker 00

  Yeah. It's purposely vague in a way, but what we really focus on is going from page attention, from going from the serving system to the whole runtime you need for intelligence.

• 42:37Speaker 02

  Wusuk, Simon, thank you so much for being here today. Thrilled to have you on the podcast, of course, and we're thrilled to be, you know, working together in the company. It feels like it's been a few years. We've already been working together, but, but yeah, great to have you here and congratulations on, on getting off to a great start.

• 42:53Speaker 00

  Thank you for having us. Yeah, thank you.

• 42:57Speaker 01

  Thanks for listening to the A16Z Podcast. If you enjoyed the episode, let us know by leaving a review at ratethispodcast.com/A16Z. We've got more great conversations coming your way. See you next time. As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see A16Z.com/disclosures.