Ep 74: Chief Scientist of Together.AI Tri Dao On The End of Nvidia's Dominance, Why Inference Costs Fell & The Next 10X in Speed

Tri Dao, Chief Scientist at Together AI and Princeton professor who created Flash Attention and Mamba, discusses how inference optimization has driven costs down 100x since ChatGPT's launch through memory optimization, sparsity advances, and hardware-software co-design. He predicts the AI hardware landscape will shift from Nvidia's current 90% dominance to a more diversified ecosystem within 2-3 years, as specialized chips emerge for distinct workload categories: low-latency agentic systems, high-throughput batch processing, and interactive chatbots. Dao shares his surprise at AI models becoming genuinely useful for expert-level work, making him 1.5x more productive at GPU kernel optimization through tools like Claude Code and O1. The conversation explores whether current transformer architectures can reach expert-level AI performance or if approaches like mixture of experts and state space models are necessary to achieve AGI at reasonable costs. Looking ahead, Dao sees another 10x cost reduction coming from continued hardware specialization, improved kernels, and architectural advances like ultra-sparse models, while emphasizing that the biggest challenge remains generating expert-level training data for domains lacking extensive internet coverage.

• 0:00Speaker 00

  TreeDow has driven many of the most important breakthroughs in AI over the past years. He was a lead author of flash attention, which has been a key reason why model inference costs have gone down so much. He was super involved in safe space model work via his work at Mamba, the alternative architectures to transformers. He's the chief scientist at Together, and his work's been so impactful that Semi Analysis basically noted that a huge part of the Nvidia mode is the fact that Tree is doing all this work in the Nvidia ecosystem. It's a real privilege of unsupervised learning to get to sit down with someone like Tree and just ask him anything that's top of mind, including the percent of workloads that are going to run on Nvidia chips in two, three years. We talked about the future of AI hardware and where the opportunities are. We talked about how AI coding products have made Tree 50% more efficient in his work, which was way higher than I'd expected. And we also talked about, you know, his thoughts on the future of alternative architectures

• 0:44Speaker 00

  to transformers and what might really work. Just a fascinating opportunity to talk with someone who's so brilliant in this field. Really, really enjoyed this conversation, and I think folks will too. Without further ado, here's Tree.

• 0:56Speaker 00

  Well, Tree, thanks so much for coming on the podcast. Really appreciate it.

• 0:59Speaker 01

  Yeah, yeah, sure. For sure. I'm really excited about this.

• 1:01Speaker 00

  I feel like when I started this podcast, you know, two and a half, three years ago, you were on my like dream guest list. And so I think that was like when kind of flash attention, you know, was was initially making the rounds and people were just oohing and aahing over the impact you were having on the space. And I was like, if we really can build this up over time, maybe we'll convince Tree to come on. And so

• 1:19Speaker 01

  Yeah, yeah, I mean, I'm very honored to be here. I mean, this should be really fun. Yeah.

• 1:23Speaker 00

  Yeah. So I think we'll hit on a bunch of different things, you know, that I think will be of interest to our listeners. I figured one place to start would just be like a classic VC question. So you'll have to forgive me. The Nvidia story over the past years has just been, you know, unbelievably impressive to get to the scale they're at. And obviously, you've, you know, contributed a tremendous amount to that ecosystem and making, you know, that hardware, you know, better and better for folks to use. Are we going to see competitors to different parts of the Nvidia ecosystem on the chip side, on the kind of systems, the package GPUs? Obviously, you've got, you know, Google and Amazon with their own chips, but, you know, for the for the vast majority of the rest of the industry, I'm sure this is something you think about a lot.

• 1:57Speaker 01

  Yeah, it's been a fair amount of time thinking about the chips. I think certainly there's lots of competitors are trying to enter this space. I mean, AMD has been here for a while. Obviously, Nvidia Nvidia is dominant for a couple of reasons. They make, they design very good chips and they build very good software. And that creates this ecosystem where people build on that. People build software on, on top of that. But I think certainly we'll, we'll see a lot of competitors entering the space. As the workload starts to be a little bit more sort of coalescing around on the architecture side, you know, transformer and M O E and so on. It becomes a little bit easier to design the chips for that kind of workload.

• 2:43Speaker 01

  Right. So if you just focus on inference, I think AMD has certain advantages. You know, they have larger memory and so on. So we're starting to see folks picking that up. On the training side, it's a little bit more difficult. I mean, networking is one of the main bottleneck and Nvidia certainly had there. But I think people understand what are the, the challenges to build good training chips? What are the challenges to build good inference chip? So it comes down to execution. So I would say this space is actually really, really exciting. I talked to lots of folks who are designing new chips, either inference or, or, or training. So I would expect in the next couple of years, maybe some of the workload will become multi silicon.

• 3:25Speaker 01

  They'll, they'll probably run on, on, on different chips rather than, right now it's a, I don't know, 90% on, on Nvidia.

• 3:31Speaker 00

  Do you think the, the architectures are like stable? Cause obviously I think one of the fascinating parts of chip design is you're basically making a bet two, three years in the future of what, you know, inference workloads and training workloads will look like.

• 3:41Speaker 01

  Right.

• 3:41Speaker 00

  And so obviously the more certainty you have around that, the easier it is to make optimizations.

• 3:45Speaker 01

  Yeah, yeah.

• 3:46Speaker 00

  Does it feel like there's enough, you know, stability to make those bets? Or is the idea that basically, look, there's dozens of startups that are, and companies that are making bets and one or two of them may work out.

• 3:55Speaker 01

  Right, right. Yeah. So I think on the architecture side, from a very high level, it looks like it is sort of stabilized around transformer. But once you look a little bit closer, there's actually quite a bit of change around that. So the most recent one in the last couple of years has been mixture of experts. So where you, you make the model much, much larger, much more, more number of parameters, but sparser.

• 4:17Speaker 00

  Yeah.

• 4:18Speaker 01

  And so that has certain. Trade offs, you know, you need more memory, but maybe the amount of compute you do is a little bit smaller. And so even that, that presents a difficulty for some of the chipmaker because maybe they were designing for a dense model where it's very uniform compute and now you have something sparse, so it's a little bit harder to design for. Or some of the stuff changing around, you know, attention has been around since 20, you know, for 10, 10 plus years.

• 4:46Speaker 00

  Yeah.

• 4:46Speaker 01

  But there are changes here and there that actually make some of this stuff difficult. So, you know, DeepSeek has this thing, multi-head latent attention that looks a little bit different than attention. For example, they have very large head dimension and, you know, if you're a stock, right, the matrix multiplication engine has a certain size, maybe it doesn't fit, right? Like weird things like that, once you look underneath the cover. So I would say that's one on the architecture. And then on the workload side, I think people are using these models for very different kinds of workload, right? Do we have the traditional chatbot? Traditional has been in the last two, three years.

• 5:23Speaker 00

  It's been around for years as traditional in AI world.

• 5:24Speaker 01

  Right. But there are new, you know, new workloads, like this kind of coding workload, like cursor and wind surf and so on, is a kind of agent workload where it's not so much as like running the model, but also making tool calls, you know, running Python interpreter, doing web search and so on. That presents challenges of how you design the chip. If the chip is very much focused on like just running the model as fast as possible, you might neglect things like, oh, how are you going to connect to the host to run the web search, that kind of thing. So I would say even though from a high level, the architecture has kind of stabilized, but underneath the cover, there's still a lot

• 6:06Speaker 01

  going on. And part of it is the workload is also changing. So it's a, it's a, it's a constant raise of how fast can you execute and adapt to these new workload.

• 6:17Speaker 00

  So if 90% of these workloads are on Nvidia chips today, where do you think we are in kind of two, three years?

• 6:21Speaker 01

  Yeah, I think on the inference side, some of it will diversify. And we're starting to see companies like Cerebras and Grok and Samenova really presenting a serious challenge. Like, they were pitching, hey, we can get very, very low latency inference. And that's fantastic for some of the use cases. So we talked to customers who really, really care about as fast, as low latency as possible, and they're willing to pay more for that. So I think we're starting to see more of these niche areas where people really care about low latency, or some people really care about very, very large batch, high throughput

• 7:03Speaker 01

  inference, those things like just massive data processing, or synthetic data generation, or RL training, where you have to just roll out and generate as many trajectories as possible. So I would say it's going to diversify simply because the workload is going to diversify. Totally. Whether low latency or high throughput or something else, maybe video generation that will require sort of a different profile in terms of how much compute do you need and how much memory do you need.

• 7:33Speaker 00

  If you were like a startup today, it kind of feels clear in the moment, hey, here's all the different optimizations one might want to make, given all these use cases. And then you kind of decide, hey, we're going to make a chip that optimizes for these things. And you wait for tape out and the actual ability to produce. And by the time you've done that, it's a few years down the line. Do you think that it's clear enough, like, what the different optimizations are going to be, such that you can kind of make that forward-looking bet? Or is there such a risk that actually, two, three years from now, if we're doing this episode, you're like, well, actually, the optimizations that matter are like these five things that we didn't talk about?

• 8:01Speaker 01

  Right. So I would say if you're a startup, you have to make a bet. You invest, so you know this, and you have to make an outsized bet. So you could bet that, hey, maybe chatbots will go away, and the thing that people really, really care about is, I don't know, video model, video generation model, or world model, or robotics, or something. And you make that bet, and you say, okay, that's probably going to be, I don't know, 50% of the workload. How would we design for that workload? And you hope that your bet will be correct. I mean, that's the role of startup. And I think that's probably The way you could make an impact, I think if you just, if you don't place a bet and you say,

• 8:43Speaker 01

  I'm just gonna optimize for just the general workload, then the incumbents are gonna just out execute you.

• 8:51Speaker 00

  I was excited we're doing this episode today because yesterday I was just scrolling Twitter as I'm about to do and I thought there was this really interesting tweet from Semi Analysis and like reaction to some of your recent work. So obviously you've done so much on optimizing models on Nvidia chips and I feel like, you know, one could try and quantify the value you've added to the broader Nvidia ecosystem, but it's clearly like massive. And I think the Semi Analysis folks were like, God, like you've added so much value to this ecosystem. Well, why wouldn't AMD or, you know, Google or some of the other folks that have chips, like, try to pay you to get you to do similar optimizations in those space? And it kind of, you know, obviously it reminded me of the LLM world where, you know, you've seen kind of massive packages for, you know, the few researchers that are really able to build and train these models. I wonder what your reaction was to that tweet and generally what

• 9:34Speaker 00

  you think about like similar dynamics in the hardware space?

• 9:37Speaker 01

  Yeah, yeah. I personally work with a lot of, lots of folks from Nvidia, AMD, Google, and Amazon and so on. I spend a lot of my time on Nvidia chips simply because, you know, this is what we have right now. That's what most people use. I think they're designed very good chips and they design very good software for that. So that allows me to sort of do interesting things. You know, that's kind of what I'm after. Like, can you do interesting things? We've worked with the folks at AMD on, for example, they had a version of flash attention and, you know, we worked with them to sort of integrate that in the public repo and so on. So certainly we work with some of them. I don't know what

• 10:21Speaker 01

  the best arrangement would be, but nowadays I'm thinking a lot more about like what are the abstractions we need, not just for Nvidia chips, but for sort of GPUs and accelerators in general. So I spent a bunch of time at a lowest level trying to figure out, get the maximum performance out of these chips. But as we You know, scale at together, we think about, oh, how do we get other people that we we we hire to get productive? Right. And part of it is building abstractions that will work on Nvidia chips and potentially other chips as well.

• 11:01Speaker 01

  The other thing I'm excited about is how do what kind of abstractions do we build so that AI can can do some of this work for us? And and I think the the the answer isn't quite clear.

• 11:13Speaker 00

  Yeah.

• 11:13Speaker 01

  Yet, but I think as as as you know, as humans or as as a technical leaders, like it's about building the right abstraction so that other people can get on boarded really, really quickly, so that, you know, the the kind of things you you do could work across different kinds of chips or different kinds of workload.

• 11:30Speaker 00

  Do you think we're at a place where there are abstractions that do allow, you know, you to do work across the different set of chips out there?

• 11:36Speaker 01

  I think we have some, right? But this this is a usual trade off, right? I, you know, Triton is really, really good and it's, you know, it supports Nvidia chips, supports AMD GPU and Intel GPU and so on. And that requires them to design sort of a front end and then there's a back end and different companies will contribute code to the to the back end. I think Triton is actually quite good and some a lot of companies are betting on that, you know, Meta, for example, PyTorch, Torch compiler will just generate Triton code.

• 12:06Speaker 00

  Yeah.

• 12:07Speaker 01

  And then rely on on Triton to generate code for Nvidia or AMD. But that's a usual trade off, which is, hey, if you don't control at the lowest level, maybe you give up some performance. And it's about like where that where that trade off point is, you know, if you give up 5% performance and you you become, you know, 3X more productively, for sure, that's, yeah, that's an acceptable trade off. But if you give up too much performance, then maybe people would go with something that's a little bit more.

• 12:37Speaker 00

  Yeah, especially a pretty competitive inference market.

• 12:39Speaker 01

  Right. So I'd say designing for humans is is really, really hard. I think I would say hardware portability is kind of a myth. Simply, you know, even for our Nvidia chips, like generation to generation, they change a lot. And I think that's kind of the only way we get more performance. Out of these chips, it's not like a CPU where, oh, year to year, maybe you get a five to 10% faster. And you know, your code, your old code just kind of works. Even for Nvidia, every generation, they essentially have to rewrite all the code.

• 13:13Speaker 00

  Yeah.

• 13:13Speaker 01

  Because you know, the way to get more flops is you, you know, build more specialized components, maybe lower precision, maybe a different way to synchronize this in different parts of the chips, right? So there's Nvidia also essentially rewrite their software every two years or so. So even part of hardware portability, even between generations, isn't quite there.

• 13:35Speaker 00

  So those abstractions will be valuable even just to help you with each subsequent generation of chips from the same manufacturer.

• 13:41Speaker 01

  Right, yeah, so I think Trident has a very compelling abstractions. And they even have something that's a little bit lower level, things like, I think this is Super Neos, GlueOn, where it exposes more of the of the hardware at the cost of maybe being less, less general, right? The folks at Modular has been building this language called Mojo.

• 14:00Speaker 00

  Yeah, what do you think of what they're doing?

• 14:02Speaker 01

  I think it's super cool, like, I think they have some of the right abstractions. It's just about execution, I would say, because you know, people are going to look at it and, and they're going to be like, okay, how fast are you on Nvidia chips? Which is a really sort of, in some sense, unfair question, but you know, that's what, that's what people care about. So they would design the abstractions and then they have to do like some of, some custom work to make it, to make their code run really, really well on Nvidia chips. And then they do a little bit of custom work on, for AMD chips, right? So it's just like, how much do you want to do the customization so for just trading off performance?

• 14:41Speaker 00

  See that market demands like the best inference on Nvidia chips.

• 14:45Speaker 01

  And we're seeing more and more of these sort of libraries or domain specific language. You know, folks at Stanford have the same Thunder Kittens, sort of trying to abstract some of this. This Tylang, their, Google has the same Mosaic GPU, I'm sure I'm forgetting some, right? But I think people are, people realize that there's a Problem of like, we don't have the right abstractions yet.

• 15:08Speaker 00

  Yeah.

• 15:08Speaker 01

  Is actually kind of painful to to train new engineers to write very performant GPU kernels. So the answer is to build abstractions. And I think we're very much in the phase where we're iterating pretty, pretty quickly. That's why we see so many of these domain-specific language coming out, right? And then as the AI model is getting better, one of the things I'm thinking about is like, how do we design for? How do we design domain-specific language or abstractions for language models? Because they operate somewhat differently from from humans and we don't know what the the right answer is. So I would say in the next one or two years, I think it's going to become a lot clearer.

• 15:51Speaker 01

  Right now, it's just, yeah, everybody's trying lots of lots of different directions.

• 15:57Speaker 00

  Where do you think those abstractions are most likely to come from?

• 15:59Speaker 01

  Yeah, I think this abstraction, I would say people approach it from two angles. One is from sort of the machine learning side. They think about, OK, what kind of workload do we have? And what are the primitives that are necessary to express those workloads?

• 16:16Speaker 00

  Yeah.

• 16:16Speaker 01

  You know, a lot of workloads, let's say inference is very much, how can I move memory as fast as possible? You know, inference is usually memory-bound, right? Or how do I do matrix multiply as fast as possible? And what are the primitives I want to express there? The other angle is people thinking sort of from the hardware side. You know, they have really cool sort of specialized components on the chips. And they're thinking about abstractions of how to expose these abstractions. So one of the things Nvidia has been really good at is this sort of designing chips that are more asynchronous. Because, you know, the matrix multiplication is getting so fast, it's ridiculous, that everything else becomes much slower.

• 16:57Speaker 01

  So it's like become, it's much, much more important to sort of overlap the computation of matrix multiplication and everything else. So how do you design abstractions that allow this kind of asynchronous execution? Things like they have the things, you know, pipelines and synchronizations and so on. So I'd say that's where the abstractions are going to come from, either from the workload side or from the hardware side. I think it will become a lot clearer in. Maybe one or two years.

• 17:24Speaker 00

  Well, you've alluded to it a few times, so I have to ask you, you talked about kind of making these abstractions suitable for LLMs, maybe, as well as kind of just using AI in these processes themselves. How much are you using AI itself in determining this stuff? And how do you kind of see that changing over the next few years?

• 17:39Speaker 01

  Yeah, yeah. I think models are starting to become useful for this stuff, actually, very, very recently. This really surprised me. On one level, what people are after is like fully automatic GPU kernel writing, right? You describe the problem and the LM will just generate the kernel for you. This is kind of, maybe we're there with some of the other areas, like, I don't know, simple Python script, data analysis, or web front end, right? You can do this, right? So could we do that for GPU kernel?

• 18:17Speaker 00

  Just five codes, some kernels.

• 18:18Speaker 01

  Right, right. I would say if that's what you want, then I think we're not, we're very, very early. So these models can generate some simple kernels, like element-wise, you know, you take in an array and you do some operation on each element, right? Or you do some kind of reduction, like summing and some normalization, things like that. So the models can do reasonably good job. But once it gets a little bit more complicated, these models just don't generate correct code. I think it's just a function of, we don't have enough training data. Training data is really, really tough, this stuff, because if you scrape the internet for kernel code, you'll get, I don't know, class projects, you'll get documentations

• 19:03Speaker 01

  that were meant for GPUs maybe three generations ago, that, you know, these are things you shouldn't be doing now, kind of thing. So training data is really, really difficult. So I think the answer to that is you probably have to start with some expert-level data, and then you generate synthetic data out of that, or hooking up to a tool like compiler and profiler, so that you can get lots of training data, or getting the right environment, right? I think it will be solved in a year or two, but it's certainly a difficult problem.

• 19:36Speaker 00

  Who has that data?

• 19:37Speaker 01

  I don't think, this kind of data, I don't think is... Private. There are a couple of places where you have expert level code. But I think it's more about sort of the workflow. How do you start from a small amount of expert level data and generate lots of synthetic data, right? So some of the, I think the folks at this Discord GPU mode, they've been really trying to do this. So they took compiler, the PyTorch compiler, would generate, you know, go from PyTorch code to Triton code, which is this lower level kernel code. And they could generate, I don't know, I think 15,000 pairs of these program of PyTorch and Triton,

• 20:21Speaker 01

  right? Like, you have to be a little bit creative. I think, you know, there's just not that much data on the internet. So you have to be a little bit creative of how are you going to generate this kind of data. So I'd say that's, that's, that's one angle. If you want fully automatic kernel generation, I'd say we're super early there.

• 20:38Speaker 00

  Yeah.

• 20:39Speaker 01

  The other side is, can it work alongside humans? Can, can models work alongside humans? And I've been pleasantly surprised that these models are actually quite useful.

• 20:48Speaker 00

  Yeah. Was there a specific moment where you're like, whoa, these models have gotten good enough to be quite helpful to me?

• 20:52Speaker 01

  Yeah, yeah. I would say maybe there were two recent milestones. One was O3. O3 has gotten really good at reasoning. You know, some of the questions I was, you know, I was brainstorming with O3 and GPT-5. I was like, hey, I have this function. Like, how would I optimize it? What are the things I would pay attention to? And it's like surprisingly good at the high level. The other one is Claude code. Somehow, it's actually pretty decent at writing Triton kernels, which, which is fantastic. Like, I, you know, as much as I love writing kernels, like, a lot of my time is thinking about the design, thinking about, oh, what kind of architecture should we design so that we can take advantage of the hardware?

• 21:35Speaker 01

  And so the implementation part of it is, you know, design is really fun, but the implementation is usually quite heavy.

• 21:42Speaker 00

  Yeah.

• 21:42Speaker 01

  Right? And Claude code turned out to be quite helpful here. Like, I would say, you know, it makes me maybe 1.5x more productive.

• 21:49Speaker 00

  Wow. I mean, that's, that's, that's pretty good.

• 21:50Speaker 01

  Yeah. So I've been a heavy user of Claude Code. If you have these models working alongside with humans, maybe they are a lot more helpful rather than just relying on them fully automatically generating.

• 22:03Speaker 00

  Yeah. What are the next like milestones you're waiting for? I guess when like a new model comes out, are there things you test that you're like, God, if the models could just get to this stage, you know, I'd go from 1.5 to 2X.

• 22:12Speaker 01

  Right. So I would say the, I think Claude Code is a good example of a kind of a step change where it's a lot more agentic, right? And somehow, you know, they post-trained Claude to do really well there. And I'm sure, you know, other folks, OpenAI and Google will soon get to a very, very similar point. So agentic here just means it can use tools really, really well and it knows when to use the tools. So it knows that, hey, oh, I'm probably not having the right API. So how do I look up the API? Or, oh, the program is not compiling, or the program is not fast. How do I get information from the profiler, that sort of thing, right?

• 22:55Speaker 01

  So I think I would say for new models, I would see how, how well they, they, they know whether they don't know.

• 23:06Speaker 00

  Yeah.

• 23:07Speaker 01

  Right. Like when do they need to seek out new information? And that's kind of a vague thing. I think people are starting to come out with benchmarks for this kind of agentic capabilities, but we're still super early there.

• 23:20Speaker 00

  You've been sort of seminal part of like so many of the improvements in the AI inference market since, you know, the launch of ChatGPT and the, you know, broader usage of a lot of these models. So I figured, you know, maybe just for our listeners, would be great to contextualize like these last three years and kind of what's driven so much of the improvements on the, on the kind of lowering costs and improving latency side? And I want to make sure to give you the opportunity to talk about a lot of the flash attention work as well.

• 23:42Speaker 01

  For sure, for sure. Yeah. I think in the last couple of years, inference costs has probably will come down maybe 100X.

• 23:48Speaker 00

  Yeah, at least.

• 23:49Speaker 01

  Since ChatGPT debut, I think.

• 23:56Speaker 01

  It's probably reflected in the API cost as well. So on one side, on the modeling, people have just trained much, much better models for the same sort of number of parameters, right? And part of it is much more data. Part of it is, I think, better architecture, right? MOE certainly helps. People invent new sorts of efficient attention and so on, right? So that's on the modeling side. Models getting much, much better for the same number of parameters. The other side is on the inference optimization. And we've just been seeing like an explosion of techniques there, right?

• 24:38Speaker 01

  I think in the early days, we didn't, as a whole, we didn't understand what are the bottlenecks of inference. And after, after, why we, you know, people realize, hey, it's a lot about data movement, you know, moving the weights in and out of the memory, moving the KV cache, which is the part of the attention operation that you need to store the history, entire history to sort of make the next prediction, right? So a lot of optimizations are around how do we reduce the data movement? So things like you could quantize the model, right? So you use, instead of typically, I think maybe two, three years ago, it was typical

• 25:21Speaker 01

  to use like 16 bits per parameter. But now people are doing, eight bits is pretty common now, and some new models are using four bits. There's some work using like one or two bits, which is, which is pretty crazy. But you see this really, this compelling trade off, you know, as you quantize the model, a lot of times you pretty much don't lose any quality. And that requires fairly sophisticated techniques, but it has worked amazingly well, right? So I think the most recent sort of OpenAI release, GPT-OS, they released. Most of them, most of the layers are in four bits, right? Which is, which is crazy. They have, I think total was 120 billion parameters.

• 26:04Speaker 01

  But you, because each is using four bits, you could fit that in like 60 gigabytes or something. Right? And that has translate to really, really good inference performance. So quantization is, is one. The one, the other one is like this co-design of model architecture and the hardware. As, as people understand much, much better, essentially like people on both sides are talking to each other a lot more. Right? I think flash attention was kind of like that, where there were people who understand the hardware really well. There were people who understand the algorithm and the model really, really well.

• 26:42Speaker 01

  But as you combine these ideas, you start to see, oh, I see exactly what the bottleneck on the hardware side is. So how do I change the, the algorithm? And for flash attention, it was like, we realized memory access was the, was the main bottleneck. And we figured out sort of a new way to rewrite the attention algorithm to reduce memory access. Right? And we're, we're seeing that a lot for, for inference. So an example is this variant of attention called, from DeepSeek called multi-head latent attention. Right? Where they realized, hey, a lot of time is being spent on compressing, moving the, the KV cache, which is a summary of the history. Right? And they realized, hey, we can make the KV cache much, much smaller by, they, they call it this, this latent

• 27:27Speaker 01

  projection, where you project on, on a much smaller space. Right? And that turns out to work really, really well. And DeepSeek, I think in the paper, they, they said they were able to serve this model very, very efficiently. Yeah. So that's one example. The other one is mixture of experts, where you don't, you know, for each query, for each token, you don't use all of the parameters in, in the model. There's some sparsity. So you use some, some part of the model. And in the last couple of years, I think the trend has been making the models sparser and sparser.

• 27:58Speaker 00

  Yeah.

• 27:58Speaker 01

  Right? I think for the first open source M L E model from Mistral was maybe a year, two years ago, they activate two experts out of eight experts per layer.

• 28:12Speaker 00

  Yeah.

• 28:12Speaker 01

  Right. So it's a ratio of 25%. But now for DeepSeek and OpenAI model, you activate, I think for OpenAI G P T O S S, it was four out of 128 or something, right?

• 28:23Speaker 00

  Yeah.

• 28:24Speaker 01

  It's a factor of 32 and we're getting sparser and sparser. So that's really, really good for if you if you serve lots and lots of users.

• 28:33Speaker 00

  Yeah.

• 28:33Speaker 01

  So people are really understanding the workload. They're really co-designing the architecture with the with the inference. And I would say that's where a lot of the a lot of improvement has come from.

• 28:45Speaker 00

  Yes.

• 28:45Speaker 01

  Is this this co-design between model and inference stack.

• 28:50Speaker 00

  And then looking forward, like, where do you think it's going to continue to come from?

• 28:52Speaker 01

  Right. So I think there's got to be another 10x or so.

• 28:57Speaker 00

  All right.

• 28:58Speaker 01

  I mean, if you're just.

• 28:59Speaker 00

  Seems to be the trend so far.

• 29:00Speaker 01

  Right, extrapolating.

• 29:02Speaker 00

  I just didn't know if we'd picked off the low hanging fruit, right? Or if they're still.

• 29:05Speaker 01

  Right. I think it's, you know, we've done, as a whole, we've picked up a lot of low hanging fruits for sure. But I think there's still lots to do. One is actually on the hardware side, which is for a while, because you couldn't really predict what the workload is going to be two years from now.

• 29:25Speaker 00

  Yeah.

• 29:25Speaker 01

  So you can specialize very well. But as you, as the, as the architecture has gotten a little bit more stable, I think people designing chips are optimizing for inference now.

• 29:40Speaker 00

  Yeah.

• 29:40Speaker 01

  Right. So, you know, low precision, get really good native hardware support.

• 29:46Speaker 01

  All the networking gets really good because as the model gets larger, you know, things like mixture of experts, the models get larger, but you don't activate all the parameters. You might need multiple GPUs or chips to store the model and then networking becomes much more important.

• 30:00Speaker 00

  Right.

• 30:00Speaker 01

  So I think on the hardware side, it would probably be two, three X just in one year.

• 30:06Speaker 01

  And on the software side, I think a couple of things. One is pushing on the model architecture. So I've worked on things like Mamba. And it's a way to, instead of storing the entire history as a KV cache, you could have the model compress that history into a smaller state vector. Of course, that's going to have certain trade offs and so on. But we've seen pretty good success with this. And I think this is going to be especially important for large batch inference, which is what people are, some of the workloads are demanding these days, where things like test and compute and reasoning, like what you really, really want

• 30:50Speaker 01

  is you want the model to explore lots of trajectories or chains of thought at the same time. I think Google came up with this, a Gemini Deep Think, I think, which that was the idea. I mean, they were winning IMO gold medals and so on, which is they explore multiple paths at the same time. And what that means is that you're doing model inference on like hundreds of sequences at the same time. And there, the KV cache become a much, much bigger problem. So you could try and get the model to sort of compress that KV cache. That's, that was sort of the direction we were going with with Mamba. And there's been a bunch of other variants coming out along that idea.

• 31:32Speaker 01

  So I think on the model side, it's going to be 2, 3X.

• 31:36Speaker 00

  Yeah.

• 31:37Speaker 01

  Right. And then on a just like kernel level, people, lots of people are getting really interested in the kernel side and some of them are very, very talented. So we're getting very, very good kernels. And that's probably another 2X or so. So taken together, I think even just probably just one year, we would probably get another 10X.

• 31:58Speaker 00

  One thing I'm struck by is obviously with this broader like diversifying of use cases and how people are, you know, what they want to run inference on and the architecture of these models, it's, you know, it seems like it's kind of expanded into, you know, there's many different end use cases and the bottleneck of each of those or the optimization you want to make each of those are different depending on.

• 32:15Speaker 01

  Exactly.

• 32:16Speaker 00

  And so I guess, what are the implications of that then, maybe to bring it back together as like an inference provider? Obviously, you want to be the best at all of those things.

• 32:23Speaker 01

  Yeah.

• 32:24Speaker 00

  Do you think that like the ecosystem, you know, remains with one vendor who is, you know, really good at all those things? Or do you see like specialization over time where it's like, oh, hey, cool. If you're, if you're, you know, trying to run a, you know, deep research, like there's one inference provider that's amazing at that. And if you're trying to run like more of a chatbot use case, uh, how do you see that kind of playing out?

• 32:41Speaker 01

  Right, yeah, I, I, I think, uh, there will probably be, yeah, maybe three kinds of, of, of workload patterns. And I think all inference providers will understand this and they will optimize for it. But there's certain, uh, you know, advantages to, uh, running at, at scale, right? So, um, so on the workload side, you know, there's the traditional, uh, chatbot. Where, you know, you need a mix of, uh, of interactivity, you know, so you, you can't be too slow. But you also don't need to be super fast.

• 33:18Speaker 00

  Yeah, sometimes it's like a little creepy if it comes like right away. You like, you want, you want to like feel like for the experience that there's someone at least to be a little bit of thinky on the other end.

• 33:24Speaker 01

  Right, right. So there's that kind of workload. There's the kind of like really low latency requirements to things like, um, I spent a lot of time with Claude code. So I always wish, hey, if this is like three X faster or five X faster, I'm certainly willing to pay more.

• 33:41Speaker 00

  Yeah.

• 33:41Speaker 01

  Right? Um, and so we'll see more of that where it's kind of like very interactive. Where if, if the model is two, three X, or if the inference is two, three X faster, uh, the person working with the model is going to be, you know, two X more.

• 33:54Speaker 00

  Yeah, it's different between like staying in flow state and not.

• 33:56Speaker 01

  Right, right, right. I mean, people are getting around this problem by, uh, running multiple of, of these models. So, uh, you know, having like four Claude codes running at the same time, that kind of thing. But, uh, for someone who, I, I personally prefer, uh, sort of, uh, deep work. And when I work with, with model, I usually just use one, which, uh, my partner yells at me for. She's like, you should be using, uh, four Claude codes at the, at the same time. But for, for this kind of workload, people, uh, Probably will be willing to pay more for this this low latency. So there's the that's the low latency sort of agentic kind of workload. And then there is the very large batch. I don't care so much about latency, but I just want as high throughput as possible.

• 34:41Speaker 01

  So this is important for things like generating synthetic data. So as I mentioned, a lot of the the way people, you know, the way people are training models now is that they have a small amount of expert level data or the human annotation data. Right? I don't know. Let's say you're you're in airlines and you're trying to get a build an AI agent to like resolve customer complaints or conflicts and so on. They have a small amount of really, really good data. And then you can generate tons of synthetic data out of that. It's kind of crazy how good the models are at at acting like humans. You can say, hey, you know, if can you act as if, you know, you're you're this kind

• 35:24Speaker 01

  of, you know, this customer from New York who is annoyed at that that their flight is delayed out of LaGuardia. I don't know. All right. And models are surprisingly good at acting as as humans.

• 35:36Speaker 00

  Yeah, I'm scared that we've taught them the capability to be angry New Yorkers at an airport. That's not like humanity at its finest.

• 35:40Speaker 01

  Right, yeah. Yeah, but somehow there's, you know, lots of that data on the internet where where the model has has learned that, right? And and the model can can then use that. I mean, they have this kind of world model internally that they can then generate lots of data that, you know, probably not as good as as human data, but you can generate a massive amount, right? And for that kind of use case, inference use case, you really only care about throughput.

• 36:05Speaker 00

  Yeah.

• 36:05Speaker 01

  Right? The other one is for this RL training kind of use case.

• 36:09Speaker 00

  Yeah.

• 36:09Speaker 01

  Where on one hand, you're training an agent to do something, you know, change his policy. But the way part of it, part of the the training loop is that as the the model has some policy, you need to evaluate how good that policy is, right? So let's say I'm, you know, I'm training an AI engineer. How would I know the current, how good the current AI engineer is? And that requires me to. Sort of sample from the model and it's called rollout. So you sample, you know, tons of completions from the model and then evaluate how good that is. Right. And so as part of this training, you really need very, very large batch, high throughput inference.

• 36:48Speaker 00

  Yeah.

• 36:48Speaker 01

  So I would say that's the third use case, very large batch.

• 36:53Speaker 00

  Yeah.

• 36:53Speaker 01

  And so for these three use cases, I think people are starting to recognize these, these patterns. And as instrument providers, we certainly do sort of different optimizations.

• 37:02Speaker 00

  How do you think about allocating your resources across the three? Because obviously I'm sure there's different optimizations that are, that are focused.

• 37:08Speaker 01

  Right. So, so I would say this is where maybe running at scale really helps. So we call it fleet level optimization, where if you're running on inference on, on thousands of, of GPUs, you can sort of dynamically change the, the cluster allocation. So as an example, this very easy, easy, simple example is running batch inference, batch API. And OpenAI has an, has an option for that. We also have an option for that where if we see that the cluster is not so busy with these more interactive queries, we can just send in these, these batch queries that can sort of soak up the, the compute, right?

• 37:53Speaker 01

  And as a result, we can provide, I think we give like 50% discount on, on batch API. And I think OpenAI does, does the same, right?

• 38:01Speaker 00

  Right.

• 38:02Speaker 01

  DeepSeek probably does that too.

• 38:03Speaker 00

  Having it at scale really, really helps.

• 38:04Speaker 01

  Right, right.

• 38:05Speaker 00

  As you, as you think about kind of like the evolution of the inference market, does it feel like there's endless optimizations to do over time? And so it's always possible to stay a little bit ahead of the, of the rest of the players? Or at some point have the low hanging fruit been plucked on each of these? And it's actually about building a broader platform to do other things on top of these workloads?

• 38:23Speaker 01

  Early on, there were lots of low hanging fruits. And if you write reasonable kernels, if you build a reasonable inference engine, you, you get much, much better results than what was out there.

• 38:34Speaker 00

  Yeah.

• 38:34Speaker 01

  But now I think the open source tooling is getting very, very good.

• 38:37Speaker 00

  Yeah.

• 38:37Speaker 01

  Right. Things like VLM and SG Lang. Some of these projects are very, very popular and they've gotten to a production level quality. Right, and we certainly, you know, work with with these folks and contribute and so on. So I think just the baseline has gotten so much better, right? But there's also just like new and newer newer use cases. So customers come to us and they say, oh, we really care about, you know, prefix caching. I really care about low latency because in our application, we we need this requirement. All right, or like, hey, we're actually not doing text. We're doing video and that's sort of a different trade off, you know, much more throughput oriented. And we work with some of those customers.

• 39:22Speaker 01

  So even as the open source tooling, the libraries are getting so much better. I think the the workloads are evolving so fast that that there's always sort of new stuff to do. Right, and there's this kind of explosion of like, the models are getting so good now that there's so many ways to extract value out of these models. That's why we see so many startups building on on these models. And as a result, the workload is going to evolve pretty rapidly.

• 39:51Speaker 00

  So you think obviously we have these like three kind of main categories of workload. Say, do you feel like fast forwarding a year or two from now, it's like 27 or like, you know, a ton of different ways and each requiring its own kind of optimizations?

• 40:00Speaker 01

  I think it's still gonna coalesce, right?

• 40:05Speaker 01

  I think the kind of agentic workload might be a killer use case.

• 40:09Speaker 00

  Yeah.

• 40:09Speaker 01

  I I think it's maybe a step change for maybe, you know, chat GPT was a step change in terms of application, right? That's that was sort of the first time users are exposed to this kind of language models. They they can talk to and maybe, you know, solve their, help them debug their code or or, you know, look up informations and analyze information and synthesize information. But I think the the next set of application is going to be agentic, which is, hey, can AI, can these models sort of take actions and collect information by themselves, right? And that is going to require sort of a different sets of optimizations, right?

• 40:52Speaker 01

  So now it's not just running the models really, really fast on GPUs. It's like, how do you connect with the the tools that humans usually do?

• 41:01Speaker 00

  Yeah.

• 41:02Speaker 01

  You know, web search is an easy example, but as as these workloads move to different verticals, let's say you're an engineer, I don't know, you're designing airplanes or something, right? You you want the model to be able to like, Access your design software.

• 41:18Speaker 00

  Right.

• 41:18Speaker 01

  Right?

• 41:20Speaker 01

  Or if you're a financial analyst, you have certain database that you want the model to tap into. Right? So I think that's this kind of workload will be one of the main workloads that we have in the next year or so. That's my prediction.

• 41:39Speaker 00

  No, and that systems level work around agents then obviously becomes a whole new set of problems to solve.

• 41:43Speaker 01

  Right.

• 41:43Speaker 00

  And so it feels like even if the original set of workloads, you know, there's great open source tooling, there's now entirely new set of workloads that also require.

• 41:51Speaker 01

  Exactly.

• 41:51Speaker 00

  You know, optimizations and getting into some of that systems work and accessing like, you know, external databases and other things is different than just optimizing inference, but it seems like a natural evolution of the product you guys are providing.

• 42:02Speaker 01

  Yeah, yeah. I think that's the sort of the main, going to be the main use case for more like professional level or enterprise level. I think on the consumer level, you know, one of the, I think one of my bets is like we'll get real time video generation.

• 42:17Speaker 00

  Yeah.

• 42:18Speaker 01

  And we're starting to see some of that. And I think it's going to really change sort of the consumer landscape to the effect of like TikTok really changed that landscape, right? You know, for better or for worse, I don't know. But it's very, very engaging, right? And so on the consumer side, I think, you know, some of the, some of the companies we work with like Pika Labs and Hydra, they're focusing on real time video generation. That's the bet, right?

• 42:48Speaker 00

  There's a whole new set of problems to solve.

• 42:49Speaker 01

  Right, right. And so it's just a lot more compute. Like video generation is very, very demanding on the compute side. So that might be fueling even more chips, even more inference optimization.

• 43:03Speaker 00

  Super interesting. I mean, obviously you're, you know, conducting a lot of research and making these bets ahead of, you know, there's all these big outstanding questions where the space is headed. I'm curious if like, if you could, you know, fast forward to the future, three years from now and get the answer to like one question in the AI infrastructure world that would really help you shape your direction today. Like what's that question that's outstanding that you're most curious to know or would most impact the direction of your work?

• 43:24Speaker 01

  The question I'm trying to answer for the next couple of years is how do we get AI to expert level? So right now, I think the models are at the sort of the median human level on some of the tasks like front-end programming, they've gotten quite good.

• 43:43Speaker 00

  Yeah.

• 43:43Speaker 01

  Actually, the models are much better than me at front-end programming for sure. Or analyzing data, that kind of task where there's lots of data on the internet. Whenever there's lots of data on the internet, these models are gonna just crush it. Right? So they've gotten to sort of the median or maybe slightly above average level on some of these tasks. But I think the economically valuable tasks are still there, right? We pay lots of money for human experts in, I don't know, designing airplanes or designing hardware or doctors and lawyers and so on, right? Those people become expert because they've spent time working with specialized tools.

• 44:28Speaker 00

  They don't have an internet worth of data that is why they're experts.

• 44:31Speaker 01

  Right. So how can we get models to get to that level to work alongside with humans at that level? I think that's where a lot of the economic value is gonna come from.

• 44:43Speaker 00

  And obviously then the hardware organizations you have to do around whatever the way to do that is. You've worked on state space models, you've spent a lot of time thinking about alternative architectures, transformers. You know, I think your collaborator Albert who said that transformers are not gonna be the final solution by themselves. Like, do you think we need architectural innovation to get us to that level?

• 45:02Speaker 01

  Right. I love Albert, right? But maybe this is where we disagree a little bit, right? So I think to get to AGI or ASI, it's possible that the current architecture we have is sufficient, right? But at what cost? Which is, hey, if you have a better architecture, maybe you get there, I don't know, one or two years sooner.

• 45:26Speaker 00

  Yeah.

• 45:27Speaker 01

  I think that's probably worth it. Or you get there with 10x less of the cost, right?

• 45:33Speaker 00

  Which ultimately flows to then the accessibility of that ASI when you get there.

• 45:36Speaker 01

  Right, right. And that's that's certainly crucial because, you know, we're spending, I don't know, $500 billion in AI infrastructure every year. Don't quote me on that, but I think like that's the sort of the right ballpark. Yeah. Right. So do we need to spend 10x that, which seems somewhat unrealistic? Or with better architecture, can we get there with the, you know, the current amount of spending or maybe even less? So I think that's where the the modern architecture is going to play a role, which is can we get to AGI with a better architecture? I think the current architectures sort of have all the right ingredients.

• 46:14Speaker 01

  And if you keep scaling, which people have been doing, you could get there, but maybe, you know, the cost is just a little, is like astronomical. What other architectures are you paying attention to? Right. So I'm pretty excited about a mixture of experts, especially getting sparser and sparser. We're trying to push that limit, like what, how sparse can you go? And we're working on that. I think it's been a compelling direction. Some some really important work done by DeepSeek, showing that you can make these things really, really sparse. There were even earlier work from from DeepMind pushing in that direction as well.

• 46:54Speaker 01

  So I think that's one compelling way to sort of get more intelligence out for the same compute. And ultimately, I think we want to optimize inference per dollar. Yeah. Right. And that means is it could be factored into inference per flop, floating point operation, and flops per dollar, which is inference per flop is is more like architecture, design, data, algorithms. And then the other side, flops per dollar is like hardware and kernel optimization. So on the architecture side, we're trying to extract as much information, sorry, as much intelligence out of the same computation. Yeah. So, you know, mixture of expert is one. I think some of the state space stuff done with with Albert has been really, really fun.

• 47:40Speaker 01

  And, you know, people are picking that up, right? We've collaborated with a bunch of folks from Nvidia. To training models, they release several of these where they show, hey, they can get really, really high quality model using this kind of architecture. You know, it's a hybrid of transformer and Mamba and so on, but with a much smaller cost or much, much higher inference performance. Right. So I think architecture is very important for this, for inference. I think that's, you know, the, the utility of architecture research. And nowadays I think about very much, I call it like inference first architecture design, which is, you know, most of the FOPS are being spent on inference anyway.

• 48:25Speaker 01

  So you really, really want to design architecture that make inference really, really good.

• 48:29Speaker 00

  Well, I'm sure, I mean, obviously you've, you know, you pushed forward, the inference side with flash attention, you, you know, with Mamba, you've, you've been a pioneer in state space models. I'm sure folks are curious, like, what research areas are interesting to you now? Like, where are you spending your time? What's the, what's the next big paper we should expect from you?

• 48:42Speaker 01

  Right. Yeah, I'm still certainly working in, in, in these areas. I'm so very much interested in these areas, but, you know, I'm exploring some, some new, new directions. Some of which involve, you're working with sort of what are the next set of, of really impactful application? And I think robotics is, is one of them, right? And robotics, like, how far are we away from having really good humanoid robots that work in, in homes? Maybe five years, maybe 10 years?

• 49:15Speaker 00

  Yeah.

• 49:15Speaker 01

  I don't know, right? And, and that's usually means a lot of interesting and exciting questions on the research side that we want to answer. So that's, that's a direction, you know, on the, on the research side.

• 49:27Speaker 00

  Yeah. What's most interesting to you within the robotics research area?

• 49:30Speaker 01

  So, so, so robots, you know, we, we can initialize a model, controlling robot with some of the foundation models that we have already. You know, you can use a language model to do the planning. Like you say, if you ask a robot to like pick, pick up a coffee cup, then the language model can say, you know, go walk over to that table and, and pick up the coffee and so on. But what's missing, I think, is this kind of interacting and acting in, in the world. Right. Cause we just don't have data for that. We have data, language, we have language data on.

• 50:03Speaker 00

  Right. And you've obviously seen some folks trying to scale simulation data, other folks are doing teller, but like clearly data problem on the, on the actual action side.

• 50:09Speaker 01

  Right. So that's, that's certainly one. The other one is I think robots kind of have to process information in a. Sort of multi-resolution, multi-ham scale way. So some of which is, if you're controlling the joints, you have to act very, very, very rapidly.

• 50:27Speaker 00

  Yeah.

• 50:27Speaker 01

  But if you're planning a route for the robot, you can do it much, much slower.

• 50:32Speaker 00

  So it's like different architectures you might want to use. I mean, it seems very analogous to some of the state space model work you've done.

• 50:37Speaker 01

  Right, right. So taking explicit, explicitly taking ham scale into account. Do I want to do very lightweight computation to just control the joints? Or do I want to do much heavier weight sort of reasoning to plan the optimal route for the robot? So I think it's going to be this sort of composite system initialized from language models, vision model, audio model, world model. But how do you stitch them together? I think that's the, that's the big question.

• 51:07Speaker 00

  With some fascinating hardware questions to be able to run this stuff locally and fast enough.

• 51:11Speaker 01

  Yeah, yeah.

• 51:12Speaker 00

  Yeah. I feel like you, you could kind of probably work anywhere you want. And I think one really interesting thing is you're obviously chief scientist over at Together, but you've also gone the academic route as well and you're a assistant professor over at Princeton. How did you think about kind of staying within academia? And then also like the set of problems that it makes sense like in this robotics space? Like you obviously, you could go work at physical intelligence or partner with like some of these robotics companies or do the work from, from an academic setting. How do you think about like what makes sense to work on, you know, within academia versus industry?

• 51:39Speaker 01

  Right, right. Yeah, that's a great question. And my answer is going to be personal, right? For every person is going to be different. So for me personally, I actually like being both in startup and being a professor. It sort of like presents sort of different modes of thinking and execution. On the startup side, it's, it's very fun because we move fast, right? Like, you know, if, if things we want to do, we could get it done in days or weeks or at most months. Like that's the level of, the horizon that we, we plan for, right? And it's very fun. Like you, you, you get a team of really, really good people. They can execute very, very fast and you accomplish great things. And I'm, I'm very, very proud of, of the work that the team has done at Together.

• 52:25Speaker 01

  So that's been very, very fun. On the academia side, I guess it's a little bit longer time horizon and the kind of problems we think about are maybe more speculative. Yeah. Right? We don't solve the problem that, that we need solution in a month. Like, we, we think about, oh, if, if the world is moving in that direction in two, three years, what are the interesting questions to ask and what are the, the challenges to solve in one or two years? Right? And, and we, we, we think in that time scale. So, uh, working with students has been, uh, very fun and, and rewarding as well, um, because we, we get to think, think about these, these questions that are, are interesting, sort of just intrinsically interesting. Right?

• 53:08Speaker 01

  Obviously, uh, there are certain, uh, trade off, like, uh, you know, the amount of compute on, in academia is much, much smaller. The way you evaluate it is, uh, you know, are your ideas interesting rather than, oh, does it, you know, work and run really fast and, and, and so on? You know, that's a different set of evaluation. Right? So in, I think in, in academia is, you have some more freedom to think deeply about, um, these longer horizon problems. Um, and I happen to like both, both modes of, of, of operation. Uh, so that's why, uh, I, uh, you know, I'm still at, at, at Princeton as a, as a professor, I'm still involved in, in, in the startup. Yeah. Right? And I, I think ultimately it's maybe a mix of kind of exploration and exploitation.

• 53:53Speaker 01

  Like, if you think at a macro level, academia's role is more on, on exploration. Right? Um, the, uh, the funding comes from usually the government. The government cares about, can you explore, uh, tons and tons of ideas to see maybe five or 10% of those ideas will, will work out? I mean, as a, as an investor, as a VC, like you, uh, that's kind of the, the, the mode you operate in as well, which is, hey, we're going to explore lots of ideas, uh, maybe five or 10 of percent of these ideas will become. Incredibly important.

• 54:24Speaker 00

  Right.

• 54:24Speaker 01

  You know, one example is attention, right? Attention, you know, even though it got famous with the Google paper, actually came out of the work at Mila with Dimitri and Yoshua Benjo and other folks, right? It was kind of coming out of academia, right? And if you think of all the other sort of components in the current architecture, you know, the optimizer, the Adam optimizer was Jimmy Bar at Toronto and other folks, right? Leonorm, also Jimmy, right? The, essentially, you know, lots of, lots of what we have now came out of academia because the role of academia is to explore lots and lots of ideas.

• 55:08Speaker 00

  Totally.

• 55:08Speaker 01

  And some of them will work, right? And then the role of big companies and startup is they take some of these ideas, they do innovate on them, they execute really, really fast, but they understand sort of the, what the market needs.

• 55:21Speaker 00

  Yeah.

• 55:21Speaker 01

  Right? What's a, what the product they should build. And they have this, they have a much, much bigger sort of financial power to make some of these bigger ideas happen.

• 55:32Speaker 00

  Right. The interesting thing about this funding environment is I feel like it's almost distorting some of that where there's plenty of venture funding available for folks to do kind of more research lab. Like if you, if you pitched, hey, you know, robotics, you know, is going to need different architectures and different hardware, you know, over the years, someone might say, okay, cool, you're not going to monetize for 10 years, but like that's, that's a really interesting idea and here's a bunch of venture money to go out.

• 55:53Speaker 01

  Right, yeah. It's, it's kind of crazy, the, the kind of funding that, that folks are getting, right? Like in Elia SSI, he was like explicitly, we're not going to build any product. But yeah, people want to give him money because he's, he's Elia, for sure. Right? And some of the folks, like you said, in robotics are, are, are doing that. So yeah, I think people are, in AI, it seems like some of these risks are starting to, to pay off. So people, you know, on the investing side, people are much, much more, more willing to bid in money.

• 56:24Speaker 00

  Yeah, totally. Well, this has been a fascinating conversation. We always like to end our interviews with a quick fire round before we get your, your take on some questions. And so maybe to start, what's one thing you've changed your mind on in AI in the last year?

• 56:34Speaker 01

  These models are surprisingly useful even for my daily work at, you know, at the sort of the advanced and expert level. Like they're surprisingly good at the. Math and coding.

• 56:46Speaker 00

  Yeah, 1.5x is way higher than I would have expected. It's pretty impressive. Do you think open source models are going to be closer or further in quality from closed source models in a year?

• 56:57Speaker 01

  I would say closer. The scaling now is more on the RL side and that relies actually more on tooling than just raw compute. So I think open source is going to do really well there.

• 57:11Speaker 00

  What development in kind of the broader AI space do you think people aren't paying enough attention to right now?

• 57:17Speaker 01

  Definitely data. I think data is always a little bit under hype. I think lots have happened on the data side, synthetic data, using models to rephrase that has huge impact that maybe people would pay less attention to.

• 57:30Speaker 00

  Do you have a favorite application that you've seen Built on Together?

• 57:32Speaker 01

  Yeah, we worked with some of these video generation companies like Pika and Hydra. They're generating like viral TikTok videos with the models they train on Together and doing inference on Together. So it's been amazing.

• 57:45Speaker 00

  Yeah, and that feels like poised for a lot of development.

• 57:48Speaker 01

  Right.

• 57:48Speaker 00

  Well, amazing. This has been a fascinating conversation. I want to make sure to leave the last word to you. Where can folks go to learn more about you, either your work at Together, your academic work, anywhere you want to point folks, the mic is yours.

• 57:57Speaker 01

  For sure, yeah. So my work at Together, we put a blog post on together.ai. I'm also on Twitter, tree underscore dao. And you know, once in a while I write blog posts on my website, so tree dao.me.

• 58:11Speaker 00

  Amazing. Well, thanks so much, Tree. This is fascinating.

• 58:13Speaker 01

  Yeah, this has been really fun.

• 58:14Speaker 00

  Thanks.