#281 Leon Song: The Research Driving Next-Gen Open-Source Models (Together AI)

AGNTCY - Unlock agents at scale with an open Internet of Agents. Visit https://agntcy.org/ and add your support. In this episode of Eye on AI, we sit down with Leon Song, VP of Research at Together AI, to explore how open-source models and cutting-edge infrastructure are reshaping the AI landscape. From speculative decoding to FlashAttention and RedPajama, Leon shares how Together AI is building one of the fastest, most cost-efficient AI clouds—helping enterprises fine-tune, deploy, and scale open-source models at the level of GPT-4 and beyond. We dive into Leon's journey from leading DeepSpeed and AI for Science at Microsoft to driving system-level innovation at Together AI. Topics include: The future of open-source vs. closed-source AI models Breakthroughs in speculative decoding for faster inference How Together AI's cloud platform empowers enterprises with data sovereignty and model ownership Why open-source models like DeepSeek R1 and Llama 4 are now rivaling proprietary systems The role of GPUs vs. ASIC accelerators in scaling AI infrastructure Whether you're an AI researcher, enterprise leader, or curious about where generative AI is heading, this conversation reveals the technology and strategy behind one of the most important players in the open-source AI movement. Stay Updated: Craig Smith on X:https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Our mission of supporting open source models gets so much attention. And I think today, this mission is no longer something that people feel like, oh, you know, this may not be that great compared to closed source model. So this really changed when major players started releasing open source models. The platform currently is being used widely across different enterprises. Many companies may have their own models and as closed source models or whatever for their internal business. But we have customers, they use our platform to do other things, right? So the key here is the flexibility that we provide.

• 0:40Craig S. Smith?

  Build the future of multi-agent software with AGNTCY. The AGNTCY is an open source collective building the Internet of Agents. It's a collaborative layer where agents can discover, connect, and work across frameworks. For developers, this means standardized agent discovery tools, seamless protocols for inter-agent communication, and modular components to compose and scale multi-agent workflows.

• 1:17Craig S. Smith?

  Join Crew AI, LangChain, Llama Index, BrowserBase, Cisco, and dozens more. The AGNTCY is dropping code, specs, and services, no strings attached. Build with other engineers who care about high-quality, multi-agent software. Visit AGNTCY.org and add your support. That's A-G-N-T-C-Y dot O R G.

• 1:50Speaker 00

  So my name's Liang Song. I'm the VP of Research at Together AI. I lead the research work that doing the R and D research, I mean, R and D elements of the company that would push out a lot of wonderful, innovative research. You're seeing power in our platform today. Um, you know, ranging from flash attention on on like kernel works like flash attention. Uh, you know, um, data works like red pajama, uh, version one, version two. Uh, we have a lot of, uh, you know, research, open source works on uh speculative decoding. A lot of the work that been used by industry widely like Medusa and others.

• 2:32Speaker 00

  Um, and we also, uh, you know, uh, have, uh, done a lot of the work that providing, uh, modeling and data optimizations to the open source community. We recently released a um, if a small fourteen B uh, uh, coder uh, model, which performs at the level of um, uh, uh, uh, Omini three level. So we just released that last week, uh, with Berkeley. So there's a lot of research in our work that are going, uh, but it's like very, uh, it's it's a product centric, research driven, products uh, centric uh, org that I'm leading. Yeah. So prior to together, I was at Microsoft.

• 3:14Speaker 00

  Um, I I was very fortunate to experience the whole, you know, the G P T, chat G P T era. I'm at Microsoft with OpenAI. I like works, uh, in a training, training system, uh, projects in, uh, in deep speed. Um, and then I created the one of the first initiative on AI for science. Uh, initiatives is, uh, centered around Microsoft, uh, partners. Uh, so using AI to address, um, you know, sci- scientific challenges at large scale. Um, and, uh, uh, and then later on at Microsoft, I was working on, um, you know, different system, machine learning system optimization techniques for inference,

• 3:56Speaker 00

  um, so that we can serve, uh, you know, the G P T series of models on Microsoft platforms.

• 4:03Craig S. Smith?

  Yeah, okay. Uh, can we go back and talk a little bit to begin about, uh, AI for science? Uh, because there's been a lot, do you still follow, I would imagine, what's happening, uh?

• 4:17Craig S. Smith?

  Generally, Anthropic's done some amazing stuff. Yeah, can you talk about where you guys, what you guys were focused on at Microsoft and where you see the field having moved since you left Microsoft and, you know, what you see in the future for that?

• 4:40Speaker 00

  Yeah, I'll talk a little bit about that. I think this is one of the things that, you know, the field is quite excited about.

• 4:46Craig S. Smith?

  Yeah.

• 4:47Speaker 00

  One time, until we get into the generative AI era, like in, in the, in the, like, early 2023 when ChatGPT came out. And then a lot of these models used to be supported by less generative AI models, based models. Where now, like, like, used to be a lot of these kind of diffusion models, or they're trying to figure out how to put different models together to address the complex biology problems, or, you know, simulating weather, climate changes and things like that. So when I was there, we had a great, I think it was a great initiative.

• 5:27Speaker 00

  So we, we started with, I think, two major projects with internal folks from MSR and also external folks. So one project we, we started is to provide this like system support for scaling protein structure prediction training. So you probably have heard AlphaFold from Google. Back then, 2023, there was a open, open sourced community work led by Columbia that called OpenFold, which is using PyTorch Lightning to, with DeepSpeed backend to train

• 6:06Speaker 00

  these types of models. And then so trying to give the community a way of studying, using open source models to study protein structure predictions. So in the past, the problem with that model itself is, is, it requires a large activation during training. So the training.

• 6:27Speaker 00

  Cannot really scale well, um you know, they couldn't do really long contacts window and they couldn't they couldn't do a larger parameter size, uh it's well bounded by the system itself. System, I mean, like, you know, the the hardware devices you're running, the GPUs, the CPUs, the large cloud. Right, so um we stepped in and we provide a lot of important kernels. So that project can now scale and can train, I remember uh uh three times longer of sequences. Uh we reduced the memory uh consumption during training by like seventy five percent or something like that, significant amount. Um and then on that project like took off, now the community, now you guys can use it, it's open source.

• 7:12Speaker 00

  Project, people are using it in academia and then um uh I think there are startups using that as well. Now they formed, I think, uh I think

• 7:22Speaker 00

  consortium actually focusing on that effort. There were um other very important Microsoft uh more internal um efforts that's uh using very creative ways to to do protein structure prediction in a different way. Um and the alpha fold and then the open fold way of doing it. Um you know, the open uh sort of the publications you can find is related to how they're using um a probability prediction way of do like looking at the equilibrium uh in in terms of a more complex uh uh you know, protein structure prediction rather than a more straightforward way that the other uh models are using. So those were um very successful projects.

• 8:04Speaker 00

  The other aspect we're looking at is uh weather uh climate models, weather models. Um that was, there was this very significant impact um at that time. Um I'm pretty sure now it's still the case uh to like the Bing service, because we have a lot of users um are using that through Bing, so yeah.

• 8:25Craig S. Smith?

  Yeah, I I I on the open fold, did you follow the same architecture as Alpha Fold?

• 8:33Speaker 00

  I think it not strictly follow, uh not strictly following that architecture. Uh I think there were a lot of changes, uh even in twenty twenty three that were when we were working on it. Um there, uh I haven't really followed very closely after my, you know, my, my time at Microsoft. But my understanding is they're trying to pull the community, the scientists into this project together and trying to ask them to contribute, uh different model components that, that, you know, you, you can, you can do mix of models, uh which is not just a one transformer architecture, uh you know, dominating, uh model. You can do a different things in that community. I think that's what they're trying to do. Um.

• 9:16Craig S. Smith?

  Yeah, and, and, and one thing that's, uh, I, uh, and I've spoken to DeepMind about open, about AlphaFold. But, um, one thing that I've never, because at the time that I spoke to them, it was still a research project early, uh, early in, in their research and, uh, these predict, um, protein structures, uh, but they're not necessarily a hundred percent accurate.

• 9:51Speaker 00

  Right, there's, um, I think, um, there's always that kind of, uh, verification, validation process needs to happen post, post-training. Uh, you can find the model to be better, uh, these days with, you know, DeepSeek, uh, being very, DeepSeek models being very creative and using, uh, RL type of training strategies to learn, um, learn its mistakes and, uh, uh, you know, uh, and then, uh, do better with the reward signals. So all of these, um, you know, if you think, if you think about protein structure prediction, uh, is it really, um,

• 10:33Speaker 00

  some, some people believe it's, it's very, uh, sort of a, like a restrictive structured, very like limited in terms of scope, what the protein can, you know, the, the amino acid can be, uh, structured in, in certain ways and other ways are not just, not just garbage. But a lot of people feel like if you, if you think it's a very restrictive. A fashion in terms of the scope, then how about make analogy to coding, right? Uh, or uh, math, right? If that's the case, then today's model does that really, really well. Um, using RL training fashion, which, which, you know, uh, can have a much more successful. Uh, uh, uh, rate of the prediction.

• 11:15Speaker 00

  So back then it was not RL based, uh, it was not reasoning model based, uh, a model. It was, uh, was, it was quite simple, uh, what they, what they studied, investigated from off and fold.

• 11:28Craig S. Smith?

  Yeah. And then, uh, because the remarkable thing i- is. Uh, you can use these models to generate novel proteins, right?

• 11:39Speaker 00

  Right.

• 11:40Craig S. Smith?

  Uh, and, and when you do that, then, uh, and maybe this is. You know, this is beyond certainly. Uh, training, uh, a model, but how then do you synthesize a novel protein? I mean, is it? And, and then do you test? Uh, how that novel protein behaves? Uh, to validate the prediction?

• 12:07Speaker 00

  Yeah, I think, uh, that was, that, that was a question a little bit out of my, I'm, you know, sort of.

• 12:13Craig S. Smith?

  Yeah.

• 12:14Speaker 00

  System researcher or computational scientist, that's sort of outside of my domain. Um, but what, what I can say is that, um, people do use these predicted structures and then they do. Uh, post, uh, post, uh, modeling, uh, you know, works like a validation and verification. It, it requires lab validated, uh, you know, uh, to be valid, to be validated in the labs. Um, whether or not this legitimate or not, or, or they're just random compounds. Um, but I, but I do sometimes see news and read some papers and they're saying like, oh, this, you know, this wonderful. Creation of the new, new structure that can mean something, and it's really, really not something we have seen, seen of.

• 12:58Craig S. Smith?

  Yeah.

• 12:58Speaker 00

  You know, that's also amazing.

• 13:00Craig S. Smith?

  Okay, and then jumping to what you're doing it together.

• 13:04Speaker 00

  Yeah.

• 13:04Craig S. Smith?

  Um, that together was started when?

• 13:09Speaker 00

  Um, I believe officially was twen- uh, late twenty twenty two.

• 13:14Craig S. Smith?

  Yeah, and and it was started by Percy, right? Is.

• 13:19Speaker 00

  Um, I think there's several, I mean, since Chris Ray, Percy, and we have uh, uh, T and Wipo, our CEO. I think there are four co-founders and Percy is one of them, yeah.

• 13:32Craig S. Smith?

  Okay, uh, and this latest model that you've come out with, can you talk about that?

• 13:38Speaker 00

  Uh, from us?

• 13:39Craig S. Smith?

  Yeah, yeah.

• 13:40Speaker 00

  Oh, yeah, so we're, I I want to say something like, we're not a modeling company.

• 13:44Craig S. Smith?

  Yeah, I know.

• 13:45Speaker 00

  So we, so, so, uh, we work with other modeling companies to come up with, uh, you know, innovative, um, m- models, sometimes with alternative architectures, not transformers, um, so that, that, uh, and then can serve ac- on actual hardware with certain efficiency, uh, control and also, um, uh, uh, you know, uh, uh, uh, economics. So the recent one is a, um, uh, a lab that in Berkeley, um, where we work with them and then train this fourteen B coder, uh, uh, model.

• 14:24Speaker 00

  This coder model, uh, uh, performs really well and it's based on, um, uh, it's, it's based on this kind of mixture of agents type of, uh, strategy that, uh, uh, you know, uh, or like, uh, and then, um, they train this model with their own data sets and the different training strategies, um, and now the, I think the coder performs really well. It's at the only level.

• 14:50Craig S. Smith?

  And what was Together's contribution of that? And in the lab, the Berkeley lab is, what, what lab is this?

• 14:56Speaker 00

  I I'm not very familiar with this particular project because this external, uh, we have one, two people were working with them, primarily on data side.

• 15:06Craig S. Smith?

  I see, yeah. Uh, okay, well, uh, t- t- tell me about the speculative decoder work. Is that something you're more directly involved in?

• 15:17Speaker 00

  Yeah, um, yeah, I I saw this is about a coverage of Together Innovations on our platform, but I think you you would like to more understand what I do uh in terms of involvement, but that's okay, uh, we can, we can go through uh those topics as well.

• 15:32Craig S. Smith?

  Yeah, and then we'll, we'll cover it together in a, in a minute.

• 15:35Speaker 00

  Sure, sure, no problem. Um, speculative decoding is, is something that um uh Together is really good at. Um, a lot of the companies in the inference provider realm that provide, provide speculative decoding. Uh, speculative decoding is a way that is, is, is a uh non, non-lousy uh way of doing inference that you don't lose accuracy, but you can boost inference performance. Um, so the basic concept is you have a bigger original model as a verifier that verifying a very, very tiny model that's a speculator, a, a speculator model, where if the outcome of the verifier and the speculator agrees, then the inference time will be dramatically reduced.

• 16:20Speaker 00

  In other ways, you would use speculator to perform the inference. Um, so you are uh basically um treating off um a, a lot of the efficiencies in that realm. Um, but the, the, the difficulty of building speculators is you want a very good acceptance rate. If your acceptance rate is very low, then your overall performance will be, still be hindered by the larger model. So you will not really get to the, the peak of the performance that you can gain. So Together um has been working in this uh prior to, I think, prior to our uh my time, I, when I joined was uh early 2003, prior to that, Together already start working on spec, speculative decoding.

• 17:02Speaker 00

  Um, and then produce several open source work to the community, like there's a work called Medusa. And uh we, in, in I Together, we have our own way, our own way of training speculator. Um, we have automatic pipeline we built um that for internally and customers that they can provide uh user data for customers, improve like user data. We have a automatic way of um training that speculator and performing different kind of model optimizations.

• 17:32Craig S. Smith?

  Okay, so I'm gonna slow it down a little bit. What is a speculator?

• 17:37Speaker 00

  It's a smaller model.

• 17:39Craig S. Smith?

  It's a smaller model distilled.

• 17:41Speaker 00

  I will. Uh, it can be, it can be, I'll, I'll give you a couple of examples to clarify. So let's say. You have a six hundred seventy B. Um. A deep seek R one model. It's a large M O E. Reasoning model.

• 17:58Craig S. Smith?

  Right, large mixture of experts.

• 18:00Speaker 00

  You serve, serve this model and you run on multiple nodes. It's gonna be really slow for you. It depends on your um decoding depths. How many tokens are you generating? The decoding process is going to be really.

• 18:15Craig S. Smith?

  Yeah, and just again for listeners, the decoding process, what are you decoding?

• 18:21Speaker 00

  Uh, so it's um auto regressive transformer is auto great, uh, all auto regressive way of generating tokens. So for instance, you have the pre fill phase where you take the prompts into pre fill. And they will generate a token where you start with that token feeding into your decoder. So anything logically coming after that token will be generated, attached back to that token. And you just keep generating the content, context. With the previous uh previous tokens that you're inputting. So that's sort of. Sort of a very easy way of explaining. So what makes transformer so powerful is this auto regressive.

• 19:02Speaker 00

  Decoding process that's that can actually generating the context. Uh, by itself. So that's the wonderfulness of the transformers and all the models that we're using so powerful today. Are based on that. Uh, particular model architecture. Yeah. All the companies are using um this uh model architecture, OpenAI, all the other um company uh modeling companies. Yeah.

• 19:29Craig S. Smith?

  Right, and and then it was, so you have a smaller model.

• 19:33Speaker 00

  Hmm.

• 19:33Craig S. Smith?

  And and then what is the speculator?

• 19:37Speaker 00

  Oh, so I'm trying to get to that. So let's say you have a large model. Um, you, of course, you don't want to perform decoding on the six hundred seventy B model, right? So, uh. Now I can give, I can train a one B model. Model with one billion parameters. And you can use it to pair up with your verifier, which is the larger model to perform speculative decoding. Um, so the smaller model inference speed is going to be really fast because you literally reduce the parameters. You know, size. By orders of magnitude, right? Now you have one B model. Now, how to make this one B model that in terms of prediction capability or acceptance rate. Get very close to the larger model.

• 20:21Speaker 00

  Right? For that smaller model, you can distill from the larger model. Right? Or distill from. Let's say this model providers provide several sizes of. Of a model, you can distill from the medium size or smaller size of model to the size that. That, uh, you know, one B, two B, three B. That you're looking for to support your model. Right? Um, that's what is being, uh. What the speculative model means, which is as much as when you run inference. Your compute is spent on the smaller model rather than the verifier.

• 20:57Craig S. Smith?

  I see. Uh. And, uh, okay, and these, these are, um. Products that you're making available to the open source community. I mean, how does Together AI work?

• 21:13Speaker 00

  Hmm. Together AI is a AI accelerated cloud company that for enterprise. So that's sort of the main thread of business. So we provide a platform like Together platform. That people, um, there are different tiers of services. One of them, the one that's open to the public. Whoever has a credit card, doesn't matter if you're startup company or individuals that you can use. Is to go to our, um, um.

• 21:42Speaker 00

  Serverless endpoint, then you can use all the models that we're offering. Um, open source models, there are now up to like two hundred models that serve on the platform. This has the the models from Meta, from DeepSeek, from Alibaba, like all these models that from different open source providers. And you can use them, uh, just, uh, you can use them is through the API where you can just type, type like in the, uh, uh, through the UI. Or you can use the APIs or other capsule toolings in your own work. You can build applications based on the together service. Now, that's just one tier of the service that we're providing.

• 22:21Speaker 00

  Another tiers are including dedicated instances where, um, let's say, companies, smaller, medium sized companies. They want to reserve GPU resources for doing different things. Um, our goal is trying to provide a entire AI life cycle from pre-training. Um, fine tuning to the point you want to make different service of your applications on our platform. A lot of companies are doing dedicated instances. They will buy on this on demand dedicated instances. When they need to use it, they can use our software to that for whatever business usage they're looking for.

• 23:02Speaker 00

  Um, the key highlight or the key distinguisher of our business. Open source, supporting open source model versus the cold source company. Uh, is that the customers own their own model. They, they own their data, right? Whatever fantastic technologies will provide you to make your model, making your speculator. Making your serving strategy, training strategy wonderful. You own that piece of the, the model, the data sovereignty, also the security of the deployment. So we have to go through a lot of the compliances, um, so that we can serve customers in worse America.

• 23:45Craig S. Smith?

  And and the models, the instance of the model that you own is on your cloud or where where does it exist?

• 23:56Speaker 00

  Ah, yes, these are open source models on our cloud.

• 23:59Craig S. Smith?

  On your cloud, but but.

• 24:02Speaker 00

  Other people can use them too. They can directly download them from parking face.

• 24:06Craig S. Smith?

  Right, but I mean, the the the uh uh an enterprise customer can have a a private instance of an open source model on your cloud.

• 24:18Speaker 00

  Absolutely.

• 24:19Craig S. Smith?

  Fine tune it for themselves.

• 24:21Speaker 00

  Yep.

• 24:22Craig S. Smith?

  Yeah, and what if build the future of multi-agent software with agency? That's A G N T C Y. The agency is an open source collective building the Internet of Agents. It's a collaborative layer where agents can discover, connect and work across frameworks. For developers, this means standardized agent discovery tools, seamless protocols for inter-agent communication and modular components to compose and scale multi-agent workflows.

• 25:00Craig S. Smith?

  Join Crew AI, Langchain, Llama Index, Browser Base, Cisco and dozens more. The agency is dropping code, specs and services, no strings attached. Build with other engineers who care about high quality, multi-agent software. Visit agntcy.org and add your support. That's A G N T C Y dot O R G.

• 25:33Speaker 00

  Different things with us. For instance, you say, okay, I'm looking at um.

• 25:40Speaker 00

  I want this model to be better, right? But I don't know how to do it. I don't, I don't know how to do it in a sort of the frontier way. So we'll have a team and helping you to get the strategies up from the, the pre-training data all the way to post-training data pipeline, make your model look better, performs better. And now you all the way coming to the inference side. We can also help you to customize inference solutions so that you reduce your cost and you make your performance and also the accuracy to be in a very satisfactory range that the customers demand. And that's what we do.

• 26:18Craig S. Smith?

  Yeah. And the, the, the open source models, I mean typically have lagged the proprietary models.

• 26:29Craig S. Smith?

  Can you bring open source models up to compete with? You know, I, I think you mentioned O three mini or, or.

• 26:38Speaker 00

  Absolutely.

• 26:39Craig S. Smith?

  Yeah.

• 26:39Speaker 00

  Absolutely. Let me talk about the field and why our mission of supporting open source models gets so much attention. And I think today, um, you know, uh, this mission is no, isn't, is, is no longer, uh, you know, something that people feel, feel like, oh, you know, this may now be, uh, that great compared to close source model. So this really changed, um, when, uh, major players are started, started releasing open source models, which is super aligned with our original vision of creating the company. So models today like DeepSeek V three model, nine reasoning model and DeepSeek R one model, uh, Lama four models that just released, uh, last week, multimodality, state of our multimodality

• 27:24Speaker 00

  models in different sizes, right? A super long context window and, uh, um, and, uh, you know, all these models are very, very competitive. When R one was released from DeepSeek, it was the biggest news for weeks because the model is so good, super competitive to open, uh, to close source models, like open API models, right? Uh, in many aspects is better. So, uh, open source community finally can use these models for their own way of creating their own enterprise or businesses opportunity, right? Open source model is really head to head with close source modeling companies. And really, I think really making a lot of them very nervous because.

• 28:09Speaker 00

  Deep, the DeepSeek, for example, DeepSeek, they, they sort of not only open weights, but they also tell you exactly how they using creative way of train the models through reinforcement learning, how they created a new attention kernel that's different than everyone else was using. Um, you know, what everyone else was using multi-head attention, they created something called ma- ma- uh, multi-head latent attention, which does difference in compressed KV cache really, really well. Um, to a very small size. So all of these creations, open source, the open source libraries or communication libraries and all of that, um, really strengthened the community's, you know, uh, mission to, you know,

• 28:52Speaker 00

  one day we'll have the, the best models from open source community. And these days where really, uh, customers, users are really enjoying using this open source model because they're either better or the same level of the closed source models.

• 29:08Craig S. Smith?

  And what do you think's gonna happen to the closed source models? I mean, it's uh, th- these companies spend an enormous amount of money on research.

• 29:17Speaker 00

  Right.

• 29:18Craig S. Smith?

  Um, th- they have to be used through an API, which th- they lose a huge chunk of the enterprise market, anybody who, who needs to, uh, keep their data on, um, premise.

• 29:35Speaker 00

  Yeah.

• 29:36Craig S. Smith?

  And, and now, uh, if, if these cheaper open source models can do nearly the same thing, um, it just intuitively seems that, uh, the, the proprietary, uh, system will eventually disappear because you can't keep investing.

• 30:00Craig S. Smith?

  Tremendous amounts of money. I mean, particularly if it's primarily just in in scaling.

• 30:08Speaker 00

  It came, um, you know, um, let's first not deny the amazing, um, you know, work that on in the transformer era that open and all the all the other pioneer companies that delivered since twenty twenty three. That was really the biggest innovations, I think. Um, you know, for this decade or more. Um, now in January, it was when deep sea models released, it was really the moment that the entire community realized that, okay, I can do this in the same level. Yeah, it's a good question. So my prediction is that these companies will have to figure out their model releasing strategies.

• 30:53Speaker 00

  Um, I think more and more of them, including OpenAI, will likely to release their open source models. Now, what likely happen is you have open source model that has a somewhat a gap of a cold source model. Now, this open source model they're releasing will be more for, let's say, smaller devices, something more commodity. So they're still maintaining the sort of the cloud level, large system level reasoning models at trying to do that as much as they can, which is very competitive, very hard these days, right?

• 31:34Speaker 00

  Now, but I think their business model has to transform, but um, my understanding or my feeling is they have to compete in the open source market.

• 31:44Craig S. Smith?

  Yeah.

• 31:45Speaker 00

  They have to maintain that top position. They have to open source model and beat everyone else. So that the community understanding is, okay, these companies still can do the best model. But that game just getting hotter and hotter and then

• 32:00Craig S. Smith?

  Yeah.

• 32:01Speaker 00

  Dinguish in products is very difficult. Yeah.

• 32:04Craig S. Smith?

  Yeah, uh, together is you're in M, you're a, um, I guess ML Ops or Gen AI Ops.

• 32:14Speaker 00

  ML system, yeah.

• 32:15Craig S. Smith?

  ML system, yeah.

• 32:19Craig S. Smith?

  Company, does, do any of the proprietary model makers use Together AI's solutions? Or are you really focused on the open source community?

• 32:34Speaker 00

  Um, I think the, I can, that's related to the sort of the, the business NDA. But I think the understanding is the platform currently is being used widely across different enterprises.

• 32:52Speaker 00

  Some companies, many companies may have their own models and as close source models or whatever, whatever for their internal business. But we have customers, they, they use our platform to do other things, right? So I think, you know, the, the key here is the flexibility that we provide, the innovations we provide. As well as the, the data sovereignty, security, model ownership that we provide on our platform. It's just a very different business model for these companies that unless, uh, let's exclude the one or two of this top open source companies. But everyone else, my understandings can use our platform and can do, you know, work on that.

• 33:38Craig S. Smith?

  Yeah, uh, I mean, it seems to me that space that you're in is increasingly crowded. I mean, I talked to a lot of companies that have, uh, you know, the accelerated training, the make training cheaper, that that can elevate open source to compete with the best reasoning models. How, how do you operate in that market?

• 34:05Speaker 00

  Hmm, it is, it is competitive, but we have the greatest reputations in this realm that provide the open source community, the key technologies, they can operate these models today. Without flash attention and other works that we contributed, we, we, we could have just closed sourced them, right? We don't have to release them, but the, the way that we did that, uh, you know. First, uh, we have the community trust that we always produce the best solution. And the numbers shows, uh, you can, um, uh, go to the different, uh, you know, uh, analytical sites like, you know, artificial analysis and others, and you can look at our numbers, um, and superior numbers,

• 34:48Speaker 00

  uh, superior cost economics benefits to enterprise, right? So I see this game as, you know, um, you have, really have to have a really talented research team behind this operation. Um, you can sell GPU hardwares, right? I can, if I can just go to the GPU hardware resource war with you and just selling, beating on each other's prize. Um, like some companies does that. But if you look at their members supporting these mo- models, right? And the flexibility they can customize these enterprise, uh, you, like a unique enterprise solutions for different companies. Because different companies are having their own serving solu- uh, ser- serving conditions.

• 35:33Speaker 00

  Some companies want very extreme serving conditions. They want large backside, they want a large throughput raging. They care about latency, but they probably don't really care about tokens per second. All of these different models, they fine tune or they make. How does that really perform on today's hardware? And that's the, sort of the question that, that we answer and we connect. So it's both a research leading, uh, team behind this whole operation of accelerated AI cloud. It's not about, uh, you know, uh, using, just using open source, uh, framework and then get it run on your platform. Uh, and that's not, you know, a s- enough solution for our customers, at least the ones that we have, uh, we,

• 36:19Speaker 00

  we have seen the market, that's not the, yeah. It's getting, uh, there are more players, um, but we're confident that, you know, we've been the leader in this community for a while, yeah.

• 36:29Craig S. Smith?

  Yeah, uh, and I, I, well, somebody.

• 36:33Craig S. Smith?

  An enterprise who's working with an open source model, how how do they decide on who to use? I mean, other than your your prominence in the market? And do you compete with people like Sarvam AI or Alef Alpha? Do you know these guys?

• 36:58Speaker 00

  No, yeah, I haven't heard of them.

• 37:00Craig S. Smith?

  Or RunPod or?

• 37:02Speaker 00

  A RunPod, I know, yeah.

• 37:03Craig S. Smith?

  Yeah, are they a competitor?

• 37:07Speaker 00

  Um, so I think the there there is a overlapping business there. I think there are some portion of it. Um, but um, I think we're running a very different business models in the market. Yeah, that we support. Now, you I can answer your previous question about, you know, who they choose, right? Um, my experience of talking to customers is especially enterprise customers. Um, their models are quite unique, even if they're based on the open source models. When they provide their models, you know, the process has played out where we have them to to, you know,

• 37:51Speaker 00

  further enhance their model. Anyway, um, most of the time they want these very extreme serving regimes, as I talked about. Um, with all a a a really sophisticated serving engine. Um, and without a team that behind it that can do can connect modeling, game firms, training, fine tuning together with just kind of the entire AI lifecycle. Uh, you you will have very hard time to produce that solutions for these enterprise customers. And that's what we were really, we have been really proud of that our team can do that.

• 38:31Speaker 00

  So, um, yeah, so I think that's what enterprise customers want. Now just take the open source model with open weights and say, okay, now I want to just use this model and then just find somewhere to serve it. Of course, our pricing are competitive too, very, very competitive. Um, and we're trying to give our customers not only the ownership, but also the best economics. Um, every day where our team are working very hard to get to make that happen. Yeah.

• 39:01Craig S. Smith?

  Yeah, um, and and then um, um, on the on the cloud, you you operate a cloud, is that right?

• 39:09Speaker 00

  Hmm.

• 39:10Craig S. Smith?

  And uh, is it primarily a GPU cap cloud? I'm I'm very curious about these new accelerators built for inference.

• 39:19Speaker 00

  Sure, I can, I have more knowledge in that realm.

• 39:22Craig S. Smith?

  Yeah, sure.

• 39:23Speaker 00

  The pure modeling questions. Uh, I think, yes, uh, for today, together is primarily um, operating on GPU cloud. Uh, we are um, recently become uh, this big partnership with Nvidia for the Nvidia universe of, as you know, uh, one uh, so uh, important partnership for cloud serving. So that, so the, I want to talk about the technology. Because the technology that we create is, we we have this in mind that the future, you know, it, of course, Nvidia GPUs, other GPUs are fantastic.

• 40:03Speaker 00

  But as a a a technology company, we're also looking at other possibility to mix them into a serving solutions for our customers in near future, right? What I mean is, in this serving situation, you will not only have GPUs, but you will have other kind of ASICs, right? They do different, they pro- to perform different roles in terms of how you do the economics, uh, what is the the band about, uh, the the customer's demand for latency and throughput? Um, all of these, uh, I think the solu- the complete solution would be a cloud that utilizes different kinds of a group, these hardware, group of hardwares, like what uh, being provided on the market today.

• 40:45Speaker 00

  Um, but as a company, we're very carefully evaluating, uh, because we're very experienced about user workload, very experienced about serverless and the endpoint use cases. So we carefully evaluated all these hardwares. And then eventually, our hope is to integrate this into the together universe, together serving platforms. Um, and we have a couple of ideas that we can, you know, really reduce the cost and getting the throughput, uh, you know, uh, significantly improved ways these hardware being part of the solution.

• 41:20Craig S. Smith?

  Yeah, because, um, you know, I've had, uh,

• 41:25Craig S. Smith?

  Andrew Feldman from Cerebras on a couple of times. And I've had Rodrigo Yang from San Benova. And I haven't talked to

• 41:39Craig S. Smith?

  to Grok yet, Grok with a Q. But but their argument is that their inference speed is is light years ahead of Nvidia. And consequently, cheaper. So I I'm but it doesn't seem like there's a lot of uptake. Everyone's still depending on Nvidia. I'm kind of waiting for that.

• 42:10Speaker 00

  Yeah.

• 42:11Craig S. Smith?

  Point at which people sort of shift the inference to another.

• 42:16Speaker 00

  My personal opinion, and this doesn't represent Together's opinions, purely my personal opinion on this, is that ASICs are great. But they have to overcome a lot of hurdles of being the primary and only hardware serving in a cloud setup for a large enterprise. So you can just look at the decoding number. So ASICs, when they make ASICs, the the pure.

• 42:44Craig S. Smith?

  And explain what ASICs is for for listeners.

• 42:48Speaker 00

  It's like a customized hardware that you understand a generic version of this model. Let's say transformer model. It's a you understand it like a dramatic pattern. You map that into these

• 43:04Speaker 00

  very fine grained circuits and have electrons and all this gating. And you make it the, you know, either data flow or whatever. You make it mapped that in there that does something that customized for this generic version of the model. So then you can enhance things like I can give more shared memory or on chip memory. So I know I can get much way bigger. And I can, if I know the current most popular models are like this, that customize a pattern or hardware pattern for it. Now Nvidia is an AI supercomputer, but it's not just.

• 43:44Speaker 00

  AI computer, it's a general purpose with something called a tensor core, which tensor core does the AI matrix multiplications in different kind of procedure. So ASIC's advantage is that because you purely customize for a pattern, your likelihood of doing decoding, which is we talked about auto regress uh regressive generation of tokens, that process can be really fast because you customize the hardware and you control the latency, you control the throughput, you control the size of memory for it.

• 44:18Speaker 00

  But uh there are other factors for service, right? You have to consider um you know the uh flexibility of this ASIC. If I have, if today we're serving at a range of dense model, let's say four hundred billion parameter dense model. I'm only ma- mi- mix of expert models we're serving, let's say one trillion, two trillion range. Okay, today you can spend a lot of time customizing this hardware. Now, in about a month or two, our field move on to even a different realm. Let's say I go beyond two trillion or I did an entirely new transformer engine or transformer architecture.

• 44:59Speaker 00

  Then these ASIC um vendors, they have to adjust it really quickly, which is not, you know, you can say it's it's very possible, right? You know, be because their expertise, but it's sometimes it's also less flexible. So GPUs benefits are very flexible because they're general purpose computing plus a AI core. Does that make sense?

• 45:23Craig S. Smith?

  Yeah, absolutely. Yeah, yeah. And and just uh ASIC stands for what?

• 45:30Speaker 00

  Um, that's a good question. I've been using this forever. I think it's uh um I don't want to say it wrong, but um.

• 45:40Speaker 00

  I want to say it accurately. It stands for uh

• 45:49Speaker 00

  sorry, I, yeah, I don't.

• 45:53Speaker 00

  I don't want to say this wrong.

• 45:57Speaker 00

  Sorry, I'm just can't getting.

• 45:59Craig S. Smith?

  Is it A S?

• 46:02Speaker 00

  Uh. A six, A.

• 46:12Speaker 00

  Trips.

• 46:19Speaker 00

  Oh, it's application-specific integrated circuits.

• 46:22Craig S. Smith?

  Oh, right. Okay.

• 46:24Speaker 00

  I remember the first part. I don't remember.

• 46:27Craig S. Smith?

  Yeah, I can. And is Samba Nova and ASIC?

• 46:34Speaker 00

  Yes.

• 46:35Craig S. Smith?

  Yeah.

• 46:36Speaker 00

  All of these, all of these companies, you know, Brock, Samba Nova, Service, these are, you know, ASIC vendors, Brock, yeah, yeah, mentioned.

• 46:49Craig S. Smith?

  Okay. Okay. And then what's next for Together AI? Where are you going with us?

• 47:00Speaker 00

  Yeah, so our goal is we're trying to provide this AI accelerators cloud. We want to expand the hardware capacity or diversity that we can serve our customers.

• 47:18Speaker 00

  We want software solution, provide better software solutions, especially focusing on inference. If you go visit our, you know, website, now you can see tons of applications as being using open source models that build through Together platform and Together solutions. So we want to, you know, kind of encompass all of these capabilities of our research and translating that research into our platform. So not only enterprise users that we, we really put a lot of efforts for them,

• 47:57Speaker 00

  but also the broader community that they can, they can enjoy and use our applications and serverless endpoints. For instance, recently we released something called. Um, uh, together, uh, together chat. If you go to, I was playing around it this morning, right? Um, if you go to, uh, Google Together Chat, you can register an account. And then you, you have a bunch of models that you can select. And from reasoning models, um, all the way to multimodality models, uh, multilingual for coding tasks, for image generation. So we have all these models serving within this one chat, uh, format through, power through open source models.

• 48:40Speaker 00

  And then we also gonna inject a lot of the things like what we create called deep research, right? Um, which is combining search functions with mix of agents into reap, uh, enhancing the power of the Together Chat. So users will all have all of these packaged in this, uh, uh, in these applications that they can use. Um, of course, these technologies can be taken apart and then put into our software stack that use for enterprise customers as well. And this is just one of the examples, um, that we're trying to do. We're trying to do the cloud, uh, really well, uh, you know, through AI. Um, and then we're trying to provide this super fast, uh, highly efficient,

• 49:26Speaker 00

  both in performance, control and economics. And also putting a lot of these AI research in the modeling and data into this platform uh, and transform them into products for our customers and be competitive uh, against the closed source modelings, um, through open source. Yeah.

• 49:46Craig S. Smith?

  Okay. Well, that's fascinating and, and I think I'm gonna end it there. But, uh, this, uh, Together Chat, is that, uh, is there a free layer?

• 49:57Speaker 00

  Um, I think, I think you can use it for, I, I, I'm not really sure up to what point. But right now I think it's a freely, freely offered that you can, you can use and test. Yes.

• 50:09Craig S. Smith?

  Okay. Well, I'm gonna go play with it.

• 50:11Speaker 00

  Yeah, you're gonna go play with it. It's, it's, it's quite good. Yeah.