E36: Your Model, Your Weights with MosaicML's Abhi Venigalla and Jonathan Frankle

In this episode, Nathan sits down with Jonathan Frankle, Chief Scientist, and Abhi Venigalla, Research Scientist of MosaicML. They chat about Mosaic’s custom LLMs, the customers seeking Mosaic out and what their journeys and use cases look like, and exciting developments in Mosaic’s research: including their new inference platform, as well as Mosaic’s MPT-7B-65k+ storywriter model. RECOMMENDED PODCAST: The HR industry is at a crossroads. What will it take to construct the next generation of incredible businesses – and where can people leaders have the most business impact? Hosts Nolan Church and Kelli Dragovich have been through it all, the highs and the lows – IPOs, layoffs, executive turnover, board meetings, culture changes, and more. With a lineup of industry vets and experts, Nolan and Kelli break down the nitty-gritty details, trade offs, and dynamics of constructing high performing companies. Through unfiltered conversations that can only happen between seasoned practitioners, Kelli and Nolan dive deep into the kind of leadership-level strategy that often happens behind closed doors. Check out the first episode with the architect of Netflix’s culture deck Patty McCord. https://link.chtbl.com/hrheretics The Cognitive Revolution is a part of the Turpentine podcast network. To learn more: www.turpentine.co TIMESTAMPS: (00:00) Episode Preview (06:04) Mosaic’s business model (07:28) Who uses Mosaic’s custom LLMs? What does their data look like? (09:55)Mosaic’s use cases for custom LLMs (12:47) How much extraction and summarization was done by humans pre-LLMs? (15:28) Sponsor: Omneky (21:50) The journeys of Mosaic’s customers and would a Wendy’s LLM know about a Big Mac? (25:46) The curriculum model and fine-tuning (29:10) Language models in the life sciences (33:20) How raw can data be before it becomes a problem? (35:44) Using the output of bulk pre-training process vs additional after training (38:30) Redteaming as a service (39:40) Mosaic’s inference platform (41:53) Spending one cent on 20,000 tokens, how is that cent distributed? (46:00)) Selling compute on a dedicated capacity basis (47:30) Oracle and AWS (49:50) The storywriter model and 65,000 token window (54:35) The transition from finite parameters into infinite attention matrix LINKS: MosaicML: https://www.mosaicml.com/ MPT-7B Storywriter Model: https://huggingface.co/mosaicml/mpt-7b-storywriter TWITTER: @jefrankle (Jonathan) @abhi_venigalla (Abhi) @MosaicML (Mosaic) @CogRev_Podcast @labenz (Nathan) @eriktorenberg (Erik) SPONSORS: Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive Thank you Omneky for sponsoring The Cognitive Revolution. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work, customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected]. Music Credit: MusicLM

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Who's training these models? Everybody. Really, the question you should ask is, who has interesting proprietary data? Everybody. I mean, as Abhi mentioned, there's a model size for everybody. There's a good entry point for everybody. And at the end of the day, you know, it's everything from small startups for whom the model is their main product. Companies like Replit that are very tech-forward and AI-first and recognize the power of this to the kinds of companies that if I mentioned them, you'd say, wow, that's a really big, boring company. They're doing AI. It's crazy. Yeah. I mean, they have amazing data. This is how you activate it.

• 0:31Speaker 01

  In the old days, you know, people used to say like, data is your mode. And then in the past year or so, there's been like this new kind of wave where, well, actually training the model so hard, maybe that's enough. And I said, what Mosaic's doing is like, we're making that easy again. So it's almost like making it more training boring again.

• 0:47Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, we're talking to Jonathan Frankle and Abhi Venigalla, chief scientist and research scientist at Mosaic ML. In today's world of AI hype, it seems like almost any project has at least some chance of going viral on Twitter. A few of those will prove deserving and enduring, but most will quickly fade away.

• 1:32Nathan Labenz?

  To make headline news repeatedly, however, as Mosaic has done over the last year, is something that only truly top-notch organizations can do. Mosaic specializes in creating custom proprietary language models for corporate clients. They were the first to offer GPT-3 quality models for $500,000 in September of 2022.

• 1:52Nathan Labenz?

  And they were the first to train Stable Diffusion from scratch for under $160,000 in January of this year. They then turned around and did it again, announcing Stable Diffusion for under $50,000 in April. One funny note from the conversation, at one point, you'll hear Jonathan give me a bit of a hard time for quoting the $160,000 number instead of the latest $50,000 number. I went back and checked, and it turns out that they announced that additional price drop in the few days between when I prepped for and when we recorded this episode. Just goes to show how fast things are moving. Most recently, Mosaic has released their own open source models, as well as an inference service that allows you to use their servers to power your applications. According to LMSys.org,

• 2:37Nathan Labenz?

  a leaderboard that collects human evaluations on blind head-to-head language model comparisons, their MPT-7B chat model is currently the number nine rated chat model in the world. Now, that's already an impressive accomplishment. But what's even more impressive? When you remove the OpenAI, Anthropic, Google, and Llama-derived models from the list, it turns out that MPT-7B chat is actually the number one rated open source model that is available for easy fine-tuning and commercial use. Additionally, the release of the MPT-7B Storywriter 65K Plus model, which allows actually even more than 65,000 tokens of context, legitimately

• 3:21Nathan Labenz?

  shocked much of the AI world and set a new standard for long context models, which are already quickly becoming the norm. We'll talk about the alibi technique that they used to achieve this. We only had an hour for this call. As you'll hear Jonathan say, demand for Mosaic ML services is through the roof. They are reaching the point where they're making tough choices between serving additional customers and conducting additional research. Obviously, that doesn't leave a ton of time for podcasting. But I still think this episode is a great window into who is training their own language models, why they're going that route, what they're using them for, and the techniques that are powering this trend. Before jumping into the episode, a quick thank you to everyone who has shared

• 4:02Nathan Labenz?

  The Cognitive Revolution with friends or posted a review online. We're now up to roughly 25,000 unique monthly listeners, and I am having a ton of fun sharing all of these conversations and everything I'm learning from them with all of you.

• 4:17Speaker 03

  Now, I hope you enjoy this conversation with Jonathan Frankle and Abi Venigalla of Mosaic ML. Abi Venigalla, Jonathan Frankle, welcome to the Cognitive Revolution.

• 4:28Speaker 01

  Hey, it's great to be here.

• 4:29Speaker 03

  Really excited to have you both. As you know, I've been a close watcher of and big fan of Mosaic for a little while now, and you guys have built an awesome platform and made a bunch of news recently with a number of product and model releases in the LLM space. So I've got a ton of questions and thought we could maybe kind of structure things by starting first with what I believe to be kind of the foundational layer of the business, which is the custom large language model training. Then I kind of want to get into the new inference side of the business as well. Definitely want to make sure we get to the 65K model that you guys recently released because I think that's super interesting and even want to dig in

• 5:11Speaker 03

  a little bit to how that was done and, you know, some of the new techniques that you have, I'm sure, not only applied but refined in practice. And then, you know, if we have time, we can even get into some, some bigger picture stuff. How's all that sound?

• 5:24Speaker 01

  Sounds great.

• 5:25Speaker 03

  Cool. So I guess when I think of Mosaic, what I have kind of understood you guys to be up until recently is the go-to place for presumably larger businesses, although you've done impressive work to bring the kind of entry point, you know, sticker price down, but presumably mostly larger businesses that want to create their own custom large language models, I think usually from scratch. So the first thing I wanted to just kind of sanity check myself on is, do I have that right? And then we can dig in a little deeper to like who those companies are.

• 5:56Speaker 01

  Yeah, no, totally. I think you, you got it right. And I think we've sort of been expanding from that initial position, like recently in the past, while they're still. But yes, we started off basically trying to help people build their own custom models, whether it's large language models or diffusion models, originally, you know, even verb models and the rest of the 50s even back in my day. We really want to help people who have valuable proprietary data turn that into valuable models that they own. Rather than necessarily leverage APIs and such. One way we've expanded from that is try and make it even, even easier for multiple of these custom models. So they can start from like pre-trained checkpoints. And kind of our recent like MPT models that we released out there. Or show these like public checkpoints where it's already a pretty good language model. And you just throw on top of this and continue.

• 6:40Speaker 01

  And finally, like the last component that we found is that every person who trains models with those wants to deploy them from this. So we figured, well, we should probably help them with that, too. Um, so that's also what we're trying to do. So really trying to help end to end. People go from the data to these um, kind of private models, private endpoints.

• 6:58Speaker 03

  Let's unpack that a little bit more. Who in today's world i- is building, you know, a custom large language model? And you know, how big are these models that they're building? We could talk about that in terms of parameters or like token count on the training side. You've kind of answered already, but I was kind of curious to what degree you're seeing people use a combination of like open source data sets versus just, you know, how many of these customers actually have enough of their own data where they don't even need, you know, any of the, the kind of standard data sets?

• 7:28Speaker 01

  Yeah, for sure. I mean, like, I think we have a very wide spectrum of customers. We have people concerning, you know, training, you know, sub-billion parameter models to, you know, like in the single-digit billions of parameters and some even going well beyond that. Um, you can see some of our customer stories like Replit recently. They trained like a really powerful three billion parameter model that actually in some ways is like the first code version, right? So kind of like out punching for its employee class. We really focus a lot on kind of efficiency in this way. We want people to be able to produce things that are smaller, you know, cheaper for inference, cheaper for training. Um, that still match the quality that they need. Now, in terms of data, you know, this is, this is a thing I've really focused on nowadays, which is that sometimes customers come with a lot of pre-trained data, you know, they may have

• 8:12Speaker 01

  across all of their like, you know, customer interactions or like, you know, follow logs and stuff, maybe like tens of billions of tokens, but yeah. But they want the model to also start with a general knowledge of the world, right? So there's um, a good amount of research doing that. Here I'm like, what are these data mix? The answer is spend a lot of time creating the data for the MPT models at least. Um, but yeah, I would say like, uh, I would expect customers to have a mix of both public and private data.

• 8:37Speaker 00

  I'll throw in one more thing, which is, you know, to put a really fine point on answering your first question. Who's training these models? Everybody. Really, the question you should ask is, who has interesting proprietary data? Everybody. I mean, as Abhi mentioned, there's a model size for everybody. There's a good entry point for everybody. And at the end of the day, you know, it's everything from small startups for whom the model is their main product, companies like Replit that are very tech forward and AI first and recognize the power of this, to the kinds of companies that if I mentioned them, you'd say, wow, that's a really big boring company. They're doing AI. It's, yeah, I mean, they have amazing data. This is how you activate it. So it really is everyone and everybody, you know, trying to train these models.

• 9:18Speaker 01

  Yeah, actually, you remind me of something that I thought of a lot, which is that kind of in the old days, you know, he used to say like data is your mode. And then in the past year or so, there's been like this new kind of way of what, well, actually training the model so hard, maybe that's the mode. And I said, what was next thing is like, we're making that easy again. So it's almost like making it more training boring again. You know, like we're bringing it back to a position where actually your proprietary data is what makes your model so much better.

• 9:43Speaker 03

  Can you expand a little bit on just kind of use cases? I mean, obviously, you get some names on your website and then I'm sure, you know, lots more customer names that maybe you can't disclose. But if you can kind of abstract away from, you know, the sort of identifiable details of some of these customers.

• 10:02Speaker 03

  I think people are really curious about use cases, like are we building chat type agents to like help enterprises interact with their customers? Are we doing like task automation back office? What's that kind of breadth and mix look like?

• 10:18Speaker 00

  You know, the boring answer is it's a little bit of everything. But I think the main thing we see, especially with big enterprises, kind of, I can really sum it up as two tasks, extraction and summarization. Those seem to be kind of the core workforce tasks that people want to get done. You've got a huge amount of information. You may get there because, you know, a new court case came out and it's a hundred something pages long and you want to understand what it's about right away. You may get there because you're using something like LangChain in a vector database. And you pulled up a bunch of really relevant documents and you have a lot of information that is relative to some question. But at the end of the day, you really want to extract out the relevant piece of information or the relevant passage. Or you want to get a summary and get useful information out of it. I think for a lot of our enterprise customers, it's as simple as that.

• 11:02Speaker 00

  For a lot of, you know, other customers that they are looking at some specific application. We certainly have customers who care a lot about chat. I wouldn't say it's anywhere near the majority. But we certainly have some folks who either care about chat as their main application or care about chat. You know, as a good user interface on top of one of these systems. And then, of course, you have customers like Rappler that are doing something that doesn't really fall cleanly into any of these categories. And it's genuinely useful. So really, you know, the models are as multipurpose as, you know, any other language model. But those are really, at the end of the day, you know, those are kind of the boring useful things that honestly, these models are best at. And they're both things where the model doesn't have to be perfectly right. We're getting in the right direction or just trying to get useful information out of some data you already have is the most important part.

• 11:45Speaker 00

  And, you know, you're not about to make an important medical decision on the basis of a summary that your model gives you, for example. There's still a human decision maker in the loop to make sure that, you know, the information is acted upon it is in an appropriate way.

• 11:57Speaker 03

  Yeah, I think that's great grounding. There's so much stuff going on in, you know, the corporate world that is ultimately not necessarily super flashy. But there's just a ton of value to unlock because of how much of it there is and how much, you know, more cheaply you could do it. Or potentially also like how much more you can scale it versus what you could do in the past. I wonder if you could comment on that. I see a lot of, in the context of like task automation, sometimes it's like, I have this task. I do it today in a human powered way. And it's that, you know, can be slower and more expensive than I'd like. So maybe I can take, you know, time and cost out. But then the other thing that people often kind of quickly turn to, once they start to wrap their head around this is like,

• 12:42Speaker 03

  maybe I can scale previously unscalable processes. So if it is this kind of extraction and summarization are the main things. How much of that extraction and summarization do you think in the pre-language model era was done by humans versus just like not done at all? Because nobody could get to it.

• 13:02Speaker 01

  Yeah, I would say I'm not trying to say about the lawyer and that's also where I think customers are probably excited to. Um, one thing I think a lot is actually in the context of some of our, our work in our open source repos. Um, lots of times when people use it, they have questions, right? They're like, oh, how do I actually like, you know, use the script, you know, like what should I be doing? What's the workflow? And right now, I think the best that humans can do really is to write like documentation, right? Or like write FAQs and stuff like that. Um, but there's no scalable way to say like have someone next to you as a support person helping you through it, right? But with these models and stuff, we could actually build some system, right? It seems like we can actually build higher and higher quality interactions with people.

• 13:41Speaker 01

  That previously like as we said, you know, it just could not be done. So the cost and scalability of like having, you know, one person for every customer. Um, so I think that's where I would be most excited, not necessarily like replacing, you know, things that are done today, but enabling new things that can't be done today.

• 13:56Speaker 03

  Yeah, I certainly see that in some of my task automation work. It quickly becomes like, geez. We could do a hundred X, you know, of this. And, and some of these things are just every, almost every business has certain use cases like. Often it's like scaling outbound recruiting outreach, you know? How are we going to get, you know, to all the candidates we'd like to get to?

• 14:20Speaker 03

  Most companies just don't have the resources to do as much of that as they would like. And we're starting to see that kind of thing turn out is going to interestingly lead to some probably different dynamics in terms of how that communication actually works and like what it takes to break through and be effective. But we're in this interesting moment where it's like the early adopters are getting these kind of early benefits and then, you know, presumably there's going to be a new equilibrium on some of these things. Digging a little bit deeper still into.

• 14:45Speaker 03

  Why does somebody come to Mosaic? Cause I can imagine, okay, I'm in enterprise. I've got these probably at the end of the day, fairly mundane, you know, information processing objectives. I could go to like an open AI and you know, they also make it pretty easy to do fine tuning with just like, you know, supervised input, output pair, uh, paradigm of fine tuning. What is the main reason that folks say, I don't want to go with them. I instead want to work with Mosaic. Is it like? Control of data, cost, if you're going to say all of the above again, but give me a little bit more than just the all of the above. Cause I kind of know what the suspects are, but I want to know like what the mix is and what you're seeing.

• 15:27Speaker 04

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 15:31Speaker 01

  Yeah, totally. I think I'll focus on like kind of two customer profile this evening. One is the type of customer, often startups, where they want to build a new product or like experience that truly cannot be done any other way than to produce their custom model. You know, they either have to train like a really custom domain like coding model, or they need a model that handles multiple languages. Or basically like things where it's like, you know, they've tried the APIs, it's not quite good enough, or it's not cost effective enough, and they really need to have their own custom model to do it. And so for them, it just makes sense, you know, like, well, if I need to build an element house, you know, I need to get the computes, I need to get the engineers, I need to get researchers. It's a really expensive and honestly today, pretty hard to find group of people to put this work in.

• 16:14Speaker 01

  Or kind of like the other option, you can just go to Mosaic and you build with them. And with a very small team, right, you can actually build these custom models for yourself. I would say that's like one type. Another type are people who actually are like our users of these API providers, like OpenAI, Coher, others, right? Where maybe they build the first version of their product on top of these APIs, and it's going according, you know, they've skilled users, you know, they're getting lots of revenue, that's fantastic. And now it's sort of like, okay, now it's an optimization question, right? Like, can I potentially build smaller models or custom models myself and deploy them for cheaper, right? And that's like part of what we're talking about in this inference strike. We're basically trying to very clearly separate, you know, the costs of training the model

• 16:57Speaker 01

  and the cost of deploying the model, and it's like a small team. It's actually very, very close to the actual inference GPU costs.

• 17:06Speaker 01

  Whereas opposed with OpenAI system, right, you know, you're paying for a token thing, and the token cost actually goes a lot when you fine tune. So I think like, for instance, some of the fine tune models, they cost 6X as much when you go from like the base models to fine tune. With those, there's no such thing like that, right? It's sort of, you know, we're very explicit about, you know, here's like, you know, the compute costs and cost share. It doesn't matter really quick to put them on either base models or

• 17:32Speaker 01

  So that's where I think like we're trying to be more transparent about all of this and like passing the data to customers. And, you know, give them a lot more flexibility than you potentially would be.

• 17:43Speaker 01

  Do you want to say, Tom?

• 17:44Speaker 00

  Yeah, I'll throw in, I think, you know, from the big enterprise perspective, it's really three things. You know, it is customization, control and costs. And I love that they all start with C. That was convenient. On the customization side. A lot of folks, like enterprises have huge amounts of data that you can't put into a few input output pairs. You know, we've had plenty of customers come to us and say, I have several hundred billion tokens or I have over a trillion tokens worth of data. There's no other way to get that data into your model beyond, you know, doing a lot of pre-training potentially from scratch. A lot of customers come to us and say, we, you know, we want to customize. The pre-training data for the model fundamentally, they look at, you know.

• 18:27Speaker 00

  Open source models and they say, well, I don't like that data set. I want a little more code. I want a little more of this. We basically have a menu at this point of what's available, open source and how they want to put it together. With any of the big APIs. You have no idea and that's part of the secret sauce. You have to hope that the thing you're worried about just isn't in that model. On the control side, we can do it within their cloud VPC such that the data never leaves. We can do it on-prem if they want to. And they own the model waits when they're done, not just, you know, not just an API to the model where they can rent their own model forever. But they actually own the model. If they want to take it and serve it another way or open source it, we're fine with that. That's, you know, that's up to the customer, not to us. And, you know, I'll be, I think, as comprehensively mentioned, the cost side.

• 19:10Speaker 00

  It's just cheaper to train a domain-specific three billion parameter model than it is, you know, even to try to fine tune in many cases, a gigantic. You know, I guess 1.6 trillion parameter model from what I've heard. Especially if you're going to do it on more than a few input output pairs. So that difference really matters.

• 19:29Speaker 03

  I've done a lot of fine tuning on the OpenAI platform in particular, and I've had some. Things where it's worked really well for me and then I've had other things where it hasn't worked so well. Where it's worked well has been. I have like a very defined task. And I'm kind of dialing it in, you know, I have a certain format requirement. I have certain like length, you know, output requirements. And I, you know, typically find it doesn't take that many examples to get there. But then I've also tried like trying to train it to write in my voice. And that hasn't worked so well because, you know, I only have so many tokens, for one thing. And it sort of kind of learns who I am, but it's like still madly hallucinating.

• 20:13Speaker 03

  You know, after at least the sort of data set that I'm able to muster. It sounds surprising to me on some level that there are this many established companies that are already this far in their large language model journeys. And I see, you know, like, everything's kind of happening, you know, all at once. Or we've got reports of like people banning chat GPT at work. And then we've got reports of other people like, if you don't learn to use it by, you know, July 1st, you're fired. I just saw a story like that the other day. But I am surprised that people are kind of seemingly moving on to like next generation, you know, already. So I'm curious about that. And I'm also curious about like the nature of the models that they create.

• 20:57Speaker 03

  Like, are they creating these models that sort of have a very particular point of view on the universe? Yeah, we saw, I don't think this is your customer, but like Wendy's, you know, has like fresh AI. And I'm wondering, like, does do those AIs like know about a Big Mac? You know, or do they have like just no conception of like what exists, you know, outside of kind of their, you know, corporate knowledge base kind of universe? There's a lot there, but I'd love to hear a little bit more about just kind of the color commentary of the sorts of journeys that people are on and the the nature of the models that they're creating.

• 21:35Speaker 01

  Yeah, so I think, um, I'll make the first question. Yeah, the number of companies that are building out. You know, I think we were also like, you know, pleasantly surprised by the number of companies that we found like since, let's say, last like winter or so, like when we really started doing this a long after. And in the beginning, I think it was mostly startups, but especially since the chat GPT explosion, like, like a lot of like. Basically, once the world like realizes this is possible, we've seen many, many more enterprises like want. They basically want like a chat GPT internally that they can use, like one that they can trust. One that they can actually control, like, you know, the data that's coming in, you know, the recency. Uh, one that which, you know, their internal like employees can you say, right?

• 22:19Speaker 01

  As opposed to some of these reports of like, you know, confidential data leaking into the chat GPT system. So yeah, I think it's, um, even if they're not like super far on the journey yet, um, they, they know right away that they're gonna have to develop some kind of proprietary in-house system, not necessarily just. Yeah. So I, I can't speak too much like what exactly our customers are doing, but I will say it's, it's not too diverse in that way yet. Um, it's not like people are ripping out parts of Wikipedia and trying to control things like that. Um, I think right now, a lot of it's just sort of how do we incorporate our proprietary in addition to like, you know, terminal knowledge? A lot of it is, you know, like nature, there's no harmfulness, you know, no, no toxic hay here, that kind of stuff too.

• 23:01Speaker 01

  Some of it is even just like, how do I incorporate, you know, new things like long context length or long form documents, which you couldn't do until very recently, like, you know, on the APIs. Of course, Anthropic now has like this hundred K support as well, and we're really excited about that, uh, the possibilities, but, um. I think most things haven't added it. I, I, I'm not sure I've seen too much of like the kind of like pruning away that you may be talking about. Maybe do you want to comment on that?

• 23:27Speaker 00

  Yeah, I think what we tend to see is that all of our customers want a mix of some open source data and their internal data. Uh, you know, if you were to just train it on your internal knowledge base, that model's gonna have a pretty limited understanding of the world. And, you know, the knowledge base may not even be in very clear prose in a lot of cases. Um, so a lot of customers are mixing together open source data since they think are a good fit, um, with their own internal data. I do also want to kind of, you know, popping up one level, I think you're framing this as do you use GPT for, or do you train your own model? And I think that's a false dichotomy. Um, what we're seeing for a lot of our customers is that the answer is yes. There are certain use cases where GPT four is both great, you know, and, you know, allowable.

• 24:10Speaker 00

  There are a lot of use cases where either GPT four simply can't be used in those cases. There's sensitive data or, you know, there are a lot of cases where it's just not effective because we need to know about internal business processes that it simply hasn't been trained on because that's internal documentation. Um, so what we see with a lot of our customers is, are they using GPT four? Yeah. Are they using us? Yeah. Um, and you know, these are different tools for different use cases. But I don't think we're trying to say replace GPT four with Mosaic and all. We're trying to say there are a lot of different ways to solve these problems and a lot of different tools to do it.

• 24:39Speaker 03

  Yeah, I think that's a good call out and it's, it's almost, uh, one of my mantras these days too. It's like, it's never in AI right now, either or, it's always kind of both. And, you know, that extends to like societal impact questions as well as kind of, you know, which models and tools are people going to use? So I, I totally, um, that totally makes sense to me, you know, that it's not all one way or the other. So not that, so it makes sense what you're saying there with kind of the mix of data. It makes sense then why you have these kind of checkpoints that are like a base to, to build on, like no need to rerun that sort of, you know, Wikipedia,

• 25:20Speaker 03

  pre-training time and time again, if you can kind of embody it and have, uh, the ability to branch off from there. When you do that fine tuning, are you continuing to mix in? Like what is the curriculum? It seems like we're kind of moving towards this like curriculum model where there's kind of this general, you know, knowledge base, you know, that is the checkpoint and you're laying on the proprietary data. Are you still mixing in the open source data? Or do you just like, just kind of pre-train on their data and that works out? Okay. I don't really have a sense for if you can make that kind of shift from like data set A to B and it all works out okay, or if you kind of have to continue to bring, uh, you know, some of the base forward into the continued pre-training.

• 26:01Speaker 00

  Yeah, I mean, I think the first thing I would I would mention here, and I'm gonna be really picky right now. I hate the word fine tuning. Because when you show up and you say, I want to fine tune on 200 billion tokens. Um, I don't, there's nothing fine about that. Um, it's as hardcore as pre-training, if not more so. What we tend to see is kind of a mix of things. If they're gonna build on one of our checkpoints. You know, you do continue to mix into open source data and create like mix. It may not be the exact same open source data, but often. You know, if a customer doesn't have the right representation of data anyway, simply training on a specific subset of the data they want the model to eventually know. Willie decatastrophic forgetting. And so you do want to make sure that you've got, you know, a good mixture of, you know, some more generic data and some more domain specific data.

• 26:44Speaker 00

  I think this is in some sense, a question as old as time when it comes to transfer learning and fine tuning. Um, there are papers from the BERT era, which oh, so long ago, you know, three years ago. Um, that were all about this question of how do you find to it properly? And, you know, a paper called, Don't Stop Pre-Training, which I mean, should tell you the whole story there. So there are a lot of different, you know, there are a lot of different approaches to this, especially when you get into instruction, fine tuning and RLHF. You are doing a different task. And so you do want to change the nature of your data and probably don't mix, you know, for example, instruction following. It just normal continuation of of natural language. But all this is really, I think it. The bottom line is all this is science yet to be done. We're experimenting, our customers are experimenting, we're doing things that seem intuitive.

• 27:28Speaker 00

  We're trying to test everything rigorously and we're all learning every day. Um, a lot of the best questions like this haven't even been answered in a rigorous way yet.

• 27:36Speaker 03

  Yeah, so basically, no sharp line between. Continued pre-training and fine tuning in many instances anyway. And yeah, I'm starting to see too, like some of these. What had been understood or at least introduced as kind of late-stage training techniques are also being backported into the pre-training, right? Like we're seeing, uh, just from the last week, I think, like pre-training with few shot, you know, structure. Uh, that's kind of a, you know, an earlier thing. That could probably cite, you know, several examples, but oh, I won't waste your time. But yeah, it's it's interesting that both are kind of blending together and the the techniques seem to be kind of going. Both directions at the same time as well.

• 28:18Speaker 00

  Yeah, I think especially once we start getting to this.

• 28:22Speaker 01

  The situation where like we just kind of few shot the models, I mean, each prompt the models. Um, even like the loss functions that we're using for fine tuning and featuring are exactly the same way. We're effectively just giving sequences and saying, please match the sequence, please auto compute it, right? Um, and so previously, I think some of the distinctions came from like, oh, it find two times, then it adds a, a, a new head. You know, I'm going to add a new classification or an L M F or something. Now all of that's kind of at the window and we're just continuing to train the, the original mates. With new like formats effectively.

• 28:52Speaker 03

  Yeah, the great convergence. It is fascinating that, you know, obviously the architectures have kind of unified. Yeah, but also the, you know, the the loss functions, you know, kind of converging to the same thing. Man, that is, it is crazy. Are there any, so what I wanted to ask specifically about, uh, life sciences, uh, cause I, one of my personal goals is to have no major blind spots in my understanding of the AI landscape that I, you know, can't quite ever achieve that, I don't think, cause it's moving too quickly. Uh, but I realize in, in looking at some of the stuff on your website, that that is kind of a, a gap in my knowledge right now. What is a language model?

• 29:33Speaker 03

  Maybe it's not even language models that you're training for that, but I, I had kind of understood it is, I think, language models in the context of the life sciences. What does that look like?

• 29:42Speaker 01

  Um, yeah, so I think, you know, uh, we've done a bit of work, you know, kind of on biomedical language models in the past. We worked with the team from Stanford, the, the C R F M team, uh, to build, I think, um, biomedical. And that was just like late last year, where basically we wanted to have a model of, you know, just custom domain data, which was, I think, like PubMed papers, basically. So then you can actually ask the questions about like, you know, um, medical like diagnosis and, and, and like things like that and have it actually serve well. And you're seeing like a very similar trend happening at Google right now, I think, with their MedCom models, they meant Palmetto. Um, where they're actually able to like kind of put them head to head with physicians and actually see like, oh, could this be an augmenting tool?

• 30:23Speaker 01

  You know, that like, you know, helps you diagnose patients, helps you like respond to them, you know, and, and stuff like that. Um, so that's like one part where it is just genuinely like language models, which are not like biomedical language. Um, there's another sort of direction which, you know, we haven't gone too much into yet, but I understand which is sort of like, uh, genomics or like, you know, kind of these, um.

• 30:43Speaker 00

  More chemistry kind of applications where you know, like protein synthesis and like amino acid chain stuff are also just like sequences, right? And especially with something new like support we offer long context, can we help people in those domains too? Basically, turning transformers for like protein sequences or something. I think that's like a really exciting direction.

• 31:05Speaker 03

  Cool. Any other customers or kind of, you know, perhaps surprising language model use cases that you would highlight?

• 31:13Speaker 00

  I mean, I don't know. I personally still find every use case of language models a little bit surprising.

• 31:17Speaker 03

  Fair.

• 31:19Speaker 00

  It's it's still new and so everything is new and interesting. And honestly, our customers are really creative. So I think there's a lot of really cool multilingual work that we're seeing happening. Where, you know, existing models are okay at doing multilingual stuff, but if you really want a model that's focused on a particular bilingual scenario, that seems to be incredibly popular right now. You know, languages that I would not have chosen if I were trying to build, say, a five language or 10 language language model. So I found that to be really cool and really exciting so far. I hope our customers will talk about that at some point. I don't want to talk about another place for them. But we've seen some really awesome applications in that area and it has to be exciting. It's not something I thought that about. It puts a lot of pressure on the tokenizer, which is kind of

• 32:02Speaker 00

  interesting. Again, something that I find a very fraught subject. But it's it's cool to see all that activity and to see that we've reduced the barriers to entry such that, you know, these applications are now within reach. You can build an LLM for an interesting language pair that certainly nobody would have picked right off the bat.

• 32:22Speaker 03

  Yeah, that tokenizer, uh, this has been interesting research this week, you know, that has been billed as maybe the beginning of the end of the tokenizer. We'll see if that pans out. But I've seen also the like, you know, one character from like a Hindi script, for example, might be in fact like eight tokens because of the way it's all kind of broken down under the hood. And that that is definitely something for listeners. If we want to go down that rabbit hole, you know, check out how certain like Indian, you know, language alphabets get tokenized. It's quite gnarly.

• 32:57Speaker 03

  Wouldn't, I want to move on to the inference business, which you guys have just introduced in just a second. But just a little bit more calibration on kind of this data. You know, people have these, you know, hundreds of billions, a trillion tokens. It sounds like it's often kind of raw. How raw can it be before it becomes like a problem? And then like, how, how tight did the resulting language models kind of stay to that knowledge base? Versus like, do they start hallucinating like plausible products? You know, that these businesses might have, or you know, I imagine that same problem must exist, right?

• 33:35Speaker 00

  Yeah, I mean, I think to a first approximation, you are, your model is what you put into it. If you train a language model on a web data set that has a bunch of advertisements in it, your model will learn to start spitting out advertisements in its sentence. You know, that, that sort of behavior is definitely in these models. And so the data needs to be reasonably clean. You can't just shove everything in there. Not all tokens are created equal, but to your first approximation, all tokens are created equal. You have to do a lot of work to get a high enough quality data set that you start to see this matter. You know, this is why we place such a high premium on things like Wikipedia, on things like code that, you know, tend to be relatively well curated, all things considered from the beginning. And so just, you know, punch above their weight, token for token with other data sets.

• 34:19Speaker 00

  But if you're going to pull data from the web, let alone data from your internal use cases, doing good work on that data to clean it up can be really, really important. And, you know, I imagine, you know, models like Palm, models like, you know, GPT-4, that was the lion's share of the work in some sense. Once you have a good system down for training the model, you know, data is a never ending problem.

• 34:43Speaker 00

  And to address the other question, you know, like, how do you make sure that the company's products are represented faithfully? And they don't like losing things? I think this is where like long context can really help, right?

• 34:53Speaker 01

  The more you can kind of shove into that initial prompts to the model, it's going to tend to that rather than whatever memories are stretched away much more, you know, attentively. And that way you can kind of ensure like, hey, here's a list for actually, you know. June 2023 products, you know, please refer only to these. I think that's a lot more, you know, reliable, potentially interview training every so often, so I got to. So really investigating, you know, researching lots of different ways to incorporate both phone context, potentially in the future retrievable as well, to make it so for enterprises, they can actually trust these models to handle.

• 35:26Speaker 03

  How how often are the are the customers able to kind of use something that comes out of the, you know, kind of bulk pre-training process versus layering on some sort of finishing instruction tuning or RLHF?

• 35:42Speaker 01

  I think for most enterprise applications, you have to go through some kind of like this after-training or polishing. I think, especially, you know, you can even tell from the API providers, they usually have a lot of this kind of either instruction tuning or RLHF or something like that, so that you can actually speak to the model naturally, rather than like kind of prompted very specifically to do what you want. I think we offer, you know, especially on our website, it's service, like instruction to the models mostly. If you build your own, obviously you can start with a base and like, you know, you know, however you wish.

• 36:17Speaker 01

  But my gut feeling is that instruction to models are a little bit more than this law.

• 36:20Speaker 03

  Yeah, makes sense. I was kind of, from that, I'm kind of inferring also like, you know, what the pattern of use is, where, you know, again, in like some of the very highly tailored, you know, fine-tuning, which is, you know, genuine, just like few, you know, example fine-tuning that I've done, it doesn't necessarily need to be instruction tuned because it's it's in such a controlled environment. You know, there's a high degree of kind of developer architecture that surrounds what the inputs are even going to be. But it sounds like there's also a pretty good mix of companies just kind of creating these things and then making them available to their internal users to say like, now you can use this whenever you want. Is that fair conclusion?

• 37:02Speaker 01

  Yeah, no, I mean, I think in terms of evaluation stuff, that the best place to put the API is on a people, like internal ones and and just like see how they use it. I think as you're saying, right, you know, especially if you're fine tuning a lot of data, input output format, maybe it's good enough to start from the base versus the instruction to model. I don't think there's like any capabilities necessary for instruction tuning. It's really just this kind of like massage, you know, the input outputs, so that you can talk to it as good to your friend or something like that. But yeah, so like, you know, a common path that might be, you know, you start from like a base model, right? Either one of our pre-trained ones, one that you built yourself. You deploy it in Mosaic, friends, and we have a nice little playground where you can talk to it and kind of like investigate how it's doing.

• 37:44Speaker 01

  Based on how that performs, you know, you you you share that out with your employees. Then you build a fine-tuned model, and then the next one, the next one. And it's really an iterative process to get the models to what you actually want. I can say we're putting a lot of work into evaluation. Like, you know, we started off with these academic frameworks, and we have some blow up as well. And we have to do really fast, like, evaluation of these kind of in-context learning tasks. But we have to graduate beyond that. These models are getting so good effectively, it's hard to even trust it. So, interactive evaluation, automated evaluation, all of this stuff.

• 38:24Speaker 03

  And if I understand correctly, that's kind of frontier right now. Like, most clients are not, for example, like, red teaming models yet, it sounds like. But you see that as a, the kind of thing that more will start to do in the not-too-distant future.

• 38:39Speaker 01

  I see this as something that we're going to provide as a service to our customers. At the end of the day, you know, we are the one-stop shop for, you know, making sure that you get a good model, all the way from data to training to inference. A big part of that is making sure that the model you put out there is something you're proud of. So we invest a lot of resources when we work with customers, make sure they're evaluating their models carefully. Red teaming is a part of that. And so, you know, I can't make any promises about a Mosaic and a evaluation product. You know, I'm not going to announce that today, but, you know, you should keep your eyes on that. And then I'll put a set to your trigger a little more detail.

• 39:12Speaker 03

  I'm always very interested in the, you know, the kind of red teaming, risk mitigation side of that in particular. But, you know, obviously, confirming that it does the, you know, the happy path, you know, effectively is is also super important. Let's talk about the inference business. You guys have layered on an inference platform to the business. And it's interesting because you both have like open source these, you know, foundation models. People can go do whatever they want to do with them. Obviously, you don't see that as a threat to your ability to build a good inference business. So I'd kind of love to maybe just start with like, the pitch for, you know, the practical kind of way that people understand, even though this model is right here and I could go do it, you know, the, what is

• 39:57Speaker 03

  the calculus that ultimately leads them to choose you? And then I'd love to dig into like, the pricing a little bit more and kind of understand the, the cost structure to the degree that we can do that as well.

• 40:06Speaker 01

  Yeah, absolutely. So I think kind of like, starting at the basics, right? So you've got some of these open source models, like, why not just try and like spin up a G P yourself and serve it? And I think there is, there's a lot of learning that happens once you actually start to try and do this and like serve it on API to, you know, many, many customers. You realize that like, kind of, you know, the, the libraries that you have today, you know, it's easy to like, serve one customer, but very hard to serve many, right? And you need batching, you need things like, you know, lower precision weights, you need things like multi-processing and auto scaling and stuff like that. And tuning all these things is quite difficult, especially as you have these very, very large models. So the challenges I would say is that when it comes to time that

• 40:50Speaker 01

  you actually want to serve, say, a 30 billion parameter language model, you find there's very few existing solutions out there that can actually achieve the type of performance or latency that you expect from like, oh, here and stuff like this. So that's like one, one thing. So I want to simplify all that infrastructure, the same movie. And in terms of the cost side of it, are there any particular questions I could answer?

• 41:12Speaker 03

  Yeah, so I study this somewhat obsessively. So just looking at the pricing page, I noticed that it is five one hundredths of a cent per 1,000 tokens for the new M P T seven B instruct model, which I gather is kind of the, the new like mainline workhorse based model. And, you know, to put that pricing a different way, at 20,000 tokens or 40 pages, you, you've now spent one cent. So you've got

• 41:49Speaker 03

  You know, quite a bit of room to to run there before you're spending much money. Where does that money go? Like, I don't know if this is something that you can even characterize this way. On some level, it could be, but it may be so many layers of abstraction. But like, do we have a sense for if I spend one cent on 20,000 tokens? How much of that one cent went to electricity? How much of it went to buying GPU cards? How much of it went to? You know, I don't know if you guys use like AWS or multiple clouds or even run your own data center. So that'd be interesting, but like, how much of it goes to data center, you know, management?

• 42:28Speaker 01

  Without going into too many details of the exact numbers, I can help build up kind of the stack of everything that goes into that file costs there. So, so Lozak, you know, we deploy anywhere on any computer. So you could bring your own GPUs they have if you have your own AWS account or Oracle account. You could rent compute directly from us and we have some preferred partners, mainly Oracle right now, where we like rent a lot of these GPUs. And so as you build up towards like that cost per server, right? What you're really trying to figure out is to satisfy a given workload. Let's say you need to satisfy like 10 requests per second to you or something. And you need to figure out any of the particular model sizes, say like a BT7 model. You want to figure out how many GPUs can satisfy that thing.

• 43:14Speaker 01

  And then you're basically renting those and running them full time 24/7. So that's the thing about inference, right? Is that you have these servers that are basically running all the time, nonstop, starting requests. And so it's funny, you know, sometimes training, at least, you know, the job finishes at some point, but inference is running forever. So so building up towards that, like, you know, one cent cost, you're right. You're paying a certain charge for renting GPUs. And that both amortizes the cost of the GPU, usually over about three years or something. Plus, electricity is being used for the data center. But but for most people, that whole cost gets packaged up into like the rental price per hour, right? So maybe like. Just putting a number out there for an A100, it costs $2 per hour, right?

• 43:56Speaker 01

  Maybe it's your data center provider, your cloud provider, it costs them only like a dollar to actually run the whole thing because they've got such a complex scale. But that's like the first part, right? So the price per hour. Now on top of that, you know, Mosaic Platform Service, you know, we have a few ourselves, so I put on top of that. But finally, when you're an enterprise customer, you basically just pay per hour price. So we help you figure out where your press workload, how to keep using. We help kind of point you towards the right GPU types that you want as well. Not necessarily A100s, which are really powerful, but a little bit of overkill for sharing workloads. Maybe you want A10s instead. And we help break that down effectively into a price per hour. And so then finally, when it comes to your single request, right?

• 44:40Speaker 01

  What we're saying basically is that that one request, you know, if it was happening constantly over the course of a month, it breaks down to only that little bit, one cent versus like the monthly cost can be a few thousand dollars.

• 44:52Speaker 03

  So as it stands today, can I come and buy like on sort of an API basis, like one call against that new instruct model? So you have both a kind of pay per use, pay per token model. And but it sounds like more of the business is ultimately because people have their own custom models, I guess. They can't use shared infrastructure anymore, right? When they've got custom models, because you don't want to be like, you have too much load, you know, cold start problems. So you're more often selling compute on a dedicated capacity basis. And the value add is helping people understand what dedicated capacity they need so that they can, you know, spend efficiently.

• 45:37Speaker 01

  Yeah, exactly. So we have two ways, two ways, have two pathways for their inference servers. The first is the starter series where there's a, there's a list of supported models that are basically models we build or open source models that you can just pay per, right? So you can match that as kind of like a shared service, like a free provider out there. So I think that's maybe like the place where you got like the one cent for, for 40 figures, right? That's kind of a starter. Then once you train your own custom walls and want to deploy them, that's where you're going to transition to the enterprise service where you're going to not be paying for requests, you're going to be paying this, like as you said, dedicated capacity that we help you figure out. And for there, it's like really, yeah, basically a cost per hour.

• 46:20Speaker 01

  And then that can satisfy certain work.

• 46:25Speaker 03

  And so do people, they buy that through you in a way where they don't even, like, do they know, like, oh, I'm deploying this on AWS and it's kind of a transparent model? Or are you like reselling the underlying cloud compute?

• 46:39Speaker 01

  It is fully transparent, right? Like, I, I think one of the, the, the core features of Mosaic is that we will run the workflows wherever is required, you know, based on your security constraints, based off your existing like contracts that you may have. And it's like optionally, you can rent it through us, but it's just an option, right? So we treat ourselves just like all other providers.

• 47:00Speaker 03

  Gotcha. Okay, cool. So if you had to estimate, going back to that original question, if I rent, I rent GPUs by the hour, let's say on AWS or on Oracle, what would you estimate their stack kind of looks like in terms of how much of their per hour cost goes to electricity? How much of their per hour cost goes to buying the physical hardware?

• 47:23Speaker 01

  Uh, I mean, I think the simple answer is, you know, that's their internal numbers and you don't know. Um, and I don't think it's a good contrast to speculate on that. Um, I would guess that the electricity is probably a relatively small amount compared to the hardware, but that's just a complete guess. Um, and, you know, happy to refer you to our friends at Oracle if you want to ask them.

• 47:45Speaker 03

  Yeah, all right. Well, I might take you up on that. This is something I definitely am trying to triangulate on because ultimately, you know, some of this stuff goes on the edge eventually, right? And then it's like you have no marginal cost of device arguably, and you know, it's just electricity. So I'm trying to kind of scout out a little bit to some, not that that's going to be the end all be all, but there's some part of the future that kind of ends seemingly at like marginal cost of electricity. So trying to kind of sketch out to like what that is, and it, it isn't super easy to figure it out. Anything else we should cover on inference? Um, and then my next thing is to switch gears and go to the, the new 65K context window model.

• 48:26Speaker 01

  The only thing that I say is if you're interested, please, please reach out because I think we have very competitive pricing and a lot of better work in the mall and things than the other agents out there. So you want to look at some of them.

• 48:40Speaker 00

  And I will throw it on that. If you want to, you better reach out to me. It's a fee because we're definitely getting, as, as our research team is learning, we are getting booked up very, very quickly.

• 48:51Speaker 00

  Certainly, you know, it's, we're making hard choices between research and customers right now because there's so much demand.

• 48:58Speaker 03

  Yeah, I'm not surprised by that at all. Following your trajectory, you know, from $500,000 GPT-3 quality, you know, however many months ago to the more recent, you know, stable diffusion for like, I think it was like 150K or so. And now all this new stuff as well. Like, I would expect the phone is ringing off the hook.

• 49:20Speaker 00

  Less than 50K. Got to get that number.

• 49:22Speaker 03

  Oh, less than 50K. Yeah, okay. Thank you.

• 49:25Speaker 01

  I think we crossed price tag a few times, so it's changed a bit.

• 49:29Speaker 03

  Yeah, you guys are hard to keep up with. It's a lot of releases. So, okay, let's talk about this. One of the aspects of the, the recent set of releases. The story writer model, 65,000 token window extends even beyond that. So I understand, I've read the, the paper a little bit, but I'd love to get a little bit better intuition for, it seems like this alibi approach is kind of the heart of the upgrade there. And what we're doing essentially is replacing the original old school positional embedding scheme with a new seemingly more intuitively justifiable principled positional embedding approach.

• 50:14Speaker 03

  But it still seems like the, the model like gets big still, right? Like to have that long token window, you have, you still like, attention still like scales with the square of the context window. You know, I have that right? Nothing has changed about that, has it?

• 50:29Speaker 01

  Yeah, no, 100%. It's, it's still the same amount of work as, as before. We were just figuring the positional embedding and replacing with a bias, need kind of attention.

• 50:40Speaker 01

  But yeah, so I think um I think some of the uh the nuances here is that that quadratic portion of the attention is actually not as big as many people might think. You know, for a lot of very large transformers, you know, like if you're training them with a context of like, say, two K, you actually don't have to work being done that kind of attention product. So it can be less than 10% of the role work of the network. So yes, as we're stretching out the context, so I go from two K to four K to eight K. Yes, we're stretching to 10% a lot, lot bigger, that that that piece of the pie. It's not like a ridiculous. You know, as a kind of point in the sand, when we trained our story writer model was 55 K context window, right? So we had the base model, then we're fine to the small context.

• 51:22Speaker 01

  The kind of uh work done per token uh went up by about four X. So you can think about like the cost to train, I'll say, like, you know, a couple billion tokens was four X larger for that context. Then it would have been a few changes with a two K. So yes, it is bigger, but it's not like ridiculous.

• 51:40Speaker 03

  So from two to sixty five. And it's actually even bigger than that, right? Because you allow it to go.

• 51:48Speaker 01

  Yeah, so at inference time, you can actually kind of you can technically go to any infinite context on, but practically speaking, you know, maybe like two X more than it's been a trainer, but yes.

• 51:59Speaker 03

  So how does that? So how can you have an infinite context? Is something just getting truncated eventually there? Or you're like rounding at some like distance, you're rounding down to zero because otherwise you traditionally you'd have that would mean an infinite by infinite matrix, right?

• 52:16Speaker 01

  Yeah, no, absolutely. The infant is definitely has a lot of aspects there like, you know, your hard roll around the memory at some point. It's not really and eventually you'll get tired of waiting as well. So that's not quite. But um what what is happening in alibis that there are these kind of uh? Slopes biases that are being added, you can imagine like there's a slope from negative one to one, from like the zero position always. And when we go up to like a very, very long context, all we're doing is stretching that slope out because it's a continuous slope. To whatever kind of target inference context like that we want. So we trained with two K and we went to like, you know, negative one to one across two thousand tokens.

• 52:55Speaker 01

  And inference time if we want to do four K, we would just stretch it out farther. So the bias being added, it's basically more of that way.

• 53:03Speaker 03

  So in the, in the sixty five K model that you have released, is there like, there is still some sort of hard cap of like, this is the max number of tokens that this thing can handle? No?

• 53:16Speaker 01

  Not really. So I mean, the model you can see on the human face as well, the config, we have the max sequence of sixty five K. But you can totally adjust it, right? So you download the model, you change that one line of the config, and then when the model's instantiated, it will just create its alibi bias to whatever sequence you wish. And you will be able to use inference to say like one hundred thirty or something like that.

• 53:40Speaker 00

  Yeah, I think the, the only two limitations that you'll encounter with alibi, and this is why we love alibi, is number one, you'll just run out of memory. That is your first limitation. So, you know, if you need a bigger beef, you're to be with more RAM to do this, or you need, you know, fancier parallelism or fancy caching or things like that. The only other limitation is, you know, that past a certain point, you know, empirically, if you've trained in a certain context window, alibi doesn't seem to extrapolate that well beyond about two X whatever the context window you train it on. So you'll run into kind of algorithmic issues there where the model, you know, the quality, the outputs will start to degrade if you get much longer than that. But otherwise, really, you know, alibi can just keep going, whatever amount of memory you have, sign of your memory, and that can give you a longer sequence.

• 54:21Speaker 03

  So I'm still a little bit lost, and this is one thing I do where I just like, fixate on things and keep asking questions until I try to understand it, until I eventually hopefully understand it. I've got finite parameters, but in principle, you know, if I had infinite memory, I could go to infinite tokens in the context window. How, what, can you unpack that transformation of these like finite learned parameters into that infinite attention, you know, approaching infinite attention matrix?

• 54:53Speaker 01

  Totally, yes. So maybe the best way is we could start with something that's close to transformer, something that just has no position knowledge at all, right? So we have all the learned weights, which are like the, you know, the word embeddings and the the weights of all the matrix multipliers and so on, but nothing for the attention matrix, right? And so as you feed in tokens and you kind of save their keys and values,

• 55:17Speaker 01

  you can always keep attending to more and more and more. It's just that there'll be no position like information. So you're looking at a bag of words. You're looking at like a whole jungle of words and your next token attending to them, but has no idea which which one came first. So that model, you could stretch out to any class in which, right? There's no learned parameters going on with the, with the attention now. The one thing that alibi adds is that it's this bias that is set up at initialization that can be stretched and it's not learned. It's like a fixed set of slopes that go from like negative K to like K. And so that's the kind of matrix that you can basically adjust dynamically at different time.

• 56:00Speaker 01

  Say I want to stretch this out over a thousand positions or over 2,000 positions or 100,000.

• 56:07Speaker 01

  So that's what I'm saying. So the alibi kind of mechanism is not like a learned mechanism and that's what makes it possible to use whatever sequence.

• 56:16Speaker 00

  Yeah, I think, you know, to get to the heart of your question, the attention weights are shared. So for each position, for each, you know, encoded token, you're using the same attention weight. So you've got parameter reuse. And so technically your sequence length doesn't matter. You can just keep reusing the same attention weights for any token that you have. In the same way that a convolutional network, you know, in many cases can be completely agnostic to resolution because you're just doing the same convolution across multiple different positions in the image. So that's why fundamentally, there's no issue doing an infinite sequence like other than running out of memory. You need somewhere to store the activations. But a finite parameter model, you're just reusing that same fixed number of weights over and over and over again at every sequence position.

• 56:56Speaker 03

  Anything else we want to cover on the 65K? You know, anything on the flash attention or other enhancements there that you think people ought to know about?

• 57:05Speaker 01

  We went with the model 65K. We showed some demonstrations that, you know, the whole great Gatsby and writing out an app log afterwards. But I'd say like one focus coming up soon is we're going to estimate our performance. Basically making sure that it doesn't take like minutes to write. You know, like I think one really missed a part of playing the game is recently recent releases that the model actually writes quite quickly. And you'll probably see some content from us breaking down how it's actually, you know, use long context windows and where it gets faster and slower. So it turns out reading is a lot faster than writing with these transformers. You can sit in 65K of context into the input and it will go relatively fast. But then to generate every token afterwards will take some time.

• 57:46Speaker 01

  So you'll see a lot of improvements throughout some coming like weeks of this, as well as potentially some new architecture features.

• 57:56Speaker 00

  I think I'll throw in perhaps, you know, as we get into last word. You know, keep your eyes out on things that are coming from us. We've released a lot over the past couple of weeks and that cadence is not going to be slowing down at all. I'll need a lot more work. It's going to be churning out over the next, you know, days, weeks, months. So I hope we'll be having these conversations a lot over the next while. You know, MPT 7D and, you know, stable diffusion for less than 50K. Those are kind of the, the boring baselines that we're going to be crushing over the next little while.

• 58:27Speaker 03

  Well, that's a great teaser. I look forward hopefully to a part two in the not too distant future. And it sounds like you'll have some exciting new stuff to help us break down and understand. But for now, Avi Venigalla and Jonathan Frankle, thank you for being part of the cognitive revolution. Thank you so much for having us.

• 58:47Speaker 01

  Thanks so much.