E40: Meta's MEGABYTE Revolution with Lili Yu of Meta AI

Nathan Labenz sits down with Lili Yu, a researcher of Meta AI to discuss the paper she authored: MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers. In this conversation, they discuss the architecture and breakthroughs of their research, and the opportunity to eliminate the need for tokenization. We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. RECOMMENDED PODCAST: The HR industry is at a crossroads. What will it take to construct the next generation of incredible businesses – and where can people leaders have the most business impact? Hosts Nolan Church and Kelli Dragovich have been through it all, the highs and the lows – IPOs, layoffs, executive turnover, board meetings, culture changes, and more. With a lineup of industry vets and experts, Nolan and Kelli break down the nitty-gritty details, trade offs, and dynamics of constructing high performing companies. Through unfiltered conversations that can only happen between seasoned practitioners, Kelli and Nolan dive deep into the kind of leadership-level strategy that often happens behind closed doors. Check out the first episode with the architect of Netflix’s culture deck Patty McCord. https://link.chtbl.com/hrheretics The Cognitive Revolution is a part of the Turpentine podcast network. Learn more: www.turpentine.co LINK: MEGABYTE Paper: https://arxiv.org/pdf/2305.07185.pdf TIMESTAMPS: (00:00) Episode preview (07:41) Takeaways from Lili Yu's paper: MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers (17:00) Architecture (24:59) Embeddings (27:43) Different local models (34:23) Encoder model (36:35) Transformer Architecture (48:10) Choosing patch size (01:08) What happens when you scale up? (01:19:20) Big picture for Meta AI (01:22:57) Responsible AI (01:27:02) China and AI TWITTER: @labenz (Nathan) @liliyu_lili (Lili) @eriktorenberg (Erik) @cogrev_podcast SPONSORS: Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive Thank you Omneky (www.omneky.com) for sponsoring The Cognitive Revolution. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work, customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected]. MUSIC CREDIT: MusicLM:

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  To model a 600 by 600 image, you have to have the one million tokens. And current architectures just cannot support it. That naturally introduced a different problem. Like, we need a new architecture to solve this. And that's why we have this very efficient way of modeling involved multi-scale transformer. We are very excited about be able to directly in the future model the raw format of any file, any input, any modality. I think that's like a really exciting direction for us.

• 0:33Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas. And together, we'll build a picture of how AI technology will transform work, life and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, my guest is Lily Yu, research scientist at Meta and author of the recent hit paper, Megabyte: Predicting Million Byte Sequences With Multi-Scale Transformers. I first encountered this paper via a viral tweet from OpenAI's Andre Karpathy, who called it "a promising way to potentially

• 1:16Nathan Labenz?

  move language models beyond tokenization." If you're not familiar with the term tokenization in the context of language models, is the process of breaking natural language down into a fixed vocabulary of frequently occurring strings. In the case of GPT-3 and GPT-4, this vocabulary consists of more than 50,000 words, word parts, numbers and symbols. All language model inputs are chopped up into these tokens prior to embedding into numbers. And all next token predictions are selected from this set of tokens as well. In other modalities, there are other conceptions of tokens. Image tokens, for example, might represent small squares within the larger image. Tokenization is used because existing AI architectures

• 2:01Nathan Labenz?

  struggle to work with really long sequences of data. With the best performing methods available today, the compute costs ultimately get out of control. And there is a GPU shortage, as you may know.

• 2:14Nathan Labenz?

  So since sequence length is limited, some higher level, more semantic compression of the data is necessary. And tokenization is the first pass, still sort of hacky way that that's currently done. However, it does cause a bunch of problems, which Karpati explains a bit in the tweet and also links to deeper reading on, if you're interested. Suffice it to say for our purposes that it gets super weird. So weird, in fact, that there is a redditor whose username has been immortalized as part of GPT-3 vocabulary. And there are other super rare tokens that famous models seemingly can't be made to say at all.

• 2:52Nathan Labenz?

  Karpati summed it up by saying, "The list goes on. TLDR: Everyone should hope that tokenization could be thrown away." Maybe even more importantly, we may find general purpose strategies for multi-scale training in the process. And that's what we're talking to Lilly about today. Because in her strategy to eliminate the need for tokenization, she just might have made a more fundamental contribution. I'm not an algorithm expert. As you'll hear, I ask some pretty simple questions. But recently, I have come to believe that, given the presence of web-scale data and web-scale compute, it was really only a matter of time until somebody figured out a workable algorithm. Transformers are just one architecture,

• 3:36Nathan Labenz?

  as the human brain is just one architecture. And neither is the end of history. The basic idea of this research is that the megabyte architecture operates at multiple scales. Unlike in a single transformer, where all the layers tend to be the same size, in the megabyte architecture, information is first encoded in patches. Then there is a global model that shuffles all the information around. And then, the final byte-level predictions are again made by separate, local patch models, which can be run in parallel. It seems like this architecture just might have it all. For starters, yes, the byte level prediction eliminates tokenization. Now the model looks at everything as raw bytes, and it's always predicting just the next byte.

• 4:20Nathan Labenz?

  There are only 256 possible bytes, just 2 to the 8. So each one is ultimately just a series of 8 zeros and ones. It's kind of crazy low level, to me at least, if you think about it that way. But because everything is bytes, music, video, text, all of these are bytes on some level. If this does work super well, it will naturally extend across modalities. It also has more attractive scaling laws, subquadratic, as they say, and again, it's more parallelizable, too. These advantages allow it to work up to 1 million byte-length sequences, hence the title "megabyte." But what is a megabyte? In text, it's a million characters,

• 5:05Nathan Labenz?

  which does exceed, for practical purposes, Claude 100K's 100K token length. In music, it's about 1 minute of music sound. This architecture, because it's constituted of transformers, will continue to benefit from improvements to transformers more generally. So obviously, the key question is how it performs. In their experiments, Lili Yu and her teammates show that it does appear to be competitive with the standard transformer methods. So the next step at this point is for Meta AI to take this architecture up to Llama scale and see what happens there. It sounds, as you'll hear from Lili's comments, that they are very optimistic. But this is such an experimental science

• 5:50Nathan Labenz?

  that we can't and won't know for sure until they try.

• 5:55Nathan Labenz?

  Assuming it is successful, we could see mass adoption remarkably quickly. They have open sourced their methods, after all. But we still might find performance quirks and behavioral surprises for quite a while to come. And even if they're not successful, I think we should continue to watch this sort of research space closely as the paradigm could definitely still evolve in unpredictable ways. Now, I hope you enjoy this conversation about some cutting-edge architectural research with Lily Yu from Meta AI. Lily Yu, welcome to the cognitive revolution.

• 6:31Speaker 00

  Yes, a pleasure.

• 6:32Nathan Labenz?

  Yeah, I'm really excited about this conversation. You've recently published this paper along with colleagues at Meta called Megabyte: Predicting Million Byte Sequences with Multi-Scale Transformers. And as soon as I came across it on Twitter and saw the headline figure from the paper, I said, this is one that I definitely want to dig a little bit deeper into. So this has been a theme for a couple of recent episodes. Definitely want to encourage listeners to take a second and go look at the figure just to, you know, obviously picture's worth a thousand words, kind of get that visual in your head of the shape of the architecture that we're going to be digging into.

• 7:17Nathan Labenz?

  It should only take like a minute, you know, to kind of study it for a second before, you know, going on to listen to the rest of the conversation. But I think that will be extremely helpful.

• 7:27Speaker 00

  We also have all the maths and formulas in Figure 2 for some people who are really curious and want to know exact detail.

• 7:35Nathan Labenz?

  Well, that's what I hope to understand better at the end of this hour than I do coming in. So maybe just for starters, let me kind of bounce off what I took away from the paper and you can tell me, you know, if I'm missing anything or how you would frame it differently. The big thing that I saw, and you know, I try to avoid analogies wherever possible to describe AI systems because I think they, they so often can confuse and mislead. But when I look at this architecture, it does kind of look like a fork where you've got kind of the main, you know, body, the global model, and then you've got these kind of smaller local models that branch off from that. And so it has this kind of general fork shape.

• 8:18Nathan Labenz?

  And what I was thinking is, okay, this seems like. A sort of different take on a somewhat common theme lately, which is like models talking to models in high dimensional space, except you've created a hierarchy that can be trained end to end. So now we have kind of multiple transformers in a single architecture, all operating under one loss function and one optimization function. How'd I do for starters?

• 8:46Speaker 00

  It's a good start, but first, the idea of separating the agents surprised me. Second, the interaction is a little bit different when we think about different agents communicating with each other compared with end to end training of autoregressive language model. So when we see different agents, there's like different type of action coming from different agents. Here, in Megabytes, we try to do prediction, just everything is bytes, and everything has causal relationship, like different, the local model won't follow

• 9:29Speaker 00

  each other, predicting the bytes, the sequential bytes, even though they are prediction parallel, but it's actually a sequential, like, uniform bytes. Right, but architectural wise, yes, we have this like, kind of like, different patch. We use a simple concatenate operation, which is the, which is the patch embedder, patch embed in the paper. The different patch group, different bytes together, and they go to the global model, and then they separate out, as you said, like fork out. I think that's the correct understanding.

• 10:11Nathan Labenz?

  So it seems like there's several big advantages to this, and I'd love to hear you kind of describe each one and maybe talk about which ones motivated you and which ones you think are ultimately most powerful. But the, the big advantages seem to be, one, that There is the ability just to scale to far larger sequences than a typical transformer can. So you're literally predicting up to a million bytes. In this case, the byte is sort of the unit of prediction. So it takes the place of the token that people are, you know, familiar with if they're, you know, API users of OpenAI or Anthropic or what have you. But going up, you know, anybody would who's used these sorts of products would recognize that,

• 10:56Nathan Labenz?

  hey, we have been kind of living at like the 8,000 token level for a while. Now that's starting to expand. We've got Claude 100K, but now you're, you know, taking this up to a million bytes. So that's a big deal. There's also the performance or let's say the compute, you know, efficiency advantage, because certain things can be run in parallel. They're, they're both smaller. So I guess it's compute efficiency and kind of ready parallelization. And then third, related to the fact that you're predicting bytes, you don't have to worry about tokens. So you can kind of handle everything as like largely sort of raw data.

• 11:38Nathan Labenz?

  Tell us more about all of those and maybe start with the one that you think is most important.

• 11:42Speaker 00

  Yeah, so I think for this work, it actually come to hand to hand everything, basically. So what we really want to solve is, is get rid of headache of the tokenizer. So we want a tokenizer free language model. However, it's, why, there must be a reason why people do the tokenizer. One thing is to effectively compress information. So it's easier to do the compute, it's cheaper. So that means, see, however, the tokenizer indeed introduced lots of problems. So the common problem people may experience right now is, you know, on the text. On the text space, you have this B P token either.

• 12:27Speaker 00

  The biggest headache is the space. And sometimes you also see people do certain prompt engineer by prefix some random combination of the letter. And then it's gonna talk us to certain things and the model gonna continue generation such a nonsense. That's all drawback of token either. Of course, another difficulty of token either is see, oh, one day you want to use chat G P T is very powerful. Let's say you want to use a weaker large language model. And you want to use in bio, you want to use use in chemistry, you want to use in some foreign code. And then the token either is out of distribution for your new domain.

• 13:09Speaker 00

  And then you're gonna get problem with like fine tuning or something. So that's what people experience with the text. However, it's also a big, big problem for multi model. See, you know, right now we are focused on text and then everybody have to using chat G P T. But we also know like chat G P T four is already multi model. And in the future, multi model gonna be a big thing. However, it's a big, big issue when you want to model image or you want to model audio. For example, in image, image, there are two main architecture to solve it. One is a diffusion model, which take pixel.

• 13:52Speaker 00

  But diffusion model is very expensive and the like it has its own issues. How if you want to do auto regressive image, then you you experience even longer sequence. Like, for example, in our paper, we do six hundred pixel by pixel image, that's already one million bytes. And then people use V Q G A N as image token either. So text tokenizer give you headache, but the image tokenizer can actually give you light mare.

• 14:25Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors. Omnikie uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work, customized across all platforms with a click of a button. I believe in Omnikie so much that I invested in it, and I recommend you use it too. Use Cogrev to get a 10% discount.

• 14:46Speaker 00

  Because it's lossy, it's actually truly lossy. So text tokenizer, the tokenization may not be reasonable, but if you do the input string, tokenize, de-tokenize, you get the original string. But for image, if you do image input, you do tokenization and de-tokenization, you get a slightly different image. Some details is off, some color is off, some, yeah, like finger get mixed up. And then if it's original, there's words, you cannot see it clearly. Same thing, same thing for audio. It's also a lossy process.

• 15:28Speaker 00

  So that's why we really want to get rid of tokenizer, no matter it's to enable, you know, in the future, the true multimodal, multimodal type of modeling, or to be able to mix it, easily mix image, text and audio, or to be easily adapt to new domain. We really want to get a tokenizer. So that's one of the biggest problem we want to solve. So that come to another question. Nothing is for free, right? As I just said, the reason people want tokenizer, because you cannot just handle such a long raw input sequence.

• 16:09Speaker 00

  For image, like to model a 6x6, 600x600 image, you have to handle 1 million tokens. And current architecture just cannot support it. So that naturally introduce a different problem. Like we need a new architecture to solve this. And that's why we have this very, very efficient, very, very efficient way of modeling, involve multi-scale transformer.

• 16:37Speaker 00

  So yeah, that's naturally, I think it's it's it's a one stone kill two birds situation here. We removed the tokenizer and we have this very efficient architecture to support it. And we demonstrate that across different modality, across text, image and audio, we can both achieve like state of art performance. Okay, so that's that that comes to another question, right? Can you combine the tokenizer and this model architecture to enable very, very long sequence modeling? So I think that's that's definitely part of our future future work and we're we are thinking

• 17:18Speaker 00

  very, very, very carefully. So for the current paper per se, that's not what we're trying to solve.

• 17:26Nathan Labenz?

  So that's all very helpful. So the big motivation is, and I didn't even mention the, you know, natural extension to multimodality in my too long, my already too long first question. So thank you for bringing that up as well, because that's definitely a key part of the result is that you have the same architecture working across different modalities.

• 17:48Nathan Labenz?

  Let me just try to understand this this core figure. So again, listeners, well, look at the figure. This is figure one from the paper. It's, you know, we'll put a link in the show notes, very easy to find. And and pretty easy to sort of get a general gist for, but there are a couple of things I want to dig into a little bit more. First, the input side, everything is bytes, right? The from this is all bytes in and bytes out and the architecture itself doesn't care what these bytes represent, right? So I got that. What is a little less clear to me is what are the patches

• 18:27Nathan Labenz?

  doing on the input side? Because I if I if I understood correctly, and maybe I didn't, I understood that the embedding was lossless. So then I was kind of like, well, if it's lossless and it's all just going into this global model that kind of sits at the middle of the whole thing.

• 18:47Nathan Labenz?

  What is the meaning of those patches? Or what is the function of those patches? As opposed to just saying, you know, here's all the bytes that are inputs. Just feed them directly into the global model. Like what's happening in those patch embed modules?

• 19:04Speaker 00

  So there are a couple of things here. See, like in figure one, right? We have this whole string as input and they are actually input as bytes. So we have sixteen, sixteen bytes. So the sequence length is sixteen and we separate them as patch. So it's four by four. That means each of the, either the local model or the patch embedder. So we have four duplicate of them and each has input of four.

• 19:42Speaker 00

  And the global model, the Go blow model, they only see the patch's input.

• 19:50Speaker 00

  So each patch take four bytes as input and they're gonna give you back one vector. So the four bytes already merged into one vector. And then now the global gonna only handle four, uh, gonna only handle four vector instead of sixteen. So that's why the global model was able to say like much shorter sequence. Or equivalently, if it's shorter sequence and you have the same compute, you can make the model bigger. So that's a big difference. If you just put everything into the global model raw, it's say sixteen, uh, six, sixteen sequence length, sixteen as a sequence length.

• 20:34Speaker 00

  And now with the help of the patch embedder as a local model, the global model only needs to see sequence of four. So it's much shorter and much efficient. So one thing, another thing which uh which I think is also relevant to your later question is this padding. The padding is actually really, really critical.

• 21:02Speaker 00

  So in current language model is doing next token prediction. So how they do next token prediction is actually by padding. So if you look at the upper part, the local model, you have this, for example, megabyte. Uh, the mega, the M E, the mega, the full uh full capture input. So for the local model, you actually need to pad the beginning, this padding token. So the model, how it works is they see the padding token and they predict the M letter. And then they say the M letter, they predict the E letter.

• 21:47Speaker 00

  So that's uh make sure they don't see future, they don't see future. Everything they predict, they only see things before it. That's the essence of the next token or in or here in our paper is next byte prediction. So that's why we they need to add this token. So so what about the, but however, because our local model can see all the preference patches. So when we, so the input to the global model, you are not only going to patch a single letter. You are, you have to pad a whole patch.

• 22:27Nathan Labenz?

  Are all the patch embed modules the same? Are they all identical? Like even down to, you know, the lookups, the the sort of conversion of a byte to. I guess it's even, it's a little different than what I normally think of as embedding, right? Because I normally think of a token having a sort of long form vector representation. But here, because everything is bytes, there's only 256 bytes, like they're just represented.

• 22:57Nathan Labenz?

  Explicitly, literally, right?

• 22:59Speaker 00

  Ah, I see. So actually, we still have embedding. Again, as you said, but embedding matrix is much, much smaller. So for example, in a normal large language model, the vocab size is either thirty two K, sixty five K, fifty K, depends, it depends on the, uh, you know, how the research choose pick for their task. So for that, you have a large vocab map to a embedding space. It's actually the same. It's just the vocab is so much smaller. The vocab is only two fifty six. And for the two fifty six, you map to a embedding.

• 23:41Speaker 00

  Again, as you said, because the vocab is much smaller, so the embedding size we need should be also be smaller. Right? Like, otherwise, it's it's such a waste if you have like four thousand uh embedding dimension to capture a vocab of two fifty six, that doesn't make sense. So in our paper, actually, the embedding size we pick as thirty two, which is very, very small. So you are mapped to one hot vector uh of two fifty six. That's what people to represent a naive vocab is one hot vector. And then you change that to a dense vector of thirty two,

• 24:26Speaker 00

  like one dimensional dense floating point uh vector with dimension of thirty two.

• 24:34Nathan Labenz?

  It's an eighth uh as big.

• 24:38Speaker 00

  Yes, yes. So we found that it's not a big, uh, it doesn't really impact that much. We can actually go even smaller, but um it's actually a very small module. It didn't optimize that much.

• 24:53Nathan Labenz?

  So there's still this embedding and you're learning your own embeddings as part of this process? There's not like an off the shelf embedding for this, I suppose, right?

• 25:03Speaker 00

  Yes, everything is not end to end. So basically, you know that matrix, that's more matrix.

• 25:08Nathan Labenz?

  The patches are the same. So it doesn't matter if I'm in patch one or patch two, patch three. If I have the same bytes, that gets converted to the same embeddings and the same. Data flows into the global model regardless of which patch that is coming from.

• 25:28Speaker 00

  Yes, hundred percent. So there's there's slight difference. The difference is the position embedding. Yeah, yeah. So position embedding, there's like separate position embedding for the local model and for the global model. You're gonna know which patch you are coming from and which letter inside the patch. So the model actually know your position fully. But the embedding is the same, actually. It's exactly the same for transformer because, you know, transformer has no sequence. Everything's parallel. The matrix has no idea of the sequence. And every position information is not from this position embedding.

• 26:12Nathan Labenz?

  Helpful and clarifying. So I'm up to, I think I understand everything now, up to the global model. And the global model, if I understand correctly, is basically a transformer, right? It is so and it has all the normal features that we would expect from a transformer, which is to say your residual stream, your attention blocks, your MLP blocks, your nonlinear, you know, kind of gate. And the difference in this case is that its output

• 26:46Nathan Labenz?

  gets directed to all the, instead of being the final output, it's its output is the input to each of the local models that are going to do the final set of predictions.

• 27:00Speaker 00

  That's perfect. Yeah, that's very, very correct.

• 27:03Nathan Labenz?

  Going back to that panning concept. So, I mean, here's a really naive question. When I'm thinking about predicting text, and maybe the answer is going to be like, don't think about text, maybe think about a different modality for this. But I'm looking at the at the diagram and it's like you've got kind of megabyte, tran are the sort of, you know, 12 letters that have been predicted. And then, you know, as it works its way through, then the output is that the next, you know, the the fourth patch is predicted. And now at the end, you've got, you know, four more characters. Megabyte, trans, and the last four are S F O R. So you predicted those that kind of chunk of four letters.

• 27:43Nathan Labenz?

  My first question is, why do we even need all of the different local models? What are they doing? Like, what is even happening there? Because my naive, what I'm, there's got to be something wrong with my understanding. But what I'm kind of thinking is, okay, you take all this stuff. You do the embeddings, you feed it into the global model. The global model does its thing. But it's really only the last patch that's predicting new content.

• 28:12Speaker 00

  Aha, I see what you mean. Yes, yes, yes. Okay. So this actually come to, this actually come to how language. So, so, okay, this is a language model, how language model works. Even with tokens, right? Even with tokens, you predict every words. So with a sentence, for example, today is sunny. You still predict today with empty input. And you predict is with today and so on. So you don't only, you are not only predicting the sunny. So it's the same here.

• 28:55Speaker 00

  So basically, the model need to know that. See the first ever, the left column, for example. The model need to know that with a padded patch. The very red, the most red color. The pad with a padded patch with basically empty patch. It's need to predict this mega. That's equivalent with you say empty and you predict today. And they say the patch plus mega is need to predict the byte, the B Y T E. The one.

• 29:35Speaker 00

  Efficient thing about the autoregressive prediction is they predict every word. That's why this learning is efficient, so. And that's done by this, uh, that's done by two things. One is this, uh, padding, the other one is tri- triangular masking, the causal mask. Yeah, so, so, so basically, we also want to make a prediction on every local patch. It's the same, so we want to predict, so that the loss has rich information about the modeling.

• 30:14Speaker 00

  We want to, so there are four local model and every local model's prediction gonna compare with the ground truth and get the loss. So we're gonna have, uh, if we simplify it, we're gonna have sixteen loss. So every predict, cause there are sixteen, um, character. So each character is, uh, has a ground truth to compare into and we're gonna get a loss and, and in the end, the loss is average.

• 30:43Nathan Labenz?

  So maybe just starting with a, you know, vanilla transformer that I'm most familiar with and everybody in the audience will be more familiar with. I don't know how well understood this is, I'm, you know, it's, it's easy to kind of forget and just kind of background this fact. But when you do a forward pass through a transformer, you are not just predicting the final token. But you are actually generating a full set of predictions, which, you know, in the case of like a GPT three type of model would be. Full distribution across the entire vocabulary, fifty thousand tokens at every token position.

• 31:27Nathan Labenz?

  Now, you're not using that at the end because you already know ground truth there for what the, you know, what all the current, you know, what all the input tokens were, but you, you can kind of examine those if you want, you know, assuming you have access to the outputs, which increasingly, you know, you have to run your own thing to have that, but you can see along the way that, oh, this token was a surprising token, this token, you know, was an expected token, et cetera. So there's lots of kind of interesting things you can glean from that. But then also in the vanilla transformer, there is this kind of look back mechanism that happens through every layer, right?

• 32:10Nathan Labenz?

  So I guess the kind of, when I think about a traditional, you know, traditional transformer, I'm thinking there's two reasons to do that. One is what you said, which is because I'm making a prediction at every single token, I have that many more predictions that I can feedback and use to power my optimization. But then also with the regular transformer, up until the last attention layer, you have some possibility, right, for information from earlier tokens to influence the last token that you're actually here to predict, right? Okay, now flipping over to the megabyte architecture, it seems like maybe if I'm

• 32:54Nathan Labenz?

  understanding correctly, one of those holds, but maybe not both. So you would still have, you make the prediction for every single byte, which in this case is you're making a 250, because there's 256 possible byte values. You're making a prediction for every possible byte value at every possible byte position.

• 33:16Nathan Labenz?

  Because you're doing all those predictions, you have all those loss measures that you can then use to power your back propagation optimization. That's presumably particularly important when, particularly useful for like informing the global model. But in this case,

• 33:40Nathan Labenz?

  Once the, when you're actually doing inference, once you're past the global model, now there's no more communication, right, between the local models. So I get kind of one of those two benefits, but not.

• 33:54Speaker 00

  I see. So, so, so, so I, I have to comment one on this. This one comment is like, in the autoregressive model, I feel like it's underappreciated that we make prediction every token. So basically, if you have sixteen token, you make sixteen prediction. And that's different with the representation, representation model, which is BERT or Roberta. So that's like encoder model. So GPT three, GPT two, GPT three, uh, extra show, they are decoder only model. And we are decoder only model two.

• 34:34Speaker 00

  So for the encoder only model, the loss is only on the masked tokens. That's only fifteen percent of the tokens. So you do one forward pass, you only get fifteen percent, uh, loss on fifty percent of the, fifteen percent of the tokens. While for decoder only model, you do one forward pass, you get loss on every tokens. I think that's why, uh, personally, I feel that's one of the big reason why decoder only model gradually gets so and so popular and being able to enable, you know, very, very big model, very, very, very powerful model, very, very uniform model.

• 35:16Speaker 00

  That's one of the big reason, like, because it can learn so efficient. And, uh, you get rich, uh, loss information and you are able to leverage the data really well. So that's a side comment, basically, we, and, and of course, that's what we want to leverage to make a prediction on every token. And then that's supported by transformer. So basically, think about this. This could be sequential. So what that means. During inference, you have this padded patch as input. And that goes to the local model. And the local model gonna see this letter one by one.

• 35:59Speaker 00

  They gonna see the pad, they predict one letter M. And then they take the M as input. M and pad together as input, they predict E. And then they take the pad and the two letter as input and predict G. So theoretically, it's supposed to down sequentially. They predict one by one. But thanks to the parallelism of the transformer, we're able to during training. During training, you can do the feed forward and get every loss. And then you can get the loss all together. So that's one advantage of transformer architecture. That's also one why like everyone is using transformer.

• 36:42Speaker 00

  Of course, I'm not saying other architect, like people are also making efforts trying to make other architecture scalable and the parallelism. But transformer is native for this and it works really, really well.

• 36:56Speaker 00

  And then, as you said, for inference, yes. So basically inference, we decode patch by patch. We decode, see if we want to decode the trans, T R A N. We need to decode the first patch out and take the first patch as input to fit into the global and predict the second patch and take the first, first two patch output as input to predict the third patch.

• 37:26Nathan Labenz?

  And then within each patch, of course, byte by byte.

• 37:29Speaker 00

  Yeah, yeah, yeah. And within each patch, byte by byte. So that's a small difference between, you know, how model works in training and how model works in inference.

• 37:37Nathan Labenz?

  I'm with you on the training portion. If we are using the megabyte architecture for inference, though, is there any reason that I have to run?

• 37:52Nathan Labenz?

  The earlier patches, or if I'm if I'm just concerned with inference, could I just run the final local patch?

• 38:02Speaker 00

  So, yes, if you know the previous patch, you can run the last patch, right? So that's come to see how you want your language model to work. So this is a question, for example, if you ask your chat GPT, I don't know, a coding question, and then the model gonna generate like the coding, like the the whole coding answer. So that means your question, they don't need to run one by one. They can run that parallel. But everything the model generates has to run token by token. So it's the same here. If you know the previous patch, you can input that and run that parallel.

• 38:47Speaker 00

  But everything the model don't know and the model need to continue generate is need to run byte by byte.

• 38:54Nathan Labenz?

  So, but I'm stuck on this point only because it will either help me confirm or, you know, poke a hole in my understanding. But if I am doing new inference

• 39:08Nathan Labenz?

  with the Megabyte architecture, I have to run the whole thing in a loop because I'm predicting one patch at a time and I can only predict one patch at a time because any future patches depend on that that patch first. That's basically the meaning of autoregressive. But when I look at figure one and I'm thinking, okay, there's four patches here. Do I in practice, could I just like not run the first three patches at inference time? And just embed, you know, do all my local embeddings, do my global model, and then just run the final patch?

• 39:45Speaker 00

  Do you know about the first three patch? Is that given?

• 39:48Nathan Labenz?

  Yeah, so let's say I, let's say I'm doing, you know, exactly this example. So I've my first three patches are Megabyte, Tran, and I input those. They get embedded, they get passed into the global model, the global model does its transformer thing. It then produces outputs that would feed into all four patches. But I'm trying to confirm what I think I understand, which is that at that point, there's no further information exchange between the patches. And so if I'm only concerned with the new patch output, then I could just say, okay, I'm going to ignore those first three patches of output from the global model and only process the fourth patch from the global model because that's all I really need the output for.

• 40:34Speaker 00

  Yes, that's hundred percent. See, like after this global model, right? You get four labeled as edge global out in our, the hidden representation of global out. At this point, you can, you can only take the last one. You can only take the last one and then you're gonna, you're gonna use that together with the local model to decode the last bytes you need. Yes.

• 41:00Nathan Labenz?

  Okay, cool. But in training, you do have to do, or you don't, I guess maybe you don't have to, but you do run all of them because they all contribute to the presence of all these loss values that then you can use to optimize the model end to end in the first place.

• 41:18Speaker 00

  Yes, yes, we want the loss from every, every byte. Yes.

• 41:22Nathan Labenz?

  So that is really interesting. This is not like a strong theory of mine, but increasingly I see kind of a lot of things and I'm like, a lot of these things seem like isomorphic to other things or kind of nearly so. So I was thinking, is there a sort of modified like vanilla transformer that I could kind of imagine that would be akin to this? And what I came up with was like, you know, typically the transformer has like the same width, right? Throughout it has kind of the same dimension at each block. But here you're kind of shrinking the dimensionality of the input in order to have a more efficient

• 42:05Nathan Labenz?

  global model and then kind of chunking in order for kind of more efficient parallelization.

• 42:12Nathan Labenz?

  Do you think you could kind of do something similar where if you devise a transformer, what you said, like the early layers, you know, are gonna be wide, but then they'll go through like a bottleneck and then at the end they'll get like particularly narrow. Does that seem like it would potentially have some similar value?

• 42:33Speaker 00

  Yeah, yes, yes, I think, uh, so there are a couple of things. I think one thing we really want to do is try to take advantage of the powerful transformer. So in some sense, the global model is intact transformer and the local model is also intact transformer. So that's something actually by design because as you said, there are many, many optimization of the transformer architecture and some of thing is very memory efficient.

• 43:09Speaker 00

  Yeah, some of thing is flop friendly. But in the end, what we really care is how well it scales. So I think there are lots of great work, but they need to test at large scale. So, you know, nowadays, I think most people do large language models still using the dense transformer because it's most well tested and it's perform well when you have a lot of model, a lot of data and a lot of compute. I think that's why we don't want to touch that. And in many sense, see one day people found efficient attention mechanism, we can just swap in, right?

• 43:50Speaker 00

  Because we can swap in the attention in local model and swap in the attention in the global model with this new attention mechanism. So that's by design. Second is this auto regressive prediction and this masking is actually very, very important. So basically, if you want to do also auto regressive transformer, you have to guarantee that. Otherwise, you have information leaking.

• 44:14Speaker 00

  And you you have information leaking and then your model will learn nothing. So you know, like the large language model learning, you need to make the task also challenging. If you either know the ground truth, I just copy from my input. When you attach the transformer locally, it's really hard to guarantee that. So basically, because we have this triangular masking on the input. If you arbitrarily uh rescale your inner dimension, or as you said, make the model wider or narrower,

• 44:55Speaker 00

  you have to add very, very complex masking strategy inside, and it's really, really hard. So that's the thing. And third thing is, there's something quite pretty about how we design this global model. So basically, the global model has shorter sequence. So a shorter sequence has not, actually has a couple of advantage. One is on attention. That's what when when people can't think about the sequence, that's what people care the most. Because apparently, this quadratic memory scaling give people issue when you have long sequence. But if you make the sequence shorter, it also reduce the feed forward. We also discussed in the paper.

• 45:40Speaker 00

  So in nowadays, feed forward is most compute uh expensive part of your model. See, if a G P T three, it's only uh the attention only take two, three percent of your whole compute. Feed forward is a big chunk. And the way we do it for the global model, it's one over P. Like the uh normal flop of a transformer, of a naive transformer. So see, we take the passes of eight. Now our global transformer only need to take one, one, one ace of the compute. And we can make it bigger and more powerful. I think that's why we stick to this current design for many reasons.

• 46:22Nathan Labenz?

  Yeah, that's really interesting. And then I suppose, too, it also sort of suggests the potential for.

• 46:30Nathan Labenz?

  Modularity, right? I mean, we see all these projects where certain, you know, a language model is frozen and combined to something else. And I can, as you're, the more you're talking about this, I can start to easily imagine freezing different parts of the model, you know, having different kinds of local models that could perhaps be swappable, depending on, you know, you may only need to fine tune the, the local models for certain tasks or, you know, who knows what, right? But it seems like that would be, this architecture would lend itself quite nicely to modularity downstream.

• 47:06Speaker 00

  Yeah, yeah, yeah, yeah. I, I totally agree. I think there's like, there's actually in the VOM, like video language model, there's such architecture to, to adapt to like, say, a pre-trained, um, so you're adding like, uh, image understanding and maybe even image generation functionality on top of, uh, pre-trained text, uh, language models. There's, it's, it's, it's quite, uh, in essence, it's quite similar. Yes.

• 47:35Nathan Labenz?

  Yeah, almost like a different head, so to speak. If you had a similar, maybe even the same concept, but yeah, you have a, i- if you can train a single global model that can handle different kinds of inputs and represent them in a meaningful way, then you could have different local models that create different sorts of outputs based on that single understanding.

• 47:57Speaker 00

  Yes, hundred percent.

• 47:58Nathan Labenz?

  Yeah, this is really interesting. So, and it's no, uh, the more I learn about it, it, it makes sense why, uh, some of the leading thinkers in the space have been very excited about this paper. You've kind of covered a little bit already, the, how you choose the patch size. It seems like that's basically a trade-off, the way I was thinking about it. And you can, you know, complicate this, but I was kind of thinking in the limit, if you just had one giant patch, then you basically just have one giant model or two kind of stacked, you know, global models, which would have all the normal downsides of that. If you, on the other hand, you know, took the other limit, then you would have,

• 48:40Nathan Labenz?

  you know, kind of an insane number of patches. And I guess I don't know exactly what goes wrong there, but it seems kind of silly. So you're kind of looking for this happy medium in the, in the middle that, that trades off these two. Extremes and that's an experimental process, it seems.

• 49:03Speaker 00

  Yes, yes, a hundred percent. As you said, either you choose patch size as the whole thing, or you choose patches as one. That's like both two extreme. We just become a normal transformer, basically.

• 49:20Nathan Labenz?

  Okay, so the patch size of one, yeah, it doesn't have to do anything. Then it's just the global model choose does the output. Okay, that makes sense.

• 49:26Speaker 00

  Either ends, we become the naive transformer and our model actually also. Oh, I mean, in our experiment, we also uplift that. And actually, we found out with combination of both two is really, really helpful. Then it comes to the question, how to choose a patch size? So theoretically, we should like, if you want to optimize the um memory of your attention, we have a formula how to help you how to choose the patch size. But we we actually pick the patches quite heuristically right now. See for text, we try to stay, don't stay uh too much away from B P token.

• 50:07Speaker 00

  So right now, B P token normally have uh one B P token corresponds to do around about four bytes. So we we cannot pick a byte, we pick patch size of eight to be not far away from that, but also compute efficient. And for image, we kind of like pick patch of eight by eight or sixteen by sixteen. That's what people do with uh the image tokenizer. So basically, how we pick the patches is kind of like inspired by how people do the tokenization. But we don't want the retokenization uh at all uh so in our paper.

• 50:45Speaker 00

  So that being said, it's actually uh good. In our paper, we also studied uh how to pick certain patch size. But I will say that's not comprehensive at all. Like, we do need more study wide range of that. And especially, uh that's going back to the long sequence story. If we make the local embedding, local model as strong as a normal transformer. See two thousand tokens, right? The local model, two thousand tokens. And and we only have see eight or sixteen of those patch.

• 51:27Speaker 00

  Then that means we're training a sixteen K or thirty two K like normal transformer. So I think I think there's a whole wide spectral how we can pick the path size. But interestingly, overall, we find a right range of patches actually works quite well for our task. So that's uh that's something definitely needs more analysis, you know, which region and how we uh do the motor size to.

• 52:00Speaker 00

  We do find it's quite stable to the past size uh at this point.

• 52:05Nathan Labenz?

  Yeah, it's interesting. These um.

• 52:08Nathan Labenz?

  I guess the the intuition for that would maybe just be that there's some curve and you know, it's kind of relatively flat near the local minimum. And so you have kind of a range there that you won't see too much difference with. Maybe you could just kind of describe the different multimodal experiments because we've talked largely about text, but there's also stuff pertaining to image and even music. Maybe just give like a little rundown of kind of even just what those data sets are, because I don't think people have a great sense, you know, when they see a name of a data set, you know, what even is that task? Uh so give us just a sense of kind of the generality that you've been able to demonstrate.

• 52:48Speaker 00

  Yes, yes, yes. So basically, uh this auto-regressive language model has been very, very popular on text. But people also have been doing that for both image and uh audio for very, very long time. However, for image, uh due to the challenge I just described, the image has very, very long sequence. Uh it's not as popular, but even earlier, right after transformers, image transformer. However, they have to compress inputs, so it's also a lossy process. The same as audio, like back in, I believe it was twenty sixteen, there's like WaveNet that you predict the audio wave um one by one.

• 53:31Speaker 00

  So the task, basically, it's the same. It's predict something, the something's byte, predict byte one by one. And for image, it's normally come with image size, say, you know, 4K image. So the 4K image, when you read the raw image, it's 4K times 4K. So that's like the square, times 3, 3 is normally the RGB channel. That's many of bytes, that's your whole image. And for us, it's like, we just take it, and this is like 2, this is a 3D matrix. One image is a 3D matrix, and we just flatten it out.

• 54:17Speaker 00

  And that's your input sequence, that's then become a 1D sequence. And now you can just treat as text, and you predict one by one, the one by one. Uh, there's a little bit caveat of that, we actually didn't, it's purely flattened out. We take, uh, you know, a patch by patch, like 8x8 patch and 8x8 patch. Uh, to, and then flattening it out, that's illustrated in the image. But, um, but for simplicity, it's like, for image, you just take this, uh, input pixel value. And you sit, you make it 1D, and you predict one by one. It's the same for wave, so when you read the, or audio, audio is represented as wave.

• 55:01Speaker 00

  So you can read the wave input, just change, that, that becomes a 1D byte sequence. And you predict one by one. So basically, that's a, that's a, that's a bigger, uh, difference. And for image, image, like, uh, you just like, uh, predict one by one, and compare with a ground truth. Then that's give you per bit, per, uh, bit per bit, per dimension. There, there are many way you call the metrics, but it's all the same, basically. How well can you, uh, accurately predict, uh, one value out of 256?

• 55:42Speaker 00

  And then we we actually achieve state of art for the smaller image, which is actually the same state of our value, matching state of our state of our value on image sixty four. And of course, we don't want to stop that. The reason we compare image sixty four is people rarely do longer sequence of image. It's just because like earlier model didn't support that. So but to illustrate the capability of our model, we did image of two fifty six as well as image of sixty six four, six forty pixel value. That's equivalent to one meaning bytes when we flatten it out into one D.

• 56:28Speaker 00

  So that's the image experiment. And on the audio experiments, basically, we take some audiobook data where basically audiobook data as well as some speech data. Basically, think about in the end, it's just wave and you want to know how well your model can accurately predict or recover those wave information. So there's, there's actually very, very interesting thing here is normally the audio is encoded not by the eight bytes, not by the big bytes of eight, which equivalent to two fifty six

• 57:11Speaker 00

  value is actually sixteen bit steps. So that means like you read one value and that's not pick one out of two fifty six. That's actually pick one out of sixty five thousand. That's actually give people earlier in, in, in earlier research, that's actually give people very much headache. Because the softmax over one over sixty five thousand is very, very expensive.

• 57:43Speaker 00

  So what people have been done is either uh do a cluster based approach to map each of the value into one out of two fifty six. Or they do hierarchical decoding. So on the decoding, they do like tree like decoding. Because into something and smaller being value and then further decode. So that has been a big issue. But what we are doing here is like, we just read the file bytes.

• 58:14Speaker 00

  So we ignore like, okay, what's the what, you know, the in particular encoding and decoding of audio file. We just ignore those. We take the file and read the bytes and model that. I think that's actually very interesting and that's the future direction we want to explore. Basically, for certain file, forget about its image, its audio or its video. You just read the bytes. You just read the byte value and then you model that. I think that's that's gonna be very, very interesting.

• 58:50Nathan Labenz?

  I want to kind of cover results and then I wonder if there's any, you know, things that you would flag in terms of like, you know, what at a high level, you know, are there trade offs here? The results you kind of present in two ways, as I understand it. One is basically that you kind of compare a, I believe it's a compute matched experiment where you say, okay, I have a certain compute budget. I'm going to train a transformer, you know, classic transformer. And also you do the perceber architecture and then you've got the, you know, the new megabyte architecture, but same compute budget. I'll train them each as well as I can on this given compute budget and then I'll compare how they do in the end. And

• 59:36Nathan Labenz?

  and that is done on a byte by byte basis, right? So we're using kind of the same uh encoding in that experiment. And finding that when you're dealing with bytes, like the megabyte architecture just blows away the others. But then the other question is, okay, well, people don't usually do transformers in that way for all the reasons that we've discussed. So then the other approach is let's take a data set, same data set. And here I was a little bit confused, but it seemed that you're taking the same data set, training a transformer, perceber AR and a megabyte. On that same data set, you know, say it's a text data set.

• 1:00:19Nathan Labenz?

  I wasn't quite clear how the compute budget compared across those different architectures. But the key punchline is the results are comparable.

• 1:00:31Speaker 00

  Yeah, so it's actually, uh, it's actually experiment hard for us to do for the, I mean, the latter case. Let me first explain why compute match is so important because it's actually quite well, quite, quite known to everyone, basically. For a quite scalable and capable of model architecture, adding more data as well as adding more compute hours. You know, with the right setup, it normally gets better as you train longer and, yeah, train longer. And adding parameters to it. So that's why we think compute match is always, is very, very important. It's just like opportunity cost.

• 1:01:14Speaker 00

  If I take the local transformer and I just add on top of the global transformer. That's, that's for free, right? It's definitely like most of cases is gonna get better, but what we want to consider is opportunity, opportunity cost. If you want to add a global model, you have to look, make your local model smaller because you only have such compute. Uh, so that's the same, so that's the same, that, that, that, that's a general idea. We want to do compute a match. Uh, so that's why we implement this perceber AR in-house. Of course, we run the experiments to validate. We get a comparable experiments from the original paper.

• 1:01:56Speaker 00

  But in the compute experiment, we run the baseline, which is perceber AR. Of course, and another baseline, which is naive transformer. Uh, we run the three model all ourselves. And the three model are seeing the exactly the same data. They are training for exactly the same, same hours. So we think that's a fair comparison. And that's what really makes us happy to get a confidence why a megabyte is good. However, as you said, most people don't do this way. When they publish paper, they don't tell you exactly how many hours they train on which GPU. What's the best size, et cetera, et cetera.

• 1:02:40Speaker 00

  So we also want to convince, also let us know, do our way comparable with other implementation? We want to compare to more models, but many model, we don't know the detail. So in that case, uh, for the benchmark run, we try to run the model longer. Until the model converge, that means like the validation loss is not uh going down again. And we take that value to compare with what the people has been reporting in their paper. But again, we don't know exactly what's happening in their paper. But that's idea, we want to at least reach a ballpark or at least uh we know we are not using more compute than them.

• 1:03:25Speaker 00

  But we're still get comparable performance. So I think that's uh in high level, two sets of experiments we are running.

• 1:03:33Nathan Labenz?

  So this is table three, right in the in the paper. And the so the bottom section, the bottom three lines are all the experience that you ran with the byte level encoding. And you're basically showing there megabyte dramatically outperforming the other architectures with the byte level encoding. And then comparing to the top section, it's that validation column. Where and I guess also the test column where you're sort of saying, okay, now. And this is where I'm a little bit unclear as to exactly like, maybe you're also saying like, it's not fully clear. They it's not always known exactly what somebody else did.

• 1:04:16Nathan Labenz?

  But is if I compare the megabyte, you know, row at the bottom to the. Let's just take the, you know, the top one, the transformer XL from the top. Do we know?

• 1:04:29Nathan Labenz?

  The relative compute budgets of those? Or we just know like the data set it was trained on, like what is the constant in that comparison?

• 1:04:37Speaker 00

  Yeah, the the constant in the comparison is only data set. Unfortunately, many of the work, we don't know how much they, how much compute they used. But the one thing is like PG nineteen is a small data set. So basically that means you run multiple epochs of it. For other in-house experiment, for example, see table two. Those data set are much larger than PG nineteen. But unfortunately, uh,

• 1:05:09Speaker 00

  people didn't really report all those data sets. So yeah, we we are definitely doing like confined optimization here.

• 1:05:17Nathan Labenz?

  So again, the bottom line is if you use the byte level, or byte level encoding, byte level prediction, then the megabyte architecture is blowing the other side of the water. But if you're kind of comparing on, you know, let's say, something like opt, you know, each each architecture working in its optimal condition, then you can get kind of comparable performance. And you know, then the punchline is, but you still get all these other advantages of the of the megabyte architecture that we've, you know, that we've discussed.

• 1:05:50Speaker 00

  Actually, uh, that's not a full story. I think for the first story is for bytes. We believe megabyte is one of, is the is the best. But what people are interested in too is like, see, if you have byte input, what's a, what's, what's the optimal? We already know, uh, megabyte. Uh, we are pretty sure. But another question is like, on byte prediction task, we are the best. But is a byte prediction task as a language model task? Work as well as when you have tokenizer for the language task? I think that's what we try to get inside from table three.

• 1:06:32Speaker 00

  So we didn't really answer that in this paper, but we're working on that right now. It's like, see, you know, byte level has its advantage, but we want to know, can we just replace the tokenizer as a whole in the future? And get wide, wide adoption, so we don't need to worry about tokenizer like with all the troubles I just mentioned at the beginning of the, of our, our conversation. So that's, that, that's something we are currently working on.

• 1:07:02Nathan Labenz?

  So there's a conversion process there where you're basically saying, we're going to re-score our byte level prediction output as if it were delivered in token form. And then we can kind of compare on a more apples to apples basis.

• 1:07:19Speaker 00

  Yes, yes, that, that, that's exactly what, what happened here. When you see table three and, and, and table two, like their value is like in totally different range, right? Because table two also have PG nineteen results and the B P B is around one.

• 1:07:36Speaker 00

  But when you see table three, that's converted, that's converted to, to token way like word level popularity. And it's in the range of forty two. So there is a conversion between them. They are actually, see table three, that's a word level popularity of forty two. Actually is equivalent to point eight B P B. So that's, uh, maybe that's a confusion. Uh, the reason is because the top session, the top other work, they don't report the B P B value. So there's no way we, uh, we can compare, so we have to convert our results to word level popularity. Yes.

• 1:08:21Nathan Labenz?

  The upside of this hopefully is pretty well established. You've got these, you know, the avoidance of all the tokenization, you've got the, you know, natural flexibility to handle all these different modalities. There's the compute efficiency, the performance is all there. What do you think happens as you scale this up? I mean, there's a couple of interesting little wrinkles in the, in the paper, particularly around the strided inference. Where there seems to be this.

• 1:08:51Nathan Labenz?

  There is some sort of performance penalty associated with the boundaries between the patches. And so you have an interesting strategy for overcoming that. But then I've also seen, you know, most of the commentary in this paper online has been like effusively, you know, positive and really hyping it up. But one of the more interesting things I saw, and I really don't have an intuition for this myself, but I wonder if you do, was somebody said, you know, this looks amazing. I do wonder if it will demonstrate the same sort of in-context, you know, few-shot learning type behavior that the vanilla transformer does.

• 1:09:30Nathan Labenz?

  Maybe it wouldn't because of the, you know, the sort of early localization of information, perhaps. So maybe just for starters, tell us about like what you kind of observed in the, in the patch boundaries and how you overcame that so far. And then how that, you know, may develop and, you know, what you expect as you think about scaling this up much bigger.

• 1:09:52Speaker 00

  Yeah, so basically, again, as you said, see, if we want to answer this question, will people adopt this in a wide range? I think the only way to test that is scale it up. Because nowadays, people don't really care about if you train a model that's less than one billion and you only train less than, let's say, 100 billion tokens. People don't really care about that because we already know there, there is a region that model can be more powerful. So that's what we currently working on.

• 1:10:28Speaker 00

  See, can we train using similar computer of Llama and get matching performance on Llama? And then we don't have, we don't need the token either. That's what we currently work on. And we think that's ultimate test, if that's gonna work well and being able to get adopted and, and being wisely used. One thing about the boundary is, so that's something really, really interesting. And we have a couple way of thinking like how to solve it. So one thing we explore a little bit is the convolutional layer. The, the convolutional layer actually improved the model slightly, but it didn't improve too much.

• 1:11:13Speaker 00

  Ultimately, our hope is that's gonna be actually solved with scaling.

• 1:11:19Speaker 00

  When the model seeing enough data, the model should learn the environment across the boundary. So I think that's that's exactly the ultimate goal and and and we're very hopeful like to see more data model can learn that. Uh, on the other side of in context learning, I think in constant learning, you you shouldn't really to be be worried. Because it's all showing in the publicity, right? Like ultimately, if the model can understand and predict the next word will, which is showing in the publicity. Then your in context learning shouldn't be a problem. So I think ultimately, again, I think both question should be answered.

• 1:12:04Speaker 00

  When we have this large compute and large models as experiments on.

• 1:12:10Nathan Labenz?

  There's a lot of surprises in these, you know, systems, right? Like in some sense, it's all a giant surprise that transformers have scaled as well as they have. And, you know, seem to be, uh, you know, even even Yan Le Cun has recently said some things about like, yeah, they probably do have some, you know, nascent world models. And he wouldn't go as far as calling it a, you know, a full world model, whatever. And obviously, there's a huge debate around that. But it does seem like these things kind of keep surprising us on the upside in general. Do you have any sense for how the surprises with this architecture? Might be different from the surprises we've seen with kind of, you know, the the vanilla transformer?

• 1:12:54Nathan Labenz?

  Or is that just something that we have no way to guess, except to go scale and find out?

• 1:12:59Speaker 00

  I will see if it's a surprise, it means emergent behavior like, yeah, most of time, you know, just let us that let us get surprised. But one thing we hope that it can better at is actually code and math. It's less talk about, but actually coding as well as math problem. Uh, that's something like, uh, you know, kind of like big issue with tokenizer.

• 1:13:29Speaker 00

  And a- as well as like sequence lens, like the coding, uh, when you need to, you always need longer sequence for you to be able to model, uh, like the, the, the, um, a coding problem. Also, the way you tokenize it's just, just really, really unnatural. See, we get on a par to the popularity of the, of a normal language model, say NAMA. We are also like really looking forward to see how this skilled, uh, megabyte work on math and coding problem. Yeah, that's one of the domain we, we want to really watch closely on.

• 1:14:11Nathan Labenz?

  Yeah, the math in particular definitely makes a lot of sense. Uh, some of the explorations of like what a typical language model has to deal with in order to figure out math at all is, you know, it can be pretty crazy sometimes where it's like, these long integers, you know, people try to get it to add together, are actually being parsed into like two and three digit tokens and it's like, oh my God, no wonder it can't do the math. You know, when it's looking at it like that, it's, it's definitely coming at it from a, you know, serious, uh, deficit for starters. The coding one, I have a little less of a intuition for just because it seems like

• 1:14:54Nathan Labenz?

  tokenization there, I would guess is usually less weird, but.

• 1:14:58Speaker 00

  Yeah, yeah. So, so, so one thing, one, one interesting experiment is like, again, token, like at the very, very beginning, we talk about why people need the tokenization to compress, right? If you take your domain and you compress it really, uh, you, and then you run your BPE, you can get a different tokenizer and that may give you a different compression rate. So what we do is like in one of the earlier work of my colleague, the encoder, basically it's a, almost like a codex model, uh, developed by Meta2. So that's like code specific, specific language model.

• 1:15:39Speaker 00

  If you take the common BPE, which is like GPT-2 BPE tokenizer, compare with, let's say we train an in-house tokenizer, we can get thirty percent less of token. Let's just tell you like a GPT to tokenize their own code is not efficient. It's like, you know, twenty, thirty percent less efficient. So it's very, also very domain specific. You know, also think about in code, sometimes there is like multiple space or something. It's really hard to figure like how BP token handle that and how that's actually meaningful. Yeah, I think, I think there are, there are issues with current way how people do that.

• 1:16:23Nathan Labenz?

  Yeah, this probably also seems like it connects to, I mean, they haven't published a ton about this, but it's kind of, you know, gradually come to public light that the latest OpenAI models have a lot of code early in their training. And presumably, this is kind of related where there's like, with presumably more code friendly tokenization, it has an easier time learning certain logical structures. And, you know, then that can kind of benefit all tasks in a way that is really hard to get, you know, to happen with just traditional natural language and traditional tokenizers. So, yeah, that makes me actually start to think that

• 1:17:07Nathan Labenz?

  maybe this is going to just be, this might just be strictly better, which I'm sure is what you're hoping to.

• 1:17:13Speaker 00

  Yeah, yeah, I, I, I think, I think, uh, to many sense, even we are happy with the simplicity part. So I don't know if you're familiar with the Galactica work also from Meta. So it's, uh, it's particular trained on archive and science. There are actually a whole strategy, how you tokenize it. So there's like, oh, it's like, in this case, we use BPE. In that case, we do byte. In that case, we do digit splitting. In that case, we do this. So there are three or, like, I, I believe, like, three or four rules combined together to just to do the tokenization. So that's actually, you know, users won't really feel the problem because.

• 1:17:59Speaker 00

  You know, like, maybe like the only concern, as you said, is that you round the token of there and know how many tokens you are. And then that's going to change your expense. But for developer, actually, it's quite tricky to pick all these rules, how to handle, how to handle correctly and effectively. And when you decoding, you also have like different way of decoding it. It's it's actually a hassle. Yeah. So so we are hoping this can simplify everything. Yeah.

• 1:18:29Nathan Labenz?

  Anything else in this work that you want to highlight that we haven't got to so far?

• 1:18:34Speaker 00

  We are very excited about the either, you know, scanning the app or say to be able to directly in the future model the well, format of any, any, any file, any input, any modality. I think that's like a really exciting direction for us.

• 1:18:52Nathan Labenz?

  Yeah, so I'll be looking out for the next paper. I just looked the, uh, this one was published May nineteen. And it's, uh, we're recording here about a month later. Knowing the size of the clusters that you guys are working with over there. You know, I I feel like I can set my stopwatch, uh, and it shouldn't be, uh, too much longer before we get some of these questions answered. I think it'll certainly be very interesting to see how that, uh, that comes out. Maybe just zooming out for a couple of kind of big picture questions. Meta AI has really been on a sort of incredible role lately. And has been the

• 1:19:34Nathan Labenz?

  subject of quite a bit of speculation here. There's like the Google memo that says, you know, and I don't, I don't, uh, you know, necessarily endorse all the conclusions of that memo, quite the contrary. But the idea is that, you know, Meta is winning because it's got this big open source community. What's it like to be at Meta AI right now with kind of hit after hit? What is the vibe? What is the kind of big vision? It seems like it's probably very different from some of the other places, but I'd love to just kind of hear your reflections on what it's like to be a part of that.

• 1:20:04Speaker 00

  I have to tell from my personal angle. On the other hand, I'm also working on like in general multimodal, multimodal language model. So I think as I said, that's one of the inspiration why we did the Megabyte. And in our team, we also believe the future belong to multimodal or mixed model. We are not think about now we have CHGPT, now we have a journey. Why can't we have a unified model that's being able to do everything? That's definitely one big trend we're working on and it's really, really exciting. I cannot ask for more at this stage.

• 1:20:42Nathan Labenz?

  You know, it's been striking to see, obviously lately, so much focus has shifted toward AI safety. And different organizations have kind of come out at the leadership level and signed on to, you know, the extinction risk statement. OpenAI, DeepMind, Anthropic have all, you know, leadership at those, at those companies have signed on. Is there a, is there like an active dialogue about this within the Meta AI team? I imagine there has to be, but we haven't really heard, you know, as much about the kind of, how do we develop all this technology without, you know, losing control of it from Meta?

• 1:21:25Speaker 00

  Yeah, of course, of course. We have a, we form like couple of like responsible AI team. And we are always very, very careful about the data. We are always very, very careful about the bios coming of the model. We have very, very strict, like, you know, how we do open source. Even though at this right now in, in my team, like, like, like the big org, the

• 1:21:56Speaker 00

  Meta Research Lab, we, we still have open source as our, open science as our major goal. But the process of doing open source is very, very strict. We have to parse all this bios filter. We have to answer all these questions. So that's basically embedded to every project.

• 1:22:18Speaker 00

  Of course, like, we do hear, like, every day, like, everyone is complaining, why so hard to, to use this model, why so hard to open source, the, sorry, use this data or open source this model? But it's a world we have to live, live in right, right now, we have to do responsible AI. And again, on the other side, there's like particular team forming. So I think in Meta, we are very much following in this. But no matter how hard it is, uh, the researcher here are interested. It seems like the leadership is online with open size. We want to do open size, I think, you know, again, we believe that

• 1:23:04Speaker 00

  um, to be able to do responsible AI, we shouldn't let few, that's only represent my own opinion, which we don't necessarily need only a very few number of company to do that. Uh, we should open sources and everybody help debug, help develop more safe model. So, yeah, yeah, I think, um, it's definitely heated. I will say everyone has different opinion, but a baseline is like every open source model, it has to have some like internal filter, like it cannot be biased, yeah.

• 1:23:41Nathan Labenz?

  Do you think we're gonna start to see a sorting of researchers across these orgs? Because it seems like the, you've obviously published this paper and, you know, along with that comes some fun stuff. Folks at OpenAI, you know, they still still do publish some, but like, obviously not nearly as much. Um, if they had invented the megabyte architecture, I doubt they would have published it at this point. So it almost seems like there may be, be developing a kind of divergence of like, people that, you know, value the open source ethos and, and want to share.

• 1:24:17Nathan Labenz?

  Go to places like Meta and people that maybe have a more, you know, we want to build a world-changing product, or, you know, we feel like we have to keep this stuff secret because it ultimately is too dangerous to share. They go to these other labs. Do you see kind of a polarization or a self-sorting starting to take place?

• 1:24:38Speaker 00

  Yeah, yeah, hundred percent. I think that, I mean, that's avoidable. When the AI industry is mature enough so that you can have quick prototyping or quick productionizing certain model or certain idea. I think it's also understandable for certain company to say they don't release their model. It's just unfortunate a little bit to the community. Um, for us, we we are here, we choose Meta AI. One of the reasons we believe open science and we're sticking to that. But I do say I agree with you. I do see this polarization.

• 1:25:21Speaker 00

  Another thing is it hasn't happened to Meta, but it may happen is when the product team and the research team is doing like the work is more and more similar. The resources could be pulled into the product team easily. Um, you know, because the ultimate customer-facing thing, like ultimate like customer experience thing, we as a research, as a pure research thing, we are doing research that's far ahead. So in such a competitive space,

• 1:25:58Speaker 00

  the product like the product team has more priority. So once that fact could be the resource shifted to towards that, towards the product team and we get less.

• 1:26:08Nathan Labenz?

  And you mean by resources there, you mean just compute, like access to GPUs?

• 1:26:15Speaker 00

  Hundred percent.

• 1:26:16Nathan Labenz?

  Yeah, well, it seems like so far so good on that.

• 1:26:19Speaker 00

  So far so good.

• 1:26:21Nathan Labenz?

  I think one of the most sort of unfortunate and problematic dynamics developing in the world today is the US-China rivalry, which seems like it's constantly escalating and it seems like nobody can do anything about it. And, you know, that would be bad enough unto itself. But now it seems like it's kind of feeding back into the AI discourse. And almost every other conversation I have about big picture AI questions, you know, are we rushing into this? Like, is this, you know, do we have a good plan to keep ourselves safe as we build these more and more powerful systems?

• 1:27:01Nathan Labenz?

  So often, the conversation kind of ends in, well, but, you know, China will do it if we don't. And so, you know, there's just kind of this low, low trust, obviously, and kind of, you know, almost fatalism around like, what other choice do we have? So for one thing, I always look to just highlight any positive, you know, connection or collaboration that crosses the US-China, you know, divide. And you haven't grown up in China, going to university there, then coming to the United States, working here, you know, in some sense embody that. What is it like for you right now to be at kind of the center of both of these, you know,

• 1:27:44Nathan Labenz?

  super hot topics? Like, AI is obviously super hot, US-China is super hot. But the, you know, center of that Venn diagram is like maybe the most focal thing in the world. And here you are, you know, publishing research right from the kind of center of all of that. What is that like? And, you know, do you have any feelings about that?

• 1:28:03Speaker 00

  It's actually, to me, it's not between US or China. It's like every country can have different policy about how to develop AI model.

• 1:28:15Speaker 00

  And that could dramatic impact things. One or two day ago, Israel, as well as Japan, had a loss in like AI model could train on any data. So that's not even in China, right? It's just like in Israel and Israel nowadays have so many startup. I can imagine a scenario is like now everybody or the startup are gonna be a Israel company so that they can train with any data or they they can have more freedom to develop their AI model. Yeah, so again, I think this indeed is a big concern. Um, in the end, I, the easier way is everybody, of course, it's hard, everybody on similar page.

• 1:28:56Speaker 00

  So they follow this uh rule of develop safe AI. But I will say go beyond, definitely go beyond US and China. It's like worldwide. Uh, every country could be, you know, the one who allowed to train the widest AI things there.

• 1:29:15Nathan Labenz?

  Uh, I just hope that, uh, you know, clearer heads can prevail. And it seems to me that, you know, any, again, any flicker of positive relations between US and China should be celebrated and, uh, you know, elevated and made more visible. And, you know, certainly, I've always been a, you know, a big believer that

• 1:29:39Nathan Labenz?

  to the degree that the United States can attract people from around the world and including from China. It's like a great thing to bring people to, you know, the, the same research environments and have this kind of cross-cultural collaboration. So I don't expect you to have all the answers on that by any means. You've got your hands full. You know, obviously just doing the research itself, but boy, I wanna, you know, I hope we can continue to, uh, you know, to have this the degree or hopefully even more of the sort of academic and intellectual collaboration that has existed and, you know, continues to exist, but seems like it's under some threat, you know? And I, I just really hope that that can be, um.

• 1:30:22Nathan Labenz?

  Built upon and not, you know, not something we turn our backs on.

• 1:30:27Speaker 00

  Okay, I think personally I haven't feel this threat or I, yeah, I think overall what I have feel is a quite friendly environment.

• 1:30:37Nathan Labenz?

  I think this is great, phenomenal conversation. I really learned a lot from it. I'm glad to hear that you have not, you know, felt any of that sort of thing personally. I don't know to what degree people are feeling it personally, but just you look at kind of macro numbers. You know, the number of grad students coming is down, the number of like Americans living in China is way down. And, you know, just in general, there's this kind of decoupling phenomenon that, especially as we head into this, you know, AI future, I would like to see more coupling and not less. So that's, you know, just kind of my, my overall point of view. But I'm very actually heartened to hear that you have, you know, have not found this to be a, a practical issue in your daily life. So that's good news as I see it.

• 1:31:23Nathan Labenz?

  Well, we'll leave it there then for now. Again, phenomenal work on the megabyte architecture. I'm looking forward to the next paper and seeing if you've just, you know, changed the, the AI game, you know, across the board, or if there are any limitations or surprises that are unveiled. I'll certainly be keeping a close eye out and, and reading your next publications with all that in mind. For now, Lilly Yu, thank you for being part of the cognitive revolution.

• 1:31:52Speaker 00

  Thanks. Thanks for this nice conversation. It's a pleasure.

• 1:31:56Speaker 02

  Omnikie uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work, customized across all platforms with a click of a button. I believe in Omnikie so much that I invested in it and I recommend you use it too. Use Cogrev to get a 10% discount.