RWKV: Reinventing RNNs for the Transformer Era — with Eugene Cheah of UIlicious

The AI Engineer Summit Expo has been announced https://twitter.com/aiDotEngineer/status/1696532657197256815/retweets/with_comments, presented by AutoGPT (and future guest Toran Bruce-Richards https://twitter.com/SigGravitas!) Stay tuned for more updates on the Summit livestream https://www.ai.engineer/summit and Latent Space University https://www.latent.space/p/lsu-beta. This post was on HN for 10 hours https://hnrankings.info/37368264/. What comes after the Transformer? This is one of the Top 10 Open Challenges in LLM Research https://huyenchip.com/2023/08/16/llm-research-open-challenges.html that has been the talk of the AI community this month. Jon Frankle (friend of the show https://www.latent.space/p/mosaic-mpt-7b!) has an ongoing bet https://www.isattentionallyouneed.com/ with Sasha Rush on whether Attention is All You Need, and the most significant challenger to emerge this year has been RWKV - Receptance Weighted Key Value models https://huggingface.co/blog/rwkv, which revive the RNN for GPT-class LLMs, inspired by a 2021 paper on Attention Free Transformers https://arxiv.org/abs/2105.14103 from Apple (surprise!). What this means practically is that RWKV models tend to scale in all directions (both in training and inference) much better than Transformers-based open source models: While remaining competitive on standard reasoning benchmarks: swyx was recently in Singapore for meetings with AI government and industry folks https://aisingapore.org/, and grabbed 2 hours with RWKV committee member Eugene Cheah for a deep dive, the full recording of which is now up on Latent Space TV https://youtu.be/dvk6X5zeIfY: Today we release both the 2hr video and an edited 1hr audio version, to cater to the different audiences and provide “ablation opportunities” on RWKV interest level. The Eleuther Mafia? The RWKV project is notable not merely because of the credible challenge to the Transformers dominance. It is also a distributed, international, mostly uncredentialed community reminiscent of early 2020s Eleuther AI: * Primarily Discord, pseudonymous, GPU-poor https://www.semianalysis.com/p/google-gemini-eats-the-world-gemini volunteer community somehow coordinating enough to train >10B, OPT/BLOOM-competitive models * Being driven by the needs of its community, it is extremely polyglot (e.g. English, Chinese, Japanese, Arabic) not because it needs to beat some benchmarks, but because its users want it to be for their own needs. * “Open Source” in both the good and the bad way - properly Apache 2.0 licensed (not “open but restricted https://www.alessiofanelli.com/blog/llama2-isnt-open-source”), yet trained on data taken from commercially compromised sources like the Pile (where Shawn Presser’s Books3 dataset has been recently taken down https://www.theatlantic.com/technology/archive/2023/08/books3-ai-meta-llama-pirated-books/675063/) and Alpaca (taking from Steven Tey’s ShareGPT https://lmsys.org/blog/2023-03-30-vicuna/ which is technically against OpenAI TOS) The threadboi class has loved tracking the diffusion of Transformers paper authors out into the industry: But perhaps the underdog version of this is tracking the emerging Eleuther AI mafia: It will be fascinating to see how both Eleuther and Eleuther alums fare as they build out the future of both LLMs and open source AI. Audio Version Timestamps assisted by smol-podcaster https://github.com/FanaHOVA/smol-podcaster. Different timestamps vs the 2hr YouTube * [00:05:35] Eugene's path into AI at UIlicious * [00:07:33] Tokenizer penalty and data efficiency of Transformers * [00:08:02] Using Salesforce CodeGen * [00:10:17] The limitations of Transformers for handling large context sizes * [00:13:17] RWKV compute costs compared to Transformers * [00:16:06] How Eugene found RWKV early * [00:18:52] RWKV's focus on supporting many languages, not just English * [00:21:24] Using the RWKV model for fine-tuning for specific languages * [00:24:45] What is RWKV? * [00:33:46] Overview of the different RWKV models like World, Raven, Novel * [00:41:34] Background of Blink, the creator of RWKV * [00:49:55] The linear vs quadratic scaling of RWKV vs Transformers * [00:53:29] RWKV matching Transformer performance on reasoning tasks * [00:54:31] The community's lack of marketing for RWKV * [00:57:00] The English-language bias in AI models * [01:00:33] Plans to improve RWKV's memory and context handling * [01:03:10] Advice for AI engineers wanting to get more technical knowledge Show Notes Companies/Organizations: * RWKV - HF blog https://huggingface.co/blog/rwkv, paper https://arxiv.org/abs/2305.13048, docs https://wiki.rwkv.com/, GitHub https://github.com/BlinkDL/RWKV-LM, Huggingface https://huggingface.co/docs/transformers/model_doc/rwkv * Raven 14B (finetuned on Alpaca+ShareGPT+...) Demo https://huggingface.co/spaces/BlinkDL/ChatRWKV-gradio * World 7B (supports 100+ world languages) Demo https://huggingface.co/spaces/BlinkDL/RWKV-World-7B * How RWKV works in 100 LOC https://johanwind.github.io/2023/03/23/rwkv_details.html, RWKV overview https://johanwind.github.io/2023/03/23/rwkv_overview.html * EleutherAI https://www.eleuther.ai/ - Decentralized open source AI research group * Stability AI https://stability.ai/ - Creators of Stable Diffusion * Conjecture https://www.conjecture.dev/ - Spun off from EleutherAI People: * Eugene Chia - CTO of UIlicious https://uilicious.com/, member of RWKV committee (GitHub https://github.com/PicoCreator, Twitter https://twitter.com/picocreator) * Blink/Bo Peng https://github.com/BlinkDL - Creator of RWKV architecture * Quentin Anthony https://www.latent.space/p/transformers-math#details - our Latent Space pod on Eleuther, coauthor on RWKV * Sharif Shameem https://www.latent.space/p/sharif-shameem#details - our Latent Space pod on being early to Stable Diffusion * Tri Dao https://www.latent.space/p/flashattention#details - our Latent Space pod on FlashAttention making Attention subquadratic * Linus Lee https://www.latent.space/p/ai-interfaces-and-notion - our Latent Space pod in NYC * Jonathan Frankle https://www.latent.space/p/mosaic-mpt-7b - our Latent Space pod about Transformers longevity * Chris Re https://cs.stanford.edu/~chrismre/ - Genius at Stanford working on state-space models * Andrej Karpathy - Zero to Hero https://karpathy.ai/zero-to-hero.html series * Justine Tunney ("Justine.lol https://justine.lol/") - mmap trick https://justine.lol/mmap/ Models/Papers: * Top 10 Open Challenges in LLM Research https://huyenchip.com/2023/08/16/llm-research-open-challenges.html * Retentive Network: A Successor to Transformer for Large Language Models https://arxiv.org/pdf/2307.08621.pdf * GPT-NeoX https://github.com/EleutherAI/gpt-neox - Open source replica of GPT-3 by EleutherAI * Salesforce CodeGen https://github.com/salesforce/CodeGen and CodeGen 2 https://paperswithcode.com/paper/codegen2-lessons-for-training-llms-on * Attention Free Transformers paper https://arxiv.org/abs/2105.14103 * The Pile https://pile.eleuther.ai/ * RedPajama dataset https://github.com/togethercomputer/RedPajama-Data * Monarch Mixer - https://hazyresearch.stanford.edu/blog/2023-07-25-m2-bertRevisiting BERT, Without Attention or MLPs Misc Notes RWKV is not without known weaknesses - Transformers do well in reasoning because they are expressive in the forward pass https://twitter.com/karpathy/status/1593417989830848512?s=20, yet the RWKV docs already note that it is sensitive to prompt formatting and poor at lookback tasks https://wiki.rwkv.com/#tldr-vs-existing-transformer-models. We also asked pointed questions about RWKV’s challenges in the full podcast. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

• 0:00Speaker 00

  Hey listeners, today we have a very special episode for you. There's been a recent paper on the top 10 open challenges in LLM research that has consolidated a lot of intense debate. Today, we're going to talk about RWKV models, Receptance Weighted Key Value Models, with Eugene Chia, who is both part of the core RWKV team, CTO of a low-code AI test automation platform, and active member of the Lanespace Discord. The RWKV architecture has the potential to solve three of the top 10 open LLM challenges: increasing context length, making LLMs faster and cheaper, and designing a new model architecture. What is particularly appealing about it is that it does so by reviving the recurrent neural network, which even I have argued has been obsoleted by the transformer. It rejects the idea that attention is all you need, and replaces multi-head attention

• 0:43Speaker 00

  in feedforward networks with new concepts called a time mix and channel mix respectively. It has been trained up to 14 billion parameters, they're getting help from Eluther and Stability AI to scale it up even more, and shows competitive results on reasoning benchmarks, the same benchmarks we covered in our Benchmarks 101 episode, with similar-sized models, yet with linear costs and speed curves instead of quadratic ones. In a way, RWKV are promising the room temperature superconductor of LLM architectures. In other words, the parallelizability and performance of transformers without the quadratic cost. Obviously, the topic of what happens after the transformer is, in the finance terminology, what we call a low delta outcome. It's probably not going to happen, but if it does, it will be very, very big.

• 1:26Speaker 00

  We even discussed this a little bit in our episode with Jonathan Frankle of Mosaic ML. But since the RWKV paper was published, the idea has been somewhat independently validated with Microsoft Research putting out the RETNet, or the Retentive Network, which has similar veins of what it's trying to do. And it also, of course, competes with other alternatives to the transformer, like the state space models coming out of Chris Rais' group at Stanford, S4H3 and the Monarch Mixer that was recently announced. However, RWKV is so far the most validated of all these ideas because it is already trained up to 14 billion parameters with multiple models that you can download and generate text with today. As podcasters, we want to be the first place that you hear about new things in AI that you'll be using in your work or personal life as AI engineers and enjoyers. So this

• 2:10Speaker 00

  presents us with a problem. We have to be early on consequential topics and things, but also high signal. Some of our favorite compliments so far, which by the way, I've added to our about page if you want to check that. Your pods are a legitimate highlight of life for me. They're amazing. From McKay Wrigley. And from the AI Safety Memes Twitter account, which is always fun, they just simply said, we're the highest signal pod for them. So we're very proud of this and want to keep it up while taking risks. Because even though we crossed a quarter million downloads just five months into the podcast. We're still very young and trying to figure out what kind of podcast we want to be and what kind of audience we want to have. Today is going to be one of the riskier pods for a few reasons. One, it is the first pod we're doing on a non-traditional architecture with no large Western institutional backing.

• 2:53Speaker 00

  Two, it is the first pod we recorded outside of the US without a regular studio, so the audio is not as good. And three, finally, it is the first pod we're doing with my Singaporean accent. I'll address each of these in turn. One, there's a significant institutional bias in Western media coverage when it comes to AI. In that if you don't come from Stanford or Oxford, or you don't work at Google or Facebook, your ideas have trouble getting attention. However, by far, the most admired organization that our guests have repeatedly mentioned is Eluther AI. Which spun up as a decentralized Discord community that independently trained the first GPT class LLMs without a prestigious background. Eluther has since spun off organizations like Stability AI and Conjecture.

• 3:34Speaker 00

  But I suspect that the RWKV community is working like the early days of Luther. And we have a rare opportunity to capture an oral history of it live rather than two years after the fact. Two, part of the latent space magic is that we try to get to know our guests and be in person with them to establish rapport. Like we did in New York in our Notion AI episode with Linus Lee. Despite the lower audio quality, I think we got a much better interview with Eugene because we were able to interact with each other in person. Three, part of the joy of audio is that you get to hear the full diversity of humanity. And neither are less you or I are American. And I like to showcase the broader world through our AI lens whenever we can. As you'll see, the RWKV story is also about the non-English rest of world, self-organizing to build the LLMs that the English-centric West has neglected,

• 4:20Speaker 00

  and doing so relatively successfully. We've aggressively edited this interview down to one hour for the audio podcast. But for those who are interested in RWKV, head over to the new LatenSpace TV YouTube channel, which has the full two-hour interview, including a screen share while through the RWKV models paper and Discord, as well as digressions on Eugene's background, discussions on diffusion models and the token crisis, and the relationship between open source AI and the AI waifu hasmundo community. I hope you enjoy our conversation with Eugene Chia of RWKV. And if you liked it, reach out to him at Pico Creator on Discord or on Twitter and let him know. Okay, so I'm here with Eugene. We are in Singapore. We, this is the first time I'm podcasting in Singapore. It's the first time I'm podcasting with my Singaporean accent.

• 5:04Speaker 00

  Eugene has been a very valued part of our LatenSpace Discord for a while, and also diving deep onto RWKV. I think you're actually the first person that brought it to my attention as like a potential transformers alternative. You're also CTO of UiLicious, which is a UI testing company that's in Singapore here, which is a low-code platform. I got the first demo maybe four years ago.

• 5:25Speaker 01

  Yes.

• 5:26Speaker 00

  And I was like, okay, fine, you know, you're doing testing. There wasn't an obvious AI angle. I mean, now that you explained it, it was great. But like, what was your personal like, okay, I'm gonna be the dedicated AI guy for UiLicious?

• 5:37Speaker 01

  Okay, so one of the things, one thing that I found very interesting with the huge transformer boom right now, right? Is that, is that traditionally, right? When you tell companies that you need, when you want to build your own AI, you need a really large data set. And over time, actually, the amount of data sets that you need is actually scaled down because you can just now find.

• 5:57Speaker 00

  Foundation models.

• 5:58Speaker 01

  Yeah, finding foundation models. And when we started UiLicious, we always knew at that time, because a lot of our other companies that were launched at the same time were dealing with neural networks, that at some point, the data that we've been collecting data on, let's say, how to do testing website. It's just a very specific focus. Basically, every single test that has run on our platform, unless our customer has. Opt out or delete the account, basically privacy related stuff, we actually still retain the test data.

• 6:26Speaker 00

  Um hum.

• 6:26Speaker 01

  And that's something that that we always felt that was useful in the long run to be able to actually build a huge training model. The irony of that was that even though we were building all those data sets, as as the threshold came in and and the transformer boom happened. We realized we don't actually need that big of a data set anymore to actually get a functional A I. One of the key insights, especially for people who is like trying to build on top transformer models.

• 6:48Speaker 00

  Yeah.

• 6:48Speaker 01

  Pre transformer, large language models is is we would we always be thinking of in terms of like hundred gigabytes of data, one terabyte of data.

• 6:56Speaker 00

  Right.

• 6:56Speaker 01

  Like millions of record for all the different examples. Post transformer, it's history, you probably need only like a thousand or ten thousand enough like. Data that you can literally get an intern a few weeks to just get it done. And you have a working model, it may not be that great, but frankly, every piece of data you add after that is a is a diminishing returns.

• 7:19Speaker 00

  And because it's it's a language model, it doesn't actually have any inherent understanding that it is automating the browser. So it's presented as like a prompt answer pair, like question answer pair.

• 7:28Speaker 01

  Cool, at least for our internal model that our users are using is is presented as here's the prompt, describe your task or what you want to modify the code. It and then subsequently generate the code for you, then hindsight is now basically co-pilot.

• 7:40Speaker 00

  Yeah.

• 7:41Speaker 01

  Yeah, I I think now co-pilot is adding that chat widget, are they fully launched yet?

• 7:45Speaker 00

  Yes, I actually downloaded it yesterday, I haven't actually used it yet, but it it is a separate VS code extension. So there now there are now three co-pilot extensions shipped by GitHub because they have shipped their org chart. I'm quite friendly with that team, but it's very funny. But just to come back to you, so did you implement this with G P T three?

• 8:02Speaker 01

  So we based it off the Salesforce Codegen model.

• 8:05Speaker 00

  Okay, right.

• 8:06Speaker 01

  So that was the foundation model that we built on top. We are looking into replacing it in parts, but that becomes a longer conversation.

• 8:14Speaker 00

  Codegen being the first really credible, open source, code specific.

• 8:21Speaker 00

  Language model that was released by literally anyone, I think about three years ago.

• 8:25Speaker 01

  Yeah.

• 8:25Speaker 00

  And then they recently released CodeGen two.

• 8:27Speaker 01

  Correct.

• 8:27Speaker 00

  Any opinions on CodeGen two and just while we're on this topic?

• 8:30Speaker 01

  In terms of like CodeGen, one big appeal for the CodeGen and even CodeGen two model is that Salesforce took a very clear and clean approach to the licensing.

• 8:39Speaker 00

  Meaning they were very, very clear that everything that they trained on was open source.

• 8:43Speaker 01

  Yeah, MIT, they didn't, they didn't, they didn't touch the problematic licenses.

• 8:47Speaker 00

  Yeah, yeah, yeah, yeah.

• 8:48Speaker 01

  So, and you can imagine.

• 8:50Speaker 00

  And you, you think that Copilot did?

• 8:53Speaker 01

  Knowing Microsoft's statement on how liberal they were about GitHub data and they were saying they used the term that is under fair use.

• 9:01Speaker 00

  I see.

• 9:02Speaker 01

  Yeah.

• 9:02Speaker 00

  I see.

• 9:03Speaker 01

  I, I have no reason to believe that they didn't, but this same problem happens to actually a lot of existing CodeGen models. And, and that was actually the main appeal for me for running, for actually building on top of the Salesforce CodeGen model. Mostly also because like, like for us, we deploy on premise into enterprises.

• 9:21Speaker 00

  Yeah.

• 9:22Speaker 01

  In Europe and they ask questions.

• 9:26Speaker 00

  So what, what, what does this deploy on premise mean? Like you, you, you pack UI lashes into a container and you give it to them?

• 9:32Speaker 01

  Yeah.

• 9:32Speaker 00

  And then you date, it's like a license fee or something.

• 9:34Speaker 01

  Correct.

• 9:34Speaker 00

  Okay, cool. That's very interesting. Yeah, okay. I, I don't know if I have any other questions on, based on that. Anything else before we go into alternate, like the reasons for alternative models?

• 9:46Speaker 01

  Okay, so, so I, anything else do I have there? No, I don't really have much, right? For alternative, yeah.

• 9:52Speaker 00

  So let me, let's set the premise, right? Like transformers have won for now.

• 9:57Speaker 01

  They have slipped the neural networks?

• 9:58Speaker 00

  Yes, and you know, it, it, it seems like you have had a history since with machine learning since before transformers. And now they're, they're at kind of like the peak of their power. And I, I see that, you know, there's, there's a desire for alternative models for a number of reasons. But I'm try, I'm very curious as to what drives your personal interest in alternative models?

• 10:18Speaker 01

  Ah, so, so first things first, to be clear, majority of our AI is still based on training, at least within my company.

• 10:24Speaker 00

  Yes.

• 10:24Speaker 01

  But what drove me into alternatives beyond transformer? In essence, once we actually managed to get our bots to generate UI testing code. The most obvious next thing that our customers started asking, hey, this, let's say the test fail. Can your AI now analyze my website and then tell me what's wrong and tell me what to change? Basically, they're getting crazy and crazy.

• 10:47Speaker 00

  Yeah, yeah, yeah.

• 10:48Speaker 01

  And that has to be.

• 10:48Speaker 00

  Humans are very good at moving goalposts.

• 10:50Speaker 01

  And we had something working for toy websites. But the one thing that we do and internally is that we look at the, I think, cause the list, top one hundred, top one thousand websites.

• 11:00Speaker 00

  Okay.

• 11:00Speaker 01

  And we basically just run or we actually do run our, our test platform against that to see, make sure that our code works against like any front end platform.

• 11:08Speaker 00

  Well, what do you mean run your test platform, right? Because you don't have tests for them.

• 11:13Speaker 01

  Oh, yeah, we have some very rudimentary basics, like go to websites, see something, click something, add to cart. Yeah, that's it.

• 11:19Speaker 00

  That's it.

• 11:19Speaker 01

  The, the idea is more of like, because there's so many frameworks out there and.

• 11:23Speaker 00

  You just wanna make sure you cover all of them.

• 11:24Speaker 01

  Yeah, and so we did the same thing for our AI. And the first thing that it, it died on was literally Amazon.

• 11:31Speaker 00

  Why? Oh, five megabytes.

• 11:32Speaker 01

  Yeah, I, yeah, I think you heard me mention that. So, so it, when you are trying to analyze a website, we've been talking about increasing token count size, right? But for e-commerce websites in particular, even if you strip off the CSS, even if you strip off the JavaScript, having the entire HTML in megabyte size is not unheard of.

• 11:48Speaker 00

  Yep.

• 11:49Speaker 01

  And that's where it's like, how am I supposed to solve this in terms of an AI point of view?

• 11:55Speaker 00

  How, how many tokens would that be? Like?

• 11:57Speaker 01

  Oh, my gosh, you could easily be looking at over a million tokens.

• 12:02Speaker 00

  I see, which is still too much even for today.

• 12:04Speaker 01

  Yeah.

• 12:05Speaker 00

  Did you look into making your own tokenizer?

• 12:08Speaker 01

  That's some, that's something that we explored. We, I think what we found more realistic was to actually parse the HTML into a more token friendly format.

• 12:16Speaker 00

  Yeah, right.

• 12:17Speaker 01

  So this way, this way you can still build on top of existing models, but yeah, we, we are exploring that as well. But, but back to the often people, so like the key things for me and, and was at that point. And as, and subsequently, I think I showed you like the experiments with like English compiler and things like that, right? AI agent generating code, you also have your own small there, was that, was that the context size is a real problem. And transformer inherently by its nature, at least, but we're not transformer. I know there's transformer XL and some other attempts, is that.

• 12:54Speaker 01

  It quadratically scales with the context size.

• 12:57Speaker 00

  Um hum.

• 12:58Speaker 01

  So we, so if we scale to like, let's say a hundred thousand, that's already requiring a shit ton of compute and V RAM. And I don't even want to imagine what happens to one million or ten million. And and that's that's where I I needed, I was like, okay, this is a fundamental problem that needs to be changed. If not, we will not go past this. And I think there's also now like a lot of people who are very interested in like models that can handle large context size. Because they also wanted to be able to use in use cases where they never never need to find, because finding things are pain apparently.

• 13:32Speaker 00

  Yes, that said, okay, well, there there's there's there's issues with just throwing everything in context, right?

• 13:37Speaker 01

  Yeah.

• 13:38Speaker 00

  The it's shown that retrieval is only best when the the item that's relevant is in in front or in the at the back of the context window. So it's basically, I'm just like, maybe we've just tapped out. Context is working memory and maybe it's like, maybe transformers are very similar to humans in that our working memory is only of a given size. If you try to artificially extend it, you would just have, you just make it very lossy.

• 14:00Speaker 01

  Yeah, so so so that's where I ended up landing on the R W K V model. Because in that sense, right, so so you, one thing that I always found really weird for transformers. But I mean, it's by design, is as you infer each token, you are you will re-computing everything upfront. That's the quadratic part. And and and while while you while you're mentioning about the working memory problem. In theory, with enough attention heads on it.

• 14:29Speaker 00

  Um hum.

• 14:29Speaker 01

  And and and people seem to be trying to cram more and more attention heads into the process. It could scale that way, ignoring compute cost.

• 14:37Speaker 00

  Okay.

• 14:38Speaker 01

  And yeah, ignoring compute cost is like, just like a very liberal, that's as true as much as anyone just, it doesn't make sense, but. But R K V what is still was fundamentally a neural network as it's called. It ends up scaling linearly as it goes through the tokens. It also it will still suffer from the memory issue. So so in so like, within the R K V, we do we do like measure two separate things. So like one, we call it the perfect memory. So in the model, we have only a certain amount of capacity where it can remember things perfectly, just like humans. And then there'll be then beyond that, that is where it will start to discard things from its perfect memory.

• 15:18Speaker 00

  Right.

• 15:19Speaker 01

  And and I felt that this was actually a lot more in line with our goals and commercially. And also what I felt was that was more useful in the long run because it's cheaper compute and it could be potentially paralyzable for a very long run.

• 15:33Speaker 00

  Right. So we're going to go into our R W K V paper in a in a bit. But one thing I wanted to ask, you kind of glossed over how you found it in the first place.

• 15:41Speaker 01

  How did?

• 15:42Speaker 00

  Because you're not a researcher. You're not like, I don't I don't imagine you're like reading papers every day or something.

• 15:47Speaker 01

  Until recently.

• 15:48Speaker 00

  Until recently. How do you find it?

• 15:50Speaker 01

  How do I find it?

• 15:52Speaker 00

  And how do you know like this is the one to bet on versus there's a bunch of other alternatives, right?

• 15:56Speaker 01

  I think what was quick, I think it was rather quick after I concluded that Transformer as it is. We're not scaled to ten million tokens.

• 16:09Speaker 00

  Okay.

• 16:10Speaker 01

  And.

• 16:10Speaker 00

  And so by the way, you you mentioned Transformers Excel.

• 16:13Speaker 01

  Yeah.

• 16:13Speaker 00

  We also did a a episode on flash attention, which which helps to make part of it sublinear at least.

• 16:18Speaker 01

  Yeah, but that is like way, way after I already died in the R K V. So history wise, at that point in time, Transformer, we are talking about like the the when four K was the limit that everyone knew.

• 16:29Speaker 00

  Right. And this was last year. I mean.

• 16:31Speaker 01

  Yeah.

• 16:31Speaker 00

  Just to set context. Okay.

• 16:33Speaker 01

  Yeah.

• 16:33Speaker 00

  Okay. And then, yeah, so so and and you just kind of were searching around and you found our our our R W K V presumably like, did you do you go straight into the Discord? Was it like primarily a GitHub repo like what was it? Because.

• 16:48Speaker 00

  As far as I can tell, there was no paper until maybe about two months ago.

• 16:52Speaker 01

  Oh, and I and I talked about it before the paper, right?

• 16:55Speaker 00

  Yes, so you found it before they did any publicity.

• 16:58Speaker 01

  Hmm.

• 16:59Speaker 00

  Which is weird. It's not normal. So what what happened? What did you do?

• 17:03Speaker 01

  So, uh, why did? Okay, so it was basically, I I believe. Okay, so it's a mixture of things because it's like, I was searching guitar, I was searching like forums, other discords. And also like blogs, actually. I was just like getting all the, because everyone was just creating lists of lists, right?

• 17:21Speaker 00

  Yeah, yeah.

• 17:21Speaker 01

  And this, and I believe you also have a list of this somewhere.

• 17:24Speaker 00

  Yeah, but mine is very, so I would consider myself very trad. In the sense that I I would just follow the large model labs. Whereas the kind of list that you have to follow in order to get to something like R W K V before they've done any publicity. Is the non trad, like. You know, the kind of people that is not working on, no Hermes, wizard, you know, that like no credentials. I don't even know who the hell they are. But they're just working on it.

• 17:48Speaker 01

  Oh, this is all for good memory and I might be hallucinating this because there was too many lists. But I believe the list that actually what brought me to R W K V was that. Beyond OpenAI's model and and beyond ChatGPT and Claudia, the the two big models, right? Outside of the English speaking nations, right? A lot of the open source models really fall flat.

• 18:12Speaker 00

  Hmm.

• 18:12Speaker 01

  And that is why when you actually go through like list or app for like doing things with in other languages.

• 18:18Speaker 00

  Hmm.

• 18:19Speaker 01

  R W K V actually stood out at that point.

• 18:21Speaker 00

  Hmm.

• 18:22Speaker 01

  And just on the basic premise, and I mean, we're not even talking about architectural advantage. It's just the basic premise that they imported the data set in other languages in the training data.

• 18:32Speaker 00

  Yeah.

• 18:32Speaker 01

  And.

• 18:34Speaker 00

  Was that a, because I mean, I imagine ninety nine percent of your customers are English.

• 18:37Speaker 01

  Yeah.

• 18:38Speaker 00

  Was that really a driver for you?

• 18:39Speaker 01

  It was not a driver, but that's.

• 18:40Speaker 00

  You're just trying to explain it.

• 18:41Speaker 01

  Yeah, that's how I landed onto like all these blogs and they go.

• 18:44Speaker 00

  And can you say, when you say fall flat, the main one that I know about is there's a tokenizer penalty for non English.

• 18:51Speaker 01

  Yeah, that's it.

• 18:52Speaker 00

  Right, so the Chinese is up to Chinese or Japanese or Thai or something like, it's like sixteen times the number of tokens. For a typical English sentence.

• 19:00Speaker 01

  Yeah, but even before that, right? Because. I mean, yeah, I think you understand like a lot of community users, they want to not use the commercial APIs.

• 19:08Speaker 00

  Okay.

• 19:09Speaker 01

  So they try to find open source models.

• 19:10Speaker 00

  Yes, and we'll talk about the not safe for work people. We, I really want, because you've actually talked to them, I have never talked to these people, but like, when I discovered them, they are. Huge community, they're extremely passionate, and they're actually good.

• 19:23Speaker 01

  Yeah, they're really good.

• 19:24Speaker 00

  They're good at this. So let's talk about them, right? Yeah, we can talk about it later.

• 19:28Speaker 01

  Yeah, so, so they don't want to use the commercial models, and they want to use the open source model. And there is a tokenized penalty, which is true, but I think on the more fundamental basis, right, if you look through the data sets. And, and this is also partially import because the way we set up our evals, all evals are written in English. And at least for the majority of them, and if we are racing towards building AI models at least right now. As you see, all the companies as they build their open source model, and they just want to narrowly focus on the evals. Adding in a foreign data set is actually a loss because as you, once you are below a certain parameter, so we're talking about the seven and fourteen, right?

• 20:11Speaker 01

  The more you add, there's more in line with your evals, the more you'll degrade and, and they just exclude it. And so the model just.

• 20:20Speaker 00

  Priorities, English, yeah.

• 20:21Speaker 01

  Yeah, the, the model just fundamentally didn't support.

• 20:23Speaker 00

  So what's the trade off? Like, I mean, okay, so English and Chinese, or, you know, there's, there's all these other languages, what do you pick, like?

• 20:31Speaker 01

  So, so our K B started, we started, we, also, in context, the main person leading the other K B project, Blink, is from China. So he naturally has an interest to make sure it supports Chinese. Yeah, so.

• 20:44Speaker 00

  And, and there are a fair amount of bilingual models, essentially, that are English and Chinese from the major universities in China.

• 20:49Speaker 01

  So, so we started from basically English, Chinese, Japanese, Korean. Frankly, this is large part mostly because there were fans in those communities that came on board. And then subsequently, we try to import other languages as well.

• 21:04Speaker 00

  Yeah, but these people are, again, not researchers.

• 21:07Speaker 01

  Nope.

• 21:08Speaker 00

  No money, like.

• 21:09Speaker 01

  Nope.

• 21:10Speaker 00

  Training on their home GPU lab or whatever, right?

• 21:12Speaker 01

  Partially true, but also how I see it works out for a lot of the other languages was that we have the foundation model. And this is the foundation model where we just kind of say, if ours be them, let's just make sure to include all the other languages.

• 21:26Speaker 00

  Okay.

• 21:27Speaker 01

  And, and, and when we include the other languages, right, the model works for most parts.

• 21:34Speaker 00

  Mm hmm.

• 21:34Speaker 01

  Uh, for the other language. Subsequently, these individuals who wanted to use these models in. For their respective use cases, we will then fine tune respectively.

• 21:44Speaker 00

  Hmm.

• 21:45Speaker 01

  Because it's easier to fine tune in another language for your use case than, I mean, this is classic fine tuning.

• 21:51Speaker 00

  Yeah.

• 21:51Speaker 01

  Than to train the language from scratch.

• 21:53Speaker 00

  Okay.

• 21:54Speaker 01

  And I think more recently, and this model is not hundred percent trained yet, more recently, we, R W K V has released the, what we call the world model. Where, where we go the next step of even including all the translation data sets that we can find. Even for minority languages that people and in our Discord. Cause the goal for them, the goal, the long term goal for us, at least internally, is that we wanted an AI model for everyone. And everyone does not mean USA, it means the world.

• 22:25Speaker 00

  Uh, well.

• 22:27Speaker 01

  So there are a lot of languages in there.

• 22:28Speaker 00

  Well, is it a, is it Asia biased or, you know, I, give me a sense of like.

• 22:34Speaker 01

  It's probably, no offence, it's probably still gonna be US biased in terms of like knowledge. Because what we are doing is still power of red pajamas for the knowledge.

• 22:43Speaker 00

  Okay.

• 22:44Speaker 01

  But in terms of language, we add all the other languages wiki and translation set. So it's hard, I mean, we haven't fully evaluated the bias yet, but I'm quite sure that when disproportionately knowledge is. Still within the English universe, there's the bias there, but frankly, we are still at the stage where, okay, support the other languages.

• 23:04Speaker 00

  Yeah.

• 23:05Speaker 01

  And and and and I think I mentioned this, this is this is one of the interesting parallels that that sometimes I have, right? Is that I can be in the, I can see in the iludor forms and all that. And then we're talking about alignments and and like we're talking about it in very.

• 23:18Speaker 00

  Which is, yeah, very keen on safety and all that, which is great, but like it's not your goal as as the RWKB community.

• 23:25Speaker 01

  Yeah, and and when when you talk to like members of the community that came on board, it's like, oh, I want to get this to work for Korean, Japanese, Thai, Arabic languages and and so on and so on. They just want something that worked, yes. They don't want it to be, they they are not after the big model that does everything. They just want something that they can play with in their language. And that was very important to them.

• 23:48Speaker 00

  Yeah, and these are literally just hackers doing it for personal enjoyment.

• 23:54Speaker 01

  Correct.

• 23:55Speaker 00

  Not yet for work.

• 23:56Speaker 01

  Yeah.

• 23:56Speaker 00

  Or maybe some of them for work, you you don't know.

• 23:58Speaker 01

  We don't know that.

• 23:59Speaker 00

  Yeah, okay.

• 23:59Speaker 01

  I mean, the whole character A I category, there's quite a number of them using it for for that. So so professionally.

• 24:09Speaker 00

  Professionally, okay, as in they they run character companies.

• 24:12Speaker 01

  Yeah.

• 24:13Speaker 00

  Let's call it. Shall we pause here and then I'll switch to the screen?

• 24:15Speaker 01

  Sure, sure.

• 24:16Speaker 00

  Okay. All right, so we have it pulled up. We are going to screen share for the bulk of this. So if you're listening on audio, might be a good time to switch to the YouTube channel. So we're just going to start with an intro. What is RWKB?

• 24:26Speaker 01

  So RWKB is a modern recursive neural network with transformer like level of L M L M performance, which can be trained in a transformer mode. And this part has already been benchmarked against G P T new X in the paper. And it has similar training performance compared to transformer models of the same data set and parameter. So specifically the G P T new X model. So the the key thing is that even though it's matching performance while trading go trading goes to G P T new X, it's doing all this without attention layers. And in the process, right, it's actually having a much substantially lower compute based on its design.

• 25:06Speaker 01

  And also because it's a neural network, which we will dive into later, why, why that's substantially lower in both training and inference. And this is back to like I mentioned previously, transformer, it traditionally transformer until until we found out about transformer X L and things like that. Tends to scale quadratically based on the context size. And this applies not just in inference, but in training. And due to how to like, due to how this is still a neural network in its heart, even though it can train like a transformer. It's able to do so much more efficiently and faster. Especially like when you hit context size of eight K, sixteen Ks and above. And and once you do like quadratic and linear, the differences start to like go of course crazy once you once you scale the numbers up.

• 25:50Speaker 01

  And that was the main benefits of the R W K V model, let's say. There were a few prominent researchers when they actually reviewed through the archery theory paper when it came out. They did highlight an important question of like, is this like evidence to literally maybe all that really matters is that we need a large data set and a scalable model.

• 26:12Speaker 00

  That makes sense, obviously, to some approximation. But you are still using attention?

• 26:20Speaker 01

  No, we are, we don't use attention inside.

• 26:22Speaker 00

  Okay, yeah, let, could, could, maybe, maybe let's rewind a little bit.

• 26:26Speaker 01

  Oh, specifically attention as you understood it, yeah.

• 26:29Speaker 00

  Okay, tell, tell, tell us more.

• 26:31Speaker 01

  So, so we, we, we use weighted receptors and.

• 26:36Speaker 00

  And if there's any diagrams I should pull up, let me know.

• 26:38Speaker 01

  Oh, okay, let's, okay, so we are using A F E. So this attention free transformer and this is, this paper was written by.

• 26:46Speaker 00

  What the hell is an attention free transformer?

• 26:50Speaker 00

  Okay, this is on, this is unusual.

• 26:52Speaker 01

  Yeah, so we use the weighted retention weights and we compute over it. And in essence, this is like the classic like stacking more layers once you do on top of it, like you don't really need. Attention, wait, once you have enough weights and layers stacked on it.

• 27:12Speaker 01

  Okay, it, it, it, I don't know whether we want to go into the deep dive or A F T.

• 27:17Speaker 00

  Sure.

• 27:17Speaker 01

  But, but.

• 27:18Speaker 00

  That's interesting. I've never heard of this paper.

• 27:19Speaker 01

  Yeah, so, so this is, this is, this was written by Apple. And subsequently, we integrate, at least blink, the creator of R W K V, took this, took this and applied it to a language model. And scaled it up.

• 27:32Speaker 00

  Right.

• 27:33Speaker 01

  And, and that is, and that is how, how we, we landed on R W K V that doesn't use attention. So sometimes within the community, we use the word light attention. Because what happens is that these layers and these weights will will still play the role of attention.

• 27:49Speaker 00

  I was gonna say, you you end up approximating attention.

• 27:52Speaker 01

  Exactly. So it ends up like looking at the tokens or parts of the memory and then applying it to the output.

• 27:58Speaker 00

  Um.

• 27:58Speaker 01

  And the key benefits is that, because remember the attention models are multi-hit part.

• 28:02Speaker 00

  Um hum.

• 28:03Speaker 01

  It will need to scan all the tokens back and forth.

• 28:05Speaker 00

  Yeah.

• 28:05Speaker 01

  This removes that requirement.

• 28:07Speaker 00

  Um hum.

• 28:07Speaker 01

  And hence it reduced the overall compute count.

• 28:10Speaker 00

  Um hum.

• 28:10Speaker 01

  I might be jumping back and forth a bit, but that's the that's the one the key essence of the W K V segments. And and we call it light attention. I and this is the part where I would disagree with the RWKB communities in some parts. I think that was a bad name.

• 28:25Speaker 00

  Ah, whatever.

• 28:26Speaker 01

  Because because it's cute. Why is it a bad name? Um, because when the RWKB paper came out, right, and then we talk about like we use this. And we call it light attention, but by design, it's really nothing like your existing attention hit models. And it ended up like sidetracking the Hacker Noon debate on like one corner. It's like, no, this is technically attention, approximating attention. Then another goes like, no, this is not attention.

• 28:53Speaker 00

  I see.

• 28:54Speaker 01

  And and but I'm like, propose a better name because I have no idea what to call it.

• 28:59Speaker 00

  Okay. What else should people know? Maybe we can explain what it what RWKB stand for.

• 29:05Speaker 01

  Oh, receptive with the key values.

• 29:07Speaker 00

  Okay.

• 29:07Speaker 01

  Yeah.

• 29:08Speaker 00

  And each of these are like actual things that you model in the code, right?

• 29:11Speaker 01

  Okay, so so we we can go into that. So.

• 29:14Speaker 00

  Which attention historically is like query key value.

• 29:16Speaker 01

  Correct.

• 29:16Speaker 00

  Okay.

• 29:17Speaker 01

  So so so do you want to jump straight into the layer architecture?

• 29:21Speaker 00

  Should we should we cover something else first?

• 29:22Speaker 01

  Anything.

• 29:23Speaker 00

  I mean, we can, any anything like.

• 29:25Speaker 01

  High level.

• 29:25Speaker 00

  High level. Okay, there's a seven B, there's a fourteen B, there's.

• 29:28Speaker 01

  Okay, so so so let's.

• 29:29Speaker 00

  One of the assets or the the artifacts.

• 29:31Speaker 01

  Okay, so on a, okay, before we go into the nitty grittys of how the layering and everything works, on the high level, right? Currently, RWKB architecturally as a model, it can be. What we have already proven is that it can be scaled and trained like a transformer.

• 29:45Speaker 00

  Um hum.

• 29:45Speaker 01

  How I do so, we'll cover later. And and this this can be scaled to as many parameters as we want. Currently, what we have is a dominant, our main models is the seven B model and the fourteen B model.

• 29:56Speaker 00

  Um hum.

• 29:57Speaker 01

  Which you can find on hugging face or respectively our demos. We also have a, that'll be the. There'll be the R A V A N models. These are also instructionally tuned.

• 30:09Speaker 00

  Okay, so there's world, there's Raven, there's music. Oh my god, this novel. What is all this?

• 30:13Speaker 01

  Okay, so be before we were the the current main models is R A V A N K V for R A V A N the P A O and Raven. So this so P A O is basically just a P A O plus model.

• 30:28Speaker 00

  What is P A O plus? I know about P A O, but what is P A O plus?

• 30:30Speaker 01

  Random data sets that the

• 30:34Speaker 00

  How many tokens worth?

• 30:35Speaker 01

  We I I would just say slightly one point one or one point two times the power.

• 30:39Speaker 00

  Okay.

• 30:40Speaker 01

  Yeah, this is not instruction tuned and and and stuff. Yeah, the plus one is typically all the other languages. Subsequently, Raven are the instruction tuned model. This is the current main complete models. We that we subsequently have.

• 30:57Speaker 00

  Any instruction data sets are from?

• 30:58Speaker 01

  Typically G P T for all, but then we scrub it for every move or the.

• 31:04Speaker 00

  As a large. Yeah, this would be the uncensored. There's there's someone, there's some other project that's kind of doing something similar and they call it uncensored, but really they just scrubbed it as as a large.

• 31:13Speaker 01

  Correct, yeah.

• 31:14Speaker 00

  So that makes it technically breaking T O S of of of Open A I, right?

• 31:20Speaker 01

  Yeah.

• 31:20Speaker 00

  Okay, but yeah.

• 31:21Speaker 01

  But that's a, I mean.

• 31:23Speaker 00

  That's a later problem.

• 31:23Speaker 01

  Listen, frankly, let's be honest, if.

• 31:26Speaker 00

  Yeah.

• 31:26Speaker 01

  If we, even if we don't remove it, someone is going to remove it.

• 31:29Speaker 00

  I mean, so there's ways around this, which is you you get it, you get clean data sets that are not G P T four, like, so the one that I typically mentioned is Yannick Kultur's Open Assistant.

• 31:40Speaker 01

  I believe that was included subsequently as well.

• 31:42Speaker 00

  Yeah, okay. Yeah, obviously, all these release orders are all over the place.

• 31:46Speaker 01

  Yeah.

• 31:47Speaker 00

  Uh so, okay, Raven, world.

• 31:48Speaker 01

  So Raven is the instruction tuned model.

• 31:50Speaker 00

  Yep.

• 31:51Speaker 01

  There and then subsequently, the world model is a new model that we are training. It's not hundred percent complete yet.

• 31:57Speaker 00

  Okay.

• 31:57Speaker 01

  With the focus on a new tokenizer and all the languages. So so what?

• 32:03Speaker 00

  All the languages.

• 32:04Speaker 01

  All the languages that we can grab from the internet, all the all the wikis in all the respective languages.

• 32:09Speaker 00

  Hmm like, what do you mean when you say all languages, one hundred languages? Okay, fine.

• 32:12Speaker 01

  So one hundred languages, it wasn't really a very precise sign. We just basically, whatever the the the wiki tool that allows us to download the X X wiki languages. If it works, it's in the set. If it doesn't work, skip. Yeah, and and you and all the major prominent Oscar translation sets. So as you can see, power, red pajamas.

• 32:33Speaker 00

  What is Oscar?

• 32:34Speaker 01

  Oscar, Oscar is just a common term that we use in, you can just search Oscar in Hugging Face data set and just means translations.

• 32:41Speaker 00

  Okay.

• 32:42Speaker 01

  So you can find like English X.

• 32:45Speaker 00

  I see.

• 32:45Speaker 01

  Yeah, all the respective pairs.

• 32:47Speaker 00

  Okay.

• 32:47Speaker 01

  Yeah, so and then all training data can find.

• 32:50Speaker 00

  Okay, so seventy percent English, fifteen percent multilingual, fifteen percent code. Is there, is there a strong grounding for why fifteen percent code?

• 32:57Speaker 01

  No, it was just, it was already there.

• 33:00Speaker 00

  Yeah.

• 33:01Speaker 01

  So the focus of the world model was not to improve everything else. It was literally that fifteen percent multilingual. We wanted to.

• 33:08Speaker 00

  Right, it was, it was English and code, and then you just added multilingual.

• 33:11Speaker 01

  Yeah, we had Fabia multilingual, but we wanted to bump it up.

• 33:14Speaker 00

  Right, so this, this is primarily English? Whatever, okay.

• 33:19Speaker 01

  Yeah.

• 33:19Speaker 00

  Alright, what I would like is like basically like a visual of like, here's all the building blocks and here's how they combine to to create all these things.

• 33:25Speaker 01

  Ah, so, so the, so we had the R M K architecture code.

• 33:30Speaker 00

  Yeah.

• 33:30Speaker 01

  So that's the main model building block. And basically we feed it the data, power plus red pajama, then subsequently not some of the code, code data. For the world model, we subsequently add on top of that all the transaction Oscar sets and so on.

• 33:44Speaker 00

  And so you're training these things. You've mentioned that you're intentionally taking a hit on evals, on traditional evals like M R U or whatever.

• 33:52Speaker 01

  I wouldn't say intentionally, also to clarify, like, I am not training it.

• 33:56Speaker 00

  Yeah.

• 33:56Speaker 01

  I'm just part of the community, the co, uh, the community and bringing someone training. But I would say it's more of like the lack of care for the evals.

• 34:05Speaker 01

  So we, the reason why we add things to the data set was never about improving evals. It's about directly in response to user feedback.

• 34:15Speaker 00

  So, oh, not good enough at this, so they're, okay, just throw, throw you in.

• 34:18Speaker 01

  Yes, literally, along those, and so, so, so take, take for example, right, like, we didn't, we didn't, for, even for Raven and the world model, as we go through the training stages, right? We specifically asked.

• 34:31Speaker 01

  People in other nationalities, within our Discord community, to test it for their language. And our, our rule that we set is that, informal rule is that, the only person can decide if whether this improved world model is better in Japanese or Thai or whatever is, is a native speaker.

• 34:48Speaker 00

  Yeah. Where, where does it take place?

• 34:51Speaker 01

  So it's mostly in, within a linguistics sense, but sometimes we do a shout out in general as well.

• 34:55Speaker 00

  Okay, linguistics.

• 34:57Speaker 01

  Yep.

• 34:58Speaker 00

  So why don't, so do you have like a appointed ambassador, like you have a hundred languages?

• 35:03Speaker 01

  Yeah.

• 35:03Speaker 00

  You, you just have like a czar of Japanese, a czar of Thai.

• 35:08Speaker 01

  It's not so pointed, it's more of like, hey, this is the Japanese model, please try.

• 35:15Speaker 01

  It's, it's, it's not.

• 35:16Speaker 00

  But there, there's no the Japanese model, there's, there's one model, there's this world model.

• 35:20Speaker 01

  So if you go to world model, I don't know whether it's in, in, no, four, sorry.

• 35:24Speaker 00

  Sorry.

• 35:24Speaker 01

  Five is, you should never put five on top, because five is fully experimental.

• 35:28Speaker 00

  Okay.

• 35:29Speaker 01

  So under false, false implications.

• 35:31Speaker 00

  I see, I see, I see.

• 35:32Speaker 01

  So, so there's, you see there's a Japanese specific tune, uh, Chinese tune.

• 35:38Speaker 00

  Arabic.

• 35:38Speaker 01

  You, then for all the other smaller languages, we actually pass them from the base world model.

• 35:44Speaker 00

  Yeah.

• 35:44Speaker 01

  With itself, so, so feedback on that. So we actually released previously like ten percent train, fifteen percent, twenty percent, like as it goes through the stages and then it's like, hey, is this okay? Is it regressing? So it's like evals, but we.

• 35:59Speaker 00

  Done by, done by real humans and not, not systematically. Is there a reason that you release, you also, so you mentioned seven B, fourteen, fourteen B, by C also zero point one B, zero point four B, three B, one point five B, like what? Is, is, is that useful for people or?

• 36:12Speaker 01

  Yeah.

• 36:13Speaker 00

  Is it just for research or?

• 36:14Speaker 01

  Zero point one and zero point four is frankly more for research, but some people do try to make use of them, nothing stopping them.

• 36:20Speaker 00

  Well, I mean, is, is extra, like, these are just different architectures, different dimensions.

• 36:26Speaker 01

  Yeah.

• 36:26Speaker 00

  So it's actually extra cost to you to provide these things.

• 36:29Speaker 01

  Oh, but suppose really for the world model, what, because we are trying a new tokenizer, we are in, in. The reason why we we're trying a new tokenizer is that as as as I think I'm cut is that. One thing that we found more like I found surprisingly frustrating in existing tokenizer was that it was very English centric.

• 36:51Speaker 00

  And the existing tokenizer you took from G P T Neo.

• 36:53Speaker 01

  Yeah.

• 36:53Speaker 00

  Okay, and and just to I need to backtrack a little bit, just for people who are not following along. G P T J was the original Luther reproduction of G P T three.

• 37:01Speaker 01

  Mm hmm.

• 37:02Speaker 00

  And then G P T Neo was the bigger G P T J.

• 37:05Speaker 01

  Yeah.

• 37:06Speaker 00

  I can bring my.

• 37:06Speaker 01

  Twenty B.

• 37:08Speaker 00

  Something like that.

• 37:08Speaker 01

  Yeah, I do believe they have a twenty B model.

• 37:10Speaker 00

  Okay.

• 37:10Speaker 01

  And there's there's a cheat for do I mean for those outside of the open source.

• 37:16Speaker 00

  Yeah.

• 37:17Speaker 01

  Space in particular for the transformer. I think one thing significant for G P T Neo X was that it was one of the major models that had everything fully documented and the. Like why they make this change in the architecture and so on and so forth.

• 37:29Speaker 00

  Hmm.

• 37:29Speaker 01

  And that became like a basically a reference note for all other subsequent open source models because they were the early ones that. Was were like make doing a good transformer model.

• 37:41Speaker 00

  Yeah.

• 37:42Speaker 01

  And at least for the for the large language model.

• 37:46Speaker 00

  So G P T two was actually open source, you didn't, people didn't find that useful?

• 37:50Speaker 01

  No, people do find do reference it as well, but it's like the code is there.

• 37:54Speaker 00

  Uh huh.

• 37:54Speaker 01

  And why do you do this?

• 37:57Speaker 00

  Oh, I see, it's not documented.

• 37:58Speaker 01

  Yeah, yeah, yeah, yeah, yeah.

• 38:00Speaker 00

  So in that sense, was O P T from Facebook useful? Because I I've heard very good things about the logbook of O P T.

• 38:07Speaker 01

  Mm hmm.

• 38:08Speaker 00

  Where they they had the daily logbook and they just published that.

• 38:10Speaker 01

  Yeah, those were useful as well.

• 38:12Speaker 00

  Yeah, okay.

• 38:13Speaker 01

  The I think one thing that Neo Neo X had going for it, especially the editor committee, is that. It it's not just logbook, it's just like you could just go to just go, hey, why you do this?

• 38:23Speaker 00

  Right, and the person who trained it will tell you.

• 38:26Speaker 01

  Yeah, someone there for.

• 38:28Speaker 00

  Hopefully, hopefully.

• 38:29Speaker 01

  One of them, so that's why we had the zero point one and zero point four models because we we were just in uncharted waters here, we. So like a lot of existing tokenizer took space as a major denominator to detect this bit.

• 38:41Speaker 00

  Hmm.

• 38:41Speaker 01

  And the tokenizer we are using is actually a a lot more simplified, so. As it tokenizes, they I mean, they scan all the tags, they do a. Set this is a model of like what what what pairs well with what and so on and so forth, right? We did a similar approach, but we instead of using like this token pairs well with this and should be paired with that. We just made it a a trio list, so. So basically.

• 39:06Speaker 00

  We see try try the data structure.

• 39:09Speaker 01

  Yeah, so we just find the longest matching string.

• 39:12Speaker 00

  Uh huh.

• 39:12Speaker 01

  In that eh matching string that we had trained inside our token list.

• 39:16Speaker 00

  Uh huh.

• 39:17Speaker 01

  And then we just use that token. It's a drastically simplified tokenizer. And it doesn't use spaces as an assumption, which I know.

• 39:25Speaker 00

  Which is good.

• 39:25Speaker 01

  Yeah, and that helps a lot of the Japanese, Chinese and.

• 39:29Speaker 00

  Ah.

• 39:29Speaker 01

  Character model because they don't have spaces.

• 39:31Speaker 00

  Mm hmm.

• 39:32Speaker 01

  And I would even and I I would even argue to F S A like if you look at like like. The really large models like being OpenAI or CalD AI, right? Tokenizers are not really a thing. I mean, the in the sense that the model can work even if you tell it character by character. It's maybe inefficient.

• 39:52Speaker 00

  Did someone try it?

• 39:54Speaker 01

  I mean, there was that that jailbreak where, you know, the system prompt, you put the character, then enter, enter, enter, the you remember that jailbreak?

• 40:00Speaker 00

  No, I didn't see that one.

• 40:01Speaker 01

  Yeah, so so you can literally like instead of like left to right, you can do it up to down.

• 40:07Speaker 00

  Okay.

• 40:07Speaker 01

  And you're just eating tokens for every character. No, actually, you're eating two because there's also the new line. Eh and and the model understood it because there's enough dumb data on the internet. That it has learned how to deal with this kind of formatting.

• 40:23Speaker 00

  Got it. Okay.

• 40:24Speaker 01

  And if these models are already understanding things at the character level.

• 40:28Speaker 00

  Okay.

• 40:28Speaker 01

  Everything else is just improved compute.

• 40:31Speaker 00

  Okay.

• 40:31Speaker 01

  Because we we jump the multiple tokens.

• 40:34Speaker 00

  Do you have any idea of your dictionary size when you had when you use this tree data structure?

• 40:39Speaker 01

  Yeah.

• 40:39Speaker 00

  Because the the typical tokenizer is like eighty thousand tokens. Dictionary size. I presume you yours will be bigger.

• 40:46Speaker 01

  Yeah, it's around. I can't remember offhand. Our our previous tokenizer is around fifty thousand. Is the new X tokenizer, then subsequently, I believe this is around the same size.

• 40:54Speaker 00

  It's not bad.

• 40:55Speaker 01

  Yeah.

• 40:55Speaker 00

  Pretty good.

• 40:55Speaker 01

  We we didn't want to change too much on that size, but we just wanted to like just change the format.

• 41:00Speaker 00

  Mm hmm.

• 41:01Speaker 01

  Yeah.

• 41:01Speaker 00

  Cool. I actually kind of want to establish the credentials of this thing. So who is blink?

• 41:08Speaker 00

  Is Randall on the internet or like if you say again, never heard of this guy until he published.

• 41:14Speaker 01

  That this is real me.

• 41:15Speaker 00

  Right. And you had like, I have this paper to work with, but it was only published in May.

• 41:20Speaker 01

  Yeah.

• 41:20Speaker 00

  You found this before the paper. And and so I think it's very unusual for a researcher to effectively launch to the to the wider public without a paper and just get some kind of pretty decent community going and then publish the paper.

• 41:38Speaker 01

  I I think like a few years back, once with G P T two, transformer started to pick up steam.

• 41:43Speaker 00

  Um hum.

• 41:44Speaker 01

  And I guess the whole world is starting to think, let's just abandon neural networks. So so we haven't even gone into the code part, but like, like, so the the main reason why neural networks were bad compared to transformer was that when you when you train it, let's say you just input a token and train a token for for data sample, you have to wait for the compute to finish for that token, take the state, and then you train the next token.

• 42:06Speaker 00

  Um hum.

• 42:06Speaker 01

  And we'll get into how R W K B solves that. Uh and but basically, the whole world at that point does conclude, yeah, neural networks, it cannot scale as well. Transformer, let's just abandon it. And and everyone just went in that direction. And Blink or Bo Peng, his actual name, decided, basically, as an individual literally at the Eluter AI forum, decided that, hey, I think we can modify recurrent neural networks, no neural networks, based on the Apple paper, the the light engine that I showed previously, to make to scale this up without to make neural networks scalable and parallelizable in the same way transformers work.

• 42:46Speaker 01

  Because the reason why we branch away and focus on transformers is because neural networks were slow to train. It was never, it I mean, it wasn't so much about whether it was good or bad. It was just, no one wants to wait a hundred years for for their billion tokens train finish, even if they can throw a G P U farm at it.

• 43:03Speaker 00

  Um hum.

• 43:03Speaker 01

  And and that's where he started taking, looking into it, like, how to make the neural networks trainable in parallel. And.

• 43:12Speaker 00

  And specifically R N Ns.

• 43:13Speaker 01

  Yes.

• 43:14Speaker 00

  Okay.

• 43:14Speaker 01

  And subsequently, it was the A I and and I believe there was also a few others. Like, because he was doing it very publicly there, came on board to sponsor. And the the G P U computes required, because even though it, I mentioned that on a large context size, it is substantially cheaper. I think, I think, especially if you if you run an open source Discord forum for A I model. That every, it's like every day there'll be someone who thinks that they can train a twenty D model on a single G P U coming in.

• 43:44Speaker 01

  It's the skill is still large, even though it's like one three four one ten compared to transformer, it still needs a large G P U. So so that's why it was A I and and the rest, stability, I believe also is involved, stepped up. And donated the A one hundreds needed to train the basic models that R W K V had. And and and then, so before that, before those models were trained, right? We were like only having in theory like the toy models or the small models that this can match transformer. We have no idea whether it can match transformer at that.

• 44:17Speaker 00

  Scale.

• 44:18Speaker 01

  Yeah.

• 44:18Speaker 00

  Yeah.

• 44:19Speaker 01

  And subsequently, with the larger models, the fourteen D models and all that, that and we can compare it directly with new X model. And that's where this paper came out. So so that's the history behind it, it's like he wasn't really doing it in silence, he was doing it from a little. Then he branched out. Be be because this became a big project on its own.

• 44:42Speaker 00

  Um.

• 44:42Speaker 01

  And that's where other people started coming in, this is all. So so the the part where we say that R W K V is a neural network that can be scaled, can be wrote out as a transformer, right?

• 44:52Speaker 00

  Yep.

• 44:52Speaker 01

  The the key thing that you want to, you will want to see, right, is this diagram here?

• 44:56Speaker 00

  Yep.

• 44:56Speaker 01

  And this should be in the paper, should, no, sorry.

• 45:02Speaker 01

  Yeah. Accordingly, so what what you get, so when you do, when inference, when you are running inference mode. Ideally, you should run it as a neural network, so this this is a layer. So it as per, so classic neural networks is that you have a state, the state could be start from blank. You you process a token, you output a state. And then you rinse and repeat. And then as as it keeps doing the output, it it makes a prediction. In in in that, one thing that so subsequently for R W K V, what happens here, right, is that the we can roll out this neural network side by side. And then it it runs similar to transformer. But the key thing here is that the states are split across the layer. So this is what we call, in this diagram here specifically, this is what we call the timings and channel mix.

• 45:47Speaker 01

  These are operations within the layer. Depending on how you view it, you could view this as individual layers. Or as how we view it, we view like this collection of layers as one layer block. And each layer block pass the states to its sibling subsequently down the road. As you process the next token.

• 46:06Speaker 00

  Which is a similar R N N type.

• 46:08Speaker 01

  Correct.

• 46:09Speaker 00

  Feature.

• 46:09Speaker 01

  However, the key thing is, it you do not need to wait for the upper layers to complete.

• 46:14Speaker 00

  Right.

• 46:14Speaker 01

  Before you can go to the next token. So what happens in practice. If I to jump to the diagram, like there's this graphic here. This is not hundred percent how it run behind the scene.

• 46:28Speaker 00

  I like it.

• 46:28Speaker 01

  Yeah.

• 46:29Speaker 00

  Whoever whoever put time into this, kudos.

• 46:32Speaker 01

  I made it.

• 46:34Speaker 01

  So so so so this is how you can visualize it. So this, the first layer is the layer norm. The layer norm doesn't, this is standard layer normalization that it doesn't need to, it just does doesn't on token and doesn't need to wait for the other layers. But if you notice, right, subsequently to the right and the top, these tokens, these blocks, right, need to wait for the blocks on the left. And and this is like, once you go past the first few tokens, right, this cascades very rapidly. Especially like this is only like one, two, three, four layers. Most models have like twenty, forty plus layers. And and the cascading patterns are happening. And in practice, once you start cascading there, you just saturate the G P U.

• 47:15Speaker 01

  And that's how it starts being parallelizable to train. You no longer need to train in slices like traditional R N Ns. That was one of the key things.

• 47:22Speaker 00

  What else is the key thing?

• 47:23Speaker 01

  Other, so other things is that, so I think you you are familiar with L S D M, right? Uh, this is, this is how like traditional neural networks keeps things within memories. Within other K V, we have two channels, we call it the channel mix and the time mix respectively.

• 47:38Speaker 00

  Uh, is there a formal definition of channel mix and time mix?

• 47:41Speaker 01

  Yeah, you can see the data from the respective time mix and channel mix move to the next, to the next segment. How time mix is designed per se was that it it's how it retains, similar to L S D Ms, right? Where where it processes state and the input, it may decide to discard certain states and keep new things in the state. Time mix does the same thing, but with a different formula, so it replaces the L S D M in a sense. And and it can decide to keep things indefinitely, so this represents a long term memories if you want to view it that way.

• 48:12Speaker 00

  But classically, the problem with that is that it struggles with long distance.

• 48:19Speaker 01

  Correct.

• 48:21Speaker 00

  This has the same issue?

• 48:22Speaker 01

  So that's, that's subsequent, it struggles with long distance because it also needs to keep track of both near term memory and long term memory.

• 48:31Speaker 00

  So you split it up?

• 48:32Speaker 01

  Yeah, effectively split it up, so channel mix is subsequent.

• 48:34Speaker 00

  Is this the perfect memory?

• 48:36Speaker 01

  Yeah, this is the closer to the perfect memory, that is the short term. So so uh L S time mix, it has trainable weights on what decides to keep and discard.

• 48:45Speaker 00

  Okay.

• 48:45Speaker 01

  Channel mix, it has a very strong bias in it towards like just the next token. So so you, so subsequently, it was just like, as uh like memories are stored in the lower layers, it just slowly shifts upwards through the channel mix, and this is the short term memory, which at some point, as it just shifts all the way up, it will just disappear into the void. At that point subsequently, then time mix will should be retaining the longer term memory.

• 49:14Speaker 00

  So we took a break for a bit, but now we're trying to cover like what is the big aha moment for you? And you said it was something to do with cost.

• 49:22Speaker 01

  Correct.

• 49:23Speaker 00

  So we have this chart on screen, there's literally a chart of quadratic scaling versus linear scaling in terms of G P U time spent in text generation, and you said it was a training time and an inference time, they're just basically in everything that matters.

• 49:35Speaker 01

  Correct, so so I mean, so look back to how R N N works in, from a high level. We do an O one operation. On a token, create a state, O one operation, create a state. So, so these are skills, you want to throw a thousand tokens at it. It's just on inference, it's a skill in your. Subsequently, for transformer, you, you, you taking a token, you, you, you, you infer, you process your first token. It may be O one here, subsequently, when you generate your third token, you need to compute your second and first. And then, right, so you do your one thousand token, you need to compute back your nine hundred ninety nine previous tokens. And as this keeps growing, growing, this is your quadratic scaling.

• 50:19Speaker 01

  And this is why, why we had, we had this graph of the amount of cumulative GPU time that you need to spend to generate all these tokens respectively. And this is fundamentally just transformer versus neural networks. Yeah, on inference, the reason why, and, and subsequently, like, neural networks did have the disadvantage of, let's say, not being able to parallelize during training. But as I, as I covered, R W K V kind of solved that by splitting, effectively splitting the layers, allowing you to train different parts in parallel. And like, some people will go into the academic debate of like, technically, the second and third token is not paralysable until the first is done. But once you get into like, I can saturate a GPU land.

• 51:04Speaker 00

  It's just way better.

• 51:05Speaker 01

  It's, it's a academic debate, we are done. And so, training, in essence, has always, I mean, this is big for transformer. On your network is, I need to do an inference pass, I look at the logits. I then I backprop to see what went wrong and I update the weights. Yeah, so the inference is the forward pass, you still need to, it's part of the training cost. As you backprop as well, as you backprop as well, like, having needing to only look at the current cell tokens in the state instead of like everything, also reduce the amount of things that you need backprop. So it's just that, it's like, there's so many factors involved in just reducing the overall inference and training time. And that was something that appealed to me because in the long run, I mean, almost once a model just run blazingly fast, right?

• 51:48Speaker 00

  Yeah, and also on minimal hardware.

• 51:51Speaker 01

  Oh, yes.

• 51:52Speaker 00

  Which, as far as I understand, you still have fourteen billion parameters, that's not going away. You still need the, the RAM to store fourteen billion parameters worth of stuff, that's not going away.

• 52:02Speaker 01

  Yeah.

• 52:03Speaker 00

  Okay, so RAM is unchanged.

• 52:05Speaker 01

  Yeah, on the RAM side. But the working memory is reduced. So typically, you need more than fourteen for transformer. I mean, let's not, let's not touch quanti- quantization. But in this case, we don't need to keep. Like, if you really, really want to, like, save RAM. You, it is possible for you to do token by token inference, so that you don't need to keep your states in history. You only need to keep your current token state and your next.

• 52:32Speaker 00

  Yeah, yeah.

• 52:33Speaker 01

  Okay. And, yeah, and, and there's actually like one segment on our committee, just purely putting other K B to C plus plus base model.

• 52:43Speaker 00

  Who and the next.

• 52:44Speaker 01

  Yeah, and running it on pies and stuff.

• 52:47Speaker 00

  Raspberry Pis.

• 52:47Speaker 01

  Yeah, it's interesting to watch.

• 52:49Speaker 00

  It is a chart about performance and it shows that RWKB is competitive or, or actually better in some of the reasoning challenges. Which, that's something I definitely would look for, right? Like, and it's, it's fine if, like, your, your speed is faster and all that, but if the reasoning quality sucks, then it's not a very useful language model.

• 53:08Speaker 01

  Exactly.

• 53:09Speaker 00

  So.

• 53:10Speaker 01

  So this is like literally us saying, there's.

• 53:13Speaker 00

  No trade offs.

• 53:13Speaker 01

  Yeah, you don't lose out in that process.

• 53:16Speaker 00

  Okay. Big question then, why isn't RWKB a bigger deal right now?

• 53:21Speaker 01

  So, one, we are not a commercial organization.

• 53:25Speaker 00

  Okay.

• 53:26Speaker 01

  This is literally the pure open source play, so.

• 53:29Speaker 00

  But you could have done the stable diffusion thing. Which, you know, stable diffusion launched, it was by a bunch of nobodies before that.

• 53:37Speaker 01

  Mm hmm.

• 53:38Speaker 00

  It's from, like, literally split out from Luther. And, but they definitely had some hype. They had, they definitely, like, you know, I interviewed Sharif, Shamin. The, the reason I ask you so many things about how did you find RWKB? Cause I think the, the generalizable skill is how to be early in AI.

• 53:53Speaker 01

  Ah.

• 53:54Speaker 00

  Cause being early in AI is very valuable. Cause then you were there to see how things developed, in- instead of like picking it up later like me. Anyway, so. Yeah, why is it not a big deal?

• 54:05Speaker 01

  You want me to be frank?

• 54:06Speaker 00

  Yeah.

• 54:07Speaker 01

  We just suck at marketing.

• 54:08Speaker 00

  Okay, that's fair.

• 54:09Speaker 01

  I mean.

• 54:10Speaker 00

  This is part of it.

• 54:11Speaker 01

  Yeah, this is part of it, like, so like, maybe.

• 54:14Speaker 00

  But I don't think that is entirely the cause.

• 54:17Speaker 01

  Yeah, I'm sure you definitely, I think the other major segment right now as well is that, is that we were really late on the paper.

• 54:25Speaker 00

  Okay.

• 54:25Speaker 01

  Like, one of the weirdest thing right now is, I, I, weirdest thing right now, I feel that is that R K B is starting to have its moment right now.

• 54:33Speaker 00

  Okay.

• 54:34Speaker 01

  Is that ever since the initial paper came out, there was ResNet, there's a, I think there's two more, there's a few more additional papers coming out. One from Microsoft, one, one from other organizations that are literally exploring the whole idea once again of scalable neural networks.

• 54:50Speaker 00

  Okay.

• 54:51Speaker 01

  And they are citing R K B as power B as well.

• 54:53Speaker 00

  Okay.

• 54:53Speaker 01

  And, and I think for most, almost like, I think it's existing, why switch to this model? When even though we have proven that, yes, it's scalable to seven and fourteen, and that it can match transformers at similar param and training size. But all this is very academic because the, the, the community, right? The commi- the community at large, especially for the English community, speaking community, right? They don't really care about this. They care about what's the best model that I can run on my computer, at least within the open source space.

• 55:32Speaker 01

  And by that, and even though we match in performance for things in the same data set, the key word is same data set. Like, this benchmark is not, it's not even red pajamas yet. It's, it's the power.

• 55:45Speaker 00

  Mm hmm.

• 55:46Speaker 01

  And, and when you have models that are like, being like Falcon, being trained on much larger data set.

• 55:51Speaker 00

  Mm hmm.

• 55:52Speaker 01

  For, especially for an English use case, it makes more sense to use it.

• 55:56Speaker 00

  I see, so there there will be another paper coming that is RWKV trained on red pajama.

• 56:02Speaker 01

  And for larger data set, yeah, and so on and so forth. So so I think that's the, we are still in the stages of reaching that point where we train on the larger data set. The only reason why we have a bigger outsized impact compared to like the other models is frankly because half of our Discord came in not for English, it's for other languages.

• 56:23Speaker 00

  Yeah, that's great. And there is a definite very US and English centric bias towards these models and and it's it's to me kind of poetic, like there's nothing in in the architecture of RWKV that particularly biases it to be really good at other languages.

• 56:40Speaker 01

  No.

• 56:40Speaker 00

  It's just that as a community, you decided to prioritize it.

• 56:43Speaker 01

  Yeah.

• 56:43Speaker 00

  In your tokenization in your data sets. That's it.

• 56:46Speaker 01

  Yeah, that's it. I I I I would even argue that I'm surprised, more surprised that, especially on on the European side of things, that we don't have more more more models that actually focus on even the European languages. Because there is like a softer jump to character, Japanese and Chinese characters.

• 57:08Speaker 00

  Because they're all romantic.

• 57:09Speaker 01

  But I think back to the benchmark, what excites me most still about this is that it just means that we just need to scale. We just need to scale this model and with the right data.

• 57:19Speaker 00

  To like forty B?

• 57:20Speaker 01

  Forty B, sixty B.

• 57:22Speaker 00

  Yeah.

• 57:23Speaker 01

  I I mean, it's parameters is one thing, it's data sets.

• 57:26Speaker 00

  Okay.

• 57:26Speaker 01

  And GPU time.

• 57:27Speaker 00

  Yeah, so you and I are talking offline about ideas for getting data, getting compute.

• 57:31Speaker 01

  Hmm.

• 57:32Speaker 00

  And all this, okay. So this is like a project that's that's ongoing. Okay, anything else for the future of RWKV?

• 57:39Speaker 01

  And the biggest one would be, okay, so this is back to how, remember I said evals doesn't hide or doesn't highlight everything. Being realistic on another weakness on RWKV's side is that now with the rise of like, let's say, hundred K or thirty two K context size windows, transformable model. RWKV currently is trained to handle, let's say, eight or even some people have already trained it to sixteen K.

• 58:05Speaker 00

  Yeah.

• 58:06Speaker 01

  Sizes. It has, and well, it will, as a neural network, it will happily keep going on for infinite context length. It will just keep generating. Does it do well? That's the the the answer is no, because you you didn't train it to handle that situation. And that's actually a challenge. So, for example, if like the prediction, the the power test loss, right? It does improve over time, let's say, if we go down the context length. But this is if we train it. And what is not seen here is that if we were to do, do, let's say, run it further, it'll just go back up. Because it was not trained to handle that. Well, it technically can run. It it suffers from the longer context length. And that that's the part where other K V, especially in like Q and A tasks and huge documents.

• 58:51Speaker 01

  Like, you get Claudia to summarize giant documents. None of this is fundamental. It's just you need more money. Yeah. That's it. And and no, there is actually a fundamental part. So what what one of the things that I was doing and am actively helping within the committee right now is that we found that the existing way to scale the memory was not that efficient. And we were just being realistic ourselves. If we want to hit a hundred K, we need to change this. So so one thing, one thing that I'm actually looking forward to right now is actually those experiments. We have already, we have already started scaling, scaling things to be able to to handle things at transformer scale. Be it the four K, eight K, in terms of like how it handles memory really well. And we want, and we found it, we want to like extend it to be like sixteen, thirty two and sixty four.

• 59:35Speaker 01

  And we we that is within our roadmap. And that's the exciting thing, because once we have that, like it able to handle long term memory within those sizes. It removes what many people in the committee felt, right, was the last architectural limit. Because once it's able to handle memories like context length with the same as transformer. We no longer need to do all the, like, you know how existingly like people do like long conversation transformer, they just discard the rest. And the sliding window, this is like the better version of sliding window you have. The model can handle the sliding window perfectly, where it can keep remnants behind it. Sure.

• 1:00:15Speaker 01

  And that's something that I'm really excited and invested towards, because this is back to the full circle. What how I came into R K E. I want my model to handle hundred K tokens, four megabytes of H T M L, whatever I throw at it and be able to process it. But it'll be it'll be lossy. The the later half will be lossy, but the key thing is extending the non lossy part. And we are aiming to extend the non lossy part.

• 1:00:43Speaker 00

  So, you know, you have displayed today an impressive amount of knowledge just across the, you know, all this stuff. And you're, you don't have like a research background. You advise to A I engineers getting as deep as you, who want to get as deep as you.

• 1:00:59Speaker 01

  So I think your article articulated very well that there are, there's going to be divisions within how we approach this. So A I engineers, model trainers and data set curators and M L scientists. So I'll loosely, loosely define it as a tree. I ignore the full stack because every company needs it. So within this tree space, there is actually a lot of ways anyone can come in without knowing anything. So let's just start with A I engineers. Don't be like, even though this, this whole topic, we even dived into how the layers work. We also showed how the math works. Frankly, for an A I engineer.

• 1:01:34Speaker 00

  Yeah, not required.

• 1:01:34Speaker 01

  You don't need it. Do, your, your main, your main thing that you needed to do was to frankly, just play around with chat G P T. Of all the alternatives, be aware of the alternatives, because be very mercenary. Swap out the cloud there if it's better for you, or swap out the open source if it's better for you, and just play around the prompts. Learn, learn their prompting techniques like one shot, two shot, few shots. And, and then from there on, you can start building your agents, stacking, stacking your prompts and in sequences and stuff like that. And you are able to build applications that do a- anything in, in terms of the A I space. And all this without knowing all this nerdy stuff with all the en- hard engineering.

• 1:02:18Speaker 01

  Because that's all you really need to actually build a product for the user. Remember, you are supposed to focus on making it for the user. They don't care if it's R W K V or transformer underneath the hood. They just care that it helps them. And, and I would say like motion probably is like probably one good example of how they use it. Because we know underneath the hood is OpenAI, but it's, you really use.

• 1:02:41Speaker 00

  It's good.

• 1:02:41Speaker 01

  Yeah.

• 1:02:43Speaker 00

  No, but so I obviously agree with all of that. Let's just say that people are there already and they're just curious. They they wanna do what you did.

• 1:02:50Speaker 01

  Oh, so that's where you start going down the layers.

• 1:02:53Speaker 00

  Yes.

• 1:02:54Speaker 01

  So the the next layer, the next layer you go down in is is subsequently training the model from scratch, fine tuning and incurring the data set. And this is this, this is from, this is where you still do not need to know the math, but you need to know like, you need to have a rough sensing on how the model works. And how the certain models, and in in this even within the transfer, open source transfer, certain models are better trained in certain sequences with certain learning rates. And you just need to get a few of it. So this is just like, collect the data set, try it, see the loss.

• 1:03:28Speaker 00

  You you literally did this.

• 1:03:30Speaker 01

  Yeah, at least for R R K B and the.

• 1:03:32Speaker 00

  That's a lot of work.

• 1:03:33Speaker 01

  Code gen models. Yeah, it's not a cheap work too, because you need G P U's.

• 1:03:36Speaker 00

  Okay, and that take, that took you how long?

• 1:03:39Speaker 01

  Oh, I do, I think I code gen alone was like six months and then this R R K B, I've been doing this for like another six months. And and that is the, it's just pure experimentation, like there's no right or wrong, because like, especially if it's in a different domain, like. Like recently, I was like helping someone on the R G B D D S C O T regarding the music, music generation domain. And my assumptions for learning rate and all the patterns, right, were just completely thrown out the window, because the music model just fundamentally is different in in those sense. So that is the. The exciting thing is, because it doesn't really have any specific rules and guidelines, until you get cer- until you try and error to a certain space.

• 1:04:21Speaker 01

  It also means that you coming in is as fresh as anyone else coming in last year. It's really that kind of uncharted space for for everyone and and especially as you start exploring to new domains. Your existing, your existing knowledge may actually matter, because sometimes, like, I mean, I think a few papers already covered this, that's like, how you train your model in certain sequences. Also matter, like you wanna train a certain set of knowledge and then and then you extend that knowledge subsequently. But if you're talking about material science or genetics, how am I supposed to know what is foundational knowledge and what is extended knowledge? I have no idea. Maybe you do.

• 1:05:00Speaker 01

  Um, I'm just picking an example, yeah. So and the same thing for music and so on. So those are things where even though you're outside the space. It's where you can come in, just at the data set level. Now you want to peel off to the next layer, let's just say, let's just say you want to look into modifying the model, the, the, the, the, the, the foundations of it. I think one of the beauties about this current boom is that even though I dipped my toes early, like before the transformer wave and into an early neural network phase, frankly, almost everything that is, that matters was basically in the past four years.

• 1:05:40Speaker 01

  Like, there were a lot of things that be in academics that were before that and in all, and they were mostly dealing in models that were under a billion parameters. They pretty much no longer matter. And can you be more specific? Like, okay, I know I'm shooting myself in the foot because I, I don't have a career in neural network, but if you're just trying to get transformers to work, you don't need to know R S D M.

• 1:06:05Speaker 01

  Yes. Yeah, you don't need, yeah, there's a lot of like pre-knowledge in neural networks that is irrelevant in the transformer era. And maybe some of it will have a resurgence, but to get up and running is not a requirement. And I think this is where you could either go the very academic way or reading papers and stuff. But frankly, what I found was way more useful was Akabatiya, his series of videos. Series of hero. Yeah, that is really, really good. I think even if I, even though I read some of the, read some of the papers and guides before that, it really helps that it starts from zero because you can see how it happens part by part.

• 1:06:50Speaker 01

  And, and even though we will not use how the exact same code they use, because like he re-implemented the back problem and all that, and we are just going to use Torch for that. Yeah, that, that, that's where you get the aha moments on how these building blocks work and how it formed to place. And like, like I had fundamental misunderstanding of how backprop worked until I actually watched this video. Oh, really? Yeah, and, and I think that's the scariest and craziest thing about AI models sometimes is that you can actually have fundamental misunderstandings, but as long as you make the building blocks and you connect and, okay, loss is great, it works.

• 1:07:26Speaker 00

  Yeah, well, so you know, even the gods of the industry, you know, I don't know if you read the Swigloo paper. So there's these like, there's all alternative activation functions, like this ReLU, and then there are people are always looking for different slopes. And very famously, the Swigloo paper had this line in there that was like, yeah, we don't know why this works, but it works.

• 1:07:49Speaker 00

  Can't explain it.

• 1:07:50Speaker 01

  Yeah, it literally happens here and there on this card too. One of the funny things that I'm doing right now in R W K V five experiments is that, okay, we are going to do this change. We're going to run this train, make your prediction. Will this model beat this model in this loss curve?

• 1:08:08Speaker 00

  As a game, as a betting?

• 1:08:09Speaker 01

  It's a very informal, it's it's it's it's literally a body kind of like, kind of bet, but. But it it it, the fact that, the fact that we can do this kind of bets, even though we like, understand the code. It's like, it just goes to show that how often like, oh wait, this didn't go as I predicted, no one. And and and that's why, that's why even if let's say you don't have a PhD or so on so forth. Heck, even if math is not your special agent, you're coming in as a developer. I'm going to come in, I'm going to say frankly, like, I didn't come from the research right now. The extremely math heavy stuff is what I struggle with.

• 1:08:48Speaker 01

  My, uh, what I do sometimes is I copy and paste the math into G P T four and ask it to explain to me, which is good in plain old language.

• 1:08:55Speaker 00

  It's very good at that.

• 1:08:56Speaker 01

  Yeah. Yeah, and so, but the thing is, there is lots of value add beyond that. One thing that I realized, and this is not specific to R W K V, this is, this also happens, happens across a lot of console models, is that.

• 1:09:12Speaker 01

  A lot of M L S like this, when they, when they really build this stuff, the focus was more of like, always get it to work. It was never about getting it to work efficiently or getting the code documented or organised.

• 1:09:23Speaker 00

  Um.

• 1:09:24Speaker 01

  And stable diffusion literally went through this whole journey. They had the giant, they had the code and the model that worked.

• 1:09:30Speaker 00

  Um hum.

• 1:09:30Speaker 01

  And the community just started and and engineers that came in with zero machine learning background. Started picking it apart, it's like, no, this this could replace this with this that does the exact same thing, but it's more efficient. Like, one of the major breakthroughs, for example, for for G G M L recent, and this happened, happened some time back for a bit. Lama models and that was that while someone external from the A I committee went in and implemented memory mapping.

• 1:09:59Speaker 00

  Yes, I saw that, yeah. I forget her name, but yeah, just Justine dot Law is her URL.

• 1:10:05Speaker 01

  Yeah, and she didn't come in as an A I expert, she came in as a software engineer.

• 1:10:10Speaker 00

  Yeah.

• 1:10:11Speaker 01

  And.

• 1:10:12Speaker 00

  These are all just very, very straightforward. You you know, in her world, this is this is normal, whereas for the researchers, they will be like, I don't know.

• 1:10:19Speaker 01

  Wait, what is memory map?

• 1:10:21Speaker 00

  Yeah, exactly.

• 1:10:22Speaker 01

  Yeah, and there are a lot of things like, like, one of the jokes that I have right now is that every month there is a research M L scientist that is rediscovering the number thirty two.

• 1:10:32Speaker 00

  Why?

• 1:10:33Speaker 01

  Because, uh, be it like or someone in the committee writing the inference code, because G P U's, uh, especially, especially CUDA, GPU's. Tends to work really well when they align to the batch size of multiples of thirty two.

• 1:10:47Speaker 00

  Oh.

• 1:10:47Speaker 01

  And if you've been in the gaming industry, especially when you write shader code, right, this is like, like, well known, like, just just given knowledge. And and and and people are just constantly rediscovering, oh, maybe if I just adjust my data set or my data size to to fit this. Batch size, suddenly I get ten percent improvement. And and yeah, and it's like, these are things that.

• 1:11:13Speaker 01

  Once again, because they were so focused on just making it work, that they wouldn't know outside space. And that's why I would say, right, if anything, right, now is the best time, I mean, that you don't know AI. To have people from different backgrounds come in, because your contribution could be from data set level, how to train the knowledge. To shade a code, to heck, how to memory map, how to cache data, there's so many gaps.

• 1:11:38Speaker 00

  Cool, great. So yeah, thanks so much for being very willing to get on and talk with no prep. Oh, we did some prep, but it's very unusual podcast episode, but I really enjoyed it.

• 1:11:48Speaker 01

  We literally just met yesterday in Singapore.

• 1:11:50Speaker 00

  Well, but I know you've been on the Discord for a while and I I could tell you like you're very serious about all this. I think it's very unusual for someone, like you have a job, but but this is like a second job, essentially.

• 1:12:00Speaker 01

  Yes.

• 1:12:01Speaker 00

  But you are really enthusiastic and passionate about it, and I think that's very rare, and I I do want to encourage more people to to do it. And so thanks for sharing.

• 1:12:09Speaker 01

  Yeah, thanks for having me here.