The Quest for Community-Trained Open Source AI Models

Bowen Peng and Jeffrey Quesnelle of Nous Research join a16z General Partner Anjney Midha to discuss their mission to keep open source AI research alive and activate the community of independent builders. The focus is on a recent project called DisTrO, which demonstrates it's possible to train AI models across the public internet much faster than previously thought possible. However, Nous is behind a number of other successful open source AI projects, including the popular Hermes family of "neutral" and guardrail-free language models.

• 0:00Speaker 0

  We're lucky right now because the state of AI as a science is incredibly new and it truly has a wide green field. And unlike lots of other areas of science where if you look at something and you think, why hasn't somebody done X, Y, and Z, unfortunately, often the reason is someone did try it and there's like a reason it doesn't work, right? That really isn't the state of play right now with AI. Like there that really you can shake a stick at almost anything and make groundbreaking novel research in this area.

• 0:41Speaker 3

  News research

• 0:42Speaker 0

  is an open source AI accelerator effort. So our goal is to uh accelerate artificial intelligence and to do it in a way that uh brings it to everybody. And when we say bringing in to everybody, it doesn't just mean that everybody can use it in the sense that like it's a product that we bring, but it's that everyone can also touch the technology underneath it. So if you just want to use the like the AI, you can just use the AI. If you want to open up the code and see the code and use it, we believe that's as important as bringing the access of the actual using it, is also bringing the underlying technology to everyone. We've seen that open source innovation, it's it's a multiplier in the technology stack. Every kid who wants to learn to do something starts by going with the open source free thing that they can get on the internet when they're you know 1012, what can I do and touch this?

• 1:33Speaker 0

  So our goal is to make sure that that those the building blocks of this transformative technology are are kept open and that we're able to do the research to bring uh state-of-the-art AI to everyone.

• 1:45Speaker 2

  What are you guys trying to do in terms of the roadmap of the set of milestones you guys have decided to focus on to get there?

• 1:53Speaker 1

  I think for my purpose at news, we are doing fundamental research, we try to push the boundary of what we can do with uh as little compute as possible, contrary to what usually others do. So we are really a split, like uh we are really really split, like all kinds of people, right? It's like it's not a centralized, really um cohesive group where we're doing a lot of stuff at the same time. So we explore really uh a lot of alternatives. Um we try to do a lot of

• 2:28Speaker 0

  cool stuff. It's interesting because the AI space right now, if you view it from like a scientific perspective, unlike a lot of other areas of science that are highly ossified. So like if you want to be in biology or chemistry, like you have to go through the academic process, and like maybe you get your PhD, and then like maybe if you saw like study for a long time, you can make like a tiny contribution to like the the store of knowledge. We're lucky right now because the state of AI as a science is incredibly new, and and it truly has a wide green field. And unlike lots of other areas of science where if you look at something and you think, why hasn't somebody done X, Y, and Z? Unfortunately, often the reason is someone did try it, and there's like a reason it doesn't work, right? That really isn't the state of play right now with AI. Like there that really you can shake a stick at almost anything and make groundbreaking novel research in this area.

• 3:17Speaker 0

  And so because we have that, because we have that, we're in that time and in that space right now. We bring together a lot of people who have a lot of different views of the world. Um, and maybe for whatever reason didn't go through the traditional academic sort of, you know, not to disparage it, sort of your state your typical Stanford MIT kind of like path, and we bring other like divergent people together, and we say there's this incredible thing that's happening where we can do research and and make fundamental breakthroughs for everyone, and you can do it as who you are, not just as this like one monolithic piece of a machine. Uh so we consider ourselves very highly individualistic and and personal versus like we are this one organization. Um but that's what we're doing. We sit in a unique time in space right now, and so we're we're taking advantage of that. And

• 4:02Speaker 2

  how did you guys get here? Like what are your backgrounds?

• 4:05Speaker 0

  I actually uh I grew up in Detroit and worked in automotive for like 15 years, um, working on like autonomous driving and stuff like that. And I actually I did that for 15 years and I enjoyed it, but I'm actually not like a car guy. And uh at one point uh I actually got um like crypto pilled. Um I was uh I was working on my master's uh research on Zcash, which is a cryptocurrency, but it was kind of like static, and I eventually found Ethereum, which is this like programmable blockchain. I'm like, this is awesome. And so I started writing all this smart contract code at night, and uh while I was still doing my job, and my wife was just like, this is what you love, you know what I mean? Like this is what you love, and she really gave me the push to be like, why don't you just go do this? Um so I spent I I left behind like the nine to five, the healthcare, like all of that, left it all behind to sort of strike out on my own and just follow my interests and see what where that took me.

• 4:56Speaker 0

  And through that process, I discovered I'd always been sort of AI adjacent because our customers were at my previous company were doing autonomous driving. Um and then I I I saw I was just you know on the internet and I saw stable diffusion, when the first stable diffusion came out. And I was like, okay, there's there's something happening here, and I need to find out what's happening. So through that uh curiosity and wonder of being like, I need to really understand how this works, and spent maybe just a year just going through the scientific process of learning everything there is to do about like the modern actual AI, the math behind it, all everything behind it, and I had that freedom now to be able to do that. And through that curiosity and wonder, um, you I'm one of these people who wants to be able to touch the technology that I use. I want to be able to step through the code because that's the way I interact with the world by like stepping through the code.

• 5:47Speaker 0

  And what I what I discovered at the time was that uh ChatGPT had come out, and I'm looking at I'm like the open the state of open source AI at the time was way behind the closed competitors. And I was just like, how are we ever going to like how's open source people like me? How are we ever gonna get to that pinnacle on the hill of like what's happening at open AI and some of these other places? And I just said, well, what are some of the things that are stopping that? Is there a reason that we can't have this from like a scientific perspective? And it turns out it was just some people needed to do the work, but and and uh we started by I was just an anonymous person on Discord and doing some research and sharing, started sharing with other people who had a like-minded thing and a like-minded approach, and uh there was someone invited me to this Discord called news research, and it was just other anonymous people hanging out, exchanging ideas, and and sort of out of that came what we are today.

• 6:39Speaker 0

  So that's how it started from here to there.

• 6:42Speaker 2

  How'd you guys need what's your record? How do you fall into it?

• 6:44Speaker 1

  Yeah, for me, like that's a long story, but at first, you know, when I was a child, I really loved tinkering around stuff, right? I I love toys, electronic toys, and those uh those like there was some kind of like electronic Lego where you could do like um create a circuit, like create AM radio with like just Lego blocks, and those type of toys was like so interesting to me because I love tinkering. So when I went into um university, college, I wanted to work with computers because computers are so like um you can do everything, right? That that's a promise of like technology is that using computers, you can basically do everything a human can, and that was like my my dream, basically, right? And when I was um surprisingly, when I was in uh university, my first actually it was one of my first classes, uh it was called it was like programming, one of the programming classes.

• 7:38Speaker 1

  The teacher was um a um he's called uh Aaron Courville. He's a professor in Mila, so he was actually a really really good um amazing prof in deep learning. And he showed us like the first class, he showed us what machine learning can do. And he showed us slides on like generative models. That was back in like 2000, uh it was like 10 years ago. And that was amazing. I'd never seen such a thing, like how computers can generate images, right? That was back in like 2014, that was GAN's, right? At that time. The GANs was the newest thing, like nobody knew about GANs, but he did. And he showed us this like really crazy slides about machine learning generative models, and I was like, I have to study this. This is my this is the future, right?

• 8:23Speaker 1

  Like you could generate music, art, right? Like all of these types of things, I was so amazing. And then I just went into machine learning. Uh I also did my master's was in computer graphics and machine learning because I always loved computer games. I wanted to make a game or some kind of thing, but as AI progressed and as ChatGPD came out, right? Um those um disstable diffusion models, I was like, yes, this is the future. Like generative models are going to change the world, and I really want to be part uh part of it. And then I did some like freelance research work, and I met Jeff. We met on actually on

• 9:01Speaker 0

  uh just on Reddit. He uh he uh we were working on sort of the same area of of research, and I made a post on uh Reddit on our local llama, and he'd made a post, and I still remember I was going to pick up a pizza and like I just got an email and I opened it up and he had just called emailed me like, hey,

• 9:18Speaker 2

  I saw this stuff you wrote, like look, I have this new

• 9:20Speaker 0

  results too

• 9:21Speaker 2

  that got

• 9:21Speaker 0

  together, and and sort of out of that became, you know, what we are today. Yeah.

• 9:26Speaker 2

  Maybe it would be helpful to just first first talk through some of your earlier work. What kind of projects have you guys released so far leading up to this

• 9:33Speaker 0

  troop probably we're best known for our Hermes series of AI models, which are our models that we train that um are sort of neutrally aligned. So we don't, it's not like um we are making uncensored models that we like think, oh, you need to be doing all these terrible things with it. But rather it's a series of AI models where what the user directs the model to do, the model feels the model will do that. Um and so this is maybe in contrast to some other. Other closed providers who have to put sort of this, these guardrails around it. And uh and and those guardrails exist for maybe for good reasons, especially as a centralized sort of US-based company, maybe or something like that. Um but what's interesting is that most AI models that people interact with today take on the persona of what we call the helpful assistant.

• 10:22Speaker 0

  Um hi, I am in you have used a chatbot, you might the helpless harmless assistant, sort of the the front door secretary sort of of AI. Um and that's sort of the very neutral approach. And it will sort of, oh, that's a good idea. Have you thought of this? That's nice. But it has been adopted to this specific persona. Um what we try to do is make models that you can instruct to take on any persona that you derive.

• 10:46Speaker 2

  So

• 10:46Speaker 0

  we say, instead of training the model to be like, you are an assistant under these under these constraints, help them the best you can. We make models where we say, take what the user says and adopt that worldview. Really adopt the world view of whatever the user wants you to take on, give that as the a priori thing to say is the truth, and then play out the scenario from there. Now, obviously, the helpless harmless assistant is one of those personas that an AI model can take on, but it's not all of them, right? And certainly we try to create that expressiveness too. And it comes from like this individual individual individualistic uh approach that we take, which is that everyone should be able to interact with the system, not in a way where it's it's moralizing down to you, but it's empowering you as a person to make to be a better person.

• 11:36Speaker 0

  I actually gotten excited about language models because I love sci-fi and fantasy stories, but I'm a terrible writer. And and these uh helped me to like write my own stories and come up with my own plot lines. And and right now, if I try to do that with some other models, they might say, well, I can't, that's copyrighted material. I can't, you know what I mean? And so, but like this this mod these models that we make help me express myself. And so we try to make models that for whatever that whoever you are, they will help you express and like be an extension of you. And then we also did have done other research uh like fundamental research. So we just do um looking at specific technical problems. So uh Bone here's the lead author of a method we developed called Yarn, which is a context window extension method that uh we we released and did the research on.

• 12:27Speaker 0

  It is now used by every every model you use nowadays, everything everything ChatGPT, Llama, DeepSeek, all of them, they all used the methods that we pioneered in in the Yarn paper. Um and that came out of this idea of uh the open source AI at the time only had these very, very narrow context windows. Like they can only deal with like a very little bit of text that you could talk to them, then they the model gets amnesia after that. And we were looking at what OpenAI had with ChatGPT, and they could take like 4,000 words. And right now the only open source models could take like 500. And we're like, how are we gonna bring this to everyone? We want to bring this to everyone. And so we just did the technical research to sort of knock down those barriers that kept us from offering the current state of the art to everyone.

• 13:16Speaker 0

  So we do research on that side, we also have our models, and that's kind of where we've um we've been previously, but when it when it came to distro, we had the same sort of attitude, which was what keeps the open source community from being able to create their own state-of-the-art AI from the ground up. Um we are in the open source world at least, uh, very beholden to the goodwill of several organizations. And number one of them probably is is Meta. So what they do with the Llama uh models is uh uh phenomenal, and I like I I I love Mark, you know, I like I love Mark. But we had this, we looked at the world and we said, what happens if he can't do that anymore?

• 13:55Speaker 2

  And

• 13:55Speaker 0

  that may not even be like up to him, you know, in particular. There's lots of things happening in the legal space, all around that. So we said, well, so suppose tomorrow every company said we can't release open source models anymore. I'm sorry, we just like it's just it's just bad business, you know, it's not like an anti-thing, but it's just bad business. Um what where would we be in the open source AI space? Um and so what are the technical problems that would keep us from being able to replicate that on ourselves? And and it turns out the real big problem is that when it comes to training models, the current paradigm for training models requires that all of the GPUs that train the model, these you know, the these computers that do the training, they all have to be like in the same room. And like that seems very counterintuitive, but like that is just like a cold hard fact, and that cold hard fact is because the way the technology stack has grown, it it assumes that all of the uh little brains that are training the model can talk to each other very fast.

• 14:52Speaker 0

  That's like the most the simplest way to do it is they can just all talk to each other very fast and they're able to train it. And that's okay if you put it all into a warehouse. And if you're a single entity who can just put it into one data center, of course, you're gonna take all your GPUs, put them in one data center, here's your data center, it's all good. But we, and sort of the the collective we of the open source AR movement aren't one entity. So how can we cooperate to actually train a state-of-the-art AI that we all own? And there were a bunch of technical problems that ultimately led down to the fact that the bandwidth on the internet is much smaller than the bandwidth between these GPUs that they train in these data centers. Um and that may or may not have been like uh uh something that was uh insurmountable. And in fact, for a lot of times people thought it people thought it was insurmountable, that there really wasn't a way to do this.

• 15:44Speaker 0

  But we sort of uh, and Bowen can talk about this too. It turns out that most assumptions in the AI space right now are a product of that's just how things had been done when when there was not nearly as much as much energy and attention to it. So someone made an assumption maybe in the early 90s that everyone just kind of has continued to, you know, go with, and we now like can revisit it and look at those and see that there's actually a lot of of growth there.

• 16:09Speaker 2

  There's

• 16:09Speaker 0

  so

• 16:09Speaker 2

  many different problems that a research team like you guys could work on between Hermes, uh you know, Yarn and now distro, what is usually what is your criteria for deciding what to work on?

• 16:21Speaker 1

  I think the the biggest criteria is that this research should be um as fundamental as possible because when you get into the engineering, the actual training or the actual um data collection, that needs really big skill, right? Right. And as a small group, you cannot go to that skill. So if we look at everything that's very very mathematical, very like that you can tweak a lot of hyperparameters can change, then that's really good because we can do smaller experiments and iterate from there, right? So that's

• 16:52Speaker 0

  I

• 16:52Speaker 2

  see.

• 16:52Speaker 0

  And it's really those things that are kind of like the 10x power-ups. Yes. Like what would be if like like you look at all of the sort of blockers and then like you realize there's this one little like load-bearing piece that if we could just solve that, all these other things on top of it would like collapse down. And now those things that we don't have the scale to reach the mountain because we don't have 20,000 of the most expensive computers. But it turns out that like if we just knock this one little brick out at the bottom of the Django set, this whole thing like collapses down and sort of looking for those like kind of important pieces where if we just pulled this one piece out and solved it, it would enable a 10x or 100x multiplier from for for people in the open source space.

• 17:30Speaker 2

  And when in the case of Hermes, what was the bottleneck that you feel like that that effort was trying to solve?

• 17:35Speaker 0

  Uh that particularly was with how the data is collected. Uh so the early uh models, especially Jeep uh like CHEP GPT 3.5, were trained through tons and tons of like human data, human data collection. Human data collection is very slow, it's very expensive. And if you wanted to make like a model at the time, the idea was you had to go spend all this money to get like this human curated data, which which is very slow. So Hermes was very early to the uh the idea that you could have synthetic data, which is that you could actually make, you could make a better model by taking an AI model, having it generate words and text, and then training a new A model on that AI model on that output. This is now like fully accepted, everyone's doing it. Uh but two and a half, three years ago, there was this giant open question of like, is that actually not going to just be this reductionist thing that collapses?

• 18:23Speaker 0

  Like, how could the how could the student ever eclipse the master kind of question? Um and we said, but but we don't have access to the resources that would enable us to go get all the human data. So what if we just try to figure out how to like you it's kind of like I always like to say, uh, I like to tell our team, we're like the astronauts in you've seen Apollo 13, where they like he like they dump all the stuff out on the table, right? And they're like, we gotta make this fit into this using only this. And that's kind of how we look at the world. Suppose all we have is on the table, how can we make all these pieces fit together to like you know get us to the moon and back?

• 19:00Speaker 2

  When was it that you guys start to shift your focus to this? Why, and then why do you how you know what is the core idea behind distro?

• 19:06Speaker 1

  I I think we haven't shifted per se, it's more like we have so many people working on different things, and then we prioritize those that we find that are the most promising, right? So everyone's looking kind of like at everything, and then we have this data pipeline that showed a lot of promise, and we look into it, and now it's like this show. So the data collection, data parts are also being still worked on, right? It's still being improved, and Hermes 4 is probably going to be really great. And now we just have another team, like my team, which works on distro. It's in power, though, right? So then other ideas could pop up, and we would have the another team working on it like in powerless.

• 19:48Speaker 0

  Well, yeah, but the the start of it really was this this idea of what if we don't get llama for. Exactly. What it like, like that was a challenge we were appreciating, like, what if we don't get llama for? Um, and uh and and we don't get Mistral log to you know like what if we we don't get that and what are the things that really would what would we what would we and because that for us if we're just fine-tuning models that's like an existential threat, right? That that's like an actual existential threat because the clothes providers will continue to get better and we would be like dead in the water in a lot of sense. So we sort of asked, is there any real reason we can't make llama for ourselves? Ah, I see. And there is a real reason, which is that we don't have 20,000 H100s. We don't have, I don't know I think Elon's got a hundred thousand H1 uh H100s now.

• 20:33Speaker 0

  Um so we don't have that right now. Uh you know, God willing in the creek don't rise, maybe we will one day, but uh but we don't have that right now. So we said, but what do we have? We have a giant activating community who's passionate about wanting to do this and would be willing to contribute their GPUs, their power to it, if only they could.

• 20:51Speaker 2

  If

• 20:51Speaker 0

  only they could. So we have the community who who's willing, but we don't have the ability to activate that willingness into actual action. So what are let's just look at it, what why can't it work? And it turns out really people have been trying to do this for a long time, but there were these very specific technical problems that just made it intractable. Um there was uh a group called Big Science who who made HiveMind who who worked on trying to do this, but they just had technical limitations because they couldn't uh ship all the information over the internet. Right. The only way people are connected is over the internet, right? And so anything that isn't sharing over the internet is is not gonna work. And so that was the sort of initial premise that came out of it is what if we don't get llama for and then what do we have that we could use to create llama for?

• 21:37Speaker 0

  And if we can't, what are the technical problems that if only we slayed that one technical problem? Now sort of the dam of our community can now flow and actually solve the problem.

• 21:47Speaker 2

  So to summarize, you you know, distro is research showing that it's possible to train um highly capable models with nothing more than a standard internet

• 22:00Speaker 1

  connection. Yes, right, as opposed to

• 22:03Speaker 2

  the kind of status quo, which is high speed interconnects without any uh sort of dramatic reduction in performance of the model. Is that roughly what you say a fair summary?

• 22:15Speaker 1

  And we would say actually uh the performance is equivalent. Right. So we've tried to make sure that that was the case before claiming that. So we made a lot of uh experiments just to make sure that it is actually uh better. Not better, but for the same bandwidth is better because now you need 1,000 times less bandwidth. Right.

• 22:36Speaker 0

  No one's interested in something with like a trade-off where you can do it, but it's not as good. It's gotta be like A1, it needs to be as good as the centralized way of doing it, you know, no apostrophes, you know, no asterisks to it. And then you don't have to do it, you don't it doesn't have to be in a data set. Right. Yeah, yeah.

• 22:54Speaker 2

  So the benchmarks are pretty impressive. But before we get to that, um, could you just explain maybe for folks who are listening who may not be super familiar with AI infrastructure, you know, why does it matter that you guys have demonstrated a way for models to be trained with with you know everyday internet connection?

• 23:11Speaker 1

  Yeah, yeah. So Lama 405B was trained on a huge data center of like 40,000 H100s, right? Those are highly interconnected and use a lot of power because every GPU needs to be connected to every other GPU. So if you have 40,000 GPU, then that's like a quadratic scaling, right? Every GPU you add, you have to connect to even more and more. So that requires a lot of power, a lot of cooling, a lot of everything, right? And basically uh to train those models, you need those high high-speed interconnects between the GPUs. But what we found is that when we train uh bigger and bigger models, the data that you actually need to transmit between the GPUs become relatively less and less. So not l it not doesn't become smaller, but for the size of the model, as you grow the model, the the size of the communication grows slower.

• 24:03Speaker 1

  Doesn't grow as fast. So now you don't need those big, big interconnects. You probably can have maybe four data centers or right or like 10 data centers separated by the internet and train the same equivalent network using uh those data centers. And if you push that boundary to the extreme, then you could have your home computer, right? Your 4090 or like some 4080, I guess, any GPU consumer or even your Apple, uh your phone, your laptop, right? Everyone could potentially uh be connected into this huge web of of uh the internet of things which train this single network, right? Right, right. So uh

• 24:42Speaker 2

  so just to give people a sense of how dramatic of a change that is. Could you what how how many folks, how many organizations do you think in the world today have the capability to train a co-located model, right, of the scale of a llama you you mentioned about 40,000 H100s? I mean,

• 25:00Speaker 0

  I would it would probably be in the number of ones on my hand and they probably wouldn't use all my fingers. Right. You know, yeah. I mean, you basically have open air, anthropic meta, uh Google, and then you have a few Mistral and then Deep Seek Um and a couple of other places that are maybe like a specific country are like backing it. But certainly we would not we could name them all and not run out of like appendages, you know, appendages to get there, yeah. Right.

• 25:26Speaker 2

  And so the the second and third order, I guess, um, you know, effects of of needing this high-speed interconnect between GPUs in a single location has basically restricted the number of people who can run those. Exactly, yes. I see. And so the insight was um you guys found a way to decouple the scaling of the um you could say performance of the models from the scaling of the interconnect required. Those two things are gonna affect you guys have essentially tried to decouple those two things. Yes, yes.

• 25:55Speaker 0

  I think

• 25:55Speaker 1

  you nailed the decouple, yeah.

• 25:57Speaker 0

  And at one point that almost becomes like a human coordination problem, right? Like if you're a single entity, you are aligned with your own coordination, right? Like you're just like, I we're gonna do what we want. Right. But even if you if you could, how would you bring all these people together to work together on one thing? And that's really like we think that has an unlock to still be like discovered and researched what that looks like as like a realization. Right. But the idea that the entire world could work together to create one AI that's representative of the whole world that everyone's contributing to. I mean, there's probably a lot of power in that. And and so we're interested in seeing sort of where that that could go given this.

• 26:36Speaker 2

  How did you guys um embark on the first step? What was the first sort of training from a research and systems architectural perspective? What was the first thing you guys decided to do to then try to uh to figure out if this was even possible?

• 26:48Speaker 1

  I think uh I I thought about it for like a couple of weeks because there's a lot of implications in like the mathematical side, actually. So we we met in New York, I think we had a long discussion on like is this even possible, right? And we came to the conclusion that this should be possible because there are some um training dynamics in neural networks that can be uh taken advantage of, which allowed you to do this type of thing. But that was all purely theoretical, right? So then we've just wrote the code and tried it. That was a really big leap of faith.

• 27:27Speaker 0

  To spend like months being like, let's just because it it took months before we got like our first data point of whether this was this was really gonna be true. And there actually were several false starts with that turned out to have been wrong. Um but the the core mathematical properties actually like it's funny, a lot of people have discovered that like everything we have in the AI was like invented in the 90s and no one noticed it. Or like was met in the 80s, it was just like this like a few like in math insights that truly were now that we know everything we know, those insights from uh maybe 15 or 20 years ago that people that seemed innocuous at the time turn out to like be extremely powerful like when applied to what we know. And we did have a lot of we did a lot of false starts with it too. And then there was also that just the fact that we had many false starts that didn't work, and we said, okay, like let's let's go to the next, we gotta keep going.

• 28:15Speaker 0

  Like, you know, like to have that sort of faith. And a lot of that comes from having the freedom from a business perspective, kind of like where we are in the space to be able to explore that, you know, and to be able to be comfortable with not having to deliver something right away and and to just just work it out till it's end.

• 28:31Speaker 1

  I think for distro, we hit not the wall, we hit like a bullet of glass. It's like we saw hints of it working on the other side, like we saw the other side, but there was like at first a lot of problems that we couldn't solve, and uh we worked on it for months and months, and one of them was actually scale. So one of the biggest breaks we had was just Jeff training on a bigger model, and that was the last step. That was just like the amazing like results we had. Like we at first we didn't believe it even. Yeah, like we thought this was a false, like we did some mistake, or like we used the wrong data, like it was too good to be true, right? And but it did work at the end. And we did more benchmarks, and that was I I think this uh the work paid off at the end, yeah.

• 29:15Speaker 2

  And so you know, in that phase when you're trying to go from theory to hypothesis testing, what is the first major hypothesis that that you guys tried to test or metric that you're trying to measure or you're looking for for indication that from a practical perspective this would work?

• 29:27Speaker 1

  So at the beginning we tried fine-tuning because fine-tuning is really fast. Right. So you can take a really small model and just fine-tune it on a small amount of data. And I I found out that with fine-tuning could actually decrease the bandwidth. Uh it was it was kind of like iffy because it's hard to judge uh the quality of a fine-tune. There's a lot of benchmarks, but then when you have like five uh tenth of a percent, like is that meaningful, right? You don't have mm, you don't have a lot of indicators in in that kind of um of uh scenario. But as we scaled up and we started doing pre-training, when you Had more and more compute, we had we got access to more compute, then we could definitely say that this is better than the previous one. Right, got it. And

• 30:08Speaker 2

  the results you guys ended up publishing were sort of staggering, right? I think you have there's a as there's an 857x. That is the worst case scenario. Reduction in bandwidth requirements, and that's a conservative estimate. It's

• 30:21Speaker 1

  a really conservative estimate. So

• 30:24Speaker 2

  almost about a thousand X reduction in the bandwidth requirements at the conservative end of the spectrum. What would what would be the optimistic side of the spectrum of bandwidth requirements?

• 30:35Speaker 1

  I think we're seeing hints of being potentially like 2,000, 3,000. So even a little bit more with this method and potentially using like quantization, you know, other things that people have like so many things people could try with this method that we haven't tried, right? That could unlock another like 10x. Right. So that could go. But we we can't really say this already. I I would try to be conservative, right? We want to be sure that it doesn't degrade the model in some other way because the benchmarks we have are great, but they only measure in ways that we measure them, right? Yeah,

• 31:09Speaker 2

  could you talk a little bit about what you guys benchmarked

• 31:11Speaker 1

  too? So we there's for example cross-enter and cross-entropy loss, which is the difference of the output of the model with the ground truth, which is the real data and uh what it predicted. There's also perplexity, which is the same measure, like but in a different way. There's like benchmark like uh Hela Swag, MMLU and those type of question answering benchmarks. But those are like fairly limited, right? They don't benchmark everything. Right. So it let's say let's say in the future LMs are being used in robotics, then these uh benchmarks won't really matter, right? Right. So it's really hard to guess whether the distro method would degrade any of the future potential benchmarks. We really want this to be fundamental and not like limiting the network to a subset of uh potential uh applications.

• 32:00Speaker 1

  Right. We want this to be optimized, like just a general optimizer that can do everything.

• 32:06Speaker 0

  But what's interesting is it tells us that uh what's actually happening in the learning, the like what we call like the instrumentability problem, or like just knowing what's actually happening when these models, what the models are being trained, it says that the actual training procedure that's happening, perhaps it it's not so much about the whole model learning, but that there's these like key insights that are being learned that are actually almost narrower in thought. And then that you don't that like the real things that matter, there's only a few high important signals that are within the learning that's actually going on.

• 32:38Speaker 2

  And

• 32:38Speaker 0

  that that itself just tells us something that we didn't know before. We know that like what we what needs to be communicated between these things, the two the different nodes are just these few key pieces of information. And that is necessary, uh that is a necessary condition, or rather a sufficient condition to get the equivalent behavior of sharing everything you know. And and that is uh an insight, you know, that is worthwhile in helping us continue to understand what's actually happening when these these models learn. Right. Because that it is as for as much research as we've done, it's still very much like a giant question marker like what's actually happening inside these models.

• 33:16Speaker 2

  Yeah, so so that that that's one of the um you

• 33:20Speaker 1

  know

• 33:21Speaker 2

  double-edged swords of of generative modeling, right? Is it's it's a fundamentally empirical space. The distro results that you guys probably were so dramatically shocking. That of course, I would I would say the primary reaction the community have was one of disbelief. Yes, yes, yes. And we expect that.

• 33:37Speaker 0

  Yes, yeah, and and we would have if someone had said the same thing to me and I hadn't done it, I would have been skeptical as well. You know, so let's talk through that.

• 33:45Speaker 2

  Yeah, let's let's what could you guys present the bull and the bear case, right? So now pretend to be let's red team the results. Yes. Yeah. Yeah. What would you say the biggest critics, if you had to chat roleplay being a critic and a disbeliever in what Noosa just pulled off? What would be your biggest objections today?

• 34:01Speaker 1

  So one of the first could be your baseline was wrong, right? So that is really easy to claim. Uh training large language models is not easy. It's not really like just you just do one thing and you get the best model, right? So you have all of these hyperparamers you have to tweak. Right. And if we didn't do the reference right, that might be just by chance that distro is better, right? Right. Right. It could be distro is actually worse, and the baseline is just worse than usual. Right. And then distro is better than the worst thing. So it's inconclusive, right? Yeah. Yeah. I

• 34:36Speaker 0

  think another piece would be someone would say it doesn't scale. You know, is that is that okay, it works at smaller sizes. Do you, if you scaled up to a trillion, do you get the same parameter? If you get the same thing, and you're like, well, there's a lot, and this is actually, I will say it is a valid criticism. There are lots of false starts that we've had just in space where people come up with an idea. It works kind of at a small thing, but when you try to like make it big, it's more a byproduct of some weird dynamics at a small scale that isn't uh true for all, you know, all instances and all sizes and all network types, even. So like AMW, like it just works for all networks. And transformers, they just work all the time. And and so what you don't want is sort of like a fragile situation, like Bowen said, where it kind of works for like LLMs of these types with this, but it doesn't work. So that would be perhaps like another consider, like a bear like criticism that people would have.

• 35:25Speaker 2

  And just to ground this in numbers, what was the scale at which you know you guys we've

• 35:28Speaker 0

  gone up through 7B now, like 7B models. So we still have lots of work to do, and it could be that it doesn't scale infinitely. Um but what we have seen empirically is that as we make it bigger, the differential between distro and NMW actually gets wider. And so this is very like encouraging, which is that if you if you saw it like starting to narrow as the scale went up, you would fear that it would eventually equate and then like maybe be worse at the end. Um but that isn't the case empirically with what we see so far. So we're quite hopeful that it will continue to scale up, and that's why just from our uh practical perspective, we want to find a now actually bring it to the community and say, okay, well, let's run the giant one together, because we don't have the 10,8200s to just do do it all at once and like prove and know for sure.

• 36:14Speaker 0

  Yeah.

• 36:14Speaker 1

  I I really think the important part of our research right now is to prove that this row can come like reduce the amount of communication and not really about the loss differential because the loss differential of distro right now is like it's better and it's unexplained. This is probably because of some side effect of this row, right? That we haven't explored. It's probably not because of the uh compression that we're doing. So uh, or maybe it is, we don't know. But then the focus should really not be on the loss differential, but on the compression part. Because as we scale up uh the models more and more, you can compress more. But the loss differential might be like changing, right? It could be wider, it could be narrower. It doesn't matter that much at the end as long as the models are trained as equivalently as possible, right?

• 37:02Speaker 1

  To AMW, the state of the art uh optimizer right now.

• 37:05Speaker 2

  And you know, the critique about um the baseline maybe not being replicable. Um it's been now about a month and a half, I think, since you put out distro. Has the community been able to replicate the baseline?

• 37:15Speaker 0

  So what we actually did was we threw away everything we did and we went to an entire new pre-training. We we started from scratch again on a second time. So we had been using our own uh one based on we we had implemented it twice, once our own, once inside of Hugging Face's nanotron framework. And uh we threw everything away, and I said, okay, let's do it exactly all over again using um uh uh it's called from Olmo from Allen AI, right? Which is the very highly reproducible, like they they publish every data index of every tokens that you could like a hundred percent reproduce exactly what they did down to like everything.

• 37:50Speaker 2

  And

• 37:50Speaker 0

  we've re-implemented now a third time in their frame in their framework and were able to reproduce their training run exactly and then did it again with distro, got the exact same results we got with Nanotron and stuff. So we've you know it's kind of one of these things where like you have to want it to fail. You have to be you have to be willing to do the scary thing and like see if it fails, but we I now have done that for the third time and uh in the distro paper that we're gonna be publishing next month. Uh that's the data, all the data is coming now from the Olmo framework for

• 38:18Speaker 1

  the R. And we will have the code, and it's also like already the baseline is not trained from us. So we took a baseline from this OMO group. So that argument can be like uh settled, right? Right. Because it's not that we didn't. We didn't come up with it. It's like they use ADAM, they assume we assume they are training using the best hyperameters, which they did because they did a huge ablation on like smaller models and train a 1B. And then we just took the same code and replaced only the optimizer to distro. Right. And then we saw the exact same curve. And then I was actually convinced because I'm the biggest skeptic of distro, actually. Like from the beginning, I I just didn't like believe that this was possible. But as we um seen more and more hints, uh as I said, it was like a bulletproof glass, right?

• 39:04Speaker 1

  We're hitting that glass. And then this was the real breakthrough, then we're like, yeah, this row is real, right? We should just announce it, right? Right, right. And yeah, and we should just finish the paper and hope for the best.

• 39:17Speaker 2

  Yeah, yeah, it's just moving so fast. Um, there's so much uncertainty about who wins and who loses, etc. That the natural tendency, I I've seen a natural tendency for people to go um to start sharing less openly with the community when they have a breakthrough or results. Um things go closed source, people stop sharing. Um but ironically, I think for for the for the kind of effort that distro is having being as public as you guys have been with the open source uh release, the paper publishing the uh the first few application results, um, and then which then enable somebody like the Almo team, the Allen team to then adversarily red team you guys and try to prove you wrong, um, makes the the you know, it makes makes distros um accomplishments that. That much more impressive. Which then I think you know is kind of the dream, right? Is that it's kind of what happened with synth data for so long, there was so much disbelief.

• 40:07Speaker 2

  But the minute people started reproducing the Hermes results, suddenly the entire space switched over into the synth data approach you guys were using. And so my my hope is actually that some of the people listening to this podcast will go try to replicate and prove you guys wrong the other side.

• 40:23Speaker 1

  We we encourage it, right? And we want people to try to use distro because it is so hard for us to think about everything, right? We we're such a small group, we only have so much so many ideas. And with this distro thing, I really hope that people just start seeing that we've really pulled out the Jenga tower, right? And now we have to start over and think about this new way of training things, which is much cheaper and much more efficient, right? Right. Because you don't want to have uh huge, like power hungry data centers in like rich countries, and then like everyone else doesn't have anything. Right, right. And then this thing would allow everyone to participate in training. Right. It's a different mindset, really. Yeah.

• 41:02Speaker 2

  If distro works, right? Um knee jerk response would be, oh wow, that's terrible for NVIDIA. Right? Because one of the largest drivers of value in Nvidia's um, you know, enterprise value and market cap and all and revenue has been these massive contracts for co-located data centers, right? People buying 10,000, 20,000 soon for you mentioned beyond buying 100,000 H100s, you know, to co-locate in a single place. Um is that true, or do you think people there's there's more to the story that actually the second and third order effects of something like distro working are actually net positive for somebody like NVIDIA in the ecosystem.

• 41:38Speaker 1

  It's not immediately that like bad for Nvidia because the Jenga tower is still there. We've just tried we're trying to pull at it, right? We haven't like just like pull it out completely. And these there's still like years of work to make distro actually uh scale towards like those type of really really large training runs. I think uh there's a lot of things that Nvidia still has, which is the CUDA stack, all the GPU hardware, right? Those like non things that are adjacent to interconnect that can also work with distro, right?

• 42:11Speaker 2

  Right.

• 42:11Speaker 1

  So it's it's it's hard to tell whether this would uh like really affect NVIDIA or not.

• 42:18Speaker 0

  I think the the bigger effects will be maybe at like the social scale and the business scale of like what it means to not have a single entity have to do it, even if everyone's still using NVIDIA's chips to do it, it's that sort of that that second middle layer that may see like a more pronounced effect because ultimately uh we want still want the the back, you still have to do back prop, you still have to like load memory and and and that uh it may be that uh the I was think what might become what might happen sooner would be uh a redesign of the types of chips that NVIDIA or someone would make. Okay, under this model, we can dedicate more VRAM versus there's like this question of how much VRAM versus how much processing power is on a die. Right. And that that could like change sort of the the dynamics of what that optimal looks like. And I think that is probably more like a it's just it's a new meta for them to then have a product space for for as well.

• 43:07Speaker 2

  As I hear you talk about the implications of if if your distro works, there's so many um analogies that could be drawn to these earlier distributed systems at home efforts. If you remember if you have guys ever did folding at home or stuff, I was a setting, yeah. So was that ever part of the inspiration for what you guys are doing?

• 43:23Speaker 0

  Yeah, it certainly was, but we actually what's interesting is that we didn't know if that would activate with a lot of people. Like it was something we were excited about, but sort of internally, there was like does any we would like kind of like we've been our own critics on this. Like, does anyone really like care about like if I have to care about training a model at their home and something like this? But and it may not have like the biggest impact right now if you were to go talk to your grandma about like what we're doing. But there's certainly a large community of people who who feel that and want to be a part of of working and contributing to AI and that team effort of like the whole world. And it's and it's certainly something we were excited about internally, but we didn't know if it would like actually like hit or stick. Right. And the second we came out with it, it's like everyone being like, oh, immediately SETI at home. Like I want to do that, like I want to be a part of that.

• 44:08Speaker 0

  And there's something aspirational about it about reaching for the stars, being part of this, you as ear one person being part of this giant big thing that we hoped would be there, but we didn't know until we sort of put it out there.

• 44:19Speaker 2

  Right.

• 44:19Speaker 0

  The current distro uh experiment still

• 44:21Speaker 2

  uses H100s. Yes, right, which is which is a pretty expensive card. You know, I think market prices are coming down a little bit, but

• 44:28Speaker 0

  still in that 30 to

• 44:28Speaker 2

  40,000

• 44:29Speaker 0

  dollars per card

• 44:30Speaker 2

  range. Um and so it's expensive and hard to get. You know, can we let's say we can we can mitigate all you know the need for all the the specialized and highly tuned bandwidth infrastructure, the interconnect in interconnects in particular. Is it is access to sort of high-end GPUs, you think still going to be a requirement for training in a distributed fashion? Like

• 44:53Speaker 0

  H100s? Well, we've been right now, I don't I think people don't actually realize that like a 4090 and like an H100 are are in a lot of ways the same card. So for the non-gamers like the 4900. The chip that's inside of them is almost identical. The chip, the actual compute chip is actually almost identical to an H100.

• 45:11Speaker 2

  Right.

• 45:12Speaker 0

  Um and what you're actually buying is the memory around. That's actually the expensive piece, the HBM3 memory that they put around the die. Um and so uh they're uh maybe like an enterprise markup that you're getting in there, you know, that uh that NVIDIA is charging for like the H100s knowing that you need to use them for training versus someone who wants to you know just use this for gaming. Um and hopefully that dynamic doesn't shift now that we introduce distro. But um we because you're able to distribute it so wide, I think the gaming GPU angle is really gonna be like the sweet spot. If as long as there's continued to be sort of like higher-end gaming GPUs, and those are on comparison with the high the high-end uh training GPUs, even if they're half as fast or a third as fast, we can marshal a lot of those together and it's sort of make up for it in scale.

• 46:00Speaker 0

  So it's it's actually like okay if we are just using the gamer GPUs, I think, for for the for the short short term. Right. Um so we we certainly do not want the sort of minimum viable entity to be someone with an H100. That that cuts nearly everyone off. And also like what Apple's doing with our MLX platform and their and their Apple Silicon, like that's amazing. And if we could activate just all those that latent compute uh towards training, that is a lot to it too. And so we're making sure that also like the code we're writing to help to actually do this training is agnostic to the hardware, is able to communicate and operate, you can have an Apple device and an NVIDIA device training together. Right. And this is actually the the for just practical reasons when training happens right now, it sort of assumes that all the GPUs are the same, they're all in the same sort of organization.

• 46:50Speaker 0

  So we're actually just writing this like fault tolerant training code where like you know, one of the GPUs can go down and that's okay, and it'll continue on training, even if some of them like fall off and are different. And you just sort of didn't do that before if you were just one entity who owned everything and controlled everything. Yeah, there

• 47:09Speaker 1

  was no use for it,

• 47:10Speaker 0

  right?

• 47:10Speaker 1

  It's like my my point of uh of this is that um as you have this technology being unveiled, distro, that allows people to think about possibilities of new methods of training, right? Right, which is uh different from the status school. And those chips probably right now, like the H100s and 4090s are not like H100s are suitable for training and 490s are not, but as people become aware of better architectures, right, better training code and different ways of training LLMs or even different architectures of LLMs, they can start fitting those models into smaller VRAM or like gamer GPUs, right? Or maybe there's also um pressure and demand. So if Nvidia sees a bigger demand in like um less interconnect GPUs such as gaming GPUs, they could potentially sell those more, right?

• 47:58Speaker 1

  So produce more of them. Right. So it really depends on it's it's really a balancing act, I think. And as we uh push this row more and more, people would start to develop an ecosystem of training around it. Right. Otherwise, right now we have to use H100s, right? But if you don't try something new, you would always stay on H100s. Right, right. You

• 48:19Speaker 2

  know, this was your first release. Right. Um can you just explain uh what comes next? What are your biggest priorities? What do you guys think you need to do next to actually get you know get built continue building more momentum?

• 48:30Speaker 1

  So we will release the paper and then the source code of distro. So this would allow people to just start iterating on it like immediately, right? That's like that's for October. So the ICLR conference, we would want to also try to publish. Um after that, maybe Jeff, you can talk about it. Yeah, and

• 48:46Speaker 0

  then the question after that is actually building something that can like use this together. So we're we're starting to work on code to be like what will it actually look like to practically use distro to so that everyone can come together and train models. So that's still very much in like the researchy phase like that. But um if we if we sort of rewind back to what I said before, which was that we looked at what we had, which is our community of people who are interested in this, a very large community, that was what we had. What we didn't have was the huge stack of H100s. So now that we've now actually kind of gone down the path, solved it, we now want to bring it back and say, okay, here's an actual full stack tooling that can actually you can use to actually do the thing. Because what we're gonna release, like the first release is very uh is gonna actually be very academic.

• 49:32Speaker 0

  It's gonna be the the the paper with the proofs with the abilations and sort of the reference PyTorch source code that's like here's how it actually works here at all this stuff. But there's still a long way to go from an academic paper and reference PyTorch code to everyone in the world training an AI model together, right? And so that is the the second that the next phase is is building that that second piece of it. I see. And

• 49:54Speaker 2

  and um can you talk a little bit about what are parts of that um you know. Product productizing the research, so to speak, and turning it into an actual optimization library that can be dropped in and used really easily in a in a training run or in a training um job. Is that is that the shape of the product? I'm

• 50:11Speaker 0

  I wouldn't think that like it's going to be like a product that you buy off the shelf and like drop into an existing thing. I think it's really more about what does it mean to have everyone working together on a model? How do you how do you reward them? How do you give them actual access to it? And then what is community ownership of that even mean? So we're still very much this is still very much in the like ideation phase of the productizing of it. And you know, we're we're we're lucky to be in a position where we uh feel comfortable releasing the sort of the secret sauce before we maybe have like all of the uh the where that value capture is sort of accumulated to us because ultimately uh it's science, it's math, it's math equations. And and trying it's like the grandmouth target, like the more star system shadow holding your hand, the more we'll slip through your fingers.

• 50:57Speaker 0

  So we have a good idea about how we can um you know have a tool chain that can execute and help everyone train together and and work on it, and and we're gonna release that after we release the source code to everyone, and and then it will just be a process of seeing what works and what doesn't with the community because again, another one stupid line, but like everyone's got a plan, you know, until like it's out there and is real, and we see what do people actually want to want to build with this. So

• 51:24Speaker 1

  we we've also like been really open source with everything because we the open source community gives back a lot, right? So everything is built upon open source, so we are giving back to the open source community. And this will benefit us in the long term because you know LMs will be trained better, right? AI would advance quicker. So one

• 51:42Speaker 0

  thing I think will this will allow in the short term is like more experimentation for experimental architectures like this. Yes. If we have a substrate upon which it took us having to go get our own H100s to actually do this, you know, we actually went out and bought like 64 of our own H100s just to do like the distro testing. But the next group of people who have an idea about uh a new architecture or new or a new novel idea, now they can use this distro network to actually try that out and have and have the ability for experiments because there's if you're inside of a large organization, you have like sometimes maybe a fear of trying something new, because like what is there only we have to get something out next quarter? But but now sort of creating an environment where all of these, like I said before, there's so much area for innovation, you shake a stick and you'll make an innovation in the AI space, having like a place where other people can actually get access to the compute to do these train to do these studies.

• 52:36Speaker 0

  And if a bunch of people are like, hey, I want to try this weird um transfusion, which is a paper from Meta, which we're really excited about. Um model with that's using bitnet that's got like all these weird things, then people are like, yeah, I'll put my G like clear, click this button, here's my GPU, like go ahead, use it, and now you've solved the coordination problem, and now we can try out all these new things and see what the results are.

• 53:00Speaker 2

  It sounds like you know, to uh again, if you if you look at the

• 53:04Speaker 0

  you know,

• 53:05Speaker 2

  we call it we called it sort of the first middle and last mile of frontier model production, right? Um the first mile being pre-training, um, the middle step uh being all of the stuff that happens you know uh in the post-training and alignment phase, and the last mile being inference optimization, like actually hosting and so on. It sounds like you know, this paper was a breakthrough in the first mile. You should you demonstrated that you could um you could dramatically reduce the bandwidth required in the pre-training step. One of the things we talked about when when you guys put up distro was kind of the impact this is gonna have on if it works, right? Is on is on regulation. The the grand challenge for distro was inspired by the question of hey, what happens if we never get an open source llama four? Yeah, right. In a not so distant future where Meta decides, hey, it's just the regulatory risk is too high.

• 53:54Speaker 2

  The argument would be, well, that's okay. Thank you, Meta for everything you've done so far. Now it's time for the community to step up, uh, run a system like distro to allow just like massively decentralized training, right? How far are we do you think from a if if let's say um based on the results you guys are seeing, Lama 4 is not open sourced. How far are we from a model that's at least as good as Lama 3, 4 or 5B being trained by the community?

• 54:18Speaker 1

  Oh, I I think 7B would be just possible immediately. Like a 7B model, I think it's not too far fetched with the first iteration of the code we put out. Uh it could be possible to train 7B to like four trillion tokens with maybe 1000 H100s, like just rented all over RunPod, right? Just somebody could do that. Uh for four or five B, what do you think?

• 54:41Speaker 0

  I think I think it's still is it it would still be like a next year sort of environment that we would have to do. There are some uh scaling problems or not scaling problems, but technical things about how you shard the model. Because at that point, you get to this um problem where you have to put the model on more than one GPU or more, and and like when you get to sort of that threshold, now you there is a communication requirement between those ones to even shard it. Well how we're looking at what act how the activations were, like how you could shard those pieces to make it work. So there's still technical work to be done to scale it up infinitely to those like that those super large model sizes, but it's not intractable of anything. It's it's an engineer, it's truly just like an engineering question that we need to we need to tackle. And by bringing it out to everyone, I'm highly confident that there are a lot of like, there's also a ton of amazing smart people who work at all these closed labs, right?

• 55:29Speaker 0

  And who can contribute how they want to contribute, you know, as not as anonymous, but at night, like here's my idea, I love what you guys are doing too. Here's a PR that implements tensor parallelism in this, you know, like there's a lot that we hope to activate for that side. But you know, to get it, uh it seems that we're in the open source space, we're always like a year playing catch-up, like a yeah, it's like a year and a half, a year to year and a half behind like the closed providers. That would be if I had to like ballpark it would be you know later at the end of next year, would be like when be at maybe that scale.

• 56:00Speaker 2

  Yeah.

• 56:00Speaker 0

  I spend far

• 56:01Speaker 2

  too much time on local llama, but let's pretend you know you're talking to the local llama crowd who who is not there 24-7 like I am, but like is um is broadly interested as a developer or in in making sure that open source keeps marching forward. Um if you could talk to them. I mean, what are the problems that they could help with to accelerate the progress you're talking about?

• 56:22Speaker 0

  Um I think right now there's going to be a lot of these engineering open questions when we release it. And we're going to need help with, for example, uh NVIDIA has this library called nickel, which is used internally to actually orchestrate when you look at even a uh centralized data center run. They do actually they create these like rings and trees to try to like efficiently move the data around the data center. And they also do uh use things like GPU direct to copy the memory the information in the gradients out of the GPU and like stream them to the next GPU. And we're gonna have to kind of throw all that away. We can't use that because it was designed for a centralized infrastructure. Right. So we're going to like it maybe that distro is like two or three times slower than like doing it traditionally, but that's not a requirement. It's just because we kind of had to throw out a lot of the crutch, not crutch, like all the things that NVIDIA has built up in their tech stack, and we're gonna need to replicate that on that site.

• 57:16Speaker 0

  And that's really just engineering work, but engineering work is still work, and it'll need to be done, yeah. So

• 57:22Speaker 2

  why don't we just take uh you know a few minutes to talk about how it actually works and what were the weirdest insights you guys discovered from an engineering and technical perspective during the project?

• 57:30Speaker 1

  So basically uh if you think about uh the usual training of AW is that you have all of these GPUs that have a copy of the model, like simply just let's say the simplest scenario. So they all have a copy of the model, and when you train them using different data, you want to synchronize them at the end, right? So you give uh one different book to every GPU, let's say, and then you train one step. So now all the weights are different. Like everyone has some different training because they've seen a different book. And now you want to synchronize all of these uh GPUs into a single model. So you have to do this copy of like the whole model into the other one. That's why it's so slow and needs those high-speed interconnects. With this role we found is that we can actually let those uh each of the GPUs train by themselves.

• 58:17Speaker 1

  So they don't need to be uh synchronized, so to speak. Like you don't need to copy the model like over and over so that everyone stays in the same state. You can actually just let each model train with their own books, right? Just uh let it train. And then every step instead of synchronizing, you just transmit what you've learned, what was the most important thing to you learn to everyone else. So it's kind of like you have this space of a lot of points in the in the cloud where everyone trains and everyone goes in different ways. And then you have this uh thing that tries to pull them together, right? Tries to g get them to a single point. But it will never reach there because your bandwidth is really limited. It's like one megabyte. The model is like two gigabytes. You're you're transmitting one megabyte every time.

• 59:04Speaker 1

  But with this one megabyte, you can try to make them as close as possible, right? In this in this cloud. And after you train them more and more and more, um, you start to see that the every model actually trains some in somewhat sense like the same amount. Like everyone trains as if they were like in the same point. And you could just take any of those models, they would have uh the similar performance. So it's kind of like the whole cloud is moving together like in in space. Right. So that's like the intuition of how distro works. But that wasn't like uh how we originally thought distro worked. This is like after we've done extensive testing, we done like we looked at actually we looked at the widths, we looked at how the distance between every every GPU, like the actual distance, and we see that at some point in training, they become like bounded, they stop diverging.

• 59:54Speaker 1

  So that's like that's why as you train more and more, we think that the compression can get higher and higher. Higher because they start to diverge this. So

• 1:00:04Speaker 0

  this concept of there being like the one model actually comes sort of from like an engineering detail where when we when people started writing code for neural network training, they just had like the one computer you're running it on. And you're just training the one model. There's just the one model, there's just the one weights. And then they said, well, we want to be able to train faster and we want to train bigger. But the way this was done was to abstract that detail away. Where the fact that you were on multiple GPUs was like hidden. And when you even like now, it's amazing. You write your PyTorch, like PyTorch code, right, to train. You write like this one line that's like do this. And like that this is actually happening on like 40,000 different computers all at one. But like you, as the like developer are acting like you're just training this tiny little, you're writing code as if you're training this one little model.

• 1:00:49Speaker 0

  So that abstraction that there's just the one model being trained is has been maintained, right? Right. And so now the implementation of this maintaining is that as you have each of these different GPUs train on different data, you have to do this all reduce operation where you're basically like you go learn, you go learn this learn, and then literally we're gonna like take everything everyone learned, we're gonna average it all together and and and start back to the same point. So it's like everyone goes off in their direction, they all come, they all come back home to mom, they all like merge back into one you know note, and then the next step we go out, we go out and do it again. So with distro, what we found is that rather than bringing everyone back home and averaging it back together, what you want to do is give each of those little nodes that are searching for the lowest point in the lost landscape the freedom to move around.

• 1:01:35Speaker 0

  And they aren't actually coming home and all synchronizing. They each have the freedom to move around. But what you don't want is the freedom to move around and just go off on a tangent. So we sort of have like he mentioned this rope. It's literally like this, they're all connected by like this bungee cord. And if one of them starts to fall down like a really good lost landscape, they'll start pulling the other ones. But that diversity of search space where we actually aren't training one, we're not we were breaking the paradigm of there being one model that's being trained. There's actually n number of models being trained, each of them getting to do their own X little exploration, but within a bounded space, so that they're kind of like all looking around, and instead of like all coming home, they all phone home. And they just kind of say, here's what I've learned, that's the best insights from what I've learned, versus being here's let's merge all together back into one.

• 1:02:22Speaker 0

  And that was sort of the the dynamics that were that are at play. But it was interesting because this was actually a post-fact realization. Like this was not the uh thesis that we went into it with. But when we looked at it and we said, okay, it's better, we're getting better loss, like we have to know why. And so we then went through the instrumentation and saw what actually was happening and seeing we found this bounded behavior, but where the diversity of exploration uh actually is the contributing factor to why it works.

• 1:02:48Speaker 1

  We're actually taking advantage of like having multiple GPUs instead of assuming that everyone should be doing the same thing, right? Where now we're just saying every GPU has the copy of the weight, so let's let's explore differently. Let's do other things that might be possible, right? That's that's how I think that's why this show is working so well, I think. It's one of the contributing factors. So

• 1:03:10Speaker 2

  I think it's very counter-intuitive. Yes, right. Could you just explain why it's so counterintuitive, you know, to folks who might not be as familiar with with training models. You know, why is this such a surprise that you can actually have bounded um bounded search?

• 1:03:23Speaker 0

  The fact that normally if you would do this, each one of them would go off in their own direction. Right. And that's not what you want, right? Because like as the models become different, they lose the ability to communicate what they've learned to each other. It's like they are each or going off into their own country and they're starting to develop their own language. And if they get too far apart, they can't phone home and talk to each other and tell each other. So the simplest, easiest thing is to just like don't let that happen at all. Bring everyone together and everyone's just one, but the diversity of a of the ability to kind of go off on and do a year in London. Like, you know, like each one of them are getting to go off and do their own thing, right? And then come back and say, here's what I've learned, and that somehow that is better than this one monolithic thing. But the one monolithic thing was actually just like a technical bottle. Like it was from the fact that like we had PyTorch and then they like, you know, or like Keras or any of the other ones, and they're like, well, if you want, you can train on multiple GPUs, and you don't even have to change your code.

• 1:04:12Speaker 0

  Right. So like it needed to maintain this like paradigm of there being the one model. So that's what I talked earlier about there being like all these things that we've just kind of always done, and that now we're at specific scales, we can like go back and rethink about like whether those are the right things.

• 1:04:26Speaker 2

  Uh as you guys were describing the postdoc realization that actually um having them all phone home is much much more efficient than having them all come home. Yeah, right. Um that makes me wonder uh from for for folks to complete the analogy. Could you just describe you know, in your next, as you guys are like making improvements to system, how are you approaching who phones home, you know, who's the orchestration node, the reward function, the objective function for who can who's the conductor of the orchestra.

• 1:04:53Speaker 1

  Yeah. So but like simply it's like when you use the all reduced operation, everyone just communicates to everyone else. But here is uh the operation is just smaller. So just everyone can uh communicates to everyone else, and you try to like come to an agreement, an agreement of like what is home, right? Right,

• 1:05:11Speaker 3

  right.

• 1:05:11Speaker 1

  So you don't actually have a home. So everyone's like uh off in uh any country, but as they phone each other, like with this tiny amount of data, they kind of understand what is home. Right. And everyone could like stay together in some in some sense. But

• 1:05:25Speaker 0

  it may be that this isn't even like the most optimal configuration. You know, um, where we're interested in uh like the and a we're thinking about like an asynchronous version where they're not even phoning home every step to each other, and like what do you mean if some phone home to others and others don't? Like, and this is where we're excited about because that is an experiment, like we can't do it all ourselves. So we're gonna like bring it out. So we actually were very conservative in this first one where we were like everyone talks to everyone, like, but that might not even be the optimal configuration, and it may be that the training will be we might and there will be like latency problems if you try to do this all over the world, right? So would it be possible to have like an interconnected like hub in every continent where they're communicating more frequently and then there's less communication to sort of the more disparate elements from like the North the America to Europe, you have like an American side and a Europe side, they're communicating more, they and then the nodes in between like communicate some summation of it.

• 1:06:20Speaker 0

  And these are all like different configurations that when you free yourself from the paradigm of thinking of it as one monolithic organization doing this one task, you then find a whole new sphere of optimization. Right.

• 1:06:30Speaker 2

  And and um, you know, between the two spectrums of command and control uh and and complete anarchy.

• 1:06:37Speaker 0

  Yeah,

• 1:06:37Speaker 2

  right. Where how are you guys uh approaching sort of the ideal system design you know for the next iteration such that there's sufficient coordination, uh let's call it uh coherence.

• 1:06:47Speaker 0

  Well, obviously it's bounded by the realities of the physical world. Like, and that is gonna think I think that will be the thing that drives it, which is that you have asymmetric network connections from people who are at home. So like home network connections, for example, are quite often um you know have a lot of down bandwidth but very little up bandwidth.

• 1:07:03Speaker 2

  Right.

• 1:07:04Speaker 0

  So like that is one realistic, if we want people to be doing this at home, that is a realist consideration that we need to be thinking about. So I think just in the it'll be driven by like the actual facts on the ground of the network topology that we inherit from like the world, then we'll have to optimize towards that network topology being what it is.

• 1:07:21Speaker 1

  Right. We also are mostly in the centralized size, like we're going towards the anarchy side. Right. It's like a lot of optimization algorithms start like with everything being like asynchronous, right? But that's like really hard to get because you sacrifice a lot, to sacrifice the uh speed, you sacrifice the convergence, right? Like actual efficiency of the algorithm. Right. So by starting on the other side, we just break through this like key um this key blocker, which is the communication, and then we can look at every other possibilities. Right. For like, for example, as you said, like having different bubbles training like independently, and then occasionally like trying to merge them back together. That that could be an interesting thought.

• 1:08:03Speaker 2

  Yeah, what one of the um you know I guess observations of how a bunch of decentralized protocols have developed over the last few years, right? Um, is that often you have a protocol design where the North Star, of course, is fully decentralized, right? We don't have any single node that's more um uh is not able to influence the outcome of the of consensus anymore than the others. In practical terms, what that ends up happening is you have a few validators, right, who are like large contributors of validation, and it's usually have one or two organizations or three, or call it five, only ordered five to ten organizations, right? That run like sort of clusters of node validators. And when they vote together, they can kind of steer the direction of the protocol. Is that roughly where you guys think we'll go? But if you if you kind of chart the number of data centers in the world by the number of co-located chips they have, there's there's only a handful that have call it 20K and above, but they're but the the tail is quite fast.

• 1:08:54Speaker 2

  Yeah, it's it's uh exactly simple. And so um before we get to the longest tale of that part of that tail, uh longest part of that tail, which is you know, time individuals.

• 1:09:03Speaker 0

  Yeah,

• 1:09:03Speaker 2

  there's this fat middle of people who run data centers who might which might have you know 2K cluster uh H100s. Do you expect like where we'll go next is that um having you know a model that's sharded across, call it a hundred two K H100 clusters.

• 1:09:19Speaker 0

  I think there is you know, immediately coming out, what you'll see is the ability for even centralized actors who might have multiple data centers to now like just use them in a more efficient way. Like just have n equals two, no, you know, like any and each one of them is like a whole data center is a node on the network. I mean, and that that's just an unlock, because especially like for a lot of these data centers, they have like maybe 100 gigabit interconnect between their own data centers, right? And it's like just the fact that you could now treat each of these two as one. I think just from like a practical standpoint, probably is like the biggest unlock that starts uh beginning. But that but that's the thing about the method is that it it is can be decentralized, right? But it doesn't necessarily have to be decentralized. Right. And centralized Actors can still break the benefit because they for that.

• 1:10:04Speaker 0

  Or they just don't have to buy InfiniBand anymore. Like they just can use a hundred gigabit Ethernet or 10 gigabit Ethernet when they're laying out their data center. So I think there might be a lot of like just simple kind of like things, the effects that happen quickly. And then sort of the dream of the full decentralized one may progress slower, but will ultimately eclipse. Right.

• 1:10:24Speaker 1

  I think also we should like separate different, like have multiple networks that can take advantage of like the H500s and then have a smaller one that take advantage of it. And later tell, right? So you have different rectangles. Right. You could fit them three rectangles instead of having one square like somewhere like in the middle. Right.

• 1:10:43Speaker 2

  Let's keep going on technical details that community's eager to hear about from the best. What are some of the other weirdest discoveries that that uh and decisions and trade-offs you guys made during the uh the first uh that

• 1:10:57Speaker 0

  you still need backpropies? Uh that back prop is still king. Uh yeah, that's that one's fun. Yeah, so we actually started this researching something called zeroth order optimization, which is where you don't do backprop. Which is where you actually train a model only through forward passes. Because that was like we were like, well, what if you because we wanted to target a 4090? That's like our quintessential, you know, but it's like I have a 4090. Like everyone makes it, it's ultimately a product you know for themselves. And like, so I have a 4090. What would it mean for me to like be able to do this? Okay, well, we can't do we can't train these huge models, but like what if we could just do it with with forward passes? So we actually, our very first iteration was the zeroth order optimization mechanism. Um that is a very interesting field, you know. But uh what we discovered is that uh backprop is still king.

• 1:11:43Speaker 0

  Like you really do still need to be doing backpropagation to to find the optimal point of the loss. And it's just like zeroth order is like what, like a thousand, like it worked, but it was like you needed like a thousand times or a hundred times the community. What was the even rememberable?

• 1:11:58Speaker 1

  Yeah, it was about about like one thousand. So that was for fine-tuning. And uh the problem with that is that right now on specialized uh unspecialized hardware, like general hardware, like Nvidia H100s, the inference time and the backprop time is almost the same. It's like is backprop is slower, but like it's like 2.8 times slower, not 1,000 times slower. So this inference only um zero order optimizer, if you had the specific hardware for it, like if you had some hardware that can do inference 10,000 times faster, right? Then that would be uh actually useful. Then you could train neural networks just by the forward pass. Right. And then you estimate the gradients, like that's a really rough estimate, and you do a lot of forward passes, and then you can train those networks. And what's interesting with with uh zeroth order optimization is that in some sense, like in the future you might not need uh general hardware.

• 1:12:48Speaker 1

  You could only you could only just need uh specialized hardware for inference and then do And then

• 1:12:54Speaker 0

  actually training. Actually train the network. So

• 1:12:56Speaker 1

  you you would have your cell phone that has inference and then just train on once in a while, right? It's just not always, yeah.

• 1:13:02Speaker 2

  Yeah, so that that is pretty uh wild because you're basically saying you can unleash the market of ASICs on training, right? And ASICs don't take 18 months to tape out, right? Um like like GPUs, they because they don't have to be as general purpose, so they're much faster to tap. So you can change the architecture on them much faster, they're much cheaper to do. Yes. Um is that uh sort of how how I guess um constrained to what to what what are the caveats on that statement? Is it basically that as long as the you know the transformer mechanism turns out to be the most efficient you know workload, uh then you know the future essentially with with if distro continues down its path is is that most training runs actually are are are running on on ASICs, not on GPUs.

• 1:13:46Speaker 1

  That could be possible. I wouldn't like assume it right that way, but it's possible in the future where you would have neural networks just training using inference forward passes. I mean we've

• 1:13:56Speaker 0

  seen chipmunk groc, other chipmaking, you know, people who are making inference-only hardware. Um and I also think that there's a lot of uh so like the core operations still are flowing point you know, matrix multiplications. There are things like BitNet which provide the hint of being able to do non-floating point multiplication that uh which is like the the amazing thing about this method called bit net is that because all of the weights are either one, zero, or one, the multiplication disappears and it uh becomes just addition because all the all the multiplier coefficient terms are just one. So and then it's just it's addition, and then it's a lot faster. And so you know there's possibilities where if you use that and then you had an ace, you could then make an ASIC who could do inference a thousand times, that could like actually get you like a multiply scale, and then that same chip could be used to do inference on the device and do the training just as fast.

• 1:14:43Speaker 0

  And so there's a possibility to world in that, but kind of from a practical standpoint, I mean like we're gonna come up the method that if just the whole world makes all the chips we tell them they make, it's gonna be perfect. So we sort of had to like you know take the take the piece. But what's nice about it is that in the future, if we're able to build up this network that actually has the participants, right? You can just shift that over to doing that in the future. And it's really that community activation key that's like the P piece, key piece, yeah.

• 1:15:10Speaker 1

  And also I would add that inference hardware training using inference hardware could unlock just more capabilities because it doesn't need to be um faster than backprop. It just needs to be fast enough that it's worthwhile to train using those. So for example, your smartphone, you're using it all day, right? Right. So while you're using it, while you do inference, you can actually just save a tiny bit of additional information and send it back. So they could use that to train. So if this inference thing is really, really fast, as more and more people use it, you could start to see some companies or like some entities try to uh take this additional data to train. So that that could be training would be like a byproduct of the byproduct of inference, exactly. Essentially,

• 1:15:50Speaker 2

  inference eats the world, right? So it's the future. Yeah,

• 1:15:53Speaker 1

  yeah, because usually people train once and then they infer, like they deployed the model, like everyone uses it. Like ChatGPT, it's not training all the time. Right. They could be training in one model like all the time, but the model you're using is not training. Right. But with inference training, it could be possible that you train the model at the same time you're using it. Well, I can't wait for the release.

• 1:16:12Speaker 0

  Any

• 1:16:12Speaker 2

  final things you want to say to folks who are listening?

• 1:16:14Speaker 0

  Uh just come along for the ride. We're trying to make AI that is representative of the whole world and that is personable to you. So that excites you. Like let's make like City at home but for AI. Let's make set a home for AI, yeah. Yeah. Awesome.

• 1:16:26Speaker 2

  All right. Thanks, guys. Thank you. Appreciate it. Good luck.