Data synthesis for SOTA LLMs

Nous Research has been pumping out some of the best open access LLMs using SOTA data synthesis techniques. Their Hermes family of models is incredibly popular! In this episode, Karan from Nous talks about the origins of Nous as a distributed collective of LLM researchers. We also get into fine-tuning strategies and why data synthesis works so well. Sponsors: Read Write Own https://readwriteown.com/?utm_source=changelog&utm_medium=practicalai&utm_campaign=changelog – Read, Write, Own: Building the Next Era of the Internet—a new book from entrepreneur and investor Chris Dixon—explores one possible solution to the internet’s authenticity problem: Blockchains. From AI that tracks its source material to generative programs that compensate—rather than cannibalize—creators. It’s a call to action for a more open, transparent, and democratic internet. One that opens the black box of AI, tracks the origins we see online, and much more. Order your copy of Read, Write, Own today at readwriteown.com https://readwriteown.com/?utm_source=changelog&utm_medium=practicalai&utm_campaign=changelogFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Featuring: Karan Malhotra – LinkedIn https://www.linkedin.com/in/karan-s-malhotraChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Nous on Hugging Face https://huggingface.co/NousResearchNous Research https://nousresearch.com/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. We just dropped Dance Party, our third full-length album on Change Log Beats. Buy it on Bandcamp and iTunes, or stream it on Spotify, Apple Music, and the rest. Link in the show notes. Thank you to our partners at fly.io. Launch your app close to your users. Find out how at fly.io.

• 0:43Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am the CEO and founder at Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How you doing, Chris?

• 0:58Chris Benson?

  Doing great today. It was nice seeing you a few days ago in person.

• 1:02Daniel Whitenack?

  In the flesh.

• 1:03Chris Benson?

  In the flesh.

• 1:04Daniel Whitenack?

  Yeah, that was great.

• 1:05Chris Benson?

  I think you posted a picture on LinkedIn, so if anybody doesn't know what we look like and has some crazy reason to want to know, there's a smiling mug of us on Daniel's profile.

• 1:16Daniel Whitenack?

  Yes, yes. And the reason we met is I was on a client visit onsite, and we were prototyping out some stuff like chat over your docs and natural language to SQL stuff, and all sorts of things with Prediction Guard. And one of the models that we were using was from News Research, and that works out great because we have Karin Moholtra here, who is from News Research, co-founder and researcher there. So, welcome. Glad to have you, Karin.

• 1:48Speaker 03

  Hey, y'all. Thanks for having me. I'm extremely excited to chat with you guys.

• 1:53Daniel Whitenack?

  Yeah, like I said, I'm a huge, well, this is our first time meeting, but I feel like we're already friends because I've had so much of my own benefit and interaction in working with models from Noose Research. A lot of amazing models that you've posted on Hugging Face and research that you're doing. I'm wondering if you could just give us a little bit of a background about Noose specifically and kind of how you came together as researchers and started, to me, from the sidelines. It seemed like, oh, all of a sudden, there's these amazing models on Hugging Face. And I don't know who these people are, these Noose Research people, but they're amazing. So give us a little bit of the backstory there.

• 2:37Speaker 03

  Absolutely, yeah. So just as a general overview, we are one part like open source research organization. We put these models out for free. We put a lot of research out for free, some data sets. So people can build on top of these open models. On the other hand, we're very recently a company as well, a C corp. So we've been working pretty hard after getting some seed funding on building together some exciting stuff. I won't go too into during the overview point, but we're continuing to do our open source research and development and release of models indefinitely. The way we started is very interesting and it would be pretty out of nowhere to the outside, for sure. It was extremely fast for us.

• 3:21Speaker 03

  We are a collective of people who have been playing around in the open source and language model space for a while, ranging from like GPT-2 release to Llama release to like the first Transformers paper. We've got people from various eras of Gen AI of when they came in. And for myself, it was GPT-2. I stumbled upon a Colab notebook and started fine tuning, made some Edgar Allan Poe and Lovecraft tunes.

• 3:48Daniel Whitenack?

  I've done the same. That's awesome.

• 3:50Speaker 03

  And we just got pulled into this world of look at these next token predictors that are just managing to smatter together the most wonderful and amazing stories. That slowly turned into a deeper and deeper dive of, well, how can I use this for learning information? How can I learn to use this for production and automation, right? It's evolved over time. For us, we started off just working with different open source collectives, actually. Once OpenAI kind of released GPT three and had closed sourced it. You know, we were used to open source GPT two. We were like, oh, man, what are we gonna do? Like, how are we gonna continue to play with the level of customization and interactivity that we had with GPT two? Then Eluther had released GPT J six B.

• 4:34Speaker 03

  The cobalt AI community, this community of people who tune models and inference models started to pop up. I think around twenty twenty, uh, twenty twenty one, uh, within in the face of this. So a lot of us started to have places to centralize and play with these models. We got to contribute and learn how to become better open source AI developers, et cetera. Eventually, there was a need for more concrete organizations to do this kind of focused work on the the creation of these models. We were stuck with like, okay, architectures for a while, like Pythia. But thanks to Meta, you know, we wouldn't be here without Meta. I'll say that first and foremost, like.

• 5:16Daniel Whitenack?

  The great Lama.

• 5:17Speaker 03

  Yeah, yeah, like prior to Lama, right? Like, everyone's like, oh, Facebook evil, like my data, et cetera. And here we are, like, they are kind of like the shepherds of this new era of the open source AI movement. So when Lama came out, there was a paper that came out called Alpaca by Stanford Lab. Right? And this was about distilling data from bigger models like GPT three, ChatGPT, GPT four. And being able to train smaller models on that distilled synthetic data, something they call the instruction data. So the Alpaca format really opened up the playing field for everybody to start making these instruct style models, these actual for prod use style models.

• 6:00Speaker 03

  So there was an idea I had in my head of, well, the Alpaca guys are using only GPT three point five outputs. What if I only generated GPT four outputs? It'll be a little expensive, but you'll probably get a better model out of it than Alpaca. At the same time that I was looking at this, there was a guy on Twitter named Technium, who had just started putting together his own synthetic dataset based off Alpaca, and the GPT-4 only as well. So I was working with a group at the time called Open Assistant, under Lion, they're a really big nonprofit. And while I was working on that, we had some GPUs that were cool with us using towards the development of new models. So I reached out to Technium and I said, hey, I have a little bit of compute.

• 6:44Speaker 03

  You have GPT-4 data in the same format? I have GPT-4 data in the same format? Let's train a model. So we trained a model called GPT-4-X Vicunia. This model was on the Vicuna fine tune. We fine tuned to fine tune, basically. The Vicuna model was a Alpaca style fine tune and we tried our dataset on top of it. It was good, it was okay. Then we thought, you know, we'll probably get a better result if we just train on the base Lama model. And the resulting model was the very first Hermes model.

• 7:17Daniel Whitenack?

  Gotcha, the OG.

• 7:18Speaker 03

  The OG, and that's kind of how it started to come together, was we both had a data thesis on use GPT-4 only and follow Alpaca, and we trained on Lama and we got Hermes. And we didn't know what benchmarks were. We didn't know anything about any of this stuff. We just made a model and it got a ton of attention. We put it out under this name, Noose Research. Noose comes from the Greek word for intellect. We thought it would be a good name for an AI company. But it was, it was just a place for, you know, fun projects and fine tunes and stuff. It was just a name we were using for our collaboration. And people started swarming and asking, you know, what's Noose Research? Like, what's this sudden, like mystical, like,

• 8:03Speaker 03

  uh, open source organization that like put out this like best model? And we're like, best model, like we just, you know, we just tried something. It was, it was really organic. And it got to the point that people started telling us, you know, you must have trained on the benchmarks, like these are doing too well. And we were like, what's benchmarks?

• 8:22Speaker 03

  We're not really like coming from an academic place as much as from like a enthusiast that became so committed that it became our life, right? It became our day to day.

• 8:32Daniel Whitenack?

  Yeah.

• 8:32Speaker 03

  So from there, people started to ask us, can I join news research? Now, there wasn't a news research to join. It's just two guys, right? What ended up happening was we formed a private Discord server and we thought there's a lot of people who range from somebody who's like 16, 17 years old, savant on Twitter, hasn't even been to college yet, insane at transformer stuff, to mid 30s, you know, working a really, really good Fang-esque job and just wants to really create and let loose. That was another class of volunteer. And then you have, you know, older gentleman who has already exited a company or something who has just been playing with code for a

• 9:15Speaker 03

  while and wants to jump in and hang out. So we ended up being this really eclectic group, you know, we don't know what your name is, we don't know what your race is, we don't know your gender or anything. It's just Discord profile picture, Twitter profile picture, right? So we came together, grew to about like 40 people, all working together on various different projects like Hermes tunes, data synthesis, the Cappy Bar series, context length extension, et cetera. And just from this kind of interaction between Twitter and Discord and bringing people in that we thought were cool, we ended up becoming what people will call an open source research org.

• 9:50Daniel Whitenack?

  Yeah, you sort of stumbled into creating this amazing research organization, which is ruling the world, which is awesome.

• 10:00Chris Benson?

  It's what OpenAI might have been.

• 10:02Daniel Whitenack?

  Oh, well, yeah.

• 10:04Speaker 03

  That's really sweet. Thank you, guys.

• 10:06Daniel Whitenack?

  Yeah, and I love it. It's so cool to hear that story and that background. And I see like in my own sort of little snapshots here and there, like connecting that in my mind over the past couple of years as I've, as I've seen you all post different models and that sort of thing. This is something I, you know, we've definitely touched on on the show before, but some of our listeners might not kind of fully. When you say this sort of like synthetic data sets that you were focused on in this alpaca format, could you kind of explain a little bit like we've talked a lot about fine tuning and you know, preference tuning and RLHF and different things, but what does it specifically

• 10:48Daniel Whitenack?

  mean that like you would take synthetic data? What does that mean in in your case? And like, why, why does that result in something good in in fine tuning and open model? People might think, oh, this is synthetic data. Why should I expect it to like be any good? So could you kind of help explain that subject a little bit?

• 11:08Speaker 03

  Yeah, absolutely. So I mean, out of context, synthetic is like as meaningless as like artificial, right? It could data is data. But in this case, it's referring to a particular class of data that's been generated by another language model or another AI, another diffusion model, etc. That can actually be used to further train models. Now, you might say, why would you want to do something like that? How is it helpful? What was important to us is we're all GPU poor, right? We're all running on laptops or maybe a 3090, maybe a 4090 like as individuals, we don't have data centers. So training or even tuning like a large model in the early days, like 70 billion parameters, something like that was just unfeasible for

• 11:50Speaker 03

  us. And knowing that GPT three is like something like 175 billion parameters and 3.5 and four can only go up from there. The question became, how can we make these small seven billion parameter models even compete with these massive, massive ones? These ones that I want to run offline, these ones that I might want to run on edge device, on a phone, on a drone, etc. Right? Like, how can I make them even useful? So there's two things to talk about here. One is synthetic data and the other is distillation, right? So synthetic data is just referring to like any kind of data that's created by a model in this case. And the reason that's

• 12:29Speaker 03

  useful is in particular, distillation. So if I told you to go study CompSci for 10 years, for example, and put in that massive time investment and really focus on general programming, and then I told you, you know, now it's time for you to learn about AI and transformers and stuff and put you through all the math prerequisites, etc. Like, you're going to come out with like a really strong foundation. Of how to do the work, but the problem is you've put in a massive time investment. Now, if I take that guy who spent ten years doing engineering, then another five years doing AI, and I ask him, hey, can you teach somebody like just really important like compressed tidbits that'll help them just get up and running to do the work?

• 13:13Speaker 03

  That's data distillation, right? That's knowledge distillation. So you look at these big models like Claude or a 70B model or GPT-4, and you can see like they're amazing. They're brilliant at everything. They have a bunch of high quality data they're trained on, and they have a bunch of low quality data they're trained on that they can interact with and express in a high quality form. So instead of me having to read a massive ten pager for why some chemical reaction or some like tax based process, whatever you want it to be, like instead of reading a massive document on that and then feeding that to a language model, we can just have that really smart model that already understands it really well,

• 13:54Speaker 03

  compress that information into an instruction or into a conversation until like two sentences, three sentences, five sentences, like half a page. And we can just train a much smaller model on that compressed information, and it will learn the compressed information, you know, to the degree that a language model learns something, you know, not perfectly. But because of that, what the Alpaca guys did was they generated a bunch of seed tasks from GPT three point five on various different domains and topics, and created these kind of compressed instructions with the instruction, an input question from the user, and then an answer. So the instruction could be like, given the following math equation,

• 14:39Speaker 03

  explain step by step why this is the answer. And then the input is the equation, which is your question. And then the output is the compressed answer. So all of that, we can take as one sample in the data set, and we can make hundreds of thousands or millions of samples like that of various different domains and various different tasks. So the Alpaca guys did this, less than a hundred K examples, I believe. And they trained the Llama models on these, and they found massive boosts to performance that this distilled information, like a human, successfully compresses and transfers over.

• 15:15Speaker 03

  So when I saw that, and then independently when Technium saw that, and then independently when many others saw that, we were like, this is so intuitive. This is exactly how I've learned anything by just going on Discord and Twitter and bothering people to give me the compressed bit of how I do something. We should try doing this with even higher quality models than 3.5. So we created, I can't remember the exact number at the moment, but at least 50,000, maybe 100,000 examples originally for Hermes one like this, just using GPT-4. And then we trained on that and ended up getting performance that was extremely, extremely like massive boost compared to the other models that were not

• 15:58Speaker 03

  trained using this kind of method. So without these giants that have already established themselves in the space, we wouldn't be here. Like without OpenAI, without Meta, like we literally wouldn't have the model and the data to do the kind of work that we did to make Hermes. What it allowed for us is like for local models to finally be like comprehensible and for us to finally have like offline capabilities to kind of take the good stuff from something like GPT-4 or something else and make it uncensored. So it still has all this understanding of all these topics, but it doesn't have all that RLHF inside it necessarily that safetyizes it. So that when people utilize the model

• 16:40Speaker 03

  has all this intelligence, but it has more freedom of thought to kind of converse with you on topics that OpenAI may reject.

• 16:46Chris Benson?

  Gotcha. One of the things I was curious about as you were going through that was a few episodes back, Daniel and I were kind of talking about the effect of model licensing, you know, on the community and the different kind of licensing concerns that were coming out from whether it be, you know, Meta, OpenAI, you name the organization. Is that ever a challenge for you since you're kind of using those to get started in terms of the inputs? Has that been a concern or do you anticipate it being a concern?

• 17:13Speaker 03

  I think that of course, like generally, like US international regulation on this stuff is evolving. The conversation is evolving very much. So naturally there's like You have to keep it top of mind. You have to think about these kind of things. But thankfully, because all of our model releases are like open source and we don't profit from them. Like if somebody goes off and creates a product using our model, you know, good for them. But we don't necessarily take on the liability or that worry of saying, hey, like, we're going to sell you this model that was created with GPT four outputs. We, we actually actively try to stay away from doing that. But because the data distillation paradigm is so effective, you know, if a model comes out that's better than GPT four

• 17:56Speaker 03

  and it's open source and I can use it locally. And in their TOS, it says, you know, you can use this to make a commercial model. Then we can apply the same techniques that we've been preparing and researching and understanding from these closed models and use it there. So right now, like, we don't stand to or try to or have any plans to profit from using any of these outputs. We're not about that because we want to be careful and respectful of these model creators. But that and these companies, but that being said, we're learning all these techniques and developing all these techniques that will be useful for when that time comes and for when that's available. Especially with the advent of something like Mistral. Uh, if we do distillation from a Mistral model, like Mistral medium or something like that.

• 18:40Speaker 03

  Uh, that's completely, from my understanding, you know, barring their TOS saying otherwise, but I believe it, it doesn't. Uh, it's completely okay in that situation for us to create models like this that can be used commercially, et cetera. Regarding the TOS stuff, though, like as much as we err on the side of caution, I'd find it hard to see a company enforce their TOS when these larger models are likely trained on

• 19:11Speaker 03

  not all copyright free stuff. Like I'd find it hard pressed to believe that these closed source companies, their models are, you know, totally copyright free and totally copyright clean. So if some other company that was feeling a little more rancious than ourselves was to say, you know, we are going to commercially release on this. I imagine it'd be difficult for them to be come after without the other group opening their books. And there's actually a pretty interesting interaction that happened regarding this between Google and, uh, OpenAI, if you guys are familiar. So.

• 19:47Daniel Whitenack?

  Yeah, I saw this interesting picture the other day. It was like the interesting web of AI. And it was like how Microsoft, Google, OpenAI, like it's like on one side, there's the ones. And it shows how they're connected to the other ones is like this visualization and like how many of them overlap in these strange ways between like, whether it's Together or Mistral or Meta, Google, Microsoft, OpenAI is sort of very interesting web of connections that probably makes some of these things rather difficult.

• 20:21Chris Benson?

  Leave it for the lawyers to sort out.

• 20:23Daniel Whitenack?

  Yeah.

• 20:24Speaker 03

  Yeah, that's the thing is like, we can look at an example, right? Like you hear that phrase like good artists copy, great artists steal, right? Like, so the data distillers, we're copying, right? Like, we're just distilling this information. Like, we're trying to like, make our models more like those. And we don't really plan to commercialize, we're just doing it for free for everyone. But the great artists are, you know, Google, you know, like, you look at Bard, and it tells you, you know, I was made by OpenAI. Now, it's fine for our open source model to say I was made by OpenAI, because we're very transparent about this is trained on GPT outputs. But when Bard violates the TOS with a paid product.

• 20:59Chris Benson?

  Bold.

• 21:00Speaker 03

  Yeah, that says like, I was trained by OpenAI, right? You'd think that OpenAI would come after this multi-billion dollar company like immediately, right? Instead, you see a tweet from, first, you see Google deny it. Then you see a tweet from Sam Altman, which was something along the lines of, I'm paraphrasing, or something along the lines of, I'm not mad that they trained on our outputs. I'm mad that they lied about it. And I'm sitting there like, okay, you're mad about this, but like, don't you, aren't you going to pursue the legal action in your Terms of Services? No, no, because everyone would have to open their books up too. That being said, I don't condone the commercial use of that kind of stuff. Like, they release, like, making a paid model from GPT-4 outputs,

• 21:44Speaker 03

  like, I wouldn't advise anyone sell a model made with them, just because, like, you know, we want to respect people's, like, TOS and stuff, they worked hard and spent billions to make this stuff, or hundreds of millions, however much they spend. But there is certainly room for hypocrisy in that realm of the large corpse. But that's my thoughts on the licensing stuff. And that's definitely my own individual thoughts. Like, we're a pretty decentralized collective at News. So you'll find people with all sorts of opinions all over the place. And as a company, we don't hold any view whatsoever on that.

• 22:21Daniel Whitenack?

  Yeah, I'm wondering, maybe this gets a little bit to the distributed nature of this. But I know that there's sort of various collections of what the News Research Group has done over time. You mentioned Hermes, but then there's these other kind of categories of things too, like the yarn models, capybara, puffin, obsidian, just looking over the hugging face now. I'm wondering if you could just give us, like, from your perspective, a little bit of a map of these different things? And like, how people might categorize the different collections of what News has done? I definitely want to talk about like, the future things and ongoing things as well.

• 23:01Daniel Whitenack?

  But as it stands now, what are the kind of major categories of what the collective has invested in their time and over time?

• 23:10Speaker 03

  Certainly, certainly. So within the stuff that's viewable on Hugging Face at least, we've got the Hermes series, of which, like I told you guys, the initial story of how it went down. But from there, Technium kept going. I haven't personally had any interaction with the Hermes model since the initial. From there, Tech just continued to create more and more synthetic data, collect from more and more sources, use more and more open data sets. And he's just got the, I guess, award-winning like, data thesis. The guy really knows how to go about curating and synthesizing good data. So Technium, it's his baby, the Hermes project. So everything you've seen since is really his work and anyone who's kind of collaborated with him.

• 23:52Speaker 03

  But almost like, you can't call anything a solo project because of the open data sets we use too. Like, everything is built on the shoulders of giants and the shoulders of each other as little people. But Tech really has helmed the Hermes initiative so far. I think that's our most popular model series. And he released the Open Hermes as well because we had some data in the original Hermes that we never released publicly. And we wanted to make that kind of an option for everybody, so. That's Hermes, still follows the same kind of philosophy of synthetic data and it now uses the ChatML format instead of the Alpaca format is what we kind of upgraded to. Then you've got a Capybara and Puffin, which are both done by a volunteer and, you know,

• 24:36Speaker 03

  OG member LDJ, who you may be familiar with Luigi Daniel Jr. So the Capybara series was using an amplify instruct method, this novel method that LDJ had worked on alongside another one of our researchers, Jay. So LDJ and Jay can get confusing, but the two of them worked on the Capybara series, created the dataset, trained the models. And then Puffin was the idea of using handpicked, smaller samples from some of our larger data sets to make sleek data sets for an easy tune and see how that works, kind of in the spirit of the Lima paper, where they just used a few examples to get really good results.

• 25:18Speaker 03

  Those are really the popular tunes using synthetic data for like general use. Yarn is this novel context length extension method at the time of creation by Emozilla, also known as Jeffrey Canell, and Bowen Peng, also known as Black97, alongside Enrico Shapole and Eluther AI. So what happened there was these guys were already looking into contests like the extension for a while and when we kind of came under the noose banner to do the work, it opened up a little bit of resources from compute sponsorships. It opened up a more centralized place for them to be able to do that collaboration. I had no hand in, in the yarn models whatsoever.

• 26:03Speaker 03

  And that's the exciting thing is everyone really gets to work in their own spheres and their own kind of autonomous circles. And then we just check in and see, you know, how's the research going? How's it coming along? Because we really work with people that we heavily believe in and we believe in their idea. So if we don't already have an idea, we're kind of just say, you know, please freely create because we brought you in because what we will freely create will push forth our agenda anyway. So I think those are our big model releases and series that we have available. Outside of that, we have a bunch of stuff on our GitHub as well, stuff that's being worked on, stuff that hasn't necessarily come out yet. There's a lot of that.

• 26:44Chris Benson?

  So I got a question for you as a follow up. It's pretty fascinating, the story that you've been telling us here because of that kind of organic, you know, creation of the organization or collective. And I'm wondering, as you've done that and you kind of went through and talked about the different model groups and kind of talked about, you know, the owners or spiritual owners, if you will, of each of those families. How do the different members of the collective interact to kind of share? Like, how do you each push each other along or share information or give ideas? So that cross-family efforts can kind of benefit from the overall collective. And as you said, now a C corp and you guys are more organized at this point. So what kind of culture has developed around those communications and learnings?

• 27:29Speaker 03

  Yeah, absolutely. I mean, when it started, it was just like a small discord, maybe like 10 people. From there, like, we kind of created more channels as people wanted to work on more things. And we had initially split up into like three, four different topics or sectors that people could assign themselves to. One being data synthesis, of course, so we can kind of find new novel methods and formats for distillation and the creation of synthetic data. One being training, like people who are just like really good at training hyperaram stuff and people who will come up with new architectures and new techniques. Another being agents, a group of people who want to actually try to build tools and do autonomous work with this stuff. And then we had this one category that it was a prediction for the future of simulation.

• 28:14Speaker 03

  So we had people that were very interested in kind of bringing this stuff into simulation, into unity, into kind of seeing how all these things came together. And it was interesting because the training built on the data synthesis, the agents build on the training, and then the SIM would build on the agents was kind of the idea. So everybody needed to work together because all those things are so intrinsically connected. But people would have specializations on kind of where in that workflow they wanted to work. We didn't end up doing a lot on the SIM side of things. Now, recently, there's a lot more interest because we have a lot more capability generally as the AI community does. You know, but as we've grown to, we went to 40 people, it was fine. Now we've gone to like 5,000 people in the Discord.

• 28:58Speaker 03

  It's a little unwieldy there. So what we do is we kind of tier people in. You come into the Discord, you can see maybe two channels. And then we'll give people a developer role. But we don't really let people select their own roles because we want to make sure we can kind of sort through people we know to kind of let them through. And even as we do open source research, a lot of it is unreleased and we want to make sure that it's kind of protected before release. So we create this developer role so people can then see like way more channels of just general development and development conversation. And from there, as we see, you know, contributors who have started to do more work or show more passion towards contributing to news in a particular field, or who have some reputation

• 29:42Speaker 03

  or some portfolio in a particular field, then we'll assign them one of those roles. And that will open up the family of channels relating to those roles and our current projects surrounding that role. So like data synthesis projects, agent projects, training projects, et cetera. So we kind of just tier it out so people can interact. And people have been around for a while or people we consider fellows or part of the cohort, they can usually see pretty much everything. So they're pretty effective in serving as coordinators for the cross communication between these different channels and groups. Even if something has like a particular, someone has a particular role or some channel has a particular role it's supposed to be a part of, like, it's still Discord and we're still very chill.

• 30:25Speaker 03

  So like, people will still work on like various different overlaps inside of just one channel as well.

• 30:43Speaker 04

  If you're listening, you know that artificial intelligence is revolutionizing the way we produce information, changing society, culture, politics, the economy. But it's also created a world of AI-generated content, including deep fakes. So how can we tell what's real online? Read, Write, Own: Building the Next Era of the Internet. A new book from entrepreneur and investor Chris Dixon explores one possible solution to the internet's authenticity problem: blockchains. From AI that tracks its source material to generative programs that compensate rather than cannibalize creators. Read, Write, Own is a call to action for a more open, transparent, and democratic internet,

• 31:26Speaker 04

  one that opens the black box of AI, tracks the origins we see online, and much more. This is our chance to reimagine world-changing technologies to build the internet we want, not the one we inherited. Order your copy of Read, Write, Own today, or go to ReadWriteOwn.com to learn more.

• 31:57Daniel Whitenack?

  I have a selfish question, which now that this is one of the advantages of doing the podcast, I get to talk to all the amazing people doing amazing things and learn from them. But I'm wondering, as a person who is also trying to fine-tune some models, either just for my own enjoyment and learning, but also fine-tuning models for specific tasks and in specific customer use cases and that sort of thing. There's a lot of people out there, I think many of our listeners, who are thinking like, since you, being part of this collective, have worked for, you know, since the sort of dawn of these many, you know, the proliferation of fine-tunes from Llama and et cetera.

• 32:41Daniel Whitenack?

  And as you've seen all that, as you're doing more and more fine-tunes now, as you're looking towards the future, do you have any kind of good advice? Or? Things to keep in mind for all those like fine tuners out there that are thinking about grabbing something off of Hugging Face, creating their own versions of these models. Maybe they have their own ideas about a specific take on on a model. Any general tips that you've found to be really useful over time or like pitfalls that you'd like to highlight?

• 33:13Speaker 03

  Yeah, I mean, I can I can try to think of a few off the top of head. I'll say that hyperparameters are really important and it's important to try to get that right. It's going to vary from model to model, but a lot of the time, some people think hyperparameters like don't really matter as much to like obsess over. And some people think it's like a secret sauce as well. So I'd say like, try to do a lot of research into like good hyperparameters, a good learning rate. Like, I'd also say like, I could be totally wrong about this as I am not the trainer of Hermes today or a lot of these models, but something I personally believe in a lot is like, ignore like people telling you to only train for like X amount of time. Like, if you're not overfitting, like just keep going, like if you can, if you have the compute, like keep training and keep

• 33:58Speaker 03

  going, like train for more tokens, more epochs. Like, that's something I heavily believe in. In terms of trainers to use, there's a lot of people who make their own scripts for specialty stuff. And there's, of course, like, you know, you can just use Hugging Face, but the library we use is called Axelotl, A-X-O-L-O-T-L, like the animal, by Cassius Wingleon of the Open Access Collective. We think Axelotl is probably the best general purpose trainer for LoRa's, QLoRa's, fine tunes, etc. It like any open source repository is buggy and stuff you're gonna have to work out, but it's, in my opinion, probably the easiest and most effective trainer to use for like,

• 34:42Speaker 03

  pretty much any model architecture available right now. So I definitely point everybody towards Axelotl.

• 34:48Daniel Whitenack?

  Awesome. Yeah, that's super useful. We'll share some links in our show notes as well. So people make sure and check that stuff out. Another kind of interesting question, as you see, you know, I think we saw these waves of models that came out maybe around synthetic data, fine tunes, or other types of fine tunes. I see this like interesting sort of thing happening over the past however many months, not that long in the scheme of things, but in the AI world, maybe a while, where we're kind of now, like, there's a lot of interesting approaches, more so than just fine tunes, but like mixture of experts and merging, and of course, multimodal stuff coming out.

• 35:33Daniel Whitenack?

  Now I see news kind of dabbling in that. You don't have to answer for the whole collective, but as there's so many of these things coming out and different approaches, what are some of the things within that, doesn't have to be one of those, but what are some of the things on your mind kind of moving forward, or on Noosa's mind kind of more generally?

• 35:54Speaker 03

  Sure. I'll try to go from like, simple to complex on the kind of stuff.

• 35:59Daniel Whitenack?

  That sounds great.

• 36:00Speaker 03

  I think that definitely just like straight up instruction tuning is great. There's other ways to tune, like the EVOL instruct method. I would advise people to try to create new instruction methodologies that allow us to make even better formatted data. People don't spend enough time trying to create new instruct formats. And we've definitely been swamped with not doing that as well. So I think towards the general community, it's a really easy place to get started. You don't need to really know how to code so much as think about how a human might more effectively phrase something or format something and kind of remix from there. I think that's like probably the easiest place to start. Then there's a model merging, right? Model merging is great. You can just

• 36:42Speaker 03

  like take two models and Frankenstein them together to question mark results. You know, you gotta just try and see what happens and feel it out. Then from there, I would say there's stuff like DPO. There's RLHF, DPO, this kind of rewards things that can let you like enable rejections, or create censorship, or put some kind of general concept or attitude towards a model. We found that to be pretty effective with the latest Noosa Hermes mixture DPO. It seems like people really like it and prefer it over just the SFT. So that's another thing that I'd heavily recommend. From there, we get a little more complex. We have some reward model stuff we're working on that I won't speak to just yet.

• 37:26Speaker 03

  Outside of saying we're working on it that we think is going to be like pretty big for reasoning boosts. Of course, there's techniques like chain of thought and tree of thought for like multi-step prompting. Creating data sets even out of that for any of these purposes I've already mentioned is going to be really effective. Now to stuff that maybe not everybody can, actually, a lot of people would already be able to do this. There's like something that we like to call over at news activations hacking. Where you're kind of messing with the way that a model, I'm trying to think about how to say this in like the most layman's terms. Like you're trying to mess with how a model like generally vibes about something. So rather than just doing a system prompt or something like that.

• 38:09Speaker 03

  You can actually like change the the model vectors to kind of be like more political about something, less political about something, more terse, more specific. And it has far more effect and control over a model than a system prompt. It's basically like a system prompt that like tells it to embody certain characteristics, but it's not something you can really jailbreak or get around. As far as my testing has shown, certainly not as easily as a system prompt. Like we have no problem jailbreaking even the most censored closed models today. Like it can be done by anybody with the right words, right? But um this activation stuff, it really creates a bit more of a robustness and fidelity to the concepts that you're trying to tell it to embody.

• 38:51Speaker 03

  There's a few more I'm trying to think of that would be useful for people. Uh, one thing is soft prompting. It's not really around anymore. It used to be pretty big during the GPTJ, like pre Lama days. And the Cobalt AI guys really pioneered the use of it in the open source community. But a soft prompt basically takes like massive prompt and and compresses it down to like way less tokens. So you can give your model like a huge prompt, like huge system prompt or huge amount of information and use like way less tokens. So soft prompting is cool. It's not going to be too difficult to like update it for like Lama, Mistral, like today's architectures. It's just like, nobody has really done it that I've seen. So, you know, to the community, if you guys do that, please share.

• 39:38Speaker 03

  Um, that's actually much easier than the activation stuff, I think. And then finally, probably the hardest unsolved is like, uh, sampling methods, like today we use like top K, top P, like, you know, nucleus sampling, et cetera, whatever, like, there's better ways to pick tokens for sure. There's better ways to judge the value of tokens for sure. Uh, everyone has been too kind of concerned with higher levels to get that low and do whatever the magic math is that I can't do. That would, uh, you know, enable some steering and some, uh, even beyond steering, like alternative sampling paradigms. And I think that would probably bring the biggest change and transformation

• 40:21Speaker 03

  to literally all models, regardless of the tune, regardless of the architecture, et cetera, get pulled off. So really looking forward to something like that happening in the space.

• 40:32Chris Benson?

  That was a lot of really good advice that you have there. I was sitting there trying to take notes while, while you're talking through it and everything. Wait, but he said that too, and he said that too. No, the really good answer there. Um, thank you for that. As we're starting to wind up here, uh, wanted to ask you, I know about, as we're recording this, it looks like it was just over three weeks ago, about four weeks ago, when we release, uh, this episode, you guys announced, uh, your $5.2 million seed financing round. So congratulations on that. That was pretty amazing.

• 41:04Speaker 03

  Thank you.

• 41:05Chris Benson?

  And I'm kind of wondering, so like you've kind of started with this kind of fairy tale story of kind of organically building from the ground up. You know, yourself, you connect with somebody else, a few other people join. You get to thousands of people contributing, you find, and, and really producing amazing work. And then, uh, you're incorporating and now you got the seed round coming. Where does that lead you? It's kind of a sky's the limit kind of scenario, it seems, you know, that now that you're, you're kind of launching and, uh, you know, on that, you know, as a corporation, as you said, where can you go from here? What do you anticipate over the next couple of years? Or even several years out, you know, what's the vision? What do you want to achieve? You've come a long way so far.

• 41:50Chris Benson?

  What's next?

• 41:51Speaker 03

  AGI, no, I'm just kidding.

• 41:54Speaker 03

  I believe you if you said it, actually.

• 41:58Speaker 03

  I mean, like, you know, someone will do it, but uh. And then you'll distill the knowledge. Then we'll distill and then you'll run the AGI on your on your neural link on your contact lens or something.

• 42:11Speaker 03

  But for us, like, there's a huge focus on locality. There's a huge focus on offline. There's a huge focus on take the power back, run the model yourself, do everything at home. Like, that's big for us. And at the same time, of course, we believe in scale. But there's this idea that, you know, there's so much unsolved at the small model size. Why don't we do that before we go to a trillion per amps? Because we can scale those realizations. But for us, like, there's certainly, you know, a transformation and change in attitude and in pressures from going from pure open source volunteer to as well having kind of this more corporate branch get created as well. But that being said, it's been pretty consistent, our ethos and our motivation for why we do this.

• 42:53Speaker 03

  And like you said, it really was organic in the sense that like, we're a product of the times. We're a product of the atmosphere of the AI community, like, people have said nice things, like you guys are setting the trend and it's not really true so much as the truth is like, we are one of many embodiments of the sentiment that the community has and that the world has, we think. Like, there's more than one noose research in this world, you know, there's alignment labs, there's Pygmalion, there's Cobalt, there's people who have been around before us, people who will come along the way, people who have already formed since we have. And there's lots of people who have kind of embodied the noose research ethos. And it's not really just our ethos as much as the overall community's ethos.

• 43:33Speaker 03

  There are people who have come before us, people who will come along the way, uh, who do very, very similar style of work as us, this kind of open work. And I think that's got everything to do with the fact that like, this is what the people want. We are just the everyman, just like everybody else. We're not like billionaires or super like all ex-Facebook or anything like that. We're we're just a bunch of people who really, really care about this, who who want to see. Everyone have access to language models, everyone be able to automate their lives, everyone be able to push their understanding of any topic to the next level. And our work, as we become an organization that's looking to, you know, be a company

• 44:18Speaker 03

  and create revenue, et cetera, we won't let it tamper or hinder any of the open source work we do. In fact, we want it to empower all of that work because we believe that the tools and the developments and services that we will be providing as a corporation will only serve to better feed the entire open source community. We're not really looking to suddenly make like a closed Hermes or something like that. We're more looking to create tools and do research that makes your open Hermes far more effective, far better, and, uh, you know, good enough that you may want to pay for that tool.

• 44:58Daniel Whitenack?

  It sounds like something I would pay for, that's for sure.

• 45:02Daniel Whitenack?

  Yeah, it's super inspiring. I, I really appreciate you, uh, taking time, Curran, to talk with us. I thoroughly enjoyed this because I am such a fan of, of everything you all are doing and the community that you've built. So, thank you for saying true to that culture and, and what you're doing. And I'm really looking forward to seeing what, what happens in the future and, and where things head. And I, I hope that we can talk again and have, uh, news back on the show and in a year when of course, everything will be different in the AI world. And I'm sure you'll still be doing, uh, interesting things. So, yeah, you're always welcome back on the show.

• 45:37Speaker 03

  Thank you so much. It's been a pleasure to chat with you guys. Thanks for being so candid. I'm glad we were able to kind of push our message forth more. And thanks for the validation you and the community have given us to keep doing this great work.

• 45:49Daniel Whitenack?

  All right. Thanks. We'll talk soon.

• 45:51Speaker 03

  See ya.

• 45:59Speaker 00

  That is Practical AI for this week. Thanks for listening. Subscribe now if you haven't yet. Head to PracticalAI.fm for all the ways. And don't forget to check out our fresh changelog beats. The Dance Party album is on Spotify, Apple Music, and the rest. There's a link in the show notes for you. Thanks once again to our partners at fly.io, to our beat freakin' residents, Breakmaster Cylinder, and to you for listening. That's all for now. We'll talk to you again next time.