Ep 17: Nomic AI Co-Founder Andriy Mulyar on "GPT-4-All", LLMs in Video Games, and Apple's AI Strategy

Jordan and Erica sit down with Andriy Mulyar, Founder & CTO of Nomic AI, and discuss his motivation for creating GPT4ALL, the importance of data-centric AI, the use of LLMs in video games, and which technology companies are well positioned to “win” in the GenAI market long term.

• 0:00Speaker 00

  Welcome to Unsupervised Learning, Redpoint's AI podcast. I'm Jordan Siegel, joined by Erica Brescia. And today, we spoke with Andre Mulyar, the founder and CTO of Nomic AI. Nomic is a startup that recently announced a $17 million investment after its data maps produced by its Atlas product blew up on Twitter and its open source model GPT-4all achieved an astounding 50,000 stars on GitHub and nearly 25,000 Discord members. We talk about how LLMs are being used in the video game industry.

• 0:25Speaker 01

  There's been full-on games that have GPT-4all integrated into the NPCs and they all have their own different prompts.

• 0:30Speaker 00

  Andre discusses the long-term strategy and vision behind Nomic.

• 0:33Speaker 01

  Our goal is to really make AI systems more accessible to everyone. It's as simple enough that your grandmother can go in and manipulate the data that goes into an LLM.

• 0:40Speaker 00

  And he speculates about the role that Apple will play in the LLM space.

• 0:43Speaker 01

  I would envision in a couple of years, you will get a MacBook delivered to you with an LLM already baked into it.

• 0:48Speaker 00

  Hope you enjoy the episode.

• 0:52Speaker 00

  Andre, we're really excited to have you here. Thanks so much for joining us.

• 0:55Speaker 01

  Thanks for having me, Jordan. Nice to meet you guys.

• 0:57Speaker 00

  Awesome. Well, to start, you know, I'm sure our audience would love to just hear a little bit more about yourself and what got you into AI in the first place.

• 1:03Speaker 01

  I've been interested in computer science and computers since day one. I think started programming when I was 13. I really got into AI, specifically NLP. Right when I went into college, I joined a research lab working on natural language processing as it applied to biomedical text. And so I just realized that the direction computing was headed was to be AI-powered. And so that's where I started dedicating all my thoughts and time. Really, everything started off in relation to what we're building at Nomic. Actually, at NERPs, which is this large AI conference, happens yearly, where I actually met my co-founder, Brandon. So he was just fresh out of grad school. And it was a meetup for NLP at NERPs. And what's funny is that in 2019, everyone was all about images.

• 1:46Speaker 01

  NLP was still this field that hadn't really gone into the mainstream. It was a conference of about 10,000 people. And the meetup that we went to for NLP was only about 40 people. And we sort of looked around. And what we realized is that we were the two youngest people there. So we just naturally congregated. And it turns out Brandon just got out of grad school. And he was actually hiring for this new startup he was working at. And they were basically training large language models on medical text. And I sort of hopped on board there. And that's how we kind of learned the problem space of like, what we're actually working on and solving at Nomic. So we learned all the difficulties that come in with training large language models back in 2019.

• 2:25Speaker 02

  So maybe you can complete the rest of the story to date for our listeners. Like, how did you go from meeting at that conference to starting Nomic, which I'm now learning how to pronounce correctly for the first time? And tell everybody a little bit about what you're building with GPT for all and also with Atlas.

• 2:43Speaker 01

  Sure, yeah. So one of the, one of the things when you do machine learning for a while is you realize that there's only so much alpha you can get on like trying new models or like making modifications to the actual machine learning algorithms. Uh, the important part is the data. Uh, this is something that most people who have been in the game for a while, known for a while. And this is everyone who's entered since like ChatGPT's launch and everyone who's sort of focused on AI since then is now starting to sort of realize. Brandon and I, uh, we were lucky to realize this really early. Uh, because of this opportunity that we got to sort of work on building models to do text manipulation, LLMs way back when. Um, so the basic, the basic problem that we were working on there was automatically sort of like summarizing radiology reports for radiologists. That was the product that the company was building called Rad AI.

• 3:27Speaker 01

  And what, uh, we realized is that, you know, you could spend a month iterating on the models, trying to improve, create a better transformer, create a, create a, create a better model. Or you can spend a week spending time cleaning the data and you would just get like a much larger improvement from the latter than the former. Uh, the problem with that is that nobody wanted to spend time staring at text files of data. Uh, that, that sucks. Um, and not only, not only is it something that is not fun to do, it's also something that's like really exhausting. But it's also the place where like the most benefit could come out of the whole like sort of modeling and machine learning process. And so like what we really set out to do at Nomic was to solve that one problem. We realized all parts of the machine learning pipeline were clearly becoming sort of replicable. Uh, deployment of models, training of models, you know, you have companies like Mosaic

• 4:10Speaker 01

  that's sort of like claiming to automate the whole training process of LLMs for you now. The only part that isn't really automatable, uh, the only part that's difficult is, uh, the actual data curation. What data do you put into your LLM such that you can program it to complete the task that you want to do? And that was sort of what we set out to solve and that's kind of what we solved with Atlas. So Atlas is this massive interactive viewer for unstructured data sets. So if you have 60 million documents of text and you're trying to use that document, those documents to program your LLM. By training it, how do you figure out what you put in? What do you take out of the L M? It's really important. You put in the wrong documents into the L M. It learns garbage. You get garbage out of your model. And that's sort of what we set out to solve there. So Atlas, it's its viewer for unstructured data. You can upload tens of millions, actually with the new launches we have coming up. It's going to be hundreds of millions of documents of text, anything you can embed,

• 4:55Speaker 01

  text, image, video, audio, and you can interact with it. So it groups together things by semantic similarity. And anything that kind of comes out in that models, in the viewers embedding space, you can see visually. So it allows somebody who doesn't have like a machine learning background or machine learning PhD to be able to very quickly do the data work required to be able to get these large models training. And that was sort of like the thesis behind everything we were doing. Every, every single piece of data in the world is going to be either put into large language models or machine or whatever foundation model that exists in a year or two, or it's going to be spit out by them and the whole internet's going to be full of machine generated text. We need an interface to be able to interact with these massive quantities of instruction information. And that was sort of like the whole thesis behind the company. That was the bet.

• 5:39Speaker 00

  Awesome. That's incredible that you're going to get to hundreds of millions of documents. I mean, how do you achieve that scale in the browser?

• 5:44Speaker 01

  So it's a few things. Number one is you need to find this one NYU professor who in his free time was building this library and let you visualize billions of points interactively in the web browser. And once you set your company to convince him to join you. And so there's a whole process involved in that. But once we had that convincing, we had sort of like the front end, the front end piece. And so this is library called DeepScatter. It's open source. What it allows you to do, as I said, you can visualize billions of points in the web browser using a bunch of fancy graphics powered web interactions. Essentially, five years ago, this would be impossible. Like we've chatted with some of the authors of the original sort of dimensionality reduction papers that a lot of our core tech is based on.

• 6:28Speaker 01

  They had a lot of the same ideas, like they wanted to build Atlas 10 years ago. It just wasn't possible because the browser tech wasn't there. So like we got the browser tech into the right state. And then there's essentially this whole entire problem in the backend that needs to happen where you need to build essentially the world's best embedding database and have things be able to ship to the world, to the front end web browser. And that's sort of like the kind of the key ingredients that go into building a giant unstructured, unstructured data viewer. Under the hood, the way you view unstructured data interactively is you want to view it through embedding spaces. So like what Atlas technically is, is the world's best sort of like embedding database and embedding database viewer.

• 7:03Speaker 00

  So finding a famous professor with a famous library and getting him to join your team, very repeatable strategy for startups there.

• 7:10Speaker 01

  Yep, easy.

• 7:11Speaker 00

  No, that's awesome. It's super impressive, the visualizations. I mean, A, they're beautiful, but B, there's obviously a lot of really practical applications for them. Would love to maybe dive in there and sort of, how are your users using it? Is it more for sort of mislabeled data and being able to refine that for fine tuning? Is it more for sort of topic modeling and clustering?

• 7:33Speaker 01

  Yeah, yeah. So there's kind of a few, kind of two main types of people who utilize Atlas. They're all people who really have this problem where they have large collections of instruction information and they're either using that data to go in and then feed it into downstream machine learning models, or they're just in a situation where they have large amounts of data and they don't even know where to get started with using it. So some of these use cases might be machine learning model training. We actually have like VCs in there who are going through and like piping deal data to be able to organize, like basically be able to get like mental maps of like what is the current state of like flow coming in. It's essentially akin to like an Excel but for instruction information. So if you don't have tabular data, but you have a lot of instruction data, text, image,

• 8:17Speaker 01

  video, audio, that's where it really shines. On the machine learning side, this is sort of like what we built the product out from the start to be. We didn't really anticipate all these other use cases that sort of emerged. People just, we made it very easy to upload data, sort of get started and get working with the product, and people just started uploading all types of data. And we started noticing a bunch of like companies uploading all types of data and that sort of like led to conversations and so forth. But the machine learning use case is like what was near and dear to our heart and what we sort of built out everything for. And there, it's really an iterative data refining process. So the whole thing I mentioned to you was like, data is like the most important part for the machine learning workflow. And what you're able to do with Atlas is you can go in and visualize the inputs, so be able to see what kind of information is going to my machine learning model.

• 9:01Speaker 01

  And then more importantly, I would say even visualize the outputs. So one of the few things that we were able to do with this tool, and it's kind of what we did as a demonstration of like Atlas's power, was we were able to train this model called GPT for all. In in March, and this was sort of like a large language model where we carefully curated a large collection of Dext, fed it to the model and surprised the model was good because, you know, we didn't put garbage in and garbage didn't come out. What's what you're able to do there is by visualizing the outputs of the model in Atlas, you can actually go in and say and pick out. And with every single model output, there's an associated score which says how confident the model is that the output is like the correct thing that it should have generated out.

• 9:41Speaker 01

  And we were able to do a lot of iterative refinement of the actual model by associating each model output alongside this score. And in Atlas, actually viewing it as a heat map, so you would just see regions of the whole data space. And you can clearly tell the model was very, very uncertain of. And just by literally doing manipulation, so going in and removing those regions or emphasizing them if you needed to, you can adjust how the model was sort of reacting to the data that it was being fed. So like, how do how do how do people use it for machine learning tasks? Well, it's a really like a like an iterative refine sort of process. We were doing it internally a lot, and now it's exciting to see people sort of realize that the utility of it externally as well.

• 10:17Speaker 00

  I mean, we'd love to understand a little bit. You mentioned sort of data is is the most important thing as part of the model. You've also talked a little bit in the past about hallucinations and how how that ties to Atlas. I think our viewers would love to hear that.

• 10:30Speaker 01

  Yeah. So L L M's, as everyone now knows, but no one knew a year ago, have this tendency to make stuff up. We call this hallucinations. And what that is a artifact of mainly is spurious correlations in the training, the pre-training sets or fine tuning sets of the machine learning models. So what happens is when you have examples where, for instance, you've gone in and shown the model that you should be able to correlate, for instance, spitting out something that isn't true in situations where you're also spitting out things that are true. The model will learn to replicate that. And this is just like an artifact of machine learning systems today. It's very difficult to prevent this. There's many approaches that people take.

• 11:14Speaker 01

  But one of the sort of easiest ones to iteratively reach a model that on average hallucinates a lot less is having very fine grain, fine grained control of the data coming in. So the reason, for instance, like what we had to do at Rad AI when we were working on these medical L L Ms is you couldn't have hallucinations coming out of the model. If you had hallucinations, you didn't have a product. And this is the interface that we really wanted back then. This is what we were doing manually, going through, searching through all the data, finding examples of outputs that were hallucinations, pinning them back to input documents, and figuring out how we can go in and remove all those input documents. And that was kind of like the iterative process. And what Atlas allows you to do is instantly be able to replicate that process. The second you find an example of a hallucination coming out of a model, you can wrap it back to

• 12:00Speaker 01

  bad inputs. So by looking at the input that came into that generation, you can map it back to the bad inputs, and all the bad inputs are organized and grouped together semantically. So that's sort of like one way you can use Atlas to sort of debug

• 12:14Speaker 01

  an L L M artifact, this artifact being hallucination. But in general, like L L Ms, I sort of, I sort of like ask people to think of them sort of like these computers which you program with data. And they're all going to be biased in various ways. So like you can bias an L L M to hallucinate a lot by, you know, putting in a bunch of crappy data into the model. You can bias the L L M to always tell you that it's not gonna, it refuses to respond to your inputs because, you know, it doesn't agree with like the ethical or moral guidelines of some organization by biasing the input data. Atlas is a place where it makes it really easy to sort of manipulate and program your L L M by just manipulating that input data that goes in.

• 12:50Speaker 02

  Got it. And, you know, going to G P T for all, you just mentioned, of course, you released it in March. It was wildly successful. I think you went to fifty thousand stars, virtually overnight, not exactly, but close. You said that that was really to showcase the power of Atlas, but, you know, it kind of seems to have taken on somewhat of a life of its own. At the same time, obviously, there have been releases like Lama two and other open source models to come out. So how do you think about the future of G P T for all and how core is it to the overall nomac strategy?

• 13:25Speaker 01

  Yeah, that's a great question. It's, it's funny. One of the, one of the lessons that we, we've been learning over and over again, and now we finally like really, really integrated into our practices and like how we run the company, is the most important thing is to make software, and whether that software is a bunch of machine learning models or whatever it might be, the most important thing is to make it super accessible to people who are not technical. Um, and that was sort of like the lesson that GPT for all really like instilled in us as an organization. Uh, but also just like what we learned, the, the world really wants. Um, there's a lot of ways you can go about using software, which is like really not intuitive. And there's a lot of ways you can go about using machine learning models, uh, where you have to, you know, spend thirty minutes installing a bunch of dependencies to even get your model to run.

• 14:09Speaker 01

  Uh, and GPT for all was sort of like a, a, a, a mixture of all these ingredients put together. So what we were able to do, right? So it wasn't just the model. It was sort of the platform for actually running the models as well. Um, so we obviously, as I said, uh, got a, got a, got, got an exam, got a model out. That was, that felt like chat GPT, uh, but was able to run locally in people's machines. And that local process for actually running it was super, super easy to get going. So you could have a machine, a computer that was from twenty thirteen with, you know, uh, very, of it, with no graphics card, but only, only CPUs, uh, with the low, low amounts of RAM and you could run the actual model.

• 14:46Speaker 01

  Um, and this sort of level of accessibility is what caused, you know, that sort of like, uh, like meteoric rises as, as, as, uh, people, people have said. But what the important part there is not just like, hey, there's these models, you can run them. That's really cool. Uh, they feel like chat GPT, they're a little bit worse. The most important part is the fact that you can run them anywhere. You can run them on edge devices, you can run them on machines that don't require, you know, a ten dollar per hour GPU plugged in to even, you know, play, play around with the system. Or like a subscription to some, to some service which, which hosts it for you. Um, and that's sort of really the kind of direction that we hope to take the company. Like our goal is to really make AI systems more accessible to everyone. Uh, the interface that we're building Atlas for the data that powers and comes out of AI systems.

• 15:28Speaker 01

  Uh, our goal is to make that as simple as, you know, uh, simple up that your grandmother can go in and manipulate the data that goes into an LLM. The same thing for GPT for all. Uh, if you have a device that, you know, is like a, an old smartphone, uh, that has, you know, a very, very bare minimum set of requirements. These devices should be able to natively run large language models without having to plug into external servers and APIs. Uh, that's sort of like the direction we're taking it. If you go look at the models right now, uh, we sort of have this curated list of, uh, minified models that exist there that can run in really low, um. Low resource devices. We're actually in a few weeks, I guess the show in this podcast is coming out, but I'll say it anyways. In a few weeks, we're launching one of the sort of like biggest releases that we've had so far, enabling edge GPU support.

• 16:11Speaker 01

  So we've integrated with this open standard called Vulcan, which is this open standard for running accelerated accelerated inference across many different types of graphics cards. So right now, the majority of models in machine learning can only run pretty much on Nvidia Nvidia graphics cards. And it will be enabling the ability to run these these models on AMD GPUs, Qualcomm GPUs, Samsung GPUs. So it's really an accessibility angle. You want to be able to make sure that people can actually use this technology, not have it, you know, concentrated in the hands of a few hardware providers or a few companies.

• 16:46Speaker 02

  Awesome. I'm sure there's a lot we could dig into there, but before we get to it, just on the accessibility front, because you've made, you know, a really powerful model, so much more accessible to developers. Like surely there have been some surprising things that have come out of that. Like what's one of the more interesting use cases or success stories? Or something that you've seen happen because people can build locally with with GPT for all?

• 17:11Speaker 01

  Yeah, I mean, the biggest one is, I would say, like, there's this whole blossoming industry right now of people building LLMs into video games. And I think that that was just straight up not possible before you were able to run the models locally alongside the games. So for example, previously, if you wanted to go bake in, you know, some like LLM component into your games, maybe you have like characters in a game that were LLM powered. You would have to go in and use a third party API. That would mean each of your users would have to pay for the outputs of that LLM. But now developers can actually bake the model in with the LLM they ship out. And magically, you know, it becomes something that's very feasible as opposed to being this sort of like feature that you would have to pay a bunch of money for.

• 17:53Speaker 01

  And people are figuring out how to how to integrate these in very, very creative ways. So like a lot of a lot of times that we see a lot of this traffic in our Discord of people just like posting the cool things that they've built. There's been there's been full on games that have GPT for all integrated into like the NPCs and they all have like their own. Different prompts and and but it's running locally on the machine of the people. So that part is really cool. Um, and also it's like around development cost, I would say like one of the, one of the biggest things that I've seen is I, and I, this, this happens every day. Somebody comes in and is just like, yeah, you know, in production, I'm using one of these, some of these larger, these larger models that, you know, are, are a little bit better. Uh, in performance because they require like GPUs to run. But my last three months of dev work was that we're all spent using your models because, you know, it's, it's cheap.

• 18:35Speaker 01

  Um, and it doesn't require any sort of additional resources than the ones I already have. Um, so those are kind of like the two big, I would say like wins. Um, there's just this whole, this whole, this whole area of new possible use cases for this technology that just wasn't possible before. And it also developers actually have access to building with these without breaking the bank.

• 18:54Speaker 00

  That's awesome. I'm a huge gamer, so thank you for making games better. I mean, would love to just sort of take a step back to, to March, when you first built GPT for all and maybe go through a little bit more of the development process that was done in terms of, you know, how'd you get the data and what did you do that with that data with respect to Atlas? And sort of the, from beginning to end, how you created the, the first version?

• 19:16Speaker 01

  Yeah, yeah, sure. So, um, this is a fun conversation. Uh, so essentially the best large language models at the time in early March, uh, were all pretty much handled. So the best models that people can interact with and sort of be able to, that were able to do useful instruction following or like chat assistants, uh, for individuals, uh, were owned by one company, uh, OpenAI. And you could access them through the API, uh, that they had. Um, and while that was cool at all, uh, that was obviously like an issue from just around like who gets to own software in the future? Who gets to control what, how people use these models? Um, who gets to control the privacy around them? Uh, so the kind of the whole like thesis behind even like playing, play, playing in the waters of like, hey, let's get our own LLM trained. Uh, maybe it'll be a good demonstration of Atlas being powerful.

• 20:01Speaker 01

  Uh, but also I think at the time, so my thinking around it was essentially centered around this idea of, you know, it's not actually that hard to train a large language model. Uh, we have experience doing it. Uh, it seems like the time is ripe to be able to like actually make a demonstration that, you know, it's not an unachievable thing. And as you've seen over the last four months, right? There's like a whole batch of YC startups that are all just like LLM training. Uh, it's, it's, it's very much feasible thing, but in, in March, it was unclear to most people that there was a thing that was possible. And we had sort of like the unique positioning where like, we knew how to do it, so we did it. Um, and the way you do that is you need a lot of high quality data. Uh, and the easiest way to get data that people. Have experienced being outputted from an L11 and recognize that like, ah, this is like

• 20:43Speaker 01

  a useful thing the L11 can do is actually by doing distillation from a larger existing model. And the best one out there was OpenAI. So we actually went in, gathered a large data set of OpenAI outputs. So this was like basically the outputs of ChatGPT. You take what the input is, you take what the output is, and then you go in and train your L11 over top of these inputs and outputs. Obviously, this raises some like concerns around OpenAI's privacy policy that says you can't build a competing product to it. This is why we went in, released the model. It was non-commercial licensed, but it was more of like a demonstration of like the power of what you could do. If you actually go in right now and go on a hugging face or go anywhere else, you'll see that every single one of these sort of top open source models that you can use, use this exact same procedure. They go in,

• 21:26Speaker 01

  they take data from the outputs of GPT-4. It's not something that frankly, OpenAI can really enforce. But obviously, as an organization, we had to play it safe and make sure that, you know, all the asterisks were covered. But essentially, we go in, we get in this large data set, a couple million OpenAI generations. And then we put it into Atlas, and then we take a couple million generations, and we reduce it to like 300, 400,000 by removing all the junk. And the way you define junk is what are examples of bad inputs and what are examples of bad outputs? And usually, the easiest manipulation you could do is just remove them. The nice thing with Atlas is you had the ability to, once you found one example, you found all of them, because they were organized semantically

• 22:05Speaker 01

  through the embedding space of all the text that consisted of the inputs and outputs. So once we actually had the model trained, this is something we did over a weekend, it didn't take long. The biggest thing was like, how do we actually put this out to the world? And again, this kind of accessibility angle that I mentioned earlier, the sort of key thing that we did, there was like a new development happening at the time where this guy, this Bulgarian guy, Georgi Gerganov, was building this library called GGL. They were top of it, Llama CPP, which sort of delivered the core building blocks that allowed fast CPU inference for specifically like the Llama machine learning architecture. And what we did is we, like, in the early days before that,

• 22:47Speaker 01

  Library like even had like a thousand stars. We went in and we like integrated with it directly. Uh, and made sure our models, uh, so we forked the repository and we made sure our models, uh, there's, there's some custom stuff in there that we had to handle. Uh, with like the model architecture, we made sure that it worked with it so that anyone could go in and actually take that model that we produced. With that very, very high quality data, we got that open AI, uh, supplied, um, and, uh, was able to run it. Uh, and that's what we did. When we launched, we released the model, uh, openly, uh, and then we also released this sort of like one line command line script. You can go in and it popped open this terminal and magically you had a chat GPT in your computer. And that I think impressed a lot of people, uh, because previously, uh, they were, they, they, they would just assume that this is like a technology that required, you know, millions of dollars of GPUs to train and run.

• 23:31Speaker 01

  Uh, and that was just frankly, not the case. And what you did is a good demonstration of the opposite. Um, and then once people realized that it was possible, uh, obviously all the developers in the world had started spinning all the things you can do with this now. That you don't have to, you know, be tied to the internet. Um, and that was sort of like the sort of like or, or origin story around, uh, how the, how the whole thing came to be. It was really a reaction to like open AI's GPT four paper. Um, if you had to ask me personally, uh, when, when we read that paper, uh, well, when we read that PDF. Um, and we went in and sort of saw, saw those lines about, you know, they're not going in and releasing, uh, models openly anymore. Uh, even though that's kind of like, that's something, that's kind of like the core thesis around machine learning.

• 24:10Speaker 01

  Uh, like in the, in, in, in the early two thousands, there was this conference called the, um, there was this journal called the Journal of Machine Learning Research, which like separated from Elsevier. Because people weren't releasing like machine learning work openly. Um, there's been a whole tradition of open, open access and open, and, and, and open research in machine learning. Um, and that was sort of like the main sort of reason we, we even embarked on the project. Um, but that's sort of like, hopefully gives you a few, a few, uh, a few insights into how the whole thing came to be.

• 24:36Speaker 00

  Yeah, no, I love that story and it's super cool how you use distillation and then use Alice to complement it. And it's just, uh, it's a really cool process you went through. I mean, would love to dive a little bit deeper on the distillation side. There's been a lot of research there and would love to hear your thoughts on sort of the pros and cons of that as, as a strategy for, for model building.

• 24:55Speaker 01

  Yeah, I mean, like. I guess the question is like, what is your goal, right? If your goal is to get the best, most high-quality large language model, or frankly, other any machine learning model that you have, that for like your own purposes, especially if they're commercial purposes, you want to think very carefully about the data that you're putting into it. Sometimes you can benefit from taking a larger model and distilling from it into your model, because maybe there's information that the larger model knows that your model should also know. Plus, you want to augment it with a little bit of your information. But distillation is really just a process like any other of making a machine learning model improve or be biased in ways where you and be biased in ways where you want it to be biased. One of the very first sort of like distillation papers in the deep learning era

• 25:40Speaker 01

  was like Hinton's 2015, like teacher-student distillation work. And it's just a another way of training machine learning models.

• 25:50Speaker 01

  It's very, it's very useful for LLMs, because it is the case that the best LLMs are ones that you can't actually access yourself. You can only access the outputs coming out of them. And it becomes useful if you want to train, let's say, an LLM to be, to act like a, acting like an assistant. It becomes less useful, for example, if you want to train an LLM for purposes that aren't like building something that follows instructions or does assistant-like behavior. It really depends what you're building for.

• 26:16Speaker 02

  So we've talked a little bit about how you got to this point and a little about the upcoming release that you have. You know, where does Nomic go from here? And how do you balance over time your focus on, you know, Atlas versus GPT for all?

• 26:32Speaker 01

  Yeah, yeah. So we get this question a lot, and the answer is pretty easy. When you think about building software that's accessible to people, you want to make sure that all parts of it are accessible, right? You want to make sure that, in our case, Atlas, which is the interface that we have for data manipulation. And then we have this interface for interacting with the models that eventually get produced out of this data. You want to, and we have efforts on, in both ends of our company to make sure that these ecosystems are maintained. The software is high-quality. We're obviously improving it. We have a lot of efforts that we're kind of considering and thinking about around maybe unifying these two components.

• 27:13Speaker 01

  But really it comes down to a question of like, what is best for, there's like, so we have this open source arm. We have this arm which is centered around actually like making revenue for the company, Atlas is a paid service. And we don't want to sort of hinder either of them, right? There's a lot of, a lot of important things that you have to do to make sure open source community actually stays open source. You don't want to go and start injecting sort of like ads or like paid features into the existing open source software. People don't like that. And obviously on the business side, you want to make sure that, you know, you're meeting the obligations that you need as a company, right? The thing that allows us to maintain GPT for all, contribute developer effort to it is the fact that we have a business, right? It wouldn't be possible any other way.

• 27:57Speaker 01

  And there's a lot of efforts that we're doing internally to make sure that both of these can kind of shine in their, shine in their room lights.

• 28:04Speaker 02

  And on the Atlas front, you said, you know, it is a commercial product. Are you selling it today? And like, what does pricing look like? How do you, how do you charge?

• 28:15Speaker 01

  Yeah, so we sell it mainly B two B. If you go on right now to the actual Atlas website, you can find some language around pricing, but it's mainly usage based. So the more data you store, the more, the more you get charged for it. The more sort of like individuals you collaborate on your data sets with, you get charged for that as well. And we're planning a lot of sort of new features in the future that allow you to, for instance, produce models off of your data. Get increased access to ensuring that, you know, you can use it in a production setting if you wanted to build services against it. Which all come with additional charges, as you can imagine. But sort of like the main pull that we're getting right now is sort of like a business, as a business is around making, making Atlas very compatible for launching on the infrastructure of existing companies.

• 29:01Speaker 01

  Most companies just don't have tooling for large data sets. Their closest solution is like Parquet files and their data bricks or Snowflakes clusters. And that just like doesn't work because it doesn't work with data scientists. And Atlas is just an interface that's really intuitive for people. So that's sort of like where our efforts are focused on.

• 29:18Speaker 00

  You mentioned, you know, a lot of companies don't have the associated tooling required to deal with L L M's today. What are the sort of highest priority blockers or gaps in the L L M space at this point?

• 29:28Speaker 01

  So here's the kind of, here's the tooling that I see right now. You have. Organizations like Hugging Face, which have done an amazing job at sort of like stewarding the ability for people to like host their models, host data sets that can be put into models. But oftentimes, a lot of the data sets that you see there are kind of like the standard benchmarks from like, from, from, from, from the research community. Another thing that they've done really, really an amazing job at is, is the metrics there. So like, if you want to go about building your models, that's one of the best places to do it. When it comes to actually deploying the models, that's where people get like, kind of really confused, I think. There's so many options, there's so many ways to do it wrong, and you get charged an arm and a leg

• 30:09Speaker 01

  for something that gets like, you know, one hit, one hit an hour, but you're paying like $3 an hour for that, for, for, for that hit, or like request your LLM. And then there's like this whole area of companies doing like LLM ops. So you see them building firewalls to make sure that your interactions with LLMs, you know, don't do bad things to your employees, or your employees aren't sensing sensitive data out. There's model, there's companies focused on LLM evaluations. So there's people like all across or like the stack. But I think really what it mirrors is the standard sort of like ML ops space that has existed for the last few years. It just has like this like LLM flavor to it. The biggest underserved place is companies that are, I mean, I don't, I don't mean to plug us, but like companies doing what we're doing, focusing on the data.

• 30:53Speaker 01

  To name a company that isn't us, another really good one is Argila. Argila, Argia, it's, I don't, I'm not exactly sure it's pronounced, I think it's Spanish. But they are doing like very like data centric AI focuses and like the things they are building. Like I said, the, the most important thing you could do as somebody who's building an AI model is to spend a lot of time with your data. And if you're not, you're going to get, you're going to waste a bunch of money. You're not going to get the bang for your buck when it comes to the resources, the GPUs that you spend. You can spend thousands of dollars on GPUs to train your model, but bad data in, the model's going to be bad coming out. And anyone who's building tooling around making sure that you can get it right the first time, making sure the data going in is the things you're comfortable with. Those are, that's a thing in the area that's like very underserved right now.

• 31:35Speaker 02

  I think, you know, just speaking of that for a minute, you're in a really interesting position because so many companies are using your product across like various parts of of their process and pipeline. So like, what else aside from, you know, the the data management is really interesting to you right now? Like, what parts of the stack are interesting? And, you know, are there any other companies that you would highlight that you think are really well positioned to, you know, kind of either take or continue on leadership positions moving forward?

• 32:08Speaker 01

  I mean, that's kind of hard to say. Like, as you said, it's the early days of of this whole thing. I think the most important thing to focus on is maybe not exactly what individual companies are building, but looking at sort of like the backgrounds and sort of like the thought processes behind the people who are actually doing the building. As we know, like, a lot of what happens at organizations is driven by the ability for like founders to like pivot and react to like new environments they're coming in to like what in the in the area that they're building. I your question is extremely hard to answer because I don't think there actually is is an answer. There's there's big organizations out there that are pushing forward the best models. Those models probably for specific use cases aren't

• 32:52Speaker 01

  the best options. Companies will have to go build their own models, I think. And there's still kind of be a whole ecosystem of software that supports that. There's sort of situation where, for instance, GPT for all works really well when you want to run your models on edge devices and you can't connect to the Internet. The situations where, you know, you could be like a character AI who's serving, you know, millions of requests an hour and you need certain types of infrastructure that mimic more like OpenAI serving infrastructure. You really need to make sure that whatever you end up building actually is something that is like the way you build it is aligned with the way people are going to use it when it comes to, you know, not burning all your money on GPUs. And also, you want to make sure that you build it right first time, I guess.

• 33:34Speaker 02

  Got it. And you know, I'm sure you have a strong point of view on this, given that GPT for all is open source. And we spent so much time talking about accessibility and how important it is in empowering more developers to actually be able to build cool things with AI. How do you think this plays out over time, right? There's obviously like kind of two big factions right now. Some folks who think that AI should be developed in, you know, behind closed doors for quote unquote like safety reasons. There is others that are very adamantly in the open source camp. I guess you can tell which camp I fall into with my air quotes. But how do you think this evolves over time?

• 34:16Speaker 02

  Do you think everything veers more towards open because of market pressures? Like, do you think there will always be a few big proprietary models? And players that have so many resources that they kind of stay out ahead? Like, how do you project this forward?

• 34:31Speaker 01

  I mean, I think, I think like to give a hypothesis like what the next few years look like. It's going to be the case that a lot of these sort of like big model, these sort of foundation model companies that you've seen come into the sphere of the last few years. OpenAI, Coher, Anthropic, this big funding round that Mistral raised to build like a European centric foundation model company. What will happen is if they're able to ship out high quality models, they have a bunch of money to be able to, you know, demonstrate that these models are more powerful than maybe out of the, these models are more powerful out of the box, like without doing any sort of customization over top of them than alternatives that are open source. They're going to have a really great time and easy time convincing like existing industry players and enterprises that,

• 35:16Speaker 01

  you know, rather spend a couple million dollars to fix their AI problems instead of like build it themselves. And they're going to be able to, you know, convince them that, you know, they're the best option to go with. But I think the majority of the ecosystem that is building software that is like LLM powered or like AI powered, they're going to end up running the models themselves. And what that might mean by running it themselves, I mean, they might be running on it on the infrastructure of other people, using other people's tools to go in and build those models. But they're not going to be, you know, sending their data and API calls like a centralized model company and getting the outputs and then just like. Go in going forward with that. They'll be customizing the models. They'll be they'll be implementing

• 36:00Speaker 01

  custom features over top of them that that you just don't get out of these sort of like like like one size fits all inference APIs that you kind of see right now. I don't know. I one one big area that I have just kind of like a lot of working experience in it. That's just like ML and like the medical space. There, for instance, like you only have two options if you're a medical AI company right now. You either build the model yourself, or you strike a very, very expensive deal with OpenAI to be able to access LLMs. Those are your only two choices. Guess what the majority of companies are doing who aren't, you know, the giant multi-billion dollar healthcare companies. They're building things themselves.

• 36:40Speaker 01

  And it's gonna get easier. It's going to get cheaper. And also, it'll just be the option that I think makes most sense to most people. Because the iteration costs are sometimes really expensive with these foundation model APIs as well. Like, I I really, I would be really surprised if the costs of all of these foundation models for like inference APIs stays the what they are right now. I have this hypothesis that like everything is being sort of like underpriced right now to be able to grab as much as much sort of usage in people's apps. It it feels unreasonable to me, especially if like the amount of the demand doesn't stay that high for a lot of these like larger models like GPT-4. It's becomes cheap. It is actually like financially viable to keep serving them at at at scale for free, for example.

• 37:25Speaker 00

  We we've talked a lot about, you know, running models locally or on edge devices with constrained resources. Would love to share your thoughts on Apple and and sort of what their role will be in the LLM space going forward and how they're enabling LLMs on device as well.

• 37:39Speaker 01

  Yeah, I mean, I think historically, right, Apple always, they weren't usually first. But when they did launch, they were best. I think you might see that with this as well. There. So we've done a lot of low level work internally at NoMac, working with their like Apple silicon chips for running machine learning models. And they're extremely good. And I'm sure right now, they're very quiet, but. I'm sure they're building lots of interesting stuff that'll be shipped in the boxes like like I I would envision in a couple of years you will get a MacBook delivered to you with an L M already baked into it. That Apple has sort of like customized to run super fast on their on their chips and everything.

• 38:17Speaker 01

  It's just the bet that they're making, uh, build the hardware from the ground up and build it specifically geared towards uh where computing is headed and computing is headed that most. Most operations on your computer are going to be run through AI models before that operation is fed back to you. So um they've done a really good job of positioning themselves, I think, for that world. I I say that as somebody who uh had a Lenovo think pad for his entire life until about last year, so.

• 38:43Speaker 00

  And then, um, you know, switching gears just a little bit uh to a question that I'm sure you've thought a lot about and get asked a lot. Um, what are your thoughts on AGI? You know, will it happen? When will it happen? And what do you think the implications uh if it does happen?

• 38:57Speaker 01

  I guess when it comes to it it happening, uh, I think you would need some fundamental shifts into just like how. Machine learning models are trained and how machine learning models work for it actually happen. Like, I don't think the current paradigm of solving large convex optimization problems to find machine learning models is going to lead to AGI. Uh, no matter how many agents you plug those models into. Um, I don't think that I don't I don't I don't think the current paradigm of doing uh deep learning is is is where it needs to be to be able to achieve something like that. Um, but when it comes to scaring the public to be able to achieve goals, I think a lot of companies are doing a really good job at it. So that's uh that's one of my thoughts there.

• 39:37Speaker 01

  Uh, but like if it does happen, I mean, like, you know, I it just like in the other technology, I mean, this this whole like L L M sort of craze that that's been going on the last nine months. Um, if it's going to change a lot of people, the way a lot of people work. Uh, developers are already working differently, like a lot of a lot of our engineers, like they use you see they use the models daily. Uh, they're they're the little rubber duckies that, you know, don't make fun of you when you ask stupid questions. Um, and it's going to change the way a lot of people work, uh, but uh it's also going to lead to benefits. And, you know, that's just like with any technology that comes out, uh, you just have to, I guess, deal with it, uh, unfortunately, or fortunately, who knows?

• 40:14Speaker 02

  Yep, we are, we are a resilient bunch, humans. We've proven that time and time again. All right, before we wrap up, we have a little rapid fire session that we like to do with a couple of questions. And the first is, what do you think is overhyped? You talked about like data and the focus on data quality and data tooling being maybe a little under, underhyped. But what would you say is overhyped in AI right now?

• 40:39Speaker 01

  I would say one of the biggest overhyped things, and man, I'm going to get a lot of, a lot, a lot of burn from this from my friends if they listen to this podcast, is agents. Every time, every time I run around to some meetup, everyone's talking about agents. And on average, I think things don't work. I think there's a couple of very sort of interesting problems that people are solving. But like, like the definitions of agents are unclear. I heard one the other day where it was basically like a sequence of inputs and outputs to an, to an LLM, where there's no human reading the LLM outputs in between as like a definition of an agent. And I really think that there's a lot of overhyping is the actual capabilities of these systems. Cause errors, they compound, usually they don't work.

• 41:21Speaker 01

  I mean, go, go to one of these agent systems online right now and try to have it do the thing it's advertised for, like booking you an airplane ticket. I would bet you a hundred dollars, it's not going to book you an airplane ticket. But there's a lot of people doing a lot of very interesting things in the space as well. When it comes to like actually training their own models to be, be able to do a sort of like, like couple step agentic behavior. But that, that, that would be what I call out there. And I'm probably going to get a lot of hate for saying that, but I'll say it.

• 41:47Speaker 02

  Those are the most fun answers, number one, and also a great answer. I don't think we've heard that one yet, and I think there's a lot of great insights in there. Okay, second one before we wrap is, obviously, regulation is a pretty hot topic in AI right now. Like, what is your take on, on regulation and how we should be thinking about it? Will it be good for the AI ecosystem? Or is it going to slow us down?

• 42:12Speaker 01

  I mean, isn't the definition of regulation to like slow things down and make sure things happen correctly? Of course, it's going to slow things down. Um, is that necessarily bad? Uh, not, not a hundred percent, I think. Um, there's definitely ways people are using these technologies that is definitely new ways to be able to use these technologies to do a lot of harm to people. Uh, that shouldn't be allowed. Uh, for instance, like the, the marginal cost to spread propaganda to millions of people to like influence a 2024 election. Is I think at its lowest in like history, uh, because you can just get an LLM to go do that for you. Um, and that's scary. Uh, things should be like put in place to prevent, you know, situations like that from occurring in mass. Uh, but I also think a lot of the existing regulation kind of accounts for it, right?

• 42:54Speaker 01

  At the end of the day, uh, it's a computer, uh, generating an output for you. It just happens that output, you know, feels a little bit more human like than it did a year or two ago. Um, and obviously there's this whole angle on the regulation side around just like companies attempting to deepen, deepen the moats around them and prevent other. Uh, prevent sort of like new, new people from coming in, knocking the incumbents out of their space. And I think that's a big part of it as well. Um, kind of the biggest thing around regulations is something that actually we're caring a lot about, Nomic about is licensing. Um, I think like one of the issues with machine learning models is, uh, the way that software licensing works is I think incompatible with the way machine learning models get released.

• 43:36Speaker 01

  And you see organizations like Hugging Face taking a couple of first steps at this, creating their own like custom licenses for machine learning models or machine learning model data sets. Uh, but I think there needs to be a lot of work done on the regulation side, um, and kind of maybe more on the legal side than regulation side. Just around defining what it means for a machine learning model to have a license. Um, what does that actually entail in terms of uses? Uh, because I think just right now, the way it's the, the software that runs the machine learning models, which is that the license actually gets applied to and the machine learning model waits. Uh, those are just two completely different objects and that's something that we spent a lot of our time thinking about it, Nomic too.

• 44:11Speaker 00

  Amazing. Well, this has been an awesome conversation. We've covered a whole bunch of different topics and I'm sure our listeners will want to learn more about Nomic and hear more from you. What's the best way for them to do that?

• 44:22Speaker 01

  Yeah, the best way to find us, uh, nomic underscore AI on Twitter. Uh, you can go to uh, nomic.ai, uh, dot com and, uh, find us on LinkedIn as well. Um, you could also just email us. Uh, we actually respond to our emails.

• 44:33Speaker 00

  Amazing. Well, Andre, thank you so much for the time today. It was, it was a really, uh, really interesting conversation and we really appreciate it.

• 44:40Speaker 01

  Awesome. Thank you so much, Jordan Erika.

• 44:42Speaker 02

  Thanks so much, Andre.