The Data Factory: Inside the $100B Race for Post-Training Supremacy, with Labelbox CEO Manu Sharma

Manu Sharma, founder and CEO of Labelbox, explains how frontier AI training data has evolved far beyond simple labeling to sophisticated reinforcement learning environments where domain experts create "gyms" for models to develop complex skills. With every Western frontier lab now spending over a billion dollars annually on training data, the conversation traces the shift from supervised learning to reinforcement learning from verifiable rewards, particularly for coding, mathematical reasoning, and computer use. Sharma reveals how Labelbox operates as a vertically integrated data factory, conducting over 2,000 AI-powered expert interviews daily and paying top specialists more than $250,000 annually. The discussion provides essential insights into the red-hot training data market that's reshaping AI development following major deals like Meta's $15B acquisition of Scale AI. Sponsors: Oracle Cloud Infrastructure: Oracle Cloud Infrastructure (OCI) is the next-generation cloud that delivers better performance, faster speeds, and significantly lower costs, including up to 50% less for compute, 70% for storage, and 80% for networking. Run any workload, from infrastructure to AI, in a high-availability environment and try OCI for free with zero commitment at https://oracle.com/cognitive The AGNTCY: The AGNTCY is an open-source collective dedicated to building the Internet of Agents, enabling AI agents to communicate and collaborate seamlessly across frameworks. Join a community of engineers focused on high-quality multi-agent software and support the initiative at https://agntcy.org NetSuite by Oracle: NetSuite by Oracle is the AI-powered business management suite trusted by over 42,000 businesses, offering a unified platform for accounting, financial management, inventory, and HR. Gain total visibility and control to make quick decisions and automate everyday tasks—download the free ebook, Navigating Global Trade: Three Insights for Leaders, at https://netsuite.com/cognitive PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to the Cognitive Revolution. Today, my guest is Manu Sharma, founder and CEO of Labelbox, a data factory that supplies frontier training data to all of the top Western AI labs and many enterprises that are pushing the performance frontier with task-specific, fine-tuned models. This conversation really couldn't be more timely. In the wake of Meta's recent $15 billion deal with Scale AI, which of course saw Scale's former CEO Alex Wang join Meta to lead their superintelligence team, other frontier model developers have been left scrambling to secure their own training data pipelines, and the market overall is still in the process of realignment. These headline-making developments have really spotlighted just how critical

• 0:43Nathan Labenz?

  super-high-quality training data is to today's frontier capabilities, and also how difficult and expensive it can be to create. Post-training budgets are growing rapidly as labs race to imbue their models with differentiated capabilities. And as Manu explains, every Western frontier lab is now spending over $1 billion annually on frontier training data. Such tremendous investment is required because modern data work has moved far beyond the simple labeling and preference indication tasks that many are familiar with from years past. And so today, with Manu as our guide, we'll be tracing the evolution of post-training, from supervised learning from human examples, to reinforcement learning from human feedback, to the reinforcement learning from verifiable reward paradigm that's ascendant today,

• 1:27Nathan Labenz?

  and unpacking how all of that has shifted data creation work away from tools that facilitated the collection of human preferences and reasoning, and toward environments, which Manu calls gyms, where models can develop new skills, starting with coding, mathematical reasoning, and computer use through a process of trial, error, and highly automated feedback. The bottom line is that today, when you think about training data creation, you should envision not the data sweatshop of the past, but highly qualified and comfortable domain experts working to create sophisticated reinforcement learning environments and verifiers that teach frontier models to solve complex, long-horizon tasks.

• 2:06Nathan Labenz?

  A quick disclaimer before we begin. Labelbox will be sponsoring the show for at least the next month. And so this does qualify as a sponsored episode. Nevertheless, I can sincerely say that this is a conversation I'd have wanted to have anyway. Because whether you're tracking AI progress analytically or trying to achieve superhuman performance in your particular AI application, understanding the best practices for training data creation is essential. And besides, the scale and scope of Labelbox's operation is genuinely remarkable. Their most recent capital raise was $110 million in early 2022. And today, Labelbox operates as a vertically integrated data factory. They're scaling their expert network in part by conducting over 2,000 AI-powered interviews per day.

• 2:49Nathan Labenz?

  And their most in-demand experts are already earning more than $250,000 per year on the platform. I haven't yet had the chance to sign up to earn some of that money myself, but I am looking forward to the AI interview experience. And assuming Manu is right, that the system does recognize me as a qualified expert, I'll report back, subject of course to any required NDAs, on my experience as an AI trainer. Now, without further ado, I hope you enjoy this deep dive into the ongoing evolution of the red-hot training data market with Manu Sharma, founder and CEO of Labelbox. Manu Sharma, founder and CEO of Labelbox, welcome to the cognitive revolution. Thank you. So I'm excited for this conversation. Your world is in chaos.

• 3:33Nathan Labenz?

  The AI world is going through a bit of a reorganization or realignment right now. And when I say your world, I mean the world of data, data creation, human sources of data. You've been in this business with Labelbox for a number of years, and we'll have a chance to dig into all the different facets of that. But obviously, the news that has sent various forms of shockwaves, I think, through the industry in the last couple of weeks is... Zuck and Meta doing this weird deal with Scale where they're getting Alex to come over and lead the super intelligence team or something along those lines. Then, of course, we've got people leaving Scale and it seems like, generally speaking, it's chaos. So what's your report from the front line? Is this like driving a lot of opportunity for you?

• 4:17Nathan Labenz?

  Are people confused? What are the takes that you are hearing that you think are interesting? Just super interested in kind of your gonzo report from the front to get started.

• 4:26Manu Sharma?

  Yeah, so this is one of the most exciting times in AI industry and just generally across the board, right? We can see when you kind of really look into the innovations and pace of progress. I believe we are experiencing the maximum innovations we've ever seen in per day or per week kind of time period. And it's an AGI race. It's a race among a number of companies and groups. And it's really interesting to see there are like this big AI labs that are playing and pushing out awesome capabilities of base models as well as products and product experiences. But then you have a number of teams that are just emerging with some new ideas and

• 5:10Manu Sharma?

  taking a bet on alternative techniques and so forth. One thing, one thing clear underneath all of that is that these teams need essentially three components to develop the frontier AI systems. One of them, of course, being compute. We've been seeing just incredible investments in CapEx across the board. The second, obviously, is the AI talent. And so these are obviously the researchers who know all the know-how of crafting these neural networks and the architectures and how to train them and what to train on and so forth. And a third, equally important piece is actually data. And we are very much in a regime where a lot of the emphasis is going towards

• 5:56Manu Sharma?

  Post-training, so kind of like in the AI land, you know, the way these AI models have trained is you have initially a first phase where you're essentially taking all of the data from the web and from the sources where you can get access to. And you're essentially training a base model that can understand the patterns of all data generated by humans across over these, let's say, hundred years or so. And then after the base model is trained, there is this thing called post-training. And post-training is where you really emphasize some aspects of the AI models that we interact with. It really turns these AI models into systems.

• 6:36Manu Sharma?

  And within post-training, there has been a number of techniques over the last three or four years that have become emergent or very important. And with your question, it is really awesome to see like a lot of limelight thrown at this industry where like, how do companies produce this post-training data? And so I think the news about Meta is essentially emphasizing that the data is very important for building these AI models and AI capabilities. And our industry, generally speaking, has been going through a number of exciting step changes.

• 7:15Manu Sharma?

  So when we started Label Box in two thousand and eighteen, supervised learning was in the main swing, right? Everybody was really putting a lot of, creating a lot of training data, which was essentially for supervised training. So you have like humans around the world who would tag images and videos and some snippets of text. And they will essentially train these models to mimic that behavior. And in around when the transformers came out, we started to see the inflection to towards unsupervised learning, where most of the learning was happening was on this sort of like large-scale data sets. However.

• 7:55Manu Sharma?

  As we emerged through that new paradigm, then the RLHF became a prominent way to actually make these models useful in everyday to day life. And this also required yet new forms of data, very specialized data sets with expertise, because now these models have very strong base capabilities and it needed to be, it needed to learn the ways, the skills, if you will, to interact with knowledge work and knowledge workers. And so RLHF, SFT became sort of a main thing. And now we are yet in another new paradigm of reinforcement learning.

• 8:35Manu Sharma?

  So nearly every AI lab that I am aware of is really emphasizing on creating data sets for reinforcement learning. And it's like reappeared back in the days, we saw all these amazing innovations of DeepMind with AlphaGo and so forth, and they were really trained on reinforcement learning. And it reappeared now, but in a new flavor, new form. So that's where, like, you know, at a high level, that's the trend we have seen. And all of these things actually require specialized data sets. And they need a lot of that. And I think we are, I would say, still in very early days of actually making these AI systems more and more capable for everyday tasks that power the economy across the knowledge work, right?

• 9:18Manu Sharma?

  And so that's what's going on at the macro level. And these AI players who are building these large flagship AI systems or models, they are very aware of like how important it is to have access to ways to producing a whole bunch of data sets in all of these interesting specialized domains that ultimately really get into these useful business domains where the AI can learn about, let's say, what professionals, let's say, software engineers are doing in everyday to day tasks when they're building companies and software. Or if you have, let's say, domains like healthcare, insurance, all these amazing industries that

• 10:00Manu Sharma?

  we have, like, how do you really learn from like core workflows, produces data? Produce data sets to create these models. It's going, it's a generally very interesting, exciting times and it is very common to expect, let's say, in an industry where M and A's happen and certain teams make these decisions as this he fit for what the opportunities are. And Labelbox has been behind the scene, providing these data sets to most of these AI labs. And it has shifted the workloads maybe a bit, you know, maybe sometimes quite quickly. Sometimes it takes some time for teams to reorient to the new supply chains and so forth.

• 10:41Manu Sharma?

  Yeah. So I think that's the macro kind of context on what was going on. And I certainly expect even more changes in our industry over the next few years. There are new ways, there are new teams that are innovating on how do people create data sets? How can we go help these AI labs to produce these data sets? And yeah, so that's what's going on, but it is very generally very interesting and exciting times for people who are really keen in how these data sets are created and how these models are learning.

• 11:10Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 11:16Nathan Labenz?

  Let's dig in a little bit more on the kind of nature of post-training, you know, the different techniques. And then like maybe some of the subtle differences. So a year ago, maybe the prevailing wisdom was like maybe 98, 99% of compute was going to pre-training. And then this post-training, while the data was really obviously critical to make it work well, was like a small thing that you could rapidly iterate on. It seems like that has changed, but it's unclear like how much it's changed. And if everybody's changing in the same way, I'm always very confused by, are we experiencing convergence or divergence in the leading model providers offerings? There's definitely striking convergence, but there's also maybe some divergence and maybe more divergence to be expected in the future.

• 11:59Nathan Labenz?

  And I think about this anecdote from the Claude four training where they had mistakenly left out, I'm sure you've seen this, that one SIS prompt harmful data set. And then discovered behaviorally, as I understand it, that the model was following harmful system prompts. They're like, why was that? Ultimately traced it back to, well, we omitted, you know, we had a typo in our config files and this one data set that was designed to teach the model not to follow the harmful system prompts was omitted. And so that behavior was not learned as intended. And then they didn't go back and redo all post-training with the right configuration. Instead, they patched it or made an on-the-fly

• 12:40Nathan Labenz?

  adjustment later. So from that, I infer that the post-training has now become a significant enough part of the overall compute that, you know, a company like Anthropic, as committed as they, I think mostly credibly claim to be to getting this stuff right, didn't feel like they had the luxury of going back. So what can you tell us, obviously without sharing client secrets, but in a sort of abstracted way from any one strategy, how much compute would you say is now going into post-training? Is it as simple? I'm sure it's not. It used to be pre-train, then some SFT, then a little RLHF, then you're done. Or at least that was how the public understood it, the AI public. I assume now there's more interleaving, more

• 13:21Nathan Labenz?

  steps and just more complication. So yeah, tell me more about what you think that kind of post-training has evolved into.

• 13:29Manu Sharma?

  Yeah. So I'm confident that in all of the big leading labs where, including many of the hyperscalers, that they are actively researching and innovating across the sort of front of end-to-end training recipes. So I'm sure there are teams that are pursuing some new ideas on how do we make training more efficient and things like that, and write kinds of data mixtures and so forth. But you have to know that pre-training can take a long time. And so let's say if it takes a few months to actually train a very large model and so forth, you know, it is in a release cycle when these companies are shipping these products, that is like a given. Okay, it's going to take X amount of time,

• 14:12Manu Sharma?

  six months, let's say, to really get something out the door. And of course, we are seeing that some of these teams are training gigantic models as a base model. And we'll talk about it in a bit. These models are many, in many ways, used as a teacher model for student models that are actually going to ship. Now, I would say that the Very growing budgets are going into post-training. And I think the intuition that I will give here is that in many ways, these base models have very strong capabilities about understanding the patterns of data and knowledge and so forth across wide range of modalities.

• 14:52Manu Sharma?

  However, what we are finding is that these models now have to produce useful work in the form of, you know, maybe you call it agents in form of like, you know, long horizon tasks. And to solve that problem, these research teams have to use techniques to teach that capability. And so there is sufficient knowledge and capabilities in the base model, but you have to then go out to teach these models like how to do X, Y, Z tasks. But just taking a very simple example, like in software engineering, these models obviously are so awesome in terms of what they can do in coding,

• 15:36Manu Sharma?

  but there's so much still to be desired when it comes to professional software development at our scale. Like, how can these models develop an entire suite of products with minimal supervision? I would say, I think we were starting to see that, but it's not there yet. And to solve that problem, these research teams have to collect the right kinds of data sets and then do reinforcement learning on that. And so that is where a lot of focus is going across the board for these teams to test new recipes and new techniques and see if their models are getting better in those sort of capabilities. And so they have generally a team that is, or a number of teams that are exploring that frontier.

• 16:19Manu Sharma?

  Like, how do we get these models really good in all of these sort of categories? I think the two obvious macro ones is coding and math, because the fundamental way these RL systems work, in coding and math, you can verify to large extent, like, you know, did the model got the solution right or not, right? So let's take a few examples. Like in coding, let's say I come up with a PR request and did the PR actually solve the stated objective? And does my test pass in the code base, even the prior test and the new test? And in RL, essentially, the system will try

• 16:55Manu Sharma?

  Millions of attempts to to solve that problem. And if it gets right, it scores really well. And then the gradient, the weights change with the model and then they keep going to the next one. And similarly, math for logic, there's a large number of problems that can be verified by numerical answer. And so the kind of very fast progress we saw at the reasoning models and the coding capabilities in the last, I would say, nine months or so is because of that. These teams are putting a lot of emphasis on just getting that right. And it is a very valuable thing to do because coding could be part of the most lucrative knowledge work in many ways. And some teams have a belief that they can actually, if they solve that, they can actually accelerate their AI research and so forth.

• 17:40Manu Sharma?

  So it certainly seems like Anthropic believes that, many others do. And then there is new innovations or new ways to train these systems in non-verifiable reporting, where a lot of our day-to-day work is you don't really know what the right answer may be. And there may be multiple right answers. But how do you model the data where these RL systems can track their progress? And are they getting it right? And if they create a candidate solution, how can they score that candidate solution effectively? And so we have some really promising techniques right now that is underpinning a lot of these latest capabilities across the board. You see across the models.

• 18:23Manu Sharma?

  Now, I do think that at a very high level, these models seem to feel like converging in the sense like they interact and they look and feel the same from an outside. However, there is this philosophical principle where to be able to really critique a capability, you really have to be at an expert level to really understand the nuances of quality and to really kind of like then have sort of a quality judgment, like, you know, this thing, this model is really good at this thing, but not that thing. And so there is convergence, maybe at a very high level, when you look at very generic capabilities. But there is actually a lot of divergence when you go into the details. So I code quite a bit.

• 19:07Manu Sharma?

  It's awesome for me to be able to get back into coding now with AI systems. And There are many times where I am working and solving a problem, solving a kind of a very tough problem in coding and you're kind of stuck and you're iterating with these models and you are able to have an intuition. At least I get an intuition where, you know, I've hit the ceiling of this model. I'm like in a, in a trajectory where I'm not really going to get, get this model to solve this problem. And like I'm able to see that across a variety of models where I have an intuitive judgment like, you know, model Gemini could do, solve this problem, maybe because it's like very algorithmic and requires some kind of math domain to truly solve that.

• 19:52Manu Sharma?

  But maybe when it comes to really refactoring the software or architecting the distributed system, I might use Claude because it has this amazing property where it is asking a lot of questions or in an internal state and getting more knowledge from reading this file, that file, and ultimately trying to make a better kind of software decision. And from a macro, it looks like conversions, but when you actually go into the details, these models are getting optimized for very different goals, it seems like. Now, I do believe that the ambition among these AI labs is to make it generic systems that are just so great across the board.

• 20:32Manu Sharma?

  But in reality, what we are seeing is that companies are also trying to optimize experiences where their user bases are. For example, in Tropic's case, it clearly is that they're, they're very focused, they're very intentional on making, solving coding. And when you look at maybe some other models, maybe they're more optimized for consumer use cases and everyday tasks and so forth. And maybe some other labs would be focused on making a general social kind of assistant, everyday assistant. And that would have different sort of optimizations to have. So that's what, to me, feels like, what we are, we are at right now. Now, there is a very interesting question.

• 21:14Manu Sharma?

  I do think that is like, do ultimately, when these teams figure out how to make the best coding system, how to make the best researcher for scientific domains, do they, are they able to bring it all together back into a single world model? And I think that it remains to be seen.

• 21:31Nathan Labenz?

  Broadly, I guess I want to understand better, what is the role of human data today? Maybe one way to think about it would be like, if we didn't have this human data, how would things be different? How would things fail? I'll just give you a little bit of a prompt. On the math side, it seems like we've seen from things like the DeepSeek R1 zero paper that you can get a model to learn at least like highly verifiable domains like math encoding pretty well, even without the human traces, you know, that are typically used in the supervised fine tuning. But that comes with some very important downsides, like it starts to do it in strange ways

• 22:11Nathan Labenz?

  and just generally act very strangely, language switching being one, you know, example odd behavior. But I think lots more where that comes from, if we detach ourselves from human baseline and just spin the RL centrifuge more and more intensively. So I guess one value of human data is it provides this anchor that gives us some hope for some sort of alignment by default, because we kind of know where the starting point is. Another interesting juxtaposition would be, I think it's GPT 4.5 versus 0.3 mini is like the best contrast, where GPT 4.5 has really good trivia knowledge, like blows 0.3 mini out of the water on like ability to answer random questions about the world.

• 22:55Nathan Labenz?

  But 0.3 mini is way better when it comes to these sort of reasoning challenges. So we can decouple reasoning and behavior more broadly from like world knowledge. And then there's the difference again between like supervised fine tuning, where my general understanding has been like, it's really important to record the trace and you're training the model on the actual reasoning process as exemplified by the human. I've had a ton of value from that. But then in the reinforcement learning, is that reasoning trace so important? So yeah, break down kind of the roles, flavors and like major drivers of impact of human data today as opposed to, you know, if you were just doing the RL thing to the max.

• 23:36Manu Sharma?

  So the

• 23:39Manu Sharma?

  Expert data is, continues to be extremely important for all of these capabilities that we are about to see, we have seen in the past. And I think in one way, it seems like we kind of underestimate like how incredible human intelligence really is and all the world and economy that we have built and in all the things we do, you know, this is extremely special. And we are trying to slice, like in many ways, we're trying to emulate it into AI systems. In many ways, AI systems are superhuman, but on the other hand, they still cannot do many other, like everyday-to-day tasks yet. And the way I think of it is that the, let's say, in the domains like mathematics and

• 24:24Manu Sharma?

  so forth, we've taken that slice of like, okay, here's an interesting, awesome data set about the truth of how at least the logic and reasoning work in sort of purely scientific domain. And we can retrain the systems to learn from that and adjust its critique and reasoning from that source. But then with that, we've gotten these state-of-the-art systems today, and they're very incredible. But yet, when you look at, when you put them into agent, these products, they, I would say like it's a, there's a baseline established. Now, there's a reasoning behavior that is basically a foundation for all these amazing

• 25:08Manu Sharma?

  kind of, or capabilities we want to now go build. And so to build these new capabilities, let's say, take an example of a sophisticated AI system. In the future, you'd be able to simply call any company, let's say you want to change your Wi-Fi router or some, something like that, you'd, you previously would call a whole bunch of people, stay online for forty minutes and try to resolve some issue. You know, we are very soon gonna be interacting with purely AI systems and these AI systems will be agents where there's a concierge who takes your call, perhaps you do the right kind of specialized expert on how to solve that problem that you have with Wi-Fi router. And maybe the system has to now interact with the entire database to perhaps provision

• 25:53Manu Sharma?

  a replacement and things like that, right? It's a fairly sophisticated system where it's interacting with you with, with voice, but then has these capabilities to read the databases and navigate a variety of software stacks inside a company to help you as a user. That system, to really build it reliably, there is just incredible amount of edge cases that first of all exist because we are now working with tens of millions of users and use kind of ways of how people could request these things. The the base model is a starting point to develop these capabilities, but the architect of that sort of agent or that solution would need a very good data set

• 26:36Manu Sharma?

  that represents all of those kind of distribution of the use cases or the interactions that the system would have to go through. And if you were to build that system today with base off the shelf model, you're going to require a lot of orchestration on top of the base models. You would need a lot of software stack and agent frameworks and so forth, and you might still not get very reliable output. And that's just one very narrow example. And there's probably like, I mean, like this infinite number of examples and use cases like this in our economy. And all of these things are going to require data sets for these systems to be rock solid and superhuman, super reliable. And I don't see any other way to go create these data sets.

• 27:21Manu Sharma?

  You can't really invent these data sets from algorithmic synthetic approaches. You can bootstrap it, you can make the process easier by all of these techniques of compute or synthetic approaches, but you still need this quality judgment from how the authors or creators of this system want the AI system to interact. How do they want the customer service to be like, what personality it needs to have? These things are a design parameter. And these people who are going to build these systems have a choice on how the systems needs to act on their behalf of their company. And so all of this requires kind of new forms of data, whether it's like the both interactions, whether it's tool use and things like that.

• 28:04Manu Sharma?

  Now, I think there was some time period that I we saw, maybe starting last year, that a number of research attempts was to mimic reasoning traces. Like, hey, you know, let's ask expert humans, you know, all the academics, universities and so forth, to teach models about how to think about whatever their scientific domain is and break down those problems and some reasoning steps and so forth. And I think what we have found is that I think researchers are finding that it's just really hard to ask humans to really express themselves in that reasoning step by step. And in fact, like there was some research papers going back to process reward models

• 28:48Manu Sharma?

  where there was really an emphasis to teach the model how to think step by step and so forth. I think we have largely moved on to essentially the modeling the problem and a solution and how would you grade the solution? And that is the new paradigm. So the reasoning is actually an emergent property of training these models with RL. And I'm sure there are ongoing efforts to continue to make that much higher quality and you might still need some sort of traces or orders of traces and so forth. But largely where I see things going is in the kind of framework where that I described

• 29:30Manu Sharma?

  where take any problem and then solve that problem and express that in a form of a way where you know the solution. And if you don't know the solution, you at least know how the AI system could be graded on how good it is performing in that task. And everything else becomes an emergent property to achieving that goal.

• 29:52Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 29:56Nathan Labenz?

  So the shift in both data and training, I'll try to summarize back to you and tell me if I'm getting it right, is from a supervised fine tuning mode where the human is responsible for both the final answer, but also the how did you get there, spell out your reasoning, let's think step by step, et cetera. And then the model is learning to imitate that pattern of thinking in hopes that learning the pattern of thinking will get you to the right answer at a high rate of reliability. That's the previous paradigm. The new paradigm is. Maybe because we've just found it really difficult to collect that data, maybe for some

• 30:41Nathan Labenz?

  other reasons. Instead, now we're just focused on, give me a really good solution. And then let's develop a rubric for evaluating the AI solutions. And then we train with a reinforcement learning signal, presumably where the AI takes its shot. And then another greater model basically gets a, here's what the AI just did. Here's the gold standard example. Here's your rubric score, the new one with the gold standard in mind against this rubric. And then the reward signal given back is like the sum of points on the rubric or whatever. How'd I do on that? What else would you, how would you elaborate that?

• 31:18Manu Sharma?

  That's, that's, that's right, actually. And, and in many, most of the domains, you not know right solution. But what you will know from these domain experts in all of these, like that respective industry is that how would they, like, they would know what is good and great or excellent. And that quality judgment has to be somehow expressed in the data for the systems to learn. So a lot of the work is going towards now producing these kind of data sets. And you might have heard, like, they say, you've had a number of speakers who are from these AI labs and some of them are doing new, interesting work. You've talked in the previous episodes about RL informants, where it's like a, the simple thing, the way I think about it is like a gym.

• 32:02Manu Sharma?

  You, you get the model to go into this gym and practice a whole bunch of tests, like practice a whole bunch of things and it possesses a new skill. And, and so think of any domain and you want to really model that in some form of an informant, where these models can go and play and start like honing that skill. And the challenging part actually is like, how do you craft these RL informants that are representative of, you know, sometimes generalized, very generalized skills? So let's say. One way to categorize things is that games could be much more generic planning gyms. But then, let's say, coding environments or maybe other environments are perhaps more specialized on some domains and certain kind of use cases. And yeah, so that's generally the way to go about it.

• 32:47Manu Sharma?

  And now this also has a new sort of challenges, like to be able to build these systems and to make this paradigm really work. You've got to have very reliable graders, auto graders, where these are specialized models that are essentially able to align with how, let's say, an AI system will grade the progress and it will be aligned with kind of the experts and the domain experts. And so there is another work stream that goes into making sure that that is very, that is rock solid. And so, once you get that, then you can really scale up this RL training environments in the post-training routine.

• 33:26Nathan Labenz?

  One little aside, and maybe you'll have other examples too, but I've always wondered about super long context. That strikes me as one that would be just really hard to, you know, you mentioned like it's hard to get people to record their thoughts. I've personally experienced that quite a bit because I've advised a number of companies and coached people on like, okay, you don't need that much, especially if you're fine tuning for a narrow task. The dataset doesn't necessarily have to be super huge. Obviously, the broader the task and the more diversity of inputs, then the bigger the dataset needs to be. But for a lot of these narrow, very purpose-built work, defined workflows, you don't necessarily need that much data, but yet I've always had a hard time of people actually sitting down and like writing out the traces. So I've lived that and can empathize with that.

• 34:10Nathan Labenz?

  When it comes to long context, I haven't personally ever tried, but I've always wondered, you know, if you're talking in the Gemini case, right up to a million tokens. It seems very, very difficult to just drop a million tokens on someone and say like, "Okay, you're responsible for the right answer." So I've always assumed that stuff was like more programmatically done and needle in a haystack, or 10 needles in a haystack, or whatever. And then I guess that's just generalized. But maybe you have more insight. Do you have any insight into how, and especially now, like Gemini not just can find a needle in a haystack, but it has really impressive command of, in my experience, like 500,000 plus tokens.

• 34:54Nathan Labenz?

  It seems to be not just finding the anomaly, but like really grocking what is going on over these long contexts that are obviously unseen and code bases that it's never had any access to. So yeah, I don't know if you would, but any insight into how that has come online? And what role, if any, the human data creation engine has played in that process?

• 35:15Manu Sharma?

  Yeah, totally. So you're right, like when it comes to extremely large context windows, you know, how do you really verify or test? Or in other words, like create data sets that these models can express like how good they are in those long context windows. So I think there are two broad camps that I've seen. And so one of them is, of course, let's say you can model needle in a haystack programmatically, create a whole bunch of hashes and see if you can actually find a particular hash in a large number of like, you know, this noise and so forth. And so those are like, I think that's that camp of like testing kind of just more perhaps like maybe like a, at a very fundamental level, can it find the information in a vast ocean of information, data points? But then there's the other camp, which is when you test it in real world.

• 35:59Manu Sharma?

  So we're actually creating a ton of data sets that are kind of testing these capabilities of these models in long contexts. And this would include, let's say, let's take an example of financial analysts. There's a very big industry where one of the job is to Synthesize a vast amount of information like SEC filings about a company and all of the information they may have released about their company performance and things like that. And these can be, you know, 10 documents, it could be also audio transcripts of earnings calls and things like that. And the financial analyst who is really top of their game would be able to not simply just answer a question about like, hey, what was like, tell me a factual thing about,

• 36:41Manu Sharma?

  you know, where is this in 100 documents, which becomes in many ways a search problem. Rather, they would model the problem where like, okay, given all of this in different modalities, I want to accomplish a task, I want to analyze and create a kind of a model of the company so that I can forecast what maybe the company's earning potential is and things like that. And that is actually very representative of a real world task. You know, somebody may take days to go create those things. And some models, while they are very good in some, in many of these examples, largely they are, there's more room for them to actually accomplish those tasks in a reliable way across variety of domains.

• 37:25Manu Sharma?

  And so that is an example where you're really forcing the model to like, understand and reason across multiple modalities and pieces of information. And, you know, what we call it internally is multi-hop kind of capabilities. And there's some strong performance, but again, there's so much to be desired. And so that's one of the ways that these AI labs are casting this long and honing the kind of reasoning, planning capabilities across wide, large context windows. And if you really, let's say if you throw into like video and other streams of like other modalities, they probably take a lot of token space because they have to be tokenized and so forth. So, you know, in many ways.

• 38:09Manu Sharma?

  Like Jamie and I particularly is very good in, let's say, video understanding and things like that, like hey, where, in which frames something happened and so forth. But when it comes to real world applications, that is trying to kind of mimic or plug into an actual industry domain experience, because workflows, these teams still need to improve these capabilities across those vectors or domains, if you will. And so there is a way to model these things and test these capabilities. And the details is in the, devil is in the details, where like we can induce failures across all the frontier models on these kind of tasks. And that data becomes very valuable for these AI labs to then use it to hill climb.

• 38:54Nathan Labenz?

  Could you describe maybe the different modalities of interaction or of data collection? And again, I'll give you a little prompt from my own experience. This is actually, summer 2022, I spent all summer doing fine tuning of the text DaVinci 002 series of models. They never actually released, we had an early kind of preview of that product, but they never released it. What I found to work well for me, and there was no multimodality at the time, was for any given task, do 10. Then, at some point along the way, I discovered the importance of the reasoning trace, but I didn't know that at first. So I would do 10, fine tune on just 10.

• 39:39Nathan Labenz?

  Then I would have 100 queued up, I would have the model do the next 100, and then I'd basically do sort of rejection sampling or correction, and specifically emphasize the ones that it got wrong, fix those, put those back in the data set, do another 100, and basically try to get myself into some acceptable success rate through, you know, depending on the task, two to N rounds of iteration. But it was kind of a, I'd say the qualitative modes were like, initially, I'm totally on my own, I have to do the task. Later, it more is about having the AI do the task and review its outputs and find its flaws and correct. How has that evolved today?

• 40:18Manu Sharma?

  In context of fine tuning these systems and models, you mean?

• 40:23Nathan Labenz?

  Yeah, well, actually, another next question I was going to ask is around segmenting the market with respect to the hyperscaler frontier labs clearly have kind of one set of needs where they want to go out and just get all the data they can from all the different experts and be amazing at everything. And then you mentioned like financial services, and I also think like pharma, and obviously a bunch of different more specialist companies that I assume are more engaged in a fine tuning exercise. So yeah, we could add another dimension to the breakdown there as well, if that is like a fault line in how data is collected.

• 40:58Manu Sharma?

  Yeah, so I think what you shared about like this iterative creation of a small data set to perhaps for eval, and eval essentially is some sort of a holdout set from training data in a way. And that that is generally considered to be the best practice, right? And it's really surprising like how many teams or how many individuals across the industry would get that wrong, because it's sometimes the hardest part in the enterprise kind of space is to really think hard about like what the actual workflows or things that they want to go automate or build is, and how do you express that in some sort of a data set in eval? And it is emerging to be a craft in its own to kind of like really

• 41:42Manu Sharma?

  understand that trace or trajectory and express that into data sets. And generally, the best teams or the successful outcomes we see is where you're essentially starting with something and iteratively adding on to that evaluation set. And like anything else, there are known knowns and there are unknown unknowns. And so when you're in that journey, more often than not, we find that teams will uncover completely a new edge case that is never thought about. And we have to go back to the drawing board and say, what if you just do it this way or that way? So like anything else, iterative development is the cornerstone of success across the software industry, right? That's what makes us like the software

• 42:24Manu Sharma?

  teams so agile and successful in all these endeavors. I think that is so true in this new world where in rather than writing code, we are producing data sets and eval data sets for these AI systems to evaluate and train on. And fine tuning is really interesting. I don't think, I don't think that fine tuning like we used to see before is going to last that in the future. What I mean by that is like in the supervised learning paradigm, like you know, five, four, five years ago. Generally speaking, most teams would take a base model and then they would sort of specialize. They would take the data that they have, they're collecting from their sensors or their companies and they were gonna fine tune that model.

• 43:08Manu Sharma?

  And make a superhuman on that use case. I think that now in like today, twenty twenty five. The fine tuning at best is helping you make the model efficient in whatever the task you wanna do. So what I mean by that is. Maybe it is that you don't want to, maybe it's very expensive to, let's say, run a very large model. At a high frequency, you know, minutes of queries per hour on a particular use case. And you can achieve really good state of the art performance on that, but given that it is a fairly narrow task. You can actually produce the data from that large model and then distill it down to a much smaller specialized model.

• 43:49Manu Sharma?

  And you know, for that queries per minute, per hour, kind of operating parameters. There is a better, smaller model that is just gonna be better across the board. It will be achieving the same quality at a. Even like, you know, fraction of the cost. And so that to me seems like one obvious reason to do fine tuning of models nowadays. And then there is probably maybe second, I would say bucket where. Some problems truly are so unique where these base models don't have that capability and you just have to get it, have it. The only way to achieve that is through fine tuning. But if you look at like last few years. We saw companies like, I believe Google released MedPal model, which was trained on healthcare only data.

• 44:36Manu Sharma?

  And my list I was reading somewhere was that the base Gemini models at two point five outperforms a hyper specialized model that was simply trained on healthcare data. And so what does that say? It is saying that these base models, the reasoning capabilities that is learning from vast amounts of other data is actually useful to solving that domain problem. And so maybe, you know, the idea that many enterprises or many companies thought that they were, they had like, you know, a very special data sets in their companies and petabytes of data, maybe the data set actually is not that useful because it is not going to be helping you improve the reasoning capabilities of the models because you do

• 45:21Manu Sharma?

  need that broad capabilities for models to like learn from and emerge the reasoning behaviors. But perhaps fine tuning might help you to achieve that particular goal more cost effectively with the things I just said earlier. And yeah, I think we don't see as much fine tuning across the board among our customers. Now, what we do see is a lot of context engineering, right? So fine tuning, you're really training, changing the ways of the models and so forth. But there's so many problems you can actually now solve by some simple context engineering and that is emerging to be a very specialized craft in its own. So prompt engineering kind of like emerged as like, you know, you're coming up with

• 46:04Manu Sharma?

  all this way to model a problem in a prompt. But I think the most effective systems I've been tracked with today, they're actually very carefully context engineered, right? So that not only includes the prompt, but also all of the context that you might give that model or that query through retrieval mechanisms and things like that. And, you know, that is not an easy thing to do, but it is something that is tractable and becoming very effective.

• 46:32Nathan Labenz?

  Just to make sure I can pass the Turing test on that coming back. You're saying across these like high value industries, finance, pharma, whatever, the base models are so good at reasoning that you are better off. I'd be interested to hear kind of the scale of this too. Like if, you know, one of these, you mentioned the financial analyst use case. We could collect a bunch of data. We could imagine fine tuning that. OpenAI offers that on their platform. Of course, you can do it open source notably. And I had this question in mind too, like, why doesn't Anthropic offer fine tuning in any way, in anything similar to the OpenAI way? And why doesn't Google? And it sounds like your suggestion is maybe, and correct my numbers, but maybe you're going

• 47:17Nathan Labenz?

  to create 10,000 examples of what a great financial analysis looks like. And then at the time, you'll, at runtime, you'll like choose the best 10 that are relevant to this, maybe 10,000 is too much. You choose the best 10, choose the best 50, put those into context. And it's about what runtime examples you are using that is driving the end performance. But you can get better as good or perhaps even better results from that than actually fine tuning and changing model weights. Do I have that generally right?

• 47:53Manu Sharma?

  Yes, like a lot of the agents that we are seeing now in the industry does require you to have context, right? So let's just take an example of code. When you're tracking a cursor, one of the things that makes it so effective is that there is a way for base model to get all the context, like what are the coding files and what are the relevant snippets, what are the functions that are similar? And so this is one of the directory structure and all of that information is modeled and sent to the base model and for, for a quarter to achieve, accomplish the goal that they're working on. And that example, you know, you can kind of like, that is a really good framework to think about in any other domain.

• 48:37Manu Sharma?

  And so what is the task and how do you take all the right relevant information? And provide that as a context for the model to accomplish that. And you're seeing it's very effective. And the technology wave is where the base models are actually going to get better and better. And you might not need as much, maybe you won't need as like maybe verbose context, maybe the context sufficiency becomes more proof over time. But generally, like that is the arc of technology where these foundation models are racing to absorb the capabilities that the other app players are trying to develop with all this context engineering and so forth. It may take some time for these bottom base models to be just so great in everything.

• 49:19Manu Sharma?

  But meanwhile, if you want to accomplish a certain task and goal, you're really for like most effective way to do is to just context like to engineer that system. And look, there are some cases that we find to models all the time ourselves. And we do that in scenarios where we have to optimize for cost or and or where we have to, we have a very unique opinion on how we want the judgments of cases to be. So I'll give you an example. So in our industry, we have, we work with millions of domain experts around the world. And this could be, these are canonical examples, like, you know, physicists, mathematicians, Olympiad level,

• 50:02Manu Sharma?

  the software engineers and so forth, right? So across the seventy plus countries. And one of the things we do is like, how do you assess these individuals and expertise? Like it would be maybe ten years ago, if you were to approach that problem, one, you know, the teams would try to solve that by like creating tests for like, hey, you know, here's a physics test, here's a math test, and try to perform on those things. But then you can see like it's just not scalable. You have to go manually craft these things. So what we do is we actually have a AI conversational system that powered by the frontier models. And then these experts interview with us, they're interviewing with an AI.

• 50:41Manu Sharma?

  And we know about their resume, we know about their research and so forth, and they're having a conversation for thirty minutes or so about whatever domain there is. And then there's an AI system that assesses. How good the interview went? And you're essentially trying to make a judgment like, is this person going to be a great producer of data set for teaching the frontier model new capabilities? And in a base model will have its own sort of properties or characteristics of that judgment. And we don't necessarily align with that. We think that that is not good enough. And so for us to improve that judgment based on all the things we see, the outcomes of the these experts and the data sets, we have to kind of close that loop.

• 51:25Manu Sharma?

  And one of the ways we do that is by fine tuning these models because that kind of capabilities just simply don't exist, that quality judgment. And that's hopefully some examples on like how where fine tuning is helping and the kind of use cases for it. By and large, the most of the industry that I see across the enterprise space has moved towards context engineering. And we see some examples of fine tuning here, there, but it's largely context engineering across the board. And in many ways, that is really good because it is, the technology is enabling enterprises

• 52:06Manu Sharma?

  to, to now intuitively think of the most valuable workflows they want to solve and express that in some sort of like a multi-step agentic trajectories. And what perhaps the principles of like solving that in an effective way would be and build that agentic sort of architecture or system. You know, it might require retrieval, it might require multiple function tool use and calls. And in many ways, it is a software engineering task with some, with sort of a domain task mapping or domain task engineering. And that's a lot more tractable for, you know, vast amount of enterprises where they

• 52:51Manu Sharma?

  may not have the skill set to be machine learning kind of experts and train the models and go into the nitty gritty of model training and so forth. And that's, I think, one of the reasons why the AI models and the industry have taken off so quickly is because now most of the world is now kind of finding ways to implement it in a more frictionless way.

• 53:15Nathan Labenz?

  So real quick on fine tuning, since it may be a mode of attack and decline on some of these even quite highly specialized problems, what models do you see people mostly choosing to fine tune today?

• 53:29Manu Sharma?

  I would say like generally three models, Quen, Lama, Mistral, I think that's the three categories that we see in the open source world. And I don't have much visibility into the private models, but I think it usually comes down to the nature of the business in the company. And so in highly specialized industries, maybe the kind of preference for these companies is to use open source models because of the operational constraints, because they are either in air gap situation, or they want to have things on their own servers and so forth. But if you go into different industries, like, you know, let's say high tech digital natives, companies like Airbnb and Pinterest or so forth, they might be actually

• 54:13Manu Sharma?

  using the state of the art models from the cloud providers or OpenAI and so forth.

• 54:19Nathan Labenz?

  So on context engineering, then, maybe you can coach me on a task that I've been working on for a while. Every episode of the podcast, I write, and now Claude does often a very admirable first draft of my intro essay, which I'll put at the top. And the basic approach there has been pretty much constant across, I'd say, since like Claude 2 wasn't very good at doing it, but I would do the same thing. I would just take the transcript of the current podcast, and then a bunch of examples of previous intro essays that originally, you know, I was just totally writing freehand. And then I just have a very simple prompt

• 55:00Nathan Labenz?

  that's like, adopting the tone, style, voice, perspective, structure, et cetera, of the attached example essays, write a new intro essay for the attached podcast transcript.

• 55:16Nathan Labenz?

  What I've always felt a little weird about there, wondering if I could do better is, notably for all those examples, I don't have the source because I don't have enough context to provide a bunch of transcripts and their resulting essays. So it's not like I have the inputs and output pairs. Instead, I just have a bunch of outputs that I consider good and then the current input. So how would you advise me to maybe think about refining my context engineering to get better first drafts? And I do always still, for now, edit them. I've been waiting for the day when Claude just nails it. But how would you suggest I improve that approach?

• 55:57Manu Sharma?

  And where do you think currently the system is lacking? Like, you think it's not producing good quality? Or you expect it to do other things that it's not doing yet?

• 56:09Nathan Labenz?

  That's a great question. I mean, it obviously writes very nicely. It has no trouble generally following my structure. I feel like it gets me on the sort of the cadence. I have this like persona, which is genuine. It's not like an act, but where I'm both like very enthused about the AI upside and legitimately fearful of, you know, could this get out of control, et cetera? It usually does a pretty good job of picking up on that and finding that sort of balance point. I'd say the things that it doesn't do super well are, and this is where I have this lingering idea that like, maybe if I was able to give it more of the inputs for those previous outputs, it might help, but I don't have context to do that. It doesn't seem to, in some cases, like pull out what's

• 56:50Nathan Labenz?

  actually most unique and interesting about the current conversation. It sort of gives me something where it's like, yeah, that's fine. But in terms of how I would really want to tee this up and frame it for people, you know, why is this relevant now? What is most interesting about this? How does this relate to broader understanding? It's that kind of stuff where I'm often like, eh, not quite doing it. Sometimes I get better results by, often I get better results. These days I do sometimes elaborate on the prompt as well. So I'll just say, in addition to my adopting style, tone, voice, whatever, I'll say, in this case, I want to emphasize, or I thought what was most interesting about this was, try to bring to the fore, whatever. And it can follow those instructions reasonably well. I still don't typically feel like it quite.

• 57:36Nathan Labenz?

  Hits the heart of the matter a lot of the time. Obviously, that's a pretty subtle thing, right? So it's amazing that they've come this far. I'm certainly not taking it for granted. But yeah, that's maybe the best I can describe impromptu where it's not quite hitting for me.

• 57:50Manu Sharma?

  It might be that you might have to stay persistent in context engineering. Like, hey, what are the examples where it didn't get right? Or lack kind of things or gotchas and so forth. And maybe there's a better way to express that in rules and things like that. But in a use case like this, I would argue that it is a lot better for you to actually continue to context engineering and try different base models. Perhaps try different ways to acquire new information or knowledge with the tools and so forth to get a transcript, get, you know, maybe background of the people and things like that. And the reason is that every couple of weeks, you're gonna see new models and base capabilities emerge, including now that we're seeing assistance emerge, like where they have memory as a feature,

• 58:35Manu Sharma?

  things that you might have liked, not liked in that instance, it's been able to capture automatically as a, in the context in a way. And, you know, maybe things are not great right now, but it will continue to get better as the base models improve. Whereas let's say if you were to like architecturally, like fine tune something right now, you're basically freezing an investment. You have to do it right now and you have to continue doing that as the base models get better and better. And one thing for sure is that the base models are improving very fast across the reasoning and this empathetic aspects of things. Even though we saw that with OpenAI 4.5 model, it is remarkable in many aspects that are not necessarily reasoning.

• 59:16Manu Sharma?

  And so maybe that is the element that is missing, that creativity and that kind of like sheer touch of humanness and so forth. And so I think it is a macro question, like, do you really want to claim this? This is a very creative domain. Like, I think what makes your, this thing so unique is that you are editing your editor in chief in this particular workflow. And maybe you actually do want to be stay in editor in chief and you want to have just like, you know, set up correctly from the models to help you do the best work for things you, makes you very unique about it. And in these cases, maybe the outcome is not to have fully automated situation. But there will be like, there are cases where completely different domains where it's just mundane work and you just want to rather have it fully solve as something.

• 59:59Manu Sharma?

  In those scenarios, maybe finding might be a better kind of a technique to do that, solving that problem.

• 1:00:05Nathan Labenz?

  Yeah, that's interesting. I'm not looking forward to being replaced by AI, but I at least want to know when it's happening. I do expect Notebook LM and similar to compete pretty effectively with me in the not too distant future, if not already. And in asymmetric way, like I do recognize that there's a, hopefully, an appeal of a personality that you get to know over time and watch a person sort of evolve. At the same time, Notebook LM can take any topic that you want to have a podcast on at the drop of a hat, right? And I certainly can't match that. It'll be interesting to see which relative strengths and weaknesses went out. I often also think too, and I have a friend who really emphasizes this point all the time. If I just read the Claude output, I'm not sure anybody would really notice all that

• 1:00:50Nathan Labenz?

  much difference or think it's any worse. So I think there's, to some degree, I am really just precious. Like I have a sense of what is me and what is me matters to me. And it matters because I'm reading it and I'm putting my name on it and I want it to be genuine to me. But that's a bit of a different question than like, what is quality? What serves the audience best? And I can confidently say that when I edit Claude's output to make my own final thing, that it is more true to me. I cannot confidently say actually at this point that it is like truly better serving or better informing the audience. I haven't really run that test. I hope it is, but it's not entirely clear.

• 1:01:33Nathan Labenz?

  That also might fall into the category of like, do I really want to know? So far, I haven't run that experiment, but it would be an interesting experiment to literally just sometimes read the Claude essay and see if anybody comments that the thing was worse. Interesting to just compare and contrast a little bit like frontier developers, what they're doing with data and then everybody else. And you can subsegment that if you want. My general sense is like, at the frontier, they want as much data as they can get, as many domains as they can get, highest value, highest quality they can probably get. I've heard numbers of $300 an hour being paid out to physicists or biologists around the world to do this sort of stuff.

• 1:02:14Nathan Labenz?

  When it comes to that, everybody else layer, which can still be like, obviously very sophisticated companies. Obviously, they have a narrow, more narrow focus. I'm interested in what else is different. Do they, for example, primarily use like their own team members to do the data work? Or are they still interested in going outside? If you're a pharma company, my naive sense would be like, I'd want my own people to do this stuff as opposed to have you go source talent around the world. But maybe that's wrong. And then it's also interested in kind of the scale. Do they have their own people do it? Do they still want to go outside? How big of a data set do you need for something like that? How do you know? Are there like rules of thumb or guidance that you can give people? How much does that kind of cost?

• 1:02:58Nathan Labenz?

  In a way, this could be like your intro, sales pitch or like orientation of a new customer. But yeah, I want to give the lay of the land, both at that top tier, but I think I understand that better. And then especially at that kind of everybody else tier.

• 1:03:09Manu Sharma?

  Yeah, so in Frontier AI Labs, we work with almost with everybody at this point. And there is, I would say like a insatiable appetite to get access to new, novel, specialized data sets that are in the regime of kind of reinforcement learning and is helping them teach models longer horizon tasks, do that reliably, do that across the board, across a variety of domains, right? Not just including mathematics or coding, but rather all the other things as well. So that is certainly the case. It's actually getting more in demand over time. And so the budgets across the Frontier AI Labs on data is actually increasing.

• 1:03:53Manu Sharma?

  To give you a sense, like each of the Frontier AI Lab is probably spending over a billion dollars a year on data. And so that is like fundamentally increasing across the board. And these Frontier AI Labs, like they're trying all these different ways to produce these data sets. I think some teams try to hire contractors directly. And so there's an emergence of these staffing agencies that we are seeing in the world. They're all, all they do is just simply help them hire domain experts very quickly in host, like in bulk. And so it's staffing agency models have become very exciting for investors nowadays in the Silicon Valley just because like now they're actually doing things in the AI space. But more often than not,

• 1:04:34Manu Sharma?

  The AI labs do ultimately require or want really high quality data delivered to them very quickly. And to produce the data, you have to operate a data factory. As a so label boxes essentially a data factory. We are fully verticalized, so we have a very vast network of domain experts and, you know, we use awesome ways to actually screen and vet these experts, but then that is just one part of the story. You have to actually go build tools and technology to then produce these data sets. And actually, like today, the state of the art is that most of the data sets are essentially hybrid data sets where

• 1:05:14Manu Sharma?

  we have to use all of these amazing techniques with AI systems and synthetic data set approaches to infuse it with the human experts to produce these novel data sets. And to your question about, we actually are going to release a study in the coming weeks. We looked at our network and and looked at the kind of earnings across the board. Guess how, what is a yearly earning for a top quartile or like a top AI trainer? What would you guess? What they're making per year?

• 1:05:46Nathan Labenz?

  Am I assuming they're working a full time two thousand hours?

• 1:05:50Manu Sharma?

  You can assume that.

• 1:05:51Nathan Labenz?

  I mean, the three hundred an hour that I quoted seems high, but a hundred an hour doesn't seem high. So maybe I'll say if they're doing a full two thousand, that would be two hundred thousand and then I'll discount because maybe they can't get that many hours. I'll put it um, put them at one forty.

• 1:06:09Manu Sharma?

  Yeah, that's a good guess. And so our top contributors are earning well north of two hundred fifty thousand dollars a year. And that, you know, this is a power law. And so you kind of see it sort of as I'm taught and around maybe forty, fifty thousand dollars a year for other domains in other countries and so forth. But the best, most highly specialized individuals are earning. Over two fifty thousand dollars a year and we actually expect it to increase as the AI frontier goes deeper into agents and business workflows and so forth. And this is like pretty amazing to see just five years ago, the the average hourly rates for kind of the data stats was much lower than that because we were operating in a very different regime domain at the time.

• 1:06:54Manu Sharma?

  And now you have essentially these domain experts who are using maybe part time, they're working on these kind of AI training tasks and some of them are really changing their lifestyle and to be able to do this basically full time and have the freedom to do work whenever they want and so forth. So that is a state of the play with human data in frontier, in in expanding the frontier. And these are extremely complex tasks and they are not necessarily any more tasks that person would do for five minutes or so. These individuals are actually creating RL informants. And so these are a endeavors to develop debt solvers, verifiers, or these informants and or

• 1:07:38Manu Sharma?

  play in these informants to produce an instance of what a good activity would look like and so forth. Now, we actually also have a very interesting, very big business in the enterprise space. So we are one of the leaders in building these software tools and platform for producing the training data. And so we have our customers, like some of the best, the biggest pharmaceutical companies or robotic medical imaging companies and so forth. And so in those cases, they are developing models where they do have to work with their own experts. So medical coding is a really great example. It's it's something I'm, I think it will be solved with AI and I'm excited to see, you know, that their companies doing that.

• 1:08:20Manu Sharma?

  And in these scenarios, you have to have individuals who are really good experts in that domain, all right? And it is such a nuanced situation where like the insurance courts, medical courts, particularly in the United States, it's a industry in its own. And so you have to tap into those individuals from that industry or train new experts to do that task really well. And I'm sure like companies that are starting to do really well and build these AI systems, they are finding all these leverage points, like they understand that nuance of how it's done and that they're probably trying to train. A larger workforce to actually go do that in a kind of factory sense.

• 1:09:00Manu Sharma?

  And in these scenarios, these teams have to develop the entire data factory themselves, if you will. They have to build the technology and the tools. Then they have to figure out who these experts are going to be, how they're going to work in the factory to produce these data sets. So Labelbox is operating a kind of large scale data factory primarily for frontier labs. And for enterprises, what we do is we actually offer them a technology platform to do it themselves. And in those scenarios, they are using their domain experts to do it, you know, run the data factory themselves.

• 1:09:32Nathan Labenz?

  Do you have to give them a sort of Palantir style, forward deployed expert to help them develop the process? I assume that they need that help in most cases.

• 1:09:43Manu Sharma?

  In some cases, yes, in some other cases, no. It really is a fact they, it really depends on sophistication of customers. So if you're, let's say, super high tech, digital startup, AI startup, application startup, perhaps you're, you want to have full control and architect and operate the system yourself. But maybe in some other industry, you're getting more help from our company.

• 1:10:06Nathan Labenz?

  Yeah, that makes sense. One other kind of random market segment, I don't know if this really exists or not, but I just did an episode not long ago on the concept of sovereign AI. And there's a lot of different facets of what that might mean, whatever. But one big one is countries around the world with different languages, different cultures, different value systems, whatever. They might want to make an investment in whatever they can make an investment in. It's not always entirely clear what they should do to get models to be more fluent in their language, more familiar with their values, respond in more culturally sensitive and appropriate ways. Just to have more local knowledge, right?

• 1:10:51Nathan Labenz?

  Whatever the, probably all the dimensions at once they'd like to improve. I wonder if there's any business there for you and whether there is or isn't. A theory that I've had in the past is that if you are Brazil, for example, or you are maybe even India, although India is big enough that maybe they want to have their own national champion. But I've said, maybe what you should do by default is just make an investment in data collection. And then just go give that data to the frontier developers and be like, hey, you can be better in Brazil. We've done the data collection work as the Brazilian government.

• 1:11:31Nathan Labenz?

  Here's a big data dump. Please use it and then you'll be better for us in Brazil. And that way they don't have to worry about, oh my God, like, what are we going to do about data centers? And what are we going to do about frontier researchers and how are we going to afford the $100 million to compete with what Zuckerberg is reportedly offering people, although I'm a little skeptical of that number, except in the most rare cases. But yeah, is there any like sovereign AI demand coming your way? And if you were going to advise the president of Brazil or Mexico, or for that matter, India, what would you tell them they should do on the data front?

• 1:12:06Manu Sharma?

  Yeah, so I believe there is tremendous opportunity in the government to essentially, I think, what would look like a leapfrog or overhaul of entire ways of doing things to this new way of making it super efficient. And I think the different countries have a different sort of system. Like in some countries, the government is providing certain services and they are actually owning the full stack of those services themselves. And in some countries, it's more privatized model where they're essentially creating the environment and creating a spec, hey, we want private companies to be able to do XYZ things.

• 1:12:47Manu Sharma?

  And so in scenarios where like some governments are developing or have been in the business of offering services to their citizens. Which I think is most of the governments really, right? So some shape or form. There is a lot of opportunity to rethink like what would be an AI data experience here? And and it probably across by all dimensions would mean they would be able to render those services a much better experience at a fraction of cost and get out of that legacy system. And finally, we have a technology that allows them to do that. I mean, many of these governments are probably running on the legacy Fortran and Cobalt, things like that, which I believe like I was learning somewhere from Doge team

• 1:13:29Manu Sharma?

  that they uncovered this massive amount of things that are just run on legacy systems and so forth. So there's tremendous opportunities. And I think there's an opportunity for rethinking experiences in a pure digital native AI first manner. And think about India where so many citizens could benefit from basic knowledge, understanding of, okay, well, how does this things work? What are the amenities? What are the kind of services available to me and all these different towns and villages and so forth? So there is that. Now, the investments really, I would argue that the government should be focusing on the outcomes, like what the experiences they want. And then as mapping it back to how do they achieve that in a more sufficient way?

• 1:14:13Manu Sharma?

  And whether it's letting private companies to develop those solutions and implement them. And in some other areas where they are based on a particular use case, there's looking at like, how do we re-architect or develop those services? And in many cases, it will become a question of like, how do you get those three components together, compute, data and talent to go build those capabilities? And you would need certainly three of them to make them effective.

• 1:14:39Nathan Labenz?

  So no shortcuts, in other words, like you don't think they could just come to you and say, hey? We want to go like my silver, add on a silver platter for the frontier developers. Where does that break down in your mind?

• 1:14:55Manu Sharma?

  There are obviously many industries and many players in our market. They would say like, no, no, let's go invest in, in the fundamentals and we'll figure out the use cases later. But you're going to have to get your data in the right place and things like that. And I am not so sure that actually helps these teams achieve the outcomes or like the entities to achieve the outcome. Rather, let's walk backwards because now finally we have a technology that can be wielded and molded to achieving the goal. And actually, the hardest part is coming over that intention and the goal and the kind of outlines of what a great experience or service would look like. And funding that, and then once you have that clarity, all the other things are actually fairly tractable.

• 1:15:40Manu Sharma?

  Like, okay, we can help them produce the right data sets. We can help them build the end to end systems and evaluate those things and so forth. And I think more often than not, like projects fail because they didn't have that end kind of goal in sight. Rather than they were like going into a, operating in a world where they were acquiring a whole bunch of set of technologies, seeing what it can do. Rather than walking backwards from the customer experience.

• 1:16:08Nathan Labenz?

  Yeah, I think that's a great reminder and something I also, in my very ad hoc, you know, AI consulting for businesses always emphasize. What problem are you trying to solve? Because there's so many amazing new technologies that you can experiment with and get lost in and have tons of fun with and spend a lot of money on. And yet, if you don't have a clear idea of what it is you are trying to accomplish, it's probably not going to go super well for you. Sometimes people are like, we need to build a platform first and then we can build an applicant. We need to build our AI platform. And I'm often like, I would do a spike. Let's get one thing working first. We'll learn a lot of lessons that way. We'll learn what data we do and don't have. There's all sorts of problems that we can identify in the course of one spike.

• 1:16:50Nathan Labenz?

  Probably all that gets thrown away and then, you know, at some point, maybe you mature into a more robust platform. But if you try to build that without specific problems in mind, I think you do set yourself up for a lot of frustration. So I totally agree with that analysis. I do give the frontier developers a lot of credit because I think they have taken a number of steps to close these gaps over time on their own. One really notable one was when GPT four O dropped. The tokenization of non-Roman alphabet languages like got dramatically better, which brought cost down, speed, you know, a lot more to fit in context windows. So they've like definitely prioritized this to a degree, but there's still a way in which it feels like

• 1:17:31Nathan Labenz?

  if you're in India or you're Brazil, whatever, you are a second tier user because just the things are so English and so American centric in many ways that if you could just do a giant data dump, maybe, first of all, I imagine the frontier developers would take it. And then it might lead to more parity between the American experience and the sort of Brazilian or the Indian experience of using the models off the shelf. But yeah, who knows?

• 1:17:56Manu Sharma?

  I think it's really interesting because I think it will be really interesting to see how it all emerges. So before Labelbox, I worked at Planet Labs. It's a company that is now public and it has, it operates about 400 plus satellites in low orbit and scans the entire Earth every day. And I was involved in developing capabilities where we would use computer vision to extract insights from this data set. So for example, government of Brazil would be very interested in understanding what is the level of deforestation that they are seeing every year? And that there was simply no other alternative way to really understand that on a daily or monthly or weekly basis. But as a company, we could develop that insight because we would scan the entire Earth every day.

• 1:18:41Manu Sharma?

  We'd know exactly, we would basically get an image of the entire Brazil every day at a certain resolution. And we could apply these algorithms and essentially feed that as an insight to the government and to make the right decisions or policy decisions. And that's an example where, you know, there's only certain companies who can do that and share that insight. And it wouldn't make sense for Brazil to actually operate foreign satellites to just do solving that problem. And so there's a categories of capabilities or use cases where you have to get the private companies that are best in what they do.

• 1:19:17Manu Sharma?

  Bring those capabilities and then work on integration with other systems to make for the policy decisions or services that gives it to citizens. And there's some cases where the governments are very uniquely positioned to curate or cater a intended experience to their citizens. I mean, what could be an example, maybe experience with tax could be an interesting one, like, hey, can the government offer a much more intuitive way to file taxes? Or maybe there's some other, you know, for countries where there is a centralized health service, can it make it super AI driven, where citizens can query and understand or get

• 1:20:00Manu Sharma?

  basic healthcare services, conversational perhaps and so forth. And so it really becomes case by case, what are the capabilities that the government can invest in and re completely rethink and in some, in some care, other areas where it will have to really rely on a private industry to offer a bunch of these solutions. I do think that there, we seem to be going in a direction where there are going to be these foundation model companies or models that are developed privately perhaps, or maybe the open source versions, and they will be used as a base model in the sovereign context. And they're essentially then going to either develop applications on top of it, that is

• 1:20:42Manu Sharma?

  really uniquely for the sovereign use cases, or in some cases, they will have to fine tune it, or develop custom models on top of it. Because, you know, government has a scale and intent to be able to produce some specialized data, so so gather some specialized data sets that companies might not be able to do so.

• 1:21:01Nathan Labenz?

  Different topic. Why has nobody offered to pay me to install software on my computer to watch me use my computer all day? It seems like one of the obvious relative deficiencies right now of the frontier models. And like, OpenAI's operator is getting decent, certainly, but you go watch the AI Village and it's, you've got a lot of stumbling around in various nooks and crannies of UIs. It just seems like this race for data acquisition has not extended to, let's just go.

• 1:21:38Nathan Labenz?

  Watch people use their computers, record that and then bring that kind of behavioral data into the fold. I've never heard an answer to this question that has satisfied me. So what's your take on why nobody's made me that offer?

• 1:21:51Manu Sharma?

  It's coming. And I think the reason I have a very clear visibility into this kind of all, we power a lot of these computer use agents right now, the labs are building that. So, right now, I think we're seeing first versions of truly useful computer use kind of agents or computer use kind of capabilities where it's just really the first innings of it where kind of like from a movie perhaps like inspiration from her where you know, these systems are becoming an AI companion. They can listen to you, they can observe what you're seeing every day from maybe your camera on your phone. But then if you're doing work on a computer, you could just turn that on and it will know everything

• 1:22:31Manu Sharma?

  you're seeing on the screen. And I would say that these models are very poor in that capability right now. So if you were to turn on, let's say, a live AI model that can understand a lot of these kind of things on your screen and so forth, the understanding piece is really good, particularly some areas of UI or text and so forth. But when you're really doing some interesting or like domain work, they often fail. And they will fail not necessarily like in a very unintuitive ways, like they will fail maybe three minutes into the kind of session where maybe the model did not understood or had the kind of memory of what was said or what were the intended goals were in the first minute.

• 1:23:14Manu Sharma?

  And so there is a failures that are happening across the board, whether it's understanding of a screen at a time, what is it? Like, for example, most models totally get a lot of these shapes of geometry and so forth, as an example, have a very poor spatial understanding. And they also fail in the time horizon. And as part of the rollout of this AI systems, I would be not surprised at all that the freemium kind of model would be that they will be able to use the sessions and the data sets to then perhaps train and improve the models. But again, there's a lot of work happening through companies like Label Box where we're actually producing these sessions and these data sets at a very large scale

• 1:23:59Manu Sharma?

  across different languages and domain expertise. And I guess in a way you can say like the capabilities are being cooked. It's not rolled out yet, but as part of the product experiences, the companies are going to be able to take the data from the users and actually improve these systems and make it more reliable. And I'm sure like they will make a choice on what privacy and some users will want to intentionally allow their sessions to be used by the companies to improve their models.

• 1:24:27Nathan Labenz?

  Do you think I have any expertise that would allow me to be a data contributor via Label Box? I wouldn't aim for 2000 hours a year, but I wonder if I even have, because I'm kind of a highly generalist, jack of all trades sort of person. Is that profile even useful anymore?

• 1:24:47Manu Sharma?

  Absolutely. Like the way I would, there's a lot of talk in human data where we take examples of doctors, lawyers, that kind of thing. I actually think it's not that actually helpful. It's helpful to just maybe like for new audience to really understand like we're looking for this domain expertise. But actually, the way we think about it internally at Label Box is we are simply looking for individuals who have like high agency and IQ across the board. And that fundamentally correlates with being able to do very generalized tasks, even in certain domains. So you would be able to learn completely new tasks, maybe even coding or maybe in some research where you're able to do long horizon tasks far better than any of these models can do right now.

• 1:25:33Manu Sharma?

  And that means that you can actually produce the training data or signal for these models to learn from. So for sure, absolutely. I think I'll let you in.

• 1:25:42Nathan Labenz?

  All right, I'm looking forward to experiencing the AI interview as well. Because I've heard about a lot of those things getting at least experimented with. It sounds like you've scaled yours. And actually, while we're on that, any lessons from the AI interviewing at scale that you would highlight for people?

• 1:25:59Manu Sharma?

  Yeah, so we probably are, I don't know about the others, we believe we are one of the largest rolled out production AI interview on right now. We are conducting well north of 2,000 sessions or interviews a day. Right now, probably 50 people are interviewing right now with our Zara AI interview. And yeah, there's a lot of lessons. So first of all, we were very surprised how much people loved it or enjoys interacting with an AI about their experience and so forth. And in many ways, when you look at the like average rating, when we ask our reviewers, hey, how did the AI do? We go, they are like 4.6, 4.7 average satisfaction score across the board.

• 1:26:41Manu Sharma?

  And that is because A, they can do the interview at their time, whenever they feel that in the moment to have that conversation. And B, it's like incredibly patient in a sense that you're able to just dive into a variety of topics and so forth. And in many ways, it asks you, if you let's say have a PhD experience, it will ask you about your research paper, about a very nuanced thing that a human recruiter may not be able to even get to that detail in a first 30 minute session with you, right? And so people are just like love about sharing their experiences and especially if they can share the aspects of the greatness of their work in that fidelity in a very condensed manner.

• 1:27:26Manu Sharma?

  So that's one. Then the people are using it to Practice their real world skills like so we just rolled out like practice sessions of our AI interviews where simply people are using it as a way to do interview prep perhaps for other things and we rolled out with an intention like hey our goal is not to go into that sort of category or market our contributors are asked for hey this is so great I want to be able to just do practice runs for other things in my real world jobs and so forth and so that's that I also find some really clever ways people try to cheat the system I've certainly seen people literally just put an iPhone with a chat G P T best voice and have AIs talk to each other and

• 1:28:07Manu Sharma?

  things like that and so we just see this human ingenuity to gaming a system and so forth so that's just I take it in a positive way so we have to understand we have to build technology and systems to to stay ahead of that right so like if we have that like truly like system and that means we might not like we have to be really good at assessment and I did finding really like solid good players with right intent and that is a always a pursuit because the human ingenuity is just so vast and you just get surprised by ways people employ all these different tools and techniques.

• 1:28:45Nathan Labenz?

  Yeah, interesting. I'm looking forward to seeing what that experience is like and how helpful the AI cheating assistant is.

• 1:28:51Manu Sharma?

  Another thing I got very surprised about actually with AI interviews is just how good it is in not only going deeper into like your context right your resume and experiences or papers and things like that but how good it is to having conversations in all these different languages it actually is able to assess how good you are like how natively fluent you are let's say in a particular language and so forth so we are able to assess all of those things your different language skills your your domain expert skills.

• 1:29:23Manu Sharma?

  I was honestly very shocked to see how effective these are.

• 1:29:27Nathan Labenz?

  OpenAI voice for that or something else?

• 1:29:30Manu Sharma?

  We use multiple providers behind the scene and and then we have, as I told you earlier, we have part of our system is fine tuned capabilities where especially around assessing the conversation and scoring or grading these things, these are all a data engine that we have built. So the more interviews we get and the more signals we get about like how good these human experts are at producing the data, we are using that insights to improving the grading system. If that makes sense and actually the entire format of interviews and so forth.

• 1:30:03Nathan Labenz?

  Cool. I'm looking forward to checking it out. Maybe going back just to the topic we started with for a second. What is your expectation for the future of the industry? Are we enter, is this a one off where Zuck is just doing something crazy? Or should we expect this to be like the first domino of what could be a number of kind of frontier developer and data factory company pairings? What would you expect there? And you can, I understand you've got a lot of skin in the game. So not sure exactly how much you want to speculate on that, but.

• 1:30:39Manu Sharma?

  Right now, we are, one way to think about it is like we are slicing kind of aspects of human intelligence and emulating that in AI. And the question then is like, how many slices are left in the vast human knowledge? And I would argue that it is probably infinite slices out there because we don't really understand how human brain works yet very effectively. And and so far, we haven't seen systems that work like human brain yet. I think we are, I think we're getting closer and so forth. And the time in time again, across multiple paradigm shifts, we have seen from supervised learning. I mean, when I started like in my education, I used to train models with three neural networks, three neural neurons and three layers and like in math, I'm assuming link.

• 1:31:24Manu Sharma?

  And just in a decade or so, we've had multiple paradigms and time and time again, we have come back to data being very critical, essential ingredient to making these capable, capable, highly capable models. Now the frontier is where we are trying to teach these models very sophisticated knowledge work. And if you look at it like they are very capable in certain ways, but they're. Not so capable at all in actual tasks in knowledge work yet. And the question is, what will it take to go do that? And I think in the RL paradigm, we are now at least looking, how do we go create or emulate

• 1:32:05Manu Sharma?

  variety of these different domain tasks and in RL, express that in RL informants or data sets and have these models learn. So I think there's a long road ahead in terms of making these AI systems more capable, more integrated in our day to day lives, being more reliable in long horizon tasks. And all of this will require data essentially, one way or the other. And I think there's some really exciting things that are happening in synthetic data domains. I think that is going to be always a pursuit because of just how like how much effective or the potential they could be. But then you still have to have some sort of a human grounding judgment to ground the synthetic data, if that makes sense.

• 1:32:48Manu Sharma?

  Because these AI systems are to be actually useful and interact with humans at the end of the day. And so we are in a very exciting times, but also I believe the data is going to continue to be the primary mode where humans are going to supervise these AI models open and dense in the sense like they are like we want to be at the driving seat and we want to be able to command or manage millions of these AIs to accomplish a task. Managing the these millions of AIs to accomplish a task will be imparted or exchanged through some data. And Labelbox is positioned to become a continue to become a bigger and bigger data infrastructure

• 1:33:30Manu Sharma?

  provider for all of those use cases that are invented right now, but are yet to be invented in the future.

• 1:33:37Nathan Labenz?

  That's a perfect place to end, but I have one more question, which is, where does that leave you? This is maybe the hardest one of all, but if you peer into your AI crystal ball, where does that leave you on the question of a plateau around a sort of AGI level versus a prospect of a sort of takeoff from an AGI level into a superintelligence level? And I know all the arguments in all the different directions, but like some of what you've said around like, I mean, obviously just positive transfer in general and potentially this sort of. Human level coder that can be run in if it's like as good as the best human level coders and it can be run in super parallel.

• 1:34:21Nathan Labenz?

  That when I imagine that I'm like, geez, once you get to that level, you might take off pretty quick, like you might really find all these new architectural innovations at a faster rate than we have been. And the whole thing could get crazy. But then some of the other things that you've said, when I think, when I focus my attention on those, and I'm like, there's all these million slices and you gotta have data for each one. Then that leaves me feeling more like maybe we hit a plateau around the best humans. Which, by the way, I think I would prefer for safety reasons, for time to adjust. There'll be time for superintelligence, is my view. But what is your expectation? Do you think we'll have any sort of extended period of kind of human expert level, but not like fast take off from there?

• 1:35:08Manu Sharma?

  So I think in a way, both things are actually true. I think we are already in an accelerated takeoff, by all means. And we get adjusted like with every day to day, like, why are all these AIs not so great right now? Yet it hasn't done my work yet. But five years ago, you asked me, man, this is, this looks like some crazy capability that, you know, from very far future. And so in many ways, like right now, we are in a fast progress. I think the future is going to be very much more synergistic with hundreds of, think about it, like there are 30 million developers now or something like that. And I think we're probably onboarding billions of coders over the next few years with this AI assistance. And so that's really interesting and exciting that has its own properties to it. So there's that.

• 1:35:51Manu Sharma?

  The question, the fundamental question that I think, which kind of also teases out in the philosophy realm is, how can these AIs are setting quality judgment? And there's a lot of things about data, like world, like if humans are able to figure out and able to put a quality judgment. And in many, many ways or scenarios, we're not able to necessarily express it, why certain thing is so good or bad, but we know when we see it. And it's this visceral, a harder emotion that appears from that experience of reality and human mind. And like, these are the concepts explored very heavily in a books like Zenith, the art of motorcycle maintenance, which sort of brings a lot of these really interesting ideas about quality and so forth from other philosophers and so forth.

• 1:36:37Manu Sharma?

  I think the question comes down to that, like how like that quality judgment will be imparted into AIs. And so far, the way we are doing it is with humans teaching it in all these meta ways. These are ways like, you know, rubrics or grading, things like that. And I'm sure a year, two years from now will be very different techniques. But I think that's an open question. I'm very curious about that. I think a lot about that, obviously. And that would be really key for kind of a, I would say like what you described, the sort of even faster progress. If you somehow can magically like figure that out across vast knowledge of vast space of human knowledge, what makes things excellent, good, bad,

• 1:37:23Manu Sharma?

  and is going to be key for making these AI systems like really progress very quickly.

• 1:37:29Nathan Labenz?

  Yeah, taste in maybe

• 1:37:30Manu Sharma?

  That's why

• 1:37:31Nathan Labenz?

  the domain to watch. This has been excellent. Is there any other closing thought or anything we didn't touch on that you wanted to leave people with?

• 1:37:39Manu Sharma?

  No, thank you for having me. It was an honor to be here.

• 1:37:42Nathan Labenz?

  My pleasure. That's very kind. So thank you very much as well. Manu Sharma, founder and CEO of Label Box. Thank you for being part of the Cognitive Revolution.

• 1:37:50Manu Sharma?

  Thank you.

• 1:37:52Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guests and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement

• 1:38:33Nathan Labenz?

  at ai podcast dot I N G. And thank you to everyone who listens for being part of the Cognitive Revolution.