Pod Insights Unsupervised Learning with Jacob Effron Ep 68: CEO of Mercor Brendan Foody on Evals Replacing Knowledge Work, AI x Hiring Today & the Future of Data Labeling About this episode Brendan Foody is the co-founder and CEO of Mercor, a company building the infrastructure for AI-native labor markets. Mercor’s platform is already used by top AI labs to label data, evaluate human and AI candidates, and make performance-driven hiring decisions. They’re operating at the intersection of recruiting, evals, and foundation model development—helping companies shift from intuition to measurable prediction. Brendan and his team recently raised $100M and are working with some of the most advanced players in the AI ecosystem today.
Transcript Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
0:00 Jacob Effron? Brendan Foody is the co-founder and CEO of Mercor, a company building the infrastructure for AI-native labor markets. Mercor's platform is already being used to label data, screen talent, predict performance, and evaluate both human and AI candidates. It's a really interesting company at the intersection of recruiting, evals, and core to improving foundation models. Brendan's team recently raised $100 million and they're working with some of the most sophisticated companies in AI today. Our conversation today hit a lot of interesting things, including what role humans will play in labor in the future. We talked about the types of data labeling that really matter to improve models going forward. Brendan reflected on Mercor's rapid ascent and some of the key decisions he made. And we also hit on where AI does and doesn't work in the hiring process today. All in all, a really interesting conversation. I think you'll really enjoy it.
0:44 Jacob Effron? Before we get to the episode, I just have one plug. So many of you have rated the show on Spotify or Apple, and thank you so much for that. If you're enjoying this, please consider leaving a rating on the show on either platform. Ratings help us grow, which help us to continue bringing on the best guests and keep the bar really high for the conversations. Now, here's Brendan Foody.
1:03 Jacob Effron? Well, thanks so much for coming on the podcast. Really appreciate it.
1:05 Speaker 01 Yeah, thank you so much for having me on. I'm a big fan, so excited to chat.
1:09 Jacob Effron? Yeah, excited to have you here. I figured we'd start at the highest level place, which is, for our listeners, I'd love if you just contextualize, like, where are we today? What's the state of AI evaluating talent? Like, what works, what doesn't? What's going on?
1:22 Speaker 01 I'm amazed at how good it is. Like, I think that everything that a human is able to evaluate over text, the models are close to superhuman at, whether it be the transcripts of someone's interview, the assessments that they're filling out in a written way, or even the signals on their resume. And it's a fascinating dichotomy, because so little of that has actually been distributed in the economy, right? And so there's just this huge green field associated with doing that. And it's one of the things we're really excited about working on and building out.
1:54 Jacob Effron? Yeah, were there things that didn't work pre-reasoning models that, like, you know, maybe talk about the last six months, like, as these models have gotten better, what's, like, finally started to work for you guys?
2:02 Speaker 01 Yeah, I remember back in end of March of 2023 when GPT-4 came out and we were like, build our first prototype of AI interviewer and nothing worked, right? It was like the model would hallucinate every two or three questions and all of that. And so it's just been riding this incredible tailwind over time. And I think the reasoning models are just, obviously, the knowledge in the models improved a lot in sort of the first year. And then the reasoning models have made them much better at particularly handling a lot of context, figuring out what matters, what to focus on, et cetera. It's been really cool. Still, there's multimodal things that the models aren't as good at just because it
2:44 Speaker 01 historically hasn't been as much of a focus of the labs and it's a lot harder to do RL with, but we're excited about that being added soon.
2:51 Jacob Effron? Yeah, what are the milestones that like Matt, you're like, I can't wait till the model can do X or Y or?
2:56 Speaker 01 Yeah, there's a handful of things. Like there are certain things that humans are very good at, like this vibe check of whether I would enjoy working with this person, whether this person is passionate and like they seem really genuine about what they're saying, that the model, it's really hard, right? It's hard for the best humans, let alone models. And so really excited about that and building evals out for a good chunk of it. But whenever I read through like the reasoning chains of the models, I'm like, in trying to decipher things that are eval, I'm always thinking like, wow, the model seems a lot more reasonable than like our, that whatever researcher on our team was creating the eval, right? And so it's, it's really incredible how fast they've improved.
3:38 Speaker 01 And I think everyone obviously is seeing everything working in code, but, but we're just in the early innings of a lot of other domains that are taking off in an incredible way.
3:49 Jacob Effron? And obviously, it seems like a big part of what you're doing is basically, you know, coming up with evals for humans and how good they'll be at jobs. You know, obviously, we have all the people creating like AI employees now. It's like, hey, agents are going to do this, or you'll have an AI agent doing, you know, this set of tasks that an employee would do. Do you guys play into this at all?
4:04 Speaker 01 Absolutely. So, I mean, we do a huge chunk of this, maybe giving a little bit of the backstory of the company. The reason we started is that we felt like there were Incredibly talented people all around the world that weren't getting opportunities. And the primary reason is that labor markets are very fragmented. And that a candidate somewhere else in the world, maybe it's remotely in the US or another country, was only applying to a handful of jobs. The company in San Francisco is considering a fraction of a percent of people because there's this like matching problem that they're solving manually. And through applying L M's, we could solve this matching problem so that we could build this global unified labor market that every candidate applies to and every company hires from. But then we realized that there was this huge takeoff in hiring people
4:46 Speaker 01 associated with these like new knowledge work roles and evaluating L M's. And so now we hire, you know, all sorts of experts for the top AI labs that use our technology to help facilitate that, both for, you know, creating evals to evaluate our experts as well as to evaluate the models and all of these agents that you're discussing.
5:09 Speaker 02 Maybe for our listeners too, on the Mercor side, you guys obviously have a bunch of uses of AI and screening candidates going through resumes. Can you talk through some of the different use cases that you have for AI? And then what the stack looks like that you guys are building on today?
5:23 Speaker 01 Yeah, I think a good heuristic is just thinking about all the things that humans would do manually, creating evals over those and seeing how we can automate them. So similar to how a human would review resume, conduct an interview, and then rank people or decide who should be hired. We automate all of those processes with L M's. And so we have evals for how accurately are we parsing the resume? How accurately are we, you know, scoring different parts of the resume? How accurately are we asking questions in an interview, evaluating that interview? And then passing that all into model context along with the references or every other kind of data that we have on a candidate to make the end prediction around how well they'll perform.
5:58 Speaker 02 Is it mostly off the shelf models? And you're kind of curating the evaluation and context around them?
6:03 Speaker 01 Yeah, there's a lot of off the shelf models for more basic things, but particularly for the like hardest problem of making the end evaluation of a candidate is where the post-training comes in.
6:14 Speaker 02 Yeah.
6:15 Speaker 01 And learning from all of the data we get from our customers of who's doing well for what reasons? How can we learn from those signals to make better predictions? Around who we should be hiring in the future.
6:25 Speaker 02 Have you learned anything about, anything surprising about those signals of something that the AI found where you thought, you know, maybe this isn't how I would have thought about it or our humans would have thought about it?
6:34 Speaker 01 Yeah, there's, there's all sorts of things. I think that one of the key benefits of AI is that it's able to just go way more in depth about like everything about a candidate and it's able to pick up on all the small details that humans sometimes miss or like, uh, you know, the, the vibe check sort of skips over because people already have their mind made up on a candidate. And so there's all sorts of like little resume signals if, um, people have demonstrated extreme interest in a particular area where they're just doing it for fun, uh, as you would anticipate. All the way to like different signals of whether someone studied abroad in a country that, uh, is where they're doing the end job. They might, they might communicate better and, and be more conducive in a work environment.
7:20 Speaker 01 And so there's, there's lots of those little things that come up and are very specific to projects and customers.
7:26 Speaker 02 Are there certain things that you see kind of will always be done by people? You were talking about the multimodal stuff, but I guess, how do you see AI and, you know, human interviewers working together versus the world where it just kind of goes all AI assessment?
7:41 Speaker 01 Like at a simplistic level, the hiring process involves assessing candidates and selling candidates. And the assessment part, I think is going to soon get so good from LLMs that it'll sort of be like foolish to think we know better, right? Like, well, people just like take the recommendation because it'll like have proof that it is performing so much better on the eval, on the end outcome that customers care about. Where humans, I think, will still continue to play a really large roles in the selling process of like this person that we're going to be working with and spending time with. Um, and I think about it as enabling human recruiters and hiring managers to spend all of their time on the candidates they want to hire, rather than all of these interviews of people that they don't end up wanting to hire.
8:24 Speaker 01 And so really, yeah, unlocking them to, you know, help people better understand the role, better understand the people that they're going to be working with and all the things that they should be. Excited about.
8:33 Jacob Effron? Yeah, I love that. Well, will people start gaming the assessment? Like, is that something that you've seen at all? I guess the LLMs are picking up on certain things. If you put in this keyword.
8:41 Speaker 01 They all decided they studied abroad.
8:43 Jacob Effron? They all studied abroad in the, you know, in the place where they're recruiting for.
8:46 Speaker 01 Yeah, yeah, it's why sometimes you have to be a little bit secretive about the signals, right? But yeah, I mean, we, we have so many things where we, we deal with this, as every large hiring process does. And so I think the key is ensuring that assessments are relatively dynamic. Either like the problem that they're working on is changed frequently, or that you're asking them super in-depth questions about a particular part of their background. Because there's so much in the way of talent assessment that becomes possible when the models are able to do immense preparation for an interview, right? Like when I'm like doing a first interview of an executive candidate, like maybe sometimes I'll have references on them, but most of the time I look at their LinkedIn profile for a couple of minutes.
9:26 Speaker 01 I have like some preliminary notes, but imagine if I could go like, listen to a podcast that they were on, right? Go read through like blog posts that they've written, all of the papers that they might have done during their PhD, and ask about those things, right? You can get way more in-depth and nuanced in a way that's very hard to game.
9:44 Jacob Effron? Obviously, you have these models that are pretty good at predicting how well these candidates will do. 20 seconds doesn't matter that that's explainable, or like these models just have like, you know, in a black box, like, yeah, this person's gonna be good and this person's not.
9:53 Speaker 01 Yeah, I think it does matter that it's explainable for sort of two reasons. First is for customers to like understand and trust those, those claims, right? Like building trust and all the reasoning chains. And then the second is obviously making sure that the models are selecting people for the right reasons, the reasons that they, they should be considering. And so it's beneficial, but I think like the end state of the economy is probably just that like, it'll be, you know, some sort of API or interaction where people want work done, or they, they need some level of human involvement and just a confidence interval on how that, well, that person will perform on the job. And there's far less of the intermediation that humans play in the process.
10:37 Jacob Effron? Yeah, it's like an interim trust milestone on the way, on the way there.
10:39 Speaker 01 Exactly.
10:40 Jacob Effron? That makes, makes a ton of sense. And then obviously, you know, today and in kind of the first, or one of the areas where you have a lot of fit on the data labeling side, there's kind of these clean feedback loops of like, you know, I imagine you could even score like how accurately, and you probably have multiple people looking at the same pieces of data. Talk about some of the challenges maybe in translating this to like maybe more vague domains of, of human work. Totally. I mean, Like venture capital. Yeah. Wait, wait 15 years and then you get your feedback loop. Yeah.
11:06 Speaker 01 One way I think about it is like, if you have 100 people that are all doing the same job, it's very easy to stack rank them versus if you have 100 people doing a very different job, right? Like founders, right? Like they're all working on something that's nuanced in one way or another. It's very difficult to like pattern match. Like what is the thing that they said or the thing that we learned that actually translated to the outcome? Cause there's just like so many confounding variables in the equation. And so I think that it's going to be like relatively easy for the like larger pipelines of roles. Like if you're hiring 20 account executives, right? Stack ranking all of them, learning from those signals. And then the models are starting to be able to learn from these, you know, much more complicated
11:48 Speaker 01 things where everyone's working on something else. Like we're doing a ranking of a bunch of the the Teal Fellows and that's, that's like a fun case, but it definitely is more challenging and relies more underlying reasoning capabilities of the models.
12:03 Jacob Effron? Maybe just talk through like, what are some of the challenges that emerge in doing that?
12:06 Speaker 01 Yeah. Well, it's basically that oftentimes there's a lot of things that aren't in model context. And so models struggle to learn from that and people like forget to add it to model context. So maybe it's like, I heard my like friend said this good thing about using this company's product, right? And like, or these things that, you know, might not be making their way in, making sure that all like references are added, all of the like interpersonal stuff that humans might pick up on. So we found that actually oftentimes just making sure the requisite data is in model context is like a majority of the problem.
12:41 Jacob Effron? Yeah. I guess in the future, maybe we're just recording every conversation with our smart glasses.
12:45 Speaker 01 Yeah, yeah.
12:46 Jacob Effron? Easy enough to feed into the.
12:47 Speaker 01 Bridgewater had it right all along.
12:48 Jacob Effron? Yeah, exactly. Exactly. Is that where we're headed? Is it just like going to be, you know, Bridgewater at scale?
12:53 Speaker 01 We'll see. I mean, I think, of course, a lot of companies will be adverse to that. And I think there will be regulatory reasons and legal reasons people don't want to do that. But I also think there's just going to be better processes for how models help get this information context. Right? Like, maybe it's AI doing an exit interview of the manager and the people on the teams to help better understand what was going on, because all the people have so many details in their head around this that we just need to get into the models for them to be able to make these superhuman predictions.
13:23 Speaker 02 Yeah, there are certainly more and more, both founders and, you know, all kinds of people that are bringing AI to their meetings. And so I think a lot of, you know, those meetings and interactions will be recorded for AI to learn from.
13:35 Speaker 01 Totally.
13:35 Speaker 02 I think that'll be interesting.
13:36 Speaker 01 I mean, you need to take our transcripts and stack rank us against each other. But only if I come up on that.
13:43 Speaker 02 What do you think of the data labeling landscape today? Like, how do you see the different players kind of differentiating from each other? It seemed like scale was really in a position to run away, but then now there's been a bunch of kind of new players in the landscape. Like, how do you, how do you think about that world?
13:57 Speaker 01 Yeah, I think like the key thing that most people don't understand in the like data annotation and evaluation landscape is just the shift in the market and how dramatically different it is from what it used to be two years ago. Because when ChatGPT came out, it was, the models like weren't that good. It was easy to trip them up. They were making mistakes left and right. Like, even like a high school student or like college undergrad could do a lot of completions or evals to help improve the models in this crowdsourcing fashion where they run these huge pipelines to get hundreds of thousands of pieces of SFT or RLHF. SFT being inputs, outputs, RLHF being choosing between a bunch of different like preference options like you would see in ChatGPT.
14:39 Speaker 01 But as the models got really good, that crowdsourcing model started to break because you needed really high-quality people that would work directly with researchers to help them understand why is this model doing well? Why is it not doing well? How can we, you know, create this really complicated data that helps to trip up the model and actually reflects like the real world things that we want to automate? And so Our platform of finding exceptional people that you would want to work with was perfectly positioned for that. And that we can hire these really high quality people super quickly. And that caused us to take off and have all of the traction working with the big labs. And I think that trend will continue and that like the companies that are stuck in these like super high volume crowdsourcing pipelines.
15:24 Speaker 01 Are probably are certainly going to see a lot of churn. And it's going to be the new players that understand the direction the market is headed. And lean into really high quality talent, underpinning it that are going to continue taking a lot of market share.
15:39 Speaker 02 Do you think there'll always be demand for, I guess, humans in the data labeling process? There is, you know, obviously more and more that can be done with these models. Or big model gets really good one task and then can train smaller model. Like, how do you see that evolving over time?
15:54 Speaker 01 Yeah, the way I think about it is that so long as there are things in the economy that humans can do that models can't do. We will need to create evals or RL environments that models can learn how to do those things. So I think there's certain domains where that's just going to get solved sooner than others, right? Like within math or even many parts of code, like you don't need that much data. It's super verifiable. The models will solve those problems. But then there's other domains that are like much, much more open ended. What makes a good founder when we're assessing them, right? Or, you know, most honestly, like a large chunk of knowledge work domains, maybe a majority of them are these like open ended problems that are really difficult to verify and understand what good looks like.
16:37 Speaker 01 And you just need to get all of that understanding that the models don't have into the models. And that's why I expect like orders of magnitude increase in the human data and evaluations market over time.
16:49 Jacob Effron? If I understand correctly, you guys, you know, clearly, I think one of the initial arbitrages you and what inspired the companies, you have these great coders that are all around the world and like, you know, they're not getting access to some of these jobs. And obviously that ended up being really important for coding data. You know, obviously you've expanded into other areas as well. Like what, you know, coding, again, it's a perfect like RL use case is probably also really perfect for evaluation. Like, what have you had to change or improve as you've like gone into some of these fuzzier domains and recruiting people in those areas?
17:16 Speaker 01 Yeah, I think that they're leaning on a lot of the heuristics of what a human would do manually is probably a good way to do it. So for example, if you want to automate being a consultant, how do you assess consultants that can help to do that, giving them a case study? Maybe that's specific to their background, maybe it's like-
17:33 Jacob Effron? Maybe a silly question, but like you guys are all probably great coders. And so I imagine you know how to evaluate coders. Like if you're starting to get a doctor on the platform, like how do you even know how to, what the heuristics are for humans?
17:43 Speaker 01 Well, I think the point you're getting at is really interesting, which is that as you start to get into domains beyond like the machine learning team's capabilities, they need to have these experts. We need to have doctors that are helping us create our like doctor assessments and our evals for what makes a good doctor as well as, you know, a bunch of other domains. And similarly, it's what the researchers need to do with all of their technology, right? Like when we were all going on Lemsys, it was super easy to, you know, look at like the high school level physics and say like what problem was right or which one was slightly better. But when it's like PhD level chemistry and the researcher doesn't have a PhD in chemistry, it's really hard to understand what's going on, to interpret these evals, to figure out how we can improve them. And so I think
18:24 Speaker 01 that that's the other big shift to your question earlier around evaluations is that both for assessing our talent as well as the way the researchers assess models, it's just going to be this much more collaborative process and working with people to help trip up the model and improve capabilities.
18:39 Jacob Effron? Yeah, I've heard you talk before about how this kind of like short-term data label contract work is like, it's kind of the perfect initial market for what you've done. And there's a massive amount of demand and it's kind of this wedge to like eventually doing just kind of like end-to-end labor markets. I'd love to just hear you riff a little bit on like, what's the sequencing of the company look like from here, you know, toward that vision?
18:56 Speaker 01 Yeah, well, I wrote our like secret master plan that goes over this a little bit. But the way I think about it is we have, like, the reason that marketplaces are generally hard to build is that they're very network effect intensive. And so the thing that makes them defensible also makes them hard to build. And so it's important right now that we're very focused on like drilling this wedge of huge amounts of demand that we have to expand the network effects, grow the marketplace and focus on that right now. But then we're also starting to see a lot of demand for hiring high volumes of contractors from our existing customers at big tech companies. They might need, you know, hundreds of data scientists or software engineers, or whatever the role is for a particular domain outside of human data, which is really the exact same kind of request.
19:38 Speaker 01 It's just a little bit more of a legacy market where you'd be going up against like the Accentures or Deloitte's of the world historically. And so leaning into that as like the second main focus and then expanding to all sorts of full-time hiring. But but one of the key things is that like the lifetime of the business, we've been doing all of these, like even the first year of the business had nothing to do with human data. It was just like hiring contractors for our friends and for ourselves, many of which became full-time employees. And so it's it's much more continuous and there's a lot of things that unify them in that we know that all companies want more candidates. They want to be able to hire them more quickly and they want confidence that they'll perform well. And so if we just measure those things and improve them over time, that'll position us for every stage of the business.
20:22 Jacob Effron? Yeah. Was there a moment that it was like obvious to you to lean into the human data side that it was just like so abundantly clear this is where to?
20:27 Speaker 01 Yeah, I remember it was while I was still in college. So I mean, the background of the business is I met my co-founders when we were 14 in high school. We were all this speech and debate team together. They were like winning all the tournaments that I wasn't as good as them, but I was building companies. And then we started hiring people internationally at the IITs in India, like we partnered with IT Kragpur's Code Club. And we were amazed that there were these smart people, as you're mentioning, that weren't getting jobs. And we felt like we could hire them to build projects. Our friends wanted to pay us to hire them. We could take a small fee. So we hustled a lot, bootstrapped that to a million dollar revenue run rate. We'd profited 80K after paying ourselves before dropping out, which I was like very proud of. But the parents still weren't satisfied with that, of course, until we had raised money.
21:09 Speaker 01 But to your question, in August of 2023, one of our customers introd us to the co-founders of XAI while they were still working out of the Tesla office. And he said, these, Merckor has these really smart engineers in India that are phenomenal like math and coding. And then like the XAI co-founder, the next day, got or one of them or two of them got on a call with me and our team and we're just like really excited. And then two days later, they had us into the Tesla office. To meet with the entire XAI co-founding team, except for Elon, it was right before one of their meetings with Elon. We were still in college, right? Like, this is insane. And we were just like, wow, like, why do they want what we've built so badly?
21:50 Speaker 01 And it's because there was this change that was happening so fast in the market that no one else had realized yet, right? And now, of course, we've like scaled that up and are talking about it because we have, you know, a critical mass of the market share. But that was, that was the point. And then we, but they weren't ready for human data yet. And so it wasn't until, call it like six months later, that we started working with a lot of the frontier labs and really scaling up the business.
22:17 Jacob Effron? You could see the, see the tidal wave coming.
22:18 Speaker 01 Yeah, yeah. I think like, one thing I've realized over time in founders looking for product market fit is that people try to force things too much sometimes. It's like, you need to just look for the signs of the market where it's like, wow, there is like gold to be found and just like drill after that. And because like, if it's hard to get like an initial sale, then it's going to be hard to scale up the process. You need to rather like, look at like, what are the, like, the really strong pain points where the wealthiest companies will pay whatever it takes, right? And just like sniff those out and then lean into them.
22:52 Jacob Effron? I guess you've expanded beyond coding, like maybe to go back to the doctor example, because I'm struck by when you were describing it. One, you know, in some senses, like evaling what a good doctor is, is like, actually what you're, what you're eventually going to bring these people to the model companies. They're going to figure out like, you know, is this the reasoning process that a, that a good doctor would use? What are you actually doing when you're working with someone to do evals?
23:12 Speaker 01 Yeah, I think that one of the key things that humans are better, a lot better at right now is like, learning over time from the instructions, from the training, from all the feedback. And so, excuse me, looking for these proxies that people have demonstrated of like, they're, you know, asking the right questions about the problem. They're going about thinking about it in the right way. They have signals in their background that indicate they've been in these high performing environments where people are obviously learning significantly over time. And all of those translating to them finding ways to trip up the model and. And improve capabilities.
23:47 Jacob Effron? Do you guys use your own product today? And like, how does it get used in your own hiring process?
23:50 Speaker 01 Absolutely. We use it for every role except for executive roles. I mean, we still have the listing for our executive roles, but most of our executives, like I would take the first interview rather than sending them straight to the AI interview for the selling reason, not the vetting reason.
24:04 Speaker 01 Yeah, I mean, it's extremely effective. In fact, we've found that in many cases, it's like the most predictive signal. I think one thing people underestimate in hiring processes is that humans have this very strong bias towards thinking that they're right in this vibes-based assessment.
24:20 Jacob Effron? And like, hiring is like the original vibe, everything.
24:24 Speaker 01 Definitely do not suffer from that. Yeah, and it's like, it's like, let's ground everything in the performance data, who's actually doing well on the job. I remember actually, like, we, so we have this role we're hiring for, strategic project leads. And we used to have a human case study before, like, the strategic project lead onsite. And onsite is like working with us for a day to see how they would do on various parts of the job and figure out who to hire. And then we switched over to fully an AI process before the onsite, and the conversion went up on the onsites. And so it's like, through using the AI interviewer, just being a lot more objective about the comparisons, having it standardized throughout, you know, everyone who's applying to the role rather than just like mixed across like three different interviewers. It was allowing us to have a lot better conversion.
25:07 Jacob Effron? What about on the eval side? Are you guys using kind of a bunch of people that you source for your own evals? You do a lot of that internally?
25:14 Speaker 01 Yeah, we use a lot of people, or we work with a lot of people from our marketplace to create our own evals. And so it's a similar process to what we go through with our customers. And of course, we still need the researchers involved with those people and understanding what are the reasons that the model's making mistakes? How can we create our air taxonomy? Have our post-training data reflect that air taxonomy? And he'll climb on the eval. But it's all the same processes and people.
25:41 Jacob Effron? Obviously, you talked a little bit about, you know, using multimodal capabilities to determine like passion and other things. It's like, what other things are you thinking about with like, you know, incorporating like video and, you know, other things that are futuristic for the platform?
25:54 Speaker 01 Yeah, one thing that I think about a lot is what role RL will have in the sort of like timeline to improve video capabilities and that RL is really good at these search problems.
26:05 Jacob Effron? Yeah.
26:06 Speaker 01 And video is just like a huge amount of tokens. That's why like models struggle with it. And so it's like, in many ways, the search problem of like, how do we look for the signal where that person was really excited about the particular thing, or if they cheated on the interviewer, like what other things where we could find in multimodal contexts. And so I think a lot about how we can effectively, you know, create the right data to get the model to pay attention to those, as well as a lot of what the frontier labs are doing to to improve those base capabilities.
26:35 Jacob Effron? I mean, obviously, it seems like, you know, even in the course of a few years, the end labeling market changed so much. As you think like, two years from now, like, where do you think this is all going? And like, do you think this is actually a part of your business? Or in two years, is it like, only the expert of the experts that are that are required?
26:50 Speaker 01 I think it's a huge part. And the reason is like, I mentioned in the beginning, we started the business because of this notion of labor aggregation, and that it feels like the way labor is allocated and the economy is wildly inefficient, and we could make that much more efficient. But a big part of that is making a bet on like, what will humans be doing in the economy in five years?
27:07 Jacob Effron? Please tell us.
27:07 Speaker 01 Right. Which is a huge question for everyone. At least, everything I'm seeing is leading me to believe that it's far more structurally efficient for humans to create evals over the things they don't yet know how to do, or sorry, models aren't able to yet do, than it is for them to like redundantly do that task all the time. And so I actually think it's highly probable that a huge chunk of knowledge work just trends towards creating evals. And it might not be the like rigid context that we have right now of like, people working on an annotation tooling. It might be much more dynamic and then talking to an interviewer about how to solve their problem. But I think that that is going to be just like a huge part of the economy. And it's one thing that I think very, like, very few
27:51 Speaker 01 people are aware of yet, because so many of them conflate it with what's happening in the SFT and RLHF market, where a lot of those data types just aren't as useful as they as they previously were, and and budgets for them are coming down.
28:03 Jacob Effron? What do you think will be the most interesting?
28:05 Speaker 02 Skills for people to develop or kind of, I don't know, if you were to advise someone that was in school, you know, what to study or focus on, where would you, where would you steer them?
28:15 Speaker 01 I would definitely optimize for like a fast rate of learning because things are changing so quickly, right? It's hard to know, like, there's so many of these things that people didn't think the models would be good at for a long, long time that they just got really good at really fast. I would say like, work with AI as much as possible. One thing I hear from people in our marketplace is that they love the fact that they just get to play around with these models all day. Like, they get to think about, like, you know, they get to spend hours thinking about a problem that the model's not going to work at, not going to be able to do, and like, what are the things the model is like, missing out on? And they say that they build a lot of valuable skills that help them to know in their workflow as a McKinsey analyst, where should they be using AI, where should they not be using AI, etc. And so I think just spending as much time with the models as possible
29:00 Speaker 01 and getting very familiar with the things that they're good at or bad in a particular domain is, is really helpful. But it's hard to say, like, be a software engineer, be a software engineer. Yeah.
29:10 Speaker 02 Yeah, it's interesting that like, you know, obviously, yeah, like your point that so many, many more of us will be spending time like, training these models and like, you know, there's almost an infinite amount of things. Obviously, there's like, hard skills that have like, right or wrong answers. But then there's so many like, just subjective things and like, maybe in the future, I don't know, we get paid to just train our own individual models for us, like.
29:27 Speaker 01 Totally, totally, yeah. I think, I think that'll be a big part of it. I, I would say one other thing is that people should focus on domains where demand is very elastic. And so an example is, I think there's demand to build a hundred or thousand times more software in the economy, right? And maybe it's not like a thousand times as many web apps, but it's like more feature iteration on existing products, better ranking algorithms, whatever it is, versus other roles where demand is probably more fixed. Like, we only need so many accounting, accountants, right? And like, so, so much of an accounting function. And so as much as we can focus on those things that there will be vastly more demand for when we're able to increase total productivity, is probably also a safe bet.
30:11 Speaker 02 Yeah, that's a great way to put it. I had a founder I was talking to the other day, and he was like, for all this talk about software engineering going away, I really could use a lot more software engineers.
30:22 Speaker 01 I know it's something I'm really excited about like if they made our software engineers 10 times more productive, we'd probably hire more software engineers, right? And so I think that there's always like interesting curves around demand and how pricing will will implicate it over time.
30:37 Jacob Effron? I mean, obviously, I imagine when you started, like, you know, there's probably a temptation like you could have built like a recruiter co-pilot or like partner like built software for staffing agencies like you've obviously gone decided to go end to end like was that obvious from the start like how did that kind of come about?
30:50 Speaker 01 I think part of the start was just shaped by I think we had a lot of benefit of just approaching the problem from first principles because we hadn't seen how it was done like we knew the problem our friends wanted to solve is they want to work with a software engineer and so we would just like handle everything associated with getting the software engineer that will perform well to be working with them. But in hindsight, I think that there is just many more businesses that will trend towards that because it doesn't make sense to build a co-pilot for a job that probably won't exist at least in nearly the same way that it does. It probably makes more sense to have this end to end process automated in a way that it's able to learn from the feedback loops and make better predictions.
31:32 Jacob Effron? Yeah, though, obviously, in your case, I think you benefited from like this data label market is actually perfect for in at a time of relatively nascent capabilities, you can do a kind of end to end, right? And I'm sure if that didn't exist, I imagine you might have had to go co-pilot for some of these other like more complex roles.
31:45 Speaker 01 I think this is absolutely right, right? Because it's like if you're hiring full time employees, then obviously definitely people want to have them on their payroll. And so I think that is one thing that we were fortunate about is that our operating model and the way that we structured a lot of the business was very conducive to what the demand and the shift we were seeing in the market was.
32:03 Jacob Effron? Initially, it sounds like you were you were helping find, you know, contractors for your friends like, did you think like, I assume at some point you were like, this is a side project and then at some point it became like the main thing like, when did you kind of, I mean, at what point was it like, yeah, I'm actually going to build this business for the next 20 years versus like, this is a cool thing I'm doing at the start of college?
32:19 Speaker 01 Well, the background is that I was always building companies in high school. I had a company that was doing pretty well, so I didn't want to go to college. I told my parents like, no, I'm not going to go to college and they did not like to hear that. And so then eventually I appeased. Them I applied to college, went to school, but I told them like I'm always I'm gonna drop out and they didn't really believe me. They figured that it was a safe bet. Once I'd agreed to go to school and then I went to school and every semester
32:43 Speaker 02 They like blocked the term teal fellow on there. Like, please don't let this out.
32:46 Speaker 01 Yeah, every semester, you know, I tell them the same thing. And then eventually I dropped out without really giving them a heads up or telling them because I was like, I've been telling them for the last two years. Right?
32:54 Speaker 02 You gave them a heads up.
32:55 Speaker 01 I gave them a heads up.
32:57 Speaker 02 The signs were there.
32:58 Speaker 01 Long heads up, right? And so I think for me, it was that like, I knew that I I just wanted to build a company. I was like passionate about building things that have impact in the world rather than sitting through classes that didn't feel very productive. And I was in many ways just finding the the right thing to to spend my time on. I think with my co founders, it was starting as a side project, you know, wanting to wanting to make sure they had the evidence to justify their parents, their decision to drop out. And it's funny, their parent or sort part of their condition for dropping out was that we'd raise money. And even though we had this business, I was doing a million dollar revenue run rate, we'd profit at 80 cap to bang ourselves, right? It was like making a lot of progress. That that wasn't sufficient. The key was that we needed to raise our seed round.
33:42 Speaker 02 That's what keeps us VCs in business.
33:44 Jacob Effron? Parents wanting some validation.
33:46 Speaker 01 It's the credibility stamp.
33:48 Speaker 02 Well, that's a good segue. You recently raised a lot of money, a hundred million dollar round. Congrats.
33:52 Speaker 01 Thank you.
33:53 Speaker 02 What does that kind of allow you to do now? Or how did you think about, you know, when was the right time to go raise more capital?
34:00 Speaker 01 I'm sure people want to throw money at you all the time. So like, how do you think about cutting out, when to cut off the spigot? Well, it's, it's also interesting. The only time we went to raise money was really our seed round where we were like, okay, we need to raise money to justify dropping out. And then our series A.
34:14 Speaker 02 For the parents.
34:15 Speaker 01 B. Exactly. Our series A and our series B were both preemptive. And so our thought process was that we wanted to keep dilution relatively low at 5% and sort of build up like a war chest so that we can invest in the product capabilities that we were talking about of like, how do we, you know, have referral incentives and all sorts of these creative consumer products that can build up the supply set of our marketplace as well as investing in more post training data. To improve our models, performance prediction capabilities. And in many ways, one of the largest blockers on our like ML team is just creating more evals and more and more RRL environments to improve our models, which happens to be very conducive to our business.
34:55 Jacob Effron? You have a kind of customer base of a lot of foundation model companies. Like, what do you think happens to that landscape over time? I mean, some people are like, you know, it will consolidate to two or three, maybe we'll see more. You know, how many different players do you think we end up with? And how do they ultimately differentiate?
35:09 Speaker 01 It's a very good question. I definitely am in the school of thought that OpenAI is and will continue to be a product company, not an API company. I think that so many of the API capabilities will get commoditized and it's really how you integrate with all the customers, context and that over time where they're able to generate a lot of pricing power. But I think that the market is going to be so large that I could see each of them leading into a given segment that they're able to absorb a lot of value in. Like, even if one of these labs were just go all in on building a hedge fund, I bet they could make a ridiculous amount of money, right? And so I, yeah, I think it's easy to like pattern match and say these companies are overvalued. But if you really approach the problem of like automating knowledge work and like what that opportunity is from first principles,
35:54 Speaker 01 it's like, it's hard to justify that these companies with such exceptional teams, making so much progress, won't be able to build really incredible businesses.
36:03 Jacob Effron? Yeah, I mean, obviously today it feels like there's been so much just like cross domain generalization that like it feels like it's trending toward like a more of a winner take all or the top take most versus like, hey, we'll have one that's really good in this place and one that's good in that. Though I guess your hedge fund examples is interesting insofar as like you could obviously, there's a lot more to build around the scaffolding of the model to make that work.
36:22 Speaker 01 Yeah, I mean, there are a lot of value, there's a lot of value to focus. I think that having a general API is probably not a great business for multiple companies. And so I think that there's, there's going to be one player in that likely, you know, one of the, one of the top two labs right now. And then there is going to be just a huge amount of customization that happens at the application layer for every vertical and every customer use case.
36:47 Jacob Effron? Yeah. And you think like for a lot of those like custom models that, that require some Sophisticated labeling.
36:53 Speaker 01 Oh, certainly. I mean, there is like so much. I mean, imagine if every trading firm could have evals over like the particular parts of their like trading analysis that were accurate conclusions versus inaccurate conclusions that translate to a trade doing well or not. And like you had one of the top post-training teams that was just focused on like, how do we optimize having the right trading analysis for sort of mid-frequency faster than our human traders are able to get to it? I think there is a huge amount of opportunity.
37:21 Jacob Effron? I'm surprised by talking to you. It feels like some trading firms' optimal strategy should just be stop trading, spend nine months just like laser focused on on post-training a model.
37:29 Speaker 01 Maybe. I actually have been sort of surprised that a lot of the trading firms are less sophisticated in post-training than one would have anticipated. I think that part of it is just the geographic separation of all of them being in New York or having a big chunk of their core teams in New York versus the labs being in San Francisco and a lot of the top researchers wanting to work on AGI rather than making money. And so, yeah, but I think that they're going to invest vast amounts in it. And there's just going to be these sort of like nine-figure, 10-figure partnerships with Frontier Labs to help customize their specific use case.
38:03 Jacob Effron? What's like the biggest unknown question you have like in AI right now that you feel like has, you're like, God, if I knew the answer to this, it would make big implications for how I'm like running the business today?
38:13 Speaker 01 I think it's what you said earlier of what humans will be doing in like five or 10 years. Or you're like, that's such a hard question to answer. And I think about that as the mission of the company in many ways. And we have all sorts of intuitions, but the world is changing very fast. I think like so many jobs are going to get automated that getting a better understanding of that and like how we can help define humans' new opportunities and the role that they play in the economy is is one of the most important things.
38:42 Jacob Effron? Yeah. Is there is there more stuff that we should be doing from like a policy perspective around this? Or like, how do you think about the role other institutions and society should play here?
38:48 Speaker 01 Absolutely. Like, I think that so many regulators have been very focused on things that actually aren't as close to like impacting American lives and that they're focused on competition with China, which Sure, it matters, but it's a lot less close to people's day to day. They're focused on safety risks, which matter, but are a lot less close to people's day to day. I think the thing that everyone's going to start freaking out about in the next two or three years is just that there's these models that are significantly better than them at their jobs. And we need to figure out how they're going to fit into the economy. And that's something we know will happen, right? It's not just this like low probability, high impact risk. And so I think that regulators need to be much more proactive around
39:29 Speaker 01 how we can plan for that future, around how we can say like expectation management for the general public and what the world will look like in a few years.
39:38 Jacob Effron? Yeah, I guess it's just hard not knowing what we're retraining people for.
39:41 Speaker 01 Yeah, it is exactly. But but I wish that there was a lot more conversation around that, right? And a lot more focus on what that next generation of jobs is going to look like and what guidance we should be giving to to everyone as they're going through school and entering the workforce.
39:56 Jacob Effron? Yeah, well, we always like to end our interviews with a quick fire round where we get your quick take on some overly broad questions that we'll stuff in at the end.
40:04 Jacob Effron? And so maybe to start, we'd love, you know, what's one thing that's overhyped and one thing that's underhyped in the AI world today?
40:10 Speaker 01 Oh, good question. I think that evals are underhyped very significantly. Even though they're hyped, I think they're still underhyped very significantly.
40:20 Jacob Effron? One of the last bastions of human capabilities.
40:23 Speaker 01 Yeah, I think that one thing that's really overhyped is like SFT to RHF data or that like bucket of legacy data. There's companies that are literally spending billions of dollars on it that don't need to be spending or need to be spending an order of magnitude less, and that'll change.
40:37 Jacob Effron? What's one thing you've changed your mind on in the AI world last year?
40:42 Speaker 01 Interesting. I think my timelines for automating software engineering have gone up significantly. Like I used to, I used to be a little bit skeptical of of hearing from researchers what their timelines are to having a really good AI software engineer that's able to write a PR that has a higher hit rate than a human. And I think now it seems clear that that's coming later this year, sometimes in the first half of next year. And that's going to be really, really cool.
41:10 Jacob Effron? Yeah. Do you think, I mean, obviously, like, it seems like with some of these AI improvements, you know, it's like if you talked about them, what they what they were two years ago, you would have said, oh, my God, it's going to change the world. And then they happened and it's like, OK, like that kind of adjusted things, but not like, do you feel like that's this like? A wow moment where like, you know, there's just mass change in employment on the software engineering side, or is it one of those things that will feel like some 10% change or 20% change?
41:32 Speaker 01 Well, I think the thing that frames it is the elasticity that we were talking about of the role and that I'm less worried about the short time horizon of engineering jobs because I think giving them tools to make them more productive will just mean we build more software, but it'll definitely change the nature of the role and that people that are product minded, people that understand how to do the things that models might not be as good at are have more of a comparative advantage in the market.
41:55 Jacob Effron? What AI startup are you most excited about besides Merkle?
41:58 Speaker 01 I'm really excited about OpenAI's coding capabilities, even though that's not a contrarian answer. I also think that there's going to be an immense amount of custom agents. And so there's, there's a company I'm friends with that's sort of like in stealth that I'm super excited about.
42:14 Jacob Effron? All right. Well, you definitely can't share it on this podcast. When we stop recording, we'll harass you for, for what that is. Obviously, you know, like you're running a hugely impactful company. You know, let's say you were getting started today. You know, you're, you're, you were just getting, beginning in building some AI app, like totally different category. What else would you think would be fun to build right now? Or like, what else would you go spend time on?
42:36 Speaker 01 I think that I would choose a certain knowledge work vertical, probably something in finance that can be automated and build custom agents in that vertical to do so.
42:50 Jacob Effron? You could build this AI trading firm.
42:51 Speaker 01 Yeah, I would probably try to choose something that I think is more positively impactful because I think that, I think that, you know, making sure that we get to the right like valuation by the morning instead of the afternoon probably doesn't move the needle in the world. But yeah, but choose something that I feel is super impactful to automate certain capabilities. But yeah, it's a cool world.
43:12 Jacob Effron? Yeah. Well, I always want to leave the last word to you. It's been a fascinating conversation. Where can folks go to learn more about you, the work you're doing at Merkle? The mic is yours. Anywhere you want to point to our listeners.
43:21 Speaker 01 Yeah, absolutely. Go to our website, merkle.com. We're hiring huge volumes of people for ourselves that are our customers, or smaller volumes for ourselves, huge volumes for our customers and have all sorts of great opportunities. That we would we would love to work with people on. Awesome. Well, thanks so much. That was.
43:38 Speaker 02 Yeah, thank you so much. That was a lot of fun.
Previous Ep 67: Max Junestrand (CEO, Legora) on Differentiating and Pricing AI Apps & How the Legal Industry Will Evolve Next Ep 69: Co-Founder of Databricks & LMArena on Current Eval Limitations, Why China is Winning Open Source and Future of AI Infrastructure Page updated October 26, 2026 · transcript and summary are machine-generated
