Harvey Co-Founder Gabe Pereyra on the Token Pricing Reckoning Coming for AI

Gabe Pereyra is the co-founder and President of Harvey. Before Harvey, he was an AI researcher at Google Brain, DeepMind, and Meta, working on deep learning at both Brain and DeepMind in 2016 and 2017 as the field was taking off. Valued at $11B, Harvey has passed $300M ARR, 960 employees, 2,000 customers, and roughly 13 trillion tokens processed this month. Harvey has raised over $1.2B to date from Sequoia, Kleiner Perkins, GV, Coatue, Elad Gil, the OpenAI Startup Fund, and GIC, with Sequoia and GIC co-leading the most recent $200M round at $11B. Niko Grupen is Harvey's Head of Applied Research. Both he and Gabe led the open-sourcing of LAB, the Legal Agent Benchmark, the first open-source benchmark for measuring AI agent performance on real-world legal tasks. LAB covers 1,200+ tasks across 24 practice areas, built with agent-led data generation reviewed by Big Law attorneys. Initial frontier-model results from OpenAI, Anthropic, and DeepMind posted today, with Claude Opus 4.7 leading the leaderboard. In this episode, Gabe and Niko sit down with Molly in Harvey's San Francisco speakeasy to break down what LAB measures, why Harvey gave the rubric to its biggest competitors, and what the early results show about long-horizon legal agents. Gabe goes deep on the token economics now hitting application-layer AI: single queries that cost $20, contract reviews that cost $20,000, and why he says Harvey is the largest embeddings consumer for some of the labs. He also covers the multi-model strategy, why open-source matters for law firm conflict risk, & the architectural shift from chat-based products to cloud agents. Niko closes with the methodology behind LAB, how Harvey generates synthetic legal data with agent-led generation and lawyer review, and what's next for the open-source legal research community. This is the second episode in Sourcery's Harvey series, following the last conversation with co-founder and CEO Winston Weinberg. 𝐄𝐏𝐈𝐒𝐎𝐃𝐄 𝐋𝐈𝐍𝐊 YouTube: ⁠https://youtu.be/NpZbum0lmlI https://youtu.be/NpZbum0lmlI 𝐒𝐏𝐎𝐍𝐒𝐎𝐑𝐒 • Brex https://brex.com/sourcery—The modern finance platform, combining the world’s smartest corporate card with integrated expense management, banking, bill pay, & travel. https://brex.com/sourcery • Turing https://turing.com/sourcery—Turing delivers top-tier talent, data, and tools to help AI labs improve model performance—and enables enterprises to turn those models into powerful, production-ready systems. https://turing.com/sourcery • VCX https://fundrise.com/vcx—VCX is the public ticker for private tech, allowing investors of all sizes to invest in venture capital. View The Portfolio at http://GetVCX.com http://getvcx.com • Deel https://www.deel.com/sourcery—Deel is the global people platform that helps startups hire, manage, pay, and equip anyone, anywhere. Trusted by more than 35,000 fast-growing companies, Deel is the people platform that just works, so teams can scale without the chaos. Visit: https://www.deel.com/sourcery • Public http://public.com/sourcery–Investing platform Public just launched Generated Assets https://public.com/generated-asset, which lets you turn any idea into an investable index with AI. With Generated Assets, you can build, backtest, refine, and invest in any thesis with AI. Gone are the days of one-size-fits-all ETFs. https://public.com/sourcery • Merge https://merge.dev—The leading provider of customer-facing integrations and agentic tools for frontier LLMs, Fortune 500 organizations, and B2B SaaS companies. Visit https://merge.dev Follow Sourcery for the latest updates! https://www.sourcery.vc https://www.sourcery.vc/ Disclosure Paid Endorsement. Brokerage services by Open to the Public Investing Inc, member FINRA & SIPC. Advisory services by Public Advisors LLC, SEC-registered adviser. Crypto trading provided by Zero Hash LLC, licensed by the NYSDFS. Generated Assets is an interactive analysis tool by Public Advisors. Output is for informational purposes only and is not an investment recommendation or advice. See disclosures at public.com/disclosures/ga. Matched funds must remain in your account for at least 5 years. Match rate and other terms are subject to change at any time.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Models are getting more expensive and they're getting better. We're just seeing this huge explosion of usage, cost, tokens, everything.

• 0:05Molly O'Shea?

  We need to talk about tokens. Winston mentioned you guys got up to 13 trillion.

• 0:10Speaker 00

  We are consuming a ton of tokens for just serving all this stuff. The big misconception right now is I don't think people realize how expensive this is going to get. And they're going to be like, what did my agent do that cost me $10 billion?

• 0:23Speaker 02

  We launched LAB, which is our legal agent benchmark. It's a benchmark for measuring the performance of agents on real-world legal tasks. We posted our first sort of initial results of how closed-source frontier models like those from OpenAI and Tropic DeepMind perform on the benchmark.

• 0:50Molly O'Shea?

  Gabe, welcome to Sorcery.

• 0:52Speaker 00

  Thanks so much for having me.

• 0:53Molly O'Shea?

  Thanks for having us here. We're in the secret speakeasy backroom.

• 0:56Speaker 00

  Exactly, yeah, secret.

• 0:58Molly O'Shea?

  So today, we're going to walk through the latest release of your benchmarks, which is such a fun topic. Winston was so excited to talk about it, too. So I guess to start, let's break it down. What are you measuring with your benchmarks and where did we get to today?

• 1:13Speaker 00

  Yeah, so the thing we open sourced was our legal agent benchmark. And I think the thing that we're super excited about is we have done a bunch of benchmarking in the past. So before Legal Agent Bench, we had Big Law Bench, which was more of a chat-based benchmark, which was kind of a QA dataset. And we shared some of that. We worked with kind of the labs and the providers. But at the time, we were a smaller company and we didn't have the bandwidth to like fully open source it and support it. And so what we wanted to do with Legal Agent Bench was fully open source this. So, you know, anyone working on agents could run their agents and kind of work with us in the community to evaluate these things.

• 1:56Speaker 00

  And the way I would think of this dataset is we tried to really mimic what the coding agent benchmarks did.

• 2:06Speaker 00

  And the way to think of the coding agent benchmarks is with things like Sui Bench, Terminal Bench, you have a GitHub repo, which is kind of all the code for a project. You have an issue, which is someone said, hey, this doesn't work. Can you fix it? And then you have a bunch of unit tests that will tell you, you know, if you write a bunch of code and run it, the unit test will tell you, you know, you broke this thing or this thing passed. And the way the agent gets scored is, did it write all this code that passed all the unit tests? And I think there's a bit of a misconception that, oh, legal subjective, so you can't do this. And I think the thing for especially big law is a lot of the work you can actually quantify. And this is what senior associates and partners are doing when their junior associates do a bunch of this work.

• 2:52Speaker 00

  And so as maybe an example of a task in the legal agent bench,

• 2:58Speaker 00

  it would be, can you do diligence? Which is you get a data room. So you're trying to buy a company and you'll get all of that company's contracts. And what the law firm needs to do is go through all the contracts and make sure that there's nothing questionable or that might blow up the deal in there. And so the way we set up this data set, like one of these tasks, is you have a data room that's all the contracts. You have a partner request, which is a very high level and set of instructions you would get from a partner if you're an associate. And usually for something like diligence, it would be, hey, we got the data room. Can you look at it? Right? And you would need to infer, I know what I need to do based on that.

• 3:40Speaker 00

  And then I think part of the like novelty here is we use both of those to create essentially legal unit tests. And so what are all the things that a partner would be looking for when they're trying to assess an associate's work product? And so you would take that data room and you would generate a diligence memo, which is what the law firm gives to their client to basically say, here's all our findings on this diligence and the things you would look for. So one section is going to analyze all the change of control provisions, which is basically, if I'm buying a company, there's going to be a change in control of the company. And many contracts have things that can be triggered. And most of them don't matter, but there can be ones where

• 4:24Speaker 00

  this important vendor contract will change in a material way and you need to flag this. And so that would be like a unit test of like, you want to check that? And then the the challenge with scaling this benchmark is that's just for diligence. And then as you scale this to every practice area that these large law firms help their clients in, you have to think about how to do this for fund formation and IP litigation and capital markets. And so a lot of what we're building is essentially this taxonomy of here's all the tasks that a law firm would do. And here's all the subtasks that associates would do and how partners or senior associates would grade that work.

• 5:04Speaker 00

  And I think what's nice is that framing is kind of the same thing you need to evaluate these agents to do these tasks.

• 5:10Molly O'Shea?

  Benchmarking makes things a lot more competitive and people love that. I mean, AI is like the most competitive world ever right now. And so I'm really curious on that standpoint, like, how are you then like, where does this, where does this lead to next?

• 5:26Speaker 00

  Yeah, so I think the interesting thing and kind of to your point is, okay, we've released this benchmark. And now we're going to have all of the labs and everyone else building models competing on it. And I think there's obviously some risk of, oh, what if some lab providers' models are the best? Then like, why do they need Harvey to do this is maybe one of the potential risks. But what we've seen is actually every different model is good at something different. And so with the initial results, we saw Anthropic's models are quite strong, but there's areas where 5.5 is better. There's some areas where open source is better. And increasingly, it's not just which model is the best. It's which model can solve the task at the lowest price point?

• 6:12Speaker 00

  Because as these models get better, there's kind of intelligence saturation where it's like, okay, for some simple task, I actually want to know that this open source model is good enough. And so a lot of the value that we're trying to provide to law firms is you can't just use one model to solve all these tasks because it's getting too expensive. So how do you think about all the tasks that your law firm does? And which model you should use from which provider to solve those tasks?

• 6:37Molly O'Shea?

  I know you've answered this before, but your research partners are your biggest competitors. OpenAI, Anthropic, Nvidia, we have Base10, there's like a handful of across categories, right? So why open source it?

• 6:53Molly O'Shea?

  Doesn't that pose a risk to you?

• 6:56Speaker 00

  Yeah, I think we we think of all of these providers as there's obviously overlap with the labs and you know Anthropic and OpenAI have done kind of like Claude for legal, Codex for legal. But I think the problem that we're solving is different. Like, I think a lot of the products that they're building are individual productivity. And the products we're trying to build are organizational productivity for law firms or large in-house departments. And so when we think about open sourcing, I think there's kind of the initial point that I made of like, most of these law firms and enterprises can't rely on a single lab provider.

• 7:36Speaker 00

  So like, there's a big risk if you're a law firm that if you just use Anthropic or you just use OpenAI, you run into conflict risk. So imagine you're using only Anthropic's models as a law firm and you want to represent OpenAI. OpenAI is not going to let you send their sensitive legal data to Anthropic's models. And so as a law firm, you need to support all the different models if you want to represent OpenAI and Google and Microsoft and all these providers. So there's the conflict risk, there's like platform risk. Like, what happens if you pick one provider and they run out of compute, their models fall behind? And so you need to kind of abstract all these things away. And so we think of the providers similar to cloud. And I think you see the same thing in cloud with like

• 8:21Speaker 00

  Snowflakes, Snowflake and Databricks and Datadog, like all these companies built on top of the cloud, but also compete with the cloud. So I think it'll look like that. And then I think the bigger point is we don't see the general legal intelligence as where the value of kind of Harvey comes from. Because most of the valuable legal data can't go into the general purpose models. And so if you think of these large law firms, the work that they're doing is like, a very sensitive internal investigation. Like mergers and acquisitions that would move the market and they can't go into public models. And so we think of our strategy as we want to open source the general stuff and work with

• 9:04Speaker 00

  all of the providers to make these models as good as possible at general legal. And then we want to build infrastructure for these law firms and enterprises that help them own their own models and build their own systems on their unique data. And that's where we think kind of our unique value comes in.

• 9:21Molly O'Shea?

  And how are you training the models? You can't pull or pool data from your clients. So how are you learning off of them?

• 9:28Speaker 00

  Yeah, so the way I would think of the big challenge and where law firms are so different than most enterprises, like at Harvey, all of the data in our company, for the most part, is ours. At a law firm, a lot of the data is all of their clients and you can't mix this. And so a lot of what we're starting to work on with our clients is we have a product called Shared Spaces, which essentially lets a law firm and their client collaborate on a legal project. And so you can imagine a world where you have a large private equity firm and their law firm, and they're doing all of their fund formation and all of their kind of M and A together in these shared workspaces.

• 10:09Speaker 00

  Where you can start training these models is how do you use that relationship to build these unique models? And so a lot of what we want to figure out is, given all of the like ethical wall constraints, regulatory constraints, client data constraints, how can we help law firms use their data in a way that respects kind of all of their client obligations? But still be able to improve their systems based on that.

• 10:32Molly O'Shea?

  So speaking on your background, you were at Google Brain, DeepMind and Meta. What were the biggest fundamental lessons that you learned from each of them that are driving the decisions you're making?

• 10:43Speaker 00

  Yeah, I think especially Google Brain and DeepMind, when I was there, this was kind of in 2016, 2017, and so right when deep learning was starting to take off. And I think it was, it was a really cool experience seeing both of those labs because they had kind of opposite strategies of how they approached research, which I think both were correct, but led to kind of different ways of executing on kind of the vision of of where they thought AI was going. And so when I was at Brain, it was very bottoms up. It was, I mean, both had some of the smartest AI researchers in the world. But Brain, the approach was, you know, let's get all these smart people, give them a bunch of compute, and then kind of let them do their own projects.

• 11:25Speaker 00

  And I think part of the outcome there is you had someone like Noam Chazir and Ashish Visswani. Kind of the rest of the folks that invented Transformers. And so that approach worked. And then DeepMind was much more top down, where Demis just had this vision of, okay, we're going to create AGI. Here's all the things that I think are required. And so they kind of had this tech tree of how they were going to do it. And then let's have all these teams working on research projects that would solve all these milestones to get us to AGI.

• 11:56Speaker 00

  And I think our approach is a bit more inspired by the DeepMind one, just because, I think because we're in a vertical, the end goal is very clear. Like, we kind of know here's all the legal work that needs to get done. And so you can think about that North Star as, you know, here's all the practice areas that these law firms do. Here's how we would build systems that help them, you know, do large parts of those with AI. And so it feels much more defined and it's it's more of an applied problem than it is kind of a pure research problem. But I think seeing that approach really inspired like how I think about these things.

• 12:34Molly O'Shea?

  How has your role technically evolved over time? I remember I was like, I was just listening to this interview or I forget what it was, but it was, they were talking about Andre Karpathia now. Like even back at Tesla, like you'd think like these roles back then were like super glamorous and stuff, but he was data labeling all day.

• 12:54Speaker 00

  Yeah.

• 12:54Molly O'Shea?

  But, you know, now he's the almighty god of like AI. So like, well, what is the biggest difference over time?

• 13:02Speaker 00

  Yeah, I I think it's definitely been interesting at Harvey where it's like when we started the company, I was

• 13:13Speaker 00

  more focused on research than we should have been at that stage of the company where I kind of had a strong sense that what is happening now was going to happen. And so I wanted to figure out, you know, how do we train models? How do we scale this? But it became very clear pretty quickly, like, okay, first, we need to build an enterprise business. In order, because, you know, we were a bit ahead of the curve of like, like when I first, when we first started the company, I was pitching people agents. And at that, like four years ago, people were like, what are you talking about? And so we quickly had to kind of like, think about what is the right product now for customers that we can make for them that's useful, get revenue and scale up the GTM motion, kind of a traditional enterprise SaaS motion.

• 13:57Speaker 00

  And Winston and I always had kind of this vision of, we're gonna need to build two companies in parallel. Like, one company is this traditional enterprise seat-based business. And then the second will be kind of the transition you're starting to see now as these things, as these models get better, things move to consumption. And I think you'll need both of these because it takes a while to educate buyers on how to buy this. Like, buying off tokens is insanely complicated. It'll probably at some point go back to something like subscription because it's hard to like forecast these things.

• 14:30Speaker 00

  But I think my role kind of shifted from when I started the company, we were trying, I was trying to do research and I was like, okay, we need to do traditional, like, build a great product, build a great enterprise business. And then now it's gone back to research where we've put all those things in place. I think a lot of the like, infrastructure has caught up, like, the agent stuff is happening now, that now I'm full-time. And a lot of it is like data labeling and how do we create good data sets? How do we work with all these partners? But I think it's kind of back to doing the things that I really enjoyed when I was at Brain, DeepMind and Meta. And so I, I think now is kind of the time where it seems like all these application layer companies, that you saw Composer 2 with Cursor, like, everyone's starting to make this stuff work. And I think that's like super exciting.

• 15:15Molly O'Shea?

  The inference layer has gotten really hot right now. What is your general sentiment about what's, what's going on? How is that gonna evolve?

• 15:21Speaker 00

  Yeah, I mean, I think it's tied to kind of what I was just saying is like, as this is starting to work in terms of open source models are catching up, I think a lot of the like, performance from the models is coming not just from, you know, doing this large pre-training, but all of this test time compute of, you know, how do you make these models better at reasoning? How do you build good kind of RL environments for them and good harnesses? And so the more that works, I think the more that kind of shifts the ability to application layer companies to do interesting things. And then, for example, for us, We work with Base10 and Fireworks and Together AI, Applied Compute, Trajectory, kind of a bunch of these companies, Ngram,

• 16:04Speaker 00

  that are doing either inference RL as a service. And I think it's, it just feels like

• 16:12Speaker 00

  the research is starting again, where there's just so many different avenues. All of these Neo Labs that we're working with are trying different ways. We're also working with the like traditional model providers, because they're also making a bunch of progress. And so I think it's just interesting as we're able to build more of these data sets. And even like, we have a bunch of customers that want us to figure out better ways to train on their data. I think kind of all of these things are starting to take off. And then I think the inference providers, you're seeing them do very well, where there's customers like us, where we do need to move some of our traffic to open source models, because it's just becoming super expensive to serve like the largest closed frontier model. And so I think it's like a mix of like, performance.

• 16:56Speaker 00

  We have a lot of customers that want to own their own models, own their own data. And like, I think that's kind of also the inference providers will serve that. And so, yeah, I think they're in a good spot.

• 17:08Molly O'Shea?

  Sorcery is brought to you by Brex, the financial stack trusted by more than 30,000 companies, including one in three venture-backed startups in the US. Nearly 40% of startups fail because they run out of cash. Brex is literally built to help founders avoid that. Unlike traditional banks that let your money sit idle, shipping away out it with fees, Brex's designs help you spend smarter and move faster. Their all-in-one solution combines checking, treasury, and FDIC protection into one powerful account. You can send and receive money globally at lightning speeds, get 20 times the standard FDIC coverage through their partner banks, and even high yield from day one. With same-day and even same-hour liquidity, access your funds anytime.

• 17:54Molly O'Shea?

  Companies like Scale AI, DoorDash, ServiceTitan, Hims, Anthropic, Flexport, Robinhood, and Plaid, trust and use Brex. Start today at brex.com/sorcery. That's B-R-E-X dot com slash sorcery. Turing is training the next generation of AI with tasks that require real expertise and real-world judgment. That's why companies like Nvidia, Anthropic, Salesforce, and Gemini partner with Turing.

• 18:23Molly O'Shea?

  Turing builds realistic reinforcement learning environments and data systems based on real operational traces, the kind of infrastructure Frontier Labs need to train superintelligence. Visit turing.com/sourcy. I guess to dig even deeper, architecture is changing really fast. I mean, you might have missed the whole SaaS wave, but starting off in AI, it was all chat-based. Now we've completely moved to the agentic era, which is like a thousand times more cost competitive. You need compute, power, memory, all these sorts of things that are reorganizing how companies are being built and, of course, the stock market, too. So how are you thinking through

• 19:06Molly O'Shea?

  the agentic layer and building that out?

• 19:09Speaker 00

  Yeah, so I think this was like a huge transition we've gone through in like the past six months, where exactly like you said, the original product we built was kind of this chat-based

• 19:22Speaker 00

  copilot for lawyers. And then we built all this product around it that let it work at a law firm on client matters and all these things. And then maybe six months ago, when these coding models started getting really good and you could do kind of things like Claude Code and Codex, and that started working really well in the command line, we started building a bunch of the infrastructure of, you know, how do you get these agents that were running in your command line and could execute all these tools and work really well? And how to move that into the cloud? And I think now you're seeing this with kind of these managed agent solutions.

• 19:57Speaker 00

  And to your point, it's like now you need sandboxes, the models use way more tokens. We have queries that like, we have simple, like assistant-type queries where you say, you know, "Draft me a document." That a single query can cost $20. We have like a review product where you can upload 100,000 contracts and ask the models to review them. And some of those can cost $20,000. And so it is just getting incredibly expensive. And I think in parallel, what we're seeing is I think even two years ago, the models were so good with coding that most programmers would just use them and be like, this is super useful. I think we're just starting to see in the past six months, that inflection for legal where

• 20:42Speaker 00

  the models can now generate entire documents and they're starting to work in a way where most lawyers who aren't using this technology just for fun, they're like, this needs to be so much better than the way that I'm used to doing things. For me to change my routine to do it, that like, we're starting to see that absorption. And so a combination of like, the models are getting more expensive and they're getting better. They're using more tokens because they're running for longer. And they work so much better that users are using them more. We're just seeing kind of this huge explosion of like usage, cost, tokens, everything.

• 21:16Molly O'Shea?

  We need to talk about tokens. I mean, there's like so many different ways to take this question because there's obviously the margin sentiment on it. And then there's also the internal usage and efficiency on tokens. And like, I think Winston mentioned you guys got up to 13 trillion. So how are you thinking through that and the cost optimization? And then we can also talk about like efficiency internally, like how do you?

• 21:43Speaker 00

  Yeah, so I think it's, there's this kind of that inflection point I was talking about where up until recently, we've always been capability constrained. And so we always wanted to use the largest model because for the most of our tasks and like, if you look on the like legal agent benchmark, even the best models weren't good enough to complete those tasks. And so, and then the usage, you know, and we weren't at a scale that the usage, you know, it wasn't something like cursor where it was so expensive to serve these things and we didn't kind of have the latency requirements.

• 22:18Speaker 00

  But that has changed in the past six months where now we are consuming like a huge number of tokens. I think for some of the labs, we are like the largest consumer of embeddings. We are consuming a ton of tokens for just serving all this stuff. And so I think it's a good problem to have in the sense of like the usage is so high. We obviously need to think through, you know, how do we serve these customers? Because if we kind of look at our usage

• 22:48Speaker 00

  like plots right now, it's like this is just the start. And it's like, there's still so much room where it's like if everyone starts using this long horizon agent stuff, the way some of our powers are you, power users are using it. There is just going to be a lot of like cost optimization you need to do. And so I think there's like a couple of things we're thinking through. I think the most obvious is just doing better and better routing of models. So like, as the largest models get better, not every task needs to be sent to the largest models. And so that's like a great use case of the legal agent benchmark. Increasingly, like serving more traffic from open source models, post-training models. Like, a lot of these like very large frontier models are large because they're good at everything.

• 23:31Speaker 00

  And so I think a lot of the opportunity for these like specific verticals will be, okay, I probably don't need a trillion parameters if I just need the system to be good at diligence. And so how do you build products and models in a way that you can get that frontier performance without kind of all of the frontier costs?

• 23:48Molly O'Shea?

  What do you think the biggest question is people are not asking?

• 23:51Speaker 00

  I feel like the probably the big like misconception right now is, I don't think people realize how expensive this is going to get. And I don't think people realize how difficult it is going to be for customers to deal with that. Like, I think when I talk with most people, they think, oh, just move to consumption pricing and that will solve all the problems. But I think there's going to be this very interesting dynamic where there's kind of a couple of ways to price these things. And I think most VCs, what they want to see is like, can you price the work? Right? Like, can you sell the value of the work you're selling? And then the, like, that's one end of the extreme. And then the other end extreme is just like priced by tokens.

• 24:35Speaker 00

  And I think the problem you run into of pricing the work is actually the same problem that law firms run into when they try to do fixed fee pricing. Right? They're trying to say, hey, here's the fixed rate of this cost. And I think there's going to be this great irony where when we started the company, one of the questions we got asked the most was, you know, what's going to happen with the billable hour? And I think most people's assumption is just, oh, everything's going to move to fixed fee. So then these law firms can protect their margins. And I think something people don't appreciate about the billable hour and why it's such a good mechanism is it lets you price incredibly complex work at massive scale in a way that the entire industry can agree on.

• 25:17Speaker 00

  Right? Because all these law firms we've talked to about pricing changes, like whenever we talk to them about fixed fee, they're just like, we just have a 10,000 clients. We can't negotiate every engagement and price this, and everyone's different. And I actually think something similar is going to happen with this token-based pricing where you've seen a bunch of headlines of like the Uber CTO where he's like, we just ran through all of our

• 25:45Speaker 00

  coding tokens for the year, right? In three months. And so all these customers are going to start getting these consumption bills of like $10 million. And they're going to be like, what did my agent do that cost me $10 billion? And if you think about how these law firms solved it, they're like, they got a bill for $10 million from their law firm. And they're like, tell me what every associate did for every six-minute increment on the entire project. And they wrote that. And I think you're going to start seeing things like this for token billing, where there's going to become this whole ecosystem of like, how do you optimize around this? Because you kind of have these weird misaligned incentives from the model providers, right? Because they're selling you consumption, like they're somewhat incentivized to have their agents use as many tokens as possible.

• 26:31Speaker 00

  And then how do you solve that? It's like, well, we can help you benchmark that and route all these and say, oh, actually for this task, you don't need to use all these tokens. And so I think there's going to be, like, I think this is going to be much more complicated than people realize. And it's the same as like, it's why it's so hard to price legal work and all of this work where it's just so hard to quantify. Like, if I think of our token usage for our programming team, it's like, what did they use all these tokens on? I'm like, they're definitely more productive, but it is very hard to like quantify these things. And so I think that will be like a big challenge.

• 27:03Molly O'Shea?

  Wow. I've not, I've not heard someone kind of break it down that it'll, I mean, obviously you guys are biased, but that is a very rational way to think about monetization for them.

• 27:16Speaker 00

  Yeah, and I, I think the same way. So I, I think it's more complicated than that in the sense of like, when most people look at law firm billable hours, they're like, oh, the incentives here are completely misaligned. This must lead to bad pricing. But I think what people don't think about is like, okay, the billable hour creates some pricing misalignment, but the fact that these law firms need to compete with every other law firm means that that's kept in check. And so it's like, this actually converges to roughly the right pricing. And so I think the same thing will happen with the model providers, where it's like, if you look at Opus 4.7 and 5.5, Opus 4.7 is three times more expensive than 5.5, but it's, you know, 10 or 20% more performant.

• 28:00Speaker 00

  But like, they, these are going to cause pressure on each other. And then if open source catches up, you're also going to have pricing pressure. And so even though there is like, like if you just had one model provider and they owned all the models, then they would just be like, here's how much tokens cost. My agent's going to use a ton of tokens. And you're kind of stuck. But I think this is exactly why you're seeing the inference providers and all the other model providers be successful because you just are going to need a lot of options. The same way that's how you deal with kind of like pricing with professional service providers. But yeah, it is a weird, like, there's a bunch of different levels of, of like the pricing there.

• 28:37Molly O'Shea?

  So how are you personally keeping track of everything that's going on? Like, what is your research diet? Has it changed over time?

• 28:45Speaker 00

  Yeah, I think it's changed a lot. I mean, I think when I first started doing research, the thing that was so exciting about deep learning is Everything was open source and everyone published everything. And so I could just go on archive for six hours a day and I would just sit there and read papers all day. And you just, you could read every breakthrough real time. And that was the thing that initially like attracted me to the field because I was like, this is so interesting. And it's so easy to talk to everyone about it because everyone's so open about it. I think that has somewhat changed now, or not somewhat, that has completely changed in the sense like, no one really publishes what they're doing anymore. And so it is harder to keep track of like the research breakthroughs and what's going on.

• 29:28Speaker 00

  I think the way that I do it now is, you know, I have, we work closely with the labs. A bunch of the people I worked with at Brain, DeepMind and Meta are at these labs. And so, you know, keeping up with everyone that way. And then I think the thing that I feel like I'm most excited about, for example, with this benchmark is, we are going to start publishing some of our research that we're doing with the labs or the other providers and open sourcing more models, more of the work we're doing. And so I think there's, hopefully, as open source catches up, there's going to be more and more of this. And yeah, I think that's kind of like the main way.

• 30:07Molly O'Shea?

  Today's episode is sponsored by V C X by Fundrise, the public ticker for private tech, allowing investors of all sizes to invest in venture capital. Learn more at getvcx.com. Some of you may not have heard this yet, but our sponsor Public just launched something called Generated Assets. And it brings A I into investing in a way I've honestly never seen before. Here's how it works. You type in an idea like A I powered supply chain companies with positive free cash flow or defense tech companies growing revenue over 25% year over year. Public's A I then dispatches a swarm of agents that scan every single U S stock, evaluates them and instantly builds a custom index around your thesis. What really stands out is how clearly it explains why each stock is included.

• 30:52Molly O'Shea?

  And before you invest, you can even back test your idea against the S and P 500. So you're making decisions with real context, not just guessing. And beyond generated assets, Public lets you invest in stocks, bonds, options. Crypto, all in one place. They'll even give you an uncapped 1% match when you transfer your investments over from another platform. If you want to build a portfolio that actually reflects your thesis, visit public.com/sorcery. Paid for by Public Investing. Full disclosures in the description. Enterprise AI runs on Merge, the AI infra platform for integrations, agent tooling, and model orchestration, so your team's ship product, not plumbing. Mistral, Dropbox, and Drata already trust Merge in production. Start building at merge.dev.

• 31:36Molly O'Shea?

  Founders scale faster on Deal. Set up payroll for any country in minutes, hire anyone, anywhere, get visas handled fast, and get back to building. Visit deal.com/sorcery. That's D E E L dot com slash sorcery. Well, speaking of performance, I have to do a little ad slot. So Brex, who I think you're a customer of, they're all about performance, spending smarter, moving faster. And so with that question I usually ask, like, maybe this is a layer deeper, like, who are the people that you most look up to that you've learned from, and what are the biggest lessons that you've taken from them?

• 32:14Speaker 00

  Yeah, that's a good question. I think Winston's definitely one of them. I think he is in terms of like, his ability to like, scale the company, keep track of everything. I think that's something that, just watching him grow with the company has been kind of super inspiring. And I think there's like, a lot of things that I learned from him in terms of, like, when I did research, I think what I was good at is how do I focus on just one thing? And I think scaling a company is being able to keep that one thing in your head, but then solve thousands of these other problems in parallel. And I think as a researcher, like, I was not good at that. I was like, good at like, okay, I'm gonna solve like,

• 32:57Speaker 00

  you know, I'm gonna try to figure out AGI, and I'm gonna ignore everything else. And like, you just can't do that when you're like, scaling a company. And watching Winston's ability to like, do that, and also kind of build a team and all these things while keeping that in mind. I think I've like, learned a lot from him doing that. Let's see, who else?

• 33:17Speaker 00

  I think my like old roommate, uh, when I was at Brain, Barrett's Off, is someone that like I've learned a lot from. He was kind of the best AI researcher that I like worked with when I was at Brain and he he's now at OpenAI. I like ran their post-training team. And I think kind of similar ability of like, just being able to keep all of the context in his head, always kind of know like the right like research directions and things like that. I think that was like another person I was a super impressed by.

• 33:45Speaker 00

  And then I think like the obvious ones of like, I think Jensen is super impressive. Like Demis, I I think a lot of these people kind of leading the like, like Satya, leading the top like labs, cloud providers, all the like large players in the AI space. I think there's like a ton you can learn from from all of these. And then, um, I think kind of the other like top application layer companies like the Cursor folks and Brett. I I think all of these folks are are kind of doing super impressive stuff.

• 34:14Molly O'Shea?

  Given learning from all of them, what are the key traits that you look for in new hires?

• 34:19Speaker 00

  Yeah, key traits in in new hires. I I think the biggest is

• 34:26Speaker 00

  like, are you just obsessed with the topic? Like, the best hires that we've made, it's usually really easy to talk to them about what we're doing because they know so much already. And so I remember like with Nico, Daniel, Spencer, like a lot of our like early hires. When I would just pitch them, here's what we're doing, it just they immediately were like, that totally makes sense. Here's a bunch of spin-off ideas and we could just like talk forever about this. And so I think that's something that I always look for of like, usually if I can have these conversations with someone where we're just building off each other. Like, that's always been a strong indicator. Like, I think.

• 35:09Speaker 00

  I think it's somewhat unique, but I think Winston and I's like hit rate on like executive hires is like insanely high. And it's usually like if we can both have a conversation like that with someone and their, you know, background, everything's a good fit. We're like, okay, this person is is gonna work out. And I think that's been like pretty accurate.

• 35:29Speaker 00

  And you can usually just feel someone who's like insanely passionate or like obsessed with a topic versus like someone who's like, doesn't know what's going on. And so I think that's kind of one of the biggest like indicators.

• 35:41Molly O'Shea?

  What are you most excited for this year?

• 35:44Speaker 00

  Training some models.

• 35:45Molly O'Shea?

  Great answer. Nika, welcome.

• 35:49Speaker 02

  Thanks for having me.

• 35:50Molly O'Shea?

  Congrats on the recent breakthroughs.

• 35:52Speaker 02

  Thank you. Yeah, we're very excited about it.

• 35:55Molly O'Shea?

  Can you talk through them?

• 35:56Speaker 02

  Yeah, absolutely. So about two weeks ago, we launched Lab, which is our legal agent benchmark. It's a benchmark for measuring the performance of agents on real world legal tasks. So we designed the benchmark to basically mirror how legal work is done at law firms. And then today, we posted our first sort of initial results of how closed source frontier models like those from OpenAI, Anthropic, DeepMind perform on the benchmark. So we're excited about what the benchmark offers, not only in terms of making a model and agent performance accessible to lawyers, but also a number of different research directions that'll spawn from here.

• 36:38Molly O'Shea?

  So something that Winston and Gabe had emphasized a lot was there was no existing data set for this type of information. So how did you guys create that? And how did you train off of it?

• 36:51Speaker 02

  Yeah, so this was actually, I think, one of the most interesting and innovative parts of the the project. So obviously, legal data is some of the most sensitive in the world. Firms can't just freely share that for something like an open source benchmark.

• 37:06Molly O'Shea?

  Crazy.

• 37:07Speaker 02

  Yeah, so we had to, we had to get creative with how we constructed the data set. And the way that we did it was actually agent led with lawyer review. So coding agents, these agentic systems are actually so good at generating synthetic data now that even for non-public documents, like certain contract types, et cetera, they can generate a pretty good first draft. Obviously, not good enough to be lawyer passing. So the way that we approached this was we have a team internally at Harvey. That we refer to as applied legal research. They're all former big law attorneys. They come from a variety of different practice areas and subdomains of law. We basically mapped out 24 practice areas that law firms have on their websites that they have lawyers,

• 37:52Speaker 02

  associates kind of assigned to within the firm. And they went through all of the sort of tasks that the firms do, or the associates at the firms do within those practice areas. And they have the priors, right? They're lawyers, they've done this work before to write out, okay, for this type of work, we use these types of documents and they look like this. And this is what a realistic deal kind of looks like. We had agents generate the data from there. And then we sent it to sort of like our larger network of lawyers to review the outputs, the rubrics, document quality, all of those sorts of things. But it ends up being a pretty like human in the loop, but scalable way to generate data.

• 38:32Molly O'Shea?

  So for someone reading this, how should they be evaluating the benchmarks?

• 38:36Speaker 02

  So I think there's two things here. One is just making agent performance legible, right? So there's kind of overall performance we have and will continue to maintain a leaderboard of just here's how this model, this agent performs on the benchmark. Obviously, higher score is better. But I don't think that these like aggregate average performance measures tell the whole story for two reasons. One, people aren't really thinking about performance in terms of just like quality maxing anymore. Like I need to get the highest.

• 39:06Molly O'Shea?

  Quality maxing? Oh, my God.

• 39:08Speaker 02

  I mean, that's been the story of like the last two years, right? It's just like, I'm going to achieve the highest quality possible at whatever cost, whatever latency, whatever burden on my users. But people are thinking about it a lot more now in terms of what is the quality I get for some amount of money spent or some amount of time spent. And so we're able to sort of dissect model performance in these sorts of ways, right? So you can see what the trade offs are in quality for a model that costs, you know, one third of the price of our leader, which is Opus 4.7 right now. Or like Gemini 3.5 Flash, it's like seven times faster at completing work than some of the frontier models. So you can look at it that way to make like real kind of production agent decisions.

• 39:51Molly O'Shea?

  And where are you getting, outside of this, where are you getting all of your research information diet?

• 39:56Speaker 02

  Yeah, so I think there's sort of three, there's three kind of avenues here. One is I did come from research, so I still have kind of a fondness for academic research. Like, we obviously have partnerships with the labs. They're not publishing as much, but a lot of my old collaborators are still publishing archived papers and NERIPs, ICLR, all these academic conferences. Even new conferences are spinning up around agents. Twitter is just like the real time.

• 40:25Molly O'Shea?

  Really?

• 40:26Speaker 02

  Feed it directly. Yeah, I think you have to have a great filter.

• 40:30Molly O'Shea?

  Who are you following?

• 40:32Speaker 02

  You have to understand like the signal to noise ratio of Twitter. But if you're following folks like, you know, Karpathy, Noam Brown at OpenAI, Shelto at Anthropic, some of these kind of leading voices within the labs, as well as some of these academics who have a presence on Twitter as well. Like, it can be, it can be helpful.

• 40:51Molly O'Shea?

  You talked about the biggest takeaways from the benchmarks, but what are you most excited about?

• 40:57Speaker 02

  Yeah, so what I'm most excited about is, so we get this cost, we get this latency, we get this practice area breakdown, right? That basically tells us where do we need to invest to improve performance of our product for our customers. I also think that the benchmark is a great kind of way to ground research directions that we want to invest in, right? And so we've seen an insane amount of activity from the open source community, both AI researchers and legal tech, which, believe it or not, has a bustling sort of open source community, to investigate a number of areas, right? So like post-training, open weight models. I think we're starting to see early results that show that open weight models, when post-trained, kind of close the gap with the closed source frontier.

• 41:40Speaker 02

  The agent harness is like the buzziest topic right now.

• 41:45Molly O'Shea?

  Tell me more about that.

• 41:46Speaker 02

  So the harness is essentially the term, the term of art that Twitter has adopted for the infrastructure and scaffolding that goes around the model, right? So you have some model, it's making decisions. You give it some number of tools, like I can read a document, I can write a document, I can do a web search, I can write code. The harness is basically the way that you define all of these tools, skills, how agents can delegate to other agents to complete tasks. It's basically the infrastructure and scaffolding around the models. It's a really interesting area of research for us, though, because the thing that we're seeing over and over again is that specialization matters and domain expertise matters, right?

• 42:30Speaker 02

  And so we can have our lawyers, the open source community, our AI researchers, all working on legal-specific skills and tools that make the agents better at these tasks that lab and our customers are sort of experiencing.

• 42:42Molly O'Shea?

  So you're even more bullish on Harvey now?

• 42:45Speaker 02

  Yeah, I've been bullish since day one. I think it's only increased.

• 42:48Molly O'Shea?

  I know, I think I saw you have a bobblehead behind you. I know they have a fancy term, but yeah, seems like you've been here for a bit.

• 42:56Speaker 02

  That is one of the most unique three-year anniversary gifts I think I've heard of from a company. But yeah, three years and about a month.

• 43:03Molly O'Shea?

  Exciting. Okay, so as we close out, how many times are you going to beat this benchmark?

• 43:08Speaker 02

  I do think the benchmark is now a target.

• 43:12Molly O'Shea?

  You created your own competition internally. Is this good?

• 43:16Speaker 02

  We did. My perspective on the entire kind of like AI benchmarking and hill climbing exercise is the extent to which everybody is investing in making models and agents more capable for legal that is purely to our benefit. Because actually, like, the innovative work that we want to do is not just base model capabilities. And so you ask, what is the thing I'm most excited about? I think this year, one, I do think the benchmark gets saturated within a year at least. But two, I think this year is the year that we see intelligence at an individual level kind of brought into intelligence at an organizational level, right? And so for Harvey, what that means is moving one layer of abstraction up in decision-making, like, how do lawyers collaborate with lawyers on our platform?

• 44:01Speaker 02

  How do human agent teams collaborate? And I think there's a lot of really interesting product infrastructure, but also AI problems to explore there that, frankly, we haven't been able to explore yet because the models are not good enough, right? So if we make the models good enough, then we can do really innovative research, in my opinion.

• 44:20Molly O'Shea?

  So the collective IQ will go up 50 points?

• 44:24Speaker 02

  At least.

• 44:24Molly O'Shea?

  OK. Well, Niko, thank you so much.

• 44:26Speaker 02

  Thanks for having me.

• 44:28Molly O'Shea?

  Hey, it's Molly. If you enjoy our interviews, check out our newsletter, sorcery.vc, where we deliver a once-a-week top deals and tech headlines email, and also go deeper on our podcast interviews. Subscribe to Sorcery today. And don't forget to subscribe to the podcast on YouTube, Spotify, Apple, or wherever you listen. Link in description to sign up.