TWiST 500 interviews with Cortical Labs, Turing, AND Mercor | E2159

Today's show: Alex is back with three more awesome interviews with founders on the bleeding edge of innovative tech. Dr. Hon Weng Chong walks us through the basics of biological computing and Cortical Labs' first-ever commercial computer running on living human cells. Turing founder Jonathan Siddarth unpacks the secrets of LLM benchmarking, and explains why even our most advanced tests need to get much much harder right away. Finally, Mercor founder Brendan Foody on how AI is about to reinvent the hiring process, and marrying the effectiveness of recruiters with the ease of online job boards. It's three — count 'em, three — can't miss TWiST interviews guaranteed to make you smarter Timestamps: (0:00) OpenAI's GPT-5: When is it coming out? Is it going to be TOO smart? (08:06) Cortical Labs' Hon Weng Chong on the electric connection between neuroscience and machine learning (10:20) Northwest Registered Agent. Form your entire business identity in just 10 clicks and 10 minutes. Get more privacy, more options, and more done—visit https://www.northwestregisteredagent.com/twist today! (11:27) Show Continues… (15:17) The extreme difficulty of going from the lab to a shippable product (20:00) . TECH: Say it without saying it. Head to www.get.tech/twist or your favorite registrar to get a clean, sharp .tech domain today. (21:05) Show Continues… (28:15) Why data is a factor of time (29:52) AWS Activate - AWS Activate helps startups bring their ideas to life. Apply to AWS Activate today to learn more. Visit aws.amazon.com/startups/credits (31:16) Show Continues… (42:44) Turing CEO Jonathan Siddarth explains why it's so important to keep benchmarking our LLMs (47:24) What it means when a model "saturates" a test, and why benchmarks need to get HARDER (50:22) What happens with the LLMs can answer all of our smartest questions? (53:44) AI Agents train in gyms? Wait, really? (01:01:33) Coding teaches models how to think, and more training mysteries don't understand (01:03:11) Brendan Foody from Mercor explains the "matching problem" that makes hiring such a pain (01:07:02) How Mercor combines a job board's distribution with the value of a recruitment agency (01:10:49) Brendan recalls building his first AI interviewer in his college dorm (01:20:16) Mercor has the opposite of a retention problem and crazy growth Subscribe to the TWiST500 newsletter: https://ticker.thisweekinstartups.com Check out the TWIST500: https://www.twist500.com Subscribe to This Week in Startups on Apple: https://rb.gy/v19fcp Follow Lon: X: https://x.com/lons Follow Alex: X: https://x.com/alex LinkedIn: ⁠https://www.linkedin.com/in/alexwilhelm Follow Jason: X: https://twitter.com/Jason LinkedIn: https://www.linkedin.com/in/jasoncalacanis Thank you to our partners: (10:20) Northwest Registered Agent. Form your entire business identity in just 10 clicks and 10 minutes. Get more privacy, more options, and more done—visit https://www.northwestregisteredagent.com/twist today! (20:00) . TECH: Say it without saying it. Head to www.get.tech/twist or your favorite registrar to get a clean, sharp .tech domain today. (29:52) AWS Activate - AWS Activate helps startups bring their ideas to life. Apply to AWS Activate today to learn more. Visit aws.amazon.com/startups/credits Great TWIST interviews: Will Guidara, Eoghan McCabe, Steve Huffman, Brian Chesky, Bob Moesta, Aaron Levie, Sophia Amoruso, Reid Hoffman, Frank Slootman, Billy McFarland Check out Jason's suite of newsletters: https://substack.com/@calacanis Follow TWiST: Twitter: https://twitter.com/TWiStartups YouTube: https://www.youtube.com/thisweekin Instagram: https://www.instagram.com/thisweekinstartups TikTok: https://www.tiktok.com/@thisweekinstartups Substack: https://twistartups.substack.com Subscribe to the Founder University Podcast: https://www.youtube.com/@founderuniversity1916

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  It's really important to benchmark L L Ms to figure out where they are good at, where are gaps in the models so that you can generate data that could help the L L Ms get better at those specific tasks. One big challenge, Alex, that I see today in the overall evaluation and benchmarking market is that a lot of the evaluations are somewhat academic and somewhat synthetic and don't connect to real world applications or real world use. Ideally, you want A G I to progress in a way where the models get better at useful tasks.

• 0:38Speaker 01

  This Week in Startups is brought to you by Northwest Registered Agent. Starting your business should be simple. With Northwest Registered Agent, you can form your entire business identity in just 10 clicks and 10 minutes. From L L Cs to trademarks, domains to custom websites, they've got you covered. Get more privacy, more options, and more done. Visit northwestregisteredagent.com/twist today. Dot tech, say it without saying it. Head to get dot tech slash twist or your favorite registrar to get a clean, sharp dot tech domain today. And A W S Activate. A W S Activate helps startups bring their ideas to life.

• 1:23Speaker 01

  As you build and scale your business, Activate credits grow with you to support your changing needs. Apply to A W S Activate today and receive up to $100,000 in credits. Visit aws.amazon.com/startups/credits.

• 1:42Speaker 02

  Hey, welcome to This Week in Startups. You may notice I am not Jason Calacanis. He is on the road this week. We're going to do a quick show without him. I am your host, Alon Harris. Joining me as always, Alex Wilhel.

• 1:53Speaker 01

  Hello, hello.

• 1:54Speaker 02

  We have three incredible, amazing Twist 500 interviews coming up right now that Alex did all week. Don't move from that seat. But first, I do think we've got to talk about G P T five a little bit, Alex.

• 2:05Speaker 01

  Yes, yes, G P T five. So this has gone a lot of play, but Sam Altman. One of the founders of OpenAI and kind of what you might call the face of AI from the software side, Jensen being the hardware face, if you will, went on a podcast with a man named Theo Von, and they were having a pretty candid conversation about what they're building. And Sam says, you know, I was playing with GPT-5 and I gave it a challenge and it did it very quickly. And he talks about how it sat him back and like made him think about what they're building. And the quote that really went viral was, he was like, you know, there are moments in time in science when people sit back and go, what have we done? And, you know, on one hand, Lon, I have three things here. First of all, Sam Altman loves to hype things up and then get mad at everyone when they get a little overhyped.

• 2:50Speaker 01

  He's a master at PR. So yeah, part one. Two, if you go back in time to GPT-3, even maybe even GPT-2, there was a lot of worries that as this technology advanced and progressed and got better, it will be misused. And we're seeing that now when people are using voice cloning to trick elders out of their money. And AI is being used in more and more nefarious circumstances. But we also at the time had a lot of people thinking that there was going to be a lot of improvement and things were going to get much better more quickly. So when I, when I think about his GPT-5 comments, they don't actually feel that different to me than what we've seen with GPT-4, GPT-3, GPT-2. So that's, to me, this probably does feel that impressive to him as he felt blown away before.

• 3:34Speaker 01

  So to me, I don't think he's being cynical. I don't think he's overhyping here. I think we're just seeing improvement and I don't think we know exactly where it's all going.

• 3:43Speaker 02

  I mean, I, I think that sounds reasonable enough. And, and look, I, I'm sure he's playing around with GPT-5 and is genuinely impressed at what it can do. Like, I think there's two levels here. Like even, like, I was very cynical about AI early on. Even I have had experiences like that where I'm impressed by what it can do and it sort of raises the bar on what I thought was possible. I'm not discounting that experience at all. I'm sure he does have that experience. But I also think this is a weird and unfortunate narrative that we've sort of fallen in. Like, and Jason has talked about this many times, that like real science and real technology kind of.

• 4:20Speaker 02

  Chases sci fi and I feel like sci fi introduced this idea that AI is scary and that AI is going to be thinking for itself. And all of those things like from Skynet in Terminator to Scarlett Johansson and her evolving beyond Joaquin Phoenix and wanting to go off on her own. Like, I think that's always.

• 4:38Speaker 01

  Spoilers.

• 4:39Speaker 02

  So spoilers for her. Sorry, everybody. I feel like that's always the fictional narrative around AI. And I do feel like that has an impact on real world AI CEOs and they're trying to be part of that narrative and play into that narrative. Because it's a helpful narrative for them. We're creating iconic, legendary, history shaping technology. That's akin to the atomic bomb or the discovery of electricity. And I mean, like, it plays into a friendly narrative if you're the CEO of an AI company.

• 5:11Speaker 01

  That's absolutely true. No, you're not wrong. I just think that AI has put up enough in terms of real results, real progress, real products, real services. That to me, we're just talking about how correct they are versus how incorrect they are when they make those large pronouncements. But I do, I do think the context here is that OpenAI is expected to release their OpenWeights model, their kind of open source-ish project sometime soon. And then GPT-5 is anticipated, we think, sometime in August. Which means that we're going to get past the hype and more into the testing. So, you know, will GPT-5 be as good as it's expected to be? Well, we're gonna have to find out pretty soon and that's encouraging.

• 5:49Speaker 02

  Pretty soon, exactly. The media is sort of saying, you know, early August or sometime in August, the sharps at polymarket, of course.

• 5:58Speaker 01

  Yeah, yeah, we were looking at this.

• 6:00Speaker 02

  Yeah, not content to wait and see what happens. There's already, of course, wagering going on. There's already a market going on for this. 25% are betting that by August 5th, which is early next week, that would be, we are right on the verge of GPT-5 being here.

• 6:16Speaker 01

  Yeah.

• 6:18Speaker 02

  One in four.

• 6:20Speaker 01

  Yeah, one unfortunate. And then by August 15th, people think it's a two thirds chance that it's out. So I my read of this lawn is that basically middle August is kind of the over under and that after that will be a surprise earlier than that will be a surprise. But we're only a couple of weeks away. That's what's exciting to me. That's the cool thing. Remember how exciting it was when XAI dropped Grok four and they talked about Grok four heavy and then Grok four coding. Then we got the KIMI K2 family of models from Moonshot. Then we got the Z.ai. I forget the GLM 4.5 series. The point is, it's been a flurry of models and Alibaba's quid models are improving.

• 6:54Speaker 02

  Yeah, we're in the middle of that meme that you are here, the world's greatest AI model like we're right in the middle of another cycle. Yeah.

• 7:03Speaker 01

  But the question is, will this be just another iterative improvement? And if so, OpenAI is going to take a lot of egg on its face. Or is it as big of a jump as maybe GPT three to four was or larger? And if that's the case, it does change the entire fabric and landscape that the technology world plays on. But Lon, let's talk to some founders.

• 7:24Speaker 02

  I love it. Let's do it.

• 7:25Speaker 01

  All right. So we have three interviews for you today. One cortical, then we're going to talk to Turing, and then we're going to talk to Mercor. Each one of these was an absolute treat. Cortical is probably the most out there in that we're talking about kind of like hybrid biologic and digital computers. It's it's an interview I chased down for a long time. It's super fun. And then Turing and Mercor, a bit more traditional in the software space, but incredibly interesting founders, incredibly interesting companies. And especially, I think, in the case of Mercor, a vision for a company that could become incredibly instrumental to how labor functions in the future. So these are a treat. I had a blast. Please enjoy them and say hello to three of our founders from the Twist 500.

• 8:08Speaker 01

  Hey, welcome back to Twist. This is Alex. Today, we have another Twist 500 interview with the company that has been my white whale for several months, but I'm so glad to have them on the show. We're going to talk about cortical labs. Now, here at Twist, we love chips. We love talking about what Grok is building to handle AI inference. We loved that Etched is building ASICs just for LLMs. We've talked to Extropic and their thermodynamic computing paradigm. But all that's entirely silicon based, more or less. Cortical wants to bring the world of biology to the world of computing in a way that when I first heard about it, I thought was science fiction, but they've taken it out of the lab and they're commercializing it. So please join me in welcoming to the show. It's Cortical Labs CEO and co-founder Han Weng Chong. Han, how you doing?

• 8:51Speaker 03

  Hi, good thanks. Thanks for having me on the show, Alex.

• 8:53Speaker 01

  I'm so excited. Okay, so you guys are fusing biology and I would say digital computing in a way that's really interesting. But I think to help everyone understand how you got to where you are today, we should go back to the earlier days of the company when you guys were experimenting with your technology and you used it not to play Grand Theft Auto, as everyone wants to do with generative AI, but instead to play Pong. So Han, can you tell me about why you chose that? And then how the experiment went to prove out what Cortical Labs wants to build?

• 9:24Speaker 03

  Yeah, so you know, if you take a few steps back to, I guess, 2018, 2017, 2018.

• 9:34Speaker 03

  There was a, I guess, the first AI boom that happened. That was, I would call it the convolutional neural network reinforcement learning boom. I had just exited from my last business and as part of that, it was a med tech business that gathered a lot of data. I was looking at using machine learning to do automated diagnostics and really got deep into the rabbit hole of machine learning and artificial intelligence. And I came across a paper written by Damas, Sir Damas Hethabas from DeepMind, who wrote that the machine learning and AI community had to go back to its roots in neuroscience,

• 10:17Speaker 03

  because that's where most of the initial discoveries were made.

• 10:23Jason Calacanis?

  Founders, if you've got a product, maybe you've got some customers, or even just a little bit of traction, guess what? You've got yourself a startup, and it's time to make things legit. You want to be official, tighten it up. Investors don't want to wire money to a Gmail account with a P O. box. No, they want to know you're a serious person and that your company is incorporated. In 10 clicks in 10 minutes, you can follow your LLC, get an actual domain name, launch your official website, claim your business email, and even start fast-tracking your trademark application. That's right, you need Northwest Registered Agent, the service that's going to do all of that for you. With N W R A's Privacy by Default option,

• 11:05Jason Calacanis?

  they're going to use their address for all your public documents, not yours. So you're not going to get a ton of spam or junk mail. And you can also get a legit address and working phone number with their virtual office setup. So get more than just an LLC, get your entire business identity. Go to northwestregisteredagent.com/twist and show the world you're in business.

• 11:28Speaker 03

  So I did exactly that. I went to the neuroscience department at the University of Melbourne, which is my alumni, and spoke to the researchers there and asked, "What's exciting in your world? What are you working on these days?" You know, I'm thinking of applying machine learning and A I to the neuroscience world. And one thing that really kind of stood out to me was they said, "Oh, we have this device, called the multi-electrode array." It's not a new device, it's been around, I guess, since the early 2000s, where you can actually etch in electrodes in Petri dishes, where you can grow neurons on top of them. And because neurons communicate electrically, and because chips run on electricity, you now have this common language between the biological system and a compute system.

• 12:11Speaker 01

  Electricity is the shared language between both neurons and chips?

• 12:16Speaker 03

  Absolutely. I mean, if you think about it, that's how Neuralink works, right? I mean, the joke we have at Choral Labs is that we are the inverted Neuralink.

• 12:24Speaker 01

  Yes, you've taken the brain out and put it into the computer versus putting the computer into the brain.

• 12:30Speaker 03

  Correct, exactly. I had read the paper from Demis. I was inspired by it. And the first thing that Demis tried to do at DeepMind was he tried to get the machine to play pong.

• 12:40Speaker 01

  Ah, okay.

• 12:41Speaker 03

  Why don't we do the same thing?

• 12:43Speaker 01

  Yeah, yeah, yeah.

• 12:43Speaker 03

  Exactly. So quite a lot of the first few attempts were, uh, weren't really successful. But, you know, fortunately, these neurons are very malleable and they're very dynamic and they self-organize. And we realized that, you know, if we were trying to tweak the system while they were learning, you would end up with these very weird so-called attractive dynamics where, because, you know, what? It's like, how to describe it? When, when you're trying to find somebody in a shopping mall, but you, the other person's also looking for you and you're looking for them. And so you almost never meet because they're always moving around in that same pattern. That's what we were doing. So what we ended up doing was we said, well, we're not going to change some parameters.

• 13:23Speaker 03

  We're just going to leave it fixed, uh, but we are going to, you know, flash them with different like signals and so forth. Maybe hopefully that way they won't actually be continuously like, we wouldn't be wandering around trying to find it, uh, find each other.

• 13:35Speaker 01

  So, so Han, this is actually when I, I fell in love with what Cortical was doing because according to at least NPR's reporting of this, you guys provided kind of a positive and negative response electrically to the neurons that were interacting with the chip. And you gave a burst of, uh, I guess unfortunate, uh, white noise to the device if it was the wrong action and then a positive organized burst of electrical activity if it got it right. So essentially you very plainly shocked it if it did bad pong and you rewarded it if it did good pong?

• 14:07Speaker 03

  Kind of. I mean, I wouldn't say, it's not really a, you know, a reward or a shock kind of thing. Because at the end of the day, we're still giving them electrical signal. It's just the structure of it. And this was actually driven by, um, us coming across a theory developed by a, uh, uh, neuroscientist, uh, based in, in UCL in London by the name of Paul Friston, which is actually really interesting, really fascinating, um, called the free energy principle. The free energy principle kind of posits is that the Brain or all biologically intelligent systems are actually generative models.

• 14:47Speaker 03

  What we do in what was now believed is happening in our brains is that we're not reactive machines. We're actually predictive machines. And so we're actually generating hypotheses about the world that is external to the brain, in the brain as simulations. And then we're using the census that we receive and the ability to affect the world as many experiments to prove and disprove the hypotheses that are generated by the brain.

• 15:18Speaker 01

  Okay, so you guys did the Pong experiment and showed that you could, in fact, create a hybrid biologic and digital computer that could play Pong reasonably well. It wasn't gonna set world records for Pong playing. Now, take me from there to the CL1, your first commercial device that kind of brings this technology out into the world. How hard was it to go from successful experiments to commercialization and creating an actual shippable hardware product?

• 15:47Speaker 03

  Oh, really hard. I would say this is probably one of the hardest things I've ever had to do because there's some element of doing something really cool, something like really novel and so forth. But more importantly is that if you're doing it as a commercial offering, it has to work a lot, like at least most of the time, rather than some random time. So it has to work very reliably. You know, you do a lot of things that are very boring, that no one really cares about, but it's important and vital for the product and the machine to work. So for instance, life support plumbing, you can't publish that. That's kind of boring. Everyone knows what you need to do.

• 16:24Speaker 03

  But to actually go through the weeds of like developing, prototyping and testing is quite difficult. So it did take a lot of work. But what was actually the motivation for it was that we had a lot of our colleagues reach out to us after we had published the work. And, you know, they were asking us, where can they buy this machine? Or not this machine, but the machine that we had used. You know, what software did we had to write? Whether we could help them if there were any questions. And we, you know, we saw this recurring pattern. I said, look, you know, people shouldn't be continuously reinventing the wheel, right? Like a round wheel works really well and we know how to make it. Maybe you should focus on building the rest of the car and we can build the wheel.

• 17:07Speaker 03

  And then you just use our wheel. If you think about the explosion in AI, right? It had a very fertile ground because the groundwork had been laid by

• 17:19Speaker 01

  Yes.

• 17:19Speaker 03

  decades of gamers buying GPUs, keeping the likes of AMD and Nvidia alive until the AI systems came online, right?

• 17:28Speaker 01

  The venture capitalists, you're welcome for all my purchases to play games.

• 17:32Speaker 03

  Exactly. So I think this is, you know, something that we, you know, we saw that and we're like, well, you know, somebody's got to take that leap and try to support the research community. And we decided that that was going to be our thing going forward. Mind you, we still do a lot of research work, you know, we

• 17:50Speaker 01

  Of course.

• 17:51Speaker 03

  We have quite a few publications that came out. We have some very interesting stuff that's happening in the lab at the moment. But we think that this is more important, that no one company can do this by ourselves. And we need to actually expand this out to everyone who's interested and try to to reduce the barrier of entry for people to get into the space.

• 18:11Speaker 01

  Okay, I want to talk about commercial applications in a second, but first I want to give people a quick look at this. So if you're on the video version of the podcast, you'll see what I'm about to show you. First of all, Han, what is this here? I believe this is from the Pong era. It's a Petri dish with what appears to be a small dish in the middle. So what is this?

• 18:29Speaker 03

  Yeah, so this is what I was sorry about, the multi-electric array. It's got a well, so that round thing, that cylinder there is used to contain the, what we call the cell culture media. So the cell culture media is that liquid that bathes the neurons to keep them alive. It's very similar to, I guess, cerebral spinal fluid, you know, that bathes our own brain as these neurons are processing, they're consuming glucose. But at the same time, they're also producing lactic acid, just like running, right? You end up with the cramps. So, you know, one thing to be here, we try to do is prevent it from getting too acidic. And we use the CO2 as a buffer for that.

• 19:08Speaker 01

  Han, I'm just imagining now, like, you know, the dog didn't eat my homework. Instead, my computer got the cramps, so I couldn't, you know, it couldn't do anymore.

• 19:16Speaker 03

  Yes.

• 19:17Speaker 01

  I want to connect this to this image right here. This, I believe, shows a closer up of the neurons actually spread across a similar electric interface?

• 19:29Speaker 03

  Yeah, yeah. So if we remember that image of the multi-electra array with that cylinder thing, in the center of a cylinder, there is actually a CCD or CMOS sensor. This is actually the same sensor that you have in your cameras. And each of those square grids are actually a sensor that captures the electrical activity. It's also somewhat photoreceptive, so that's why we have to keep them in a dark space.

• 20:02Jason Calacanis?

  We all understand the importance of a crisp, memorable, easy-to-spell domain name. One of those names you can say over the phone and people know how to type it in without asking you the spelling. But let's get real, the good ones are either taken or there's some poacher who's holding it and waiting for some huge payday and they don't reply to you, even if you want to pay for a premium domain. You don't want to use up all your runway on a domain name. That's just the truth for a startup. You want to put that valuable cash back into your startup's operations. So you should consider this, a .tech domain. You can get a clean, crisp, super memorable name for your website and company and signal out loud to your customers and investors, we're

• 20:43Jason Calacanis?

  a tech company. That's instant branding for you. That's why over 500,000 founders have collectively raised over $5 billion in investment, building their companies on .tech. So skip the hassle, head to www.get.tech/twist, or go to your favorite registrar and grab your .tech domain today.

• 21:05Speaker 03

  And so you'll see things like the large, long piece that's going across, that's an axon, and all the stringy bits are the dendrites. So if you think about that, that's just one segment and 50, that's microns. That's actually the width of a human hair, 50 microns. And so, yeah, that's how small it is. And the level of connectivity, the level of self-organization as well is actually pretty phenomenal.

• 21:32Speaker 01

  It's so, this is, this image just, I've actually stared at this for probably 20 minutes just straight because it's such a, it's literally to me like 1980s science fiction, but just actually real and now turning into a commercial product. Like this is so cool. And I just want to show everyone, the culmination of all this work that you guys have done at Cortical Labs is this bad boy. This is the CL1. If you're watching, sorry if you're listening to the audio version, imagine a very long toaster with a clear plastic top and lots of tubes. And I believe this bad boy can keep neurons alive for six full months. Han, going back to your point about the plumbing for life support, yeah?

• 22:10Speaker 03

  We'd say six months because it's just easier to explain it that way. But actually, you can keep, you know, we as a lab and other people have kept them alive for, for, you know, months, even years, kind of thing. When you, when you're talking about trying to do large scale experiments, you need a lot of samples. You need a lot of units to get, you know, statistical significance and power through a, through a large N. We wanted that scale. We wanted that longitudinal ability to, to study the neurons. But we also didn't want to have to spend a lot of effort doing it. So a lot of, a bit of engineering went into a bit of plumbing to, to, to build this system that encapsulates the neurons in a,

• 22:54Speaker 03

  in a sealed environment, which is really important because these neurons don't have an immune system. And so, you know, if you had COVID and you coughed into it, they would actually just also get COVID.

• 23:02Speaker 01

  So your computer could literally catch the cold. That's really funny.

• 23:05Speaker 03

  Yes, pretty much. You know, don't eat bread next to it because you'll get the yeast and mold that goes into it and they'll kind of kill the cells. But we also built in, you know, our own neural interfacing system. So there's actually, you don't really see it at the base of the CL1 has a compute unit. So we have, you know, FPGA and what do we call it? General purpose compute units in there as well. That's where we digitize the signal. We, we then put it through our very tight processing loops and, you know, we're developing APIs and SDKs for this so that you know as a programmer you don't have to go

• 23:48Speaker 03

  deep into getting these things to stimulate the neurons with high precision but you know a lot of code you can just use our API and we've developed a DSL domain specific language to express how do you want to stimulate these neurons at individual channels at low latency loops.

• 24:09Speaker 01

  That that brings me to kind of I think the question on everyone's mind who's listening to us right now which is okay this is awesome you've made a digital biologic brain what's it good for and I think that this is actually where my my knowledge really kind of runs short because I'm not sure what compute loads what experiment types what commercial applications the CL one and its successors are going to be best at and so when we think about the market that cortical out is going after.

• 24:36Speaker 03

  Yep.

• 24:37Speaker 01

  How big is it?

• 24:38Speaker 03

  I think the two most interesting things that we've learned along the way about the system here is that they do two things really well well actually three things but that third thing is attached to the second one firstly they use significantly less energy than your traditional compute they're not a von Neumann based architecture the the energy is derived from the glucose in the actual cell culture media so doing it back on the envelope calculation we came across some really phenomenal numbers so for our original experiment we grew about eight hundred thousand to a million neurons and we did a very rough estimate where we just said what's the glucose content in the media how often are we changing the media so therefore

• 25:21Speaker 03

  if we change the media you know X number of times a week and there's this much of glucose and they're combusting all of it assuming because they never combust a hundred percent you know how much energy were they using and turned out it was actually 10 to the minus four watts of energy so that's 0.0001 watt of energy it's probably even less than that because that's a massive overestimation.

• 25:41Speaker 01

  But but still even with massive overestimation it's effectively zero and glucose last time I checked is sugar water so it's nice and cheap.

• 25:49Speaker 03

  Exactly I mean if you think about it you and I as walking reference use only 20 watts in our brain.

• 25:55Speaker 01

  The guys at Xtropic were telling me this. They're like, you know, we have this amazing probabilistic computer that runs on 20 watts and it's literally inside your head. And they're working on thermodynamic computing, which is different, but I've been thinking about that ever since they brought it up. Okay, so clearly this is not going to be used right now to train the next major LLM. But the people who are purchasing these L1s that will come out sometime next year, what do they want to do with it?

• 26:20Speaker 03

  I think the thing to understand here as well is that LLMs could not exist 20 years ago. Right? Even if we had the compute available, we could not make it work because LLMs required large data sets. I mean, they're called large for a reason, right? We've gotten to this point of, what is it, 40 years? What are we now, 20, 25? Let's say 40 years of the public internet being this repository of free language training data. There are lots of other domains that are not language based, that do not have large data sets that are publicly available. And this is the second point that we've discovered

• 27:03Speaker 03

  along the way is that if you do a head-to-head comparison, and there's a paper coming out, actually, this was in Europe, it's about two years ago, where we put it in the RL workshop. It's one of the posters. We took reinforcement learning agents. So we did an experiment where we took three of them, DQN, which is your classic AlphaGo. It's kind of, I guess, the gold standard, but it's kind of old now. There are better algorithms out there that are more what's so-called sample efficient. And we put that against the biological system and we get the same game. But we said, here's the one thing, we're going to constrain the reinforcement learning agents so that they get the same amount of data that the biological system gets, right? Because if you think about reinforcement learning systems, nobody really talks about this.

• 27:45Speaker 03

  But the way they actually learn in a simulation is that they spawn millions of parallel processes and they speed up the time of the game by 200, 300 fold. And so if you read the AlphaStar paper or something like that, DeepMind kind of said that if you were to train a A thing or human to play at the amount of data that AlphaStar was playing, it would take 400 years of continuous gameplay to get to that level of performance. And so what we did was we said, we're going to sample constrain it because if you think about this, the real world, you can't speed up time. Time ticks at the same rate that you and I have.

• 28:26Speaker 03

  And if data is a factor of time, right, say bitrate and so forth, then if you wanted to get, you know, say 10 minutes of data, you would have to spend 10 minutes collecting that data. Knowing that, you know, these systems use far less data, when we did the comparison with the Reinforcement Learning Agents, and, you know, data being a factor of time, we realized these things are actually potentially very good for problem sets that don't, problems that don't have a data set, problems that are real time specific. And that is actually the vast majority of the domains that have not actually been touched by language.

• 29:03Speaker 01

  This is when I get really excited and I don't, I don't want to make you project the future too much because that's not really fair. But to me, with, with neurons that we can interact with the software, it seems like we've managed to take out the best of both worlds. The programmability of, of computers as we understand them, and also the probabilistic low power consumption and kind of generalized intelligence of brains, and brought them together, Han. So to me, in time, it feels like we should be able to use both, you know, sure, our mass GPU clusters that everyone's building, but also we should have probably large tanks of biologic, you know, neurons that are also helping us do a lot of, a lot of work.

• 29:47Speaker 01

  Is that too kind of science fictiony or am I on the right path?

• 29:55Jason Calacanis?

  We're all familiar with AWS, Amazon Web Services. That's the cloud platform that powers so many of your favorite brands. But do you know about AWS Activate? That's their program for startups where they provide up to $100,000 in AWS credits for all startups, whether you're backed by an investor or you're bootstrapping. Money is time to keep innovating, time to delight customers and time for you to keep gaining traction. You need runway and AWS's Activate is going to help you with that runway. We hear this story from so many of our founding university companies. They're finding product market fit, the words getting out about their product, just a little boost. Finding some savings here or there to get a little extra time can be the difference between getting traction

• 30:37Jason Calacanis?

  and bringing in revenue, or hey, let's call it what it is, running out of money, okay? And shutting down. AWS knows this. That's why they've created the ultimate toolkit for early-stage startups looking to boost growth. With AWS Activate, you're going to get up to $100,000 in AWS credits, hands-on support and training, plus exclusive discounts with some of our favorite companies and tools. So start getting the support you need at every stage of your startup journey. To learn more, visit aws.amazon.com/startups/credits. That's right, aws.amazon.com/startups/credits.

• 31:16Speaker 03

  I think you're on the right path. I mean, the thing about it is that it's, we're good at some things that machines aren't. And machines are good at things that we aren't, right?

• 31:23Jason Calacanis?

  Right.

• 31:23Speaker 03

  So this is called Moravec's paradox, particularly in robotics. And I think this is the motivating factor for why, you know, Elon set it up, things like Neuralink, right? And the BCI people, which is that, you know, as AI gets smarter and better, we also need to skill up, right? They're getting, so if you think about it, the machines are getting better at what we reserve as traditionally human or biologically centered tasks. It's not beyond the realms of possibility that we could also go the other way, assuming the BCI has really become a thing, where not only do we utilize our brains for the low-power stochastic computation, the probabilistic computation, but we can also just, you know, have a, I don't know, an iPhone chip in there that can just give us the square root number at the speed of thought.

• 32:03Speaker 01

  Right, so I could essentially allow compute to handle the math that I don't want to do myself, but all the learning I can use my brain for, so.

• 32:10Speaker 03

  Correct.

• 32:11Speaker 01

  Once again, you guys are kind of Neuralink in reverse, so.

• 32:14Speaker 03

  Yep.

• 32:15Speaker 01

  Look ahead to just a couple of years, not super long term, but the CL1 gets into the market next year, customers, you guys are also going to have an API to let people access the technology remotely if they don't have the lab setup necessary to run this thing. What's the next generation? Like, what are you guys going to build after the CL1? What is the CL2?

• 32:32Speaker 03

  Well, I think the CL2 is, we've been thinking about it, and there are a lot of features that didn't make in the CL1 that we'll probably, you know, start thinking about putting into CL2. I mean, the things, your standard things, right? Make it like smaller, make it, you know, probably cheaper to obtain, easier to program, you know, maybe more modular units. There are quite a few things that, you know, we've been thinking about. Now, the question is, does it make it to CL2 or does it get pushed out to the CL3 is, you know, a discussion for the team. But again, you know, I think it's, it's still a little bit premature. What we really want to do is get the CL1 out to our partners, our research collaborators, and really get their feedback, what works, what doesn't

• 33:16Speaker 03

  work. So that way, we can sort of triage and prioritize what we want to do for the CL2. No point of putting in features that no one else wants. I mean, we might want it, but, you know, we have the ability to just spin up whatever, CL3, 4, whatever internally. But, you know, if there are features that are, that are missing, that we have on our list that's kind of lower down our sort of priority, but it would, you know, tremendously help the, the community get going faster or do more work for less. You know, I think that's something that we want to be prioritizing.

• 33:50Speaker 01

  You talked about using mice neurons at the top of the show, and anyone who knows anything about biology and experiments knows that a lot of, a lot of mice and rats die. So that is kind of par for the course for how we currently handle ethics. Are mice neurons and human neurons radically different? Are they relatively fungible? Silly question, but I'm not sure about the answer. Before I get to an ethics question, I thought we'd start there.

• 34:14Speaker 03

  So, and this is the really interesting thing. We, we used to think that there was no difference between a human and a mouse neuron. If you read any of the papers before, I think 2022, 2020, everyone said, yeah, there's no difference. All mammalian neurons are the same. And it turns out that I think on average, human dendrites are actually longer than mouse dendrites.

• 34:40Speaker 03

  But they have the same number of ion channels. So these are the gates that allow the potassium and sodium to go back and forth between the cell and the external environment. And that's what generates the action potential. But having spaced out more, you actually have the ability to hold more so-called electrical states. Well, that's the theory that's been pushed by some of the researchers at Harvard MIT. And so, yeah, it turns out they are different and that probably contributes to the differential in the overall performance. And we've seen that as well. The human neurons are better at playing and processing information.

• 35:16Speaker 01

  But for now, the mice neurons are sufficient for the state of technology and the progress you guys want to make. So we're not, there's going to eventually become a news story entitled, "Startup takes human neurons, puts them into computer, are we creating a new form of consciousness? Oh, oh, panic." But it sounds like-

• 35:34Speaker 03

  Well, we actually do use human neurons now. Alongside mouse neurons, human neurons is the other really big area of research because that's where a lot of the biomedical stuff happens, where we're looking for new drugs, you know, understanding disease models and so forth. We stand at the shoulder of giants and so we use a lot of the techniques that have been developed by the, what you call the synthetic biology, the biological engineering field to grow stem cells obtained from adult cells. So if we went back about 10 years ago, in Japan, there was a researcher by the name of Yamakawa. Professor Yamakawa won the Nobel Prize, I think, a few years back for this discovery

• 36:19Speaker 03

  of inducible pluripotent stem cells. And what he discovered was that if you took anybody's cells, your cells, my cells, you know, skin or blood, anything with a nuclei, and you expose them to four compounds, you can actually reverse the clock back of these cells where they go from a blood cell or a skin cell back into a naive state stem cell that can then become, turn into anything. We- Utilize that functionality, yeah, to essentially grow human neurons from stem cells. So this way, we don't kill any animals. It's continuously renewable, as long as you provide the right conditions. And yeah, that is, that's how we do it now at Cortical Apps.

• 37:04Speaker 03

  Because, you know, we don't really particularly like killing the mice as well. And so we think that this is actually an easy approach.

• 37:10Speaker 01

  Absolutely brutal. No, I'm totally in favor of that. I think there's going to come in time, some questions that appear ethically interesting with what you guys are building. But I'm also of the opinion that they're not actually going to be ethically serious. I think they're going to be mostly good for not making fun of the media, my industry, but like, great for a headline, but less dicey in practice. So I'm very bullish on the company. Just before I let you go, how has demand been for the CL1? I know you guys have a forum up. I know you guys are working on deliveries. Has demand exceeded expectations? Is it about what you thought? Just where's the business side of things going?

• 37:48Speaker 03

  Yeah, demand has actually far exceeded what we had expected. Suffice to say, we are now struggling to try to figure out our logistics and supply chain for, to balance supply and demand. You know, we've had a lot of signups, particularly for the cortical cloud. I think over 3,000 signups. We have no ability to service more than 20 or 30 people on the cloud system.

• 38:21Speaker 01

  So you have 100x more demand than you could say.

• 38:24Speaker 03

  Yeah, and so they're on the waiting list and so forth. And then, you know, for the hardware, we have had a lot of labs and research groups reach out to us. We are, you know, very excited about this, but now we're looking at the challenger of, oh God, now we actually have to ship this thing. And we're going to have to ship lots of them. And, you know, if they break in the field, we're kind of screwed. So, you know, we better make sure they don't break out there. So there's a lot of like, we're doing our best at the moment to, A, finalize a lot of the software, finalize the hardware, you know, getting our partners ready. It's also very, very expensive, CapEx wise, right? Because we have to, we don't charge our customers until we ship them, but we still have to build the units first. And so, you know, trying to figure out where the next source of funding is going to come

• 39:09Speaker 03

  from is going to be a bit of a challenge.

• 39:11Speaker 01

  Hon, why don't you just charge them half upfront and resolve some of your cash flow issues?

• 39:15Speaker 03

  Well, we could, but you know, it's

• 39:18Speaker 01

  Do it.

• 39:19Speaker 03

  Yeah, we could do that. But you know, I think for us, we want to make sure that, you know, we I don't want to do a Kickstarter kind of thing, right? And they had to start at the

• 39:29Speaker 01

  I don't think that's Kickstarterish, but I respect where you're at. I guess then the correct question to close with is this. Sometimes people say that venture capital is a little bit conservative. I think the VCs that bet on Cortical were clearly being relatively adventurous, which is good. So as you guys do need more capital, is the venture capital ecosystem in Australia, Europe, and the West at large, is it interested in putting more money into the business? Or are you guys still a little bit outside of what you might call the venture norms?

• 39:59Speaker 03

  Yeah, I think we're outside of venture norms. I don't know what it is, but the venture industry likes to gravitate to thematics. So in this case, it's been, what is it? We went from transformer LLMs to agents and something, something, whatever next year brings. It's very hard, I think, unless you fall into specific thematics to attract that. So, you know, I think we have some of the best investors in the world backing us because they're just the ones who

• 40:29Speaker 03

  are willing to take bets, which is essentially what the industry initially started out with, right?

• 40:33Speaker 01

  And Blackbird Ventures and Horizons Ventures.

• 40:37Speaker 03

  Correct, yeah. And, you know, we've had, you know, a new round that was just put together to get us a little bit more fuel to deliver our products by kind of people like 3C AGI, they've just come into the game. And then, you know, we've had players like NQTEL as well, who have backed us. It's really about, I think, trying to figure out who are the Mavericks in the space? Who are the ones who have the imagination, right? And the sci fi background, because at the end of the day, if you have 200, like, maybe not 200, maybe like, say, even 10. Fund foundational model companies, but they're all pretty much, I couldn't really tell you the difference between Claude versus Gemini versus ChatGPT versus, I don't know, whatever else, like Quen or DeepSeek. They are mostly the same. And so if they're all

• 41:22Speaker 03

  mostly the same, and I can jump between one another, I don't really see any, you know, significant stickiness or moat to it. So anyway, that's a problem the VCs do to resolve. But, you know, rather than all jumping in into one space, I think we should try to spread it out. So for us, you know, we're very focused on delivering the product and, you know, we welcome anyone who's interested, who's listening to your podcast, who understands the fundamental nature of building hardware first before you can get to software, you know, to, to, to look into the space. Because the, the thing, as we already talked about, the AI space already had that groundwork put in by the gamers, right? You didn't, we didn't need to invest in the hardware because the gamers just bought it already.

• 42:05Speaker 03

  So I think that's something that we have to bear in mind. With any new compute space, there is always going to be a hardware component, and then we, we shouldn't shy away from having to invest and build into that space.

• 42:18Speaker 01

  Well, I do, I do look forward to the eventual future when we have Canva in Sydney, and we have Cortical in Melbourne, and we'll have a good old fashioned competition about who's going to be the next future leader of Australian technology. Han, thank you so much for coming on. Thank you for answering all of my questions. And when you do have so much extra production capacity, I'll give you my address to ship my CL1, and you just tell me where to send the check, okay?

• 42:40Speaker 03

  All right. Thanks, Alex.

• 42:41Speaker 01

  Thanks, Han.

• 42:44Speaker 01

  When Meta invested in Scale AI, absorbing its CEO and co-founder in the process, startups that competed with Scale saw an absolutely huge opportunity. Now, mostly people focused on the data labeling side of what Scale had done historically, as the place that startups might actually capture the most market share. But Scale also offered AI evaluation tools to folks who build LLMs. So startups that offer AI evaluation tools may also be in line to benefit from Scale aligning with Meta, a company that competes with other firms to build the next great AI model. Why work with Scale if Meta? That owns about half of it. One startup CEO, Turing's Jonathan Siddarth, told Reuters after Scale's partial exit to the social giant, that leading AI labs now realize that neutrality

• 43:27Speaker 01

  is no longer optional amongst service providers. It's, quote, "essential." So, to help us understand the LLM evaluation market, just how big it is and how data comes into play in 2025 to build those next models, please welcome to the show, it's Turing CEO Jonathan Siddarth. Jonathan, hey, welcome to the show.

• 43:44Speaker 00

  Thank you, Alex, for having me. It's great to be here.

• 43:46Speaker 01

  I'm very impressed. Also, we're both in our home offices today, and I think this just goes to show that remote work, not entirely dead, even though everyone seems to claim that it is. I'm glad that you're here. So, starting for folks who are less aware of what LLM evaluation is, Jonathan, can you just give us the working definition from your side of the fence?

• 44:04Speaker 00

  Yeah, so it's really important to benchmark LLMs, to figure out where they are good at, where are gaps in the models, so that you can generate data that could help the LLMs get better at those specific tasks. One big challenge, Alex, that I see today in the overall evaluation and benchmarking market, is that a lot of the evaluations are somewhat academic, and somewhat synthetic, and don't connect to real-world applications or real-world use. Ideally, you want AGI to progress in a way where the models get better at useful tasks.

• 44:43Speaker 00

  So, when we evaluate models, the three dimensions to look at are complexity, like, are you evaluating them on really hard tasks? Real-world use, are you evaluating them on something that a human would actually care about? And third is diversity. You want like a wide breadth of test cases that you're evaluating the models on. It's such an exciting space, and I think of evaluation as step zero of data generation. That's why we do both evaluation and data generation.

• 45:14Speaker 01

  Okay, so let's talk about the benchmarks because there's been some commentary, I think Apple had a paper that came out and they said that, you know, one of the problems we have with a lot of the benchmarks that everyone likes to trot out their new LLM and put them up against and have the charts is that there's data contamination issues, there's overfitting. So based on what you just said about trying to solve real world problems and also the fact that we know that these benchmarks are getting a little bit, I don't know, dicey to use, is the standard way that companies announce how their LLMs perform an effective form of evaluation? Or is that mostly window dressing to get more social media hits because you're one point higher on one test or another?

• 45:51Speaker 00

  So I'd say it's the labs care about the benchmarks because it's good for bragging rights, it's good for recruiting, like who has the, who has the best model for coding, who has the best model for STEM, etc. So benchmarks serve some useful purpose, like, and you can argue that in today's hundred million dollar sort of talent wars, or maybe it's closer to 300, 400 million dollar talent wars, that those bragging rights help because the best researchers ideally want to join like the winning team, like who's already kind of close to, close to being number one. But the labs care about two things, public benchmarks and private evals. And I would argue that private evals are actually more important because there you're

• 46:34Speaker 00

  evaluating how well is your model doing relative to the competition on the prompt distribution that you care about. Meaning if you're Google, or if you're, if you're Meta, or if you're OpenAI or Anthropic, the type of queries you might get inside in a coding context might look different from what you might get on the phone in like a general chat assistant context, or what a human might put when you're searching through like a desktop. Right? So you want to optimize for your own query distribution that your users are testing your model on. So that's why these private evals are helpful. So you need public benchmarks and private evals. The challenge with public benchmarks, Alex, is if you look at coding,

• 47:18Speaker 00

  for example, there's, there is this good benchmark called Sui Bench, which is created by this lab at Princeton. Last year, we went from 2% to 50% in Sui Bench. This year, the best models are already north of 60%. We'll probably saturate Sui Bench this year. And then where do you go, right? Like, we haven't, clearly, we haven't automated all of software engineering yet. The benchmarks are getting saturated.

• 47:40Speaker 01

  Yeah, can you double click on what you mean by saturated in that context? I think that's an important point.

• 47:44Speaker 00

  Yes. So if a model does 90% plus in on a benchmark, you've kind of aced the test to some degree. And researchers call that saturating a test. Because now the test no longer tells you whether your models are improving or not. Right? So the obvious answer is you have to create a harder benchmark where the models would have even more headroom to climb, right? Even more room to improve. Which is why at Turing, we're actually creating a benchmark for coding that's even harder than Sui Bench. And it gives us deep satisfaction because we managed to have all the models start at zero.

• 48:23Speaker 01

  Oh, okay. So everyone's failing 100% of this new tech. Ah, okay. Let's talk about private evals. Because this is a core thing of what Turing offers to its customers. So I'm curious. Let's say that I'm OpenAI. I have a new model. I want you guys to take a look at it. You find a couple of places where it's not quite up to snuff, not quite where we expected it to be. So then do you turn around and say, hey, guys, here are the places where there are gaps? And then help them solve those issues? Or do you guys just point out, here's the spot where, you know, you might want to do some more work?

• 48:54Speaker 00

  So we do both. We evaluate the model and we generate data to help the models improve. But let me take a step back, Alex. And let's look at this landscape, which is super interesting right now. So as these, what's happening is as these models have gotten smarter and smarter, the data that's needed to advance the models has become increasingly harder to generate. Right? The models are advancing in depth. They're improving in coding, STEM reasoning, et cetera. They're advancing in breadth, in multimodality, multilinguality, multi-industry, et cetera. And the models are becoming agentic, meaning the models can now execute complex, multi-step workflows in a real-world business context.

• 49:38Speaker 00

  Now with the models advancing like this, what's needed in a platform is you don't just need Iron Man, you need the Avengers. Frontier models need frontier data. Frontier data needs a team of frontier humans, right? So we have like PhDs from physics, chemistry, math, biology, expert Olympiad level coders, people who are literally at the pinnacle of their fields. And sometimes you have to have them working together to break the model. For example, in physics, if you, if an expert is evaluating the model in physics to test some theory in physics, you might wanna, a PhD in physics might ask the question to test the theory. A software engineer might need to build a simulation to test that theory.

• 50:22Speaker 00

  And then a data scientist might have to analyze the results of that simulation. So you need to daisy chain really smart humans together to generate data.

• 50:32Speaker 01

  Actually, Jonathan, can I talk to you about that? Because what you're describing to me sounds like very interesting tests. Having people with different domain expertises work together to find places where the model might have a weakness or might not be able to answer something. And then you say that they're generating data. Is the data they're generating simply where the LLM in question misses or doesn't meet expectations? Because I thought this was more like, here's new information for the LLM to be trained on versus looking at a model that's already been trained. And then, does that make sense? I feel like I may have had this backwards.

• 51:06Speaker 00

  So that's a great question, Alex. So the way these models are trained is there's a step called pre-training, where you basically feed the model gobs and gobs of data. The war in pre-training is mostly over, like almost all the models are trained on the same subset of the internet. Now the battlefield is post-training, and there's a new field called mid-training, which is, I can get into that later. But with post-training, what you do is you have these human experts create question-answer pairs, where the, a software engineer might ask a question like, hey, how do you create an app that connects dog walkers to dogs? And can you write it in both Swift and Kotlin for Android, right? And now the system has to generate the app.

• 51:50Speaker 00

  So that's an example of a supervised fine-tuning dataset, where you give it a prompt and a completion, and the model learns how to, how to, how to auto, how to respond to that prompt, right? So you need experts in different fields. Now, the reason this has gotten harder is two years ago, A very low skilled contractor was capable of creating tokens that could have advanced the model. Now, because the floor has gone up, now you need PhDs from Stanford, Berkeley, MIT to figure out where the models break. You first have to break the model. You have to ask a question that literally stumps the model. And then the humans create good question pairs of questions and answers that you then feed into the model for fine tuning and then the model learns.

• 52:32Speaker 01

  Ah, it's the question and answer pairs that generates the data for the model. Okay, now that makes good sense to me. So it sounds, though, like what you've done is, back to your Avengers Iron Man point, just got together like a, like a super set of nerds and basically you're applying the smartest humans to find the flaws in the most advanced L L Ms. What happens, Jonathan, when we don't have PhDs that can ask questions that the model can't answer? Because it seems to me that we've raised the bar, but we're raising it towards a ceiling.

• 53:02Speaker 00

  Yeah, yeah. I mean, I've oversimplified it a little bit, like, so I would say like, so one example I gave is you give question answer pairs. There's another type of data that you generate for reinforcement learning where you, the humans are generating questions and verifiers. They're not generating the solution, they're generating a way to verify whether the solution is correct or not, which you can do with certain fields like coding, math, hard sciences, you can do that, right? Write a program to sort numbers in Python, you can write test cases to verify whether the, the program that you wrote is correct or not.

• 53:42Speaker 01

  But you're not telling them every single step along the way, you're just saying, I can verify that what you've done, the work you've done, does generate an acceptable answer.

• 53:50Speaker 00

  Correct, correct. And, and, and the version of this for enterprise is you create an RL gym, a reinforcement learning gym. I think this is one of the coolest things in computer science, like, just like a gym where humans go to train, this is like a gym where an agent trains. And in these gyms, you create basically clones of different websites, like a DoDash or an Uber Eats or a, or a NetSuite or a Salesforce. You create this virtual environment with prompts or workflows and ways to verify whether the task is completed. Now, the task could be, hey, salesperson, why don't you research everything you need to about this prospect and then update Salesforce in this way?

• 54:33Speaker 00

  And as long as when you create this virtual environment, you create these prompts and you create these verifiers. Now, an agent tries out different combinations. The agent is basically trying to use the tools in an appropriate way to actually complete that task. And it's a cool way to learn whether the agent is learning through trial and error, how to execute a complex, multi-step workflow. You're not teaching the agent, go to LinkedIn first, go to Zoom info first, and then check out Salesforce for prior conversations. You're not teaching it that, but it's learning. It's learning just based on feedback on, okay, if I accomplished this, I get a reward. And then it learns how to operate in a way that maximizes future rewards. It's really cool. To your question of what happens if we run out of human intelligence, right?

• 55:18Speaker 00

  Fortunately, I think we are still a significant distance away from that. We've run out of internet data, but there is lots of intelligence trapped in the minds of humans that has yet to be transferred from human minds to machine minds. I would say, Alex, even the way,

• 55:36Speaker 00

  like, I love your show. I love what you and Jason have done with the show. But when you're thinking about how to interview, interview somebody who's on the show, what type of questions to ask, what type of prep to do, I guarantee that knowledge is not distilled into the models from OpenAI or Anthropic or Meta or Google. The only way that's going to happen is if we hire somebody like an Alex or a Jason to work on Turing, to use our tools. And we didn't get into this, but we have tools that make it easy to keep the quality of the data high. And we have AIs that will assist you in generating this data so that we can make this scale.

• 56:17Speaker 00

  Pun not intended.

• 56:18Speaker 01

  No, I was gonna let that one slide. We don't allow more than two dad jokes per episode. So we have to, we have to save them, if that makes sense. So here's my question. I, I, I understand that scale being subsumed by Meta has been great for the Turing business. You told me before the show, you guys are at nine figures of revenue and profitable and growing. And honestly, hell, yeah, I love that. But if I'm OpenAI, why wouldn't I try to replicate what Turing? Has built inside of my domain. Because if there's one thing that large AI model companies have today, it's a lot of access to capital. And so as we've seen from Meta trying to buy literally every human who's touched an AI before, there's a lot of money to play with here. So are you guys just so good that OpenAI doesn't want to try to replicate what you've built?

• 57:02Speaker 01

  Or is there something else that I'm missing in that it's good to have a third party versus an internal group be kind of telling you where you might want to work on things?

• 57:10Speaker 00

  Yeah, I mean, that's a great question, Alex. Firstly, to advance towards ASI, you need three things to move in parallel. Actually, you need four things. You need research and algorithms. You need compute, you need data, and you need the application layer. Those are the four things, right? Research, compute, data, and applications. Now, the labs are really good at research, right? And kudos to OpenAI, Anthropic, Meta, Google, Apple. All of these companies are advancing the frontier forward. So they're spending their energy there. We could flip this question to also ask, why don't they build their own compute?

• 57:52Speaker 00

  And in some areas, the labs are investing in it. I mean, we've heard of Google having TPUs and other companies trying to do that. But Nvidia clearly has the edge today. And there are some good companies like Grok, Cerebras, et cetera, that are also building custom chips. So compute is also advancing. And the third pillar is the data pillar, right? Now, generating this data is exceedingly complex. Again, three dimensions. It has to be complex. It has to be realistic, meaning it has to mirror how a real human would interact with a model. And third, it has to be diverse. It's very hard for a lab to get all of this in-house. I'll give you an example.

• 58:33Speaker 00

  Like today at Turing, this month alone, we are hiring PhDs across the board in physics, chemistry, math, biology, at the level of granularity of somebody who's an expert in dark matter, somebody who's an expert in black holes, somebody who's an expert in molecular biology, right? Like these very niche fields. So it's, and you ideally need these humans part-time because the way these humans are good at the job of training the models is because they are also good at their day job, which keeps their skills sharp. So you kind of, and once you ingested that knowledge, you might want to move to the next frontier and the next frontier and the next frontier.

• 59:13Speaker 00

  So you need a platform or a partner that can scale up very quickly to elite talent, manage the talent to like, make sure the talent is generating data part-time, the data is high quality. We have to build a ton of tools, like we have this platform called Allen, where AI is used upstream of the human to minimize the work for the human. AI works alongside the human and AI does quality control. It may sound a little dystopian. It's like, you know, AI overseeing humans to improve AI.

• 59:41Speaker 01

  I'm not afraid of our AI future personally. So that doesn't, that my, my P doom, I guess, is very low. But I want to go back to your black holes, dark matter point because I'm a bit of a science fiction nerd and I'm also a bit of a space nerd. So those are topics that are near and dear to my heart. But if I spend a lot of time helping a particular LLM better understand why we think about dark matter, why we came up with the idea, you know, galaxies and spinning and not having enough matter, blah, blah, blah, blah, blah. Will that work to make the LLM smarter in that particular domain have spillover effects to other areas? Does it make it more intelligent writ large or more intelligent only in that specific domain?

• 1:00:26Speaker 01

  I'm not sure of the answer here, so I figured I'd ask.

• 1:00:28Speaker 00

  Yeah. Phenomenal question. What we know for, what we know with relatively high confidence is that when the models get better at coding and math, they seem to get better in a wide variety of other tasks that have nothing to do with coding and math. Something about coding seems to teach the models how to think in a more structured way, how to communicate with less ambiguity, how to think step by step. So coding and math, there've been some experiments in how they have out of domain performance, outside of just pure code generation. And coding and math is also interesting because sometimes when you ask complex questions, Alex, the substeps involve being able to compute stuff or calculate

• 1:01:13Speaker 00

  stuff and pass the results forward. For example, Alex, if you had a question like, hey, what are some interesting themes in investing in AI? The answer to that might involve a model that knows how to write code to query PitchBook, write some Python code to analyze the data and say, you know, AI in healthcare seems to be spiking, AI in retail seems to be going down. That requires the model knowing how to write some Python code to analyze the data. It knows matplotlib to plot the results and show you the results. So coding and math have broader applicability. But it's also true that this is one of the mysteries of these language models. The model seemed to learn some representations about the world

• 1:01:56Speaker 00

  that seem to carry over to other areas. I can totally imagine you understanding human psychology. If the models were really good in human psychology, it could help the model write better website code that designs the website to be more persuasive, maybe like more conversion optimized, maybe makes it better for generating copy. So it may be, who knows, maybe there is something to learn from studying the universe that helps people in their day to day. It could be like some indirect thing, but we don't know for sure.

• 1:02:26Speaker 01

  So Jonathan, here's to a next great couple of years as the world gets bigger and faster and smarter. And just before I let you go, where can people find Turing on the internet? And is there a particular job that you're having a hard time hiring for that you wanted to shout out to the world?

• 1:02:40Speaker 00

  Great. Thank you, Alex. If you're working on AGI research or in human data, come talk to us like we are hiring across the board. And if you're building a frontier foundation model and you need data to make the model smarter at coding, reasoning, STEM, etc., come talk to us. We're at Turing.com and you can email me at [email protected].

• 1:03:02Speaker 01

  Perfect. All right, Jonathan, thank you so much. We'll have you back on in another six months when AI is twice as smart. And until then, this is Twist. Bye.

• 1:03:11Speaker 01

  All right. So here on Twist, we have talked ad nauseam about AI and the job market, mostly about how AI might impact the job market, change who does what job, change what jobs are done. You get the idea. People have a lot of concerns that AI might take all the jobs. We'll have to see what happens. Today, though, I want to talk to a company that wants to use AI to help people not just get jobs, but get the right jobs and also to have companies find the right people. Because as it turns out, and you know this, if you've done any hiring in your career, finding the right people pretty much is terrible, even with the modern tools we have today. So to help explain how this is going to work, please welcome to the show, it's Mercor, CEO and co-founder Brendan Foody. Brendan, hey, how you doing?

• 1:03:48Speaker 05

  I'm doing great. It's awesome to be here and appreciate your having me, Alex.

• 1:03:52Speaker 01

  It's my pleasure, man. I love doing these. Talking to people who are building what's actually next for the economy literally never fails to give me a jolt like a good espresso. So thank you. Now, one reason why I wanted to talk to your company is because you have this amazingly huge vision. You guys wrote that you founded the company because, and I quote, "The labor market is the largest, most inefficient market in the world," which is a pretty big claim. So before we dive into how Mercor is going to approach this and how you're going to get into the market, tell me why you think that and what brought you to that as the problem you wanted to solve?

• 1:04:24Speaker 05

  Yeah, it really comes down to this matching problem where when you were describing earlier, the reason that it's painful is that when a candidate is applying to a job, they can only apply to a couple dozen jobs. And when a company is considering candidates in the market, they can only consider a fraction of a percent of the people available that are looking for work because they need to solve this matching problem manually. They need to manually review resumes, manually conduct interviews, and manually decide who to hire. But when you're able to solve this matching problem at the cost of software, it makes way for this global unified labor market that every candidate applies to and every company hires from. And so that's

• 1:05:06Speaker 05

  the end vision of the company in the North Star that we work backwards from.

• 1:05:10Speaker 01

  Okay, so essentially, when I go out there to look for jobs, I might look at my local area, I might look at jobs at companies that I've heard of, but no matter how far I cast my net from my perspective, I'm still missing out on most of the gigs. And therefore, most of the companies aren't hearing from as broad a talent pool as they might.

• 1:05:26Speaker 05

  Exactly. Yeah.

• 1:05:28Speaker 01

  Okay, so in a world in which we're dealing with kind of the demise of remote work, isn't the labor market necessarily constrained by geography?

• 1:05:38Speaker 05

  Well, not precisely. I don't think we're dealing with the demise of remote work. I just think that in many ways, remote work might actually become even more prevalent. And part of this is that if you automate 90% of what it means to do remote knowledge work, that means that the bottleneck to productivity is the other 10%. And so for all the domains like software engineering, where demand is extremely elastic, we'll just produce 10 or 100 times more. And so I think that humans and the role that we play in knowledge work, both remotely as well as in person, is going to become amplified with these huge increases in productivity that are starting to happen.

• 1:06:19Speaker 01

  So just jisting that down for folks, do you think that in time, remote work is not going to go away and therefore there will be kind of a global talent pool? And therefore, if you want to access the best people, you're not only going to have to have a much larger pool, but you're also going to have to look more broadly at the corporate level.

• 1:06:37Speaker 05

  This is certainly the case. And most importantly, that I think it'll be centralized, right? Instead of all of these decentralized, fragmented job searches and people, companies that are looking to hire, I think it'll be one place that everyone goes, that every company hires from, that can facilitate all of these job matches that people love and companies are finding a lot of value in.

• 1:07:02Speaker 01

  Okay, and I think you want that one place, that central hub to be, Mercor, your company.

• 1:07:07Speaker 05

  Yeah.

• 1:07:08Speaker 01

  And if you succeed, this means that recruiters, as we know them today, are going to have a hard time. Probably generic, un-AI-enabled job boards like Indeed and similar are all going to get hit. So that's the scale of what you guys are shooting for, essentially a revolution of how people find and secure jobs around the world. That's fantastic.

• 1:07:29Speaker 05

  Well, so one way of thinking about it is there are sort of these two ends of the spectrum. On one end of the spectrum, there's companies like LinkedIn or Indeed that are job boards, and they have this very broad distribution, but they only aggregate this very thin layer of the person's resume, right? And so they capture about one thousandth of the value chain associated with facilitating a hire. On the other hand of the spectrum, there's the services companies, the recruiting agencies, the staffing agencies that do all the manual work and heavy lifting to get their 30% of first year or whatever their fee structure is. But no one has been able to combine the distribution of LinkedIn and Indeed with

• 1:08:10Speaker 05

  the value capture and value add of a recruiting and staffing firm, because previously it wasn't possible to automate services, right? But now that's all changing. Now it's becoming possible to automate everything that recruiters and staffing agencies would otherwise do in this unified way that has the same kind of distribution scale of consumer platform with hundreds of millions of people.

• 1:08:36Speaker 01

  All right. I wanted to start here because I want people to know where you're going to, but I now want to go the other direction and talk about what you're doing now, because you guys have this really interesting, quote, "public secret plan," if you will, of using, helping AI companies find the right talent that they need as essentially your wedge into the market. And to help explain this to folks, you guys want to learn how to place candidates very effectively in this one area where there's a short feedback loop, if you will, so you can learn pretty quickly and then apply that more broadly over time. But I think even pigeonholed by some in the press is like helping people hire for AI, when that does seem to be kind of a small fraction of the company's vision. So what I want to know is, how does it work today?

• 1:09:20Speaker 01

  I know you guys are working on one subset of the market, but walk me through, from the candidate and the company side, how Mercor actually does do what you're describing.

• 1:09:28Speaker 05

  So candidate will come to us looking for opportunities for work. They'll see a lot of the jobs available or they can apply to one of those or talent pool more broadly. They'll upload their resume and they'll take an interview with our, the AI on our platform to then get evaluated to see what they're good fit for, what they're interested in, what kinds of matches we might be able to create for them and facilitating, you know, huge volumes, tens of thousands of these matches with no human process and involvement. On the other side of the marketplace, there's companies that will give us these requests of saying they need 100 people in a particular domain, maybe a subset of software engineering

• 1:10:11Speaker 05

  or investment bankers or whatever the professional domain is. And we will go out to the supply side of our marketplace, find all the people that are a good fit for them and facilitate all of those contract opportunities.

• 1:10:24Speaker 01

  Tell me more about the, the AI interview and I'll just say that I'm an AI bull. I think this stuff is really cool and it's going to work well. On the other hand, I do have some friends that are in less prestigious jobs who've had to deal with some AI onboarding, some AI screenings, and they've been pretty negative about it. So I'm curious what you guys have built and how well it works and kind of what candidates think of that part of the process.

• 1:10:49Speaker 05

  Yeah, so we built the first AI interviewer in March of 2023 when I was in my college dorm room. And initially I remember it would hallucinate like, like nothing else because it wasn't even GPT-4 at that point, right?

• 1:11:01Speaker 01

  Oh, God.

• 1:11:02Speaker 05

  It was 10 second latency. And it's been extraordinary to just see this tailwind of model improvement, you know, lift all boats and make these applications possible. And so a good heuristic for it is ameliorating all of the processes that a human would otherwise do, where similar to how human would review resumes and conduct interviews, we automate all of the preparation for the interview and conducting the interview and evaluation of an interview with that similar format and heuristic. And while I think some candidates obviously prefer to talk to human because they think about it more as a selling process. From the company rather than strictly a buying process.

• 1:11:41Speaker 05

  The overwhelming sentiment is that there's millions of people that apply to jobs and just get completely ghosted and don't even get the opportunity to interview, right? It'll be like less than 1% of people even get to talk with someone or show more than their resume for the most competitive jobs. And so globalizing that ability and not just internationally, but also all across the US for people to talk with the candidates and actually consider everyone for the opportunity has been really impactful.

• 1:12:18Speaker 01

  Does the AI interviewer you guys have built in its current iteration, does it work with multiple languages? Because I presume that if you're only doing English, you're still constraining yourself to a small portion of the overall possible job takers.

• 1:12:30Speaker 05

  Yeah, we start out predominantly with English. Now we're starting to do others, working with a lot of our customers to try to make sure the models improve their multilingual capabilities so that the interviewer is well set up for that.

• 1:12:45Speaker 01

  And I presume that India is a market of choice for you guys. But are there any other hotspots around the world where you're seeing a lot of talent saying, hey, we really want to be part of this new marketplace that you're building?

• 1:12:54Speaker 05

  Yeah, well, so actually, the majority, the vast majority of our hires come from the US now, over 60%. India is the second largest geography that we hire from, but also seeing lots of Eastern Europe, lots of South America. One thing to note is that the average pay rate in our marketplace is over $90 an hour. And so it's very different from most labor marketplaces in that way. It's a totally different league from what you would find with the crowdsourcing platforms like Scale and Surge that sort of built these legacy labor marketplaces with more in the range of $10 to $30 an hour pay rates.

• 1:13:32Speaker 01

  You guys are currently targeting a much more educated and rarefied employee pool. Again, as your starting point, as your wedge. So I want to talk about how good your system is at finding and placing people because it sounds good in theory, but in practice, of course, that's what matters. So what's the right metric that you guys track in terms of finding the right person for the right job? And then having both sides of that equation be happy? Is it repeat business? Is it successful completion of a contract? What's the KPI there?

• 1:14:01Speaker 05

  Yeah, I think about it as a two-sided retention problem. And there's sort of leading indicators on each of those retention problems. So the supply-side retention or applicant-side retention is, of course, if they come back looking for work. And so we have models that predict what is the probability that they're going to be interested in a particular job opportunity that we present them with, based on all their prior experience, so that we can ensure we're retaining candidates very well. And then on the demand side, it's how well are those candidates performing? Where we collect all of the performance reviews of who's doing well for what reasons, and use that as our eval set, as our benchmarks to go internally on. How do we predict, based on the interviews, the kinds of people that are going to perform well,

• 1:14:45Speaker 05

  that are going to translate to a lot of value for our customers? And that's why our demand-side retention is so large. So now we're working with six out of the mag seven. We have 1,605% net revenue retention on an annual basis.

• 1:15:02Speaker 05

  And so it's sort of nuts.

• 1:15:05Speaker 01

  1,605%, so it's 1,600%, so 16x net revenue retention. For folks out there who don't know why that's funny, mature software businesses will often turn in a net revenue retention number of 115% or plus 15. So the number that you put out there is humorously large and implies that customers are buying not just a little bit more of the services, but 16 times as much over a one-year basis.

• 1:15:29Speaker 05

  Yeah, the two-sided retention problem is working well.

• 1:15:31Speaker 01

  Yeah, no, two-sided retention solution. I don't think problem is the right word. Okay, so to me, AI companies that need to hire experts is a very... It's a market that has a lot of reason to do well, to invest in new ideas, to try to find the right people, because they're in a massive race to build the best stuff and therefore capture the market. How well do you think the Mercor model is going to translate when you go wider and you're dealing with people that might be less individually, resume-wise impressive, and are more...

• 1:16:05Speaker 01

  Average folks applying for more average jobs because to me, difficult job, difficult person might actually be easier to match than finding the right person from a more general pool for a more general gig.

• 1:16:17Speaker 05

  Yeah, well, we do hire a lot of people from general backgrounds as well. Like, it's really, really broad across literally every industry and the economy. But maybe stepping back a little bit, the background of the company is initially, we were automating the processes of hiring people for our friends. And then Scale AI came to us and they used our platform to hire over 1,000 people. And what we realized was that there was this huge transition in the market away from the crowdsourcing paradigm of low and medium skilled talent towards this, you know, medium and high skilled sourcing and vetting problem, not only with higher caliber people, but also people that work directly with the AI researchers to help them interpret evals and push the frontier of model capabilities.

• 1:17:02Speaker 05

  And that is one of the core reasons that the performance data we collect and all the things we learn and how we facilitate better matches is actually much more similar to a general work environment than most people realize on face.

• 1:17:16Speaker 01

  When you say, Brendan, that you're collecting the evals, my impression of how this worked was I would come to you, resume, interview, job, and I would go work for someone else for some period of time. But if you're collecting the evals to ingest back into your process, does that mean that when I land a contract or a gig for Mercor that I'm actually working for you guys?

• 1:17:36Speaker 05

  So technically, the work product that our contractors are producing ends up going to the customers. What we own is just the performance review on like, did this person do a good job on the project?

• 1:17:50Speaker 01

  The end customer tells you how the employee did, and that's part of the overall arrangement. So the feedback does make it, even if they're working for someone else?

• 1:17:58Speaker 05

  Yeah, yeah. And one interesting thing is we facilitate the entire payment stack. So we learn from who's getting bonuses, who's getting raises, who's getting dismissed for what reasons, all as part of the data flywheel.

• 1:18:10Speaker 01

  Okay, so people love to talk about the usefulness of data and how the more you have, the smarter you can be. So how much better have your systems become after ingesting increasing amounts of performance reviews, bonus information, contract retention and so forth, all the stuff we've talked about? How quickly does that filter back in and actually generate a material and measurable improvement in your ability to place candidates?

• 1:18:32Speaker 05

  Yeah, pretty much on a weekly cadence. I mean, we went from where it's been most impactful and our focus has been is predicting people at the high end. And the reason is there's this dynamic on a project where if we're providing 100 people to work with a given customer, the top 10% of people are going to drive majority of the value. It's similar to like if you have a team of 100 software engineers, probably the 10 core people are these like 10, this notion of 10x software engineers that are driving.

• 1:19:00Speaker 01

  Are you telling me that power laws exist?

• 1:19:03Speaker 05

  Power laws exist, right? And many, in many knowledge work verticals, but the implications of that from our standpoint are profound because if we're able to build interviews and assessments and all this technology that can predict these power law outcomes, the amount of value that we drive for our customers and performance of those people and the quality is extraordinary.

• 1:19:25Speaker 01

  Okay, so let's play devil's advocate here because I'm always on one hand, a capitalist technologist, and on the other hand, I'm a person who has friends who, you know, have to make rent. So if you can help people find the 10x engineers and the 10x engineers only, like let's say you can just say, look, here are the top five people and here's 45 others you can hire if you want. Aren't we going to end up with a labor market in which companies are able to hire the best and get more out of them and then they're going to need fewer total people? I'm just worried about folks who are like B students.

• 1:19:57Speaker 05

  Yeah, well, so I sympathize with a concern, but one thing I've come to appreciate is that people sometimes over index on the dimension of how exceptional someone is and under index on how relevant their experience is. And that if you match people with the right thing that they're extraordinary at, you can unlock these phenomenal outcomes.

• 1:20:16Speaker 01

  All right, now I want to talk about results because you guys raised, I think it was a series B earlier this year, quite a large round, quite a large valuation. And I think TechCrunch said your revenue was somewhere in the realm of 75 million ARR. You guys have also talked about growing, you know, 50% a month at one point in time. So one, since your last funny round, has growth stayed as hot as it had before? And then when do you think will be the right time to use the wedge to expand your remit and try to take on a wider range of jobs and also longer tenure jobs?

• 1:20:48Speaker 05

  Yeah, so I'll answer the first part. The business has grown by an order of magnitude since we received our term sheet for the Series B. So the growth has been incredible.

• 1:20:58Speaker 01

  Happy investors, very happy investors.

• 1:21:01Speaker 05

  Well profitable the entire time. And so we have more cash in the bank than we've ever raised, which is, you know, unique for an AI company.

• 1:21:11Speaker 05

  And then to your second point of timing the expansion to new markets. The context for why we focus on the AI labs is we realize that there's more of a comparative advantage when we focus on hiring someone for five weeks versus five years. Like when we're hiring someone for five years, you want to get dinner with them, build a trust and a relationship. Five weeks, you want this fast, efficient AI interviews automated process. And so we're leaning into that a lot now in the wake of the scale news with them no longer in the market. There's so much market pull that we're focused on capturing that. But what a lot of people don't realize is that throughout the duration of the business, we've still been doing lots of contract hiring outside of the market for AI labs and lots of full-time hiring for ourselves and our friends.

• 1:21:55Speaker 05

  We still have the customers from 2023 before we even started working with AI labs that are continuing to grow with us. And so are starting to also ramp up investments and a lot of those other kinds of hiring work.

• 1:22:11Speaker 01

  So it sounds a little bit like I asked a question that was more relevant maybe a year ago. But instead you guys are expanding your work with AI labs and doing other things. So it sounds like the wedge is wedging and you guys are already growing your remit. Okay, now, before you go, two things. One, where can people find the company on the internet? And then two, is there a particular role that you're looking to hire for? And want to shout it out into the broader world to see if the right candidate is waiting for you? Which is ironic, I know, given the conversation.

• 1:22:36Speaker 05

  Absolutely, people can go find us at mercor.com, M E R C O R dot com. And we're hiring for a huge volume of roles, particularly lots of software engineers. So for any software engineers that are looking to join our team full-time, we're super eager to talk to you for anyone looking to join our marketplace, whether it's part-time or full-time as a contractor. We pay exceptionally well. We have phenomenal satisfaction on the marketplace and retention as we talked about. And so would love the opportunity to work with you.

• 1:23:09Speaker 01

  Do you use your own software as the place you source some of your own internal candidates?

• 1:23:14Speaker 05

  Of course, absolutely.

• 1:23:15Speaker 01

  So you do like the taste of dog food. All right. Well, Brendan, thank you so much. We'll have you back on in, I don't know, probably six more months when you do eventually raise more money, then I can ask you about that. But in the meantime, thank you and we'll see you soon.

• 1:23:25Speaker 05

  Yeah, for sure. See ya.