Weekly Recap | Elon vs. Trump, Ukraine's Drone Attack, Cluely Update & OpenAI CRO

(00:00) - Intro and Overview (02:57) - Ukraine’s drone attack on Russia (03:21) - Soren Monroe-Anderson (Neros) (09:02) - Connor Love (Lightspeed Ventures) (13:45) - Erik Prince (Off Leash) (15:20) - Elon Musk Vs. Donald Trump (21:44) - Delian Asparouhov (Varda) (32:48) - Cluely Update from dropout Roy Lee (46:52) - Kian Sadeghi (Nucleus) (01:00:12) - AI Day on TBPN (01:00:14) - Mark Chen (OpenAI) (01:28:49) - Sholto Douglas (Anthropic) TBPN.com is made possible by: Ramp - https://ramp.com Figma - https://figma.com Vanta - https://vanta.com Linear - https://linear.app Eight Sleep - https://eightsleep.com/tbpn Wander - https://wander.com/tbpn Public - https://public.com AdQuick - https://adquick.com Bezel - https://getbezel.com Numeral - https://www.numeralhq.com Polymarket - https://polymarket.com Attio - https://attio.com Follow TBPN: https://TBPN.com https://x.com/tbpn https://open.spotify.com/show/2L6WMqY3GUPCGBD0dX6p00?si=674252d53acf4231 https://podcasts.apple.com/us/podcast/technology-brothers/id1772360235 https://youtube.com/@technologybrotherspod?si=lpk53xTE9WBEcIjV

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00John Coogan?

  You're watching TVPN. This week, what were the top stories? What were the most interesting things that we learned?

• 0:07Jordi Hays?

  Favorite interviews.

• 0:08John Coogan?

  The Ukraine drone attack, that was huge. Operation Spiderweb, a whole ton of drones were smuggled into Russia in shipping containers. They emerged and went out and attacked bombers. We had Soren Monroe Anderson from Neros on the show to break that down for us. We also talked to Connor Love at Lightspeed, who does a lot of defense tech investing. And of course, a couple of weeks ago, we had Erik Prince on the show, the founder of Blackwater. And he had kind of predicted that the Ukrainian military was perhaps underrated. And we might be seeing like something like this in the future. And so that was interesting to see to see play out.

• 0:51John Coogan?

  So we will take you through those kind of interviews, recaps of those. Then obviously, we had the absolute meltdown between President Donald Trump and Elon Musk. That unfolded on X and Truth Social, the two leaders.

• 1:03Jordi Hays?

  Dueling.

• 1:04John Coogan?

  Dueling social media.

• 1:05Jordi Hays?

  Beaner, social platform.

• 1:06John Coogan?

  Exactly. And although it's a highly political story, there are big business implications.

• 1:12Jordi Hays?

  Totally.

• 1:12John Coogan?

  Talking about what's going to happen in space between NASA, Boeing, different launch providers.

• 1:18Jordi Hays?

  Yeah, the implications for Tesla, SpaceX, Neuralink, even the boring company, right?

• 1:22John Coogan?

  There's a lot of stuff.

• 1:23Jordi Hays?

  Many of Elon's businesses are heavily regulated.

• 1:26John Coogan?

  Yep.

• 1:27Jordi Hays?

  And the potential impacts are substantial.

• 1:30John Coogan?

  Yeah, then we also had had some earlier stage founders on the show. Roy from Cluely came on and went pretty viral, put on a show. You're surrounded by journalists, hold your position. Cluely is a is a service to help you cheat on everything. We got to actually try the app. Someone was asking, was pushing us to try it and see how good the product is. But regardless of the product, he's also a phenomenal marketer and he came on and put on an absolute show.

• 1:59Jordi Hays?

  And he's printing, apparently.

• 2:01John Coogan?

  Yeah, he's doing great.

• 2:02Jordi Hays?

  He can't spend all the money that they're bringing in.

• 2:05John Coogan?

  And so we'll give you an update on Roy and see where that business is. And then Keon from Nucleus came on to launch Nucleus Embryo, which he calls the first ever genetic optimization software that helps parents give their children the best possible start in life.

• 2:21Jordi Hays?

  Quite a lot of controversy there on the timeline this week. A lot of people hate it. A lot of people love it. And we'll let you kind of decide for yourselves.

• 2:31John Coogan?

  And then we had a whole bunch of AI experts on the show from Google, OpenAI, and Anthropic. Got to the front leading edge of the debates around AI, AI, LLM.

• 2:41Jordi Hays?

  Poor John yesterday. You were fighting as long as you could. You didn't want to talk about drama. You got dragged into it, but we did get some great coverage from Mark Chen at OpenAI, as well as Sholto over at Anthropic.

• 2:56John Coogan?

  Yeah, it was a lot of fun. The big news over the weekend was the Ukraine drone attack on Russia. They shipped shipping containers into, deep into Russia, at which point drones flew out of the containers and hit strategic targets. We're going to have two guests on the show today. Soren Monroe Anderson from Neros to talk about that, and also Connor Love from Lightspeed to talk about that, and also defense tech investing generally. Today, we have Soren from Neros, who builds drones and has been to the Ukraine. And so we'll bring him into the studio and ask him how he's doing. How are you doing?

• 3:30Jordi Hays?

  There he is. Welcome.

• 3:31Speaker 02

  Great. How are you guys?

• 3:32John Coogan?

  We're good. We have a new soundboard, so expect some wild, some wild cards.

• 3:35Jordi Hays?

  Wild stuff. Could you give us a high-level overview of the history of drone warfare in Ukraine? Because I understand it's been progressing super rapidly on both sides, and it'd be helpful to understand kind of the different stages.

• 3:50John Coogan?

  Did they ever have like Predator drones, like the global war on terror type of drone? Or did they jump straight to quadcopter and kind of like leapfrog the technology?

• 3:59Speaker 02

  So, you know, you've had this Russian aggression war in Ukraine since 2014. Obviously, the full-scale invasion was 2022. But even during that, that period before the full-scale invasion, there was some usage of drones for surveillance and dropping explosives. These are primarily still like small drones, like what you're seeing now. But this was not a proliferated technology. Then, when the full-scale invasion happened, within a few months, Ukrainians started thinking about all these ways that they could use, you know, inexpensive drone technology to get an asymmetric advantage. And that is where FPV drones started becoming a really, really big deal. So they pioneered the

• 4:41Speaker 02

  Really this idea of, you know, putting an explosive on a racing drone and using that as a precision strike weapon. There were instances of this happening in other places, but they really scaled it and they've really refined it. And then Russia was was much slower to take it seriously, although now they tend to in some ways outproduce Ukraine and they have a much, you know, more direct line to China, where most of these components are coming from. But since 2022 and FPVs just starting to get used, now it's reached an unbelievable scale. It's estimated Ukraine is going to produce four and a half million FPV drones this year. And those are ranging from, you know, ones that are this big to 15 inch propellers, fiber

• 5:24Speaker 02

  optic controlled drones, many different types and sizes of warheads, different configurations. And I can talk more about the drones that were used in in Operation Spiderweb as well because those were really interesting. But what we've seen is just this vast technology landscape where new clever ideas like fiber optic are, you know, going to be the the hot thing for a few months and then they sort of just become another tool in the tool belt and it's just this constant arms race.

• 5:49Jordi Hays?

  Yeah, talk about, talk about this, this attack was was unique in a bunch of different ways, but is this something that had been, to your knowledge, or or just, you know, more generally known to be something that had been attempted multiple times or, you know, maybe like, I'm curious to know, yeah, kind of the backstory on on this type of attack because it seems, you know, it's massive difference to be using this technology way behind enemy lines versus using it, you know, at the front line.

• 6:22Speaker 02

  Yeah, so primarily FPV drones are used on the front line, say the kind of 30 kilometer band across the zero line. What was so unique here is that it was FPV drones, short range drones, being used 4,000 kilometers inside of Russia. It was this unbelievable application where, you know, you've seen Ukraine using long range one way attack drones that are going, you know, 1,500 kilometers to strike targets deep inside of Russia. But here, these were small drones actually driven in on on trucks, basically in the tops of shipping containers. And I don't know of any operations that were similar to this beforehand. I think it was.

• 7:07Speaker 02

  Not something they wanted to give away. And the drones were actually operating on cellular. They were not operating on local, like the the normal low latency local radios you use for FPVs typically. And so I think, you know, this is going to be something that a lot of people are going to look at and and see. If you have drones that are operating on cellular, you can't really tell them apart from cell phones. It's really hard to defend against, really hard to detect. But now it's going to going to be part of airfa- air-based defense is thinking about drones that are operating on cellular being piloted from basically anywhere in the world.

• 7:42Jordi Hays?

  Talk about the Russian response, the immediate response to this incident. From the footage that I saw, and I think most people saw that that tracked it. It seemed incredibly challenging to respond to it quickly, right? By the time you could sort of organize a response, a lot of the the core damage had been done. What do you think the the the question I think that every country is asking themselves now is how do you defend against this type of attack? Whether you're at war like Ukraine and Russia are, or you're just, you know, thinking, you know, long term.

• 8:20Speaker 02

  Yeah, this clearly poses a massive threat to critical infrastructure. I mean, being blatant, the US does not have any defenses in place that would stop this from happening. We already know, there's already news stories about drones that are flying over our air force bases, and we can't do anything about it. And I think the only approach here has to be a multi-layered system, where you're looking at all of the different types of electronic warfare, and also considering things like satellite communications and cellular communications, where you're basically able to turn those off on the flip of a switch, which is a huge a huge inconvenience and a huge thing to build into the infrastructure. But clearly, that's going to be required.

• 9:02John Coogan?

  Welcome to the stream, Connor. How are you doing?

• 9:05Connor Love?

  I'm good. I'm doing all right. Good to be back, guys.

• 9:07John Coogan?

  Yeah.

• 9:08Jordi Hays?

  Oh, he's got a suit this time.

• 9:09John Coogan?

  Oh, looking great. We love to start.

• 9:11Connor Love?

  I won't say I dressed up just for you, but you know, I would have taken the suit off far before this, you know, if I wasn't coming on.

• 9:17John Coogan?

  Fantastic. Thanks so much for jumping on. Have you been tracking the Ukraine story closely? Any insights there? Anything in the portfolio that's at all relevant in the defense tech world? Do you expect a response from the US government or guidance or change to any strategies? Really, any takes on that?

• 9:37Connor Love?

  I mean, first shit, what a, what a time to be alive. I mean, you know, I'm sure your Twitter feeds and your group chats were going up, pun intended, over the weekend. I mean, it's pretty crazy. I mean, let's be honest, like, first, I'm not shocked that the Ukrainians did this. I mean, the execution seemed to be flawless from what we can pull from, from open source intel. I do think, though, I mean, again, it's not a surprise. The Ukrainians have been mastering drone warfare for the last handful of years and, you know, you want to call it that, you know, they called it spider web. Like, this was their, this was their Trojan horse. This was their, you know, Israeli beeper. And the outcome is, is, is pretty impressive, to be honest. I mean,

• 10:18Connor Love?

  what, what, from the outside looking in, like, the Russians woke up over the weekend and they thought they were getting their $4 team orders and what did they get? They got 1,000, you know, FPV drones, you know, blowing them to smithereens. So, it's pretty impressive. I mean, my takeaways from this are really twofold.

• 10:34John Coogan?

  Please.

• 10:34Connor Love?

  The first is like, there's never been a clear signal of where warfare is going. And to be clear, what, you know, what I, what I view this from, you know, both the entrepreneurs in my portfolio, but also from my perspective, I mean, the world is about, you know, cheap, attributable, a lot of times autonomous systems and that's, you know, playing out in warfare, that's playing out in other areas of life. And then the second thing is, you know, candidly, it's like, It's really hard to defend yourself at the pace at which things are changing. And again, like, I know we do some things here in the United States and are trying to be on the front end of a lot of this innovation. But when this happens, I think this almost just resets everyone again and says, all right, how do we respond to it?

• 11:17Connor Love?

  And I think it's, to your point, it's not a, it's not a direct US response. It's more of, hey, what do we need to buy? What do we need to develop, you know, for our own fight in, you know, in some way, shape or form?

• 11:26Jordi Hays?

  Yeah. What do you think, obviously, you're a venture capitalist, not a geopolitical strategist, but what's the right Russian response to this? Is it, hey, we suddenly need to be wary of having cell coverage anywhere near strategic military assets? I mean, it seems like Ukraine, in Ukraine, in Ukraine's perfect world, they could run this style of attack a bunch and copy and paste and hit other targets. But it feels like something that was dependent on cellular technology that that's something that the Russians can revoke, you know, fairly, fairly quickly. Sure, it'll be inconvenient, but I'm curious if you have, have a take.

• 12:05Connor Love?

  Yeah. I mean, you know, to be honest, when I think about, how do you defend against this? I think there is, you know, I wouldn't call this the easy answer of just, you know, turning off, off the cellular network. I actually think the only way to do it kind of practically is in layers or in a multitude of, of different ways. Because, you know, yeah, the, you know, the reality is if you looked at how the Ukrainians carried out this attack, they did so on the local, you know, Russian cell network. Which, again, I don't think any Russian kind of defense unit, any of these bases, was ever thinking that they would have to turn off their own cell network. And then there's just the practicality of how you do it. I mean, I think there was, what, four or five different attacks that hit all at the same time? What do you, what do you do?

• 12:49Connor Love?

  You, you turn off the network for tens of thousands, hundreds, thousands of people? And oh, by the way, this is like a dirty little secret that nobody talks about. You know, yes, you have your military systems that are protected and all that, but a lot of coordination is happening through WhatsApp, a lot of coordination on, and so all of a sudden, you turn off the cell networks, you're actually inhibiting your own defense, your own response, the first responder, you know, the, you know, getting your own people out of there. So I think it's a bit more complex than that. And then the last thing I'd say is just like, Even if you do this in layers, you know, you need to be resilient in a way, but you're not going to stop everything. I mean, this was just brilliant masterclass of, you know, again, if maybe there was a plan, we didn't know this, but maybe there's a plan

• 13:31Connor Love?

  for, you know, 100 bases and we only hit five of them. And so if you think about just the broad, you know, geopolitical, you know, geographic coverage you have to have, I think to be 100% certain on anything is just you, it's impossible. You can't do it.

• 13:46Speaker 04

  The most capable military in Europe right now is the Ukrainian military. The lessons learned that they have are very significant. The drone tech is far and away the best. Their ability to do fight against and to even conduct electronic warfare and even close air support in this environment is leaps and bounds ahead of even what the US military is. So that's the military to learn from. Ukraine does have a corruption problem. I hope sincerely that Trump is able to get a ceasefire in place and to stop this killing because it's absolutely pointless. It's just Slavs killing Slavs at this point

• 14:28Speaker 04

  and it's, it's, nobody's going to advance. And you have a blend of old and new. I mean, if you look at the pictures of the front there now, it's almost indistinguishable from the Battle of the Somme, right? Artillery duels, static lines, bunkers, all the rest. Now, the problem is somebody can fly an FPV into your bunker on the other side, but between tens of millions of landmines, which make armored breakthrough very difficult, it slows down any attack so that the FPVs and artillery can get to it, you're not going to see any kind of blitzkrieg, Hanskiderian maneuver warfare there until some significantly different

• 15:13Speaker 04

  weapon systems come along. So, look, Europe needs to get serious about it. They're far from it at this point.

• 15:21Jordi Hays?

  Elon posts four minutes ago, "The Trump tariffs will cause a recession in the second half of this year." Wow. Somebody else was saying, "Can I finally say that Trump's tariffs are super stupid?"

• 15:34John Coogan?

  Who's that?

• 15:35Jordi Hays?

  Somebody else is posting, uh, Mads posting is saying it's Xi Jinping. He says, bro, you seeing this? And it's Putin on the other end. He's just looking at it. Hold up, got a line and it's, we'll start pulling some of these up.

• 15:51John Coogan?

  Ridiculous. Uh, what else is going on here? This is the president versus Elon.

• 15:57Jordi Hays?

  Naval says Elon's stance is principal, principled. Trump's stance is practical. Tech needs Republicans for the present. Republicans need tech for the future. Drop the tax cuts. Cut some pork. Get the bill through.

• 16:13John Coogan?

  This is so crazy.

• 16:14Jordi Hays?

  Antonio Garcia says, remember, there's F you money and then there's F the world money.

• 16:21Jordi Hays?

  Will Stansel says, imagine being the ICE agent suiting up for your biggest mission of all time right now. People are saying that Trump's gonna deport Elon.

• 16:31John Coogan?

  Back to South Africa.

• 16:34Jordi Hays?

  Will Depew says, time to drop the really big bomb growing Daniel is in the Epstein file.

• 16:40John Coogan?

  Turn it into a coffee pasta.

• 16:41Jordi Hays?

  That is the real reason.

• 16:44John Coogan?

  Oh, no.

• 16:48John Coogan?

  Delian.

• 16:50Jordi Hays?

  We had a question from a friend of the show said, the real question is if Tesla is down 14%, how could SpaceX and OpenAI be trading if they were, how would they be trading if they were public? The, the real thing here is it's bad for everyone, right?

• 17:06John Coogan?

  It feels bad for everyone, right?

• 17:07Jordi Hays?

  Coin is down. Nobody's really winning here. China's up.

• 17:11John Coogan?

  Yeah.

• 17:12Jordi Hays?

  Uh, Sean McGuire, I mean, I'm just saying like at a high level.

• 17:15John Coogan?

  Yeah, yeah, yeah.

• 17:16Jordi Hays?

  You know, China is the big beneficiary here of, um, uh, Sarah Gross says, if anyone has some bad news to bury, might I recommend right now?

• 17:26John Coogan?

  Yes, yes, yes. If you have, if you, uh, what, what, what's the canonical bad startup news? Like, oh yeah, you missed earnings or something. Drop it now.

• 17:37Jordi Hays?

  Inverse Kramer says, Bill Ackman is currently writing the longest post in the history of this app.

• 17:45John Coogan?

  And we have, we have a video from Trump here if we want, I can throw it in the tab and we can share it on this, on the stream and, and react.

• 17:54Jordi Hays?

  Lex, Lex Friedman says to Elon, that escalated, quickly triple your security, be safe out there, brother, your work SpaceX Tesla XAI Neuralink is important for the world.

• 18:06John Coogan?

  We need to get Elon on the show today. If somebody's listening and can make that happen, I would love to hear.

• 18:11Jordi Hays?

  Max Meyer says, so I got this wrong. I didn't say it never happened, but I thought it wouldn't. I'm floored at the way this has happened.

• 18:17John Coogan?

  Yeah.

• 18:18Jordi Hays?

  He didn't think they would have a big breakup. Many people didn't think they would have a big breakup. Even just earlier this week, it seemed like they might just have a somewhat peaceful exit. Trump just posted a little bit ago, I don't mind Elon turning against me, but he should have done so months ago. This is one of the greatest bills ever presented to Congress. It's a record cut in expenses, 1.6 trillion dollars and the biggest tax cut ever given. If this bill doesn't pass, there will be a 68% tax increase and things far worse than that. I didn't create this mess. I'm just here to fix it. Anyways, lots going on.

• 18:56John Coogan?

  Let's go to this, this Trump video. I want to see.

• 19:00Speaker 05

  The criticism that I've seen, and I'm sure you've seen regarding Elon Musk and your big, beautiful bill. What's your reaction to that? Do you think it in any way hurts passage in the Senate, which of course, what is your seeking?

• 19:11Donald Trump?

  Well, look, you know, I've always liked Elon and it's always very surprised. You saw the words he had for me, the words of, and he hasn't said anything about me that's bad. I'd rather have him criticize me than the bill because the bill is incredible. Look, Elon and I had a great relationship. I don't know if we're well anymore. I was surprised because you were here, everybody in this room practically was here as we had a wonderful send off. He said wonderful things about me. You couldn't have nicer. Said the best thing. He's worn the hat. Trump was right about everything and I am right about the great, big, beautiful bill. But I'm very disappointed because Elon knew the inner workings of this bill.

• 19:53Donald Trump?

  Better than almost anybody sitting here, better than you people. He knew everything about it. He had no problem with it. All of a sudden, he had a problem. And he only developed the problem when he found out that we're gonna have to cut the EV mandate, because that's billions and billions of dollars. And it really is unfair. We want to have cars of all types, electric, we want to have electric, but we want to have gasoline, combustion, we want to have different, we want to have hybrids, we want to have all, we want to be able to sell everything. He hasn't said bad about me personally, but I'm sure that'll be next. But I'm, I'm very disappointed in Elon. I've helped Elon a lot.

• 20:28Speaker 07

  Mr. President, I just want to clarify, did he raise any of these concerns with you privately before he raised them publicly? And this is the guy you put in charge of cutting spending. Should people not take him seriously about spending an hour? He's saying this is all sour grapes.

• 20:41Donald Trump?

  No, he worked hard and he did a good job. And I'll be honest, I think he misses the place. I think he got out there and all of a sudden, he wasn't in this beautiful oval office and he was, and he's got nice offices too. But there's something about this when I was telling the chancellor, this is where, as people come in.

• 20:58Jordi Hays?

  Breaking news, Delian Asgharouhav, who's joining us in the temple for some live reactions.

• 21:06John Coogan?

  Surprise guest. I can't even spell surprise guest. I'm so excited about this.

• 21:13Jordi Hays?

  Yeah, in other news, 11 Labs dropped a new product.

• 21:20John Coogan?

  Another news, $2 million seed round. Stop it, stop it. We love 11 Labs.

• 21:28Jordi Hays?

  No, they'll, they'll.

• 21:30John Coogan?

  Keep grinding, but.

• 21:31Jordi Hays?

  Just launch again tomorrow.

• 21:32John Coogan?

  You're going to have to launch again. Start shooting a new vibe reel, start shooting a new, writing a new blog post, because no one's going, Lulu says yes, delay the launch on TVP.

• 21:43Jordi Hays?

  So basically, right now, I can just pull up and just refresh. I'm gonna just be refreshing Truth Social.

• 21:51John Coogan?

  So, okay, Jordy's on Truth Social, I'll be on X. Give us a reaction, Delian, what's going on?

• 21:57Delian Asparouhov?

  I mean, at some point, I was like, I'm just, you know, sort of scrolling X and I like tuned into you guys like an hour ago. And I was like, they're talking about some AI thing. I was like, at some point, they're gonna switch to like breaking news. And then I was watching and I was like, okay, like.

• 22:08Jordi Hays?

  John resisted.

• 22:10John Coogan?

  I fought it for like, for like a half an hour, but we couldn't do it. But yeah, give us your quick reaction.

• 22:18Delian Asparouhov?

  I mean, I'll always, you know, sort of give it from the, you know, sort of space angle. You know, it's amazing that, you know, how much the world has shifted since, you know, Friday of last week, whereas you sort of presumed the Jared Isaacman was going to be the, you know, sort of NASA admin to today. It was released that the Senate reconciliation package re-added budget back into NASA largely for the SLS program, which was basically the program that, you know, Jared and Elon were, you know, sort of largely advocating to, you know, sort of completely shut down. So, you know, the, it is already show, like, you know, the sort of counter reaction, you know, is already showing up, you know, in, in policy.

• 22:57John Coogan?

  Sorry, SLS program, is that Space Shuttle or no?

• 23:00Delian Asparouhov?

  Uh, sorry, that's the SLS launch rocket. It is based off of old Space Shuttle hardware, but it is basically the internal, you know, sort of NASA run competitor effectively to like a Starship heavy, you know, launch rocket.

• 23:13John Coogan?

  Yeah.

• 23:14Delian Asparouhov?

  So, you know, because it was, you know, sort of generally behind budget, behind schedule, and there are so many commercial heavy lift rockets coming online.

• 23:20John Coogan?

  Sure.

• 23:21Delian Asparouhov?

  The default was canceled. That is largely, you know, sort of a Boeing based program. And so, you know, if you look at, you know, you know, three months ago, you know, when they were announcing that 47 program, you know, Elon walks into the Secretary of the Air Force's office, obviously he'd been, you know, sort of ranting against, you know, manned fighter jets and believing that that shouldn't be what, you know, be what the department is prioritizing. Thirty minutes after that meeting was when they announced the F-47 program. And so now you're seeing basically like the equivalent in space where, you know, you know, that was obviously awarded to Boeing. Boeing was the, is the largest prime behind, you know, SLS. You know, Boeing basically, you know, is going to be the biggest winner of, you know, NASA refunding, you know, sort of SLS and Jared Isaacman not being NASA administrator.

• 24:01Jordi Hays?

  So tying this back to the timeline, Trump posted less than 30 minutes ago, "In light of the president's statement about cancellation of my government contracts, SpaceX will begin decommissioning its Dragon spacecraft immediately." Break that down.

• 24:15Delian Asparouhov?

  I mean, that just means that we no longer have a vehicle that can go to the International Space Station. We no longer have a vehicle with astronauts up and down. You know, we also don't have a vehicle that can de-orbit the International Space Station safely, right? The Dragon was expected to be able to do that. So what that means is, you know, if you guys remember all the memes about stranded, you know, from last year around Boeing Starliner, it now means that the space station, you know, itself is basically, you know, sort of stranded. And that's like, you know, one of the government contracts, obviously, that you know, SpaceX is involved in. Elon, I've heard generally, like, just wants to shift all things to Starship anyways. And so in some ways, was probably kind of looking for an excuse to, you know, sort of shut down Dragon and refocus energies. There's also a part of it where it's like, look, he is like, kind of independent in the space world

• 24:56Delian Asparouhov?

  in that, you know, Starlink's total top line revenue is going to be passing the NASA budget in the next year or two. And so in terms of like, size of, you know, state actor that can influence space, you know, his own company is basically about to become, you know, as large of an actor as like, the entire United States. So I don't think there's going to be like, a de-escalation here. Like, you know, my, my, you know, estimation is like, on both sides, it's going to continue to escalate. You know, if we thought that we lived in dynamic times, you know, when Trump got into office, it's going to be even more dynamic when there's like- The dynamism will continue until morale improves. Elon, the center, AOC, the progressive populist, and Trump, the, you know, sort

• 25:38Delian Asparouhov?

  of conservative populist. And, oh, man, it's, uh- It's remarkable times.

• 25:44Jordi Hays?

  I mean, I just have so many questions, right? How does this impact Golden Dome?

• 25:49Delian Asparouhov?

  Yeah, what's Boeing's stock doing?

• 25:51Jordi Hays?

  Is, is, will Golden Dome even be a viable project without SpaceX? It's, it's-

• 25:57Delian Asparouhov?

  I think there's just going to be more resistance, probably, to working with, you know, sort of upstarts, because they would be ones that would probably be more likely to collaborate, you know, sort of with, you know, SpaceX. And so-

• 26:07John Coogan?

  Well, it feels like Boeing would be a logical beneficiary of this turmoil and yet they're down today. They haven't really popped.

• 26:16Delian Asparouhov?

  Oh, really?

• 26:17John Coogan?

  Yeah.

• 26:17Delian Asparouhov?

  I mean, I'm not obviously, you know, one to give like, you know, public.

• 26:19John Coogan?

  Yeah, I know. I'm just trying to work through it myself and it's surprising.

• 26:24Delian Asparouhov?

  It's like Tesla dropped and Boeing to pop, basically.

• 26:27John Coogan?

  Yeah, yeah, that would be the expectation, but there must be something here because it feels like this is purely interpersonal between Elon and Trump and not, it's not like, oh, Boeing was secretly behind the scenes the whole time lobbying even more effectively. It doesn't, well, where's the tinfoil hat?

• 26:43Delian Asparouhov?

  That's fair.

• 26:45John Coogan?

  Maybe we need a tinfoil hat segment, who knows? But yeah, I mean, when you're in Boeing world, it's like, hey, we're only down 1%. Let's go. The coup of the century.

• 26:54Jordi Hays?

  My question is, has there ever been a crash out of this magnitude ever?

• 26:59John Coogan?

  In history?

• 26:59Jordi Hays?

  In internet history?

• 27:00John Coogan?

  When Elon and Trump became friends?

• 27:03Jordi Hays?

  Honestly, world scale.

• 27:04Delian Asparouhov?

  I actually. Probably world history equivalent. I feel like there was something in like the building era in the United States where, you know, there was some.

• 27:11Jordi Hays?

  Crashing out, crashing out used to mean calling up the New York Times and just ranting. Now you can just live post like all your reactions and it's just all real time. This is like crash outs are actually intensifying. You actually want to be long crash outs.

• 27:26John Coogan?

  Yes, definitely.

• 27:27Jordi Hays?

  Over the next 20 years.

• 27:28Delian Asparouhov?

  And you have their own social media platform that they own. So, you know, you gotta be on both X and Truth Social to like stay on top of things.

• 27:35John Coogan?

  Yeah, I actually did like a deep research report a while back on like, has the richest man in America ever been close with the US president? Going back to like, you know, was Rockefeller particularly close?

• 27:47Delian Asparouhov?

  Yep.

• 27:48John Coogan?

  And because the narrative was like, oh, this is like so unprecedented. And in fact, it is unprecedented.

• 27:53Delian Asparouhov?

  Oh, really?

• 27:54John Coogan?

  Yeah.

• 27:54Delian Asparouhov?

  Interesting. I would have guessed that like Rockefeller was close.

• 27:56John Coogan?

  Me too. Me too. That's what I was going for was like, no, I imagine this is always, it is always close. But no, I think because the president has become more powerful globally, your point about, you know, mayor of America, dictator of the world, like it becomes increasingly valuable for the richest man to have a close alliance. And so it's become more, I don't know exactly. How accurate that research was is totally possible that like behind the scenes Rockefeller was really close to the president at the time. And we just didn't write about it in the history books, but there certainly aren't very many anecdotes about the richest man in America going on.

• 28:30Delian Asparouhov?

  Yeah, so Pavel had a. For AP US history 2050, you know. Yeah, so this is. Where Elon Musk called the president at the time, a potential pedophile. Was it A. Epstein Island, B about a cave in the Philippines, C.

• 28:47John Coogan?

  What a mess.

• 28:48Jordi Hays?

  No, so Pavel had a good post. He was quoting the the big bomb from Elon. He said hypothetical question about the USA's power structure. Is the man with the most access to capital more or less powerful than the political head honcho? Purely hypothetical. It's a good question to ask.

• 29:06Delian Asparouhov?

  I mean, I think both like archetypes have grown both in absolute power, but also in relative power to the rest of the globe. Since the Gilded Era, right? If you think about like the president of the United States in 1925, I'd say pretty darn powerful. But like there was clear like, you know, it was a sort of multipolar, you know, sort of world. Argentina was pretty darn rich at the time. Obviously, Europe was still, you know, sort of recovering from World War One, but UK was generally, you know, sort of doing well. Like it was not, you know, it was clear there was a, you know, sort of huge, you know, outweighed effect. And then if you look at probably the, you know, sort of biggest, you know, industries at the time. You, I don't think you could claim that even like Standard Oil at its peak, I'd have to go look at the exact numbers.

• 29:47Delian Asparouhov?

  But that like it had the size of budgets relative to like, you know, the like US government in terms of, you know, sort of budgets, right? Versus I feel like now for the first time, you both have, you know, sort of US president extremely, extremely powerful. And then you have like, you know, sort of mag seven effectively like the size of, you know, sort of, you know, huge states. Like they're, you know, fucking their own state governments.

• 30:07John Coogan?

  And then also just more bureaucracy, more red tape. So like I, when I think about the 1920s, like Robert Barrens, it's like it is the, it is the you can just do things era. And so you want to build a railroad like, yeah, you might need to get like one rubber stamp, but it's not going to be 10 years and tons of lobbying and all this different stuff. So you can kind of just go, you can just go wild.

• 30:25Jordi Hays?

  You know, it's bad when Kanye is saying. Rose, please know, we love you both so much. It's just like the voice, the voice of reason is Kanye West.

• 30:35John Coogan?

  Yes, thank you.

• 30:36Delian Asparouhov?

  We need to bring them together and, uh, you know, form a peace treaty.

• 30:39Jordi Hays?

  Nikita Beer just added his pronouns back to his bio.

• 30:42John Coogan?

  Let's go.

• 30:43Delian Asparouhov?

  He's got a rubber band. Elon's got a rubber band all the way back to, you know, sort of extreme wokeism. Straight back to, uh, you know, sort of super climate change and.

• 30:51Jordi Hays?

  Wow, somebody's, somebody's sharing, uh, resharing the picture of the, the cyber truck blown up in front of the Trump tower. I guess it's just like this.

• 30:59John Coogan?

  This is real.

• 31:00Delian Asparouhov?

  It was foretold. Yeah.

• 31:03Delian Asparouhov?

  But it was a question of like when and what magnitude, not if.

• 31:09Jordi Hays?

  Always bad if, if Vladimir Putin is operating to negotiate between, uh, President Trump and Elon. I think, I think a lot of the world is, is waiting for Roy Lee's take, clearly, and the clearly army.

• 31:24Delian Asparouhov?

  That's who we believe.

• 31:25Jordi Hays?

  They want that. People have been asking him to get involved with geopolitics.

• 31:29John Coogan?

  Wow.

• 31:32Delian Asparouhov?

  I love the, uh, Shil Mohad put up a, uh, you know, sort of meme about, uh, Narendra, the like, uh, Prime Minister of India. Um, you know, he basically, uh, copied and pasted the, uh, Trump Truth Social post about negotiating peace between India and Pakistan when it wasn't like actually fully negotiated. And, uh, you know, posting about, you know, uh, you know, negotiating a ceasefire between, uh, Elon and Trump.

• 31:53Jordi Hays?

  Funny thing is like Truth Social, you can just read all of Trump's posts without creating an account. It truly shows that like, I would think that you would have to make an account to read them all, but they just, it's not gated at all.

• 32:05John Coogan?

  It's.

• 32:06Jordi Hays?

  This could be the biggest, but you know, they clearly, I don't think they care about monetization. Bitcoin is actually, uh, falling alongside.

• 32:15Delian Asparouhov?

  Falling.

• 32:15Jordi Hays?

  Falling.

• 32:17Delian Asparouhov?

  Wow. Bitcoin falling, Boeing falling, Tesla falling. Who's the biggest winner of the day?

• 32:20John Coogan?

  I, I think it's China.

• 32:22Delian Asparouhov?

  China.

• 32:23John Coogan?

  Yeah.

• 32:23Delian Asparouhov?

  China, China.

• 32:24Jordi Hays?

  Sean McGuire.

• 32:25John Coogan?

  Really sold off. It's, it's down 3% today at 101K. So still up, but you know, uh, rough.

• 32:32Delian Asparouhov?

  Winnie the Pooh just dipping his, uh, you know, hands in that pot of honey, just snacking away, watching from the sidelines.

• 32:38John Coogan?

  Yeah. Let's see Chinese stocks, U S stocks, Chinese, I can't find.

• 32:43Delian Asparouhov?

  Okay. That's probably my commentary on the day, boys.

• 32:46John Coogan?

  Anyway, this is great. It was fantastic having you.

• 32:47Delian Asparouhov?

  Thanks for jumping on.

• 32:48John Coogan?

  Thanks for hopping on so quickly. Next up, we have Roy from Cluely coming back for an update. He's hired 50 interns, I think, or something close to it.

• 32:59Jordi Hays?

  He said they're bringing every intern on.

• 33:00John Coogan?

  They're bringing every intern on.

• 33:02Jordi Hays?

  We got every intern coming in.

• 33:04John Coogan?

  Well, welcome to the studio, Roy. How are you doing?

• 33:06Jordi Hays?

  Boom, let's go. There they are.

• 33:12John Coogan?

  I think we're overpowering you. Can you hear us?

• 33:16Speaker 09

  Yeah, yeah, we can hear you.

• 33:17Jordi Hays?

  Yeah, make sure we're zoomed out all the way so we can see everybody. We got a small army here.

• 33:21John Coogan?

  This is incredible. How big is the team? Kick us off. How many you got at this point?

• 33:26Speaker 09

  The team is 11 full time plus the interns.

• 33:29John Coogan?

  How many interns you got so far?

• 33:31Speaker 09

  Interns, bro, we're closing in on 50, brother.

• 33:37John Coogan?

  That's amazing. Congratulations. What are they all doing? How do you manage everything? Is it just, is it purely social media? Is that what you want them to focus on? Growth?

• 33:48Speaker 09

  Yeah, yeah, growth, marketing, like the only goal of the company is get 1 billion eyeballs on to Cluely. So you have unrestricted creative freedom and permission to do anything and everything. Just make the company go viral. Every single person you see behind you has over 100,000 followers on some social media platform.

• 34:05John Coogan?

  Wow.

• 34:06Speaker 09

  10,000 plus.

• 34:07John Coogan?

  That's remarkable.

• 34:09Speaker 09

  Me too now.

• 34:10John Coogan?

  Yeah.

• 34:11Jordi Hays?

  There we go.

• 34:11John Coogan?

  Oh, yeah, you probably popped. What's working? What platforms have actually been, been driving the most growth? I mean, I'm sure you've run a lot of tests. What have you learned that you can share?

• 34:22Speaker 09

  Bro, Ben, take it away, bro.

• 34:24John Coogan?

  That's it.

• 34:25Speaker 10

  UGC has been really good. We just hit 10 million views today.

• 34:28John Coogan?

  10 million views.

• 34:29Speaker 10

  Eight days.

• 34:30Jordi Hays?

  Wow, there we go.

• 34:32Speaker 10

  Hoping to get 100 million views in the next month.

• 34:34John Coogan?

  What platform specifically are the most fertile ground for targeting your specific customer? Because you can imagine that there's a lot of folks who are AI curious on X, but then there's much broader, more viral audience, more general audience on platforms like TikTok, YouTube, Instagram. What's working and what is the next, next platform that you're going to be focused on?

• 34:56Speaker 09

  Yeah, well, we're trying to go viral on every platform regardless. Um, but the main thing right now is Instagram reels.

• 35:02John Coogan?

  Oh, Instagram reels. Interesting. And what is the main value prop that you're hitting people with? Is it still the cheat on test thing? Or have you evolved at all? What? Still?

• 35:11Speaker 09

  Lucky interviews. Yeah.

• 35:12John Coogan?

  Interviews. Okay. And, uh, has there been, i- i- i- this was controversial when you launched it. Is it still controversial in the comments? Are you getting flamed? Has anyone big dunked on you? And has that driven virality? Is that actually a net positive?

• 35:28Speaker 09

  I- Instagram is not like Twitter. Like you could post the craziest shit on Instagram and they will still not think it's controversial.

• 35:34John Coogan?

  Really?

• 35:34Speaker 09

  So how to make it controversial? Like we have to engagement bait some other way. Like it's cheating tool is controversial on Twitter, but on Instagram, you could, you could have like a white guy say the N word 10 times and it's still not controversial enough. Like, like you need crazy shit on Instagram. That's what we cracked. Every single person here has like very great viral sense.

• 35:50John Coogan?

  Yeah.

• 35:51Speaker 09

  And watch the reels that do go viral. You see, there's like ways that we've engagement baited the videos and this is what we'll keep doing to a probably a billion views a month is what.

• 35:59Jordi Hays?

  How long does it take to figure out if an intern is cracked? Is it like an hour, two hours? How much time do you need?

• 36:05Speaker 09

  For me personally, me personally, probably like 10 minutes, but for anybody watching, probably would take like one, one or two weeks.

• 36:10Jordi Hays?

  There we go. There we go. Uh, how do you guys think about, how do you guys think about product marketing? Obviously you're just going viral everywhere, getting all this attention. How do you make sure that it, that it, uh, he's shaking his head.

• 36:23Jordi Hays?

  Doesn't think about, it's not about the product. It's about the attention.

• 36:28Speaker 09

  You guys can make anything go viral.

• 36:30Jordi Hays?

  Yeah.

• 36:32Jordi Hays?

  Yeah, but, but, but how do you? The side of the street, you know, you, uh, make some UGC videos, make some Twitter posts, you know, you can sell anything. You know, in 2025, product doesn't matter. You know, I could jack off off the side of a building, sell some videos of it for 20 bucks each. Make $2 trillion. It's crazy. Two trillion. That, that's intense. Um, how do you guys think about burn? Uh, is it on your mind? Is it on your mind at all?

• 36:54Speaker 09

  I don't know if you saw the last tweet, but as of literally like two days ago, we're still we're still cash flow positive. We're still fucking profitable.

• 37:01John Coogan?

  We're so profitable.

• 37:02Jordi Hays?

  Let's give it up for profitability.

• 37:03John Coogan?

  Let's hear it.

• 37:04Jordi Hays?

  It sounds.

• 37:06John Coogan?

  So you're charging for the products and people are paying. Are they at all satisfied? Or did they feel like they got scammed?

• 37:12Speaker 09

  Bro, like the product works. You're either using this as a consumer and it's working because like, like you're passing your interviews and/or if it doesn't work, you're not going to complain to me because I'm gonna go right to your employer and tell them, yo, guess who's complaining about using the product? Like, I'll get you blacklisted if you complain really strongly.

• 37:29Jordi Hays?

  Where, how are you thinking about, how do you, how are you guys thinking about product evolutions? What do you want to add to the product? Obviously, you want to help people cheat on everything. Where, where are you going to help people cheat next?

• 37:39Speaker 09

  We don't care about, like, the product is going to be led by the virality of the content. We have video ideas right now that we're going to try to push for different use cases. We're going to see which ones go consistently the most viral. If you can make something go more viral, then like you can just build the technology after you have all the attention. So we'll figure out the exact use cases and exact niches we're going to quintuple down on once, once these guys get to work.

• 38:02John Coogan?

  What, what, what formats on Instagram reels are like the most modern in terms of consistently viral? Like you mentioned, like man on the street interviews, what do you do for a living? That's always been fertile ground. What about, I see a lot of those like mobile game ads that look like, you know, you're fighting down some sort of bridge and then you go into the game. It's actually just a match three. What, what, what are the different formats that you like to pull from?

• 38:28Speaker 09

  Every week, there's two new ones and at any point, there's probably 10 to 20 viral trends that is happening. And these cycles so quick, you need to keep your finger on the pulse. These things will like expire immediately. Need to be on the ball. And like, if I, if I told you right now, by the time people watch this on YouTube, like it would all been expired.

• 38:45John Coogan?

  Well, they were live. So, so give us, give us the latest and greatest. Like what's going viral today?

• 38:51Speaker 09

  Well, right now we got 10 million views using a Snapchat format.

• 38:55John Coogan?

  Okay.

• 38:56Speaker 09

  Viral for like the last three years, to be honest.

• 38:58John Coogan?

  Okay.

• 39:00Speaker 09

  And, and I, I think that like, we just have to get people who continuously scroll TikTok like six hours a day.

• 39:06John Coogan?

  Yeah. But what's the, what's the actual format that you use? Like, describe the video. What is the hook? Like, break it down for me like you're explaining the, like, the art behind the viral format.

• 39:19Speaker 09

  There's a caption. It starts with a face, usually a handsome dude or a pretty girl. They're saying, damn, this interview started with the interviewer started with the hard questions. I should have been a CS major, not a business major. That's in game because people are saying like, bro, like CS is way harder than business. Then it turns around, the interviewer asks like, like, hey, how are you doing? Why should we hire you? And then this guy uses Cluey to generate a response, but he can't fucking read the response. So he reads it hella autistically, like, oh, I revel in detail. And then that's, that is like another conversation point. Like, people are cooking on the guy because he, uh, he can't read properly. The guy is like a, doing a really dumb interview using Cluey.

• 39:54John Coogan?

  That's great.

• 39:54Jordi Hays?

  How are you guys using AI-generated content internally? I know a lot of these, the videos that you guys are creating are just typical social media, vertical video. Do you have an intern that's just generating, uh, basically copy and pasting, making a bunch of other?

• 40:09John Coogan?

  VO3 or any of these tools relevant?

• 40:11Jordi Hays?

  Anything clicking?

• 40:12Speaker 09

  Not, not yet. I think there's still like a 10% left before they cross the uncanny valley. And the biggest thing is that people need to think your video is real. That, like, like, that is the difference between 100K views and 10 million views if people think it is real.

• 40:26Jordi Hays?

  Clearly, AI CEOs perish on AI.

• 40:30John Coogan?

  What about, uh, yeah, what?

• 40:33Speaker 09

  Google needs like 10 more Chinese researchers to like figure it out. And once, once they push out the latest update, then, then VO3 will be there. But right now we need real people.

• 40:41John Coogan?

  Yeah, yeah, yeah. Uh, well, I mean, what about just using AI as like stock footage replacement? Not, not as the lead in for the video, not the entire video, but just like sprinkled in to illustrate a point. You know, an establishing shot of like a building, a helicopter pulling into a building like that. That historically has been kind of something that you would reach to, uh, you know, Adobe Stock video for. VO3 feels like it's there, but are you not drawing on that at all yet?

• 41:10Speaker 09

  If there's a viral format when we need it, maybe we'll use it. But right now, like it's, it's really brain dead to go viral on Instagram.

• 41:17John Coogan?

  Yeah.

• 41:18Speaker 09

  Formats are not hard. You don't need a helicopter. You need a guy, a camera, a really shitty camera. You need a computer.

• 41:23John Coogan?

  I mean, what about like those, those kind of like AI mashups like Harry Potter, Balenciaga, or the, uh, the kangaroo with the plane ticket getting on the plane? Like, like AI content can go viral when it's really, uh, when it's like inspired almost by a human. It's not, it's not entirely AI generated, but it's using the tools effectively to create something that's like still catchy. Uh, do you think you'll be using any of that anytime soon?

• 41:47Speaker 09

  Prob- probably very soon. We're scaling up. Like what you see right now is probably about less than 1% of what the size will be by the end of this summer. Like we are profitable. We're not trying to be profitable. We just keep making so much money. We can't help it. So we're really scaling this shit up. I'm not even trolling you. 1,000 creators are going to be shipping on content. We're doing a complete internet takeover.

• 42:07John Coogan?

  Okay. So, so, so, so, so why in-house? Why, why, like, like, why, why do they even have to be employees? What, couldn't you turn this into like a multi-level marketing scheme or something? A pyramid scheme?

• 42:16Speaker 09

  That's what we're gonna do. That's exactly.

• 42:17John Coogan?

  Oh, that's what you're gonna do? Okay.

• 42:19Jordi Hays?

  MLM.

• 42:20John Coogan?

  MLM.

• 42:22Jordi Hays?

  Are you guys worried that you could be infiltrated by journalists? I'm sure they're circling the house right now.

• 42:28John Coogan?

  The hit pieces are gonna come. Yeah. We're, we're doing a softball interview right now.

• 42:32Jordi Hays?

  I mean, the, the person that's brave enough to try to do a hit piece on the Cluely Army is...

• 42:38Speaker 09

  I bet they're dying to. Look, more, more eyeballs is better. There's no companies that ever died from a founder being too controversial. You got Deal fucking infiltrating with genuine spies and they're still doing fine, bro. You got fucking Worker 17 guys. They're, they're, they're still kicking. Like no company ever dies from being too controversial. You die because you don't make enough fucking money.

• 42:59John Coogan?

  Yeah, yeah, yeah, yeah. Uh, speaking of making money, uh, what, what's the pricing model right now? Are you doing anything on price discrimination? Is there a super high tier if you get a whale? What does a Cluely whale look like? Can I spend $2,000 a month on this service?

• 43:11Jordi Hays?

  Yeah, you should add a tipping feature too.

• 43:13John Coogan?

  Yeah.

• 43:14Jordi Hays?

  People should be able to tip you guys if, if they have a good experience, maybe they get the job.

• 43:18John Coogan?

  Like really financialization, pay as you go, high interest rate loans, just really push it. Make it sports gambling in there. Maybe there's, throw it all in.

• 43:25Speaker 09

  Yeah. I mean, it's $20 a month for a hundred dollars a year and, and our, our top line revenue is really being driven up by enterprise. And we're gonna have to talk to the sales team to get a custom quote, but you know, like... There's a lot of

• 43:38Jordi Hays?

  Are you serious? What, whatever, is that more on the sales side? What, who are the enterprise?

• 43:44John Coogan?

  So you sell the SDRs?

• 43:46Speaker 09

  You guys laugh because you think I can't sell enterprise because I'm

• 43:48Jordi Hays?

  No, I, I, I don't believe it. I trust.

• 43:51Speaker 09

  Like these 20, these Fortune 500 CEOs, like these are like 35 year old dudes who sit there and put are laughing at my posts.

• 43:57John Coogan?

  Yeah, yeah, yeah. Yeah, no, it seems, it seems legit. It makes sense.

• 44:00Jordi Hays?

  No, I, I, I believe it.

• 44:01John Coogan?

  But, but I mean, you're, you're not going even higher tier? Like, what's the $2,000 a month clearly vision?

• 44:07Speaker 09

  For, for consumer, there's a lot more we can do with more compute, but right now we're like, to be honest, I didn't expect to grow this fast. The edge team is quite small. I'm like, spend a lot of time trying to hire more, more competent engineers. We have a lot of backlog tasks that we need to fill out, especially for this last contract that we signed. So we're full-time focusing on the one big guy that we got right now. And after that, um, then, then we'll, we'll try and scale this up. But right now we're focused on the one, one big client that we signed.

• 44:32John Coogan?

  Yeah.

• 44:32Jordi Hays?

  Uh, talk about your compensation strategy. The people want to know, uh, you, you said, uh, you can raise infinite capital and you're so confident. I believe you. Uh, but, but I'm curious to, to get some more insight there.

• 44:46Speaker 09

  Bro, I feel like it's so retarded to be a company. Sorry, am I allowed to say that?

• 44:49Jordi Hays?

  No, you're not allowed.

• 44:50John Coogan?

  No, this is a family-friendly show.

• 44:52Speaker 09

  It's very stupid to be a company. I try to race to the bottom to see how little you can pay your employees. Bro, if I'm making hella money, we're all making hella money. Like, like it's, I'm trying to pay them more to see if, man, like maybe tomorrow we'll start being like cash flow negative, but I just can't make a lot of money, bro. Like I, I would like to pay these guys what they're worth. And the output is fucking insane. We did 10 million UGC views in what, like eight days? Like, like you don't see this sort of traction in any company. And you don't see killers like this in any company unless you're paying these motherfuckers like what they're worth, bro. Like, I don't know what like 130

• 45:24Jordi Hays?

  Maxed out contracts, maxed out contracts.

• 45:26Speaker 09

  Exactly, exactly.

• 45:27John Coogan?

  Yeah. Uh, what about devices? I mean, it seemed like this would be a natural fit for some sort of AI wearable or other platform. Um, is there an app coming or are you interested in what's happening with Johnny Ive and OpenAI? What's your take on the device world?

• 45:43Kian Sadeghi?

  We're very interested in the hardware space. We've got like a million things cooking on hardware. We've got people in the garage right now working on shit you don't even know about, bro. Like, like, we're bringing manufacturing back to America and it all starts at the Cluey Garage.

• 46:00Jordi Hays?

  Let's go. I love to see it. Nobody, you know, they doubted, but you guys are re-industrializing America. You guys really are the

• 46:07Kian Sadeghi?

  They're working on brain chips down there.

• 46:09Jordi Hays?

  Yeah, brain chip.

• 46:09John Coogan?

  Brain chips. That's the future.

• 46:11Jordi Hays?

  There we go. There we go. The new Neuralink. Yeah. I mean, I, you know, there's a world in the future where you guys actually just roll up Neuralink and OpenAI.

• 46:18John Coogan?

  For sure.

• 46:19Jordi Hays?

  The Cluey umbrella.

• 46:20John Coogan?

  Yep, definitely.

• 46:21Jordi Hays?

  It's possible.

• 46:21Kian Sadeghi?

  I'm excited to offer acquisitions for, for both of those companies.

• 46:24John Coogan?

  Yeah.

• 46:25Kian Sadeghi?

  It's in the roadmap.

• 46:26Jordi Hays?

  It's on the road. All right. This has been a lot of fun. I'm excited for you guys. Uh, it is, uh, and, uh, I have no doubt that you'll go from, you know, 10 million views a week, 10 million views a week to a hundred. Uh, and I'm excited to see you guys hit that billion view mark very soon. So keep it up. We are all very entertained and, uh, rooting for you.

• 46:47Kian Sadeghi?

  Xie xie, xie xie.

• 46:49John Coogan?

  I love the energy. Thanks, man. We appreciate you joining.

• 46:52Jordi Hays?

  Better guys. Keep having fun.

• 46:53John Coogan?

  Bye.

• 46:55John Coogan?

  Next up, we have Kian from Nucleus coming on, uh, with a big announcement, something like 10 years in the making, close to it, maybe seven years. Uh, we'll bring Kian in. Let's play some soundboard. How you doing? Welcome to the show. Wow.

• 47:11Kian Sadeghi?

  Tweets are flying. Oh my God. You guys seen this?

• 47:14John Coogan?

  Yes.

• 47:15Jordi Hays?

  You seen this? Seen this?

• 47:18John Coogan?

  Break it down for us. Explain, explain what's happening.

• 47:21Jordi Hays?

  There's nothing like a launch day.

• 47:23Kian Sadeghi?

  I'm trying to figure out, guys, is this, is it Gattaca or is it Theranos? Because people can't, they can't make up their mind.

• 47:28John Coogan?

  Oh, yeah. We're gonna find out.

• 47:32Kian Sadeghi?

  What's going on? Let's give some context to the audience. Nucleus has launched Nucleus Embryo, the world's first genetic optimization software. Basically, parents can, you know, give their children the best start in life. They can pick their embryo based off of physical characteristics like eye color, IQ, they can go to disease risk like cancers or heart disease. Basically, we really believe parents can get all the information that exists about their embryos. And they can pick however they want. Um, for me personally, you know, it's been 10 years in the making. The journalist I actually covered it today in the Wall Street Journal was a journalist that covered my gene editing in a warehouse in Brooklyn 10 years ago.

• 48:05John Coogan?

  Yes, wow, overnight success.

• 48:09Kian Sadeghi?

  You know, it's a long time in genetics.

• 48:11John Coogan?

  Yeah. So, so, so break down the state of the art because like embryo screening exists. I think most parents in America, at least if they have the means, do some sort of screening while the embryo is growing. Is this purely for IVF? Is this just going a layer deeper? And then is, I want to talk about the regulatory and FDA component as well.

• 48:32Kian Sadeghi?

  Yeah, let's talk about it. So basically if you go to an IVF clinic today, you're a couple. The vast, vast, vast majority of clinics, the first thing I should understand is that the IVF process is principally controlled today by clinicians or doctors. Honestly, couples don't have as much liberty in our perspective as they should. It's their baby, it's their embryos, they should have the right to those, that information, and they should have to pick off any vertical. However, today in the clinic, what generally happens is people test embryos for very rare and severe genetic conditions. For example, like chromosomal abnormality, like Down syndrome, for example, or even a condition like cystic fibrosis or Tay-Sachs or PKU, right? These are conditions that are very rare, um, that maybe someone might have a carrier for cystic fibrosis, but again, it's, it's pretty rare. Then there are conditions that we've all heard of, heard about, things like breast

• 49:16Kian Sadeghi?

  cancer, things like coronary disease, the things that actually kill the vast majority of people today, right? Chronic conditions kill the vast majority of people today. Those conditions are just not tested for in the clinic, even though we have very good science actually that can make those predictions. How do we know this as a, as a DNA company? Well, that's what we do, right? We build models that predict disease and the way you test those models in adults. So we go from adults to embryos is actually because we can basically well validate these models to show that they work in both the embryonic context and in the adult context. And so what we're really doing is we're going from, okay, instead of just looking for really severe like Down syndrome or cystic fibrosis, why not do breast cancer? Why not do heart disease? Why not do colorectal cancer, right? Why not do schizophrenia? Why do Parkinson's? But then why stop there? And this is really the important thing because ultimately, you know, if you think about

• 49:58Kian Sadeghi?

  diseases and traits, the extreme version of any, uh, a trait is actually a disease, right? Height is a great example of this. One extreme end is like, you know, John, for example, he's like Marxism. Then the other end is like me, dwarfism, right? Both ends, okay? So, you know, so, you know, IQ is another example of this. One end is like, you know, autism. The other end, it can actually be some sort of, you know, cognitive basically challenge that people have. And so when you think about it, when you start realizing that people have drawn a line in the sand saying, you can get, you know, rare diseases, you can get common diseases, but then they really said you can't, can't get any traits like height, even though the best predictor we have today, actually in the world, the best polygenic predictor is for height. So as a company, we've kind of completely reimagined this and said, wait a second, what's going on here?

• 50:42Kian Sadeghi?

  You should have access to the entire stack. Rare diseases, we do. Cystic fibrosis, common diseases like breast cancer, and also traits all the way up to something like IQ.

• 50:51John Coogan?

  Yeah, so, I mean, that test, are you just giving people the data? Because I imagine that once you get into particular recommendations, that's more of what I would expect a licensed doctor to need to do.

• 51:05Jordi Hays?

  Well, yeah, my sense is that you can allow people to get the data from their doctor and then, and then feed it into Nucleus. Is that correct?

• 51:13Kian Sadeghi?

  So that's, that is correct. And actually, we have a couple, there was like 10 announcements today. You know how we do it. We like to do 10 announcements in one day. We are actually very, very excited to announce a huge partnership with Genomic Prediction. Genomic Prediction is actually the oldest embryo testing company that exists. They've done genome-wide testing embryos for almost a decade at this point. And I think they've done over 120,000 couples for PGT, which is a specific kind of test. And so we're actually partnering with them. So we make it very easy for Genomic Prediction customers to request their files and actually port it over to Nucleus. But really, this isn't just for Genomic Prediction customers. Anyone who's undergoing IVF can go to their clinic and say, I want my embryo's data. You can take that data, you can upload it to Nucleus, and then all of a sudden, you know, the application of DNA

• 51:56Kian Sadeghi?

  makes this technology universally, basically universally accessible.

• 52:00John Coogan?

  Now, how much of, how much of the benefit is, is actual algorithmic analysis bringing in other data points to contextualize the data versus just better UI and better hydration of existing tech? Because we, we, we, we had, we had a friend on the show who was talking about Getting some medical results from a doctor, the doctor's office was closed. It took two days to until the doctor was going to be able to interpret the results. He was able to just take a photo, upload it to ChatGPT and say, hey, is this, is this, you know, is this really, really bad? Should I be panicking? Because it seems somewhat out of the range. And ChatGPT was able to say, hey, you still got to talk to the doctor, but this isn't, this isn't the craziest thing I've ever

• 52:43John Coogan?

  seen. This is way out of distribution. And so that's almost like a pure UI layer, but extremely valuable. I know it might not be like the right narrative for some people that it's like not as innovative, but I think that like all that matters at the end of the day is giving people benefit, right? It's always both.

• 52:58Kian Sadeghi?

  You, you, you fundamentally, technology, just for technology's sake, it's not siliconized about, right? Siliconized about making something that people want. Okay. And people can actually use. Exactly. Think about the nucleus innovation. It's, it's two pronged. Okay. One is in the informatics, right? You know, I've been doing this for five years. I almost, I would argue to myself that I probably spent too much time, you know, developing the science, right? Science in a nutshell isn't actually very useful. You need to expand in access to it. So on that point, we do multiple different kinds of analyses. They make it such that we can actually provide the most comprehensive analyses that exist today. But moreover, and this is really the, I think a key point to your point, John, is people understand them. People can see them. I mean, you can pull up the platform. I'm not sure if you guys have shown already, but it's very easy to sort, compare your embryos. You can

• 53:42Kian Sadeghi?

  actually name your embryos, you can stack rank your embryos, you can understand what the score means. We lead with overall risk, or we tell you, for example, instead of saying you're in the 99% off for genetic risk for a condition, which, you know, what does it actually mean? We say, hey, you have a 5% chance, or the like of, let's say schizophrenia, or some other condition. In other words, by leaving overall risk, people have much greater intuitive understanding of the results. We're communicating to them. We have genetic counselors on hand. So this really is, what are we showing here? Are we showing something? Are we showing the?

• 54:05John Coogan?

  Yeah, yeah, we pulled up your website.

• 54:07Kian Sadeghi?

  That's another thing. That's a, that's a fun one. That's an Easter egg.

• 54:12Kian Sadeghi?

  That's the that's the kind of approach that we're taking here. And I think consumers are responding to it, right? People want to have access to their data.

• 54:18John Coogan?

  Yeah.

• 54:19Kian Sadeghi?

  The clinician, the doctor shouldn't decide what embryo to implant. You should.

• 54:22John Coogan?

  Okay, so talk to me about what requires FDA approval. Obviously, new medical devices. Like, if you were developing a machine to take in an embryo and sequence the DNA, I would expect that the FDA would want an approval for that medical device. But if you are taking data and just showing it to a customer in a different UI, that feels like probably a very light FDA process. And then there's probably a continuum in the middle where, once you're making a recommendation, they have rules around that, right?

• 54:54Kian Sadeghi?

  We, as a company, do not tell you which embryos to implant.

• 54:57John Coogan?

  Sure.

• 54:57Kian Sadeghi?

  You know, basically, parents, the couple has complete agency to decide how they want to use the information to implant their embryo. Moreover, let's be clear. Height, right? I mean, can a height analysis be a medical device? You know, that doesn't even make sense, right? IQ, height, these traits, for example. We all, you know, traits are something that I don't think actually belongs in even the kind of infrastructure of thinking about medical care, right? These are things that go beyond medical care. These are things like, you know, that that people just kind of intuitively know and that there are DNA tests done every single day, D to C, for these analyses, because they're not disease analyses, right? So we do both diseases and traits, to be clear. My point is, many of these innovations, you have to wonder, like, you know, should the government say if someone can or cannot pick their embryo based off height? That doesn't seem right to me. I think it should be in the complete liberty of the individual to decide that.

• 55:42John Coogan?

  Yeah, but I mean, we're a democratic country. And so if, you know, a huge swath of the population says that the FDA should review that type of test or that type of analysis.

• 55:53Kian Sadeghi?

  Height analysis?

• 55:55John Coogan?

  It could happen. I mean, the FDA reviews all sorts of different stuff. And so I guess the question shifts to like, do you expect a change from FDA on the way these these analysis tools are regulated?

• 56:12Kian Sadeghi?

  I think right now, the most important thing is just putting these high quality rigorous scientific results in people's hands and then helping them basically have healthier children, helping them give their child the best start in life.

• 56:23John Coogan?

  Yeah.

• 56:24Kian Sadeghi?

  Um, you know, I, I think that generally speaking that, you know, people should have more liberty, more choice in, in medicine. I think the broader longevity trend actually touches on that point as well.

• 56:34John Coogan?

  Yeah.

• 56:34Kian Sadeghi?

  Um, so that's what we're excited to do at Nucleus.

• 56:36John Coogan?

  Yeah. I mean, the fact, the fact that you're partnering with a, a company on, on actual, on the actual like medical device side, like they are doing the sequencing of the embryos, like that really takes it out of the Theranos question entirely in my mind. I think you feel like you should be beating the drum there a little bit more. It's like, like we didn't say we created some new device, but I don't know.

• 56:54Kian Sadeghi?

  We ship. We ship. That's the difference. We ship. It's locked in stock, baby. Don't look at it. Go use it. That's, that's the evidence. Who is there putting, okay?

• 57:02Jordi Hays?

  Um, I love the visual of John and his wife selecting between embryos and it's like six, 10 or seven, two. Tough choice. Well, if we go with the six, 10, uh, he has, you know,

• 57:18John Coogan?

  Potentially fly commercial once in his life.

• 57:20Kian Sadeghi?

  Actually, we can actually play this game right now. Okay. Here, we're going to play a game right now. I'm going to put in the chat, pick your embryo.com. Okay. Everyone listening to this, pick your embryo.com. I'm going to go to it. Oh my God. Here we go. Little Easter egg here. Okay. Let's see what's more important to you, John, intelligence or muscle strength? Come on.

• 57:37John Coogan?

  Oh, absolutely. Muscle strength. Let's go. We're the future's bodybuilder.

• 57:40Jordi Hays?

  Okay. John would, John would take a, he would, he would happily have a five, two son if he had, you know, top point oh, one percent bodybuilding genetics.

• 57:50John Coogan?

  Exactly. Yeah.

• 57:51Kian Sadeghi?

  Okay. So what's your lifespan or height? Come on, lifespan.

• 57:55John Coogan?

  Lifespan. Let's go. Let's go.

• 57:57Kian Sadeghi?

  Let's go.

• 57:58John Coogan?

  Maybe low depression. You gotta be golden retriever mode. You gotta be, uh,

• 58:02Kian Sadeghi?

  You need low depression.

• 58:03John Coogan?

  You need low depressions. Let's go low OCD. I don't mind bouncing around a bunch.

• 58:08Kian Sadeghi?

  Okay. Risk taking anxiety.

• 58:10John Coogan?

  Uh, let's go high risk taking. There we go. Okay.

• 58:14Kian Sadeghi?

  Okay. We're analyzing.

• 58:15John Coogan?

  Is this some generally stuff going on? This is great. I got Nadia too. The enduring athlete. Let's go. Physically strong, cautious, built to last. Yeah, this is great. Is this driving a lot of, a lot of attention, a lot of downloads? Is this going viral yet? This seems like something that's designed to be shareable.

• 58:33Kian Sadeghi?

  I think we just dropped it right now. Technology Brothers, we got you the exclusive.

• 58:35John Coogan?

  Let's go, let's go. Let's put it out there.

• 58:37Kian Sadeghi?

  They can pick your embryo. People say, what's it like? Maybe not doing IVF yet. No problem.

• 58:41Jordi Hays?

  Funny, only, only 9% of people choose Nadia.

• 58:44John Coogan?

  Okay, well, we're, we're contrarian. We like that here. That's fun. It's great. Oh, well, well, congratulations on the news. Congratulations on the launch.

• 58:55Jordi Hays?

  Yeah, the pace is wild. Last, last thing, what's going on with, have you seen these Just Blood billboards?

• 59:00John Coogan?

  Oh, yeah.

• 59:00Jordi Hays?

  They're all over LA.

• 59:03John Coogan?

  So, so there is, there is someone who's running a campaign right now, justice for Elizabeth Holmes, claiming that Theranos was not the scam people think it was. And there's a, there's a documentary coming out and there's billboards all over LA for Just Blood. Like, it's just blood. It's not that big of a deal.

• 59:19Kian Sadeghi?

  And, and John, just to be clear, there's an exclusive on Technology Brothers next week about, from this person, right? They, they're gonna tell their story next week, just to make sure. You, you invited them already. I hope.

• 59:28John Coogan?

  We're, we're, we're, we are, we are toying with the idea that, that someone reached out to kind of connect us. We're, we're thinking about doing it, but we're not, we're not 100% sure that it would be appropriate for the show.

• 59:38Jordi Hays?

  Based on the website, I don't know if, I don't know if it's appropriate.

• 59:41John Coogan?

  Yeah, it doesn't look like it was designed with Figma, so I don't know. We can't quite do it. It's a little bit.

• 59:46Jordi Hays?

  The team definitely doesn't use linear.

• 59:48John Coogan?

  Yeah, but they, they, they claim that, that Elizabeth Holmes has been proven innocent. And so it's a bold claim. We like, we like to see people making bold claims.

• 59:58Jordi Hays?

  By what, uh, jury is my question.

• 1:00:00John Coogan?

  Yeah, jury of someone who knows HTML.

• 1:00:04Jordi Hays?

  Keon, uh, the energy is off the charts.

• 1:00:07John Coogan?

  Electric.

• 1:00:08Jordi Hays?

  Electric.

• 1:00:09John Coogan?

  Thank you for coming on, firing us up. Congratulations on the launch. We will talk to you soon.

• 1:00:13Kian Sadeghi?

  I'll see you on Twitter for sure, okay?

• 1:00:15John Coogan?

  Yeah, we'll see you there.

• 1:00:16Kian Sadeghi?

  Bye, guys. Bye.

• 1:00:18John Coogan?

  We have someone from OpenAI here. We're going to stick to technology and business, but welcome to the show, Mark Chen. Good to see you.

• 1:00:25Mark Chen?

  Good to see you guys.

• 1:00:26John Coogan?

  Awkward day, but I'm excited to talk about deep research. I am excited to talk about AI products. Would you mind introducing yourself and kind of explaining what you do? Because OpenAI is such a large company now and there's so many different organizations. I'd love to know how you interact with the product and the research side and anything else you can give to contextualize this conversation.

• 1:00:47Mark Chen?

  Yeah, absolutely. So first off, you know, thanks for having me on. You know, I'm Mark. I am the chief research officer at OpenAI. So in practice, what that means is I work with our chief scientist, Jakob. And, you know, we set the vision for the research org. We set the pace. We hold the research org accountable for execution. And ultimately, we really just want to deliver these capabilities to everyone.

• 1:01:09John Coogan?

  That's amazing. In terms of research, I feel like a lot of the, what happens in the research side is actually gated by compute. Is that a different team? Because what if the researchers ask for a $500 billion data center? That feels like maybe a bigger, a bigger task.

• 1:01:24Mark Chen?

  Yeah, it is useful for us to factor the problem of research and also kind of building up the capacity to do that research. So we have a different team. Greg leads that, which really thinks holistically about, you know, data center bring up and how to get the most compute for us. And of course, when it comes to allocating that compute for research, you know, Jakob and myself do that.

• 1:01:45John Coogan?

  That's great. And so what, what can you share that's top of mind right now on the research side? There's been this discussion of pre-training scaling wall potentially, the importance of reinforcement learning, reasoning. There's so many different areas to go into. What's actually driving the most conversations internally right now?

• 1:02:08Mark Chen?

  Yeah, absolutely. So I think really, it's a really exciting time to do research. I would say Versus two or three years ago, I think people were trying to build this very big scaling machine.

• 1:02:19John Coogan?

  Yeah.

• 1:02:20Mark Chen?

  And really, the reasoning paradigm changed a lot of that, right? You know, like reasoning is really taking off. And it really opens this new playing playground, right? It's like there are a lot of kind of known unknowns and also unknown unknowns that you're all trying to figure out. It kind of feels like GPT two era, right? Well, where there's so many different hyper parameters you're trying to figure out. And then I think also, you know, like you mentioned, you know, pre-training, that's not to be forgotten either. You know, today, we're in a very different regime of pre-training than we used to be, right? Today, we can't treat data as this infinite resource.

• 1:02:55John Coogan?

  Yeah.

• 1:02:56Mark Chen?

  I think a lot of academic studies, you know, they've always kind of treated, you know, you have some kind of finite compute, but infinite data. I don't think there's much study of, you know, like, you know, finite data and infinite compute. And I think, you know, that also leads to a very rich playground for research.

• 1:03:13John Coogan?

  Do we need kind of a revision to the bitter lesson? Is that a refutation of the bitter lesson? Or do we just need to re rethink what the definition of scaling laws looks like?

• 1:03:27Mark Chen?

  Uh, no, I don't think of anything as a refutation of the bitter. Really, like, our company is grounded in, we want simple ideas that scale. I think RL is an embodiment of that. I think pre-training is an embodiment of that. And really, at every single scale, we face some kind of difficulty of this form. It's just like, you gotta find some innovation that gets you past the next bottleneck. And this doesn't feel fundamentally very different from that.

• 1:03:50John Coogan?

  Mm hmm. What is, what's most important right now on the actual compute side? We heard from Nvidia earnings that, that we didn't get a ton of guidance on the shift from training to inference usage of Nvidia GPUs, but it feels like it must be coming. It feels like this inference wave is, is, is happening. Are those even the right buckets to be thinking about tracking metrics in terms of the, the, the story of artificial intelligence? Because, yeah, I mean, it's like if if the reasoning tokens are inference tokens and and but they're what

• 1:04:31John Coogan?

  lead to higher intelligent, more intelligent models, like it's almost back in the training bucket again. What bucket should we be thinking about and and and or are we, how firmly are we in the the

• 1:04:46John Coogan?

  the applied AI era versus the research era?

• 1:04:49Mark Chen?

  Well, I think research is here to stay and it's for all the reasons I mentioned above, right? It's such a like a rich time to be doing research, but I do think, you know, inference is going to be increasingly important as well, right? It's such a core part of RL that you're doing rollouts. And I think, you know, we see 2025 as this year of agents, right? We think of it as a year where models are going to do a lot more autonomous work. You can let them kind of be unsupervised for much longer periods of time. And that is just going to put big demands on inference, right? When you think about kind of our overall vision, right? We we lay it out as a series of steps and levels on the way to AGI, right?

• 1:05:30Mark Chen?

  And I think the pinnacle, really, that last level is organizational AI, right? Like, you can imagine a bunch of AIs all interacting.

• 1:05:39Mark Chen?

  And, yeah, I think that's just going to put huge demands on inference, right?

• 1:05:43John Coogan?

  On that, on that organizational question, I remember reading AI 2027. And one of the things that they proposed was that the AIs would actually like literally be talking to each other in Slack. Does that seem like, does that seem like the way you imagine agents playing out? Like, using the two, the same tools as humans instead of

• 1:06:04Jordi Hays?

  One agent says, I'm gonna go talk with Teams and I'll talk with Slack and I'm gonna do a little negotiating.

• 1:06:09John Coogan?

  But maybe it just happens super, super fast 24/7? Or or is there like a new machine language that emerges?

• 1:06:16Mark Chen?

  Yeah, I mean, I think one thing that's really helped us so far in AI development is to come in with some priors for you know, how humans do things. And that's actually

• 1:06:28Mark Chen?

  You know, if you bake those priors and they typically are great starting points. So I could imagine like, maybe you start with something that's Slack like, and give it enough flexibility that it can kind of develop beyond that and really figure out the way that's most effective for it to communicate. One important thing, though, is, you know, we want interpretability too, right? I think it's very helpful for us today that what the agents do is, you know, easy for us to read and interpret. And I don't think you want that to go away as well. So I think there's a lot of benefits, just even from a pure like debug the whole system perspective. Let the models, you know, speak in a way that is familiar with us. And, you know, you can also imagine like, we might want to plug in to the system too, right?

• 1:07:12Mark Chen?

  So, you know, whatever interfaces we're familiar with, we would ideally like our model to be familiar with as well.

• 1:07:20Mark Chen?

  I think it's also pretty compatible with, you know, we hit a big milestone. We got, I think, three million paying business users.

• 1:07:30John Coogan?

  Let's go.

• 1:07:31Mark Chen?

  Yeah, there we go. Let's go.

• 1:07:38John Coogan?

  Three gong hits for three million.

• 1:07:40Jordi Hays?

  The gong will keep ringing for a while.

• 1:07:44John Coogan?

  I was hoping you would drop a number.

• 1:07:47Mark Chen?

  Yeah.

• 1:07:48John Coogan?

  Congratulations. That's actually huge. That's amazing.

• 1:07:51Mark Chen?

  Yeah, yeah, yeah. But I think one big part of that is, you know, we have connectors now, right? Into like G-drives. And I think, yeah, you can imagine, you know, like Slack integrations, things like that. I think we just want the models to be familiar with the ways we communicate and get information.

• 1:08:07John Coogan?

  Yeah. Can you talk about benchmarking? It feels like we're potentially.

• 1:08:11Jordi Hays?

  Yeah. Do you think about benchmarks at all?

• 1:08:13Mark Chen?

  Oh, yeah, a lot. I mean, but I think it's a difficult time for benchmarks, right? I think we used to be in this world where you have these human written benchmarks for other humans, right? And I think we all have these norms for like, what are good benchmarks, right? Like, we've all taken the SAT. We all have like a good conception of what it means to get, you know, whatever score on that.

• 1:08:37Mark Chen?

  But I think the problem is the models are already at the point where for even the hardest human written benchmarks for other humans, it's really near saturated or saturated, right? I think one clear example here is the AMI, like probably the hardest auto gradable like human math eval, at least in the in the US. And yeah, the models are consistently getting like ninety plus percent on these. And so what that what that means is I think there's a kind of two different things that people are doing, right? They're they're developing kind of model based benchmarks, right? They're not kind of things that we would give to an ordinary human, things like humanities

• 1:09:20Mark Chen?

  last exam, things like, you know, Epic AI that are really, really at the at the frontier of what what people can do. And I think the the hard thing is it's not grounded in intuition anymore, right? Like, you know, you don't have a lot of people who have taken these exams. So it it makes it harder to kind of calibrate on whether this is a good exam or not. One of the exciting things that's on the flip side of that is I really do think we're at the era where models are going to start innovating, right? Because I think once you've passed the last kind of like the hardest human rating exams, that's kind of at the edge of innovation. And I think you already see that with the models, right? Like, they're helping to write parts of papers. And and I think the other kind of way that people have shifted is, you know, there's

• 1:10:05Mark Chen?

  these, you know, ultra frontier evals, but there are also people kind of just indexing on real world impact, right? You look at your revenue, kind of the value delivered to users. And I think that's ultimately what we care about.

• 1:10:18John Coogan?

  Can you can you bring that back to interpretability research? Like with these super, super hard math evals, for example, are we doing the right research to understand if the thought process mirrors not just not just one shotting the answer, oh, you you you memorized it or you magically got it correct, but you actually took the correct path, kind of like, you know, you're graded for your work, not just the answer if you're in grade school? And and, you know, Dario said that interpretability research will actually contribute to capabilities and even give a decisive lead. Do you agree with that? What's your reaction to that concept of interpretability research being very important?

• 1:11:01Mark Chen?

  Yeah, I mean, we care a lot about it here at OpenAI as well. So one thing that we care a lot about is interpreting how the model reasons, right? Because I think we've had a very kind of specific and strong view on this in that we don't want to apply optimization pressure to how the model thinks so that it can be faithful in the way it thinks and to expose that to us, you know, without any kind of incentives to cater to what the user wants, right? I think it's actually very important to have that unfiltered view.

• 1:11:36Mark Chen?

  Because, you know, oftentimes, like, if the model isn't sure, you don't want to hide that fact, right? Just for it to kind of please the user. And sometimes it really isn't sure, right? And so we've really done a lot of work to try to promote this norm of chain of thought faithfulness and interpretability. And I think it gives you a lot of sense into what the model is thinking and, you know, what are the pitfalls that it can go off into if it's not reasoning correctly?

• 1:12:04Jordi Hays?

  That's such an important point, because if you have somebody on your team and they come to you and they say, hey, you know, I think this is the right answer, but we should probably verify it. It's like, it's still valuable.

• 1:12:14John Coogan?

  Totally.

• 1:12:14Jordi Hays?

  It puts you on the right path. If somebody comes to you 100% confidence, this is, this is the truth. Being wrong is like, well, like, trust is just destroyed.

• 1:12:22John Coogan?

  Yeah, totally.

• 1:12:23Mark Chen?

  Yeah, don't you guys feel like, you know, safety felt a lot more theoretical a couple of years back, right? But like today, you know, like the things that people were talking about a couple of years, like scalable oversight, really having the model be able to tell you like, and convince you that the work it did was right. It feels so much more relevant right now, just because the capabilities are so strong.

• 1:12:42John Coogan?

  Yeah, I mean, just personally, I've completely flipped from being like, oh, the safety research is not that valuable because I'm not that worried about getting paper clipped. It just seems like a very low likelihood that that's kind of like the bad ending, like immediately in this boom and all this crazy gray goo scenarios were just so abstract and sci fi. It just felt like economics will fall into place and there will be like a cold, like the nuclear ending, which is like, we didn't build nuclear plants and we just stopped everything because we, humans seem to be good at that. But now that we're actually seeing things.

• 1:13:15Mark Chen?

  Yeah, it's crazy how fast it's been, right? Like, I think my, like, my personal story is, it's like, you know, what got me into AI was AlphaGo, right? Like, just watching it get to that level of capability at Go. And you were kind of like, it was. Such an optimistic and also kind of a little bit of a sobering message, right? When you saw Lizardal get beat. And I just remember, you know, like we we saw the coding models, you know, when we first launched, like, I think, very OG Codex, you know, with GitHub Copilot. It was maybe like under, you know, a thousand Elo on on Codeforces. And I still remember the meeting where I walked into where the team showed my score and they're like, hey, this model is better than you. And it's like, you know, you come full circle and it's like, wow, like I put decades of my life into this.

• 1:14:00Mark Chen?

  And, you know, the capabilities are there. So like, if, you know, I'm kind of at the top of my field in this thing and it's better than me, like, what can it do?

• 1:14:09John Coogan?

  Yeah, yeah, that's amazing. I have so many more questions. On AlphaGo, are there are there lessons from scaling? How scaling played out there that you can that we can abstract abstract into the rest of AI research? What I mean is, as I remember it, the AlphaGo training run was not 100k H200s. But what would happen if we actually did an AlphaGo style training run? I mean, it would be an economic money pit, right? Like, they've had no economic value to do. But let's just say some benevolent trillionaire decides I'm going to spend a billion dollars on a training run to beat AlphaGo and go even bigger.

• 1:14:52John Coogan?

  Is is Go at some point solved? Would we see kind of diminishing scaling curves? Could we throw extra RL? Could we could we port back everything that we've doing in just general AGI research? And and and just continue fighting it out in the world of Go? Or does that end and does that teach us anything?

• 1:15:10Mark Chen?

  Yeah, yeah, honestly, I feel like if you really are curious about these mysteries, join our team. That's the thing I want to say. So, yeah, I mean, really, like, kind of the the central problem of today is RL scaling, right?

• 1:15:22John Coogan?

  Yeah.

• 1:15:23Mark Chen?

  When you look at AlphaGo, right, it's it's a narrow domain, right? Yeah, I think in some sense that limits the amount of computing you can pump into it. But even kind of small toy domains, they can teach you a lot about how you scale RL. Like, what are the axes where it's most productive to to pump scale in? I think a lot of scaling research just looks like that, whether it's on RL or pre-training. So you identify a lot of, you know, different different variables under which you can scale. And like, where is kind of where you get the best kind of like marginal impact for for pumping scale there? I think that's a very open question for RL right now.

• 1:15:59Mark Chen?

  And I think what you mentioned as well, it's just like, you know, going from narrow to broad, right? Does that give you a lever to pump a lot more scale in as well? I think when you look at our reasoning models today, they're a lot more broad based than, you know, just being able to kind of an expert system on go. So yeah, I really do think that there are so many levers to scale.

• 1:16:21John Coogan?

  What about move 37? That was such an iconic moment in that AlphaGo Lee Sedol match. They play move 37. It's very unconventional. Everyone thinks it's a blunder. It turns out not to be. It turns out to be critical. It turns out to be innovation. Do you think we are, we're certainly post touring test in language models. We're probably post touring test in image generation. But it feels like we're pre move 37 in text generation in the sense that there hasn't been like a fully AI generated book that everyone is just, oh, it's the new Harry Potter. Everyone has to read it. It's amazing and it's fully aged and it's fully generated or this image.

• 1:17:03John Coogan?

  The images, they do go viral, but they go viral because they're AI. Move 37 in the context of Go did not go viral because it was AI. It felt like it was actual innovation. So is that the right frame? Does that make any sense?

• 1:17:16Mark Chen?

  I think it's not the wrong frame. So I think some, some quick thoughts on that. I, I think kind of when you have something that's, you know, very measurable, like win or lose, right? Something like, like Go. Yeah, it's like very easy for us to kind of just judge, right? Like, did, did the model do something right here? And I think the more fuzzy you get, you know, it, it is just harder, right? Like when it comes to, is this the next Harry Potter, right? Like, you know, it's not a universally loved book. I think it's fairly universal, but you know, there's, there's some haters. And yeah, I, I think it, it is just kind of hard when it comes to these human subjective things.

• 1:17:56Mark Chen?

  Where it's really hard to put down in words, like what makes you like Harry Potter, right? And so I think those are always going to lag a little bit, but you know, I think, you know, we're, we're developing more and more techniques to attack kind of these more open ended domains.

• 1:18:15Mark Chen?

  I don't know, I I wouldn't say that we're not at an innovative stage today. So I think my biggest touch with this was when we had the models compete on the I O I last year. So I O I, it's like the the international, basically Olympics for for computer science. Basically, the the top four kids from from each country go and compete. And these are really, really tough problems.

• 1:18:41Mark Chen?

  Basically selected so that they require some innovative insight to solve, right? I think.

• 1:18:49Mark Chen?

  And we did see the model come up with solutions even to some very head hog problems. And and so I think there was a lot of surprise for me there, right? I was completely off base about which problems the model would be able to solve the most, right? I think like I I kind of categorized there there's six problems, some of them as more kind of like, oh, this is standard. A little bit more standard, this is a bit more out of the box. And it's like, it's not going to be able to solve this more out of the box one, but it it did. And I think I think it really does speak to kind of these models have the capacity to do so, especially trained with R L.

• 1:19:26John Coogan?

  Now, now, now, put that in context of what's going on with Arc A G I. Obviously, OpenAI has made incredible progress there, but it just, when I do the problems, it seems easy. And when I look at the I O I sample problems, I think this would be a 20 year process for me to figure out how to achieve that and I can do the Arc A G I on my phone. Is this the spiky intelligence concept? Is this something that a small tweak in in algorithmic design, just one shots A G I or or Arc A G I? Or or is there something else going on there that we should be aware of?

• 1:20:00Mark Chen?

  Yeah, I mean, I think part of this is the beauty of Arc A G I as well, right? Like, I think. I'm not sure if there's another kind of like human intuitive, simpler benchmark, which is for the models. And I think really that's one of the things they really optimize for on that benchmark. I do think when it comes to models, though, like, there's just a little bit of a perception gap as well. Like, you know, models aren't used to this kind of native, you know, like, just screen type input. I think there's a lot we can bridge there. Actually, even O4 Mini, it's a state of the art multimodal model in many ways, including visual reasoning. And I think, you know, you're starting to kind

• 1:20:41Mark Chen?

  of build up the capacity for the models to take images, manipulate and reason about them, or generate new images, write code on images. And I think it's just been kind of under focused. But I think when I talk to researchers in the field, they all see this as a part of intelligence, too. And we're going to continue to focus there.

• 1:21:01John Coogan?

  Yeah, is R KGI kind of in the, if we're dropping a buzzword on it, is like program synthesis? Is there a world where, I know that, I know the tokens, like, the images, we see them as renderings of squares and different colors, but the, when they're fed into the L L M, they're typically just a stream of numbers effectively. Is there a world where actually adding a screenshot is what's important? Like visual reasoning?

• 1:21:28Mark Chen?

  Yeah, yeah. So I think, I think that could be important. It's just like, kind of, you know, whenever it comes to like textual representation of grids, models today just don't really do that well. Right? And I think it's just kind of because humans don't really ever write down textual representations of grids. Or like, you know, we have a chessboard, like, no one really kind of just like types it out in a grid. And, and so the models are kind of like, under trained a little bit on, on what that looks like and what that means. So, you know, I, I think with more reasoning, we'll just bridge the gap. I think with better visual perception, we'll just bridge that gap.

• 1:22:09John Coogan?

  Yeah.

• 1:22:10Jordi Hays?

  How are you thinking about the role of non lab researchers in the ecosystem today? I'm sure you try to recruit some of the best ones, but the ones that don't join your team.

• 1:22:21John Coogan?

  Tell us about the one that got away.

• 1:22:23Jordi Hays?

  Yeah, the one that got away.

• 1:22:24Mark Chen?

  Yeah, no, I mean, I think it's still actually a fairly good time for for specific domains, right, to be doing research. And, you know, I think the style is just very different. And you do feel the pull of non lab researchers into labs, because I think they feel like a lot of the burning problems in the field are at scale, right? And that's kind of one of the unfortunate things too, right? Like, when you look at reasoning, you just don't see that happen at small scale, right? There's like a certain scale at which it starts becoming signal bearing, and that requires you to have resources, right? But I do think, you know, a lot of the really good

• 1:23:05Mark Chen?

  work that I've seen, you know, there's experimental architectures, I think a lot of good work is happening in the academic world there, like a lot of study in optimization, a lot of study in kind of like GANs, you know, there's certain fields where you see a lot of fruitful research that that happens in academia.

• 1:23:23Jordi Hays?

  Yeah, that makes a lot of sense. How about consumer agents? How are you thinking about them? You talked earlier about sort of B two B adoption, and that's all very exciting. But how much do you and the research org think about breakout consumer agent products?

• 1:23:40Mark Chen?

  Yeah, that's a fantastic question. I think we think about it a lot. I think that's the short answer. You know, we really do think like this year, we're trying to focus on how we can move to the agentic world, right? And when I think about consumer agents, I think like ChatGPT proved that, you know, people got it, right? It's like people get conversational agents, when they conversational kind of models. But when it comes to consumer agents, we have a couple of theses and that we've tried out in the world. I think one, one is deep research, right? I think this is something that can do five to 30 minutes of work autonomously, come back to you and really like kind of synthesizes information, right? It goes out there, gathers, collects, and kind of

• 1:24:25Mark Chen?

  You know, compresses the information in a form that that's useful.

• 1:24:28Jordi Hays?

  But a little bit of a little bit of pushback there. Like, I can see that as a consumer product when someone like Aiden is like, I want new towels. And he uses deep research to like, figure out like, what is the best towel across every dimension? But when I think of deep research, yes, it has applications with students, but it's often.

• 1:24:47John Coogan?

  Some of them might just be paradigm, because I feel like.

• 1:24:48Jordi Hays?

  Consumers being like, give me a deep research report on this country and where to travel and things like that.

• 1:24:53John Coogan?

  We keep using this flight example, but I don't, I haven't actually tried to book a flight with deep research. It's totally possible that it could go and pull all the different flight routes and calculate all the different delays and all the different, all the different parameters of, if I fly to this airport, I can park or I can use valet here or something like that. Yeah.

• 1:25:11Jordi Hays?

  Yeah. And I guess like, when I think of agents, it's, it's deep research is like, you know, curating information on which you can take action on. But it's like, at what point is action a part of that sort of loop, right? Where you can not only curate a list of flights that you want, but then, you know, actually go out and have agency.

• 1:25:31Mark Chen?

  Yeah. I think one of our explorations in that space is operator, right? It's where you kind of just feed in raw pixels from your laptop into, or you know, from some virtual machine into the model. And it, it produces, you know, either a click or some keyboard actions, right? And so there it's taking action. And I think the trouble is, you know, it, you don't ever want to mess up when you're taking action. Right? I think the cost of that is super high. You, you only have to get it wrong once to lose trust in, in a user. And so we want to make sure that that feels super robust before we get to the point where we're like, hey, look, here's a tool. I.

• 1:26:13John Coogan?

  That's so different than deep research because like, you can wind up on some news article and read a one sentence that it gets a fact wrong. Or the commas in the wrong place and the numbers off. And but that's just the expectation for just text and analysis. And if you delegated that, yeah, you're going to expect a few errors here and there. Oh, that's actually a different company name, or that's the, that's an old data point. There's new data, but very different if I book a flight and you book the wrong flight and I can wind up in Chicago instead of New York.

• 1:26:42Mark Chen?

  Exactly. And I think the reason why we care so much about reasoning is because I think that's the path that we get reliable agents through.

• 1:26:49John Coogan?

  Sure.

• 1:26:50Mark Chen?

  Um, you know, we've talked about like reasoning, helping safety, but reasoning is also helping reliability. Right. It's like, you imagine like, what makes a model so good at a math problem? It's like, it's banging its head against it. It's trying a different approach. And then it's like adapting based on what it failed at last time.

• 1:27:07John Coogan?

  Yeah.

• 1:27:07Mark Chen?

  And I think that's the same kind of behavior you want your, your agents to have. It's like, yeah, tries things like adapts and, and keeps going until it's.

• 1:27:14Jordi Hays?

  And that's, that's the, uh, humans do this every day. You're booking a flight, you keep hitting an error. It's not which or which form you missed, right? And you're just sort of banging your head against the computer and eventually it says, okay, you're booked. Right. Yeah. I think that's a great call out.

• 1:27:28John Coogan?

  Yeah. Um, I mean, there's so many more questions we could go into, but, um, I'm, I'm interested in the scaling of RL and kind of the balancing act between pre-training RL and inference. Just the amount of energy that goes into getting a result when you distribute it over the entire user base. How is that changing? And I guess, um, is, is, are we post like really big, really big runs? Is this going to be something that's like continually happening online? Or it feels like we're moving away from the era of like, oh, some big development, some big run happened. And now we're grouping the fruits of it versus a more iterative process.

• 1:28:12Mark Chen?

  Um, yeah, I mean, I don't see why it has to be so, right? I think like, if you find the right levers, you can really pump a lot of compute into RL as well as pre-training. I think it is a delicate balance though, between all of these different parts of the machine. And, you know, when, when I look at my role with Jacob, it's just kind of like figure out where, um, how, how this balance should be allocated. Where the promising kind of like nuggets are arising from and, and resourcing those. Yeah, it's, it's kind of a, in some sense, my, I feel like part of my job is a portfolio manager. Yeah.

• 1:28:43John Coogan?

  That's a lot of fun.

• 1:28:46John Coogan?

  Well, thank you so much for joining. This is a fantastic conversation. We'd love to have you back and get deeper.

• 1:28:50Jordi Hays?

  Great hanging mark.

• 1:28:51John Coogan?

  We'll talk to you soon.

• 1:28:52Jordi Hays?

  Yeah, peace. Have a good one.

• 1:28:55John Coogan?

  Next up, we have Sholto Douglas from Anthropic coming on this show.

• 1:28:58Jordi Hays?

  I'm getting so much.

• 1:28:59John Coogan?

  Jordy is giving us the update.

• 1:29:01Jordi Hays?

  No, I'm just getting a lot of messages saying, why, no one cares about AI. About the drama on the timeline. Well, we do care about AI. We care a lot about AI.

• 1:29:14Jordi Hays?

  But it is a mess out there.

• 1:29:16John Coogan?

  Wow. Yeah, the end of the Trump Elon era. Uh, I don't know. Well, maybe we have to get some people on to talk about it, uh, tomorrow or something.

• 1:29:25Jordi Hays?

  Gotta do it today.

• 1:29:26John Coogan?

  Anyway, uh, we have Sholto from Anthropic in the studio. How are you doing?

• 1:29:30Jordi Hays?

  What's going on?

• 1:29:32Sholto Douglas?

  Good to see you guys.

• 1:29:33John Coogan?

  Hopefully, uh, you're staying out of the chaos on the. Don't, don't open your time. Don't open. We'll do you a favor.

• 1:29:40Jordi Hays?

  Sweet child.

• 1:29:43John Coogan?

  Everything. Stay focused on the application.

• 1:29:46Jordi Hays?

  Stay focused on the mission.

• 1:29:47John Coogan?

  Stay focused on the next training run.

• 1:29:49Jordi Hays?

  We really, humanity really cannot afford for any AI researchers to open X today.

• 1:29:54John Coogan?

  What a hilarious day. Anyway, I mean.

• 1:29:57Sholto Douglas?

  24 hours, guys.

• 1:29:57John Coogan?

  Yeah. How are you doing? What, what is new in your world? What, what are you focused on mostly day to day? And, uh, maybe it's just a way of an intro.

• 1:30:06Sholto Douglas?

  Yeah. Um, so at the moment focused really hard on scaling RL. And that is the theme of what's happening this year. And we're still seeing these huge gains where you go, you know, 10x compute increase in RL. We're still getting like very distinct linear gains on the basis of that. Um, and because RL wasn't really scaled anywhere close to how much pre-training was scaled at the end of last year.

• 1:30:25John Coogan?

  Yeah.

• 1:30:25Sholto Douglas?

  We have like a basically a gamut of riches over the course of this year.

• 1:30:30John Coogan?

  So where are we in that, in that RL scaling story? Because I, I remember the, the, some of the rough numbers around like GPT-2, GPT-3, we were getting up into like, it costs a hundred million dollars. It's going to cost a billion dollars. Like it just rough order of magnitude, not even from Anthropic, just generally like, what is a big RL run cost or, or how many, are we talking 10K H200s or 100K? Like, are we going to throw the same resources at it? And if so, how soon?

• 1:30:59Sholto Douglas?

  Yeah. So I think in Dyer's essay at the beginning of the year, he said that a lot of runs were only like a million dollars back in like December. Um, I think you have like DeepSeek V3 and this kind of stuff like R1. Um, which means that with that's like at least two rooms just to get to the scale of GPT four and GPT four was two years ago. Yeah. Right. Um, RL is also perhaps a bit more naively paralizable and scalable than pre-training. You're pre-training, you need everything in one big data center ideally, or you need like some clever tricks. Um, RL, you could like in theory, like what the prime intellect folks are doing, scale it all over the world.

• 1:31:29John Coogan?

  Yep.

• 1:31:29Sholto Douglas?

  Out of it. And, and so we, you are held back like maybe like you're held back far less than you are.

• 1:31:35John Coogan?

  Sure. So e- everyone and their mother has a billion dollars now. There, there, you know, hundreds of thousands of GPUs getting pumped all over the place. I, I, I feel like we're not GPU poor as, as a, as a society. Uh, maybe some companies need to justify it in different ways, but it sounds like there's some sort of, uh, uh, like reward hacking problem that we're working through in terms of scaling RL. What are all of the problems that we're working through to actually go deploy the capital canon at this problem?

• 1:32:07Sholto Douglas?

  Yes. So I mean, think about what you're asking the model to do in RL is you're asking it to achieve some goal at, at any cost, basically.

• 1:32:14John Coogan?

  Yeah.

• 1:32:15Sholto Douglas?

  And this comes with a whole host of like, uh, behaviors which you may not intend. Um, in software engineering, this is really easy. I like to, it might try and hack unit tests or whatever. Um, in much more longer horizon, real world tasks, you might ask it to say, go make money on the internet. And it might come up with all kinds of fun and interesting ways to do that unless you find ways to guide it into following the like principles that you want it to, to obey, basically. Um, or to align it with your like idea of what's sort of best for humanity. And so it's actually, it's a pretty intensive process. It's a lot of work to find down, hunt down all the ways these models are, uh, hacking through the rewards and, and, and patch all of that. And yeah.

• 1:32:53John Coogan?

  Yeah. How, how, uh, are we going to see scaling in the number of rewards that we're RLing against? If that makes sense, I would imagine that, uh, at, at a certain point, we, unless we come up with like, kind of like the, the, the genesis prompt, go forth and be fruitful or something and, and multiply. Uh, the, you could imagine, uh, training runs on, on just knocking down one, one problem after another. And is that, is that kind of the path that we're going down?

• 1:33:24Sholto Douglas?

  I very much think so. There's this idea in which like, you know, the sort of world becomes an RL environment machine in some respects, because there's just so much leverage to making these models better and better at all the things we care about. And so I think we're going to be training on just everything in the world.

• 1:33:40John Coogan?

  Got it. And then, and then does that lead to more model fragmentation, models that are good at programming versus writing versus poetry versus image generation? Or does this all feed back into one model? Does the idea of the consumer needing to pick a model disappear? Are we in a temporary period for that paradigm?

• 1:34:03Sholto Douglas?

  I think the main reason that we've seen that so far is because people are trying to make the best of the capital. Like, we are all still GPU poor in many ways.

• 1:34:13John Coogan?

  Okay.

• 1:34:13Sholto Douglas?

  And people are focusing those GPUs on the sort of like spectrum of rewards that I think is most important. And I'm a bit of a big model guy. I really do think that, similar to how we saw with large pre-trained models before, where small fine-tuned models made it, like had gains over the sort of GPT-2 era, but then were obsoleted by GPT-4 being generally good at everything. I think, to be honest, you're going to see this generalization and learning across all kinds of things. That means you benefit from having large single models rather than specialization or area fine-tuned models.

• 1:34:47John Coogan?

  Can you talk a little bit about the transition from, or any differences between RLHF and just other RL paradigms?

• 1:34:55Sholto Douglas?

  Yes. So RLHF, you're trying to maximize a pretty, like, lossy signal. Things like air-wise, like, what do humans prefer? And I don't know if you've ever tried to do this, like, judge two language model responses.

• 1:35:07John Coogan?

  I get prompted for that all the time.

• 1:35:09Sholto Douglas?

  Right.

• 1:35:10John Coogan?

  And I'm always like, I don't want to read both of those. I'll just click the one on the left.

• 1:35:13Sholto Douglas?

  Exactly, exactly. And, you know, I click one of the random ones sometimes.

• 1:35:16John Coogan?

  Yeah. Or I click like the one that just looks bigger, or I'll read the first two sentences. But yeah, I'm not giving straight. I'm not, I'm not being, I'm not doing my job as a, as a human reinforcer.

• 1:35:26Sholto Douglas?

  Exactly. Human preferences are easy to hack.

• 1:35:29John Coogan?

  Yeah, totally.

• 1:35:30Sholto Douglas?

  Environments in the world are much truer, if you can find them. So something like, did you get your math question right? Is a very real and a true reward.

• 1:35:38John Coogan?

  Does the code compile, right?

• 1:35:39Sholto Douglas?

  It compiles, exactly.

• 1:35:42Sholto Douglas?

  Did you make a scientific discovery?

• 1:35:44John Coogan?

  Yeah.

• 1:35:44Sholto Douglas?

  We got very little rewards right now, but pretty quickly over the next year or two, you're going to start to see much more meaningful and long horizon rewards.

• 1:35:53John Coogan?

  You're going to see models bribing the Nobel Committee to win Nobel Prize.

• 1:35:57Sholto Douglas?

  Well, there's a good reward hack.

• 1:35:58John Coogan?

  There's reward hacking.

• 1:35:59Sholto Douglas?

  But that's something you want to prevent, right?

• 1:36:02John Coogan?

  Exactly. Yeah, that's the real nightmare. What about, like, there's so many different problems that we run into that feel like it's just really, really hard to design any type of eval. The, my kind of benchmark that I use whenever a new model drops is just tell me a joke. They're always bad. And, or, or, or even, even the latest VO3 video that went viral was somebody said like, stand-up comedy joke. And it was kind of a funny joke, but it was literally the top result for joke Reddit on Google. And then it clearly just took that joke and then instantiated in a video that looked amazing. But it wasn't original in any way.

• 1:36:47John Coogan?

  And so we were joking about like the RLHF loop for that is like you have an endless cycle of comedians running AI-generated materials and then, and then, you know, speak, microphones in all the comedy clubs to feedback what's getting laughs.

• 1:37:04Sholto Douglas?

  I mean, honestly, that would work pretty well.

• 1:37:06John Coogan?

  Yeah.

• 1:37:07Sholto Douglas?

  Many comedians want to hook us up with an RL loop. I mean.

• 1:37:10John Coogan?

  Yeah, yeah. But, but I mean, for, for some of those less, like, as you go down the curve, it feels like each one gets harder and harder to actually tighten the loop. We see this with like longevity research where it's like, okay, it takes 100 years to know if you extended a human life. Like the, yes, you could create a feedback loop around that, but every change is going to be hundreds of years. And so even if you're on the cycle, it's irrelevant for us in the context that we talk about AI. So talk to me about like, are you running into those problems? Or, or, or will there be like another approach that kind of works around those?

• 1:37:43Sholto Douglas?

  So there are a lot of situations where you can get around this by just running much faster than real time. Like, let's say, the process of building like a giant app, like building Twitter, right? It's something that would take human months. But if you got fast enough and good enough AIs, you could do that in several hours, right? I create heaps of AI agents, they're all building, right? And so you can get a faster reward signal in that way. In domains that are less well specified, like humor, I agree, it's really, really hard. And this is like, why I think in some respects, like creativity is like at the at the top end of the spectrum, like true creativity is much, much harder to replicate than the sort of like analytical scientific style reasoning.

• 1:38:17John Coogan?

  Yeah.

• 1:38:18Sholto Douglas?

  And and that will just take more time. You know what? The models actually are pretty good at making jokes about being an AI. This feels weirdly fresh. Like, everything else is kind of a weird copy of something like it's like, it just, it feels like it's derivative, basically. It's trying to infer what humor is, and it doesn't really understand it. But jokes about being an AI are quite funny.

• 1:38:37John Coogan?

  Yeah, I I I think this also might be, I don't know if it was directly reward hacking, but I noticed that one of the new models dropped and a bunch of people were posting these like 4chan, like be me memes. And and and they were, it seemed like they were kind of hacking the humor by being hyper-specific about an individual that they could find information on online. And so you're laughing at the fact that it's like, oh, wow, that is like something that I've posted about. It's making a reference, but it's not really that funny to me. It's other than it's just like, wow, they really did its research. Like, it really knows Tyler Cowen intimately, which is cool. But I didn't find it hilarious. Yeah, yeah, yeah, very interesting. Let's talk about

• 1:39:16John Coogan?

  some sort of deep research product, projects and products. We were talking to Will Brown, and he was saying, like, AGI is here with some of the bigger models, but the but the time that AGI can feel consistent, it diverges. And so you could be working with someone who's, you know, 100 IQ, but they will stay consistent for years as an employee, or they'll they'll keep, you know, living their life. Whereas a lot of these super smart models are working really well. And then after a few a few minutes of work, the the agents kind of diverge and kind of go into odd Paradigms, it feels very not human. It feels like a, like just a, they're hyper intelligent

• 1:39:59John Coogan?

  one way and then extremely stupid in others. What's going on there? What is the, what is the path to extending that? Is that more like having more better planning and better, better like dividing up the task? Or, or will this just kind of naturally happen through the RL and scale?

• 1:40:15Sholto Douglas?

  Yeah. So there's that jaggedness, right? Which is what you're seeing is how we call it. And I think that is largely a consequence of the fact that maybe something like deep research is probably being RL'd to be really good at producing a report.

• 1:40:27John Coogan?

  Yeah.

• 1:40:27Sholto Douglas?

  But it's never been RL'd on the like act of producing valuable information for a company over a week or a month, or like making sure the stock price goes up in like, you know, a quarter or something like this, right? Like it, it doesn't have any conception of how that feeds into the broader story at play. It can kind of infer it because it's got a bit of world knowledge from the, you know, the base model and this kind of stuff. It's never actually been trained to do that in the same way humans have. So to extend that, you need to put them in much longer running, much like, like your long horizon things. And so, so deep research needs to become, you know, like deep operator company for a week kind of thing.

• 1:41:00John Coogan?

  Sure. Is that the right path? Like, it feels like the road might be, there's a, like, the longest running LLM query used to be just like a few seconds, maybe a few minutes. And I remember when, when some of the reasoning models came out, people were almost trying to like stunt on it by saying like, oh, I asked it a hard question, I thought for five minutes. Now deep research is doing 20 minutes pretty much every time. Is the path two hours, two days? Or are we going to see more efficiency gains such that we just get the 20 minute model, the 20 minute results in two minutes and then two seconds?

• 1:41:37Sholto Douglas?

  Yeah. So this is somewhere where like inference in many respects and prioritization becomes really important. So both like, how fast is your inference? Is that literally affects the speed at which you can think and the speed at which you can like, like, you know, do these experiments? Also, how easily you can prioritize becomes really important. Like, can you dispatch a team of, of subagents to go and do deep research and like compile like sub reports for you so that you can do everything in parallel? These kinds of like, it's, it's both like, there's an infrastructure question. Here that feeds up from the hardware and the chips and this kind of stuff to like designing better chips for better inference and all this. And and an RL question of like, you know, how well can you prioritize and all this? So I think we just need to compress the timelines, compress the time, compress the timeframes.

• 1:42:20John Coogan?

  Yeah. So if I'm, if I'm like an extremely big model and I'm running an agentic process, like how, how much am I hankering for like a middle sized model on a chip? Or like baked down into silicon that runs super fast because it feels like that's probably coming. We saw that with the Bitcoin progression from CPU to GPU to FPGA to ASIC. Do you think we're, we're at a good enough point where we can even be discussing that? Because I, every time I see like the latest mid journey, I'm like, this is good enough. I just want it in two seconds instead of 20. But then a new model comes out, I'm like, oh, I'm glad I didn't get stuck on that path.

• 1:43:01John Coogan?

  But, but yeah, like, how far away from, how far away are we from, okay, it's actually good enough to bake down into silicon?

• 1:43:09Sholto Douglas?

  Well, there's a question here of baking it down to silicon versus designing a chip, which is like very suited for the architecture that you care about. Right? And baking on silicon, unsure. Like, I think that's a bet you could take, but it's a, it's a risky one because the pace of progress is just so fast nowadays. And I really only expect it to accelerate. But designing things that make a lot of sense for the sort of transformers or architectures of the future should, should make a lot of sense.

• 1:43:34John Coogan?

  Wait, that is a big gap, though. Transformers or architectures of the future? If we diverge, there's a lot of companies that are banking on the transformer sticking around. What is your view on transformer architecture sticking around for the next couple of years?

• 1:43:47Sholto Douglas?

  I mean, look, they stuck around for five years, so they might stick around for a little while. But there's, there's different, you think about architectures in terms of this balance of memory bandwidth and flops, right? One of the big differences we've seen here is Gemini recently had actually a diffusion model that they released.

• 1:44:01John Coogan?

  I was about to ask you.

• 1:44:02Sholto Douglas?

  Yeah. The other day, right?

• 1:44:03John Coogan?

  Yeah.

• 1:44:03Sholto Douglas?

  So diffusion is inherently extremely flops intensive process. Whereas normal language model decoding is extremely memory bandwidth intensive. You're designing two very different chips, depending on which bet you think makes sense.

• 1:44:13John Coogan?

  Yeah.

• 1:44:13Sholto Douglas?

  And if you think you can make something that does flops like four times faster than diffusion and like four times cheaper than your other code, diffusion makes more sense. So there's like, there's this dance, basically, between the chip providers and the architecture. Both trying to build for each other, but also like build for the next paradigm. It's risky.

• 1:44:30John Coogan?

  Do you, I don't know how much you've, how much you've played with image generation, but do you have any idea of what's going on with images in chat GPT? It feels like there's some diffusion in there. There's some tokenization, maybe some transformer stuff in there. It almost feels like the text is so good that there's like an extra layer on top almost and that it's, it's almost like reinventing Photoshop. And, and I guess the, the broader question is like, it feels like an ensemble of models. Maybe the discussion around, around just agents and text based LLM interactions shouldn't necessarily be transformer versus diffusion, but maybe how will these play together? Is that a reasonable path to go down?

• 1:45:09Sholto Douglas?

  Well, I think pretty clearly there's some kind of like rich information channel between, even if there are multiple models there, there's like, it's, it's conditioning somehow on the, on the other model because we've seen before, like, let's say when, uh, you know, models use mid journey to produce images, it's never quite perfect. It can't perfectly replicate what went in as an input. It can't perfectly like adjust things. So there's a link somehow, whether that's the same model producing tokens plus diffusion. I don't know. Like, yeah, can't comment on what OpenAI is doing there.

• 1:45:38John Coogan?

  Yeah. Yeah, yeah. Are there any other kind of like super wild card, long shot, uh, research efforts that are maybe happening even, even in academia where, I mean, this was the big thing with,

• 1:45:55John Coogan?

  what was his name, Gary, uh, he was talking about, I forget what it's called, symbolic, symbol manipulation was a big one. And, and I feel like, you know, you can never count anyone out because it might come from behind and be relevant in some ways. But, but are there any other research areas that you think are like purely in the theory domain right now that are worth looking into or tracking that, you know, low, low, low probability, but high upside if they work?

• 1:46:24Sholto Douglas?

  This is a tough one. This is a tough one. But I will say, it's not a symbolic thing.

• 1:46:27John Coogan?

  Please.

• 1:46:27Sholto Douglas?

  It's crazy how similar transformers are to systems that manipulate symbols.

• 1:46:31John Coogan?

  Sure.

• 1:46:32Sholto Douglas?

  What they're doing is they're taking a symbol and they're like converting it into a vector and then they're manipulating and moving stuff like information around across them.

• 1:46:37John Coogan?

  Sure.

• 1:46:38Sholto Douglas?

  Like, there's this whole like, uh, debate that all transformers cannot represent symbols and they cannot do this. It's, it's, yeah, it's not real.

• 1:46:46John Coogan?

  So Gary Mark is underrated or overrated, I guess?

• 1:46:51Sholto Douglas?

  Overrated.

• 1:46:52John Coogan?

  Yeah, yeah, but, uh, but I mean, if you, if you twist the, uh, if you twist it so much, you wind up with saying like, well, really like the, the transformer fits within that paradigm. And so maybe it's, you know, it, it's, you know, it, like the rhetoric around it being a different path was maybe false the whole time.

• 1:47:12Sholto Douglas?

  Yeah.

• 1:47:13John Coogan?

  Something like that. Um, but, but I, as I remember that debate, it was really the, the idea of compute scaling versus almost like feature engineering scaling and, and will the progress scale with human hours or GPUs essentially? And that has a very different economic equation.

• 1:47:32Sholto Douglas?

  It does.

• 1:47:32John Coogan?

  And it, and it feels like there's been, there's been some rumblings about maybe with a data wall will shift back to being human labor bound. But do you think that there's any chance that that's relevant in the future? Or is it just algorithmic progress married with bigger and bigger data centers in the future?

• 1:47:52Sholto Douglas?

  So I'm pretty bitter lesson built, uh, since that I do think removing as many of our biases and our like clever ideas from the models is really important, just like freeing them up to learn. Now, obviously there's like, there is clever structure that we put into these models such that they're able to learn in this extremely general way. And that, uh, but I, I am more convinced that we will be compute bound than, than we will be like human researcher, uh, our, human researcher bound on, on this kind of thing. Like, we're not going to be feature, feature engineering and this kind of stuff.

• 1:48:23John Coogan?

  Sure.

• 1:48:24Sholto Douglas?

  We're going to be trying to devise incredibly flexible learning systems.

• 1:48:28John Coogan?

  Yeah, that makes sense. Um, on the scaling topic.

• 1:48:33John Coogan?

  Part of my worry is that the ums get so big that they turn into these mega projects that are, that at a certain point, you're bound by the laws of physics because you have to move the sand into silicon chips and you have to dig up the silicon.

• 1:48:50Sholto Douglas?

  It's very certain sand.

• 1:48:51John Coogan?

  Yeah, there's only so much sand and like the math gets really, really crazy just for the amount of energy required to move everything around to make the big thing. Where are you on how much scale we need to reach AGI? How, whether or not we will see like the laws of physics start acting as a drag on progress because it certainly feels exponential. We're feeling the exponentials, but a lot of these turn into sigmints, right?

• 1:49:20Sholto Douglas?

  So I think we've got what like two or three more ums before it gets really hard. Leopold has this nice table at the end of his situational awareness where I think like 2028 or something is when under really aggressive timelines that you get to 20% of US energy production. It's pretty hard to go exponentially beyond 20% of US energy production. Now, I think that's enough. Every indication I'm seeing says that's enough. Now, there might be some complex, you know, data engineering, raw engineering, this kind of stuff that goes into, there's still a lot of algorithmic progress left to go. But I think that with those extra ums, we get to basically a model that is capable of assisting us in doing research and software engineering.

• 1:50:05John Coogan?

  Yeah, which is the beginning of self enforcement.

• 1:50:08Sholto Douglas?

  Yeah.

• 1:50:09John Coogan?

  Interesting. Is that just a coincidence? Like this feels like one of those things, this feels like one of those things where like the moon is the exact same size as the sun in the sky. It's like, oh, it just happens that AGI happens within this time. Like, who, have you unpacked that anymore? Because it feels convenient, not to, you know, I know last.

• 1:50:26Sholto Douglas?

  There's a lot of weird conveniences or like weird, it's a good sci fi story, let's say.

• 1:50:31John Coogan?

  Totally.

• 1:50:32Sholto Douglas?

  You know, we've got Taiwan in between China and the US and it produces the most valuable material in the world and it's locked between the two.

• 1:50:38John Coogan?

  Incredible plot. Yeah. Yeah, really bad for the people that don't think of, that don't believe in simulation theory. It really feels like, oh, this is scripted.

• 1:50:47John Coogan?

  Fascinating. Talk to me more about getting to an ML engineer in AI and kind of that reinforcement. I imagine that you're using AI code gen tools today and Anthropic is broadly and everyone is. But but what are you looking for and what are the what's the shape of the spiky intelligence? Where do they fall flat? And what are you looking to kind of knock down in the interim? Before you get something that's just like, go.

• 1:51:17Sholto Douglas?

  Yeah. So I mean, we definitely use them. The other night I like I was a bit tired. I asked her to do something, just sat watching it in front of me, working for half an hour. It was great. It was truly weird experience, particularly when you look back a year ago. And we're still copy pasting stuff between a chat window and, you know, a code file.

• 1:51:35Sholto Douglas?

  What I like meter's evals for this kind of stuff. So they have a bunch of evals where they measure like the ability to write a kernel, the ability to run a small experiment and improve a loss. And they have these nice progress curves versus humans. And I think this is maybe the most accurate reflection of like what will take for it to really help us at doing progress. And there's a mix here, like where they're not so great at the moment is like large scale distributed systems engineering, right? Like debugging stuff across heaps and heaps of accelerators and like the way the feedback loops are slow and you're actually like the if your feedback loop is like an hour, then it's you spending the time on on doing something.

• 1:52:11John Coogan?

  Yep.

• 1:52:12Sholto Douglas?

  Feedback is 15 minutes.

• 1:52:13John Coogan?

  And for context there, the hour long feedback loop is just because you have to to actually compile and run the code across everything.

• 1:52:20Sholto Douglas?

  Exactly. Spin up all your machines or you need like, I would like you need to like run it for a while to see if something's gonna happen. Like at that point in time, you're still cheaper than the chips. So you're you're you're sort of, it's better that you do it.

• 1:52:33John Coogan?

  Yeah.

• 1:52:34Sholto Douglas?

  But for things like your kernel engineering or for like, you're actually even just understanding these systems incredibly helps. Like I, one thing I regularly do at the moment is in parts of the codebase in like languages that I'm unfamiliar with or some stuff like this, I'll just ask it to rewrite the entire file with comments on every line. Game changing.

• 1:52:50John Coogan?

  Wow.

• 1:52:51Sholto Douglas?

  It's like, yeah, or just come through like thousands of files and explain how everything interacts to me, draw diagrams, this kind of stuff. It's really. Yeah.

• 1:52:59John Coogan?

  Yeah. How important is a bigger context window in that in that example you gave? That feels like something that's important and yet it just naively like Google's the one that has the million token context window. I imagine that all the other frontier labs could catch up, but it seems like it hasn't been as much of a priority as maybe like the PR around it. It sounds like i- is that important? Should we be go- should we be driving that up to like a trillion token window? Is that is that just going to happen naturally?

• 1:53:26Sholto Douglas?

  There's a nice plot in the Gemini 1.5 paper where they show the like loss over tokens as a function of context length. And they show that the loss goes down quite steeply. Actually, as you put more and more and more like of a code base in the context, you get better and better and better at predicting the rest.

• 1:53:39John Coogan?

  Yeah, that makes sense.

• 1:53:40Sholto Douglas?

  The context length, it's a cost. You know, the way transformers work is that there's, you know, you have like this, this memory that is proportional, the KV cache is proportional to how much context you've got. And so you can only fit so many of those into like your various chips and this kind of stuff. And so a longer context actually just costs more because you're taking up more of the chip and you're sort of like, you could have otherwise been doing other requests, basically.

• 1:54:05John Coogan?

  Bringing it back to the custom silicon, is that a unique advantage of the TPU? Is is that something that Google has has thought about and then wound up to put themselves in this advantaged position? Or is it a durable advantage even?

• 1:54:17Sholto Douglas?

  Yeah, so TPUs are good in many respects, partially because you can connect hundreds or thousands of them really easily across really great networking. Whereas only recently has that been true for GPUs.

• 1:54:27John Coogan?

  With NVLink?

• 1:54:29Sholto Douglas?

  Yeah, with NVLink and like the NVL 72 stuff. So it used to be like eight GPUs in a pod and then like you connect them over a worse interconnect. And now you can do 72 and then it breaks down. With Google TPUs, you can do like 4,000, 8,000 of a really high bandwidth interconnect in one pod. And so that is helpful for things like just just general scaling in many respects.

• 1:54:51Sholto Douglas?

  I think it's doable across any chip platform, but it is an example of like somewhere that being fully vertically integrated is a is a benefit.

• 1:55:00John Coogan?

  Yeah, that makes sense. Talk to me about Arc AGI. Why is it so hard? It seems so easy.

• 1:55:06Sholto Douglas?

  It does seem easy, doesn't it?

• 1:55:08John Coogan?

  Well, it certainly seems like more more evaluatable than tell me a funny joke, right?

• 1:55:14Sholto Douglas?

  Yeah, I mean, I think if you are old on Arc AGI, then it would you probably get superhuman at it pretty fast. But I think we're all trying not to RL on it so that it functions as like an interesting held out.

• 1:55:25John Coogan?

  Sure. Okay. Wait, is that just an informal agreement between all the labs?

• 1:55:30Sholto Douglas?

  Yeah, we're trying to have a sense of honor between us.

• 1:55:33Jordi Hays?

  That's amazing. How many people on Earth do you think are getting the full potential out of the publicly available bottles? Because we're now at a point where we have, you know, billion plus people are using AI almost daily. And yet I have to admit, my sense would be it's maybe like 10,000, 20,000 people on the entire planet are getting that sort of full potential. But I'm curious what your assessment would be.

• 1:55:57Sholto Douglas?

  Yeah, I completely agree. I mean, I think that even I don't get the full potential out of these models often. And I think as we shift from you're asking questions and it's giving you sensible answers to you're asking it to go do things for you that might take hours at a time, and you can really like paralyze and spin that we're going to hit like yet another inflection point where even less people are like really effectively using these things. Because this basically requires you to like, it's like a like Starcraft or Dota, like it's going to be like your APM of like managing all these agents. And that's going to be a process. Yeah.

• 1:56:30Jordi Hays?

  So I think Starcraft is such a good example. You think you're just absolutely crushing it. And then you realize like, there's an entire area of the map you're just getting destroyed on. It's such a good, it's such a good comp.

• 1:56:44John Coogan?

  That's great. Anything else, Jordy?

• 1:56:46Jordi Hays?

  I think that's it on my side. I mean, I would like this to be an evolving conversation.

• 1:56:50John Coogan?

  Yeah, this is fantastic. We'd love to have you back and keep chatting.

• 1:56:52Sholto Douglas?

  Absolutely. It's really fun. Love to be back on.

• 1:56:55John Coogan?

  Yeah, we'll talk to you soon.

• 1:56:56Jordi Hays?

  Cheers, Sholto. Have a good one.