AGI-Pilled Cyber Defense: Automating Digital Forensics w/ Asymmetric Security Founder Alexis Carlier

Alexis Carlier, founder of Asymmetric Security, explains how assuming AGI-level intelligent labor should transform cybersecurity from reactive triage to proactive, continuous digital forensics. He breaks down today’s threat landscape—from “spray and pray” cybercrime to nation-state IP theft and North Korean “remote workers.” The conversation explores Asymmetric’s AI agents for deep investigations, their services-first approach to business email compromise, and how specialized digital forensics may differentially accelerate defensive AI capabilities. Nathan uses Granola to uncover blind spots in conversations and AI research. Try it at ⁠granola.ai/tcr⁠ with code TCR — and if you’re already using it, test his blind spot recipe here: ⁠https://bit.ly/granolablindspot⁠ Sponsors: GovAI: GovAI was founded ten years ago on the belief that AI would end up transforming our world. Ten years later, the organization is at the forefront of trying to help decision-makers in government and industry navigate the transition to advanced AI. GovAI is now hiring Research Scholars (one-year positions for those transitioning into AI policy) and Research Fellows (longer-term roles for experienced researchers). Both roles offer significant freedom to pursue policy research, advise decision-makers, or launch new initiatives. Applications close 22 February 2026. Apply at: https://www.governance.ai/opportunities Blitzy: Blitzy is the autonomous code generation platform that ingests millions of lines of code to accelerate enterprise software development by up to 5x with premium, spec-driven output. Schedule a strategy session with their AI solutions consultants at https://blitzy.com Serval: Serval uses AI-powered automations to cut IT help desk tickets by more than 50%, freeing your team from repetitive tasks like password resets and onboarding. Book your free pilot and guarantee 50% help desk automation by week four at https://serval.com/cognitive Tasklet: Tasklet is an AI agent that automates your work 24/7; just describe what you want in plain English and it gets the job done. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai Claude Claude is the AI collaborator that understands your entire workflow, from drafting and research to coding and complex problem-solving. Start tackling bigger problems with Claude and unlock Claude Pro’s full capabilities at ⁠⁠https://claude.ai/tcr⁠ CHAPTERS: (00:00) About the Episode (04:20) Defining AGI and jaggedness (12:27) Modern cyber threat landscape (Part 1) (19:10) Sponsors: GovAI | Blitzy (22:17) Modern cyber threat landscape (Part 2) (29:58) AI-powered cyber defense (Part 1) (33:31) Sponsors: Serval | Tasklet (36:20) AI-powered cyber defense (Part 2) (42:20) Inside digital forensics workflows (51:52) Bootstrapping AI cyber defense (59:17) Shaping the capability frontier (01:08:44) Future of automated forensics (01:17:59) Outro PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Before getting started today, I want to take a moment to introduce our newest sponsor, Granola. You probably know Granola as a leading AI note taker, but it's in fact much more than that. Because it works at the operating system level, it can capture all audio in and out of your computer, allowing it to take notes not just on meetings, every podcast you listen to, every video you watch, and if you choose, everything you say. Right now, to help new users make the most of the platform, Granola is featuring AI recipes from entrepreneurial thought leaders, including several past guests of this show. There's a Replit recipe that converts discussion notes to a Replit app build brief, a Ben Tassel

• 0:43Nathan Labenz?

  recipe that creates content production plans, and a Dan Shipper recipe that looks across multiple sessions to build an unspoken company culture handbook. My own recipe, which you can try now on Granola, is a blind spot finder. It looks back at recent conversations and attempts to identify things that I am totally missing. This has already proven useful in the context of contingency planning for my son's cancer treatment, though I am very happy to report that he is still doing extremely well. And over time, I expect it will become invaluable for suggesting AI topic areas that I've neglected and really ought to explore. For today, my guest is Alexis Karliay, founder and CEO of Asymmetric Security, which just recently came out of stealth.

• 1:27Nathan Labenz?

  In response to the launch announcement, Logan Graham, who leads the red team at Anthropic, described Alexis as one of the most AGI-pilled founders in the space. So naturally, I had to find out what that means for the future of cybersecurity. As you'll hear, Alexis's motivation is increasingly familiar, but nevertheless profound. If we assume that AGI is coming and that it represents a near-infinite supply of intelligent labor, the question becomes, how should we redesign our cyber defenses from the ground up? His answer is to move from a paradigm of reactive emergency triage to one of proactive, continuous digital forensics. We begin the conversation by describing the current threat landscape, distinguishing between

• 2:09Nathan Labenz?

  the spray-and-pray tactics of financially motivated criminals, the more sophisticated ransomware attacks of cybercrime gangs, and the patient, high-stakes IP theft operations conducted by nation-states like China.

• 2:22Nathan Labenz?

  Alexis also shares fascinating details on the North Korean remote worker phenomenon, where state-backed actors infiltrate Western tech companies not just to steal secrets, but also to earn salaries that actually fund the regime. From there, we turn to how Asymmetric is building AI agents capable of performing the deep investigative work that was previously only available from a very limited number of expensive human experts. We discuss the jagged frontier of current model capabilities in the security domain, and why, though off-the-shelf models can already achieve 90% accuracy on many investigative tasks, Asymmetric is going to market with help from insurance companies with a services-first business model focused on business email compromises, both to ensure that they deliver

• 3:07Nathan Labenz?

  consistently for customers and to build the proprietary dataset needed to close that final gap in performance and reliability. Importantly, Alexis also makes the argument that while most AI technology is inherently dual-use, the fact that people who specialize in these investigations don't tend to become particularly outstanding hackers suggests that digital forensics could be a domain that allows us to differentially accelerate defensive AI capabilities. And arguably, it's this strategy of intentionally shaping the AI capability frontier that is Alexis's most important contribution, and I would love to see this approach ported to other domains. The opportunity to build specialized datasets, evaluation methods, and training environments

• 3:51Nathan Labenz?

  that can help harden society's defenses seems both increasingly tractable and urgent. So if you're thinking this way and building, whether in cybersecurity, biosecurity, mental health, or anything else where a DAX strategy could make a big difference, please reach out and tell me about it. With that, I hope you enjoy this overview of the cybersecurity landscape and preview of the future of automated digital forensics with Alexis Carlier of Asymmetric Security.

• 4:20Nathan Labenz?

  Alexi Carlier, founder at Asymmetric Security. Welcome to the cognitive revolution.

• 4:25Speaker 02

  Thanks for having me.

• 4:27Nathan Labenz?

  So yeah, I'm excited about this conversation. An interesting little endorsement of you and your point of view that I saw online on Twitter from Logan Graham at Anthropic, who leads the red team efforts there and really kind of is at the intersection of national security concerns and what can models do and has been deep down the rabbit hole of how weird the world might get in the not too distant future. He said about you in response to the announcement of the new company that few people are more AGI-pilled than you. So I wanted to start off by just getting a little bit of your worldview. You can tell maybe a little bit of your background too and kind of how you came to be one of the most AGI-pilled people around, but how did that happen?

• 5:09Nathan Labenz?

  And even more so, like, what does it mean to be so AGI-pilled?

• 5:13Speaker 02

  Yeah, I saw this. I suspect what he's getting at is I've taken throughout my career professionally, done a lot of stuff that's been a really premise on the idea that AI and AGI is going to be a big deal.

• 5:27Speaker 02

  So a few years ago, I was on the early team of the Center for the Governance of AI, which is a big AI governance thing, especially prominent in the UK and US and elsewhere. And this was like in the pre-ChatGPT, I don't know, 2020, 2021 days when AI people were talking about it, but it wasn't really this big, this big thing it is today. And back then, we were trying to think about what would happen as we saw that it was going to be a bigger and bigger deal. And there were just very few people doing that at the time.

• 6:00Speaker 02

  And today, sort of continuing that with Asymmetric Security, where we have quite a distinctive thing we're doing in the space of cyber defense. And I think because we're taking AGI very seriously, like one frame on what we're doing is we're like assuming something like AGI will exist in the world. And then we're like, okay, what makes most sense from a cyber defense perspective, assuming that? And that takes you to quite a different place than like lots of other security companies. And if we're wrong, you know, the company is like, maybe it's still valuable, it's way less valuable than it would otherwise be. And so it's like these big costly bets on like, yeah, this is the thing that's happening.

• 6:37Nathan Labenz?

  And what does AGI mean to you? Because we've had multiple rounds now, like Tyler Cowen at the 03 launch said, okay, I'm calling it. This is AGI. Friend of the show, Dean Ball, called it with Claude code with Opus. And I have to say, it's certainly getting there, right? Where do you think we are on the, is AGI here or not? If not, what is missing in your perspective that you think is kind of going to tip us into a stranger future?

• 7:08Speaker 02

  Yeah, that's how I think about AGI. It's not here yet. I have in mind more the like drop it remote worker style version of things where you're really fully substituting for remote work that humans do, including for like long horizon types of tasks and you can really fully sub in for humans at the moment. You know, we have this weird jagged frontier of capabilities where the models are geniuses at some things and then just cannot like string together tasks over long enough horizons. And I think that sort of explains why we have this weird paradox of, you know, on some dimensions, clearly they should be human, but then they clearly are not having that much economic output just yet. And so I think for me, we will see it in the economic output and GDP statistics and so forth, when we really have this like you're actually subbing in for human work.

• 7:54Nathan Labenz?

  I find the GDP question so confusing because so many of the most impressive moments that I've experienced with AI have been GDP destroying because it's when it's effectively substituted for something that I would have otherwise had to go out and hire done, that I'm like, this is where I'm really feeling the AGI. And so I have kind of a strange confusion, I guess, around like more output, sure, you know, I would expect more services rendered, right? More legal services, more medical second opinions, obviously lots more code, but potentially at a lot smaller prices than previously. So that's kind of a weird, I don't know if you have a point of view on that, but I still kind of find myself

• 8:39Nathan Labenz?

  just like, I don't know, the frontier model developers revenue is exploding, but we don't quite see it in the GDP statistics, but I'm not even sure what I would expect to see. Like in some ways, I might expect to see GDP shrink.

• 8:53Speaker 02

  That's interesting. Yeah, I'd say probably the more relevant thing to be tracking actually is just output. And there, I think very clearly, we should expect a lot more than we're seeing today.

• 9:02Nathan Labenz?

  Yeah, it obviously becomes a little tricky. How do you measure output? Because we have, you know, dollars are the standard way and it's, yeah, gonna be tricky. I guess, just because I've been so deep down this, right, like, like so many, I don't know where your Claude code or open code or, or bot journey is at the moment. But I have been, you know, like many, trying to figure out how I can create leverage for myself. I've done, over time, like, tons of task automation type things where I break down a task and make a little eval set and build a workflow. But that doesn't feel like AGI. That feels like task automation. Now I'm kind of like, oh, we're getting definitely a little closer with

• 9:44Nathan Labenz?

  this Claude code setup. Like, it can figure things out in really remarkable diversity of different situations. The one thing I have noted, and I'd be interested in your reflections on this, is, and it feels like it's probably going to be pretty easily fixed, but the one thing I've noticed recently is that the Opus 4.5 in Claude code has a strong tendency to try to write code with heuristics for things that I really just wanted to spend tokens on to understand with its own fluid intelligence. So, a random task was backfilling transcripts for the podcast onto the website, because we didn't do that in the early days of the podcast. And I was like, well, it'd be really, be nice to go back and have transcripts for

• 10:28Nathan Labenz?

  all those episodes. But so many times in the backfilling process, I see it like, write a little Python script with guesses about how things are going to be. And then that doesn't quite work. There's too many edge cases, whatever. And I'm just like, all capping it, use your fluid intelligence, just read the file. You know, if you just read the file, you'll know what to do. Don't, you know, write these kind of roundabout Python scripts. That's, I definitely think that speaks to jaggedness, and it also speaks to, in my mind, like,

• 10:56Nathan Labenz?

  How subtle and kind of honestly minor some of the major weaknesses still are like it just has this strong impulse to try to code up heuristic based guesses about how things should work when it's like if you just read it, you know, you're definitely smart enough, Claude, to to know what to do. But it just has kind of some wrong impulses in a few different ways. I mean, maybe we should save your response to that and contextualize it in what you're actually building. But if you have any immediate reactions, I'd be interested to hear them as because I do. I mean, Logan commenting that you're one of those AGI filled people out there. I think that is that is a strong statement, you know, knowing who he spends his time with. So I am interested in kind of your take on how you understand

• 11:40Nathan Labenz?

  jaggedness and how soon it might be resolved.

• 11:44Speaker 02

  Yeah, I think my I don't my take here is roughly that RL is a big part of what's going on. Like with the pre training paradigm, you were getting a lot of generality kind of for free because you're reading all the internet and the model was just improving in this very predictable way. And I think a lot of what's happening now, and I think Kepathi and others have said this as much publicly, is because you're you're sort of relying on verifiable rewards, the areas where they're getting good are really like kind of predictable things like coding and math. And so I actually suspect it'll be, you know, until we have some big other paradigm, I don't think it's going to be to get that great at writing like poetry, for example, for quite a while. That's that's generally the like high level.

• 12:28Nathan Labenz?

  Well, put a pin in that. We'll come back to some of those related questions as we get deeper into the problems you're solving and the way you're solving them. I'd love to take a little survey of cybersecurity broadly and how AI is beginning to impact that and how you expect it to impact it more and more over time. Again, this is an area that really confuses me. So this is a great learning opportunity. On the one hand, I kind of work from the assumption that like, everything is hacked all the time. And I've gotten so used to it that I would confess to not my like key, I've, you know,

• 13:06Nathan Labenz?

  Privileged passwords or passwords I don't reuse that are like for key accounts like my Gmail and so on. And then I just have kind of throwaway passwords that I use for like long tail stuff. I'm like, ah, if anybody compromises any of these services, I don't, I could probably live with that. And I've just gotten to the point where I like click ignore when Chrome pops up that one of my passwords has been found in a breach. So that's the level of security I'm operating with. And I would have to assume that like, you know, it puts me in pretty good company as being certainly exposed in some ways. And yet, life mostly goes on and like, not only does like critical infrastructure continue to run, but like even I, you know, with my terrible practices, like mostly don't have any problems. So I'm

• 13:46Nathan Labenz?

  a little bit confused about the landscape today, including like, who are the attackers? What do they want? Who are they attacking? How are they attacking? That's kind of a baseline. And then, you know, how is AI starting to change all that? I'd love to just get kind of a survey rundown from you.

• 14:01Speaker 02

  Sure. Yeah, I think broadly, the majority of attacks are by like relatively unsophisticated amateurs or criminals. Maybe this is like eight, roughly 80%, something like that. And often these are financially motivated. And then the other end, you have these like very sophisticated attacks from nation states, like a small minority from a volume perspective, but often they're going after like really critical IP or things like this. And so some of these like low sophistication groups will do things like spray and pray attacks, sending out like phishing emails to as many targets as possible. They'll be like script kiddies, so like using malicious programs that others, more sophisticated folks have come up with, but just like sending them out on mass and hoping it works out. That's sort of one end.

• 14:44Speaker 02

  That if you, you know, if you were being targeted yourself, that's probably what you would be seeing, probably because you don't have, you know, a ton of super sensitive IP or stuff that really sophisticated folks are going to come try steal. And probably you're just like, you're not going to click the phishing email and you're fine. And then the other end is like, you know, the nation states. And here it's a very different ballgame. Like a few important actors here, you have China, a big one, obviously the CCP. A big focus of theirs is IP theft from R and D heavy industries in the West. AI is almost definitely like a huge focus now. Russia is another big player. They're interestingly a bit more focused on political disruption. So famously, there was the 2016 US election disruption. And another big one here is actually North Korea, which has very different motivations.

• 15:29Speaker 02

  Actually largely about funding the regime. And a big thing they do here is they have this like North Korean remote worker program where they send literal North Korean operatives to just get tech jobs as remote workers in foreign companies just to fund the regime. That's like a pretty unusual one, but it happens a lot.

• 15:45Nathan Labenz?

  I've heard a little bit about that, but I didn't realize it was like a...

• 15:50Speaker 02

  Yeah, it's a thing. We actually just had, we just brought on someone from CrowdStrike who works some of these cases where they, the companies in question realized that something was up with their employee. And they were both sending back money and sending back various bits of IP.

• 16:06Nathan Labenz?

  So the play is to get the job, get access to either information or some sort of financial

• 16:12Speaker 02

  Exactly.

• 16:14Nathan Labenz?

  tooling that they can then siphon money off.

• 16:16Speaker 02

  Exactly.

• 16:17Nathan Labenz?

  Get the Brex card, get the ramp card ASAP. Yeah, that's fascinating. Okay, so we got China, we got North Korea, Russia, obviously, a player in this as well. Is there a middle? There's also kind of, like, who is it when a hospital, this recently happened in a, to a medical system locally here. I went to see my primary care physician and he was like, oh, it's been a nightmare, you know, our stuff is totally inaccessible and there's some, you know, cyber gang somewhere that is demanding ransom or whatever and they don't want to pay it. And so I'm like taking notes on paper. Who's that?

• 16:57Speaker 02

  Yeah. Yeah, that's a, that's a, one of the most common types of attacks actually is ransomwares. And these are often organized criminals who are like moderately sophisticated and act in this coordinated way. In particular, what they tend to do is they like find ways of getting access to systems, encrypt it all, and then just put up a, like a notice on the screen, be like, okay, if you want your data back. Sent us a bunch of money. This is like, it was a huge pop off during COVID, actually. And this economically is one of the like most damaging things. Like relatively recently Jaguar had a, had a big ransomware case. And if I recall correctly, they needed like a $2 billion loan from the UK government. After this happened to avoid going under and their operations were down for like

• 17:42Speaker 02

  a month, two months, something like that. So very significant economically. Interestingly, the, the like type of, the type of like attack they're doing means they, they operate very differently to nation states. Whether the motivation of the ransomware focuses to be discovered as soon as possible, once they've encrypted the data. The nation state, like more sophisticated guys on the other end are just trying to stay sleuthy for as long as possible. And so you end up with these like very different modes of operation.

• 18:07Nathan Labenz?

  Yeah, okay, that's quite interesting. When you talk about like a $2 billion loan. That makes me wonder how much of the, you know, in like business, there's always this question of like, how much consumer surplus are you creating? How much of the value that you're creating are you, you know, as a business able to capture? How much of the damage, I guess, that the cyber criminals are causing? Do they manage to capture for themselves? Like if Jaguar, if the damage to Jaguar was X billion dollars? That doesn't mean they paid X billion dollars to the hackers, right? Is it like a 10 to one ratio, 100 to one ratio? Do you have any sense for how much they're actually able to get from the damage they do?

• 18:51Speaker 02

  Yeah, I don't know the specific numbers, but it's incredibly negative, so. I think order of 10 to one sounds right. It's just incredibly economically damaging. And I've heard, I've heard some economists claim that if you sort out this sort of cybercrime, it's actually a huge effect on growth. Cause you just, it's so value destroying.

• 19:06Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 19:10Nathan Labenz?

  In terms of the baseline situation, how, of, of the attacks that are happening? And maybe you could, I don't know if it makes more sense to segment this by like volume, probably not, maybe more like by damages to the degree that we can, how many of these attacks are of the social engineering variety versus?

• 19:30Nathan Labenz?

  Finding purely technical exploits where somebody left, I don't know, a port open or whatever the kind of common things are. And how many of them are like these real cinematic type of things where people are really figuring out exploits that like nobody knows about? Obviously, that's got to be relatively rare. But how should we understand like how much damage each of those kinds of things is doing?

• 19:50Speaker 02

  Yeah, I think a bit hard to say in terms of damages. A bit more clear on volume. So the majority is social engineering or phishing. Hard to say exactly, but say like roughly 70 to 80%. The reason for this is kind of simple. Just attackers choose the path of least resistance and there's no reason to burn like say a valuable zero day if you can just get away sending a convincing email. And so that's what most people do to start with. And in terms of the technical risk, most actually comes from known issues. So it's like vulnerabilities that people are aware of and just most systems have not been patched. Yeah, just the like a large fraction of what happens is just people haven't done the basics. They haven't patched what's known. It's like very solvable stuff.

• 20:33Speaker 02

  Zero days, on the other hand, are a lot more rare and tend to be exploited by the most fiscated actors like nation states. But you hear a lot more about them than you would expect based on what's like what you're actually seeing in the wild in terms of what attacks are going on all the time.

• 20:45Nathan Labenz?

  So what works in terms of defense? Like turn on your two factor off? How much does that get you? Update your operating system? How much does that get you? If I do those things, am I good? You know, or how much risk remains if I do kind of the the basic common sense things?

• 21:05Speaker 02

  Yeah, it depends a lot on who like good against who. For most people who are not, you know, the targets of nation state attacks or or aren't they corporations we're going to get ransomware, doing the basics like MFA, regularly updating your software, like checking your your supply chain of vendors, checking their security, having some sort of like automated monitoring systems, doing occasional compromise assessments. So like proactively looking to assist them to see if it's been compromised. That will, you know, get you pretty far against lots of these amateurs and some organized crime as well. Frankly, on the other end, it's just extremely difficult to defend against nation states. It requires, like, if you already want to do this and like in a very determined way, if you've got some IP that's super important,

• 21:49Speaker 02

  you need things like extremely strong limitations on software and hardware providers. Like, you probably just couldn't use most SaaS, for example. You need to be super careful about who your employees are and vet them in a way that's just super outside the norm and like plausibly illegal in some contexts. You need to be way, way more stringent on proactive security, like searching for zero days in your system, doing compromise assessments all the time. It's just like very, very difficult to do. And most organizations are just not there, which is why most nation states just have a lot of ongoing intelligence operations that are stealing IP all the time.

• 22:25Nathan Labenz?

  I've been advised, I didn't make it this past summer, but I want to go to China and participate in some form of intercivilizational AI safety, you know, dialogue and hopefully mutual understanding building. And when I was thinking about going this past summer, people advised me, okay, you can't take your devices. You have to get burner phones. You know, the second you get back, you got to throw that phone away or you could, you know, maybe sell it on or something. But like, absolutely, don't take your devices. Don't log into anything while you're there. You know, do all, take all these like extraordinary measures. Do you think that's necessary if I am just a regular American kind of AI yapper, taking a trip to China?

• 23:09Speaker 02

  Certainly, if you didn't do it, you would probably get people reading over your stuff.

• 23:14Nathan Labenz?

  And by that, do you mean like they demand my phone at the, at the airport and take a look at it? Or that they have a way of getting into an Apple device and gaining?

• 23:25Speaker 02

  Yeah, you just have, you should assume if you do that, that people are monitoring your communications in an ongoing way. And maybe that's fine, like for normal people, maybe that's often fine. I suppose most people would not love the idea of the CCP reading all your stuff. But yeah, I think people generally write that if you if you go to China, you should expect stuff like that to happen, especially someone like you with a public profile and so on.

• 23:44Nathan Labenz?

  And that would continue, like, just to understand the architecture of the surveillance, it would be something they would be putting on the device such that when I take the device home, that comes with me, not something that like, because I'm on the local network there that they would have access to only while I'm there. Is that the correct understanding?

• 24:03Speaker 02

  Yeah, you should assume they should have some way of getting persistence mechanism. This might be a physical thing, or it just might be they've got access to your own credentials or something like this, but I assume they'll have persistence.

• 24:16Nathan Labenz?

  So that that's a little bit like, how should I sort of reconcile the idea that I'm not that important, for one thing, with the idea that there aren't that many zero days to be burned? Presumably, Apple, I think I should trust in terms of, you know, they've got a big team of people that are working on this all the time, right, and identifying these things and closing them down. So how is it that they have so many of these mechanisms that they would use one? I can't be that high on their power rankings of targets, right? So are there just a lot more zero days than I have been led to believe? Or like, how do I resolve that that seeming tension where I just don't feel like I should be worth

• 25:01Nathan Labenz?

  using these things? Because presumably, that gives Apple some ability to like, figure out what's going on and and close it down each time they use it, right?

• 25:09Speaker 02

  Yeah, Apple's security is generally pretty good. I think broadly what's going on here is just, it's extremely difficult to get visibility on what nation states actually have access to from an offensive security perspective. And the amount of effort in terms of like, people going into this is, you know, estimated, I think, in the hundreds of thousands for the CCP. So there's just a lot of man hours going into finding zero days and finding ways to break into systems. And often, like, there are many cases where access that nation states have had weren't discovered publicly until decades later. And so we should just expect that there's a lot of forms of access that very sophisticated organizations like these have and are using all the time. And

• 25:52Speaker 02

  Yeah, it is the case that, you know, for most normal people, you won't be hiding the priority list. And so they're not likely to, you know, spend these like very, these zero days if they expect to be discovered. But certainly it's within the realm of capability. And in many cases, they probably can just get away with stuff without anyone knowing for a very long time.

• 26:11Nathan Labenz?

  So how do these things tend to come to light? This starts to get a little bit into what you're doing. So I don't know if it would make sense to talk about how people discover things first, or maybe just talk about how the landscape is is changing in light of AI. Because I think pretty much everything we've talked about so far has been

• 26:32Nathan Labenz?

  baseline, like you could have said all the same things pre GPT four, right? So yeah, maybe what's kind of what's changed since GPT four class models have come on the scene? How has this landscape started to shift, if at all, in a meaningful way? I again, it doesn't feel like it's changed my life yet, but everybody's telling me it's going to.

• 26:52Speaker 02

  Yeah, I think for the most part, it hasn't actually had like a huge effect just yet.

• 26:56Speaker 02

  There's been very, you know, it helps with obviously you could automate phishing emails and things like this, but there hasn't been this like really big uplift from an offensive perspective yet.

• 27:05Speaker 02

  I think that's about the change. Or it sounds like on on most measurements that the labs and others are doing, the models are already on the precipice of these offensive security capabilities. And broadly, I think the way to think about the effect here is like bringing down the threshold of sophistication necessary for any given kind of attack. So attacks that pretty, you know, say moderately sophisticated ransomwares, assume that a load more people will be able to do this all of a sudden.

• 27:31Speaker 02

  That's not coming on the line yet, but will be coming on soon. And from a defensive perspective, I think also hasn't been super transformative yet. And again, I think probably about the change and that's part of what we're we're building at least.

• 27:44Nathan Labenz?

  So how do we measure these capabilities? There it's one of these weird things. I mean, I guess this is becoming increasingly common, right? Where it's like across the board. When the tasks were easy and anybody could tell if the AI was doing a good or bad job, things were pretty easy. We're now in so many domains in a world where very few, you know, because the performance is so good, very few people can even really critique it. I'm staring down the barrel of a, an interview next week with the founders of Harmonic, who created this Aristotle model system that got the IMO gold alongside, obviously, OpenAI and DeepMind. And it's like, damn, you know, I would score zero on that test.

• 28:29Nathan Labenz?

  So how do I even understand what's going on? I basically feel the same way in cybersecurity and it feels like there's a very, it seems like very limited number of data points we can get to measure how good these things are when, you know, I know there are some things like, well, there's been some zero days since the, since the training cutoff, so we can like see if they can do those. But my sense is those are like pretty small number. So how do we even get a handle on what the model capabilities are in a way that we can like, you know, trust or base our plans on?

• 29:06Speaker 02

  Yeah, I think this is super tricky in cyber for a reason that you were sort of gesturing at. One big thing going on is unlike, say, software engineering where you have a ton of like code on the internet, most cybersecurity stuff is just not public. Like in the context of incident response, for example, when a company gets hacked, they're not going to share the like logs from the email getting count publicly. And so it's very tricky to actually get a sense of like what, how exactly the models are performing in these contexts, because you've got nothing to benchmark on. And I've talked to some of the folks at the Frontier Labs, this is a big bottleneck right now on trying to evaluate because you just have nothing, nothing to benchmark on. You're not seeing all the attacks that are happening out there. So this is a big problem.

• 29:49Speaker 02

  And secondly, also, yeah, it is the models are getting sufficiently good that you do need a bunch of subject matter experts who themselves are very good on this, good at this. And there aren't just aren't that many such people. Like CrowdStrike, one of the, one of the top incident response companies in the world, they have a team of roughly 60 people, it's not huge, who can do this really to sort of deep forensic defensive side security investigations. So you're just really constrained on talent and really constrained on data. And it does make it very hard to know what's going on. By default, I think. I think there are ways to solve this, but by default, things are tricky.

• 30:19Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors. So I appreciate you taking a, a long time here to just give me the kind of baseline lay of the land. How are you going to make this better for us so we don't have to worry about this? And maybe, you know, could start with a little bit of like, what does it look like today when a company starts to get the inkling that they have been pwned? How does that come to light? What do they do today? And obviously you have a different vision for how that can work in the future. And I think being more thorough, proactive, you know, playing to AI strengths are a big part of that. But take us through the kind of before and after of how things are and how you hope you can change them to be.

• 31:01Speaker 02

  Yeah, so the problem we're trying to solve basically is make, make it much, much easier and much more accurate to detect when you've been breached and have that happen as quickly as possible. Today, roughly speaking, you have these monitoring systems, detection systems that are based on like static rules, like basically software that hard codes, oh, this thing is maybe suspicious. And this kind of works a little bit, like it's, it does do something. But you end up in a situation where you get a lot of false positives, a lot of alerts going all the time. And the reason for this basically is that a lot of activity that could be suspicious is also

• 31:43Speaker 02

  just could be normal. So like if you're logging in from different places or devices, yeah, that is maybe weird, but also maybe you're just traveling to bed this week. That happens. Maybe you're just up late in the middle of the night and that looks suspicious, but it also could be normal. Maybe you're downloading files in bulk. Again, sometimes you just download files in bulk. And without ability to like reason over the forensic evidence in a lot more detail, you just have no way of distinguishing between these things. Also, by the way, on the, on the other hand, there's this like genuinely suspicious behavior that looks normal. So if you're trying to exfiltrate data, you might just over the course of months, exfiltrate like very small amounts, which are just not very noticeable. And then that typically detection systems at the moment won't pick up on at

• 32:25Speaker 02

  all. And again, if you had something like a human who suspected there was something going on here and had the time and energy to reason over all this evidence very deeply, you'd probably be able to figure it out. But that's not how this stuff works right now. And so generally, status quo, if you have some alert go on in, in say, an enterprise, the security operations center that handles all this stuff will do triage. So an initial basically rapid assessment to figure out when a lot fires fast and quickly, like, is this something worth prioritizing? Is this a real alert? Is it a false positive? Can it be ignored? And then in the minority cases where they're like, yeah, this is actually

• 33:06Speaker 02

  a suspicious thing. Let's pull out and do digital forensics, which is this sort of this methodology of doing very deep security investigations, where you're trying to figure out in a lot of depth, what exactly happened, looking at every evidence source you need to, to figure out all the details. The intuitively is just like, you know, a detective, like a forensic detective, and you're, you're sort of reasoning over all the evidence you need in this, in this sort of like judgment based way. So that's the status quo. I think what will happen, you can sort of think in principle, what would be ideal here? We know that digital forensics is the sort of best way of doing this if you had sufficient time and sufficient energy and sufficient labor to do this. The reason you can't right now is because you're just bottlenecked on people. There are just very few people with this expertise, they're very expensive and they

• 33:50Speaker 02

  work slowly and it would just be impossibly expensive to do this. And so at the moment, it's this thing that's most used reactively after a breach. That, I think, will not stay the case going forward. Like one more general intuition here, I think a lot of people, when they think about what to do with AI, like when you're building AI agents, what should you focus on? Have this frame of, oh, this is like a human workflow that's really important in the world. And what if I could just automate that work with AI? And that can work, like obviously, you know, you have, you're automating legal work, you can do stuff like this. But I think a better question is like, what would you do if you had 100x the intelligent labor at your disposal, something like that?

• 34:32Speaker 02

  And I think the answer is that for many kinds of work, you can perform work that was previously too slow, too expensive, you can perform it at scale now because of the change in economics. And that's the bet we're making in the context of defensive security. This domain that is currently super accurate and super costly only really applied proactively. You could just imagine, you know, eventually at the limit, getting this done to the cost of compute and you just have these sort of close to continuous proactive assessments of systems in a very deep way. And that's, that's what we think will happen eventually. The question is how quickly can we get there and can we get there soon enough that we can deal with a lot of the security risks that are going to come on with, with AGI and, you know, offensive security changing and that kind of thing.

• 35:18Nathan Labenz?

  Yeah, I like the, the frame of, okay, you could take, usually for guidance, usually say, you can probably save 90% time and money on some task that's being done by humans if you automate it with AI. But yeah, I think you're totally right to emphasize that the other side of that coin is what can you scale that was just previously impossible to scale? And

• 35:46Nathan Labenz?

  This is an interesting one. Can you just, can you describe a little bit more what the humans do? So let's say I'm breached, I don't know, maybe set the scene in terms of like, who your, you know, ideal customer profile is and how it comes to their attention that they've got a problem and then who they go to and like how long it takes these people to engage. And then if I just were to sit there and watch what they do, what are they doing? Are they just like grepping through logs, kind of using the intuition they've built up over time? Or like, what is that sort of story? And then again, we'll, we'll, we'll switch to the, the future state story next.

• 36:25Speaker 02

  Yeah. So to make this concrete, so we're, we're currently focused on what's called business email compromise, more and more true to just email based cyber attacks. And broadly, what this is, so like the common modality here is an attacker, someone got access to an email account. Maybe they've sent out a phishing email that someone's clicked on. Maybe they've got access to your credentials some other way. And they're trying to use that access to trick people into doing stuff like sending them information or sending them funds. So this can often is financially motivated, but can also be quite different. So like when the DNC was hacked by Russia in 2016 was an email based attack. And so,

• 37:07Speaker 02

  we asymmetric, so we come in and we, we respond to these attacks as do other folks, cybersecurity folks. And what this generally looks like is, okay, you come in and you pull down all the email logs from an environment. So the, the person, the enterprise has been hacked, they might be using Microsoft, they might be using Google. And you look through the logs and first of all, you're trying to figure out like, how did they get access? Like, when did they first get access? Did they then escalate their access in various ways? Did they get access to another account, something like that? And very concretely, it's like looking through email logs and trying to be like, okay, this is like a weird location to be logging in from. And maybe like the user was in two places at the same time, that's impossible. And so this is, this suggests something, something is suspicious here. So

• 37:49Speaker 02

  you figure out what, how they got in, then you figure, okay, what did they do? You again, looking through the, the email logs, be like, okay, did they send emails? Did they read them? Did they delete them? Often people will set up like automatic inbox rules. So like automatically forwarding emails, automatically deleting invalid emails. Are they looking at files in the Google Drive? So you're painting this like very comprehensive picture of what happened. And also whether, you know, whether the attacker is still in the, in the network. And then finally, you're looking at, okay, how did this happen? Was it a phishing email? And at this point, you also sometimes look at the email bodies themselves. And so you have a pretty clear like sequence of things you're trying to do. But you're doing this in this sort of very like flexible, human way of like, okay, this looks kind of suspicious. This looks funny. Let me pivot off this bit of evidence. Let me think what comes next.

• 38:33Speaker 02

  And so it's been hard up until very recently to actually automate this in a meaningful way.

• 38:37Nathan Labenz?

  How much does access to? Like if you're running your own enterprise email server versus your like a Google customer? And your employees have Gmail. How much does that change? How this goes down? If you're a Gmail customer, you obviously, or I would presume, you know, you can't just get that level of access, right? So do they have a team that you work with to try to resolve that sort of stuff? Or how does it work when when you've got like a a SaaS provider like a Gmail?

• 39:10Speaker 02

  So most tenants, like a user tenant, will have a bunch of logs that you just use. So actually, they do have all the logs you need. You just get admin privilege to those to those accounts and you can go from there. The provider does actually matter a lot. So for example, there are far fewer Google compromises of emails than Microsoft's. But like, I think at least an order of magnitude. So big differences based on how they've configured the environments.

• 39:34Nathan Labenz?

  Do we know what causes that? Or is it if it's that big of a difference? Like, why isn't why has Microsoft not managed to close that gap?

• 39:41Speaker 02

  Yeah, I can't recall the specific reason, but it is like this. It is this very tractable thing they could be doing. That I just are not have not been doing for reasons that are kind of baffling to me. I guess it just doesn't really affect him in this big way.

• 39:55Speaker 02

  But yeah, I know some folks who have considered working in this space and building very spits of tech to help deal with these attacks. And they were like, now probably we shouldn't do this because Microsoft is going to solve it one day. But yeah, up until now, it's not been the case.

• 40:09Nathan Labenz?

  Yeah, that's funny. Okay, so how do we get AI to do this? You know, and where are we right now in terms of? You know, how good is Claude Code? And, you know, where does it fall short? And what are you building to make sure it actually works?

• 40:25Speaker 02

  Yeah, so I can tell you better about what we're building in this context. We are a full-stack AI digital forensics and incident response company. Where the mission is to accelerate AI cyber defense, in particular by trying to automate this field of digital forensics as quickly as possible. And we're doing this broadly by having these human AI teams doing these kinds of investigations. And so we're currently focused on these email investigations. And so concretely, what this looks like is we go in and do the kind of things I was just talking to you about, but on this AI platform we've built. That ingests all the logs, broadly, we have an agent that will do a first pass analysis over these logs. And then the human investigators will click through and basically try to do something like a QC, most of the time, to just check the quality of the reasoning.

• 41:10Speaker 02

  And in a minority of cases, they do need to meaningfully change what's been done. And so based on what we're seeing in these initial cases, and for context, these email investigations are like on the much simpler end of investigations. The models are like pretty good already, even without doing anything fancy, without training on specific data, without anything like this. You can get maybe something like 90% accuracy, something of that order of magnitude out of the box. That is super helpful for like speeding up the process of this investigation. It's totally insufficient for actually automating this work. This is a context in which you need like very high accuracy for various reasons, which means you do completely need the humans and you do, like it's also an area where the lines of reliability just matter a lot.

• 41:54Speaker 02

  So for the foreseeable future, and certainly as we get to more complicated kinds of investigations, it does feel to me like this would be an area where there's this long tail of needing humans to be around to really like push up the lines.

• 42:08Nathan Labenz?

  Okay, that's really an interesting kind of possibility, you know, production possibility frontier that I'd like to understand better. But just as a little context, the

• 42:22Nathan Labenz?

  I think that, you know, this is going to play out, I suspect, in very different ways in different domains. And I'm always kind of like, well, jeez, if AI can do 90% of the work, how much does that leave for humans? Is it 10%? Or is it still like, can they finish it in 10% of the time that they would have had to spend? Or do they still have to spend half the time that they would have had to spend, even though in some sense 90% is done? You can imagine that last 10% taking longer. And then, of course, there's also the question of how many nines do you really need? Or can you even measure in a given context? And then there's the question on top of that of like, how much latent demand is there for different kinds of services, depending on how much the overall price can

• 43:06Nathan Labenz?

  come down, based on how much can be automated and how, you know, how much more productive the humans can be, and whether or not they can hit key reliability thresholds? So my usual kind of somewhat tongue in cheek way of saying this is like, on the one end, we have dentistry, where I do not want any more dentistry, no matter how cheap it gets. I want the minimum, you know, ideally zero. And it's never something I want. And making it cheaper doesn't, doesn't really entice me. On the far end, you might put something like massages, where I potentially get a massage every day if it was, you know, close to free. I have limited, I mean, I guess one of the big bets that you have is that

• 43:47Nathan Labenz?

  there is a ton more demand for this than is currently served. I would think about the thresholds that matter in terms of, you know, the ratio of human productivity that would then, you know, enable, subject to certain, you know, reliability thresholds that would then enable like the vast explosion of this market that you're obviously trying to unlock.

• 44:15Speaker 02

  Yeah, so I think a couple of ways to think about it. From the perspective of just like applied reactively in the context of incident response, that I think is a lot is not of the form that the demand will hugely grow here. Or rather, it's not of the form there is a bunch of latent demand. I do think demand will grow because there will be a bunch of AI-enabled cyber attacks. But it's not of the form people, if they could have more of this, they would want more. It's just like, after you've been, you know, you're bleeding, you're hacked, you need to deal with it. That's basically the demand. On the other hand, if you get to this, this paradigm where you're using this sort of investigative reasoning but applying it proactively, then I think it sort of ends up substituting for the current approaches to doing detection in cybersecurity, which are not at all based

• 45:00Speaker 02

  on this digital forensics-type approach. And it's a much, much bigger market on the order of a few hundred billion, something like this, maybe five, 500 billion. And so I think as a sort of like a bottom line here, at minimum, you're substituting for this existing huge market of detection. Then there's additional question here that I think is sort of an open question of if it becomes really cheap to get high levels of security, like much better than what you currently have in detection, what is the demand for that? That, I think, is more uncertain and depends probably a lot on the type of actor. There'd be some, some organizations where it's just you're just doing, you're getting what you need from security and it's like a meet the minimum bar thing. Then there'd be other organizations, say like AI labs who are trying to protect really

• 45:41Speaker 02

  sensitive IP, or national security agencies and governments, who have this much high demand for like very high levels of security. And I think those are the areas we should expect this, like additional demand above and beyond just substituting for the current approaches to detection.

• 45:57Nathan Labenz?

  So, how do we get there?

• 46:01Speaker 02

  It's a great question. So this actually relates to something you brought up before around the difficulty of measurement in this space. And I mentioned these two boneless around actually having access to data or like context on what's, what's actually happening, what these incidents, what these cyber attacks look like. And then having the people that can actually like assess, assess performance.

• 46:25Speaker 02

  At the moment, the status quo is that it's very difficult, and we've talked to many of the folks at the front-end companies. It's very difficult to make the models better in these situations because you don't have these two things. It's hard to build like verifiable rewards for reinforcement learning without having a lot more color on what's actually doing, what's actually going on, like on the ground in cyber, and without having people who can assess these things. And so I think the key to really accelerating progress in this in this space is solving both of those two things. And I think broadly, the approach we are taking to solve this is actually having this whole cybersecurity team doing these investigations all the time. So we've just hired folks from CrowdStrike, from Palo Alto Networks, from some of the biggest cybersecurity companies,

• 47:10Speaker 02

  whose day job is really just to do these investigations with the AI tooling. And as they're doing this, they're implicitly like evaluating the model performance based on the use of the tooling. And then secondly, I think the tricky thing is you you need to really just like be seeing a lot of the different types of cyber attacks that are happening. And off the back of that, building evaluations that are really high fidelity, really realistic, building environments that are really realistic and that you can use to train on. And I think this just doesn't happen by default. And by default, the the model providers, I think, will lag on on these dimensions. None of the big cyber companies actually are either pushing on this either. It's not like a huge, they don't have a huge immediate incentive to do this.

• 47:51Speaker 02

  So unfortunately, if your default is just this stuff lags behind in a way that doesn't apply as much to offensive security, by the way. Because from an offensive perspective, you can just try hack something. You can just do that all the time. You don't you don't need any sensitive data. And so there is this asymmetry. Exactly. So there is this unfortunate situation where you can like, a lot of bottlenecks are improving the defensive side, a lot less apply to the to the offensive side. And I think what the space needs is basically a lot more companies trying to solve this like data verifiability evaluation problem. We're doing one attempt, but I would love to have many more.

• 48:26Nathan Labenz?

  So how do you position yourself in the market to? Because I can sort of see if you become the go to company that everybody knows is going to do the best job with this. Then as we see in, and this is kind of a general phenomenon in the AI space, right? Like the more you, more of the share of the business you win, then you have more access to data and you get kind of a positive flywheel going. So I can see how that dynamic could work once the flywheel starts to turn. But it does strike me that it's a probably a pretty hard market to enter.

• 49:02Nathan Labenz?

  So how are you thinking about entering it as a young company that obviously doesn't have the, you know, the track record, the sort of the old adage of nobody got fired for going with IBM, right? Like, I assume there's probably something similar in cybersecurity where it's like, well, nobody got fired for going with CrowdStrike. How do you overcome the just relative unknown factor and win business so that you can start to accumulate this data and get that flywheel turning?

• 49:31Speaker 02

  Yeah. The, um, the flywheel is, I think, actually especially interesting in cyber and CrowdStrike is an interesting example where about, I don't know, 15 years ago, whenever they, they got started. They actually also started as an incident response service provider initially before they productized a couple of years later. And so for the first two years, they were just doing this kind of services. The reason for that was they could solve this problem of understanding what cyber attackers are doing all the time and collect this sort of data. And secondly, build these relationships with enterprises who they were serving. In cybersecurity, trust is like really important for distribution because it's hard to assess like how all the tools are working. So for most, like, I don't know, if you've in a bit of HR software, you can just tell, does a thing like, does the thing pay my employees on time?

• 50:17Speaker 02

  In cybersecurity, you're sort of taking it on trust. Like, yeah, is this thing stopping attacks? Like, I'm not, am I not attacked or do I just not know about it? So like trust is a bigger thing. And so they figured out that there was this really strong flywheel birth from data and distribution from doing services.

• 50:32Speaker 02

  And this has remained the case since then. I think what's different now is that up and like, up and like quite recently. It's been impossible to get AI-enabled services to work really well and get these much better margins than you would otherwise. And this is what's creating this new opportunity in this space. So we are initially focusing on these email-based attacks, where we have built out this platform that basically takes the time required to respond to these attacks down from, depends on the firm, but on the order of two days to a week to these investigations, to a few hours. So like a very meaningful difference here. And this means that we can just do these things much quicker and much more cheaply.

• 51:17Speaker 02

  And that is proving sufficiently compelling to get a bunch of folks to trust us. And from that initial trust, it'll be much easier to get a lot of different types of cases here. So I think the opportunity is coming, I guess, again, available because of this technological shift, where in this context incumbents are just not adopting as quickly as startups.

• 51:36Nathan Labenz?

  And is that just the purely sociological phenomenon that we see in many places of incentives aren't quite there? Who wants to take a risk? Maybe they're billing by the hour or whatever in the first place?

• 51:51Speaker 02

  The sum of all of that, billing by the hours is part of it.

• 51:55Speaker 02

  Another piece that's more subtle is, on the face of it, it doesn't look necessarily like this massive opportunity. Like, it's a pretty big market, you know, if I recall correctly, it's like a $40 billion market of this sort of services. But from the perspective of, say, CrowdStrike, who is, what I, I don't know, close to a $200 billion company, I think, and the majority of their revenues are coming from their detection products, which are not these sort of digital forensic types of services. It's like relatively small fish. And so it only becomes this very compelling, you know, this comparatively compelling thing to like this other big pie. If you have this view that, yeah, okay, we're actually going to get to the point where we're meaningfully automating all this field,

• 52:39Speaker 02

  and then this will change, actually, not just how like these forensic investigations are working, but how detection is working. And that requires you to really take AGI seriously and to really be like, yeah, okay, this work that at the moment is like, just entirely humans, is in pretty short order going to be automated in a meaningful way and will change how all these other bits of cybersecurity works. And most of the players in the space, they just don't have that belief. In general, they're And I've spoken to some of the folks who I know who work at the intersection of AI and security. The security people generally don't, they're financially skeptical people and they don't really believe in AI in this big way. And so I think part of the opportunity is just like, do you really believe in AGI and have thought through the implications of that? And I think most of the big players are that.

• 53:23Nathan Labenz?

  Yeah, fascinating. Okay. Can you tell a little bit more about the jaggedness of models? And I guess I'm also interested in, of course, we've got, you know, the age old debate of proprietary API models, which I would presume are the best at these tasks off the shelf. But then you've also got, if you're using open weights, then you have the ability, of course, to fine tune or modify however you might like. What's the mix? What's the model mix look like? And how much do you think that this is going to be? Like, how much value is in the harness versus how much is in the training data that you can create? And how much of, I guess there's also another question of like, training data can be monetized or can

• 54:08Nathan Labenz?

  be, you know, commercialized in multiple ways. One is you could sell it back to the model providers. And I think that's, you know, from what I hear, increasingly a very good business. Or you could try to make your own models that outcompete them, you know, so you have that asset. How would you describe the kind of, what's working today? And what is your strategy to not get steamrolled by, you know, just better and better frontier models kind of winning everything?

• 54:38Speaker 02

  Yeah, on the jaggedness, I think there is actually this sort of underrated opportunity here to defensively and intentionally accelerate capabilities in various ways. So like one intuition pump here and the context of alignment for a long time, AI alignment, people have talked about the automated AI researcher being one, one important part of the way we're going to solve alignment.

• 55:02Speaker 02

  There hasn't been really been similar and analogous things in other fields. And I think part of the reason was just with pre-training and this like very strong generalization. It was hard to see how you could accelerate in this intentional way rather than just across the board. That I think with the jaggedness we're seeing is no longer the case. And the implication here, I think, is that actually you can go out and pick various areas that you think it might be important to harden the world, like defensive cybersecurity or bio defense or bits of ASH, for example. And you could very intentionally curate the data sets, the environments, the evals that you need to sort of like pull out the jagged frontier in this specific direction. And this doesn't, I haven't, like, people don't seem to talk about this very much. I think this is a mistake.

• 55:46Speaker 02

  It seems like this really huge opportunity. I mean, for many of the same reasons that AI is going to be so transformative in the first place. Like, if you're a subject matter expert in one of these areas and instead of doing bits of work yourself, you can just put this expertise into the AI systems and then scale it immensely more. That's like an incredible thing you could be doing. And yeah, it strikes me that very few people are doing this. And a few reasons, I think, in some contexts, there's like, like in the context of cybersecurity, for example, people often have this idea that actually it's just inherently dual use. That's, I think, partly true, but also overstated. I guess I saw Sam Momen tweet this the other day.

• 56:27Speaker 02

  There are some areas like pen testing and vulnerability discovery that are in fact, yeah, they're just dual use, like you can patch holes or you can exploit them. But this is not, you know, cyber is a diverse field and this is not true of all areas in cybersecurity. And so digital forensics, for example, I think is very much not like this. Where fundamentally what you're doing is trying to detect where the stuff is broken into a system. You're asking, is there evidence that an attacker is here rather than can I break into this? And so that's just like, it's a defensive application. And I think there are other areas of cybersecurity too that have this feature. And so generally, I think there'd be much, it'd be good if there was much more of this kind of stuff out there. On your other question around like the models we're using, how much is the scaffold, all that, that kind of stuff.

• 57:10Speaker 02

  Generally, the way I sort of think about this, at least from how we as a company should be going about this. First, we should just see what we can get from the learning through another box.

• 57:19Speaker 02

  You know, pick the learning through, test the different models, see how they go. Build minimal scaffolds, see where the performance is. Make sure we have sophisticated evals that are working well here and that we can really tell what's working. And then only then consider doing some more fancy stuff like training our own models or things like this. At the moment, we are at the eval stage of that. And on these simple incidents that we're currently working on. Just models out of the box with some scaffolds is actually getting pretty far. Like, again, we're not, we're not at the, we're not so far down the lines of reliability, but it's actually unclear how much more effort is worth for us to get them good at the specific, like, email-based attacks.

• 58:01Speaker 02

  Going forward, these other kinds of incidents that are way more open-ended along the horizon.

• 58:06Speaker 02

  I'd be very, I'd be, I'd be pretty shocked if the models were as good or anywhere near as good. And I think that's an area where all this differentiated access to data that we use as blueprints for evals and so on. And the reasoning traces of analysts using the platform that I think would become this very differentiated and valuable asset.

• 58:24Nathan Labenz?

  Do you have an intuition at this point as to whether you license that data back to model developers? Or keep it for yourself to try to make your own specialist model?

• 58:37Speaker 02

  Yeah, both are possible. And you could also imagine doing this sort of individually where you're, you have some stuff that you've done in the past that you're then sharing. I think the benefits of sharing, at least from like a mission perspective, are pretty significant. And that you're, if you're getting the foundation models better at these tasks, you then get this immediate insane distribution. We are deployed across huge parts of the world. You're naturally deployed across like AI labs, across governments, some of the most important areas from a security perspective. And also, yeah, they will be big contracts. And relative to other data providers, just the fact we're seeing all this cybersecurity incidents all the time does make it quite differentiated. On the other hand, is it giving away some sort of proprietary thing?

• 59:20Speaker 02

  Possibly, but yeah, I think it's sort of an open question, but there are, there are certainly big benefits to sharing.

• 59:26Speaker 02

  Not the data per se, I would say, because that's, that's, that's delicate, but sharing the sort of the evaluations drawing on that.

• 59:34Nathan Labenz?

  Going back to that question of generalization, I mean, I do think the Obviously, the jagged frontier is a major factor affecting all of us that are trying to use AI to do stuff on a daily basis. And the question of how well RL generalizes seems like a pretty big question for like what the next couple of years are going to look like. What would you say is the case for like your specialists? Are they? I mean, I did the point, I definitely take the point that like it's very different to grip through logs versus try to break in in the first place. But I then would also kind of guess that like the people that you have doing the incident

• 1:00:18Nathan Labenz?

  response would probably be like very good at breaking in, right? Like they would know all the tricks of the how to make a phishing email compelling or whatever, right? Like if it would seem that if you had trained as a human at least on the response, you would be maybe not the most elite on the attack side, but like I would expect significant generalization. So how do you see that playing out at the human level? And do and, you know, is there not reason to expect that a similar kind of generalization could happen for models?

• 1:00:51Speaker 02

  Yeah, so at this human level, actually surprisingly, this is not the case where the the this digital forensic field is actually a just like a pretty distinct thing within security. And the people who do it are a different, you know, very different community to like the offensive white hat hackers, things like this. And so at least at the human level, there's not a ton of generalization, interestingly. Yeah, and I suspect this is this, there is just actually relatively limited overlap in terms of the skill sets here. And I think this would also apply to models. Again, of course, at some point, we're going to self continual learning and the models will just get good across the board very quickly. But until we're in that regime, I think there is this this opportunity.

• 1:01:33Nathan Labenz?

  Okay, cool. That's quite interesting. Do you have other things in mind that you think are like that? We can get out of our domain pretty quick here, but like, is there a biosecurity analogy? Is there something that we could be differentially accelerating in biosecurity hardening that wouldn't lead to the same kind of advances in the offensive skillset? Or is this perhaps something that's like relatively idiosyncratic just to the way the cyber world happens to be structured?

• 1:02:09Speaker 02

  Yeah, I will caveat that I don't know bio very well, but you could imagine, like, I obviously you'd need robotics for this, but just producing a bunch of masks and having those ready to go, like stop buying masks, stop buying vaccines.

• 1:02:23Speaker 02

  Certainly, they seem like there are at least some, like the mask example. I don't know, you know, I don't know to the extent to which the biological research is dual use. And then on, let's say, what other areas? I would love, by the way, for someone to just go through and catalog or a bunch of important domains and try to figure this out.

• 1:02:42Speaker 02

  Yeah, I could imagine the context of AI safety, maybe there are specific things here. I'm not sure. It seems like an important thing for folks to go figure out.

• 1:02:50Nathan Labenz?

  Sorry, I recently did an episode with a couple pioneers in the formal methods space. And

• 1:02:59Nathan Labenz?

  obviously, there's like, you know, multiple. There's a general agreement that cybersecurity could become a big problem in the AI future. And then there's several flavors of like ways that we might respond to this. The formal methods angle, I mean, it doesn't really necessarily address email based hacks, but you could imagine a lot of things could get a lot better if.

• 1:03:25Nathan Labenz?

  The rate of vulnerabilities is just dramatically reduced based on,

• 1:03:31Nathan Labenz?

  you know, and the vision that I understand that they have is using formal methods as a reward signal to train coding models such that it's not just that you're like taking output from today's models and trying to verify them, but you're also like really closing that loop and getting to the point where models should be writing like super humanly secure code by default, you know, in like a super majority of cases. That seems like very plausible that we will get there because it seems like the viability of the formal methods, I mean, that's obviously its core strength, right? So it should that that flywheel should be pretty easy to get turning. How do you think

• 1:04:12Nathan Labenz?

  about like where you want to be in the in the cyber security, like, you know, if it's a defense in depth type of thing, right? Where you can you can harden your outermost defenses and then you can have, you know, better forensics when things do get through in the extreme limit, if the outer defenses are, you know, totally impenetrable, then the the forensics, you know, becomes less valuable or, you know, again, in the extreme limit, like unnecessary. How do you think about that? Like, is that just never going to happen in your mind? Or or what?

• 1:04:47Speaker 02

  Yeah, I think MoH would be great. People should totally do that. But I'd be very surprised if it would be sufficient. A few reasons. One is just for like very high levels of of defense against, say, like nationals, you know, nation states. It's just extremely difficult to defend. And it's not just like having no code vulnerabilities is sufficient. There are other ways you can get in. And the strong assumption should be stuff is going to get in. Additionally, the like regarding these other forms of of ways of getting in generally to be like very, very secure, you need to do these things that trade off with your productivity as an organization. So say, I don't know, say you're in, you're in AI lab and you have this like way of interacting

• 1:05:30Speaker 02

  with the model weights that restricts the output. That's kind of annoying, but it does protect against like weight exfiltration and things like this. And so in general, you do have this, this trade off with productivity that is like hard to get around in many areas of security. And so that's, that's the thing that's generally going to stay there. A nice feature of friends, automated forensics or detection is that it doesn't really have this feature. You can just sort of run it in the background and it does increase your defensiveness. And so generally, I think of these things as substitutes. It would be great if we could get this full, you know, in theory, full hardening. I don't think that's at all realistic. And so in general, they're just like the more of each, you can get the better.

• 1:06:11Nathan Labenz?

  So wait, substitutes or complements?

• 1:06:15Speaker 02

  I guess maybe both. Well, they substitute in the sense that if you have more better detection, you need less hardening for like a given level of security. But also having, yeah, having both together makes you most good.

• 1:06:26Nathan Labenz?

  Is there, um, I've not actually done an episode with these guys yet, but there's a company called Jericho Security. I just talked to them offline a while back, but I have their website up. National, uh, next generational security training trusted by the US government. Smarter cybersecurity training starts here. Jericho Security is designed to protect you against today's most advanced email, SMS, and deep fake threats. And basically what they do is automate the phishing attacks, more or less, the spear phishing attacks, to see like who in your organization is clicking on these links and make sure everybody's on their toes. And I don't know that there's, to my not, when I spoke to them, there wasn't like a, you know, a deep forensics. They, you, when you

• 1:07:08Nathan Labenz?

  click on the link, they, they take you to a sort of, we got you page. That's like, hey, wise up dude, like you just clicked on a bad link. And here's how you should have known, you know, and how you can, can know next time. But I could imagine that, you know, in terms of like, environment, uh, creation or something like that, there could be some interesting collaborations between a company that is systematically testing the, uh, for finding the soft spots in the, in the human, um, Defenses or lack thereof and what you guys are doing under the hood. Are there other, how do you go to market? I mean, do you have like partnerships or people that bring you in?

• 1:07:50Nathan Labenz?

  Are there other like alliances that you have that that allow you to get into the room? Because obviously people need to, this is like crisis stuff, right? So they need to like either find you real quick when they're searching or they kind of need to have known you or they need somebody to give a trusted referral. What does that look like?

• 1:08:11Speaker 02

  Yeah, the God of Buck is actually kind of interesting. It's via, for the most part, insurance companies, actually. Cybersecurity insurers. So they will insure a bunch of folks who get hacked. And when they get hacked, they have a pre-approved panel of incident response vendors who they will call in to deal with the hacks. And so actually, a lot of the the motion is going and meeting the insurance carriers and talking to them and getting, building trust there. And then off the back of that, you end up building the specific relationship with the enterprise. But yeah, a lot of the distribution is through the insurance carriers. It's quite unusual.

• 1:08:44Nathan Labenz?

  The AI underwriting, you may probably know the guys from the AI underwriting company then, I suppose. Yeah, okay. Interesting, insurance strikes again. Okay, well, let's zoom out a little bit here in closing and talk about like, what's the big picture look like? You've got this kind of, you know, initial service that's kind of your wedge into enterprises. How do you see that line of service expanding? You know, kind of paint a picture of like, as this matures and you get really good at it, you know, what can we enjoy in terms of additional security? Does it impact my, you know, day to day life at all? Or is it just something that kind of runs in the background and is sort of a, you know, guardian cybersecurity angel watching over me?

• 1:09:30Nathan Labenz?

  And what's the time, like, can I invest before you sell to CrowdStrike? When they finally wake up and realize AGI is a thing?

• 1:09:38Speaker 02

  Yeah, the way I see this broadly, imagine we have just AI systems capable of fully automated digital forensics and much sooner than we otherwise would because we've done all this this work of bootstrapping with humans and building these evaluations and so on. And the implication, I think, is that this completely changes how defensive cybersecurity operations work. Instead of you missing this huge number of breaches because there's no one investigating deeply, you actually just have this proactively on in systems and so you're catching a lot more than you would otherwise be. And this, the implication here is just that there's a huge reduction in the number of breaches that go under detected. And then hopefully this becomes, this is just deployed across all the most important parts of the world.

• 1:10:20Speaker 02

  So it's deployed across AI labs, it's deployed across the AGI spy chain, it's deployed in Western governments safeguarding things like autonomous weapon systems. And when you look back at the overall effects of AI as it diffuses and you think it happens in cybersecurity, you say, actually, no, look, it seemed like this was overwhelmingly beneficial for defenders. That is, I think, possible. It requires a lot of hard work, but it's pretty clear what you would do to get that.

• 1:10:45Nathan Labenz?

  Anything else we haven't touched on that you want to make sure people are thinking about as they try to absorb your extremely AGI-filled worldview?

• 1:10:55Speaker 02

  I think the big, the big thing for me is just, I think we are one, one important attempt of doing this, like pulling out the JAGA frontier in a way that takes seriously what the world will look like when you get these really powerful AI capabilities. But I see very little of that. And I don't see any reason why that's the case. And generally, I don't think it requires any, you know, huge leaps of imagination or things like that. It seems just very tractable, very important to do. And I would love for there to be many more such projects across cyber defense, across cybersecurity, across the whole space there.

• 1:11:30Nathan Labenz?

  Yeah, time is of the essence, it seems. We're due for the automated AI researcher intern edition in just a

• 1:11:40Nathan Labenz?

  Coming soon.

• 1:11:41Nathan Labenz?

  Yeah. Yeah, wild times.

• 1:11:44Nathan Labenz?

  Cool.

• 1:11:45Nathan Labenz?

  We'll definitely continue to follow this with interest and maybe with your motivation, I'll even get around to changing some of those passwords.

• 1:11:55Nathan Labenz?

  Sounds great. Alexis Karlyayev, founder at Asymmetric Security. Thank you for being part of the cognitive revolution.

• 1:12:02Nathan Labenz?

  Thanks, Nathan.

• 1:12:04Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G.

• 1:12:48Nathan Labenz?

  And thank you to everyone who listens for being part of the cognitive revolution.