Aaron Levie, Steven Sinofsky & Martin Casado: How Do You Secure a World of AI Agents?

Erik Torenberg sits down with Box CEO Aaron Levie, and a16z’s Martin Casado, and Steven Sinofsky to debate how the AI industry should think about safety, security, and regulation as increasingly capable agents move into the real world. They argue that much of today’s conversation is happening before we have clearly defined the risks we’re trying to regulate. Drawing on earlier waves of computing, from computer viruses and the early internet to aviation and automobiles, they ask what AI can learn from industries that developed safety standards only after understanding how their technologies actually failed. The conversation then gets concrete: agents don’t get tired, can operate at enormous scale, and can probe systems in ways human employees never could. That could require rethinking permissions, authentication, operating systems, and the security stack itself. They also discuss why AI innovation may increasingly move beyond the frontier labs and into the software built around the models.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Aaron Levie?

  If you regulate AI too early, you actually don't solve anything. You still just kind of had the same risk ultimately.

• 0:06Martin Casado?

  You will the thing into being, but you haven't figured out how to control it.

• 0:09Steven Sinofsky?

  The problem we have now is this rift between the labs and the security community that keeps coming to two conclusions. Sloppy and you're not complete in what you're telling us happened.

• 0:20Martin Casado?

  An employee is like 10% chance of species extinction. The post is very reasonable, but the atmospherics are not.

• 0:27Aaron Levie?

  Agents worms completely flip that. These are just roaming drones. But like 5,000,000 and they will easily mistake a good task for a bad one.

• 0:36Steven Sinofsky?

  So now we need a whole layer internally that just is tracking way more about what authentications are being done, what APIs are being done. The US about 15 years ago stopped leading in tech antitrust. The problem is that Europe is going to lead with that because they have nothing to lose.

• 0:52Aaron Levie?

  This could change the nature of software fundamentally.

• 0:54Steven Sinofsky?

  The center of innovation has just moved. This is the signal that.

• 0:58Erik Torenberg?

  The early internet was riddled with viruses, worms and security failures. We didn't stop building it. We learned how to make it safer. What should AI take from that history? In this episode, I sit down with Aaron Levie, Steven Sinofsky and Martine Casado to debate AI safety, regulation, and what changes when agents start operating across the software we use every day. We get into why agents could force a rethink of permissions in cybersecurity, what decades of software security can teach today's AI labs, and why regulating a technology before we understand how it actually fails can create problems of its own. And we look at a broader shift already underway. As models mature, some of the most important AI innovation

• 1:41Erik Torenberg?

  may increasingly happen outside the frontier labs in the systems and software built around them.

• 1:49Erik Torenberg?

  Guys, welcome back to the podcast.

• 1:51Aaron Levie?

  Thank you. Didn't think we'd ever do this again. I can't believe this is great. I mean, Martine's just building these $100 billion companies. Too busy for this podcast.

• 2:01Martin Casado?

  Well, at least taking credit for it. SBCs do.

• 2:03Erik Torenberg?

  Exactly. We've a lot to discuss today, but Aaron, why don't we start with you? Pacing the frontier. How have you reacted and reflected on what's happened there? And just the discourse that's followed.

• 2:12Aaron Levie?

  Oh, boy. I think we should start with Martine on this one. You were fighting lots of good ground wars. Maybe I'll say one thing that we probably all agree with, and then we can figure out where we maybe kind of fracture off. I think we would agree that any AI lab right now at the frontier should be building in the safest way possible with the highest degree of governance and security and whatever your definition of alignment is. This is incredibly important area of research. It's incredibly important area for the diffusion of AI. You're not going to have AI diffusion without extremely high quality products that can be trusted by enterprises and that aren't kind of constantly hacking systems. So when at least I read the Dario post, I actually didn't disagree with almost anything because it was all about how do you have better

• 2:56Aaron Levie?

  security of these systems, sandboxing, better testing. There's going to be some debates around the embedded nature of the testers. And do you agree with who those are? And does the industry all align on that? But I think actually all of the major points were probably salient and appropriate. Then the only question is, does this get sort of user leveraged to do things that maybe we don't agree with, which would be like a slowdown of AI dramatically because of regulatory controls that would sort of not make it easy to compete with the frontier labs? Or do politicians kind of end up sort of taking the message and run with it? And maybe even worse outcomes happen. It's used to ban data centers far faster and whatnot. And so, so I think the actual substance of the topic is actually incredibly important and I think very important for AI advancement in general.

• 3:40Aaron Levie?

  And then the question is, what do you do about it? Especially what do you do about it from a regulatory standpoint? And that's probably where the industry is going to land on very different points in the continuum. But Martine was putting up a good fight on, let's make sure that we don't use this for regulatory kind of capture, which I also agree with. But I think the ideas in the pacing conversation are important. Again, it's a little bit of a funny concept because maybe it's not even pacing as much as just good hygiene and engineering. And so with good engineering, obviously there is a slight slowdown, but it's a slowdown that obviously allows acceleration of your diffusion because you wouldn't be able to have any of the AI be diffused if nobody would trust using it. So.

• 4:16Martin Casado?

  The post is very reasonable, but the atmospherics are not, right? So an employee is like, this is going to kill whatever, 10% chance of species extinction. And you know what Dario says? I agree with him more than I disagree with him, right?

• 4:32Steven Sinofsky?

  On TV.

• 4:33Martin Casado?

  On TV, the same day that he landed these things. And so in some way, you can't have these conversations in isolation, which is of course, if he's going to agree in species extinction, this post that he has looks like this milquetoast capitulation that's totally not adequate for the task at hand. And so I think the atmospherics are totally broken. And a lot of my comments were on the atmospherics. And then I have this quibble, but it really bothers me because I'm a pedant, which is, I think pacing is the wrong way to describe this. For one, it is orthogonal to security, right? So you can very slowly build a nuclear weapon. And that doesn't make anybody feel better than a slow versus fast. So that's one.

• 5:15Martin Casado?

  The second one, it just feels like a capitulation to the pause folks without actually addressing it. So you're saying, well, we're not going to pause. We're going to pace to make them happy, but we'll also somehow make the regulators happy. And I think it makes them both unhappy. Because the pause people are like, well, that's not a pause. That's just pacing. And then the regulators are, you're still doing the thing.

• 5:36Aaron Levie?

  Right.

• 5:37Martin Casado?

  And so I just feel like my sense of what's happening is the labs are actually trying to do the right thing. And I applaud them for that. I think this is a pragmatic proposal and I applaud them for that. I think the messaging is wrong because they're trying to like split the difference between an internal kind of fringe faction, which are doomers, this little pause, and then the regulators on the other side. And the problem is they're making both of them unhappy. And I think what they have to do is they need to come out. They need to address the X risk question directly. They need to say, no, we don't think this stuff we're going to do is going to cause extinction. And then I think this becomes this very sensible.

• 6:10Aaron Levie?

  And what would you do, just to play the other side for one second? What do you do? Do you make room for the one possible Venn diagram, which is the lab researcher that is both simultaneously super scared, but also works on advancing the state of AI? Because they believe that it's so important to get right, that they want to pursue that. And then obviously the language is right now very problematic. But that person does exist. And that is a real kind of person in our industry, which is like, we have to be at the forefront of AI. I'm also very scared of it. And so that's why I'm working on this.

• 6:45Martin Casado?

  So let me address this directly. I used to work for Lawrence Livermore National Labs on a weapons program, nuclear weapons. I know what it's like to work on things that have existential risks.

• 6:52Aaron Levie?

  You were the first pacer.

• 6:55Martin Casado?

  We were part of the first pacer. So if a constituency within the labs that are the most knowledgeable people believe this stuff has existential risk.

• 7:06Aaron Levie?

  Yes.

• 7:06Martin Casado?

  The answer is to nationalize it and actually put controls that we know that work, right? Now, if they don't actually believe that, and in the private conversations I have, the most sensible people don't, it's a small fraction that do, this is an HR problem, right? So to me, an HR problem is a company problem. Like if they are worried that they can't recruit people or they can't retain people, which it seems to me a lot of this is just that concern. It's almost more of this kind of researcher currency. If that's the case, I think that is the wrong reason to cause a national level lockdown on a very promising technology. So listen, I think the post, again, I think the post is actually very sensible. I think there are real concerns around security.

• 7:48Martin Casado?

  We've had many compute epochs that have real security concerns. I don't think you can reconcile discussions on X risk with the proposal that was put out. You just can't reconcile those two things. And that has always been my primary.

• 8:00Aaron Levie?

  Right, right. Okay, unleashed.

• 8:06Steven Sinofsky?

  Okay, the first thing is, is there a schedule that they've published that says when all this stuff, whatever bad stuff is going to happen is going to happen? They haven't. So you can't pace it because nobody knows when it was supposed to finish in the first place.

• 8:19Aaron Levie?

  It also just seems disingenuous.

• 8:20Steven Sinofsky?

  It's complete nonsense.

• 8:21Aaron Levie?

  100%.

• 8:21Steven Sinofsky?

  Like you can't claim, this is like when the press reports on Apple's latest iPhone is late.

• 8:27Aaron Levie?

  From what?

• 8:28Steven Sinofsky?

  Nobody knows exists. I haven't told anybody that.

• 8:31Aaron Levie?

  Yeah, my Apple car was very late.

• 8:33Steven Sinofsky?

  Yeah, like I don't understand. In order for something to be slower, you need to know the rate at which was moving in the first place. So it's all just utter nonsense. And you can't escape that.

• 8:43Martin Casado?

  And then.

• 8:44Aaron Levie?

  By the way, this is another problem that like, again, from my little quibble on PR is.

• 8:49Steven Sinofsky?

  Nobody believes it anyway, right? And so if that's the thing.

• 8:52Aaron Levie?

  Wait, which part?

• 8:53Steven Sinofsky?

  The pacing.

• 8:54Aaron Levie?

  Which part they don't believe?

• 8:56Erik Torenberg?

  That they're going to pace?

• 8:57Steven Sinofsky?

  That they're going to pace.

• 8:57Aaron Levie?

  I see, okay.

• 8:58Steven Sinofsky?

  They've been at a dead run. They've raised more money than ever before. They've grown faster than ever before. There's no indication that they're pacing. So if this is what you're gonna hang your.

• 9:05Aaron Levie?

  I see, I see, yeah. Well, well, the issue obviously is that the model that everybody has internally far exceed what anybody else has access to. So it's really just the pacing of external releases. But again, back to your point, pacing versus what?

• 9:16Steven Sinofsky?

  Right. We don't know if the one that scares everybody also doesn't work for a legal brief. It might actually screw all that stuff up because it's so, who knows? Right. So there's that. And it's just disingenuous to claim that you're paid. Second, why do they have to announce all of this and ask the government to tell them to? Like, that's the part that starts to go, well, this is really spooky. If you are the most afraid of how everything is gonna go because of your product, just stop.

• 9:40Aaron Levie?

  Don't do it.

• 9:40Steven Sinofsky?

  Like, this is, like, I worked at a missile factory in college and we had nuclear missiles and I walked the floor.

• 9:47Aaron Levie?

  What's with you guys and missiles? I don't know. I'm just here with software.

• 9:53Steven Sinofsky?

  You were one of two people in college when we were, which was either you were protesting to keep them off campus.

• 9:58Aaron Levie?

  Or building up.

• 10:00Steven Sinofsky?

  So that was your choice.

• 10:03Steven Sinofsky?

  And like, we had, like, like, nuns from the Catholic Church show up and pour blood all over our missiles.

• 10:10Aaron Levie?

  Totally.

• 10:10Steven Sinofsky?

  You know, and I'm busy just wheeling PCs around on carts saying, here's your PC. And I'm like scared to die, knowing what was going on. And I'm like, it's a nuclear missile. And then you find out that that's what stopped the Cold War. Like, that literally, it was a Pershing missile, right? And that was what did it. And so, but you said something that I think is super interesting, which is pacing is this sort of fuzzy non-word between like going and not going.

• 10:35Aaron Levie?

  Yeah.

• 10:35Steven Sinofsky?

  And the problem is, you're exactly right. Like, there's, no one is gonna be happy with the middle road. And so one of the things that, that people, I think it's almost fun for me to watch as a sport, they think that they're, that all these people saying to the government, do this, don't do this, that they think they're gonna get what they want. And it's a complete 100% misunderstanding of how government works. Which is, you, when you talk to the government, they actually know how these things work. And they know they're just listening to you and they're listening to everyone and no one is gonna get what they want. Because they know in order to do anything in our system, it's a compromise. And so everything that, that, all the inputs can make it into the government.

• 11:19Aaron Levie?

  Yeah.

• 11:19Steven Sinofsky?

  The output never makes everyone happy.

• 11:22Aaron Levie?

  Right.

• 11:22Steven Sinofsky?

  100% of the time. It either doesn't go far enough or it goes way, way too far.

• 11:28Aaron Levie?

  Yeah.

• 11:28Steven Sinofsky?

  And, and so you can't take the view that you're gonna talk to the government and talk your way into the solution you want. So the bottom line is if they're asking for pacing, they're gonna get the wrong velocity.

• 11:39Aaron Levie?

  Well, my favorite thing is like, it was, was it David Sacks was like, I don't know, I think it was David Sacks, but someone from the government said, you're asking us to regulate you? No. Basically, the answer was no.

• 11:53Steven Sinofsky?

  Well, but the thing that.

• 11:54Aaron Levie?

  Probably just Trump, I think.

• 11:56Steven Sinofsky?

  But the thing that they know now, that you just know from experience is, once the wheels start on regulating.

• 12:02Aaron Levie?

  You cancel that one down.

• 12:03Steven Sinofsky?

  And now it's become an election issue for every party and every jurisdiction, up and down the whole government stack.

• 12:08Aaron Levie?

  Yeah.

• 12:09Steven Sinofsky?

  So there is, there's now this, this whole basket goods, basket of regulatory approaches.

• 12:14Aaron Levie?

  Well, the, the next election will 100% be a referendum on, on AI. So like, it has to happen that, that 2028 is like the AI election. And you could basically run on, the problem is, is like, it's not obvious who would run on the pro-AI story because it's gonna be too nebulous to tell that story. So then it's just, it's just basically varying degrees of how much do you regulate it? Or at least try and like avoid the topic?

• 12:40Steven Sinofsky?

  Yeah.

• 12:40Aaron Levie?

  But it is, it is too bad that we've, we, as a country, are in a spot where like the pro-AI case is just like, it sounds too, it's like, it's just like, takes too many words, you know, it's way too nuanced.

• 12:52Steven Sinofsky?

  It's defensive.

• 12:52Aaron Levie?

  Yeah, it's defensive.

• 12:53Steven Sinofsky?

  Defensive, and we, we own none of the vocabulary.

• 12:56Aaron Levie?

  Right.

• 12:57Steven Sinofsky?

  So the whole debate is, is, is pause, it's, it's swarms, it's rogue. Every word has been chosen by the people who don't wanna do AI.

• 13:07Aaron Levie?

  Yeah.

• 13:07Steven Sinofsky?

  And so it means the first thing you have to do is invent new words and to say that their words are wrong, which takes so many words that.

• 13:13Aaron Levie?

  Yeah, so we gotta be like union jobs and, and, you know, cancer and like, you know, there, we need, you know, there needs to be another word cloud that emerges.

• 13:22Steven Sinofsky?

  What I, what I don't, what I don't understand is why the labs have not taken a position on X risk. Like, short of that, I don't think this goes in any direction other than heavy-handed regulation.

• 13:32Aaron Levie?

  Yeah.

• 13:32Steven Sinofsky?

  Well, it would just be negligent of the government to be like, there's a 10% chance of species extinction. The CEO of the top company says he agrees with that. Like, how can a government not do heavy-handed regulation?

• 13:44Aaron Levie?

  Well, but what, who, but what would anybody real, knowing, knowing this, this ecosystem though, I don't know that you would be able to pin anybody down on that other than something.

• 13:53Steven Sinofsky?

  No, I would say Dario said it on.

• 13:54Aaron Levie?

  No, but I'm saying you're not gonna pin anybody down on a lower.

• 13:56Steven Sinofsky?

  Well, in fact, he, he, he does the worst thing about it, which is he agrees with people who claim that they believe that there's an X percentage of extinction happening. But he will, he specifically is out of his way to say, I'm not gonna put a percentage on it. Which I just think is the weirdest.

• 14:12Aaron Levie?

  No, but that, that is, to be, to be fair, to be fair.

• 14:14Steven Sinofsky?

  No.

• 14:15Aaron Levie?

  Okay, no. Nobody can be fair. To be fair, the, that's like, it's not 100% like, uh, like disingenuous or whatever. Like, like, he probably doesn't specifically agree that it's 10%. Or maybe he does and it's just too scary if he were to say it.

• 14:32Steven Sinofsky?

  Well, let's just play binary search.

• 14:33Aaron Levie?

  But is it more or is it less?

• 14:37Aaron Levie?

  But the whole thing isn't made up. You knew it was a made up concept that we just created. But like, nobody can, you can't quantify any of this.

• 14:43Steven Sinofsky?

  Well, that's sort of Martin's point.

• 14:45Aaron Levie?

  Well, but you just said an official position though. So the only official position would be, like, I think the only official position you could possibly get would be the PR version of this that would be, the only thing that would be intellectually honest would be, there are real risks with AI. There's incredibly positive benefits as well. We are working to mitigate the risks so they are as reduced as humanly possible.

• 15:04Steven Sinofsky?

  I don't think that's, no, I don't think that too. Listen, so.

• 15:06Aaron Levie?

  Wait, what do you think the?

• 15:08Steven Sinofsky?

  We've been through multiple epochs of technology.

• 15:10Aaron Levie?

  Yeah.

• 15:10Steven Sinofsky?

  We've been through compute, we've been through the internet, we've been through the web, we've been through social networking. We had a discussion about the risks without talking about X risk, right? So for example, one thing you can say is we do not think the marginal risk for species extinction is different than it is.

• 15:25Aaron Levie?

  But what if they do think it is higher?

• 15:26Steven Sinofsky?

  Well, then we should.

• 15:30Aaron Levie?

  I think the answer is they do think it's more than just the internet.

• 15:33Steven Sinofsky?

  We have one of two options. You believe that we're gonna go extinct and we shut it all fucking down, like shut it down.

• 15:38Aaron Levie?

  They believe that this is a chance that we could.

• 15:41Steven Sinofsky?

  Speaking of.

• 15:43Aaron Levie?

  Dario thinks less.

• 15:47Steven Sinofsky?

  No one, like the Pentagon, you know, they ran a lot of simulations on the chances for nuclear war, great movie War Games about the whole thing and all of that. But the thing was, it was, it was non-zero.

• 15:59Aaron Levie?

  Yeah.

• 15:59Steven Sinofsky?

  And so once you said it was.

• 16:01Aaron Levie?

  Can they say, can they say non-zero then? Is that allowed?

• 16:03Steven Sinofsky?

  I think as soon as you think it's non-zero.

• 16:05Aaron Levie?

  The problem is, is if you say non-zero, that could be like, you know, shit, man. You think I think 93%?

• 16:10Steven Sinofsky?

  Wait, so, so, just so we're all having our think conversation, let's talk about marginal risk.

• 16:13Aaron Levie?

  Yes, yeah.

• 16:14Steven Sinofsky?

  We're not talking about absolute risk.

• 16:16Aaron Levie?

  Right, yes.

• 16:16Steven Sinofsky?

  Right, okay. It's just that I think if, once you say it's non-zero, the only answer of like catastrophic.

• 16:24Aaron Levie?

  For marginal.

• 16:24Steven Sinofsky?

  The only answer is you have to nationalize it.

• 16:27Aaron Levie?

  Yes.

• 16:27Steven Sinofsky?

  And so what he's trying to, what the, the labs in general that have that view are trying to thread is, they want it to be non-zero as a license to do a certain set of things without the burden of just becoming a nationalized.

• 16:41Aaron Levie?

  Well, do you, I actually don't know all of the precedents. You hopefully do, but, but like, there must be some things that are non-zero, uh, let's say X risk, X risks that are not particularly nationalized, but the, but the regulatory environment around them is so heavy that it might as well be nationalized because of the KYC requirements on, like, like, I'm sure, like, to develop anthrax, you have to go to a particular kind of lab.

• 17:05Steven Sinofsky?

  Well, BSL4 labs are, but they're nationalized.

• 17:08Aaron Levie?

  They are nationalized, okay.

• 17:09Steven Sinofsky?

  Anthrax tends to be nationalized.

• 17:11Aaron Levie?

  Yes.

• 17:12Steven Sinofsky?

  Like, normal human safety, not so much, right? Like industry oversight that over time becomes federal regulation.

• 17:19Aaron Levie?

  Right. And the progression, which I think is just super important to this discussion, has been since the post-World War II era, most industries that are critical to the infrastructure, power and banking and healthcare, the trend has been to basically be nationalized.

• 17:38Steven Sinofsky?

  Yeah.

• 17:38Aaron Levie?

  By, but just like what you, like your examples of KYC and all the other stuff, the banks are for all practical purposes nationalized and the financial crisis.

• 17:46Steven Sinofsky?

  Okay, but you said all practical purposes.

• 17:48Aaron Levie?

  Right.

• 17:49Steven Sinofsky?

  They're literally not nationalized.

• 17:50Aaron Levie?

  Right.

• 17:51Steven Sinofsky?

  So, so maybe that, but this might be the intent of the labs is to look like JP Morgan or look like Verizon. And it's just like, we're critical infrastructure, we get heavily regulated, it's not good for open source, or at least frontier open source, but it's a, it is like, it's, it's like a plausible outcome for this industry in the limit.

• 18:08Aaron Levie?

  It's not necessarily good for innovation is the problem.

• 18:10Steven Sinofsky?

  Well, I think even that's fine, just don't use species extinction as.

• 18:13Aaron Levie?

  Sure, as you're, yeah, yeah, yeah.

• 18:14Steven Sinofsky?

  This literally is like the difference between the things that are like stuck in the lab. And let me just say something. I actually think the labs are moving in the right direction.

• 18:22Aaron Levie?

  Yeah.

• 18:22Steven Sinofsky?

  I actually think the actual statement was really good. You know, in my discussions with, you know, executives and leaders, like, they understand that they have like this tension and they're gonna reconcile it. So I actually am quite optimistic that the labs are both doing the right things and trying to do the right thing. I just think that grew so fa, grew so fast and just trying to figure out how this machinery works. And I think Sinofsky really hit the nail on the head.

• 18:46Aaron Levie?

  Me too.

• 18:46Steven Sinofsky?

  The political process is its own, is its own thing. And I don't think, I don't know if it's naivety or hubris, but I just don't think that they kind of know how to navigate it.

• 18:53Aaron Levie?

  Well, I think the tech industry.

• 18:56Steven Sinofsky?

  Has literally over a hundred years consistently relearned the lesson at each technology wave that that we don't understand the regulatory climate and can't navigate it. And even the companies like AT&T and IBM that were born out of basically being government monopolies from the start, never figured it out. Both got sued for antitrust, both got substantially and structurally changed as a result. And they had hundreds of lawyers in the 1960s navigating them. And that, you know, Microsoft came along, like, we were just like, what?

• 19:27Aaron Levie?

  Yeah.

• 19:27Steven Sinofsky?

  Like, we had no idea what was happening to us. And there's, you know, there's Bill Gates playing golf with Bill Clinton, and then we get slap in anti-trust laws from his administration. And Bill was like, I was playing golf, here's the picture. And that doesn't help. And like, isn't that what I'm supposed to do, was go and play golf? That's a shortened version of those things. But I think, and I think it's just, it's, I always use this example, which is the Hollywood got together during the, the red scare and all, and censorship, when they were worried about the government censoring movies for, for, for sexual content, for adult themes. And they all got together and, and of course, they were never able to win in court if they tried to, but they were threatening to do it. And they got together and they formed the Motion Picture

• 20:09Aaron Levie?

  Right.

• 20:09Steven Sinofsky?

  Association.

• 20:10Aaron Levie?

  Yeah.

• 20:11Steven Sinofsky?

  And movie ratings and all that, and they police themselves.

• 20:14Aaron Levie?

  So how do you, do you like, do you like the FINRA proposal?

• 20:18Steven Sinofsky?

  No, because FINRA is-

• 20:19Aaron Levie?

  Well, no, that's, that's as close to MPAA as you can get with more teeth.

• 20:21Steven Sinofsky?

  No, it's not, because FINRA is gonna, FINRA becomes legislation, which becomes with direct oversight.

• 20:27Aaron Levie?

  Yeah, I think MPAA might not have as much consequence in like, how society functions.

• 20:32Martin Casado?

  Well, it was only the First Amendment.

• 20:33Aaron Levie?

  Yeah.

• 20:38Steven Sinofsky?

  Did I win that?

• 20:39Aaron Levie?

  No, I, I, I, first of all, fantastic. But, but I still don't know if you won it, like, like, I agree, free speech is really important, but like, but like, you know, like, the, I just think-

• 20:51Steven Sinofsky?

  There was no societal risk.

• 20:53Aaron Levie?

  I just think what's in our movies will be like, we'll survive. Like, on like, a different continuum of-

• 20:57Steven Sinofsky?

  Always, every, history is always relative. And, and at the time, being a communist was a really bad thing, and 30% of Hollywood got fired for, for, you know, it was all this stuff.

• 21:08Aaron Levie?

  Yeah.

• 21:09Steven Sinofsky?

  So I, it's always a risk to bring up something in that kind of way, because it sounds so dumb, like, nobody thinks about movie ratings. And that's because, like, actually, they were ruled, basically, you can't constitutionally mandate them. So they couldn't regulate them on cable TV. And so we got to grow up with HBO and all this other stuff.

• 21:24Aaron Levie?

  Yeah.

• 21:24Steven Sinofsky?

  But the problem with FINRA is, that is a perfect example of essentially nationalizing risk.

• 21:30Aaron Levie?

  Yes.

• 21:31Steven Sinofsky?

  Because even though the banks all pay money into it, and that's how it's run and stuff, it's all mandated.

• 21:36Aaron Levie?

  Okay, wait, so you think we're gonna end up with a situation that is better than FINRA? FINRA appears to be the best case scenario, but like-

• 21:43Steven Sinofsky?

  It is the best case scenario.

• 21:44Aaron Levie?

  But not even anymore. Like, like, I do think this, this technology is, in the limit, again, so powerful, relative to what it, I mean, for what it can deliver, that it would be impossible for eventually Congress not caring about that. Like, it's just like, not possible.

• 21:59Martin Casado?

  My question to you guys is, at what point?

• 22:02Aaron Levie?

  If it's eventually making every recommendation in your healthcare process, and, and it's inside of your medical device as an open weights model, and it's, and it's all, and every trading system for high-frequency trading, there's just, and it's on an airplane, like, there's no chance that the government doesn't say, we need something, where FINRA actually is like, the, probably the best case scenario of what that looks like.

• 22:22Steven Sinofsky?

  Right.

• 22:22Aaron Levie?

  Yeah.

• 22:23Steven Sinofsky?

  And so now, that's, that's absolutely the most crucial point. Because if you, all the people in the Senate now, were in the Senate when, when the Communications Act was passed, and the, and liability was not passed through to ISPs and to social networks. And they sat around, the debate at the time was, they would literally say to us, the internet is so big, how did we not have anything to do with it? And that's why Al Gore gets a bunch of abuse for saying he created it.

• 22:49Aaron Levie?

  Right.

• 22:49Steven Sinofsky?

  Because he, he was actually trying to get out in front of that and say, no, the government was instrumental, and it all backfired because it made it look like he was a crazy person. But, but that, it is absolutely the case, that they felt like they did, they missed their chance to be on top of the internet.

• 23:05Aaron Levie?

  With Al Gore being on the positive side to innovation.

• 23:08Steven Sinofsky?

  That was their whole-

• 23:09Aaron Levie?

  Yeah. So who is the Al Gore of?

• 23:11Steven Sinofsky?

  There's no one. There was a couple, well, that's, it seems to be like that.

• 23:15Aaron Levie?

  Right, right.

• 23:15Steven Sinofsky?

  That, you know, the defense people sort of wanted private but not, which is exactly where they were on the internet.

• 23:20Aaron Levie?

  Right, right.

• 23:20Steven Sinofsky?

  And so it's very interesting that a lot of this is just this, like, Elizabeth Warren's, Senator Warren's tweets were all like, we missed this for social networks. And it's like-

• 23:30Aaron Levie?

  There's a lot of pent-up energy against tech that, that could end up just all siphoning into, into AI regulation.

• 23:37Steven Sinofsky?

  That is exactly what it is. That's what's happening.

• 23:38Aaron Levie?

  Yeah.

• 23:39Martin Casado?

  I think there's another problem, which is, we're all trying to predict what's bad, which is not how we've normally done things. Like, by this time on the internet, we'd taken out, like, tens of billions of dollars of, like, economic, well, we've caused tens of billions of dollars of economic damage, we've, we'd had words that took out 10% of the infrastructure, the, the internet infrastructure, which was running critical infrastructure, we had hospitals go down.

• 24:00Aaron Levie?

  We really should have blocked the internet in like, 97. How did we, yeah, okay, okay, okay, okay.

• 24:07Martin Casado?

  We, we had, I mean, I, I remember at a time, like, I remember when you would, you would, you would buy your CD of Windows 95, and by the time it was done installing, you would have a worm potentially.

• 24:21Aaron Levie?

  Way to go, Stephen.

• 24:22Steven Sinofsky?

  No, no, this was, this was the reality.

• 24:25Aaron Levie?

  Oh, totally.

• 24:25Steven Sinofsky?

  The reality of the PC until 2001 was, you could not install a PC connected to the network without getting infected.

• 24:33Aaron Levie?

  Oh, viruses were everywhere.

• 24:34Steven Sinofsky?

  For the time, and, and there were, there were two, two level, two, two rounds of congressional hearings, you had all the-

• 24:40Aaron Levie?

  That is actually, okay, okay, I agree, that's a very interesting point. This type of zeitgeist in 94 would have, we would probably not have the internet.

• 24:49Martin Casado?

  Listen, listen, I mean, the automotive, the airline industry, you always have these periods of kind of like, teeth cutting, where like, you learn about the dangers, and you learn about the technology and the internet. I mean, we were on the ground floor of this. I mean, things were down all the time, there's economic damage all the time. Like, there was like, novel, really, there was new viruses, there were new worms, they're all over the place.

• 25:07Aaron Levie?

  But Y2A, Y2K was gonna destroy the world.

• 25:09Steven Sinofsky?

  Here's the interesting thing about this.

• 25:10Martin Casado?

  But, well, it's, I just want to say, like, like, so when we created policy, and we did create a lot of policy, it was kind of like, with a bunch of very specific data points on what you're trying to do. And so you know that the policy actually fits the, the fact pattern. And in this case, I mean, even when you were talking, you're like, well, what if, you know, yada, yada, yada. And it's very hard to predict a policy when we don't know. Now, one area we can talk about is, there seems to be novel cybersecurity risk. Great. And what is nice about the discourse, and it's actually starting to evolve around that, like, you actually hear people from the lab saying, novel cybersecurity risk, this is an engineering problem, we're talking about that. So the more this becomes concrete around real identified risks, I think we can all fall in line. But that's not been the discussion to date. That's a very recent thing.

• 25:53Steven Sinofsky?

  Perfect, perfect, perfect point. Because if you look in, in, in 1986, the Computer Crime and Fraud Act got signed. It was out of a very, very specific scenario, which was, two groups of hackers broke into GTE telemail.

• 26:07Aaron Levie?

  Hmm. One of them lived in my dorm, go figure.

• 26:10Steven Sinofsky?

  And, sure.

• 26:11Aaron Levie?

  No, it wasn't you.

• 26:15Steven Sinofsky?

  He worked at Cisco later. And, and, and the problem was, that was 1983, and there was no crime. And it took two and a half years for the law to make it through. That made it very, very specific. And that's the law that says you can't, you can't access unauthorized computer systems.

• 26:34Aaron Levie?

  So I think we'd all agree that, that you, we probably, you know, we probably have like, 90% of laws already in the applied layer, like, you can't hack systems, et cetera. Do you think there's anything that should be, from a liability standpoint, in the model layer? Which of course then-

• 26:47Steven Sinofsky?

  Wait, and now you're making a technical legal talk.

• 26:49Aaron Levie?

  Legal.

• 26:50Steven Sinofsky?

  No, the, the legal, everything, my read of, of hugging face, OpenAI-

• 26:54Aaron Levie?

  Yeah.

• 26:55Steven Sinofsky?

  For, they're all absolutely blatant computer crime acts, except for the fact that there's, they carved out in amendments later, that if you're a white hat, it's not illegal anymore. And that was because people kept making mistakes. And, and they didn't want to go around arresting everybody who was actually trying to make the system better, because they messed something up. And so the Justice Department wrote a memo that said, we're not gonna prosecute for, for this. And we're also not gonna prosecute if you just, like, violate the terms of use of a system versus actually try to breach it.

• 27:26Aaron Levie?

  Uh huh.

• 27:27Steven Sinofsky?

  And, and so, I think the law is ample for this, for the scenario. And, and it was all written, this GT telemail was used by NASA and Livermore and all the labs. And so that's why it was, it caught the attention of DC, because it was federal systems that were being broken into. And the problem we have now is, is this, is this rift between the labs and the security community that keeps looking at all their postmortems and coming to two conclusions. Sloppy.

• 27:53Steven Sinofsky?

  And and you're not complete in what you're telling us happened. And so, of course, the CBE process came about in the 1980s, the computer virus and vulnerability reporting stuff out of CMU. And and for years, they worked on very structured reporting with obligations and how to and for some reason, they're not using any of that to do this reporting. And so there is this very basic stuff that I look at and say, well, until they're doing that, they really should stop talking. Like, they shouldn't do a postmortem on a breach that looks like an intern wrote it, and it's not a postmortem. It's this selective memory. It looks like exactly the kind of postmortem you do when you hire outside lawyers to investigate some random thing, and you only give them certain stuff.

• 28:37Steven Sinofsky?

  Because you don't have to give the lawyers you hire all the information about what happened. Like, where's all the Slack messages? Where's the actual details of what happened?

• 28:46Martin Casado?

  Can I can I just interject on an annoying aside? So, um, which is, which is, which is, no, no, this is good, which is very in line with this, which is, um, I've been in the security, like, the actual cybersecurity committee for a long time. And it is, you know, it's always been one of these things where, like, they just don't like practical solutions. So even if you build, like, you know, a secure system, like, well, what if somebody, you know, shows up?

• 29:06Steven Sinofsky?

  Oh, yeah, yeah.

• 29:07Martin Casado?

  You know, like, you know, can Russell Crowe can, like, break the encryption or something?

• 29:11Steven Sinofsky?

  It's Mission Impossible.

• 29:14Martin Casado?

  There's always like these kind of like whatever. So my favorite thing that happened recently was like Noam Brown. And some podcast, we've all said stupid stuff on podcasts, I've already said.

• 29:20Steven Sinofsky?

  Oh, you didn't like this one?

• 29:22Martin Casado?

  No, it was great.

• 29:23Steven Sinofsky?

  I thought it was fun.

• 29:24Martin Casado?

  No, it was fantastic.

• 29:26Martin Casado?

  I'm setting it up. So so Noam Brown was like, listen, you know, uh, you don't know what a super intelligence could do. It could maybe use like the heat of a CPU to exfiltrate itself to another computer. Which this brought me back to my, now I'm very comfortable. Great, now I can have endless pointless discussions on this. But what is interesting is you basically have the access people saying something you thought was plausible, and then you have like the security people having this kind of endless discussion. And so I think at some level, now these communities are being bridged, which is like, you know, I actually think that was a very reasonable thing from Noam Brown to say. I think you can pick holes in it, but we all say weird stuff on podcasts. I actually think exfil risk is real. I mean, I worked in secure computing environments that were highly classified, that you would,

• 30:11Martin Casado?

  the covert channels were unbelievable. So these are very real comments, but like, we actually can have a real discourse. And it was the first time I saw really hardcore systems people having pretty, like, I would say constructive.

• 30:23Steven Sinofsky?

  Well, they were calculating the bitrate and all this.

• 30:25Martin Casado?

  But it was a constructive discussion. And like, you know, you had like the kind of the typical access people were engaging. Of course, it was it was Twitter, so there's like a lot of name calling this and that. But it was actually, I felt like a real discussion for the first time. So I hope we see more of this. I hope we see more, you know, you know, cyber related things. I think the labs should talk more about it. I think it'll engage the community. And I think once that happens, we can actually.

• 30:47Aaron Levie?

  Yeah, Greg's been out there a lot more on this topic, which has been, which has been good. But I think the takeaway is Noam Brown should be doing more brainstorm on podcasts.

• 30:56Steven Sinofsky?

  It opened people's eyes to the fact that actually there's a lot of risks in security that most people don't understand. And so that means that there's more stuff that, like, you can't make baseline risk, you know, like, how is your authentication layer work? You can't just wave your hand and say that should go away and then bring up, oh, but, you know, space aliens can invade. And that's a little bit of what was going on that I felt uncomfortable with. But like, what do you mean? Like, it was sort of like, but you know, there's also this risk. And that's how I view that.

• 31:25Martin Casado?

  Of the heat thing, but but at least we're now, we're now in a, in a domain where comfortable. Like, I can talk about entropy and like, we could actually have a concrete discussion. That's not, oh, well, it's super powerful. At least we reduce it to like the laws of physics and the laws of.

• 31:38Steven Sinofsky?

  And most people did, I would say, most people had no idea that that kind of risk was real. I mean, like, when I'm walking around the Pershing missiles, I actually had to test the graphics cards in PCs because a DoD requirement is that the screen memory not be sustained when you pull the power. But not for for zero time. Like, the minute that the power went off, the memory image had to go. And if there was like a three second delay.

• 32:00Martin Casado?

  Yeah.

• 32:01Steven Sinofsky?

  You you you had, that could be read.

• 32:03Martin Casado?

  Oh, see, see, we had people that came into our offices and would literally measure the distance of the monitors to each other because of Tempest attacks, which is 100% away to use electromagnetic radiation to leak information. I've seen the same thing with spread spectrum from the BIOS. I've seen it from, I've seen it from from audio speakers. Like, we had to remove the speakers out because it's a very high.

• 32:22Steven Sinofsky?

  Our building, our building.

• 32:23Martin Casado?

  These are very real things.

• 32:24Steven Sinofsky?

  Our building had had just muzak speakers aimed at the windows.

• 32:28Martin Casado?

  Yeah.

• 32:28Steven Sinofsky?

  Just to produce interference.

• 32:29Aaron Levie?

  You need to go run safety at one of these labs. This is like, I've never seen you at more excited than heat, you know, based communication.

• 32:37Martin Casado?

  Can I tell you the craziest covert channel I've ever seen? So it turns, remember, remember the old CRT? I know, this is like one of my.

• 32:42Steven Sinofsky?

  So.

• 32:42Aaron Levie?

  I'm glad that someone's older than me, not really, but acting it.

• 32:45Martin Casado?

  Remember the old CRT? So it turns out if, like, let's say it's night and you're you're using a CRT terminal in your room, the lightest thing in the room is actually the pixel that the raster beam is on. So most people think it's like the glow of the monitor, but it's actually that given pixel. So somebody figured out that, like, let's say you're in, like, you know, a hotel room and you're on your computer. If you have something that can sample the color of the window, you can reconstruct the screen.

• 33:09Aaron Levie?

  Oh, that's crazy.

• 33:10Martin Casado?

  Right, you just do it the same hertz that the raster beam is moving. And then somebody else figured out if you can subvert three pixels, you can use those, you know, because it just looks like bad pixels, you can use those to basically send a message. So like, you could literally like sit out in whatever, like.

• 33:23Aaron Levie?

  Do your own SOS.

• 33:24Martin Casado?

  You sample the message and you can, it is a relatively high bandwidth one way communication. And so, like, what Noam Brown was saying, like, maybe heat is not the way to do it, but that level of sophistication is actually real. That's a, that's a thing.

• 33:36Steven Sinofsky?

  My, I, when I was at the missile factory, like, I had to lock my keyboard up. I, that's really an impolite word, but that's what we call it. I had to lock my keyboard up at night because they didn't want the custodians who didn't have clearance walking by and and just noticing which keys were dirtier or cleaner. And I once left it out and there's like a note from security, you know, telling me to report to security and pick up my keyboard. Like, the guard who walked the hallways just took the keyboard that night off my machine. And that's like baseline, I was not cleared, I was just had sensitive, you know, I was like, nothing, I was an intern.

• 34:08Aaron Levie?

  The big problem with this conversation is now this is all in the training data of every AI model in the future, so.

• 34:14Martin Casado?

  But that's also, but that's also the threat. Well, this is the thing. But that's, the threat, the threat model for these things is always you've got a trusted side and an untrusted side.

• 34:22Steven Sinofsky?

  NIST has, NIST has 500 page manuals.

• 34:25Aaron Levie?

  This is already out there.

• 34:26Martin Casado?

  And untrusted side, you assume basically an oracle that can do and know everything.

• 34:30Steven Sinofsky?

  Yeah.

• 34:30Martin Casado?

  And then the question is, can you get information off the trusted side? Which, by the way, is what Noam was saying, which, again, is this actually.

• 34:35Steven Sinofsky?

  Which I do think brings up a super interesting point, which I'm going to bridge, which is just that I think the thing that people really aren't wrapping their heads around and are using the language that's really confusing is just that what AI can do is it can try all of those things in a very short time.

• 34:50Martin Casado?

  Yeah.

• 34:51Steven Sinofsky?

  Yep. And and it doesn't get tired, it doesn't get bored. And, you know, and so, but the interesting thing about it is there's a whole layer of security that you now have to go look at every single API, every single service you're running internally in your network as like, you know, like, nobody thinks that their internal GitHub or their internal Slack or internal finance expense tool is is vulnerable to a denial of service attack. But swarms.

• 35:14Aaron Levie?

  Will do it.

• 35:14Steven Sinofsky?

  Look, literally look like a denial of service attack.

• 35:17Martin Casado?

  Yeah.

• 35:17Steven Sinofsky?

  And so now we need a whole layer internally that just is tracking way more about what authentications are being done, what APIs are being done. And and but that's just like, now it's just going to be basic. And and all the offsite people that are old are like mailing me like, why are we explaining to this to everyone? It's so basic because nobody did it internally.

• 35:37Aaron Levie?

  Yeah. And. Well, you didn't have to worry about it for your people. And that's the, like.

• 35:41Steven Sinofsky?

  But now your person is just, he's a software.

• 35:43Aaron Levie?

  Yeah, right, and like unlimited and and the.

• 35:45Martin Casado?

  Has a credit card.

• 35:46Aaron Levie?

  Yeah, I mean, we, you kind of got by with with information security, like, to some extent, on the fact that most people will do the right thing 95 to 99% of the time.

• 35:58Steven Sinofsky?

  Wait, is not, the malicious employee is like one in 10,000.

• 36:01Aaron Levie?

  Yeah, yeah, yeah. So so like, so all these systems are basically open to whoever wants to access them or like one tap on the shoulder and then you have access. And agent swarms completely flip that because they will, these are just roaming, you know, drones. Yes, and and but like, yeah, times 10,000 and and they will easily mistake, like, a good task for a bad one and vice versa. So so the the data security in our systems is, this is going to be a huge upgrade moment.

• 36:30Steven Sinofsky?

  I actually, Marta, like, I think that actually we're going to need a different access and security model going forward. Where are we going to go on that? Because the model we have is not granular enough and it's not performance enough to handle this stuff.

• 36:44Martin Casado?

  So I'll just step back, I'll say like a meta, like a meta point, which is, I think this is how these conversations should go. We've identified a novel risk, which is like cybersecurity, which we actually have proof points, and now we're talking about solutions. I think the entire discourse around AI can be of that form. And that's, the biggest mistake is that's not what it's been. Like.

• 37:01Steven Sinofsky?

  I think the entire industry and community is very happy to engage exactly like this. Yes. And I have something to say about exactly what you're asking, but I'm saying like, like, I think this is where the conversation should be. Yeah. So. We're, we're known for having healthy conversations that everyone should learn from, so. So that's what we do. So, so here's the thing, um, I don't think there's a, a technical limitation here. Like these threat models are very well understood and have been in the literature for a long time. I mean, like the operating systems research, the, um, MLS, the multi-layered security research has, has, has considered these sorts of things from an academic lens. The reason it hasn't been adopted is just tended to be a usability issue. I guess it's really hard to maintain and you didn't have to. So you could argue that AI solves the usability issue because it's AI is using it.

• 37:46Steven Sinofsky?

  So maybe now it's going to be a renaissance in operating systems and network and computer languages. And we should like go back to the old research and we should kind of start rebuilding systems that are secure by design. And oh, by the way, if we don't think that, you know, these things are safe to put out, we don't put them out until we have these systems built. And, and, and oh, by the way, the AI is very smart so they can help us build it. And so I think again, like computer always, like, you know, how much of the stack we had to change for the internet? Mm hmm. Everything. Right. Tell me about it. And you know, you know how not vulnerable, how vulnerable everything was like, like we could be in one of those moments like, oh shit, we got to rethink everything. And that's fine. We've done that before, but, but I think that's a conversation we should have. So I agree. It's time to think about evolving these things.

• 38:28Steven Sinofsky?

  Well, like, look at, like, here, here, you mentioned earlier, like the booting a PC and getting a virus in, in 30 seconds or whatever. So if you look at how like you take an iPhone out of the box, which a lot of people are doing this week, you know, the, what happens is it, the whole network is basically shut down, except for getting the latest version of the operating system. Because it doesn't, even though it was pressed, you know, six weeks ago, you know, there's, there's some zero day thing has been discovered since. And so actually the whole out of box process now involves first step updating, doing an update where the, the device can't do anything else. And it can never do anything else until it's updated. That's the, like, there are all these benign, things that are completely benign that we turned off in all of this desktop software of the era

• 39:11Steven Sinofsky?

  that used to be good things, like it was having a macro for Word so you could build automatic citations or something. It was like the super cool thing until it became a virus. We had a thing where you could put a CD in and it would just arbitrarily run a program. And so then what somebody did was like, oh, well, I'm going to burn a clone of that CD and replace the program with my virus. But it's going to look like the thing that's supposed to run and it's just collecting all this stuff and being evil. Yeah. And we disabled that. And so what, one of the things that's happening right now is we, there's a whole bunch of stuff that happens on, like you, on your own box corp network that actually is going to have to just change as the, the standard procedure. Two factor of five years ago was not standard in most places.

• 39:53Steven Sinofsky?

  The, the, your whole, like your whole SaaS world, like I remember in like 2015 or so, when, when you would talk to a new company about their, oh, we're going to do enterprise pricing or whatever. And then they realized the first thing they had to do would go do Okta integration and, or Google off because they could not have their own directory of how to manage it. And then that just became a thing. And now there's not a SaaS program anywhere that doesn't just launch with managed authentication. Right. And, and so there's just so many things that need to happen before you're, you're even software now. Yeah. Well, yeah, we're, uh, I mean, there's probably every layer of the stack has to evolve a, a bit in this.

• 40:32Steven Sinofsky?

  Like even the like lack of granular nature of like, you know, you, you have these modes of like, the agent will either ask you every single time, you know, if, if you want to, you know, give a permission to do something, or the exact opposite of like, it can just like delete your entire computer. Right, right. And like our OS probably wasn't built for the right level of, of granular set of, of tools you want to give the agent. Um, we, we've kind of done a lot of work, work in this space because obviously like, you know, do you want to give an agent exactly like your entire file system? Probably not. Right. Maybe in some cases you do, but oftentimes you want to have granular controls of like, in this folder, you can do read, write. And in that folder, you can only do read. And, and so how do you kind of make this all intuitive for the user? So it's, it's very difficult. And so, so there's going to like.

• 41:17Steven Sinofsky?

  What he just said is a very deep comment. I know, exactly. Yeah. Which is basically the conclusion of 40 years of. Right. No, a hundred percent. It's like you actually can't make it, but maybe with AI you actually can, like, maybe there is like. Well, I, I actually think this is, if you ask me, like, walking in, like, what I wrote down on my note, like, was my biggest fear. Yeah. Is that Europe decides that GDPR was the best thing ever. And they're going to just GDPR AI. Yeah. And, and the AI will be fine. It'll be have one, it'll have one prompt for when it text gets emitted that just says, this vendor is emitting text and it's probably wrong. Yes or no? That, that's going to be, because you can't really, at least in North America, you're not going to send your speech. In Europe, they still will. And, and they'll have filters and keywords and block lists.

• 42:01Steven Sinofsky?

  But then on any verb, anytime that an agent or, or, you know, a background agent or a frontline agent touches a third party product, I, I'm really worried that they're just going to say, we need a GDPR prompt on that. And it's going to be. Oh, back to the user through the agent. Every single write or every single non lookup. Yeah. Becomes like a, a, a safety warning, like the airbag thing in your car. And, and the regulators love that. Yeah. Because it's a liability assignment and so it has this sort of legal precedent. Yeah. And, and I really worry that that's, that's actually the mid, middle ground where we're going to end up. And, and unfortunately, because the US about 15 years ago stopped leading in tech antitrust.

• 42:46Steven Sinofsky?

  The, the problem is that Europe is going to lead with that because they have nothing to lose. Yeah. Like they're, and, and so I don't want to be sad and down about it, but I, I, I just can't get out of my head that they love prompts. I mean, look, I have to put in that browser. Your people's on prompts. Thing. Yeah. They, they, it, it, because it, it's the same, look, get into a new car. Yeah. Which I haven't done in years, but you know, like you're. A lot of prompts. Stickers off, you've got, you know, you have all of these things. Yeah. And, and someone thinks that that was success. Yeah. And it's like, has anybody ever read, like, what is this if you're, if there's a baby in this seat? And it's like, well, that's relevant for some people, some of the time, but it's this. It's the entire fabric.

• 43:27Steven Sinofsky?

  It's attached to the seat and, and they love, that, that is a very particular thing that they just love. And so I would, I would say if I were an AI now, the one thing I would be trying to avoid is, and look, we added it. Okay, FINRA for AI, we'll, we'll, we'll create that standard. I, I, I mean, look, we, we had to put this, we, on, on, when the internet was new, the big thing was to download a program and run it. And of course, if your machine is running in administrator mode, it was download a virus and take all your. And so with, with Windows XP, which was in 2000, we added this thing that prompted you and stop. Yeah. Like literally your machine stopped, user account control. And like, it was absolute assault. And we also put it in Word.

• 44:08Steven Sinofsky?

  The stupid little macro to help you write your thesis also came with a warning every time you opened your thesis saying, this has my. Everybody would just click. And everybody, and so you, you end up in this world where just like with GDPR, everybody is numb. Yeah. And so then they're like, well, it needs to be bigger. Although Mac, I mean, Mac kind of has. Nobody downloads software. That's the thing. It's a very, very usage pattern on Mac. So, and I, I wish. Sorry. I, I don't have like 10 applications on my Mac, but. Well, that's 10 and then you're done. Yeah. And it's, but it's, a lot of people still do all this stuff. Imagine if instead it was every time you go to a new website. Yes. Which you do now. No, that'd be bad. Yeah. I mean, to, to, to start from here, but to bring it kind of back to the, the macro point, like I wish this was the discussion we were having. Yeah.

• 44:52Steven Sinofsky?

  Which is like, I feel like we've dealt with a lot of these problems. I feel like when we talk about philosophical X risk, we're not solving these very pragmatic problems. I actually think this is actually a constructive conversation to have. Maybe prompts would help. I don't know. And I, I think part of the problem is, is like, people don't remember like, you know, how, how unfettered access was and how bad it was. And just how relatively benign that ended up being like, I even remember like, this is at Stanford during our PhD, I remember the oscilloscope was kind of janky. I'm like, what's going on with this oscilloscope? Like it's, you know, it's a little slow and I was measuring the network traffic and I was like, more network traffic than you would expect. And I'm like, why is there network? I didn't know the thing had a TCP stack. And somebody like broke it in because there's an old version of Windows CE and was running a porn server.

• 45:34Martin Casado?

  You know, and so like

• 45:35Steven Sinofsky?

  I don't do that.

• 45:37Martin Casado?

  It used to be

• 45:37Steven Sinofsky?

  For the record, no.

• 45:38Martin Casado?

  It it it used to be the case that like anytime you turned over a stone.

• 45:43Steven Sinofsky?

  Yeah, yeah.

• 45:43Martin Casado?

  Somebody had broken into something and like, was it this like worst case malicious, whatever? It was actually very rarely, even though the capability was there. And so if if we could just somehow tone down the rhetoric and put it in context and these still are computer systems. And yes, there's very serious stuff and people have definitely died because networks have gone down.

• 46:03Steven Sinofsky?

  Yeah.

• 46:04Martin Casado?

  There's been real issues, but like, and then can we just quibble about GDPR? That would be amazing. But the problem is like, that's not the discussion. It's not about GDPR and prompts. It's about species extinction.

• 46:14Steven Sinofsky?

  And philosophy.

• 46:15Martin Casado?

  And philosophy and and irrefutable things. And it's just

• 46:20Steven Sinofsky?

  It's also very soon. Like, I, I, I think that that's such a great point. And I, I think you, you hear people now talk about we regulate airplanes and we regulate cars. And you forget like, well, the first cars were at the turn of the century.

• 46:32Martin Casado?

  Yeah.

• 46:32Steven Sinofsky?

  And unsafe at any speed was in the mid 1960s.

• 46:35Martin Casado?

  Totally.

• 46:36Steven Sinofsky?

  And, and, you know, people have been doing, selling pharmaceuticals during the gold rush. And thalidomide happened fifty, you know, fifty or seventy five years later and not even in the US. And then there was the FDA. And, you know, people, the first pilots were flying, obviously, at the beginning of the 20th century. And it wasn't until the 1920s that you had to get a license to be a pilot. And you literally showed up with your own plane and you got a certificate. It was like, it was literally no different than driver's ed is today. And then it was 20 more years until they had anything to do with airworthiness and looking at your plane. But it was, it was minimal. And then it wasn't until way after World War II that they got involved in like what you think of as the modern FAA. And so you're looking at 40 years

• 47:19Martin Casado?

  Yeah.

• 47:20Steven Sinofsky?

  of of innovation and they were not moving slow. I mean, have you ever seen that video in black and white of like all the different planes that crashed? And that was 20 years after the Wright Brothers.

• 47:29Martin Casado?

  Yeah.

• 47:29Erik Torenberg?

  And the FAA is incredibly effective.

• 47:31Steven Sinofsky?

  And, and, and right, it's incredibly effective.

• 47:33Erik Torenberg?

  It's the safest form of transportation, like, you know, and so I do think that this process

• 47:37Steven Sinofsky?

  It's also the slowest, most difficult form of innovation. And so if you start the, if you had started the FAA in 1910.

• 47:43Martin Casado?

  Yeah.

• 47:44Steven Sinofsky?

  You'd never, you'd never have

• 47:45Martin Casado?

  You'd never have Well, I, I was, I was listening to a Nick Bostrom podcast just a couple of days ago. No, I, I

• 47:54Martin Casado?

  It was interesting. It was interesting. And, and, but no, but, but, but he actually makes his point. If you regulate AI too early, you actually basically don't solve anything. And you, you, you, you still just kind of had the same risk ultimately, but you don't understand

• 48:06Steven Sinofsky?

  Yeah, you've invented the, you will the thing into, to, to being, but you haven't figured out how to control it. Yeah. Well, we haven't figured out what it actually is yet. Like, like there are new things coming out all the time and like casting AI completely differently

• 48:19Martin Casado?

  Right.

• 48:20Steven Sinofsky?

  as than we thought of just six or nine months ago. And I think that that, like, all the innovations that can happen at the application layer is going to cause things to move in and out of the models in different ways. And like we're, we, you know, we, we thought up until last week, I think, that, you know, text prompts and text coming back was going to be the best way to interact with

• 48:39Martin Casado?

  I know, the Jeff show.

• 48:40Steven Sinofsky?

  And Jeff Chef.

• 48:41Martin Casado?

  It's so good, too.

• 48:42Steven Sinofsky?

  And then also talk about that.

• 48:44Erik Torenberg?

  Yeah, why you find it so remarkable, yeah.

• 48:46Martin Casado?

  Well, okay, so, so the way I think about it is, um, so, okay, so LLMs were kind of text in, text out, right? They generate text and, and, and they came from chat, right? It was to communicate with a human. And we've spent the last few years trying to take this thing that spits out text and cram it into a traditional program.

• 49:01Steven Sinofsky?

  Hmm.

• 49:02Martin Casado?

  Right? And, but traditional programs don't really speak text, right? And so then you end up doing this janky thing where you're like, in the prompt, you're like

• 49:09Steven Sinofsky?

  Here's the scheme on

• 49:09Martin Casado?

  Here's the scheme. But like, the thing is generating text and it kind of ignores it and it's just been super janky. And so what Jeff basically says is, listen, um, you know, generating the text on the outside is a very expensive thing, but it's also kind of, you know, it's more complicated than you need. So why don't we, we, we'll, we'll, we'll read text and we'll have all of that kind of knowledge to read the text. But then rather than generating text, which is very expensive, we will just, if you give us a set of options, we'll choose the best option. We can do that incredibly, um, we can do it incredibly fast, incredibly cheaply. But also we can do it with much more accuracy because we can train just for this. And so for all of the use cases that are not talking to, to like a, a chatbot, but are actually trying to put it in traditional software, this is a great fit.

• 49:52Martin Casado?

  And so this has probably been the fastest adoption of an AI model since ChatGPT. It's just been remarkable because we're all primed for this.

• 49:58Steven Sinofsky?

  I just want to pile on this one because I, I can't tell you how much I love seeing this exact form of, of, of innovation. And because what it does is it does the thing that's bugged me from the very beginning, which is there's been no user study ever that shows like interacting with the computer using full natural language is efficient.

• 50:18Martin Casado?

  Hmm.

• 50:19Steven Sinofsky?

  It's, it's like literally always the least efficient way. And it's very simple. And it's just like, ask yourself, how many people are asked, are really, really good at asking questions? And immediately that's like, less than half the people can ask a good question in a meeting.

• 50:31Martin Casado?

  Yeah.

• 50:32Steven Sinofsky?

  And then how often do you look at the answer and get really frustrated before it's finished? But you have to pay all this money to watch the seven paragraphs come out and then apologize that it's only a little bit. And so that's one, like having a different model. And then the other, of course, is my favorite, which is the output of it is designed for probabilistic programming.

• 50:50Martin Casado?

  Yeah.

• 50:50Steven Sinofsky?

  And so instead of saying like, is this a customer service question? Then route to customer service, otherwise route to general help desk or whatever. It's like, well, this is 80% customer service and that's exactly simulation. And it turns out there's like 50 years of computer science research in literally like probabilistic if statements.

• 51:09Martin Casado?

  Hmm.

• 51:10Steven Sinofsky?

  And so suddenly the coolest place to be in computer science is going to be in probabilistic programming, which was like all of computer science in the 1960s and 70s. So it was basically, how do, because all of computers started with doing math and it was all simulation. So it was like, let's launch the missile and hit that target, but it's windy. But wind isn't constant. So like, let's model the wind and decide where to put the thrusters and where to do the art. And so most programming through like, say, 1970, before it got to accounting was

• 51:43Erik Torenberg?

  And then we ruined everything.

• 51:45Steven Sinofsky?

  No, in accounting, there's no probability in accounting.

• 51:47Martin Casado?

  Right.

• 51:48Steven Sinofsky?

  But, but most programming was basically this modeling kind of thing. And so most programming language design was trying to figure out how to put probability into if statements or into while loops. Like do this until something happens, maybe most of the time. And, and like, so my first CS class, like the very second assignment or so was a simulation about like waiting in line at a store. And I didn't know it at the time. I actually looked all this up when I was reading about Jeff, which was like, the, the whole thing was my, my professor was like, wrote the book called The Theory of Simulation in like 1960. And I just didn't know that because it was kind of died by the 80s because it was all replaced by hyper.

• 52:26Steven Sinofsky?

  And so this, this notion of probabilistic, and, and that slide deck you shared about the future, though it's different about language models and stuff that you said was super good.

• 52:36Martin Casado?

  Oh, how reflects one.

• 52:37Steven Sinofsky?

  How was, yeah.

• 52:37Martin Casado?

  Phenomenal.

• 52:38Steven Sinofsky?

  It was a great deck. But the part that I felt was missing was that like, oh, wait, this is not all new.

• 52:43Martin Casado?

  Yeah.

• 52:43Steven Sinofsky?

  Like, like all of computer science was this probabilistic stuff. And so it's going to be very interesting to dust off all of that work because it's exactly what's going on. Like, it's not, an if statement now is if X percent, not if always. And so the way that, that Jeff worked is just to like, you, you basically, it's a custom programming language almost, which is here's the

• 53:04Martin Casado?

  Right.

• 53:04Steven Sinofsky?

  prompt, come back with a percentage.

• 53:06Martin Casado?

  Yeah.

• 53:07Steven Sinofsky?

  And then you put that in the if statement.

• 53:08Martin Casado?

  But the consequential thing is like, finally, we have a way to integrate these language models into a traditional software. And I think it's kind of funny because it, like, you ask the question, like, why haven't the labs done this, right? And it's kind of like, like, not an indictment, but a reflection on how they think. Like, they're trying to create beings and beings speak. If you're trying to create God, God speaks in natural languages or whatever, where this is really about something that's for traditional software, which is kind of not the direction that they've been taken. But one of the reasons the uptick was been so dramatic is because a lot of us software people have been trying to integrate these models into software and it just hasn't. So even before you get to the probabilistic, like, if I want a language model to drive an if statement, like, it's really hard today.

• 53:50Steven Sinofsky?

  Yeah, yeah.

• 53:50Martin Casado?

  With this model, it makes it much, much, much easier. And then, of course, this could change the nature of software fundamentally to make it more stochastic over

• 53:57Steven Sinofsky?

  Well, I think, but I, absolutely. And I think that what's so cool is that, that it is happening outside the models because that's what I think is just going to happen, which is the center of innovation has just moved.

• 54:09Martin Casado?

  Yeah.

• 54:09Steven Sinofsky?

  And it's just, and now people need to like, it turns out that the

• 54:12Martin Casado?

  Oh, you mean outside of the lab?

• 54:14Steven Sinofsky?

  Right. The platform providers

• 54:15Erik Torenberg?

  A hundred percent.

• 54:16Steven Sinofsky?

  You know, basically reach a critical, a point of critical mass where the innovation stops happening at the platform layer.

• 54:22Martin Casado?

  Hmm.

• 54:22Steven Sinofsky?

  And, and then, you know, Apple, there's this famous expression in the Apple community called Sherlocking, where Apple looks around and the, the things from the outside world become features and people complain and it's a But that's sort of how the innovation works because once you're a platform, you're overwhelmed, no matter how many people you add, you're overwhelmed with just keeping the thing running and compatibility and stuff like that. And so I think that this is the signal that now people have figured out that there's innovation to be done

• 54:50Martin Casado?

  That's awesome.

• 54:51Steven Sinofsky?

  to the model, but outside the model.

• 54:53Martin Casado?

  Yeah.

• 54:53Erik Torenberg?

  Guys, thanks for coming.

• 54:54Martin Casado?

  Okay, okay, great.

• 54:58Erik Torenberg?

  Thanks for listening to this episode of the A16Z Podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or a review and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts and Spotify. Follow us on X at A16Z and subscribe to our Substack at A16Z.substack.com. Thanks again for listening and I'll see you in the next episode.

• 55:24Erik Torenberg?

  As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see A16Z.com/disclosures.