(Preview) Understanding the AWS Outage, Resiliency Realities and Innovation Urgency, F1 Heads to Apple TV

Celebrating the return of the NBA with reactions to Inside the NBA on ESPN before turning to an extended explanation of the technology underlying the AWS outage this week and the history of US-East-1 in Northern Virginia. Then: Grappling with the trade-offs inherent to investing in resiliency to preserve the status quo, the risks that preservation comes at the expense of innovation, and Twitter as an object lesson. At the end: Questions on F1's deal with Apple, an F1 hater checks in, a sales pitch for YouTubeTV, and Ben chooses his favorite karaoke songs and explains how he learned Chinese.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Andrew Sharp?

  Hello and welcome to a free preview of Sharp Tech.

• 0:08Andrew Sharp?

  Hello and welcome back to another episode of Sharp Tech. I'm Andrew Sharp and on the other line, Ben Thompson. Ben, how you doing?

• 0:19Ben Thompson?

  Better than you, Andrew. As we all know, the NBA championship was decided last night. I am wearing my Bucks polo shirt in celebration of obliterating your Washington Wizards. Season's over. Give us the ring right now.

• 0:34Andrew Sharp?

  The Sharp Tech Bowl. Not a lot of drama in the Sharp Tech Bowl. I was talking to Bill Bishop earlier in the week and I mentioned that the Wizards were opening with the Milwaukee Bucks. And he said, are you guys going to have some sort of bet on the game? And I was like, not in a million years will I be betting on the Washington Wizards. I'm happy for you. It's great to see you in the Bucks polo, you know, beaming with joy after Giannis debuts here, turning over a new page after the Brewers' journey a couple of weeks ago.

• 1:03Ben Thompson?

  Brewers? What are the Brewers? Not familiar.

• 1:06Andrew Sharp?

  It's all about the Bucks these days. That's exactly right. Here we go. Yes. Well, basketball is back. It's been a very entertaining 48 hours and now we have a gambling scandal looming over everything. So.

• 1:19Ben Thompson?

  Now the NBA is truly back.

• 1:21Andrew Sharp?

  Indeed. Chaos in our midst.

• 1:24Ben Thompson?

  They tried to run the Clippers as like the scandal of the year, forgetting that no one cares about the Clippers.

• 1:30Andrew Sharp?

  Child's play.

• 1:30Ben Thompson?

  Yeah. So they need to go with something new. So here we are.

• 1:34Andrew Sharp?

  Indeed. Well.

• 1:36Ben Thompson?

  Actually, no. I have some comments on the NBA.

• 1:39Andrew Sharp?

  Great.

• 1:39Ben Thompson?

  Number one, like everyone else, thrilled with NBA on NBC. Imagine trying to make a broadcast look good and to make people excited about watching a product. What a concept. A video that they create before the game, an iconic sort of entrance, not running commercials up until the tip is shown, showing player introductions, like the lowest hanging fruit in the world. NBC delivered. Thumbs up to them. I'm with everyone raving about it.

• 2:12Andrew Sharp?

  Amen. I completely agree.

• 2:13Ben Thompson?

  However.

• 2:14Andrew Sharp?

  What's your next take?

• 2:15Ben Thompson?

  That was only improvement number one. And in the ranking, it's actually number two. Number one is I don't know how this happened. I mean, I knew it would happen, but I didn't fully appreciate. The biggest victory in my NBA viewing experience is Inside the NBA. Basically completely taking over the ESPN presentation. ESPN still does the broadcast themselves.

• 2:39Andrew Sharp?

  By themselves.

• 2:40Ben Thompson?

  It's unbelievable how bad the ESPN shows were. And then it really bugged me. A game would end and it goes straight to SportsCenter.

• 2:48Andrew Sharp?

  Yeah.

• 2:48Ben Thompson?

  A property that was massively important and valuable in the 1990s. What are we doing here? Like I, last night, the game's finished. I was working on my update. I had Inside the NBA, they were on fire last night. Absolutely hilarious segments. And it was so great. And you know what? It's like an experience. You watch the games, then you go to bed with Kenny and Charles and Shaq. And it was, it was so good. I can't believe somehow they're on ESPN. And by the way, there's something for ESPN to learn from. The ESPN, the extent to which they're hated is kind of incredible compared to the 90s when they were so cool. And there's a bit here where I'm sad we lost TNT. But TNT was kind of like the rebel upstart.

• 3:33Ben Thompson?

  Like that was the whole thing with the studio show. It was like not, it didn't feel highly produced, even though it's like the best show on TV. It was sort of slapstick. Like you got the late night cable feel from Inside the NBA.

• 3:45Andrew Sharp?

  Totally. It was a couple of guys hanging out for an hour after four hours of basketball games.

• 3:50Ben Thompson?

  That's what.

• 3:51Andrew Sharp?

  It was never more complicated than that.

• 3:52Ben Thompson?

  Right. What we got from ESPN was like a luxury brand knockoff. Where they're trying to present themselves as like a big, important broadcaster. But they're cheaping out on everything. Like their camera quality is the worst. Their studio shows were always terrible. The, the reason they're pushing SportsCenter is because it's cheap content that they, that they used to be able to run really expensive commercials again. Cause everyone watched it. And this was a return, like actually, even though it felt so weird to have Inside the NBA on ESPN. It felt like. A possible opening for ESPN to remember their roots and to go back to being the sports channel for people to hang out on with crazy stuff happening at 11pm at night.

• 4:34Ben Thompson?

  And huge thumbs up to me. I'm like, given the NBA a lot of grief, it still deserves a lot of grief in many respects. But wow, didn't appreciate how it really was ESPN. If ESPN is your prime broadcaster, TNT was still great. ESPN is the backup broadcaster with the slapdash show and whatnot. And then NBA and NBC like.

• 4:57Andrew Sharp?

  That can work. Yeah.

• 4:58Ben Thompson?

  It's a mullet strategy with a much more attractive mullet. You have business in the front with NBC. You have ESPN in the back where they belong. Two thumbs up from me.

• 5:08Andrew Sharp?

  Yes. Well, I am going to be writing every week on Sharp Text, a new website called Sharp Text. You can reach the site directly at sharptext.net. Last week I wrote about the trade war. This week I wrote about the NBA's new broadcasting landscape for a post that will publish on Friday. And I talked about why de-emphasizing ESPN is so healthy in the grand scheme of things. And having Inside the NBA there to salvage things on the ESPN side makes the new environment that much better. And so I'm very, very excited as a basketball fan. I didn't expect to be this excited about the new broadcasting structure.

• 5:52Andrew Sharp?

  And I was a little bit worried about Inside the NBA on ESPN. Because for the last 20 years, ESPN ruins all their programming by just overloading these shows with commercials. So people have like 60 seconds, 90 seconds to talk.

• 6:09Ben Thompson?

  Producers of like what to talk about and things like that. Like the, I mean, this is obviously one we feel particularly acutely as Bucks fans. When basically Malika Andrews was pushed by an ESPN producer to ask Giannis, minutes after they lose game six Toronto in 2019 about his plans to leave for another team. And they've just pushed this endlessly for the last six years. And just in general, all they focus on is the transaction game. What's going on there? Which? Is intellectually interesting and such a detriment to actually enjoying the game, which is supposed to be the point. Uh, I have such disdain for ESPN and basically the reason why Inside the NBA is so great is because it's so clearly ESPN is not involved

• 6:54Ben Thompson?

  at all. They slapped their logos on there and everything else. Like there was even, one of the segments last night was asking Charles Barkley which TV personalities with which network? And one of them was Ellie La Forge, who was the sideline reporter for TNT last year. And he says, oh, she's with us and ESPN. And no, actually she's still with TNT, but Charles is like, you tricked us because they're still at the TNT studios, right? Like it's just-

• 7:21Andrew Sharp?

  Oh yeah. So he's seeing her every day-

• 7:23Ben Thompson?

  That's right.

• 7:23Andrew Sharp?

  Down there in Atlanta.

• 7:24Ben Thompson?

  So anyhow, uh-

• 7:27Andrew Sharp?

  A plus. A plus to ESPN for not ruining a good thing. The greatest studio show of all time. And they said all along, we're going to leave it untouched, hands off, let them do their show-

• 7:39Ben Thompson?

  But it's hard to believe them. Yeah.

• 7:40Andrew Sharp?

  Yeah. Corporate partners say stuff like that all the time.

• 7:43Ben Thompson?

  Well, I'm here. By the way, I'm going to defend Inside the NBA before we get emails. There is a certain contingent of NBA Twitter that complains about Inside the NBA that, oh, they don't know what they're talking about. All they want to throw in the post, they diss on current players, da, da, da. Totally, totally misses the point. This is the same critique that is applied to Bill Simmons. Like-

• 8:02Andrew Sharp?

  Yeah.

• 8:02Ben Thompson?

  These are entertainment products. This applies to the NBA too. And like the whole replay fiasco, which is still a disaster. Like, like final minutes going on. Like, it's an entertainment product. Getting, being perfect, being able to cite an Excel spreadsheet is not entertaining. And it's funny when they're wrong. They also, by the way, sometimes get some stuff really, really right that the nerds miss.

• 8:27Andrew Sharp?

  Really right. That's the important point with both Bill Simmons and Charles Barkley. They often have insights that elude some of the nerdier Excel sheet crowd. So it's A, more entertaining and B, I would say equally insightful in terms of what you're getting.

• 8:44Ben Thompson?

  Right about the big things. Your big thing.

• 8:45Andrew Sharp?

  There's a lot of right about the big things coming from the Chuckster. And also the only sports show that my wife looks forward to, Inside the NBA. So there's another endorsement for you. Uh, but I'm glad that we're celebrating basketball's return with a 10-minute intro on the NBA.

• 9:03Ben Thompson?

  Oh, we're going to be back to ESPN in a little bit. Just hold on.

• 9:06Andrew Sharp?

  Indeed, we will. For now, we will start with a killer article that you wrote this week. I really enjoyed the article on Wednesday.

• 9:17Ben Thompson?

  I just want to jump in. I appreciate the enthusiasm with which you said this because I saw the rundown. You write a killer article and the problem is now that we work together, I'm not sure I can trust anything you say. So, um, okay, I feel you really turned it up there. Um, you persuaded me. So thank you.

• 9:33Andrew Sharp?

  Yeah, I convinced you it was genuine. Yes, I sat there on Wednesday. I was like, man, that was really great. You unified three big American stories over the past couple of years, two of which dominated discussion over the last two weeks. We talked about rare earths on last week's episode, so let's start with the AWS outage. Because on Monday, I went to my Starbucks app and found that it was broken. And then I turned my phone on and off three times because I thought it must be an iPhone problem. Then I showed up to Starbucks and they said, no, the app's not working for anyone. And then I finally got my coffee, went to read the Wall Street Journal to prep for a Sharp China episode. The Wall Street Journal site was down too.

• 10:19Andrew Sharp?

  That's how I found out that there was an outage that the whole country was dealing with on Monday. So what the hell happened with US East One? Because I had never heard of US East One. What's the proper way to characterize what US East One is? Like if I'm explaining it to my wife, is it a data center, a data center cluster? What do we go with?

• 10:42Ben Thompson?

  Yeah, it's a data. I would just call it a data center, even though at this point it's like a massive complex. So with these huge clusters, you go like, I'm going to serve my business at AWS. When you do that, you actually pick what's called a region. And that region is where you put the core of your application. And best practices are you're in multiple regions, but there has to be a source of truth sort of somewhere. And ideally you're doing followovers and writes and syncs and all, but this is a, it's, this is a massively complex problem. It's really, really hard. Think about your Starbucks app. You can be there in line. You can say, oh crap, my Starbucks balance is low. Press a button, money gets added. You go to the cashier, you flash the thing.

• 11:26Ben Thompson?

  Within a matter of seconds, there is money changing hands that implicated a barista right there, taking your order and making it. The level of speed, low latency, complexity of this is, it's actually insane that any of this stuff works as well as it does. The, but, but even within that, that's sort of an internet story broadly. But within that, there's, the enemy of everything in computing is latency. And you, you have this challenge of you're in some respects governed by the speed of light. So, yeah, there's aspects of me being in America where my internet experience is different just because there's literally not signals going across the ocean. Right? Like, so stuff like that can certainly matter.

• 12:11Ben Thompson?

  But there's also things like, like the complexity of if you're on AWS and you have other data on Azure. Like you have to cross over like the public internet from like AWS's internal network to the Microsoft one. You can run into challenges there. You see as little as Netflix, like I think about the Netflix whole story is Netflix puts servers in like your local ISPs that actually serve the content. So it's never actually going over the public internet. It's but the all, what's over the public internet is like your authorization and, and you're the actual like, like the catalog that's customized to you and, but all that is, is much.

• 12:46Andrew Sharp?

  But that's why Netflix is better at serving content than most of its competitors because they've built out that.

• 12:52Ben Thompson?

  There's a really, yeah, for sure. Now, all the other competitors, all the content is still on CDNs. It was content delivery network. So say like a cloudflare, cloudflare, also if you go to your local ISP office, there's a Netflix, they have a rack there. There's a Netflix bit that has all the Netflix shows that's like right there. They push it out, you know, new shows come out. So that when they release it at, you know, like midnight or whatever it is, they've actually spent the last week seeding that all over the world. So that when you hit play, it's just playing down the road. There's another bit in that server in that rack, that's a cloudflare server that holds all your content delivery stuff. There's an Amazon. On one there. So like Passport, like this podcast is going to be hosted on a CDN. So that when you hit download, it downloads super fast. It's close to you. It's not getting clogged up by the internet.

• 13:36Ben Thompson?

  And one of the huge challenges of Passport is CDNs are kind of much more straightforward and obvious. If you have one piece of content you want to serve to anyone, we want to have an authentication layer on top of it. So you can only download it if you're a Systrecre subscriber. So we, there's a whole level of complexity. How do you actually manage verifying someone? Do they have the right token XYZ? And then delivering the content that's close to them and keeping this all in sync and managing it is devilishly difficult. So anyhow, within a sort of overall cloud, you have AWS as a whole, then you have regions. If you're within a region, then having all your stuff in the region is going to operate better, is going to be easier.

• 14:20Ben Thompson?

  All be just because they're closer together physically and they're also closer together from like a network topology standpoint. Like you're jumping through fewer hoops, you're not crossing into, even if it's like an AWS to AWS, they might have dedicated fiber that goes between their things. But you're, you're, every time you go up a level as far as networking goes, you're introducing latency, you're introducing complexity. It's getting harder. The, the, just as a rule of thumb, the closer things are together, the easier it is to manage. And the less variables you're dealing with, the less problems you're likely to have. Now, within regions-

• 14:51Andrew Sharp?

  That's interesting. So from your article, my takeaway was that it's cheaper to have it all in one place. But it also sounds like it's just more efficient and you'll have fewer errors.

• 15:01Ben Thompson?

  It's so much easier. Managing multiple, managing multiple clouds is hard. Even within a cloud, managing multiple regions is hard. Now, within a region, there can then be individual like data centers and like that's sort of abstracted away. I, I'm not, I think Amazon probably has like super highly, I, I don't know. This is beyond the realm of my knowledge. They probably have like certain even higher level plans or whatever, you can guarantee stuff's close, even closer together than just being in the same region. But basically the long and short of it is, is that by far the easiest and most resilient, to use a word, way to build an application is to have it all in one region. It's to, so you have your databases there, your compute is there. Everything is in one place and it can sort of work together.

• 15:44Ben Thompson?

  And it's, and so US East One is a region of AWS. AWS has-

• 15:51Ben Thompson?

  Tens of, maybe even hundreds of regions. And this is important that you have laws about like where data has to be located. It's a big thing in Europe, right? Data has to be in Europe, that has to be in, so, but there's, but also you want fall over regions. Like you don't want all your data in one place because where that place consists by nuclear bomb? Or, or a natural disaster or whatever it might be. So you need it elsewhere. Or what if the region goes down? So what happened here, it's not clear what exactly happened other than it's some sort of DNS issue. If you think networking is hard, this extra hard part of networking is DNS. So DNS is.

• 16:25Andrew Sharp?

  DNS, here we go.

• 16:27Ben Thompson?

  So you, DNS is basically domain name service. If you type amazon.com, amazon.com is meaningless. What happens is you go to DNS server that translates amazon.com to an IP number. The IP number, the most used ones are IP four. There's also IP six, but there's IP four, which is that string of four numbers. So one nine two point one six eight point one point zero. Now that's usually like your home, uh, or then we get into NATS and like, well, that means.

• 16:58Andrew Sharp?

  How deep down the rabbit hole do we want to go?

• 17:01Ben Thompson?

  But basically, you, if you have the Amazon, you could just type the Amazon IP address into your browser and it would get there. But there's a human translatable aspect here. But actually all that NATS stuff is important because there was a huge scare a while ago. The reason why we have IP six is that IP four is just, there's a limited number of numbers. It's 10 to the power of 12, which you think about all the, every, every device in the world, everything has its own IP address. My computer has two IP addresses. It has one for the wifi and it has one for the network connection. And so my house alone has hundreds of IP address devices. What happened was this crazy hack called, uh, called NATS.

• 17:41Ben Thompson?

  There were basically, you have a, I have a router that sits and faces the internet that gets the incoming connection. And then within it, it basically creates a network inside the house and redistributes IP addresses. So your IP address of your computer could be the same as my computer and it doesn't matter because they're, they're on, on, on different NATS. Anyhow, the, the problem is within, you, you think about US East One, this huge data center with a gazillion servers, all which need their own IP addresses.

• 18:09Andrew Sharp?

  IP address, yeah.

• 18:10Ben Thompson?

  So there's all these tables and these translation layers that is happening so you can, I can sit here in my office on my laptop and access something that ultimately resides inside a database within a virtual machine in Northern Virginia, or, or whatever it might be. And something in that got screwed up.

• 18:31Andrew Sharp?

  Okay.

• 18:31Ben Thompson?

  Easy to screw up, by the way. So I mean, I'm sure Amazon will be looking to see what happened, how this happened. But as soon as they said, oh, something went down, everyone involved networking probably assumed, oh, it might be a DNS issue. They say, yeah, that makes sense. Very hard to recover from because there's, and there's all sorts of these dependencies where, I mean, even the home networking, if you screw up something with IP address or something, you can't reach the device. Like you have to go to the device and like connect to it directly to like, you know, set it to, there's D H cheaper. Like there's, there's just, it's a whole, like I've done with home networking.

• 19:06Andrew Sharp?

  When that happens for me, I call Ben and Ben walks me through it. Really sophisticated enough to handle a problem like that.

• 19:13Ben Thompson?

  Anyhow, yeah, so there's, it's really complex. It's really difficult. And you can see how problems like this happen. What's interesting, and this is sort of the point of the article, is how broadly impactful a problem like this ended up being.

• 19:30Andrew Sharp?

  Totally.

• 19:30Ben Thompson?

  So US East One is actually really interesting historically. So you go back to the early nineties and the internet's just becoming a thing. The internet starts as this, you know, ARPANET, this military product. The, the myth around the internet is that it was created to avoid a nuclear attack. That's actually not true. It was created because there were very few, uh, supercomputers in the world back then, where the supercomputer is slower than your Oppo Watch today. But they wanted to have more access to it for more researchers without having to be physically there. So the initial.

• 20:03Andrew Sharp?

  So widely dispersed was the goal.

• 20:05Ben Thompson?

  Well, no, it was just to get access to computing, which is actually cloud computing today. So you could act, now, along the way. Where the nuclear story comes in is actually that was a motivation for the creation of what's called packet switching. So instead of there being a serial connection where you have to have a direct connection computer and you sort of like send the data as a stream of bits, the way data is actually distributed on the internet is it's all chopped up into packets and all those packets are sent individually. So imagine if I'm mailing you something, instead of mailing you a big box of the whole thing, I send the box in all its constituent atoms.

• 20:42Andrew Sharp?

  Great.

• 20:42Ben Thompson?

  And like, so they're very small. They can all make it. If anyone gets blocked-

• 20:46Andrew Sharp?

  A billion little boxes on my doorstep.

• 20:48Ben Thompson?

  Yeah, it just routes around automatically. It finds another way there because it's all interconnected, right? Even if the path is longer, it can still get there. And then when it gets to your house, it reconstitutes itself into an object. That's how the internet works. Packet switching, a motivation for the creation of packet switching was, in fact, that's where the internet, the nuclear story comes in. Back in the day, the original ARPANET was sort of the serial, send it direct. If one of the nodes got bombed, you couldn't route it. After packet switching, now you could sort of route a route. And this is one of the many truths slash myths about the internet, which it is highly resilient. It can survive a nuclear attack. Like, like, like,

• 21:30Ben Thompson?

  as long as something's connected, stuff can get through. The theme here, though, and what we saw this week, is the internet is in theory resilient. And if you want to be one of the Excel spreadsheet guys that are mad at Chuck and Kenny, it is resilient. But sometimes there's a difference between theory and what is the actual experience, reality of everyone. And when you were at Starbucks, trying to access the Wall Street Journal to prep for Shark China, the internet was not feeling very resilient to you.

• 22:05Andrew Sharp?

  Life came at me fast. Indeed. Well, so my question, reading the article, reading this portion of the article specifically-

• 22:14Ben Thompson?

  Oh, sorry. I didn't finish my history of, of, of, of, of US East One. Do you want some more?

• 22:19Andrew Sharp?

  Well, I mean, I've driven out there a lot and it's unbelievable how many data centers have popped up over the last 15 years in Northern Virginia, but this is the big mama data center. So yeah, give me more US East One history.

• 22:32Ben Thompson?

  So as they're building out the internet, there is a major exchange that was built in because of Washington DC, I think almost certainly that where like you had, you were getting these initial laydowns of, of, I don't even know what it was fiber back then, whatever they were sort of using to sort of, because what happens is when you connect on my laptop, my laptop connects to my wifi access point, my wifi access point is connected to my router, my router is connected to my ISP, and then it sort of makes its way out in the public internet. There's these basically big pipes that

• 23:00Andrew Sharp?

  Yeah.

• 23:00Ben Thompson?

  interconnect, uh, there's a specific name for them that, that it will make it to one of the big pipes. It's going to Washington DC, probably to this interchange, and then it's going to your local ISP and it's filtering down to where you're at right now. Again, a miracle any of this works. And so, so the, there was this interchange that was built there, number one. Number two, you had land and pretty cheap power, particularly back in the early 90s, and you had a region that doesn't get that many natural disasters. You guys get a hurricane, maybe like

• 23:33Andrew Sharp?

  We do have that going for us.

• 23:34Ben Thompson?

  Yeah, like

• 23:35Andrew Sharp?

  Some pretty rough summers. The humidity can feel like a natural disaster every now and

• 23:39Ben Thompson?

  No, but that doesn't count. We're talking like hurricanes, we're talking like earthquakes, we're talking like

• 23:43Andrew Sharp?

  Right.

• 23:43Ben Thompson?

  tornadoes that, that, and so it was good. That's actually a really important consideration because you, you don't want your data center to go up in a tornado, but, but then also the geographic location, it's right in the middle between Europe and the West Coast. So you, you have, remember, distance matters. So if you're

• 24:04Andrew Sharp?

  That was what was fascinating from your article. You wouldn't want to put it in the middle of the country.

• 24:08Ben Thompson?

  Right. And you think about in the Northeast, it's kind of hard to build there, right? Like there, there's, you know, the, just, it's so built out. We see this with like Amtrak, right? Amtrak gets a new, got a new train. Congratulations. It's like marginally slower than the old one.

• 24:20Andrew Sharp?

  It's slower.

• 24:20Ben Thompson?

  Because that's like a track issue, right?

• 24:22Andrew Sharp?

  We're doing great in this country.

• 24:23Ben Thompson?

  That's right. It was super dense up there. It's hard. Northern Virginia had space. You had this interchange there. It was, you know, And so you have.

• 24:30Andrew Sharp?

  And I never knew that, just for the record. I always assumed that the reason the data centers were out there was exclusively because of proximity to the federal government. And that's why it was set up that way. But the latency point, that's why I enjoyed that.

• 24:45Ben Thompson?

  Well, so there is, there is a Washington Wizards tie in.

• 24:48Andrew Sharp?

  I'm aware.

• 24:48Ben Thompson?

  What happened in the late 90s is a company called AOL.

• 24:53Andrew Sharp?

  We used to be the center of all this shit back in like 1996.

• 24:57Ben Thompson?

  And now Ted Leonsis owns the Wizards and manages as well as Lady AOL. So anyhow, the, um, so AOL, you know, AOL is a great company.

• 25:07Andrew Sharp?

  Yeah, had a nice run.

• 25:09Ben Thompson?

  Had a nice run. That's a way to put it. A better run than the Wizards ever had. So you, uh, AOL is established in Northern Virginia. It sort of establishes this space as data center central. They were the biggest internet company in the late 90s. They let a huge build out and you get this sort of network density, not just of the data centers and things like that, but power is built up to support this. The grid is built up to support this. You have all the pieces that go into it. And you start getting people. There's a lot of infrastructure engineers that are in Northern Virginia. Some of them were there for the federal government for sure. That was certain. So you had a talent base to start. AOL really accelerated this and basically made Northern Virginia the center of the world as far as data centers go.

• 25:54Ben Thompson?

  So when Amazon comes along and we're going to establish this thing called AWS, which no one really understood or appreciated how important it was. There was only one place to put the first data center. That was in Northern Virginia. That data center is US East One. It's where everyone who started AWS went there. Every tutorial that was created about creating a site on Amazon talks about US East One. US East One got all the new equipment. When Amazon launched new instances, an instance is basically the type of compute you want. You get a basic instance like an Intel processor. There's the Graviton processor. There's all these have different memory configurations and how much reserve capacity you have. Like there's a gazillion, all this, the most options, the newest features, the large.

• 26:39Ben Thompson?

  Number, so the highest availability and the lowest prices because there's a just a bunch there is always been US East One. So every the default place to build anything, particularly that is built early on. And remember back then there was just AWS for a long time before Azure sort of came along, before GCP came along, was US East One. And it's funny because Passport did not go down because we're on US East Two, specifically because like US East One has always kind of been this ticking time bomb. There was a US East One issue like six or seven years, like right before we started Passport. And I said, I'm like, no, we're not going to be on US East One, which is probably somewhat delusional. It worked out this week. But um, but yeah, it's

• 27:21Ben Thompson?

  just the, it's one of those things that it's cheaper, it's the default, it's what you do, it's just the center of the internet world. And yeah, the internet is a bunch of nodes and any one node can go out. But the actual experience, again, to go back to you in line at Starbucks, is that actually one node is more important than the rest of them.

• 27:42Andrew Sharp?

  Yeah. And everybody's on US East One, or it felt like it on Monday morning, certainly. Um, my question is, obviously, this is cheaper, it's more efficient, and everything is set up on US East One. And your article was about the erosion of resiliency that's resulted from the present day state of affairs, and some of the tough choices that might need to be made across a couple of different industries in order to build that resiliency back in. With this particular situation, though, we're talking about private companies. So like, how does that dynamic change? Are they basically scared straight after an incident like Monday,

• 28:24Andrew Sharp?

  which cost God knows how much money to God knows how many companies? All right. And that is the end of the free preview. If you'd like to hear more from Ben and I, there are links to subscribe in the show notes, or you can also go to sharp tech.fm. Either option will get you access to a personalized feed that has all the shows we do every week, plus lots more great content from Stratechery and the Stratechery Plus bundle. Check it out. And if you've got feedback, please email us at email at sharp tech.fm.