Casey breaksdown AWS outage (Whiteboard Edition)

SQUASH THOSE BUGS with Sentry - the best way to monitor your bugs in production (we know you have a lot of them) and fix them with ease. Check out https://sentry.io and get started today! Casey breaks down the AWS outage in great detail with literally the coolest whiteboard demonstration you have ever seen.

• 0:00Speaker 00

  This episode of The Stand Up is gonna be extra special because Casey is gonna do the intro. Casey, what are we talking about today?

• 0:08Speaker 01

  Hello, everyone, and welcome to The Stand Up, the number forty-five, six best tech podcast on Spotify, according to the most recent, uh, something.

• 0:24Speaker 00

  True.

• 0:28Speaker 01

  Uh, anyways, I'm sorry. Today, on The Stand Up, I wanted to cover something. I'm gonna talk about the AWS outage that happened in October. But I'm doing so because I kind of wanted to talk about a bigger thing, which is the idea of actually understanding something versus saying you understand something. So like, one of the things that happens a lot, especially, I think, to people who are, uh, earlier in their programming career, like if you're, if you're a junior programmer or something, you're coming in, um, and I know this was certainly true of me, is you, you wanna seem like you know stuff, right? Like, you don't wanna seem like you don't understand what's going on.

• 1:11Speaker 01

  So there's a lot of, like, external pressure, whether it's really there or not. You feel like you should kind of say that you understood something, or, or pretend to understand something, even if it's like a little bit hazy, or you didn't quite get it. And even if it wasn't your fault, like, even if the thing wasn't explained properly, or didn't include, like, important information, you're still incentivized to basically act like you knew what it was, right? Because it just makes you seem smarter or something, or at least doesn't make you seem junior, right?

• 1:46Speaker 01

  And so one of the things that at least I've found as I got older and programmed, had more programming experience and things like that, is nowadays, I, like, almost over-ask for, for things to be explained. Like, I'll, like, I don't care about looking dumb at all. I'm like, wait a minute, go back, like, I didn't understand that part. Like, what do you mean by this? Or like, what's that term mean, or whatever? Because now I just don't really care about that, like I'm not as worried. And I want to actually know because I've had so much experience programming where I thought I knew something or I pretended I knew something and it came back to bite me. I'm like, I want to actually know. Like, I want to be sure that when I have an explanation of a bug or I think I know the reason of a performance slowdown,

• 2:32Speaker 01

  I always in the back of my head, I'm like, if I haven't really gotten to the bottom of this, it could be something else. It could be, it could be that the real thing is still hiding in there. And I just don't know because I haven't really looked at it all the way. I'm just, I'm moving on because it's convenient or whatever. And so the reason that I wanted to talk about the Dynamo DB outage is because recently, there's been kind of a string of high-profile outages. So there was like a big one that took down Google. And it turned out it was a thing where like they, they didn't handle a field being empty, right? So that their programming, the way they were programming, they were like, okay, we have this thing, we, we load some JSON and if there's nothing in the JSON, it's just we like, we deref a null pointer or something, right?

• 3:17Speaker 01

  It was like literally that, right? And then there was one with CrowdStrike where they were like, they took down the entire world with blue screens. And that was, they gave a very good, it was like a really good explanation of it. They were like, we have, here's, we do this certain array sizing thing. And we had too many rules, so it like overflowed the array, right? And so these were like pretty good when they, when they gave what they call RCAs or root cause analysis, right? When they said like, here's why we went down. When I read them, I didn't feel like there were a lot of unanswered questions in my mind. Like, maybe I didn't know like literally the line of code that, because they maybe didn't publish literally the piece of code. But they gave me enough that I was like, okay, I understand how someone wrote this code.

• 4:02Speaker 01

  And I understand the stupid thing that, that they did, right? That like, okay, don't do that thing. I understand. And I'm totally like, okay. With the Dynamo DB one, because it came up on this podcast, right? We talked about it when that, that dude at the Guitar Center, right? Was like, I overheard someone talking. Yes. At the pub, right?

• 4:24Speaker 02

  Yes. Incredible.

• 4:27Speaker 03

  Here we see the elusive programmer, a simple creature that spends most of its time working alone, often in darkness. But what's this? Someone being wrong on the internet? Our coder springs into action, reaching top speeds of 120 words per minute before flash! A light-mode website, the natural enemy of these code lovers, stuns our friend. The chase is called off. We'll have to get them next time.

• 4:50Speaker 03

  When not on their computers, they can spend hours drawing crude symbols on something they call whiteboards. Researchers have discovered thousands of dialloks, often with more than a dozen used than a single office. However, no linguist has yet deciphered what their purpose is.

• 5:06Speaker 03

  Vain creatures, their bodies have evolved over a millennia to be able to sit in unusual postures while looking at themselves online. This will often last for many hours, using the excuse they're waiting for code review, repressed to why they're so inactive. And finally, after a long day of accomplishing very little, our keyboard warriors ready for bed. Quick read and it's lights out.

• 5:29Speaker 03

  Goodnight, little coder.

• 5:33Speaker 00

  So how do I sleep so well at night? Well, I have Sentry to help me crush those bugs. And I'm not, I'm not talking about like little teen-tiny South Dakota bugs that die in the winter, I'm talking about big, mean jungle bugs. And I'm not scared of any of them, by the way, but I can squash those bugs with Sear by Sentry.

• 5:53Speaker 01

  So I was kind of a little more motivated about that one to go like, okay, let me go see like what, how much information they've posted. And I had read, I had already kind of read afterward, they had a summary where they posted an RCA, and it was very vague. Like the RCA just did not really explain very much. I then noticed that they posted a, a full presentation, like at Reinvent in December, they, or I guess I don't know if Reinvent was in December, but the video went up in December, of the Reinvent presentation where they covered this outage. So I went and watched all of that. And after having read the entire RCA and watched the entire presentation, I still was left going, "I don't see an actual explanation of the bug here."

• 6:39Speaker 01

  Right? Like, I'm trying to figure out what the actual bug was, and it just wasn't ever explained. And so what I kinda wanted to do was just talk about that, go through why I don't think they explained what the bug was, and just use that as an example of like, "I don't think people should just go, 'Oh, okay, I get what the bug was.'" Because people have like replied to me and gone, "Oh, here's, let me explain to you what the bug was." And then they just explain the same things, I'm like, "That's not the bug." Right? So everyone is like incentivized to go like, "I understand it." Because it's like, "No, if you can't tell me what the actual bug was, then we're not done here." Right? Like, we should have that fuller explanation. So does that all make reasonable sense, like what I'm saying?

• 7:20Speaker 02

  Yeah, first off, I just want to say, I knew exactly what you were saying, Casey.

• 7:28Speaker 01

  But right from the start, right, T? It's like, right away, you were like, "Okay, I know, I know what that is."

• 7:32Speaker 02

  I know exactly what you're saying, no questions on my end, no blockers, thanks, everybody, I'm great, I'll see you guys tomorrow, you know, no problem.

• 7:41Speaker 04

  I just want to say, I really like listening to Casey talk. On the podcast, when I listen on Spotify, but also just right now, like, I could listen to you talk for an hour.

• 7:48Speaker 02

  Great shout out, too, for the Spotify podcast.

• 7:51Speaker 01

  I was gonna say, like, especially when you listen on Spotify. Your choice.

• 7:56Speaker 02

  The quality is incredible.

• 7:58Speaker 00

  You also get the bonus extras, right? You get all the banter before and after the actual main.

• 8:03Speaker 02

  Extra yap.

• 8:04Speaker 00

  Oh, really? I didn't know that.

• 8:07Speaker 02

  We started posting longer, longer versions on Spotify that are like, more of the extra yap time, less of the on top, it's not on topic stuff, but a little more yapping's on Spotify.

• 8:18Speaker 00

  Yes, because the live audience gets the yapping. They get to come in here, they get to hear about trash and his Pokemon addiction, which you probably don't even know about because you weren't, you were listening to this on YouTube, right? You don't, you don't get to hear all the fun stuff.

• 8:29Speaker 02

  But that's kinda, that's kinda a hard sell for the first 10 minutes of a YouTube video.

• 8:34Speaker 00

  It's a very hard sell for a YouTube video. Be like, "Oh, I'm gonna watch four guys talk about something I don't even understand, and it's called Dynamo DD."

• 8:41Speaker 02

  Since we're starting the podcast, maybe we should introduce Adam.

• 8:44Speaker 00

  Oh, yeah.

• 8:44Speaker 01

  That's a very good point. We haven't done introductions at all.

• 8:46Speaker 04

  Hello.

• 8:47Speaker 02

  Tell us a little bit about why we brought you onto the podcast today.

• 8:50Speaker 04

  Because I am at TJ's house.

• 8:54Speaker 02

  Number one.

• 8:54Speaker 01

  There it is.

• 8:55Speaker 02

  Number one reason. Why did I opt out for this today?

• 8:58Speaker 00

  TJ requires all people who visit his house to be on the podcast. It's been awkward at a couple of times.

• 9:03Speaker 01

  Yeah. Yep.

• 9:05Speaker 00

  Okay, but Adam, who are you really, other than an AWS hero?

• 9:10Speaker 04

  I'm not even that. I was an AWS hero.

• 9:12Speaker 02

  R.I.P. Too soon.

• 9:14Speaker 05

  You get kicked out of the superhero group? Like, how's that work?

• 9:17Speaker 04

  You just don't, you don't get renewed. I was a one-term hero. And they decided...

• 9:21Speaker 05

  Oh, you, is it like a paid up thing? You pay to be a hero?

• 9:24Speaker 04

  No. No, I just, I didn't really care about AWS anymore or talk about it ever. So they were like, maybe he's not a hero anymore.

• 9:30Speaker 02

  Now he's a villain.

• 9:31Speaker 05

  Whoa. Casey looks like he's part of like some murder mystery.

• 9:35Speaker 04

  He's standing now.

• 9:36Speaker 00

  Yeah. Oh, dude, we're, we're about to get, uh, like the, uh, what is it? Nick Hill? What's the person that does all the like drawing on the board and then it shows up?

• 9:44Speaker 02

  Casey Muratori. That's what I'm thinking of.

• 9:46Speaker 00

  Muratory. Is it Muratory or is it Moratory?

• 9:50Speaker 02

  Oh, my God, you're about to do visuals, aren't you?

• 9:52Speaker 00

  Yes, I know. This is the best podcast. It's literally, this is the best one to be a part of.

• 9:58Speaker 01

  Uh, it's pronounced Muratory by my family, like, almost like there was a Y there, like Muratory.

• 10:04Speaker 00

  That's correct.

• 10:05Speaker 01

  It doesn't really make any sense because in Italian, it's an Italian name, and in Italian, it'd be Muratory.

• 10:12Speaker 05

  Muratory.

• 10:12Speaker 00

  No, no, no. It's not Muratory in Italian. It's Muratory.

• 10:16Speaker 01

  Yeah.

• 10:19Speaker 01

  Doesn't make any, so why, how it got Mur, I have no idea. That was some Italian-American, like, immigrant thing that happened, I guess. I don't know. So here's effectively what they said. They have these things called API endpoints, maybe they call them, right? And these are the domain address, like, if you look up in DNS, it's the name that you're going to look for to know who you're supposed to send, like your Dynamo DB requests to. And these things, I guess, look like this. And Adam can probably confirm this because he is, or was, a hero.

• 10:56Speaker 04

  They look like, what, oh, it's behind.

• 10:59Speaker 01

  Yeah, we're a few seconds behind because our video disappeared on Rivers.

• 11:02Speaker 04

  Yeah, yeah, yeah.

• 11:02Speaker 01

  Okay, there we go. So they look like DynamoDB.us east dash east dash one dot API dot AWS or something like this. And I guess it depends whether you're using IPv6 or IPv4, like they have different names depending on things, or whether you're using like a specific, like, they talked about governments use like a different one or whatever. So these names are like names that you effectively hard code, I guess, into your application where you're like, when I need to do something with DynamoDB, I'm going to like ask for this. Does this make sense? And does that sound right, Adam, to like, because I don't use AWS stuff.

• 11:37Speaker 04

  Yeah, yeah, that's all right.

• 11:38Speaker 01

  So, you know, you you ask for something like this and you're going to send.

• 11:41Speaker 04

  He understands perfectly.

• 11:46Speaker 04

  I do. I mean, I know what he's saying.

• 11:47Speaker 01

  Yeah. So that then is going to redirect you somewhere because obviously there isn't like one machine that's going to handle all the DynamoDB traffic in the entire universe. Even if you subdivide it by region, which you can see here, you're kind of supposed to pick a region. I guess you don't, you don't send it to some main address. You send it to a regional address, or maybe you, there is a main address you can use that will figure it out. I don't know. But anyway, at some point, you're talking to this, and this needs to point to effectively like a load balancing scheme. So this thing is supposed to point to effectively what they called a DNS tree, although they never really explained the tree

• 12:29Speaker 01

  nature of it at all. It sounded more just like a, like a weighted array, if you will, where you just said, here's a bunch of machines, and you're going to pick those machines based on weights that we set so that we can load balance, right? So if a machine gets behind, maybe we set its weight to lower. And if a machine seems kind of empty, we set its weight to higher. And so they called it a tree. So I'm assuming it's a tree. They never explained what the tree part of it was. But this name.

• 12:58Speaker 01

  Is supposed to point-

• 12:59Speaker 00

  Can I interrupt for one quick second?

• 13:00Speaker 01

  Yeah.

• 13:01Speaker 00

  By the way, someone did get their L6 promotion based on that tree. So I do think next time you should find out what that tree is because that meant a lot to somebody. Okay, there was a packet and engineers happened.

• 13:11Speaker 01

  I do agree the tree is probably important. It's just not important for the bug. And even this, that, so that, I will say, there was no need for them to explain the tree. So I'm okay that they skipped out on what the tree is doing.

• 13:24Speaker 02

  I got a quick question as well.

• 13:25Speaker 01

  Yes.

• 13:26Speaker 02

  Is it called a tree because it's a root cause analysis or no?

• 13:29Speaker 01

  Yeah.

• 13:32Speaker 00

  Okay, no more jokes, we're too off topic. I'm sorry, I'm sorry, Kay, I'm sorry.

• 13:35Speaker 01

  So anyway, this is supposed to point to that. And that, that sort of, uh, this, this load balancing scheme, basically, of DNS entries. And the way that they described this in like their presentation is they would use a thing like, I'll say, plan145.dynodb, like ddb.aws, right? Now, this is the root of that tree, I guess. Not root cause analysis, but like this tree, this would contain, like, this is the top level record of a bunch of records that allow it to do its load balancing. And I assume Route 53 kind of has this load balancing capability. I'm reading between the lines of their presentation, they didn't say that outright.

• 14:18Speaker 01

  But I'm assuming Route 53, which they're doing all this through, you know, which is their own DNS thing, is, allows that load balancing to happen by you just set stuff up in here that says how the load balancing should sort of be working right now. And then it will pick the correct machine based on like some kind of randomization in the weights or whatever. Now, what they said was this name, which really does exist, and apparently there's a tree or something like this. This name is one that they just kind of used for their presentation. They never actually used a human readable name for this plan like 145 that I've written here or whatever. It was really a hash of something. So it would really be like, you know, 0AFE12, you know, 9A or something like that, right?

• 15:03Speaker 01

  Is actually what would be there. So if you went and looked, you would not see a human readable name, or at least at that time you wouldn't, I guess. You wouldn't see like Plan 145, you'd just see that. And so the idea was, OK, a user goes to use it. They query this name. Route 53 will direct them like to here. And this thing is some kind of a load balancing tree that Route 53 can use that will allow you to get where you need to go, right? That will give you an actual machine you can send traffic to eventually. Again, they did not describe any of that, so I have no idea how any of that works. I've never touched or used Route 53, so I have no idea. But we'll just assume that that happens because it doesn't matter for this bug.

• 15:42Speaker 00

  We do have an AWS hero. So if you do, if you are confused, you can always ask Adam, and he may have further insights.

• 15:49Speaker 01

  I mean, yeah, go for it.

• 15:50Speaker 04

  Well, Route 53 does have a lot of different ways you can like split the traffic. So yes, weighted is one of them, and that sounds like what they described.

• 15:57Speaker 01

  So somehow they've set up these records with that, and they just didn't say how, but something, something in a tree form did that. My guess is there's like a weighted, like the tree has like weighted, like there's a couple of weights at the top that branch out to more weights or something like that, because that's easier for it to deal with because there's a lot of them or something, who knows? Anyway, I have no idea. Point being, this is what's supposed to be happening normally. Now, the reason that this is called Plan 145 here, even though it actually would have been some hash code, but they refer to it as like Plan 145, is the load balancing, as you might imagine, has to be kind of continuous because the Dynamo DB machines are like doing stuff all the time. They're becoming more overloaded.

• 16:39Speaker 01

  There's machines are going down or crashing or who knows what, right? Could be happening, being taken offline. New capacity can be added. And so this stuff has to be updated constantly, like all the time. So this main API endpoint that you connect to, it constantly has to have that tree that it's pointing to be adjusted. And so the way that they do that is they create another tree, the tree that they're going to move to, right? They create like, you know, Plan 146 or something.

• 17:15Speaker 01

  And they make the whole tree here. And then when they're ready, like when this tree is done, they take this, you know, this record here. And instead of it pointing to that one, they point to this one. Right? So they make the new one and they move over to it by just changing that name. Now, for some reason, and this reason is not really explained. The way that they've set up that process is they split it into two pieces. There's something called a planner, which figures out what the new tree should look like, basically. So you can imagine there's some machine called a planner. And I don't know if it's an actual machine or if it's just a process running on some machine that's running other things. Who knows?

• 17:59Speaker 01

  But there's something called a planner. And as far as I could tell, there's only one. Meaning there's just a planner that sits there and figures out what should the new plan look like that we're going to switch to. And it's constantly doing this. So it generates Plan 145, then it generates Plan 146, then it generates 147, 148, 9, you know, blah, blah, blah, blah, blah. Right? And it just keeps putting out plans for all of eternity because that's its job. Now, it never actually creates them, apparently. It, its job is not to ever make them in Route 53. It's just to figure out what they would be

• 18:42Speaker 01

  if someone were to put it into Route 53. Then they have three enactors.

• 18:58Speaker 01

  These enactors get the plan from the planner and they put it into Route 53.

• 19:07Speaker 01

  This just makes sense. Now, one planner, as far as I am to understand the presentation, three enactors. There was no explanation for why this would be the case. They said the reason there are three enactors is because it's supposed to be fault tolerant, like if one of them goes down or something. But they never explained why you wouldn't then need three planners, because if the planner went down, then the enactors have nothing to enact. So it didn't really make any sense. So there wasn't an explanation in the thing about why this structure looks the way it does. It's not really that important to the bug that it looks this way, although it kind of is, as we'll see later. So I was a little weirded out by the fact that they didn't justify this, but that's fine. So hopefully that makes sense.

• 19:51Speaker 01

  We have a planner, we have three enactors, the enactors are all trying to enact this plan, right? Now, what happens here is that, for, again, reasons that all the, the only thing they said in the presentation was, "It makes it easier to reason about." This is what, this is the only information I have. They said, "It makes it easier to reason about." Because it makes it easier to reason about, these enactors use serialization. So instead of them just trying to create records, and if the records are already there, just not creating them or something, in other words, I have three people running, we

• 20:29Speaker 01

  all want to create, you know, let's say this top-level record, plan146.ddb.aws, right? We all are trying to do that. One of us does it first, the next person tries to do it, and it's already there or something, right? We're all trying to create the same record, so in theory, we could just have three people randomly hammering on whatever part of the plan they're trying to hammer on, and in theory, it should kind of all work, right? And I sort of got the sense, although you didn't come out and say it, I sort of got the sense from the presenter that he would agree with what I just said, meaning that they could have just had them run arbitrarily, and it would or should be okay. But, he said, "They use serialization to make it easier to reason about."

• 21:14Speaker 01

  What that means is, instead of these enactors just hammering on it like that, what they do instead is they attempt to acquire a lock for whatever the endpoint is that they're trying to update.

• 21:28Speaker 01

  So in other words, if this person is trying to update one of these things, and I got the sense that it was if you're trying to update this one, but it could have been if you're trying to update this one, or it could have been on both. They never really 100% said, if I remember correctly, exactly where the locking was occurring. But the locking occurs by them going, "Okay, I'm going to create a lock that is a DNS record." And by using the fact that Route 53 has the idea of an atomic, which is, you know, I can do two things, and if they both wouldn't succeed, then it won't do either of them. They basically made a locking system that locks via Route 53.

• 22:13Speaker 01

  So Route 53's DNS records are actually the lock record, if that makes sense.

• 22:18Speaker 00

  Can I ask a quick question?

• 22:19Speaker 01

  Yes.

• 22:20Speaker 00

  You said it does this through serialization? I don't quite understand what that means, because I thought serialization is just converting from one memory to a different memory representation of some... I'm struggling on that part.

• 22:31Speaker 01

  Sorry, different serialization. So, yes, that is serialization.

• 22:35Speaker 00

  Oh, serial, okay, yeah.

• 22:37Speaker 01

  In this case, we literally temporal serialization, meaning they wanted these enactors to have some kind of a way in which they would organize their behavior into an order, rather than just being arbitrary. And the way that they did that was locking.

• 22:56Speaker 00

  Okay.

• 22:57Speaker 01

  So what will happen is, instead of this person just doing whatever it is they're going to do, like, "Okay, I'm gonna like, I finished this, I'm gonna point this guy at Plan 146 now." Instead of doing that, it attempts to acquire a lock on like this, right? And if it doesn't get the lock, it won't make the change. So only one of these enactors can be in the process of updating this at any given time. Does that make sense? Now again, exactly what they were trying to do with that was never explained. They just said makes it easier to reason about and left it there. So I don't know why they thought this was an improvement. And amusingly, it's what ends up uncovering the bug. So it wasn't an improvement. If anything, it was probably bad, but.

• 23:42Speaker 02

  So KZ, are you saying they don't have like, they don't have a good reason for, they're saying we're gonna make the enactors run almost like one at a time. Why do they have a, why do they have three enactors? I don't underst- like, why do they not just have one? They just don't say that. We don't know why.

• 23:56Speaker 01

  And they didn't quite explain, like, I didn't really hear an explanation for how you have three concurrent enactors. You expect them to be able to go down, which is why you have three.

• 24:08Speaker 02

  Right.

• 24:08Speaker 01

  But they're taking a lock? So what happens if this guy takes the lock and then goes down? Like, I didn't hear an explanation for that either. So this was all very confusing to me. Like, I, I, I'm not complaining about it as part of what we're talking about here because it's not important for the root cause to me. But as a presentation, I had so many questions. Like, I was like, I don't understand why you did any of this, to be completely honest, right? And maybe that's, again, part of it could just be that I don't use ADWF services. It might be that some of these things would be obvious if you are someone who regularly uses Route 53 or something. You'd be like, oh, it's because locks can be set to a timeout or, I mean, I don't know, right?

• 24:51Speaker 01

  But anyway, so yeah, so they're doing that. And what ends up happening, I, for, for, uh, this, but the thing that uncovers the bug is that what ends up happening is these enactors, when they don't get the lock, they just do like a back off, right? They'll basically just do like, okay, let me wait and I'll try again. So enactor, this enactor tries to get the lock, but somebody else already has the lock. So he just waits a little while, he tries to get the lock again. That's what will happen, right? And what they said happened was they hit a pathological case, quote unquote, where one of the enactors is, you know, has enacted some plan. And that plan, let's say, was pretty old.

• 25:34Speaker 01

  I think they used 110 was an example that they used. So it enacted Plan 110. And it wants to point, you know, it's like, I gotta set the API to point to my 110. Tries to get the lock to update DynamoDB.us east dot one or whatever, and fails because someone else is enacting Plan 111 or something like that. Right? Or Plan 109, could have been a previous plan. So the other enactors are doing it. It can't do it. It backs off, right? And remember, this enactor here, we're on 110. It's trying, it's, it really wants to enact it. It tries again. Someone else has the lock now. It tries again, still locked. This person is sitting on 110, desperately trying to enact it, can't do it.

• 26:16Speaker 01

  Apparently, this just happens so many times that the other enactors, and the planner is just churning out new plans this whole time, right? The other enactors, they get up to like 145 or something and 146. They're enacting plans that are like way ahead of 110. Right? And this guy's still stalled because he just unluckily never gets the lock. Right?

• 26:38Speaker 01

  Finally, at some point, after like Plan 145 has already been enacted and pointed to by some other enactor and all that stuff, Plan 110, this enactor that's still trying to do it, finally gets the lock. And he's like, yeah. And so then he says, okay, we're pointing to 110 now. Yes. Right? So now it's on a super old stale plan. But this really shouldn't be a problem, right? Because eventually, the next time some enactor has something, it's going to be a much later plan. They'll just enact Plan, you know, 146 or 7 or 8 or whatever, and we'll re-point it back to this and we're back to a fresh plan. So everyone will just have bad load balancing for like a few minutes, but then it'll be fine. Right?

• 27:19Speaker 02

  They did have bad load balancing for at least a few minutes, right?

• 27:23Speaker 01

  Yes, true. Uh, well, it's a lot worse than that.

• 29:46Speaker 01

  So that's, that's the end game of what would happen. Whether it was from asking for this or asking for that, I'm not sure. But to just get back, no records found. That's what, that's what you would have received when you're, when you were trying to call that API. So whatever like, whatever like library you were using to like use this, to use Dynamo DB, it would just be like, hey, no records found, bro. Sorry, right? So this, if you ask anyone on the internet, right? They're all like, yes, they explained the bug. That's the bug. Like the bug is that there was this race condition, right? Everyone, because everyone, as soon as you say race condition, everyone's brain shuts off. They're like, oh, okay. Well, it was a race condition. Done, like nothing to see here, right? So they're like, it's a race condition. They explained it. It's like, no, they didn't explain it.

• 30:31Speaker 01

  Because if you think about what would happen here, immediately after this, everyone's getting this.

• 30:37Speaker 02

  A new enactor.

• 30:38Speaker 01

  A new enactor will just enact a new one, right? And so the bug, right? Is why didn't that occur? That's the actual, the actual RCA that I wanted to see is,

• 30:49Speaker 00

  Yeah.

• 30:50Speaker 01

  why didn't the next enactor come and fix it?

• 30:54Speaker 00

  Can I, can I throw out something else? Wouldn't, wouldn't also be a bug? Like why write a record so old that it should be deleted immediately?

• 31:01Speaker 01

  Well, it wasn't. It was because it was, it was, this guy had written it quite a long time ago. And it was, it, the way, well, I mean, if you're asking, why didn't they write enactors with better code? Yeah, that's a very good question.

• 31:14Speaker 00

  Because it seems like if you are updating to something that should be deleted immediately, isn't like, that's like, that feels like the problem right there. You've done something wrong long before.

• 31:22Speaker 01

  Yeah, even though it doesn't really fix the theoretical structure of this thing, a simple check in this guy when after he finished backing off on the lock, he should maybe check to see whether he's about to set this to something that he would delete if he was running his deletion code is probably a good safety measure. But yeah, so 100% agree with him.

• 31:40Speaker 00

  Okay, okay.

• 31:40Speaker 02

  That enactor worked really, really hard to get that record written.

• 31:45Speaker 01

  It's been waiting a long time. And it's gonna have its Pokemon cards.

• 31:49Speaker 02

  Anyone ever waited? So just let him write the record, okay?

• 31:53Speaker 01

  So, so I want to hear about that. Unfortunately, if you look at the presentation and you look at the RCA, it's nowhere to be found. The presentation at least has one 12-second little tiny chunk where it does say where the bug roughly would be. And so let me explain what that is. So what apparently occurs alongside this. So when, when you do DynamoDB US East 1, when you point that at your plan, you also do another operation at the same time.

• 32:34Speaker 01

  And that operation is to set rollback. I think it's DDB. Is it DDB rollback AWS? I don't remember exactly what it is here. There is a rollback record. It sets that record to whatever the old plan was. So if we were here pointing at 145 and we're now going to point at 110, right? This old inactor is like, I'm moving to the 110. It attempts to set, it take whatever this name was, right? Currently and move that new, that name, which would have been Plan 145, move that so that the rollback address points at the old plan. Right?

• 33:19Speaker 01

  And this is just for debugging or, you know, it's basically just for operator ease, right? If they want to roll back to the previous plan or something like that, or if you just want to know what the previous plan was, you can see it here, right? That's part one of how the, what they said about failure. I would want to point out one thing here was this also didn't make any sense to me because I was like, okay, you're telling me that these things update every like minute or something. What good is it to have one of those? Like, by the time you even logged in, it's been updated from the one that you wanted to roll back to, to some new thing that's actually the plan you don't want because everything went down, right? Like, it's, right, you, you don't want this,

• 34:04Speaker 01

  you just want these names in a list. So you can be like, what was it at at 12:30? Like, that one, right? So this made no sense to me. I have literally no idea why, why this would ever be good, right? It did not sound like it would do the thing you actually want, which is to be able to mark a point in time and go, we need to go back to 1:00 PM because everything went to crap after that, right? Anyway, so that didn't make any sense to me. But again, not exactly specific to the bug. So I didn't ask why. I'm just saying, okay, that's what thing it had to do. So it could only roll back one version, is what you're saying? Yeah, even though the other trees do exist. So you easily could by just knowing what the name was. So all this is, is a, is putting a human readable name

• 34:47Speaker 01

  on something you almost certainly don't care about, right?

• 34:52Speaker 02

  But they don't really, they can't really store that much stuff, Casey. I don't think they can really put, like, I don't know, Adam, like, they don't have a lot of scale there, right?

• 34:58Speaker 01

  You, you, it's like eight bytes. Dog, that's a lot of bytes. If it were me, I would have just made this a timestamp, if that's what you wanted, right? I would have said, when did the planner, or, or when did this person point to this thing? Like, when you got the lock, you changed this name to the timestamp and update this in one atomic? So then you just know, if I want to roll back to 1:00 PM, I just look for like whichever had the timestamp just, you know, the, the earliest timestamp not after that time. And that's what we were running at that time. That's what I would have done, right? But, I don't know. So I have no idea why they did this. They did what they did. I, you know, maybe it might make perfect sense. Again, I have no knowledge of their system. All these things may make perfect sense. So I'm not really, I'm just saying I don't understand them.

• 35:43Speaker 01

  I don't, they might not be bad ideas, right? There might be good ideas if you understood the rest of the system. So anyway, so what they say, and this is all we get, is this operation, meaning setting the rollback to point to the old plan that was being, you know, which in this case would have actually been newer in some cases, right? So it's not really the, the previously pointed to plan, which may be older, may be newer. Doing that activity, if that plan no longer existed, meaning like it had been deleted like this, then the enactor stops permanently. So every time, like, once you get into a state where DynamoDB.usc-step1, right?

• 36:26Speaker 01

  So we do the whole sequence of steps that we said here. This plan gets deleted. So now this is pointing at an invalid, like, unresolvable name. We cannot resolve plan-110, which is actually some hex code. But whatever that was, we can't resolve that anymore. Once that state is true, then the next time an enactor comes and tries to make it point to a new plan, whatever that new plan is, it cannot, like, when it actually gets this far and tries to set the rollback, that will crash it permanently.

• 37:04Speaker 01

  Therefore, all three enactors will now stop. Because eventually, all three will try to enact a new plan. They will try to set the rollback first to point to whatever the old plan was, find that there's no plan there, and that apparently is just a hard crash. Oh.

• 37:21Speaker 01

  Now-

• 37:22Speaker 02

  That's crazy. I thought the three enactors were supposed to make it so that they have redundancy.

• 37:28Speaker 01

  Now- Again, this is why I get grumpy with people online who are like replying, they're like, "It was a race condition." It wasn't a race condition. The race condition is not necessary for this. The race condition is just why you ended up with this name being unresolvable. But if you didn't have whatever code did this badly, it would have just worked. You never would have known. You would have had a momentary, like, you know, minute outage of DynamoDB or something. But I'm guessing there are minute outages of DynamoDB from time to time, right? Like, that's not global news. What's global news is taking it down permanently, which is what happened here. And until an actual human goes and figures this out,

• 38:12Speaker 01

  resets it, gets these enactors going again, it's just gone, right? It's just out permanently, so hours potentially, right? And it was long enough, I guess, in this case, to then have cascading failures. You would never have had that. It's just a momentary out. Like, if some people momentarily got an unresolvable name or no records, right? Then, then they would just try again. That's usually what hap- like, with DNS, like, that's like your phone you went through a tunnel, right? That's all that would have been. So I want to know, what did the code look like here? How did you write something that if this wasn't a valid name? Which it wouldn't even be on standup, meaning if you were starting this system and the operator hadn't preconfigured it, it wouldn't be pointing to anything, right?

• 38:56Speaker 01

  That's the default case that you would think you'd start with. So if you're going to do this, you would think you would just handle that case because the rollback address could just not point to anything, right? Just take whatever this is, if it's nothing, set the rollback address to nothing. Done, right? So there's something really weird about the way they wrote this code. And that is what should have been in the RCA. That's the whole bug to me. This is just set dressing for how we ended up having this thing point to nothing. But the same bug would have occurred if someone had accidentally deleted this record. Like, some operator was just like, oops, crap, I set it to nothing. This same bug would have happened, according to the presentation, right? So the root cause is not the race condition.

• 39:41Speaker 01

  The race condition is an aside. Does that make sense?

• 39:44Speaker 00

  Quick question. So I'm legitimately thinking through this. And so that means the thing that sets the rollback probably assumes some sort of struct with a bunch of memory or something has been passed in. Does some sort of, like, some sort of access, it explodes? Or?

• 39:59Speaker 01

  Yeah, maybe.

• 40:00Speaker 00

  Do you think this is the same style of bug? Which is the one line that took down CloudFlare? Which is they just assume it's there and unwrap it? It's in Rust, it is memory-safe Rust, unwraps it, explodes it.

• 40:13Speaker 01

  I really don't know. My guess, like, in my head, I was like, what is the thing that I see people do a lot of times where I'm always like, why would you ever do this? But it's just because that's the way they learn to program. And I was thinking, like, if you were writing in one of these languages that likes to throw exceptions for error conditions, this would be a great example of that. So if you had a thing where you were like, oh, I went to go get the DNS record that this thing points to. And normally, in a sane programming environment, no one is throwing an exception there. If they get back nothing, they just return nothing, right? And then when the person goes to set ddb.roback-DNS, they just set it to nothing, which is the correct behavior.

• 40:54Speaker 01

  Like, nothing flows, literally the value "nothing" flows correctly through this flow. So if you were writing it to be... Since it is a core foundation service, assuming you were trying to write something that was fault tolerant, you would never do something like throwing exceptions. So in my brain, I'm thinking, I bet what happens in here is when you ask for this record, they just use some library call or something that throws an exception when the record doesn't exist. And it just threw an exception and the enacted was done. That's my guess, right? And I could be very wrong about that because I'm just wild guess, right? But this is why I want to see the RCA. What was it? It could be exactly the stuff that Trash was talking about. I mean, it could be the stuff that Prime was talking about, could be the stuff that I just said, could be anything.

• 41:37Speaker 01

  And I want to know because that's where the actual education would be here. Avoiding this race condition is completely unimportant. This race condition could have lived there. And while it was important eventually to fix it to avoid those once a year weird outages for five seconds or something, it is not actually the thing that we most want to learn. What we most want to learn is don't write this thing. And we don't know what this thing even was, so how do we not write it? This is why I think it was the bad RCA. Does that make sense?

• 42:06Speaker 02

  Yes.

• 42:07Speaker 00

  Yes.

• 42:08Speaker 01

  All right.

• 42:09Speaker 02

  What is most of AWS written in, Adam? It was Java.

• 42:12Speaker 00

  I was about to say someone from the chat said Scala. They said they worked at AWS for seven years and they said most of it's written in Scala.

• 42:20Speaker 02

  Well, that's technically Java with extra steps, so. And that will anger all of them endlessly.

• 42:26Speaker 01

  So, so that's really it for me. Like that, that, this was a thing where I was like, I don't feel like I saw the explanation. And I actually feel like it's important to hear because there was a bad programming practice at the bottom of this summer and I want to know what it was. Especially because it helps people like me when I, you know, I don't really do a lot of architecture education right now, but at some point I probably would like to do some of that because I think there's a lot of bad architecture out there. And so I kind of try to pay attention to these things, like what are the kinds of architectural mistakes that people are making? And I bet this was one of them, right? And so I'd like to know, I'd like to know.

• 43:04Speaker 05

  Yeah, I think like what I would expect is like at least like one simple reproducible example of like.

• 43:12Speaker 05

  Why it blew up like a whole like little code snippet. So like, and this is something you brought up earlier is like, kind of like how we approach these type of things like if I'm like reviewing someone's code and I see something that looks weird. I will always do my best to make my own little sandbox and like prove my theory out and then like actually show them the code like this is why this is probably wrong. Here's like a small symbol reproducible step. So I would expect something like that. And that also helps me like truly understand. A lot of people, like you said, they'll, they'll see something like that. Looks funny. But I don't know why it looks funny, but I hate, I can't stop there. I gotta like actually like build it out and then like understand. So that's what I would expect.

• 43:49Speaker 01

  And you know, like, like I said, the, the CrowdStrike and the Google outages, I thought were better at like just telling you that they were like, look, it was a null pointer deref in here. Or it was an out of bounds array because we thought there's only going to be 20 and we put 21 in the config file, right? And like, okay, I know exactly what kind of code that, you know, is causing that kind of problem, right? And furthermore, furthermore, to like an earlier comment, literally as far as I know, everyone who programs in Rust only does it so that occasionally when they see something like this, they can say, well, if they'd had written it in Rust, it wouldn't have happened.

• 44:25Speaker 01

  They were not given enough information to even make that comment. They probably made it anyway, to be fair, but they were not given it. So you have to give one rule that should be followed in RCAs is you have to give Rustations enough information to, if they so chose, correctly say that it would have been prevented in Rust.

• 44:47Speaker 02

  True.

• 44:47Speaker 01

  And this, we do not have that. We do not know whether this would have been prevented in Rust. We have no idea. It probably wouldn't have, but we don't know.

• 44:56Speaker 02

  Well, Casey, we do have a pretty good chance because it's like probably would have never shipped. So it would have prevented it.

• 45:04Speaker 00

  True. We would have zero enactors because we would be designing set enactors. Yeah. I will say that there's something, CloudFlare does really good job at this as well. They like go in and show like a lot of lines of code and say like, this is exactly what's going on. This is, you know, even though the problem's up here, this is the line that exploded due to all these previous conditions. That was me making fun of Russ with the unwrap, which actually wasn't truly the problem. Uh, but you know, it's just like all these things kind of happen. So they, they do a really good job. I'm surprised at how poor of a job AWS has done for this one.

• 45:35Speaker 01

  Well, and the other thing too is it, it was one of those things where it now it makes me, so it makes me unnecessarily suspicious of you, right? When I read this, I'm like, are you hiding something? Did you not really figure out what the bug was? Like you talked all about this race condition, but even from your own presentation, I can tell the race condition really wasn't important. That was just, that was just what led to the record having been set to nothing, but who cares? Right? Like that's, that's like something that's nice to put in the RCA as like an explanation of why this bug occurred now as opposed to some other time, but it's not the bug. So it's weird to me. Like when I see an RCA that doesn't talk about the bug, now I'm suspicious, right? And unnecessarily so, because if you actually did find it, then just tell me and now I

• 46:16Speaker 01

  know you found it, right? So it's like, I think it also is a confidence boost for the people who are looking from the outside who want to know, can they trust this Dynamo DP thing? If it looks like you actually found the bug, I have a little more confidence in you. If it looks like you have no idea what the bug was, or don't seem to understand what the bug was, then I'm, that I'm more concerned. And so I think that's also another reason to do this in your state. It, it provides confidence to your customers.

• 46:40Speaker 02

  Okay, maybe that's where they fired Adam as an AWS hero too. Maybe it's all connected.

• 46:45Speaker 00

  Could be.

• 46:46Speaker 01

  They didn't want him exposing these dirty secrets.

• 46:48Speaker 02

  Yeah.

• 46:48Speaker 01

  He was, he knew too much. He knew too much.

• 46:52Speaker 00

  Could you give a, could you give a quick like three minute summary of the guitar shop? Like what that, what that was revealing? Because I'm trying to remember what it was. Cause it involved like a single point of failure guy who was out here for this failure as well.

• 47:06Speaker 01

  So I don't know how to reconcile the two things. Uh, and of course we have no idea, we have no idea if either are telling us the truth now, right? Because this was such a bad RCA, I have no idea if it's correct or not.

• 47:18Speaker 05

  But what was the password?

• 47:20Speaker 01

  Yes, the password was wishbone12, I think.

• 47:22Speaker 02

  There you go, always trying to kill me.

• 47:25Speaker 01

  That's my recollection, anyway.

• 47:28Speaker 01

  So, yeah, that story was that, that there was, there was a thing that was designed to copy configurations. And that thing had kind of gone rogue and could not be stopped. Like, it was just, like, it was just copying configurations totally incorrectly, and it needed to be, like, fixed or repaired or something. And we don't have any more information because it was an overheard conversation, right? And so, does that comport with this? Well, a little bit, because those enactors do sound like the kind of thing that would be running a configuration copy. But on the other hand, it's not really a configuration for machines. Like, a DNS entry is a DNS entry. It's not, it's not really a configuration.

• 48:11Speaker 01

  So, I would say the two stories don't line up that well.

• 48:16Speaker 01

  And so, that's another reason why I was kind of hoping that this RCA was a little bit more believable, because I wanted to know for sure that the story was false. And I still don't really know, based on how bad this RCA is.

• 48:26Speaker 02

  What if, what if the tool that the guy wrote to copy the configs is just literally the enactor? Like, they just productionized it, and like, they haven't changed it in seven years? That was kind of my, like, connecting the dots there, was he's like, "Guys, I wrote that as a way for me to test stuff in my local environment, and you just decided to make three enactors and put them next to each other in prod? I don't, how did this happen?"

• 48:52Speaker 01

  I do, I have a ton of questions.

• 48:55Speaker 00

  Alternatively, is it the rollback? Because that's the one that did the copying of, like, "Hey, here's the previous one", right? And so, I'm gonna copy the previous one, then it gets like this null issue going on. And it just, like, the script never encountered a null, it just goes rogue and starts writing over and over and over and over again, to where you can't.

• 49:13Speaker 00

  You can't do anything.

• 49:14Speaker 01

  I don't know. All I know is that like, as far as I can tell from their explanation, going only on what they were providing, I still just don't think the race condition's even relevant. Because again, literally, an accidental update to the Route 53 endpoint would have taken down all three enactors immediately. Because according to them, all that's required to stop them is if the endpoint points at an unresolvable name. That's all you need. And so if that's really true, literally, an operator typo could have taken all this down. No race condition necessary, right? And so again, the RCA just does not do a good job convincing me that you've talked about what the real bug was. Because I can think of so many ways that you could have triggered this exact same thing

• 49:58Speaker 01

  that don't involve this race condition that you spent the entire RCA telling me was "the bug", but I don't think it is.

• 50:07Speaker 02

  Well, we'd like to extend a formal invitation to Jeff Bezos. If you want to come in here and explain yourself.

• 50:12Speaker 00

  Uh, I believe it's Andy Jassy.

• 50:14Speaker 01

  Andy Jassy now.

• 50:15Speaker 00

  Yeah, Andy Jassy is the man you're looking for. Or we can take that.

• 50:19Speaker 02

  Wait, I'm going straight to the top.

• 50:21Speaker 04

  Jassy is a previous as well. He's not at AWS anymore. He's head of Amazon proper.

• 50:27Speaker 00

  Yeah, that's what we want. I mean, we had head of-

• 50:28Speaker 01

  We want the head, we want the president.

• 50:30Speaker 00

  Yeah, Bezos was previous head, now he's just chairman. So he's no longer day-to-day, you know, he's gallivanting. So we want someone more boots on the ground.

• 50:37Speaker 02

  I want a real chairman analysis, though.

• 50:40Speaker 01

  Let's get them both.

• 50:41Speaker 00

  I want the-

• 50:42Speaker 02

  That's an RCA, is it not?

• 50:43Speaker 00

  The Sunday chairman analysis, what's that?

• 50:47Speaker 02

  The real chairman analysis.

• 50:48Speaker 01

  Armchair, the armchairman analysis.

• 50:50Speaker 00

  The armchairman. That's what I'm looking for, the armchairman.

• 50:53Speaker 01

  Dude, why isn't armchairman a phrase? Like, I'm armchairman of the board.

• 50:58Speaker 02

  I'm pretty sure that's the guy that does the chips, right? The armchairman?

• 51:02Speaker 01

  The armchairman.

• 51:03Speaker 00

  I don't get the joke. I'm sure it's-

• 51:05Speaker 01

  ARM.

• 51:06Speaker 02

  Armed-

• 51:07Speaker 01

  Armchair risk machine, yeah, ARM.

• 51:09Speaker 00

  Oh my god.

• 51:11Speaker 02

  Got potato chips, I know trash is on the pod, but-

• 51:16Speaker 01

  Oh my god dude, wait a minute, wait a minute, wait a minute, wait a minute. This is our sponsor opportunity. Potato chip companies. Make a line of potato chips where the bags are labeled by actual chip. So it's like, this is a 9950 X3D chip. Like, the bag just has that on it. And it's specifically potato chips for developers.

• 51:45Speaker 02

  Casey, I have been trying to convince Sun Chips to sponsor us forever, because think about it. Sun!

• 51:53Speaker 01

  Sun Microchips!

• 51:55Speaker 02

  Yes, that's what I've been saying!

• 51:57Speaker 05

  Oh, I was not, I was not going on that path.

• 51:59Speaker 02

  It's a match made in heaven!

• 52:02Speaker 01

  Garden salsa!

• 52:07Speaker 01

  Garden, programmers garden salsa!

• 52:13Speaker 05

  Yes, Microchips, Sun Microchips, yes, yes, yes, yes, yes, yes, yes, yes.

• 52:18Speaker 02

  That's the one that we need. If we can get a Sun Chips sponsorship, that's the perfect one. It's a match made in heaven.

• 52:24Speaker 05

  It really is.

• 52:26Speaker 02

  But Casey, I've been tweeting at them for like 5 years.

• 52:28Speaker 01

  They don't care. They're just like, who is this guy? And why does he keep tweeting at us?

• 52:34Speaker 02

  I'll see if I can find the Sun Chips, my oldest Sun Chips tweet here. This is, this is May 28, 2001, okay, we're really digging back.

• 52:43Speaker 05

  2021.

• 52:44Speaker 01

  Dude, you are really working hard to bag this chip sponsor.

• 52:47Speaker 02

  Well, I was writing a lot of Lua at the time, so I thought that it would go nicely together. Lua, like, moon, sun, right, it's like, moon does the sun, we're echoing back to each other here. So that was a classic hashtag computer chips. There you go.

• 53:01Speaker 01

  Yeah!

• 53:05Speaker 05

  That's a new terminal line.

• 53:07Speaker 01

  No, it's...

• 53:08Speaker 02

  They were not a meme.

• 53:08Speaker 05

  Oh, they were.

• 53:09Speaker 01

  No, that was part of, that was part of the meme.

• 53:13Speaker 02

  They did not reply. They've never replied to me, they've never interacted with me.

• 53:17Speaker 01

  So disappointing.

• 53:18Speaker 02

  I know. So I'm still waiting, still waiting.

• 53:21Speaker 01

  So disappointing.

• 53:22Speaker 05

  I can't believe it.

• 53:23Speaker 02

  You like Sun Chips though, right?

• 53:25Speaker 05

  Harvest cheddar, specifically.

• 53:27Speaker 02

  But you're not a Garden salsa fan?

• 53:28Speaker 05

  We made a harvest cheddar, are you kidding me?

• 53:30Speaker 00

  Oh my god, hold on, hold on, hold on, hold on.

• 53:33Speaker 05

  I am, I am a fan. When we were at the, the tower, all those garden sauces, threw them to the side, disgusting. Wow.

• 53:44Speaker 00

  Oh.

• 53:44Speaker 02

  Guys, this changes everything.

• 53:47Speaker 05

  Nope.

• 53:47Speaker 00

  Yeah, Trash, you can't be a part of this sponsorship.

• 53:50Speaker 02

  I thought we had the same taste in snacks.

• 53:52Speaker 05

  Well, I ate all the Harvest Cheddars over in that tower.

• 53:54Speaker 02

  That's actually great, because then we make a good team.

• 53:56Speaker 05

  See, that's what I'm saying. You're like, ah, yin and yang.

• 53:59Speaker 00

  I, I actually enjoy French onion also.

• 54:02Speaker 05

  I don't think, what, is there a French house?

• 54:03Speaker 00

  Green pack, green packs French on, blue packs a ridge, which I'm not a fan of.

• 54:08Speaker 05

  It is a ridge.

• 54:09Speaker 02

  Original is not that good. Adam, are Sun Chips vegan?

• 54:12Speaker 04

  I have never had a Sun Chip in my life, I don't think.

• 54:15Speaker 05

  Are they vegan? I don't know if they're vegan.

• 54:17Speaker 00

  Leave his house now.

• 54:20Speaker 04

  They may not have any Ozarks, it may not be a vegan.

• 54:22Speaker 02

  They don't have them in Ozarks.

• 54:25Speaker 00

  Oh my goodness. Okay, well, I think this is probably actually a reasonable time to end at this point.

• 54:31Speaker 05

  Yeah. I can't believe Casey wrote backwards that whole time like that.

• 54:34Speaker 02

  That's what I was gonna ask.

• 54:35Speaker 01

  Yeah.

• 54:36Speaker 02

  You're not writing backwards.

• 54:37Speaker 01

  Yeah, but you get used to it.

• 54:39Speaker 02

  Adam, it goes like this, his right hand, he writes forwards, he trained his left hand to write backwards.

• 54:44Speaker 04

  That is not how, you're not doing it.

• 54:45Speaker 02

  Yes, so it's like right hand, he did it, he does it the right way. He cannot write his right hand backwards, but you can watch, he does his left hand, he does it backwards.

• 54:52Speaker 04

  Are you guys trolling?

• 54:53Speaker 00

  No.

• 54:54Speaker 02

  You just saw it happen.

• 54:55Speaker 04

  Casey's nodding yes. I spent most of the time you were talking.

• 54:58Speaker 01

  You'd use the left hand, that's how you do it. You just, because if you, if you aren't, so if you're ambidextrous, this would be hard. But if you're used to writing with your right hand, then when you train your left hand to write backwards, that just seems normal. Right? Because you don't have, there's not another thing. Yeah.

• 55:14Speaker 04

  But you are trolling. You don't actually write backwards?

• 55:16Speaker 01

  Yeah, we're, we're, yeah, we're trolling.

• 55:18Speaker 05

  Do you, what, do you think he would actually write, actually, I can kick up those seats?

• 55:21Speaker 01

  So many people ask me that, it's weird.

• 55:23Speaker 04

  I was watching so closely your hand, and I couldn't figure out if it could be flipped. It was like, no, I think that's the right way. Like, I'm trying to like stand where you would be standing, and like, I could not get that question out of my head.

• 55:34Speaker 05

  Adam starts writing on his fake, on.

• 55:36Speaker 02

  Yeah.

• 55:37Speaker 05

  Fake whiteboard.

• 55:39Speaker 02

  For writing backwards for six months and come out of a coma, I figured out how to do it.

• 55:46Speaker 00

  I don't know how you do it Casey, so it just seems magical to me.

• 55:50Speaker 05

  It's just the mirrored camera, right?

• 55:51Speaker 01

  Thank you Trash, yes.

• 55:53Speaker 04

  Okay.

• 55:53Speaker 00

  Well, no, no, but how do I see through it?

• 55:56Speaker 04

  Yeah.

• 55:57Speaker 00

  It's glass.

• 55:58Speaker 05

  Glass? What do you mean?

• 56:01Speaker 00

  Through the glass, right? Because if I'm writing on one side, you see it backwards. So if you just mirror it, does it actually come out correctly? I haven't, I haven't done the math to know.

• 56:08Speaker 04

  Ryan, cut the podcast.

• 56:11Speaker 01

  This is a rapid.

• 56:15Speaker 01

  Wait, no, wrong, false. What did we say at the start of this thing? If you're actually want to know, you ask questions.

• 56:25Speaker 04

  Great, okay.

• 56:25Speaker 01

  You don't just fricking pretend and Prime is doing the exact thing he should be doing, which is saying, I'm not sure, does that actually work?

• 56:32Speaker 00

  Yeah, because my only question is how many times do you reverse it, right? Because you're gonna, if I draw like this. Right? If I draw like this. But then it goes through, if you're looking at it from the other perspective, for you, yours, it's gonna be flipped, right? Uh, there you go. Flip this thing. It's gonna look like, it's gonna, oh, that's vertical.

• 56:52Speaker 04

  The glass was clearly between the camera and him.

• 56:55Speaker 00

  Yeah, the glass is clear, so that's backwards looking through it. And so then you re-flip it again? Does that work out? Okay, maybe. Yeah, sure. All right, I buy it.

• 57:04Speaker 01

  Yeah. But at the end, you're writing normal. If you were, if you were, like, you, you got there right now, you wrote the letter D, right?

• 57:09Speaker 00

  Yeah.

• 57:10Speaker 01

  Of course, unfortunately, that could be letter B, so you probably want to write a letter that's only goes one way around.

• 57:14Speaker 05

  It's a different lowercase.

• 57:15Speaker 01

  Right?

• 57:15Speaker 00

  Uppercase.

• 57:17Speaker 01

  Uh, there we go. So you write the letter Z, right? And you're, you're writing it this way, you're looking at it, so think of yourself as standing and you're looking at the board, you write the letter Z, right? Now, if you were to walk around behind your screen and look back at it, you'd see Z flipped, right?

• 57:33Speaker 04

  Yeah.

• 57:33Speaker 01

  That's a C. So all you have to do is just flip it one more time and it's correct.

• 57:37Speaker 00

  All right. Can I show you guys one thing that gets people always really riled up?

• 57:41Speaker 01

  Sure.

• 57:41Speaker 00

  See how I draw my S's?

• 57:42Speaker 01

  Yeah.

• 57:45Speaker 01

  What?

• 57:45Speaker 04

  That's right down?

• 57:46Speaker 01

  Bottom up S? Are you kidding me right now?

• 57:48Speaker 00

  But then look how beautiful they are.

• 57:50Speaker 04

  I mean...

• 57:51Speaker 01

  That's not a thing.

• 57:52Speaker 05

  No.

• 57:52Speaker 01

  It's like a serpent.

• 57:54Speaker 05

  No.

• 57:55Speaker 04

  They...

• 57:56Speaker 02

  You can tell when it's done that it's wrong.

• 57:59Speaker 05

  Wha... Your Z? Do you actually write your Z like that too?

• 58:02Speaker 02

  I write my Z that way.

• 58:04Speaker 00

  Yeah, uh, sorry Matt, uh, sorry Trash, we did math. So we cross our sevens.

• 58:08Speaker 01

  Yeah.

• 58:09Speaker 05

  Okay, I do, I do my, I do my sevens like that, I don't do my...

• 58:12Speaker 01

  It's called Zed by the way, but sure, whatever.

• 58:14Speaker 05

  Oh dang!

• 58:17Speaker 01

  Oh, alright, got it, got it.

• 58:20Speaker 00

  That's fine. Uh, also, can I, can I, can I show you guys something?

• 58:23Speaker 01

  Mm-hmm.

• 58:24Speaker 00

  I have a very magical ability, are you ready for this one?

• 58:27Speaker 01

  Yes.

• 58:29Speaker 00

  I drew an ampersand first try, without fumbling it. It's considered pretty impossible by many people's standards when they first draw an ampersand, it's very difficult.

• 58:37Speaker 02

  Also, can you zoom in on the line? Looks pretty good, from where I'm at.

• 58:41Speaker 00

  Almost a perfect line right there too.

• 58:42Speaker 05

  That's computer enhanced.

• 58:44Speaker 00

  No, you can see that I, I goofed it right here, and I goofed it right here, you can see the two spots that I goofed.

• 58:49Speaker 02

  Enhance, computer, enhance!

• 58:50Speaker 05

  Yeah, there you go.

• 58:52Speaker 00

  That's why you gotta go to computerenhance.com.

• 58:56Speaker 05

  The way you write your S, that has to slow you down so much.

• 58:59Speaker 00

  Uh, I just decided to write really nice instead. I don't care, I write, I don't care that I write...

• 59:04Speaker 02

  Why would you draw a clown, Trash? Can you explain that to me?

• 59:06Speaker 00

  Yeah.

• 59:06Speaker 05

  I don't know. It's just, to me, it was just...

• 59:10Speaker 04

  Isn't that the same amount of writing?

• 59:11Speaker 00

  But Trash, is that why you hold your pencil?

• 59:14Speaker 05

  Like this? Can you do that?

• 59:15Speaker 00

  No, no, no, how do you write?

• 59:17Speaker 05

  No, that's like this.

• 59:17Speaker 00

  Okay, well, that's, there's your problem, you hold it like a weirdo.

• 59:22Speaker 02

  Dude, we were just talking about drum line.

• 59:24Speaker 05

  What, do you hold it like this?

• 59:25Speaker 00

  Yeah, like a proper, yeah.

• 59:27Speaker 04

  Neither of those, no. That's not how I hold it either.

• 59:30Speaker 02

  What are pencils and pens for?

• 59:31Speaker 04

  There's too many fingers involved.

• 59:34Speaker 00

  Right, right on top, boom. That's how you hold it.

• 59:37Speaker 05

  Can you guys do this? Can you guys do this?

• 59:40Speaker 02

  I can do that.

• 59:43Speaker 01

  That's the, okay.

• 59:44Speaker 02

  Oh, have you seen this?

• 59:45Speaker 01

  Please make that be the opening. Can you guys do this? And it goes down. And then you cut to the start of the podcast.

• 59:53Speaker 02

  Trash, you know what that reminds me of? I was, I was at my, I was at my friend's house. We were all sledding together with us and our kids. His, uh, his son goes down the hill and he goes, "And this is how a 13-year-old sleds." He jumps down the hill, immediately gets the sled caught down, face first into the snow and just biffs into the snow.

• 1:00:14Speaker 05

  He start crying?

• 1:00:15Speaker 02

  So now every time I see him at church, I'm like, "And this is how a 31-year-old runs."

• 1:00:22Speaker 05

  That is so mean.

• 1:00:24Speaker 02

  He thinks it's hilarious. He cracks up every time.

• 1:00:26Speaker 00

  All right, Trash, I'm gonna do one more little trick for you. Okay, are you ready?

• 1:00:29Speaker 02

  Yeah, okay.

• 1:00:30Speaker 00

  All right, you gotta hold a pencil like this. All right, and then you, without taking your hands off the pencil, okay?

• 1:00:35Speaker 02

  We did this one.

• 1:00:36Speaker 00

  Yeah, you gotta end like that.

• 1:00:40Speaker 02

  Ah, I knew the answer at one point.

• 1:00:44Speaker 00

  Okay, you just, you just, just don't take your hands off of it.

• 1:00:48Speaker 02

  I can't.

• 1:00:52Speaker 00

  Dude, that, that would-

• 1:00:52Speaker 02

  Do it again, do it again, do it again, do it again.

• 1:00:54Speaker 00

  That's the one that, okay, it's very simple. Okay, so you put it right here, right? And then you gotta turn it, and then you just put your hands the other way.

• 1:01:03Speaker 02

  It's like three pixels on my side.

• 1:01:06Speaker 01

  Yeah, it's very bad, the video quality. It just doesn't even look like it is a pencil.

• 1:01:09Speaker 05

  Whatever, can you do this? Boom.

• 1:01:16Speaker 05

  But you can't do that. Can you, can you spill blood with your hand?

• 1:01:18Speaker 00

  Okay.

• 1:01:20Speaker 01

  Trash, bro, you're gonna get this podcast.

• 1:01:24Speaker 00

  All right, Trash, all right, Trash.

• 1:01:26Speaker 05

  Can you Fortnite dance? Probably not.

• 1:01:28Speaker 00

  Yeah, okay.

• 1:01:30Speaker 00

  No, please don't, Trash. Trash, you don't do this. You don't wanna do this, Trash. I'm doing it for you.

• 1:01:37Speaker 00

  This is just baiting me. No, no, no, no, I'm good, I'm good. If you hit an orange justice on this podcast, you will literally never stop orange justicing from here until the universe ends.

• 1:01:48Speaker 00

  No, we're good. End it. Put a plug. Thank you, everybody, for watching. I'm terribly sorry about this. Top 50 tech podcast. Yeah, I showed off.

• 1:01:56Speaker 01

  We got out of this one. We didn't even have to show a nipple, so that was a good effort.

• 1:02:00Speaker 00

  Can you believe I told him to go to Spotify for this? Yeah.