949: Web Dev HORROR Stories + Spooky Trivia! (Spooky Stories Pt. 1)

It’s that time of year again, Scott (as Dracula) and Wes (as a big bad shark) return for their annual Spooky Stories special! They’re joined by a mysterious guest for a round of creepy coding trivia and chilling true tales of web dev gone wrong; dropped databases, haunted passwords, and more. Beware: these are real developer horror stories. Show Notes 00:00 Welcome to Syntax! 00:05 Scott’s vampire costume. 00:38 New to live-streaming. 00:54 AWS Outage. 01:50 What is Spooky Stories? 02:01 (Wes is wearing a shark costume) 02:34 Brought to you by Sentry.io. 03:30 PHP MyHeadache. 06:23 A Browser Hack. 08:30 Browser Hack Gameshow & Special Guest! 17:01 Ghost Password. 21:32 Just A Patch. 24:12 Don’t drink and Deploy. 34:47 Discount. Hit us up on Socials! Syntax: X Instagram Tiktok LinkedIn Threads Wes: X Instagram Tiktok LinkedIn Threads Scott: X Instagram Tiktok LinkedIn Threads Randy: X Instagram YouTube Threads

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Wes Bos?

  I think we're live.

• 0:02Wes Bos?

  Yep, it's being pulled in. What's up, everybody?

• 0:05Scott Tolinski?

  I'll tell you what, man, the makeup, it's a little tough. All the things are a little tough. I gotta get used to this. I spent all morning doing the makeup. I'm just rubbing it on my head and then trying to blend it. And then I would add too much black and then it's like, what you learn is that it's hard to go back when you add too much black to the mix. So then you're like.

• 0:27Scott Tolinski?

  And then on top of it, I gotta wear glasses. So I'm doing the whole thing. I look like this, which are a bit better. But when you put the glasses on, it's hilarious.

• 0:35Wes Bos?

  Oh, my goodness. That is so good. All right. Just tweeted it out. Let me get it.

• 0:40Scott Tolinski?

  Your title says at S at syntax.

• 0:46Wes Bos?

  Oh, I see. I see. Yeah, it's fine. That's fine. That's a spooky story in itself.

• 0:53Scott Tolinski?

  That's a spooky story in itself. Speaking of spooky stories, I'm happy that Riverside is even up today. Uh, given that Amazon screwed the pooch. I was wanting to test some stuff yesterday on Riverside and I couldn't get it to load.

• 1:08Wes Bos?

  Man, I, I think I missed that whole outage. All I saw was everybody talking about it. I think I was just happily doing my local dev and didn't see a thing about it. But uh, it seemed like a major outage.

• 1:18Scott Tolinski?

  Didn't affect any of my stuff. It might've affected syntax.fm.

• 1:22Wes Bos?

  Yeah, yeah, definitely did. We're hosted on Versal, which is hosted on AWS. So the whole, the whole, uh, the whole internet went down and the funniest parts were people tweeting about their like internet connected beds.

• 1:37Scott Tolinski?

  Oh, yeah.

• 1:38Wes Bos?

  It's just the, the future of us living in a place where your bed stops working if the internet goes down. Which is, is absolutely hilarious. But uh, people, people are, are piling in. Welcome, everybody. So we do this episode every single year where we do spooky development stories. These are stories that make you want to stick your hammerhead in the ground. And you, people submit them every year. If, if you have not submitted yours, please go to syntax.fm. Spooky and on probably next year we'll read it, but we collect them all year round and, and we read them off once every year. So I'm excited to get into it. How are you doing today, Scott?

• 2:20Scott Tolinski?

  Oh, I am simply dead. I, I could not go outside in the sun. So I, I'm having a great time today. In fact, in my nice little, little cave here. But folks, before we do that, let's talk about something that doesn't suck. Let's talk about Sentry. Sentry.io, the perfect place to fix all of your blood curdling, bone chilling problems on your applications. Yes, we all know that these things pop up all the time and it can simply make us scream. So what you want to do is head on over to Sentry.io forward slash syntax, sign up and get two months for free. You can fix

• 2:59Scott Tolinski?

  all of your bugs in production, all of your bugs in development, and you can do so with like the best tools in the entire world. And I say that as somebody who uses these tools every single day. So shout out to Sentry for, for presenting this podcast.

• 3:16Wes Bos?

  You know that probably about four or five of these spooky stories without our prompting have had, this probably wouldn't have been an issue if we had Sentry enabled.

• 3:26Scott Tolinski?

  Yeah, right. Yeah.

• 3:28Wes Bos?

  Those will probably write themselves, but you want to grab the first one? Let's get on into it.

• 3:33Scott Tolinski?

  Let's grab the first one. The PHP My Headache. On my very first job, we had a state of the art tech stack, PHP backend, MySQL database, embrace yourself, PHP my admin in production. We used it for everything, browsing tables, running queries, breaking things accidentally, you name it. Oh, the last of the days of PHP my, my admin.

• 3:56Wes Bos?

  PHP my admin is like the OG SAS, you know?

• 3:59Scott Tolinski?

  OG SAS.

• 4:00Wes Bos?

  If you don't want to, or not the OG SAS, the OG like CMS.

• 4:03Scott Tolinski?

  CMS backend. Yeah.

• 4:04Wes Bos?

  Yeah.

• 4:06Scott Tolinski?

  One fine day, I decided to refresh my local database with a copy from production. Yeah, I know, solid DevOps. I opened PHP My Admin and I clicked the big red, drop that database button, and got the classic warning, "Are you sure you want to do this?" "Hell yeah, I'm sure," I said confidently, "Click." Two seconds later and our Slack incendiary exploded like a fireworks show of despair. "Table does not exist, query fail, everything is on fire." Uh, my soul left my body, I had been logged into production, not local. I had just deleted the entire production database. "Hey, the site's down." "Yeah," I said with a forced smile, "Working on it."

• 4:50Scott Tolinski?

  Meanwhile, one of my vacationing coworkers messaged me, probably from a beach somewhere. "Hey, what the hell is happening? My phone is blowing up." "Nothing to worry about," I lied through my teeth. Miraculously, I had a five-year-old MySQL dump on my-

• 5:05Wes Bos?

  Five minutes, not a year.

• 5:06Scott Tolinski?

  Five-year-old, yeah.

• 5:07Wes Bos?

  Major difference.

• 5:09Scott Tolinski?

  Major difference there, a five-minute-old MySQL dump on my laptop. I imported it back into production like my life depended on it, which in sense, it did. Within minutes, the site was back online. I leaned back, took a deep breath and told my coworkers, "Everything's fine now." Nobody ever knew how close we came to digital extinction that day. Just imagine, you do not have a five-minute-old backup here.

• 5:37Wes Bos?

  Uh, it's wild that so many of these stories are people that don't have automated backups in place. They don't have any sort of guardrails where you can just YOLO PHP My Admin in production. At the very least, change the background of that production PHP My Admin to be like red or something. If you're gonna have scary places-

• 5:58Scott Tolinski?

  Yeah, no kidding.

• 5:59Wes Bos?

  -have a few guardrails in place.

• 6:01Scott Tolinski?

  Okay, how many, how many shaking bones do you give this one?

• 6:06Wes Bos?

  Oh, man, I'm gonna give that a, a six out of seven, for sure.

• 6:10Scott Tolinski?

  Six out of seven, I'll give it a six, I'll give it a five out of seven. If this didn't have a backup, you could go higher on this one. But definitely one of those stomach in your, pity your stomach moments, for sure.

• 6:23Wes Bos?

  Next one's called a browser hack. I worked for a small, literal mom and pop shop. Wife was the programmer, husband was the designer and head of company. As my first job out of college and it's had its own proprietary CMS e-commerce platform, the class of LampStack. I was a jack of all trades doing all the technical stuff from programming the front end, design work to the database and server administration. We hosted hundreds of our clients' websites on a server we rented. This was before the cloud was big and it was obvious from my comment about the LampStack. Yeah, so many of these, these horror stories come from just shoddy PHP LampStack.

• 7:00Scott Tolinski?

  Yeah.

• 7:01Wes Bos?

  Not that it's a bad programming language, but just like the, the like hygiene around being safe with production was nothing. Everyone's just FTPing and working in production. You're just YOLO-ing all. One day, our phones started ringing off the hook with a bunch of customers saying their websites weren't coming up. For us in the office, though, the sites were coming up fine. There was an utter dread for maybe half an hour, which felt like an eternity as we were struggling to figure out what was going on. Eventually, we determined that it was affecting our users below a certain version of Internet Explorer. Remember those days? After investigating further, I eventually realized I had accidentally

• 7:42Wes Bos?

  saved a semicolon at the very beginning of a PHP file that was a configuration shared by every site on the server.

• 7:51Scott Tolinski?

  Uh huh.

• 7:53Wes Bos?

  As a result, a semicolon at the very beginning of the HTML for every website before the HTML tag, most browsers completely ignored it. Yeah, browsers are super forgiving about malformed markup, but of course IE threw a fit about it. Fortunately, it ended up being an easy fix. We figured out what was going on. I am forever grateful. I don't get nightmares from having to support Internet Explorer. Literally, a semicolon took down their entire empire.

• 8:19Scott Tolinski?

  Oh.

• 8:20Wes Bos?

  Five trumps out of five.

• 8:23Scott Tolinski?

  Yeah. Oh, five trumps. Yeah. I'm gonna give this a pretty scary stuff here. This will also be five for me.

• 8:35Scott Tolinski?

  Who's here? I think I just heard a doorbell. Wes, did you hear something?

• 8:41Wes Bos?

  Yes, who's here?

• 8:43Scott Tolinski?

  Uh, maybe you should check the, uh, the waiting room.

• 8:48Wes Bos?

  Oh, are you kidding? I didn't even know about this. No chance. Please let him be dressed up. Yo, ladies! Oh! No chance! No chance!

• 9:02Scott Tolinski?

  CJ!

• 9:04Speaker 02

  My computer was frozen for a second. Hello, hello. How are you doing?

• 9:09Wes Bos?

  Do you just casually have a pirate costume on?

• 9:10Scott Tolinski?

  Wait, what is that? I hear something else. What's going on here? Let's wait a second. What is that? What?

• 9:24Scott Tolinski?

  I was working in my code late last night when my eyes beheld a creepy sight. For my code was working with no reason why. And then I saw it, to my surprise.

• 9:40Scott Tolinski?

  It was a browser hack.

• 9:43Scott Tolinski?

  With Macromedia Flash. Oh my gosh. It really saved my ass. It was a browser hack.

• 9:56Scott Tolinski?

  Gentlemen, uh, boys and ghouls, it's that spooky time of year. And in this spooky time of year, anything can happen. So I've invited our friend CJ over for a bit of trick or treat. And see, we all know that a well timed browser hack can really, really save our ass. So I've prepared for you a soul sucking game where we have eight browser hacks. You all have to tell me, is this a real browser hack? Or did I make it up? The first browser hack. The backslash nine trick.

• 10:33Scott Tolinski?

  This targets IE 5.5 and IE6 specifically by doing a backslash nine at the end of a CSS property.

• 10:43Wes Bos?

  Uh, do we both get to guess?

• 10:45Scott Tolinski?

  You both have to guess.

• 10:47Wes Bos?

  That, I'm gonna say that's a legit one. There was backslash. I don't know if nine was the one.

• 10:52Speaker 02

  I can't hear Wes for whatever reason. Riverside is messed up.

• 10:54Wes Bos?

  But, can you hear me, Scott?

• 10:56Scott Tolinski?

  I can. CJ cannot.

• 10:58Speaker 02

  Yeah.

• 10:58Scott Tolinski?

  I'm sorry. I, if you can hear me, CJ?

• 11:00Speaker 02

  Yeah, I can hear you, Scott.

• 11:02Scott Tolinski?

  I will tell you when you should talk. Wes says this is a real hack. CJ, what do you say?

• 11:07Speaker 02

  I don't think it's a real hack. I say no.

• 11:09Scott Tolinski?

  CJ says fake. It is fake.

• 11:10Wes Bos?

  Yes. Ohhh. But there was one, wasn't there? There was like a weird IE 5 and 6 one. Maybe.

• 11:20Scott Tolinski?

  Maybe. Uh, JS and CSS. You can write JavaScript inside of CSS to extend CSS's capabilities using the expression function.

• 11:31Wes Bos?

  That's fake.

• 11:33Scott Tolinski?

  Wes says fake. CJ, what do you, what say you?

• 11:36Speaker 02

  I mean, this seems really useful. Like, maybe it existed somewhere and then didn't become a standard. I'm just, I'm gonna say real. Real.

• 11:43Scott Tolinski?

  This is real.

• 11:45Speaker 02

  No way.

• 11:45Scott Tolinski?

  This is a, in all versions of Internet Explorer, 5.5 and 4, apparently, uh, you can, this blog post is really interesting. Apparently, it worked really well. It was just slow as hell. So, uh, people didn't use it. Um, real.

• 12:00Speaker 02

  Fascinating.

• 12:01Scott Tolinski?

  All right. Number three. CJ, two for two. Wes, oh for two. Conditional comments used to deliver HTML or CSS to different versions of IE if LTIE7.

• 12:14Wes Bos?

  This is real. Um, this was the move in, uh, HTML5 boilerplate. You had it and it added those conditional classes to the HTML element?

• 12:25Scott Tolinski?

  Yes. Uh, CJ, what say you?

• 12:28Speaker 02

  I believe it's real. I saw Wes saying a whole bunch of stuff, but yeah, I've, I've used this in the past. It's real.

• 12:33Scott Tolinski?

  It is of course real. For those of you who never worked in IE, this magic actually worked. And we used it all the time for various stuff, even like banners that said, "Get the hell off this browser." All right. Number four, the ping fix for IE6. A shim a PNG alpha in IE6 by replacing images using Microsoft Filter ProDX Image Transform Microsoft Alpha Image Loader. Wes?

• 13:00Wes Bos?

  This is very much real. I actually went down a rabbit hole several years ago to figure out, like, how did that actually work? To me, it was magic at the time that it figured out how to just punch out all the alpha pixels of a PNG, or all the, I guess, white pixels, but it's now, it's a, now I know how it works. It's very real.

• 13:22Scott Tolinski?

  CJ, what about you? What say you?

• 13:24Speaker 02

  Okay, I didn't hear what Wes said. He said a lot. I'm gonna say fake just because the colon in the syntax seems like that'd be hard to parse.

• 13:31Scott Tolinski?

  It's real!

• 13:32Speaker 02

  It's real! Whoa, whoa, okay.

• 13:33Scott Tolinski?

  Yep. Here's a CSS tricks article for it. PNG hacks from IE6. I think this article's from like 2009 or something. This is a blast from the past, for sure. All right. The comment float fix. This stops IE5 Mac from double float margins because it forced the parser into a Phantom Quirks neutral mode. Wes?

• 13:55Wes Bos?

  That, you tried to make it sound like that's actually something, but IE5 was not on the Mac. There was an IE version on the Mac at one point, and this is, no, I've never seen this slash thing ever. This is fake, fake, fake, all the way down.

• 14:09Scott Tolinski?

  CJ?

• 14:11Scott Tolinski?

  Real.

• 14:13Scott Tolinski?

  CJ says real. It is fake!

• 14:18Scott Tolinski?

  The margin, I made that up. Margin collapse fix. This prevents IE6 from collapsing margins, which collapsing margins when you have two margins that butt up against each other. Wes? Real or fake?

• 14:31Wes Bos?

  Margin collapse fix. Margin collapse is a thing, but the code you have is, that's, that's not real. This is, that's fake.

• 14:41Scott Tolinski?

  Wes says fake, CJ.

• 14:42Speaker 02

  I'm gonna say fake too.

• 14:44Wes Bos?

  Oh, prevent collapsing. Okay, good.

• 14:46Scott Tolinski?

  The star and the underscore hacks. You can use the star or the underscore to target specific browsers because I suppose it would be whatever, the progressive fallback there.

• 14:59Wes Bos?

  Okay, my, this is real all the time. My old IE knowledge is immaculate and these are, these are very much real. I've very much written many of those in my career.

• 15:09Scott Tolinski?

  CJ, Wes says real. What say you?

• 15:12Speaker 02

  I now believe the syntax can be anything, so I'm gonna say real too.

• 15:17Scott Tolinski?

  It is of course real. Wes is mad that I'm telling you what his answer was.

• 15:22Speaker 02

  If I could hear his mic, I would know what he said.

• 15:25Wes Bos?

  Well, you can't.

• 15:27Scott Tolinski?

  Okay. IE float double margin fix. When you float an element in IE, it doubles the margin. So this is, that's not the, that's not the real or fake part. IE actually does that. So if you do float left and then margin left, it will do 40 pixels. So the fix would be throwing display inline to fix it. Real or fake, Wes?

• 15:52Wes Bos?

  Wow. That sounds like something that would be real and putting something inline also sounds like the move because probably one of those margins is from it being boxed and the extra one is because of it being floated. So I'm gonna say real.

• 16:09Scott Tolinski?

  Wes says real. CJ, what do you say?

• 16:11Speaker 02

  Stop telling them what the answer is.

• 16:13Scott Tolinski?

  But CJ says fake. It is real. Yep, you gotta do that. And thank you for joining us for Spooky Stories, real or fake?

• 16:25Wes Bos?

  You guys like Halloween way too much. Like, not, not too much, but like such impressive costumes. And mine is just like, I just grabbed one from the kids' bin like 10 minutes before this started.

• 16:41Scott Tolinski?

  CJ, thank you for joining us. You're more than welcome to stick around or go off and do whatever you have to do for the rest of the day.

• 16:47Speaker 02

  I got some booty to be plundering, I guess. I don't know.

• 16:51Scott Tolinski?

  Yes.

• 16:52Wes Bos?

  Oh CJ, this is a family show.

• 16:56Scott Tolinski?

  Oh, I believe I also have some booty to be plundering. Okay, so let's get into the next one, Ghost Password. So I'm a maintainer of a fairly popular NPM package. Nothing huge, but it recently crossed the 2 million monthly downloads mark. Nothing huge, 2 million monthly. That's huge. Give yourself a round of applause, of which I was super proud of. For a long time, I was the only dev behind it until a few months ago. I even wrote in with a potluck question about how to start sharing ownership. I do remember that. The vampires never forget. Then about a month ago, the spooky part started. I'm

• 17:36Scott Tolinski?

  on my phone and OnePassword suddenly stopped authenticating me. You know, Face ID fails and it asks for the master password. I type in my master password, denied. Oh, man. That's the worst. Um, okay, weird typo. Tried it again, denied. Little not forms in my stomach. The next day, I tried on my Mac, same thing. Now I'm officially locked out. I'm thinking, okay, don't panic, I'll just contact support, spend a few hours going back and forth with OnePassword support team, but they're firm. There is nothing they can do on their end to help me recover the account. Is that serious, OnePassword? Oh, man. And here's the kicker,

• 18:20Scott Tolinski?

  the part that felt like a line from a horror movie. They told me, sir, we can confirm from our logs that your master password has not been changed in five years. Woo. It was like being locked out of your own house with a butler telling me the key that I've used every day for five years is the wrong one. I eventually had to give up. I started the soul-crushing process of resetting every single password for every account I own. Oh, my God. In the end, I managed to reset every single account except for NPM. Due to their security policy, an account cannot be recovered without the two FA codes or the recovery codes. They tell you to save those recovery codes for a reason. Wes?

• 19:03Wes Bos?

  Yeah.

• 19:04Scott Tolinski?

  Do you save your recovery codes?

• 19:05Wes Bos?

  Yes. Like, there's, man.

• 19:09Wes Bos?

  You should, especially your OnePassword recovery codes, or like your password manager recovery codes, you need to store those in the first place, and you need to store them somewhere safe, because you are screwed if you lose those.

• 19:25Scott Tolinski?

  So with extreme regret, I had to say goodbye to my NPM account and package I built from the ground up. No!

• 19:31Wes Bos?

  No!

• 19:32Scott Tolinski?

  An open source project with billions of downloads and now orphaned because of a password I thought I knew. I feel like... I feel like you're just typing in that password a little bit wrong, don't you?

• 19:41Wes Bos?

  Yeah, did you try, see if caps lock was on, maybe?

• 19:45Scott Tolinski?

  Yeah. Or did you try from another phone or computer or something? Because like, I know me, if I'm on, let's say, a new version of macOS, something that might now always work the right way. I was on like helium browser and my Reddit login didn't work for two days, but I didn't change the password. I just tried it again in two days and it worked, so.

• 20:04Wes Bos?

  It makes me wonder if this guy's, if something else spooky is going on. You know, like the guy's computer has been compromised, because like, everybody right now wants to get ahold of NPM packages. And one with 2 million, if you got an NPM package with 2 million monthly installs, you're certainly being targeted, so I would probably wipe your computer, because something funky is, is being put in between you and the sign in there. I think, I think that's what's happening.

• 20:34Scott Tolinski?

  Yes.

• 20:34Wes Bos?

  Yikes.

• 20:35Scott Tolinski?

  This was very spooky.

• 20:36Wes Bos?

  Send us the name of it, because yeah, that's, that's odd.

• 20:39Scott Tolinski?

  Yeah.

• 20:40Wes Bos?

  But also like, props to OnePassword for not giving anybody, random people, access to your account. I would not ever want them to be like, "Oh, sorry, I forgot it." That's the whole point of, of backup codes and, and all these awful... There's, there's many layers of, of OnePassword if you, if you get locked out. And if you can't get them all, you're probably a hacker. And unfortunately, that means some good people get caught in the toine of it because they had bad password hygiene, but it's worth it.

• 21:09Scott Tolinski?

  Word. Uh, how many spooky, uh, bats do you give this one?

• 21:13Wes Bos?

  That one's more of like a, a sad one. I'm gonna get that four out of seven spooky bats.

• 21:20Scott Tolinski?

  This one's eight spooky bats for me, cause it's very spooky, sad or not. Like sometime, sometimes Wes, things that are, are, are sad or spooky too.

• 21:29Wes Bos?

  That's true. Very. Yeah, that's, that's deep. Next one we have here, just a patch. When I started web development, I did not know everything about how SQL patches were made. There were a few requests that I worked on, bugs and new features worked fine for me. And then I was asked to patch it up to prod. I had new tables and some extra columns on existing tables. So I thought it would patch it all up at once. So I generated a script for every object in the database, not to miss anything. Yes. Drop, create scripts. The boss ran it on the prod DB and then he asked me why every table was empty.

• 22:06Wes Bos?

  That is when I realized what I did. Fortunately, there was a fresh DB backup and it was fixed immediately. Man, awful. There, there's so many of these stories where people just YOLO some script, like at least run it. Pull your data. If it's, if your database, database is not so big, pull it locally and just run it. Just, just to check or, or, or a lot of like ORMs will have like a, like a test mode where it will tell you how many things it will update.

• 22:35Scott Tolinski?

  Yeah. What are the dry run or something like that? Yeah. Uh, very, uh, very common. And, and I, are you one of those people? Sometimes I see the dry run option and I'm like, it's all good. I got this.

• 22:48Wes Bos?

  Yeah. Well, I, I've been there as well. Even just, um, my home assistant the other day, I needed to like update it. And I was like, should I back up my config? I was like, ah, it'd be fine. It wasn't fine. It, I, I had everything go missing. Luckily I had a backup from, I don't know, a couple of months ago, but always back up before you do anything where you say, it'll be fine.

• 23:10Scott Tolinski?

  Yeah, always do that. Hey Wes, this next one is a little bit long. I don't know if you've scrolled. But it, it goes on and on. So, uh, you think we should just read this in its entirety?

• 23:21Wes Bos?

  Is it? Did you read this one already?

• 23:23Scott Tolinski?

  Yes, it is a spooky.

• 23:25Wes Bos?

  For the next show, I read them all to make sure they were all good. Is it worth it?

• 23:29Scott Tolinski?

  Is it worth it? Sure. Let's do it.

• 23:31Wes Bos?

  Yep. All right.

• 23:32Scott Tolinski?

  This is a true story.

• 23:33Wes Bos?

  Recording. If it's too long, we'll cut it out.

• 23:36Scott Tolinski?

  Folks, you're getting the live feed here.

• 23:38Wes Bos?

  Yeah.

• 23:38Scott Tolinski?

  The live unedited version of this spooky stories. Wes, I didn't hit you with any spooky voice today.

• 23:48Wes Bos?

  Oh.

• 23:49Scott Tolinski?

  Does that, does that make you happy?

• 23:51Wes Bos?

  Yeah. Oh, I can tell you I'm getting really hot in this hammerhead shark costume. My wife's always talking about natural fibers.

• 23:58Scott Tolinski?

  Yeah.

• 23:59Wes Bos?

  And like, don't wear plastic, folks. This is so hot.

• 24:02Scott Tolinski?

  Oh, yeah. For JS Nation, we wore space costumes and they were like, it was like literally wearing plastic bags. I was just absolutely dying in it the whole time. Okay. This is a true story and it happened in 2019, about five months before the pandemic hit. It is a catastrophic production failure that I'm 100% guilty. Keep in mind, I was still under six months probation inflicted upon new hire. So this was a new hire, then there are six months, uh, and they have committed a crime. The company I was working for at the time was releasing to a new market. With a new market comes new payment options. Of course, we wanted to allow every single payment method in every single county, right?

• 24:46Scott Tolinski?

  So we had multiple conditions to build our final list of available payment methods for each particular customer. That check was done in FE, in, in the FE. What is the FE? What, why am I blanking?

• 25:00Wes Bos?

  Finance? Front end? No, front end.

• 25:03Scott Tolinski?

  No, no.

• 25:04Wes Bos?

  Yeah, of course.

• 25:06Scott Tolinski?

  That check was done in the front end, my domain. While the implementation of the new payment method was done by a colleague in the payments team, I'll call him Steve. He was a newer hire than I was. During the whole thing, no one else could really help us since every single senior there was too busy with some other part of the new market release. It was a whole company effort, the miracle releasing to a new market in a couple of weeks instead of months. It was me and Steve figuring out our way through microservices. We only had each other to rely on, so we pair coded most of it. Just to add some spice to my life, the requirements kept changing. We were having issues with PayPal and Visa. So for around a little over a month, I'd get different requirements and conditions

• 25:48Scott Tolinski?

  for which payment method should be available every other day.

• 25:52Wes Bos?

  Oh, this is a nightmare.

• 25:54Scott Tolinski?

  It's a nightmare.

• 25:54Wes Bos?

  I hate working with this stuff, like. People always complain about Stripe and PayPal and it's, it's the reason people hate them is because they grow so large that it is impossible to figure all of this stuff out and keep it all straight as to what's available in which country.

• 26:09Scott Tolinski?

  The conditions were also fairly complicated, depending not only on the customer country, but also how many purchases the customer had done in the past. Whether it was a high ranking customer, whether or not they had submitted documents before, whether or not the customer was considered validated. Since every time a new payment method was added, the norm had been touch as little as possible. The conditional tree was an absolute mess, with many functions scattered through the utilities file, some of which had one line in them to call yet another function. Bro, we have all been there. At some point, me and my partner in crime, Steve, couldn't tell if the new payment method was showing or not under the correct conditions because it was held just to figure out what the conditions

• 26:53Scott Tolinski?

  for the requirement sheets were already added and correct or not. Steve and I decided to refactor the whole thing. Beautiful.

• 27:04Wes Bos?

  Can we, can we stop there for a second? The, like, the legacy code that had been tweaked year, over and over and over. It's every little edge case possible. It is almost never a good idea to just scrap it and start from scratch unless you have an extremely well thought out test suite covering all of these, which it doesn't sound like they did.

• 27:27Scott Tolinski?

  Even for like markup, I, I had a, back in, back in the day, I had an old HTML site that had like, you know, 25 files in each directory that was like index forward slash hyphen hyphen new 20, you know, whatever. And, and you'd update something, you'd be like, why is this where I was not using the right one or whatever? And I decided to refactor that and that was a nightmare too. And that was just straight up HTML and CSS. Okay, we looked at each other and say at the same time, "Screw it." We yanked out the whole component, gutted the so-called conditional tree. We burned the utilities file to the ground and we added tests when there were none. Days of pair programming, blood, sweat and tears. And all that's left is to confirm in the staging server.

• 28:10Scott Tolinski?

  I have cobwebs on my mouse pad and it keeps messing up my scroll. I'm like going like this every single time. Then I lose my spot because this one's so long. Okay, we hit a couple of simple problems. First, we couldn't properly validate test customers. So we set it to manual. We use a forced response that had to be configured in certain test API. Everyone else was also testing stuff in that same staging server. And forced validation was getting in the way of the customer validation API test. Again, Steve looks at me like he's saying, "We are nice to everyone who's going to be nice to us." Again, we go, "Screw it." We hardcoded the customer is validated in the front end.

• 28:54Scott Tolinski?

  Hardcoded that the customer is validated. Fine, sure, they're all validated. This is the first fork in the validation tree. And if the customer was not validated, no payment method is available. We check, double check, triple check every situation, every country, every possibility in that FSM is accounted for. And we see and document everything is working and we test as far as we can test. See, there was another little thing. We couldn't complete payments with this new method because there was a test API for that method. And it never made it into the documentation. Steve and I couldn't find a single soul who knew about it. Maybe they were another, they were a spirit. Time was running out.

• 29:39Scott Tolinski?

  This was Wednesday morning. We had to push to code before 3:00 PM. There was one last test left to run. The case where the customer is not validated. The customer validation guys are still testing their code. So we can't use the forced response API again. We go, "Screw it," and we hardcoded the whole thing. 15 minutes to deploy the code again to the staging server. We look at it, we see it's working. We harass both of our teams to get a PR containing all of the backend and frontend modifications approved. Another 15 minutes merge to master, another 15 minutes to merge to production, and we are done. The release party is about to start. Crack open some brewskis, right? Boss drags us all into the party lounge. Get in here, boys.

• 30:23Scott Tolinski?

  I bring the laptop to the party because my name was on the production of deployment. So I was ready to be on call to monitoring certain channels, error logs for at least two hours. Steve brings his laptop, we grab a couple of beers and split the task of monitoring. We sip our beers, we laugh, we talk about how crazy all was, and we bask in the light of how much we managed to simplify that code. Then, oh, we see the curve showing payments transaction activity, flat line. Oh. I, that is like the, people who don't know, if you work in payment processing, anything, the moment that you make code changes and then all of a sudden in the actual payments window, you see a flat line, you, you are, go into panic mode immediately because you

• 31:08Scott Tolinski?

  goofed something up, right? I look at Steve, he's as pale as a ghost. I'm as pale as a ghost, hands shaking. We go to the logs and try to find an inkling of a clue what the crap is going on. There are no errors, nothing crashing. We dig deeper. We enable debug in production, knowing full well, we might just crash a couple of services. We see a bunch of logs, customer not validated, exiting out. I bang my head on the table a couple of times. I know, I say carefully. Steve confused, just staring at me. I open the frontend code and it dawns onto him. Crap, we forgot to un-hardcode the thing, didn't we? Classic. Oh, no. He bangs his head on the table a couple of times. We look around to see if anyone had noticed, everyone was partying.

• 31:53Scott Tolinski?

  I create the PR, we change one Boolean from false to true. Steve quietly approves. I try to breathe deeply while looking at Jenkins, feeling those 15 minutes stretch out over an eternity. I reach out to another beer and split second, I crack the beer open. It goes pop. I feel a hand on my shoulder. I look up and I nearly gagged. It was our CTO. Oh, CTO means business. I fill out my soul leaving my body as he says for calmly. Please don't drink and deploy. My brain was still trying to process it when the CTO bursts out laughing and leaves. Steve looks at me. We are so fired. We cheer with our beers as

• 32:34Scott Tolinski?

  the deployment finishes. We drink and celebrate our own little party as the payment transaction curves go up. Oh, man. We both had to sit there a postmortem of shame later that month to get scolded on how many millions exactly the company lost in this hiccup. Bro, if that if that loss is a milli or higher, that's a that's a scary situation.

• 33:00Wes Bos?

  That's scary, but also like.

• 33:03Wes Bos?

  If you have that much traffic and that much payment, unbelievable that that gets through. But I also to calm in these stories.

• 33:14Scott Tolinski?

  Not just unbelievable, they could super unbelievable that that gets through and gets put on the shoulders of a six month old hire. That's a process failure right there. A six month and a less than six month old hire. Oh, man. No, we didn't get fired because of that incident. The shame of sitting in that postmortem was probably more traumatizing than being fired. The incident caused a solid bond between us. He was my favorite colleague until he left. Wes, I give this one a good seven. A good seven creepy crawlies because million, if it starts with a million, that bumps it up in that, you know, higher than seven.

• 33:55Wes Bos?

  When the stakes are this high and the stories are so funny, it's a 10 out of 10 for me.

• 34:00Scott Tolinski?

  It's a 10 out of 10.

• 34:01Wes Bos?

  And like, if there's anything that helps you get your deploys down to be as fast as possible, it's stuff like this where you know the fix is in, or you hope the fix is in, but you gotta sit there for half an hour while you wait for the thing to to deploy and be pushed through, you know, because like, some of these sites are down and. Hopefully you can cut back, but it sounds like this was such a big one that there was no rolling back and you could only, only fix forward.

• 34:33Scott Tolinski?

  Yes. By the way, these fangs are digging into my lip and I'm gonna have like

• 34:38Wes Bos?

  Awful.

• 34:38Scott Tolinski?

  actual poke marks in my lip when I'm not gonna take these off.

• 34:42Wes Bos?

  Real blood, folks. This is not makeup.

• 34:44Scott Tolinski?

  It's about to be. I'll tell you that. It's about to be.

• 34:47Wes Bos?

  All right. Last question or last story we have for this episode and then coming up on Wednesday, we have a full hour of them as well. Plenty of them is called discount. You get a discount and you get a discount and everybody gets a discount. I work for a large company that offered multiple subscription based products. Man, these, these ones, I like the ones where the database goes away, but I really like the ones where real money is involved.

• 35:14Scott Tolinski?

  Oh, yeah.

• 35:15Wes Bos?

  And I really, really like the ones where products go out before they realize it's, that's

• 35:24Wes Bos?

  my task was to implement a discount feature in the CMS so that a content editor could add discounts to a product. Everything was tested and deployed to production without a hitch until 12 hours later, someone discovered that each and every product now had a discount. Products that had no discount set up used the first available discount in the list. And it happened to be conveniently sorted from high to low. This was a cause of an API in our CMS that returned a field such as object dot discount dot value, but my code only checked if object dot discount exists, which always was the case.

• 36:04Wes Bos?

  An empty object is truthy, folks. Although the financial loss was very extensive, my boss was relaxed about it and understood that mistakes happen. When I eventually left the company, years later, I got a postcard with the actual Oprah, you get a car meme as a goodbye gift.

• 36:21Scott Tolinski?

  Ah.

• 36:22Wes Bos?

  Oh, and sometimes you see these like, like pricing errors and people like freak out. Like Abercrombie had a had a major pricing error, maybe about six months ago. And I'm still not sure if it was a pricing error or if it was just like a intentional, like, marketing campaign. But when when those pricing errors go out, it's nuts. Like, once when I when I was in university, we found a coupon. In Canada, we had this website called Red Flag Deals. And anytime there was a pricing error or a coupon or a good deal, it would go up on that website and people would just destroy it. So we had this one that was $5 off $5.

• 37:03Wes Bos?

  So we went around to every it was called Pharma Plus, like a like a like a CVS in Canada.

• 37:09Scott Tolinski?

  Yeah.

• 37:09Wes Bos?

  We went around to every Pharma Plus because you're we're a college student and we have more money than brains or sorry, more time than money and brains.

• 37:17Scott Tolinski?

  Yeah, right, yeah.

• 37:18Wes Bos?

  And we just we went to every single one and bought $5 worth of stuff and then went back in and bought $5 worth of stuff and just kept doing it until they stopped us. And we just went to the next one. And I got like, maybe like $100, $200 with like deodorant and chips and all this hilarious stuff. And I just I feel bad now, looking back, like, some poor guy pushed that coupon out on a Friday afternoon, it was a Friday, and just went home for the day. And then they buy like, like, Sunday morning, they had stopped it. But like, it was running rampant.

• 37:52Scott Tolinski?

  Yeah. One thing I always laugh about is when people are like, talking about software bugs and like sports apps or something like ESPN thinks this player plays for this team. So I was like, no, ESPN doesn't think that. It's like some poor intern who pushed a wrong value in the app. And all of a sudden, you know, it's like, some some some poor person is like now like getting the heat of the whole company because of that, even though it's like, you know, insignificant little thing, right?

• 38:19Wes Bos?

  Well, at least we can you give us a laugh. Many of these stories people send us years, years later after they've the PTSD of the event has done, man, I am hot in this shark costume. But so that's the end of the first one. Should we just keep rolling? Because I don't want to end the live stream.

• 38:38Scott Tolinski?

  No, we will keep rolling, but let's at least wrap the episode. Well, thank you, folks, for joining us on this first spooky tales of spooky web developer stories where we found real scary situations that real developers found themselves into.

• 38:57Scott Tolinski?

  We will catch you on Wednesday with a full hour for this was just a first taste, the first bite of these spooky stories. Out. Peace.

• 39:13Wes Bos?

  Just kidding. It's just a costume. I didn't get eaten. Oh yeah, I was gonna ask that. You actually recorded that song? Like, that didn't sound like AI. That was really good.

• 39:22Scott Tolinski?

  Would you like to hear it again? Folks, this song was not AI. I recorded this myself.

• 39:27Wes Bos?

  Play it. I'm gonna grab a drink. One sec.

• 39:29Scott Tolinski?

  Okay. This was not AI.

• 39:45Scott Tolinski?

  I was working in my code late last night when my eyes beheld a creepy sight for my code was working with no reason why. And then I saw to my surprise.

• 40:00Scott Tolinski?

  It was a browser hack. With Macromedia Flash. It really saved my ass. It was a browser hack.

• 40:16Scott Tolinski?

  Yeah, that was me doing everything. I did one take on all the vocals. And then I did a second or third take to do some harmonies and backups. But yeah, that was me singing everything. No auto-tune, folks. I was just singing it. So sometimes I forget that I was a music major. Sometimes I forget about that. It was a different life.

• 40:36Wes Bos?

  That is very good. I don't have any sort of talent like that.