950: Even SCARIER Web Dev Nightmares (Spooky Stories Pt. 2)

In part 2 of this year’s Spooky Stories special, Wes and Scott discuss the most chilling developer horror stories—from six-month-old unprocessed donations and runaway dog-food orders to vanishing databases, DNS disasters, code that literally tore apart a mall’s ventilation system, and more! Show Notes 00:00 Welcome to Syntax! 01:33 A Bug Beyond the Grave 04:16 NHL SPOOKS 06:36 White Space Ghost Faced 10:54 Over Order Nightmare 16:50 Alaskian 21:16 Brought to you by Sentry.io 22:50 Rackspace’d Out 25:02 Fired 26:52 WordPress Woes 33:21 What does the P in VPS stand for? 34:18 Beyond the Grave II 35:39 The Hottest Hot Fix 37:54 Bad Redirect 40:03 Instead of Making Money, You Spend Money! 41:26 Certbot Certain Death 43:55 It’s Always DNS 50:02 Cache Ruins Everything Around Me 51:52 Fiber F-Up 56:18 More Spooky Stories Don’t Drink and Deploy 😬 | Spooky Dev Stories Pt. 1 Submit your Spooky Stories Hit us up on Socials! Syntax: X Instagram Tiktok LinkedIn Threads Wes: X Instagram Tiktok LinkedIn Threads Scott: X Instagram Tiktok LinkedIn Threads Randy: X Instagram YouTube Threads

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Wes Bos?

  Welcome to Syntax.

• 0:04Wes Bos?

  We've got our second episode of 2025 Spooky Stories. These are stories of web development and software development in general of just awful things that have happened in your career. You've taken down websites. We've got some real doozies coming up, just awful stories, and we are going to read them. If you have your own spooky story, please submit it at any point. Go to syntax@fam4/s spooky and you can submit your story to us and we'll read it on next year's episode. We're always collecting them and always love hearing them. So we have a sentry plug in one of these stories, so I'm going to leave it for that. Is that, you think that's all right, Scott?

• 0:49Wes Bos?

  It's actually a really good one.

• 0:51Scott Tolinski?

  I want to read the ad.

• 0:57Wes Bos?

  I want to read the ad. So if you didn't catch the last one, and if you're listening on audio, Scott has like a very good vampire costume, real blood.

• 1:08Scott Tolinski?

  Thank you. I was doing this costume and my wife said, "Do you have a plan?" And I said, "No, I don't have..." What is this? I'm losing the accent. "I don't have a plan." And so I just started mashing the face paint around and what you see here is what you get. And you know what? I think that's a pretty good job, actually.

• 1:30Wes Bos?

  You look great.

• 1:31Scott Tolinski?

  Thank you.

• 1:33Wes Bos?

  All right, let's get in on the first scary story is a bug from beyond the grave. By the way, I wrote all the titles for these ones, so please appreciate them. "Once, I built a website for a foundation that did medical research. People could donate money for special occasions like birthdays, but mostly in memory of somebody who has passed away." That's nice. "Everything worked perfectly at launch and I even tested it myself. The payment showed up as reserved on my card." All good. You know, like it reserves the money in your card. It hasn't actually gone through. Sometimes you need to do that. That's a part of credit card processing. Six months, oh my gosh. "Six months later, I'm at my summer cabin with a beer in hand and the grill going when the phone rings.

• 2:18Wes Bos?

  None of the donations have come through. A donator was complaining that their donation never came through." Strange.

• 2:28Wes Bos?

  After some panicking and digging through, all the test suites were still passing. I realized I'd done everything right, except one tiny thing. I never hit capture. The payment provider, thankfully out of business today, had a special endpoint for capturing the payments. That's Stripe is the exact same way, by the way. Auto capture was optional, but not default and very hurried in the documentation. So all those donations and memory for dead people have only been authorized, but never actually charged.

• 3:02Scott Tolinski?

  Oh, no.

• 3:03Wes Bos?

  Oh, so people have been donating for months to honor loved ones and the money never moved. Total panic. Imagine your entire business is taking money for people. You don't do the one thing you're supposed to do. But luckily, the payment gateway kept pending transactions for six months. It had been five months and 20 days.

• 3:26Scott Tolinski?

  No.

• 3:27Wes Bos?

  I wrote a quip script to capture everything and sent out an email to everyone affected. In the end, it all worked out. Ooh.

• 3:36Scott Tolinski?

  Spooky. Oh, oh, this one, because it involves the spirit of lost ones. I am going to say this one is a nine out of 10. That is very spooky to me. Oh, man.

• 3:48Wes Bos?

  I'm going to give that one a nine out of nine and a half. That was, I just, the fact that it not only is awful and includes money, but also includes like something sensitive, which is donating and memory of loved ones.

• 4:02Scott Tolinski?

  Yeah.

• 4:05Wes Bos?

  Yeah, next one's about the NHL, though. Maybe those make us feel better.

• 4:09Scott Tolinski?

  Oh, go about the NHL. Go grab my hockey stick I got over there. I got a hockey stick in my office, Wes. Oh, yes. The next one here is NHL spookiness.

• 4:22Scott Tolinski?

  I built a very nice website for the National Hockey League. A whole team of content specialists, about 10 people, start filling the site with content. The day before launch, the whole team even starts working from 6:00 in the morning for their last big push of content. I export a single table from our staging site. Not content for a special feature. To SQL and blindly import it in the production database. Oh, blindly import it in the production database. What a sentence. The whole site goes down and my new table is left. Only my new table is left. Oh, man.

• 5:03Scott Tolinski?

  It turns out the dot SQL export didn't just contain a create table query, but also a drop or all drop table queries.

• 5:18Scott Tolinski?

  That sounds like an AI style migration, if you're asking me. That's a... Oh gee, so they ran a SQL command that just dropped all the tables and then added their new table. The latest database table was from three days before, so all of the content was gone. Yes, the launch was delayed. Yes, I was ashamed. Yes, I always check my exported dot SQL files from now on. This is now more than 10 years ago and I still get chills thinking about it. Oh, there's no backup strategy here? What, everybody had to re-enter their stuff or what? It's just gone?

• 5:53Wes Bos?

  Probably. Yeah, that's awful. If that was me, I'd just be like constantly backing that thing up, you know? Just every, every 20 minutes backing it up. Cause that's, is very spooky to have that happen, especially when it's such a big, like the NHL, you know? Like there's 10 people working on it. This is, that's, that's a lot of money in, in paying people salaries.

• 6:17Scott Tolinski?

  Yeah.

• 6:18Wes Bos?

  Like, brutal. I mean, I want a seven out of 10 spooky, spooky bookies.

• 6:24Scott Tolinski?

  Spooky bookies. I'm gonna give it five Jason masks out of seven Jason masks. Uh, he did wear a hockey goalie mask, if you don't know about Jason.

• 6:34Wes Bos?

  I never know what you west, so. The next one is entitled white space ghosts faced. I'm a full stack dev that writes mostly typescript, of course, like any good full stack dev. I'm a seasoned staff with lots of experience writing backend code, managing infrastructure, writing helm charts, setting up alerts, yada, yada, yada. So I'm not afraid to jump into some Python code base and make some changes. This is sounds like something I would say. I've been writing a lot of Python lately. The Python service I've been working on was originally written by a messy ex-Google engineer, and I'm stuck with them. And a broom cleaning up the mess. Yeah, typical Google employee writing a bunch of messy Python.

• 7:16Wes Bos?

  This particularly messy Python service was hitting Google's cloud video debt generation model. Okay, so this is relatively new. So I was cleaning up some of the messes. Every time video creation fails, multi-line logs are barfed up over the standard out in using a print statement instead of our structured logger. So I updated the main.py file in our code base to use our logger instead of print. So doing so required moving around a few things, etc. Makes sense. I opened up a PR, I ping my trusted front-end dev to rubber stamp it, and logs are pretty and people can create VO videos through the API. Everything is good. All right. Three weeks go by. That's a long time, folks.

• 8:00Wes Bos?

  That's a long time for this, the just wait part. And I see a group of people crowded by a laptop on my way to the office, and the CFO is standing by them, looking moderately perturbed. They are wondering why our VO2 costs last month has shot up 5X. Certainly nothing to do with my innocuously logging change, right? Well, time to find out. My little cleanup PR introduced a line that unindented the message acknowledgements outside of the try catch block, resulting in failed video generations. Example, fail due to safety checks. Yeah, like you try to create a video with something that is is not appropriate, they'll they'll fail it.

• 8:43Wes Bos?

  To be tried, retried five times. Just for reference, a 30-second video generation on Google costs about $10.

• 8:54Scott Tolinski?

  Oh, yeah.

• 8:55Wes Bos?

  It's it's an expensive API, so this literally had cost the company hundreds of thousands of dollars. The crazy part is it was just a white space change. It didn't even show up on our code review platform, which is Graphite. So the DevI Ping rubber stamped the PR, didn't even notice the change. Friggin' Python with his indentation. I used to be such an indentation fan, you know? I used Pug and Stylus and CoffeeScript and everything was indentation based.

• 9:23Scott Tolinski?

  Same, yeah. I liked it.

• 9:25Wes Bos?

  No more.

• 9:26Scott Tolinski?

  I liked it. I don't know if I'll still like it, but I liked it at the time. That was the style then.

• 9:31Wes Bos?

  I would blame him. I would blame Graphite, but ultimately I have to be forthcoming and put my ego aside in such situations. There is only one person to blame here. The Python programming language and its effing syntactic white space.

• 9:48Wes Bos?

  He queued it up to blame himself. He just says it's the Python problem. So, yeah, lots of yapping to say one invisible line change caused us hundreds of thousands of dollars to the Google Abyss. Thankfully, my colleagues were quite forgiving and I'm still working hard to redeem myself in the finance department. I've definitely been given a shorter leash. The head of engineering.

• 10:12Wes Bos?

  Oh, you gotta have some alerts set up or something if things were to to go awry.

• 10:21Scott Tolinski?

  Oh, bro. Yeah, I get that. I get that one about four Dracula fangs of which I'm going to be taking out. So that way I don't have permanent damage done to my lips here.

• 10:31Wes Bos?

  That is not the most expensive goof up we've had. We've had a couple of stories in the past that were in the millions of dollars because they shipped out physical products to people who did not pay for them. But that is still a seven oofs out of eight, I think.

• 10:52Scott Tolinski?

  You think seven oofs out of eight? Okay, next one here. Overorder nightmare.

• 10:58Scott Tolinski?

  I had been working at a premium pet food e-commerce store. To give you an idea, we average about 4,500 orders a day with about 300 of that amount were subscription orders. Well, I subscribe to some dog food, maybe I'm a customer. The customer had an option to start a subscription of their pet food orders. This would create an order on the interval the customer set him or herself. However, the person that created the whole subscription system went into a burnout and eventually left the company. I joined when that person was already burnt out. Let me tell you, working in reoccurring billing without like tools to do it for you can absolutely

• 11:42Scott Tolinski?

  drive you to burnout, absolutely.

• 11:44Wes Bos?

  Billing. What's the worst? Can we, can we ask this? What's the worst parts to work in? I'm gonna say.

• 11:53Scott Tolinski?

  Reoccurring billing.

• 11:54Wes Bos?

  Single sign-on. Reoccurring billing. Single sign-on and maybe auth in general. Access control.

• 12:02Scott Tolinski?

  I don't mind access control either.

• 12:04Wes Bos?

  I think those are probably the worst.

• 12:06Scott Tolinski?

  Anything that involves kind of like ping ponging back and forth, but like with reoccurring billing, like testing it, you have to be, it has to be done through webhook. So you can send a test webhook and then like when an actual subscription like webhook comes in, yeah, you could have tested it, whatever, but it's still like, it's, it's still always very scary for, for like what actually is, is doing that. There was a big demand of the customers that they could choose which day of the week they would receive their food as hybrid remote work was very popular in our country. They wanted to receive their food when they worked from home. This would be for regular orders and subscription orders. For example, for regular orders, you could choose that you would receive it the first Thursday after placing the order.

• 12:50Scott Tolinski?

  This got really complex as we sold it to about 20 countries. Oh yeah. Oh, oh, oh. We worked together with different shipping companies depending on the warehouse that order came in from. These shipping companies all had different lead times for each country. Eventually, we worked this out and thinking about the implementation of the existing subscriptions.

• 13:14Scott Tolinski?

  For running subscriptions, we had to calculate what was the last day of the week they received their last order. Then we calculated the next shipping dates that would receive on the same day. But with the given subscription interval calculated into it, we had to keep in mind the subscriptions were running so that we could need to calculate the time from their last order until now and subtract that from the interval they had specified. We then took the closest day calculated from the next shipment date and would then change the shipment date for the existing subscriptions. Oh my gosh. Bro, I hate subscriptions with digital products. With shipping and then shipping dates that things have to arrive in, take me out back. I'm done. Logistics, man.

• 13:57Scott Tolinski?

  Logistics. I don't like logistics. I'll tell you about that. I don't know. I'm good at planning. Jeez, that's crazy. At the time, I had been there for about a year. The subscription system was a bit of a gray box for everyone, where we could change some things, but nobody on the team had full context on how it all worked exactly. And there was no documentation. As I said earlier, the original creator had left the company. I had been fixing some bugs in the system at my time there and felt confident to implement this new feature. I took some days to get familiar with the system and wrote out exactly how I would change it in the areas that would be touched. From running subscriptions, I wrote a migration strategy that discussed everything with the team and my manager. Everybody gave the green light.

• 14:42Scott Tolinski?

  We tested the implementation. We put this into production. Everything ran smoothly and we were getting positive feedback from our customers. Oh man, the hammer's about to drop here. But after 10 days, we were getting complaints. People who had subscriptions and had their next order delivered in these days had been paying every day since the feature launched.

• 15:05Scott Tolinski?

  That's, that's the hang I knew was waiting for it. They'd been paying every day. Turns out, we had been calculating the next order date wrong, and once the new date was calculated, the next order date was placed on the current date, as this is a running system and had been going for 10 days before we noticed. Thousands of extra orders created, packed and shipped.

• 15:28Wes Bos?

  Oh no! And it's, it's not even like you're getting like, like a hard drive or something fun. It's like, like dog food. It's so huge.

• 15:39Scott Tolinski?

  I bet. I would take 10 extra orders of dog food because you know they're going to refund it at this point. Luckily, I created a backup of the database when I launched the feature into production so we could see who had upcoming orders, and we then had to spend the next week with the whole team figuring out which customers had placed how many extra orders to reimburse them all.

• 15:58Wes Bos?

  Oh.

• 16:00Scott Tolinski?

  This, I give this one, uh, 9 demon pups out of 10 demon pups.

• 16:06Wes Bos?

  Brutal. That is, and you know, like, the poor support people as well, they're just going to be going through it for the next, like, weeks or months, and like, probably lost a bunch of money on the dog food. And I, I would be mad if I got all that extra dog food, because like, where do you put that, you know? I guess you could give it to a, like, donate it to a shelter or something, but oof. Hopefully that's the silver lining here. Some shelter got a bunch of free dog food.

• 16:38Wes Bos?

  By the way, we keep all of these anonymous. Um, this one includes the company name, so I'm just going to change the company name for the sake of anon- anonymity.

• 16:48Scott Tolinski?

  Anonymity.

• 16:50Wes Bos?

  Anonymity. The year was 2019. I was on the experimentation platform team at the time, but was given responsibility over everything analytics after being told it should be in the "keep the lights on" mode, which is minimal development, just keep it running. The service API was based on Segment.io in order to leverage Alaskan-specific wrappers. Change the name.

• 17:17Wes Bos?

  Wrappers around the clients. Alaskan developers had been given a halt order for feature development after being plagued with reliability issues in the front end that devs were flying blind to. It was decided that the best way to give front end quick visibility into performance and reliability was to leverage the analytics platform. The front end JS client used a library called segment local storage retry in order to capture as many events as possible. This is pretty common when you're sending events. You'll throw them in like a variable or you'll throw them in local storage so that if there is, if they're blocked by like a thing, if the person goes offline for whatever reason, at least they're in local storage and you can retry over and over again.

• 18:02Wes Bos?

  This uses multiple local storage entries to manage the queues for processing and to be able to recover queues from tabs that have been closed. That's another good reason for it. We started getting reports from internal users that local storage was filling up and it was due to the analytics client. After digging around, I found that the client would detect that local storage was full and switch the entire storage solution to memory, which prevented any further cleanup inside of local storage. Once we got that fixed out, we pushed it to all products only to start causing more instability to our backend analytics service. So it took us a while to realize that we would occasionally be seeing single analytics

• 18:47Wes Bos?

  event be able to be duplicated tens of thousands of times, which was also causing additional slowdowns on the other clients. It was a hard case for us to get budget to add more fixes for a service that should have been in maintenance mode, but the volume it was needed for processing grew from 1 billion a day to 3 billion with the influx of additional monitoring front end teams that were trying to be shoehorned in. Yeah, this is, this is monstrous. Like we've had folks on the past from Sentry when it's just like. Like you're essentially getting DDoS. Like being able to just receive the events from everybody is a massive, massive thing.

• 19:27Wes Bos?

  Soon, it became apparent that the fix was causing more slowdowns on client browsers and also sometimes prevented garbage collection from running due to the frequency of local storage checks. This is really interesting because local storage is a synchronous API. Because so every time you're touching it, you're, you're actually running a synchronous function, which if you do it often enough, can start blocking the main event loop. In testing, I even managed to crash my entire laptop.

• 19:58Wes Bos?

  I was thinking, we should do a video of who can crash the browser tab first. Who can write the worst code to crash a browser tab? I think that would be good.

• 20:07Scott Tolinski?

  I can crash your browser tab in seconds. Melissa, I can crash faster.

• 20:12Wes Bos?

  Oh, you know how to do it?

• 20:14Scott Tolinski?

  You just do a, I mean, just use, use effect, huh? Just use effect.

• 20:19Wes Bos?

  Use effect. We tried a few hacks to get on top of this, but at the end of the day, we were trying to treat local storage like a transactional database and it became very clear that only one thing we could do is move the functionality to index DB, which removed a lot of complexity and overhead and almost all of the issues went away after that, except for the fact that many devs kept throwing more and more load at it because the analytics pipeline turned into a hammer for many problems, but that's a different story. Moral of the story, don't treat local storage like a transactional database. Yes. Yeah. As much as, especially like you can, you can run out, you can run out of space in local storage and every time you write to it, it only holds strings.

• 21:01Wes Bos?

  So you gotta, there's like a JSON parse you gotta pay for there. And a second story here is the analytics is not designed for platform monitoring. I think there's another tool for that, like, dot, dot, dot, sentry.io. Why, thank you for the, the ad there, dear friend. I don't know what your name is, but yes, you guys should be using Sentry at sentry.io forward slash syntax and get two months for free using the coupon code Tasty Treat.

• 21:31Scott Tolinski?

  Yeah, take a bite out of your errors.

• 21:35Wes Bos?

  That's good. P.S. The main problem was Overlord Alliance on the analytics problem kept growing despite being in maintenance mode and it hit a peak of 13 billion events a day before Dedicated Team was assigned to rein in the cause. Oof, I can't imagine the bill for that, man.

• 21:54Scott Tolinski?

  Be with a billion.

• 21:56Wes Bos?

  Billion. P.P.S. I'm no longer at Alaskan.

• 22:02Wes Bos?

  To this day, there is still some code that will attempt to reclaim events from local storage despite no new analytics being added to local storage since the start of 2021.

• 22:15Wes Bos?

  Oh, man. Man, man, man. I give that nine and a half out of 11. That was good.

• 22:24Scott Tolinski?

  Yeah. Yeah, I would, I would say that is, I would say that is a nine out of 10 great white sharks that are infected with some kind of zombie disease. So they're

• 22:34Wes Bos?

  Zeke in the chat just said that's 150,000 events per second.

• 22:39Scott Tolinski?

  Oh, what?

• 22:40Wes Bos?

  It's a big, that's a big company, Alaskan. I think they have every person in Alaska as a customer.

• 22:47Scott Tolinski?

  They, a lot of them do go skiing, for sure.

• 22:50Wes Bos?

  Yes.

• 22:50Scott Tolinski?

  Alright, the next one, rack spaced out. I'm gonna little Arnold there. I run the dreaded R.M.R.F. forward slash star when I meant to run the R.M. hyphen R.F. dot forward slash star.

• 23:11Wes Bos?

  Never, never do this.

• 23:16Scott Tolinski?

  What? On the production server of a large commercial real estate company. I didn't use Git or any other version control. Backups had been offline for years. The server was completely trashed.

• 23:33Scott Tolinski?

  But the rack space tech was entertained. Can you imagine getting this email? They'd be like, hey, brother, I just ran RMRF on my entire server. 23 year old me was thoroughly impressed that the dude stayed on the phone with me for hours. Ended up spinning up a whole new dedicated server and moving over what pieces he could to the new box. I lost a few days of data in the end, but got most of it back up. Well, that's, that's a lucky end result. One time I ran get clean on my root directory. I did that one time. I think I talked about it in one of these spooky episodes.

• 24:14Scott Tolinski?

  And then I deleted a bunch of video production files I had going and kind of borked my system, but didn't. I noticed what I was doing, so I, I was really quick on the command C. But I still, it's still chomped through some stuff. Either way, very scary.

• 24:28Wes Bos?

  Everybody go right now and check that your backups are working because man, they often are not. And when you need them, it's not the time to find out they are not working. And also like shout out to like support engineers or like the technical support staff that actually know what they're doing. Cause they're able to, Sentry has a really good support team. I've been to the Toronto office several times and you just like listen in on them trying to help everybody with all their problems. And they know about everything. It's crazy.

• 25:01Scott Tolinski?

  Yeah.

• 25:02Wes Bos?

  Next one's called fired. Before I worked as a web dev, I worked in IT doing desktop network and server support. I had a job working for a financial company in San Francisco. My job was to prepare laptops and other hardware for new fire, new fires, new hires. The process involved wiping the ORM images and loading custom images on my colleague and I maintain depending on the laptop or in rare case desktops. My job was very routine and robotic. I loved it. One day, however, the CEO's laptop was brought in for servicing and definitely not a wipe sadly. In the routine of my job, I formatted a CEO's hard drive. So true, you do something every single day.

• 25:47Wes Bos?

  We have a backup of his laptop before, but it was not one of the CEO's laptop that was backed up. I was looking for work the next day.

• 26:00Wes Bos?

  Oh, that sucks. That sucks so much. Oh, man.

• 26:05Scott Tolinski?

  CEO's hard drive. I can't think of like a worse thing to make an impression with. Oh.

• 26:13Wes Bos?

  Man, but like, I don't know if these big companies can do this, but like I have rolling backups running in multiple places all the time because this is bound to happen at one point.

• 26:26Scott Tolinski?

  Yeah. Yeah. I mean, you gotta have a backup. The IT people should have been making sure that that CEO had backups or whatever, but still, uh, very spooky. Yeah. Getting fired for something very spooky. I would say that this is a, you know, a six out of eight killer snowman for sure.

• 26:43Wes Bos?

  I would give the same as well. Killer snowman.

• 26:47Scott Tolinski?

  Killer snowman. Jack Frost's, I believe. I don't, that's a movie, but I haven't seen it. WordPress woes. I was doing some freelance work for WordPress when I was on a hot streak with quite a few jobs done and all five star reviews until this job. It was actually pretty simple. It was just a WooCommerce site that needed some minor changes. So I did what I always did and created a backup, a staging site, et cetera, and got to work. I completed everything, uploaded onto staging and then got the client to overview it with a green checkmark. Awesome. They were using SiteGround, so I was able to just hit the sync button, merge the staging sites to production, forgetting that this was an ecommerce site.

• 27:31Scott Tolinski?

  I just overrode the database entries for all of the last few days of purchases. There was no way to tell who had placed the orders or when or for what. Suffice to say, the client was very upset and I still got lucky. I still got away with a three star review, even though it should have been a lot worse. I would imagine what they would have to do is like go through amounts and like guess and then contact the customers and be like, our dumbass web developer screwed the pooch here. What did you order? Yeah, that, there's no way to fix that but manually.

• 28:05Wes Bos?

  We've had this, this many times, this story of someone who did it and they've had to like re- Build everything from like emails sent out.

• 28:15Scott Tolinski?

  Yes.

• 28:15Wes Bos?

  Hopefully you can go into your, your sent emails or logs or something like that. But just, it's always a tricky one, especially like, it's not a WordPress specific one, but a lot of people hit this issue where I have config that needs to change in the database, you know?

• 28:30Scott Tolinski?

  Yeah.

• 28:30Wes Bos?

  But I also have content that lives in the remote database. And sometimes you bring the remote one locally and sometimes vice versa. And you gotta be very careful doing these migrations.

• 28:43Scott Tolinski?

  I always found that to be very scary when you have that much, like with WordPress or Drupal, like doing migrations in that way where the database is such an essential

• 28:52Wes Bos?

  Yeah.

• 28:53Scott Tolinski?

  part of the whole thing. I, yeah, there was like a Drupal features and ways that you would commit things and whatever to get things in to get. And it was just like always very scary when you did any kind of deployment process.

• 29:05Wes Bos?

  Another WordPress whoa. One time I was working on converting an old shop site. This guy submitted like 10, 10 WordPress horror stories like he'd never learned from his thing.

• 29:21Wes Bos?

  Because the guy wanted his own site. I was doing everything in WP CLI to speed things up because there was tons of data. Well, I didn't realize that WP CLI was going to trigger the WooCommerce email system as I was migrating the data over. It sent out emails to people about their purchases. There weren't actual charges made, thanks goodness, just re-triggering of email notifications. And the guy I was doing the job woke up to over 18,000 emails from angry and confused customers. Luckily, I was able to salvage our relationship and get the job done with a five-star review. This guy's got five stars. And the site is still running to this day, five years later. It made me think a little more carefully.

• 30:05Wes Bos?

  Well, apparently not, because the next one is from the same guy.

• 30:10Wes Bos?

  But like, if you worked with somebody and they really screwed the pooch, but then they fixed it, would that give you confidence in working with them in the future?

• 30:21Scott Tolinski?

  Well, you know what they say, you know, fool me once, shame on you. Fool me twice, you can't be fooled again. But

• 30:31Scott Tolinski?

  Did you ever hear that? That's a George Bush quote. He like, he forgets what he's supposed to say. So he's like, "You can't get fooled again."

• 30:39Scott Tolinski?

  It's a, it's a train wreck. But yeah, I give one time. One time you give them a break and then two times, yeah. J. Cole said something about that.

• 30:49Wes Bos?

  Well, the third time, you got the next one.

• 30:52Scott Tolinski?

  Yes. "I was working for a small WordPress-based company when we had a client who was thinking of hiring us on Retainer. So we said we would spend a lot of time to look at their site and their project to get up and up to speed and discuss what we could offer them. One of the big things they needed was help with when they approached us was dealing with Stripe. The way it was set up was not correctly doing something. I don't remember why exactly. So we asked if they had any custom code within the site dealing with Stripe and they told us no. When I pull up the site to my local machine and spun up in my dev environment at the time, and I start to look through some things, well, it turns out that they were wrong about custom Stripe stuff.

• 31:35Scott Tolinski?

  And there was some code that automatically ran a bunch of webhook stuff when loaded locally. All of their customers, around 5,000, got a second charge to their credit card for the month, no matter how recently it was paid. All in all, they lost about 500 customers by the time we talked to them the next afternoon. We were not hired on Retainer and we learned some very valuable lessons that day. You didn't even get hired and they lost 500 customers for, that's a bad day at the office."

• 32:10Wes Bos?

  You know what? They probably had the Stripe API keys in the database and he pulled the production database locally.

• 32:18Scott Tolinski?

  Yeah.

• 32:19Wes Bos?

  Holy, don't put API keys in your, I guess in some cases you have to, but man, that sucks. A 10% loss of all their customers. And on top of that, how many of those 5,000 people that got the second charge? Like filed a, like a charge back? Cause that's 15 bucks out of your pocket. Every, like regardless, if, if, if you, the way that Stripe works is if you can refund it before they do a charge back.

• 32:48Scott Tolinski?

  Charge back.

• 32:49Wes Bos?

  Then you're in good shape. But if you don't, if they file a charge back and then you immediately refund it, you still pay that $15, even if you, you win or regardless. So like, I don't know, 5,000 people, maybe of 1% of them charged back. That's still 750 bucks out of your pocket.

• 33:11Scott Tolinski?

  Yeah. If you want to get pissed off, people, your customers, start messing with their money. Yeah.

• 33:21Wes Bos?

  What does the P in VPS stand for? This is a really quick one. I switched my VPS provider and forgot to point my portfolio to the new IP address. How do you do that? A month later, a friend texted me saying, oh, my portfolio was turned into an adult website.

• 33:43Wes Bos?

  How can you not check it?

• 33:46Scott Tolinski?

  Dude.

• 33:47Wes Bos?

  Oh, I guess that IP address was reused and you just never changed where it was pointing to. That's great.

• 33:53Scott Tolinski?

  Yeah. One of the first agencies I worked for, we were doing WordPress sites. And the only site I'd ever worked on, it wasn't even my, my client, but um, it had gotten hacked somehow. I don't know how. And that's, that's what had happened to it. They just replaced every URL with a bunch of redirects to adult sites. And uh, it was a crazy day at the office. I'll tell you that.

• 34:15Wes Bos?

  Never, never do that.

• 34:18Scott Tolinski?

  Beyond the Grave 2. When I first joined my company as a junior developer, I was testing out an issue with one of our alert systems. By tagging a very popular artist to my test event, I managed to send everyone a push notification stating that artist was going to be performing in their area. Oh, no. The only issue being that that artist had died a few months prior. Oh, no.

• 34:47Scott Tolinski?

  That's a much bigger oh no than I had initially given that. Oh, my God. Luckily, I worked for a nice company. No blame was placed on me and the new guardrails were put in place before tagging deceased artists and test events dispatching real alerts. Bro, that's two now that I have dealt with people who have passed away. That is very spooky. Ah, that I give that one 10 ghosts out of 10.

• 35:15Wes Bos?

  People talk about like vibe coding and building an app in an afternoon. This is the stuff that you're not told about. How these companies have unfortunately sent push notifications about a dead artist.

• 35:31Wes Bos?

  And now there's some code in their code base that checks if an artist is deceased or not before it sends events. The hottest hot fix. I once worked on new features that introduced API calls to a somewhat complex web component. From everything my team tested, it worked great. We thought we had accounted for everything. So we push it up to our repo, get it approved, publish the version and get it installed in our front end applications. Fast forward a smidge, the reports start to roll in that tests in the front end application are failing and holding up CI and CD. The common thread amongst the reports was the test suite can't find any elements. I don't pay much mind because we ran a colossal Cypress test suite

• 36:16Wes Bos?

  for a monolithic repro. So that could be anything. And that kind of thing is happening for. It's so true. Your Cypress tests are failing. Almost nobody goes, oh, must be actual error. Your initial thoughts are just like, probably Cypress is acting up.

• 36:34Wes Bos?

  Then I see the error crap. This is our API call failing in their local dev environment because we didn't add a proxy or dummy data. We forgot to add a failure state. And because we didn't catch the error, it's throwing at webpack's red screen of death, which blocks Cypress from querying anything on the page. We rushed out a failure state, which solves a problem in our main development branches. Nope. The hotfix branch is running into the same issue. Let's get that fix out. Oh crap. They weren't supposed to get the absolute latest, so they did a hotfix on, not on their production, but on their, I guess their main branch, which is not yet released.

• 37:17Wes Bos?

  So they pushed out a bunch of code that was not meant to be pushed out yet. Suffice to say, it was a comedy of errors that took days to fix. And if we had Sentry in our build system, we likely wouldn't have run into this issue. Thank you Sentry.io forward slash syntax use coupon code tasty treat. Hope you enjoyed this spooky story.

• 37:35Scott Tolinski?

  Ah.

• 37:36Wes Bos?

  Oh, that sucks. That sucks. I'm gonna give that one a one out of, no, a two out of three.

• 37:45Scott Tolinski?

  Yeah. Yeah, I'll give it, I'll give it a five out of 14 Legos on the floor. Bad redirect. In a company I worked where everything was basically in shambles and barely making ends meet, a coworker thought it was funny to somehow magically redirect hackers to an adult site when he thought something was of malicious intent. The boss made a demo for the company. That was playing during the demo, he was redirected to the adult site several times. It was a miracle we didn't lose the contract and that he didn't get fired. You sent your, you sent your boss to an adult site?

• 38:28Scott Tolinski?

  Don't put adult sites in your work. Don't do that.

• 38:31Wes Bos?

  Never, ever.

• 38:33Scott Tolinski?

  Never.

• 38:34Wes Bos?

  Ever put a, even put a swear in. I was having some frustrations with my recording the other day. I was sent like a screen recording to some, someone and I sent it to Randy, our producer as well. And the name of the screen recording was F this. And guess what? They saw. Never, never put swears or any adult stuff or, or even like think that you understand. Cause also a lot of this, like trying to catch people doing bad stuff. You're going to eventually catch somebody who's not doing bad stuff for whatever reason. Don't do that. Yeah, that's not a good idea.

• 39:12Scott Tolinski?

  You can, you can just send them to a blank page. That's just as easy. And then like, no harm, no foul. You can trap the, you can do honeypots. You can do all that stuff. And you can still annoy them by, I had a ban system on Level Up that would like act as if they weren't logged into the site. So then they would go to try to log in again, but then the login would fail because they were already logged into the site.

• 39:34Wes Bos?

  Oh, yeah.

• 39:35Scott Tolinski?

  Like, and that way, like if, if for some reason someone ever got accidentally stuck into the, that ban system. It would just be like, hey, I'm trying to log in. The login says this. Oh, let me fix it for you. You must've tried five different credit cards, 40 different times and the system failed. You know, like you must've hit these specific criteria, but like, don't send them to an adult site. That's bad, bad idea.

• 40:03Wes Bos?

  Next one, instead of making money, you spend money. So back when I started web development, I was working for one of those sketchy cellphone game ringtone wallpaper companies. Um, and we're billing the victim, I mean customer via premium SMS messages. That's how it used to be. You want a ringtone? You, you send an SMS. For development and testing purposes, we had regular SMS numbers that we use that didn't bill our cellphone. Okay. So you had like a, you have a premium one, which you make money. You have a regular SMS, which you actually, you pay to send. One morning our CEO called us into a meeting to give us the bad news that there wouldn't be a bonus that month. As the SMS provider had screwed us over, instead of paying out the bumper sale, they were billing us for excessive use of the free SMS.

• 40:50Wes Bos?

  I'm sure you had already, guess what happened? So at a moment, I had to sheepishly confess. I expected to be written up for having left the dev mode enabled in production, but I had an amazing CEO that just said, make sure it doesn't happen again and left it at that. Like you cost not just the company, but your coworkers real money by leaving a dev flag in production. That is three bones out of three for me. That sucks.

• 41:21Scott Tolinski?

  Three bones out of three? I'll do eight cobwebs out of 10 cobwebs. Certbot Certain Death. How to Lose Hundreds of Hours of Meeting Recordings in One Easy Step. Just let Certbot fail silently and wait for the SSL to expire. Hmm. We record our Zoom meetings through their RTMP streams. All of these streams land on our loyal Injust server, which is tied to a domain name and secured with a Let's Encrypt SSL certificate. Like responsible grown up adults, we even let Certbot to handle renewals automatically. Yeah, that's, that's fine. Then one day during the mysterious deployment, probably a Friday, Certbot stopped working.

• 42:06Scott Tolinski?

  No alerts, no warnings, just quiet little failure sitting there, waiting. Days later, when the SSL certificate finally expired, boom. Every single meeting stream started failing. And how did we find out? Not through fancy monitoring or alerts. An employee casually said, hey, that little streaming icon isn't showing up anymore. Translation, congratulations, you've lost recordings. A round of chaotic debugging later, we discovered the issue. The server was alive, but the SSL was not. We renewed the cert, the streams returned, and we immediately begged the leadership to approve buying proper SSL certificate with a longer validity.

• 42:50Scott Tolinski?

  Because apparently relying on Certbot alone is a bit like relying on goldfish to guard your house. Moral of the story, if Certbot fails quietly, it's not the failure that hurts. It's the moment SSL expires that breaks your soul. What a wonderfully written spooky story value. Thank you for crafting that. That's why I just use a system, a host or something that uses Certbot behind the scenes or whatever to manage the SSLs. I don't want to do Certbot myself, but I certainly don't want to pay for SSL certificate in 2025. I'm just going to use whatever my host has got baked in. Because every host in the world's got that now.

• 43:34Scott Tolinski?

  Who's not giving you SSL renewals for free?

• 43:37Wes Bos?

  I actually just looked it up. Let's Encryp used to email you when your certs expired, but as of January 2025, they've stopped that. So they won't even, you need like a downtime, like a ping, like a pinging service would.

• 43:52Scott Tolinski?

  Ping service, for sure.

• 43:53Wes Bos?

  Easy fix that for you. All right, next one. It's always DNS.

• 43:59Scott Tolinski?

  Is this about AWS?

• 44:01Wes Bos?

  Probably.

• 44:02Scott Tolinski?

  Was that DNS?

• 44:03Wes Bos?

  Actually, it's about Bluehost.

• 44:05Scott Tolinski?

  Oh, classic.

• 44:07Wes Bos?

  It's always about DNS. I had been debating about staying with Bluehost or moving over to Netlify for my small business website. I was tired of paying for hosting and figured a free Netlify plan was sufficient for what I needed. I mean, my site was pretty simple, just a portfolio, some case studies, and why was I paying 120 bucks a year to Bluehost when Netlify could host it for free? WordPress was overkill for one I needed. Now, I should mention, I'm a designer who dabbles in front end development. I can build it, style it, make it look gorgeous, but deployments and server side stuff, that's not really my wheelhouse and I'll usually work with other developers that would handle those things. Still, I watched a few YouTube tutorials in Netlify and everybody made it look so easy.

• 44:51Wes Bos?

  How hard could it be, right? So one Saturday morning in the air of confidence and iced coffee, I decided to cancel my hosting and begin migrating to Netlify. I signed up for Netlify, connected my GitHub repo and watched my site deploy in minutes. It felt like magic. Then came the DNS settings. Now, I know DNS was important, as we all do, but when I logged into my domain registrar and I saw that wall of cryptic records, A records, MX records, C name, TXT, SPF, my brain just went, yeah, I got this, shouldn't be too hard. I updated my name servers to point to Netlify as their documentation walked through and watched the little propagation may take 24 to 48 hours pop up.

• 45:36Wes Bos?

  And I felt like a tech genius. Is this really what backend devs deal with? Super easy. As a cell certificate check working, I gave myself a little victory fist pump. What I didn't realize is I had removed my email, which was running through Google Workspace. And those MX records I casually deleted, those were the only things telling the internet.

• 46:02Wes Bos?

  For two weeks, it felt like months. I'm sitting there thinking, wow, business is slow today. Nobody's reached out. Gets us a quiet week. Meanwhile, my email service is completely non-functional. And every email sent to me is bouncing into the void. Clients trying to send me project files, potential leads and time sensitive requests. All gone, never received. I get that way sometimes where I turn on do not disturb for the entire day. And then I go, wow, nice, nice, quiet day today. Oh shit, I do not disturb on all day. Hey, I've been trying to email you all week. Are you getting my messages? They keep bouncing back. The panic like that set in. Oh no, what?

• 46:43Wes Bos?

  But that's where it gets worse. I had no idea to fix it. Remember, I didn't know backend network DNS stuff. So I did what any panic designer does. I started Googling. I spent the next two days, brother, you could have, you could have DM'd me and I would have fixed it for you in like 10 seconds.

• 47:05Wes Bos?

  Oh, going down the most frustrating documentation rabbit hole of my life. I've read Netlify's DNS documentation, which would tell me to check my email provider settings. So I go to Google Workspace documentation, which would tell me to configure my DNS records with my hosting provider. So I go back to the Netlify docs. I was stuck in an infinite loop. I even went chat GPT and gave it to me back and forth in the same flow. I watched YouTube tutorials, read hundreds of Reddit thread. All with different variations of how to fix DNS records Netlify Google Workspace. I finally pieced it together enough, enough information to understand that I need to manually add Google's MX records back into the Netlify DNS settings. I copied those magic strings of text from Google's support page. I pasted them into Netlify.

• 47:49Wes Bos?

  I waited for what seemed like another agonizing 24 hours for propagation. And then I finally emailed myself and started working again. You don't even need to wait 24 hours there.

• 48:00Scott Tolinski?

  You don't, yeah.

• 48:01Wes Bos?

  There's like two things with DNS is when you're changing the name servers, you're literally changing who manages your DNS. If you can change it from provider to provider, or you can just use your registrar to change the DNS, you know, that's what I do. I put them all in Cloudflare and then I just point them at whichever servers you actually need because that way you don't have downtime. But also like when you import a domain name into Cloudflare, it scans the domain name for as many records as it can because you could accidentally be bringing down so much stuff, more than email, if you don't know about it. That's the fun part. I crafted the most embarrassing, sorry for the technical difficulties email I've ever written and sending it to every client I could think of.

• 48:46Wes Bos?

  I had to do damage control, apologize profusely and basically admit that yes, the UX guy broke his own website user experience. Talk about updating my services to we offer development, but only front end development.

• 49:02Wes Bos?

  Oh, I actually, I did that once where I switched over somebody's name servers and I thought I had taken their email down, but it turns out that I didn't. But I was panicking for like probably an hour.

• 49:17Scott Tolinski?

  Yeah.

• 49:18Wes Bos?

  It sucks.

• 49:19Scott Tolinski?

  It does suck. Why has DNS propagation gotten faster? It used to have to wait 24 hours. Like what's changed there?

• 49:26Wes Bos?

  I don't know. How does the internet work? You know, there's, there are name servers around the world and when one of them has changed, they have to let the other ones know. And I've had it in the past where it updates instantly for me, but then you, there's, there's like a website you can visit where it will ping your website from like, like hundreds of servers around the world and tell you where it's pointing to. And you can see if it's updated. So like, it's just such a common thing and the, the infrastructure between all of these peers, as I understand it, has gotten much better. So things. Things update much more quickly.

• 50:02Scott Tolinski?

  Cache ruins everything around me. Cream, get the money. Dollar, dollar bill, y'all. Uh, first day on the job was for a small startup, serving hundreds of thousands of users daily. I picked up an easy ticket, updated some dependencies, ran fine locally, ran the test. CI was fine. PR was approved and merged. I headed out for lunch with the CEO and founders who were in the office that day. Well, a nice little day at the office. Mid-conversation, his phone starts buzzing. He checks it and goes pale. Sites down, white screen. My stomach drops. That two-line PR? No way.

• 50:43Scott Tolinski?

  Turns out, our monorepo had the same package at two slightly different minor versions, locally and in CI. Yarn's cache handled it fine, but Vercel's build didn't use the cache. White screen for a thousand users. The CEO stepped away to make some calls. I pulled out my phone, jumped into the dashboard and rolled back the deployment myself. 12 minutes of downtime. My second PR? Implementing sync pack lint to the CI to catch version mismatches. Oh, yeah. Build cache, bro.

• 51:19Wes Bos?

  People always get annoyed by like lock files in pull requests. Or sometimes they'll just be like, yeah, I'll do it myself. I'll just delete the thing and reinstall it if there's like a, like a thing. But what's happening is that like, there's a possibility that eight levels down your package tree, there's some package that got bumped from 1.0.1 to 1.0.2. And, and it introduces a major bug in your system. I've, I've had it myself once in the past and that's where lock files are, are very, very important.

• 51:51Scott Tolinski?

  Yeah.

• 51:52Wes Bos?

  Next one's called a fiber F-up. I ripped out and destroyed the ventilation system at a US mall, along with $100,000 in equipment from a single retry. A number of years ago, I was working as an embedded engineer for Internet of Things products. This specific product is designed to guide sunlight into building. It consists of a large array of approximately eight-inch lenses and a big stack of fiber wire, uh, led into the building as special fixtures, which is lamps that scatter the sunlight in the room. That's crazy. So it's sunlight coming from outside, being piped through fiber and then into lamps.

• 52:35Wes Bos?

  So like they're lighting the mall without any electricity. It's that easy. Sun hits a lens, travels through the fiber wire and lights up the room. The entire lens, okay, he just explained how it works. Yes. The entire lens. So I didn't read this whole thing because I was like, this is gonna be good. The entire lens array is programmed to follow the sun with fractions of a degrees of accuracy. Everything in these projects is expensive because of the insane tolerances required for good light. But the fiber wire especially needs to be custom made. Normal fiber for internet, audio, et cetera, do not work for visible light. They are made to transmit signal frequency data, not broad spectrum light. Interesting.

• 53:18Wes Bos?

  One of the first of these devices was installed in a mall in the US. The fiber wire was simply fed down into the mall through the air ducts. The code had validation checks for everything, including valid encoder entries. That's the position of the motor, right? For the motor that spun the entire array of lenses. If any value was wrong, the application would just stop. Last minute, we make, these are always the best. Half of these stories are last minute, we made a separate decision. The operating system should just restart any application that crashed. Well, unfortunately, one of the encoders turned out to be faulty.

• 53:58Wes Bos?

  The application would start up, validate everything, start to spin the device, realize the motor wasn't turning and crash. Then it would start up, spin for a bit and crash. For an additional context, due to the insane precision needed, the motor was mounted on a one to 65,000 gear. It took 30 minutes to rotate 180 degrees in full speed. It had crazy torque. It took a few days before the thing.

• 54:28Wes Bos?

  It took a few days before the thick fiber wires finally ripped out large chunks of the ventilation system in multiple rooms and then finally snapping in half.

• 54:42Wes Bos?

  Somehow I made those two decisions all by myself and I did not make the connection.

• 54:49Wes Bos?

  This guy caused physical damage in the magnitude of hundreds of thousands of dollars.

• 54:56Scott Tolinski?

  Physical damage.

• 54:57Wes Bos?

  From one little decision to reboot.

• 55:04Wes Bos?

  That's like fail safe. Fail safes, man, are crazy when, especially when they're real world where they could, like I often think about like self-driving fail safes and all of the code that needs to be written for that.

• 55:17Scott Tolinski?

  I think unfortunately for humanity, I don't know how many of these companies are going to put fail safes over, you know, the dollar. So we'll see how that works out once that all, because there was like some, something where like a electric car got pulled over and it's like the car can't be held at fault. So like, what are they going to do? Like, what are they going to do when something crazy happens with electric cars? I better not hear a dev horror story about a

• 55:44Wes Bos?

  No, I know.

• 55:45Scott Tolinski?

  Or a self-driving car.

• 55:46Wes Bos?

  Last minute, we decided it's okay to hit dogs as long as you're not killing a human.

• 55:52Scott Tolinski?

  As long as the dog is 15 years old, we don't mind. The dog was on its way out anyways.

• 55:58Wes Bos?

  We didn't account for the fact that the API could be down and the gas would never let up.

• 56:04Wes Bos?

  I did one indentation and the thing drove itself into the ocean. Please don't make that happen.

• 56:10Scott Tolinski?

  Our AI access to the self-driving code and had it rewrite it.

• 56:18Wes Bos?

  Awesome. Well, we had more stories than we actually needed, but please keep them coming. Syntax.fm/Spooky. We'll throw the ones that we didn't read today onto the, onto the schedule for next year, because there are some brutal ones that have, have yet to come.

• 56:36Scott Tolinski?

  Yes. Brutal. Yes. What a wonderful and spooky, spooky stories episode. Wes, thank you so much for making this happen live. I thought that was a really neat little tit you did. People who are watching on podcasts, we ended up streaming this live kind of randomly on Twitter. A lot of fun, folks. I hope you enjoyed yourself. If you're watching live, if you're listening to the episode in your car, if you're watching on YouTube, we do this every single year. So if you liked these dev horror stories, there are some millions and millions of dollar cost horror stories in the previous spooky stories episodes. So, folks, if you enjoyed these and you're looking for more bone chilling tales, check out the

• 57:20Scott Tolinski?

  spooky stories episodes on Syntax. There are a ton of them. I don't know. We've done one, two, three, four, five, six, seven.

• 57:29Wes Bos?

  Probably, we've done like seven years of them, and we've done at least two or three every single year. So there's, there's at least twenty episodes.

• 57:36Scott Tolinski?

  Like sixteen or seventeen. Yeah. Yeah, including these two, probably twenty. So check them out. Tons and tons of spooky stories. If you have a spooky story, submit it on our website, Syntax.fm, and we might read it next year. So, shout out.

• 57:50Wes Bos?

  Peace.

• 57:51Scott Tolinski?

  Peace.