How BAD is Test Driven Development?

The gang discusses Test Driven Development and laughs a lot. As usual ;) Don’t forget, ssh terminal.shop

• 0:00Speaker 00

  So today's stand-up, we are gonna talk about TDD. Okay, we're not gonna have some topical topic. We're gonna go after the big stuff, okay? TDD, is it good? Is it bad? Do you use it? TJ's just smiling, looking like an absolute champ in that brief frame. Dude, somehow TJ's the only human I know that gets freeze-framed, and it's a good picture.

• 0:21Speaker 01

  Yep.

• 0:22Speaker 00

  Yeah, look at TJ, look at that Labrador.

• 0:24Speaker 02

  I can't handle this right now. I cannot handle this right now.

• 0:31Speaker 00

  Alright, Casey, you probably have the strongest opinion, potentially, out of all of us.

• 0:36Speaker 01

  Okay. I doubt that, but okay.

• 0:40Speaker 00

  Alright, cuz I also have a strong opinion, but I'm...

• 0:42Speaker 01

  It's probably you, then.

• 0:43Speaker 02

  Yeah, I think Prime, you gotta lead up the strong opinions here.

• 0:46Speaker 00

  I'll lead up with the strong one, which is, so, I recently just tried another round of TDD.

• 0:51Speaker 01

  Prime, Prime, you have to say, "Hey, everyone, welcome to the stand-up. We got a great show for you tonight. I'm your host, the Primagen. With me, as always, are TJ, hello, Trash, hey, everyone, C. Muriatory. Great to be here, Prime, thanks. Today's topic is test... Have you ever heard a podcast before?

• 1:16Speaker 00

  Thanks, everybody, for joining. You're watching the hottest podcasts on the greatest software engineering topics, The Stand-Up. Today, we have with us Tej, who's in the basement fixing something.

• 1:29Speaker 02

  Is that a Home Depot ad?

• 1:31Speaker 00

  Look at him, he's helping us, his router. We have with us Trash Depp, broke a toe, winning a judo competition, still living off that high. And C. Muriatory, the actual good programmer among us. Oh, alright, I got a lot to live up to. Yeah, you got... Don't worry, it's not hard to live up to, at least in this crowd. I'll kind of start us off on this one. So, we've been developing a game called The Towers of Mordoria. We came up with the name from an AI, and AI generated that name for us. And effectively, I decided that, at one point, I have this kind of really black box experience. I have a deck of cards that I need to be able to draw cards out. When they're played, I either need to go to the discard pile or the exhaust pile. When there's no more cards in my hand or not enough to be able to draw cards,

• 2:14Speaker 00

  I need to be able to take all the discard cards... Dude, Tej is just flexing on us, touching grass right now. Put all the discard cards, shuffle everything up, and then draw new cards back out, replenish the draw pile. Right? This sounds like a TDD pinnacle problem. There's black boxes, there's a few operations, I'm gonna be able to do this. And so I just did this like a week and a half ago on stream. It was amazing. I did the whole thing, designed the entire API in the test, then set up all the tests, set up it all to fail, then it did work, or then it failed, then I made it work, then it worked, and it was great, and everything was perfect, and I was like, there we go, we just did it. I just TDD'd and it was a good experience. So then I took what I made, and I went to integrate it into the game,

• 2:58Speaker 00

  and realized I made an interface that's super good for testing, and not actually good for the actual thing I was making, and I had to redo it. And so it always, you know, that's kind of been a lot of my experience with TDD, is that either the thing is simple enough, and is super black box enough, that I can make it, and both the testing interface and the usage interface is the same, or I create an interface that's really great for testing, and it's actually not that great for the practical use case. And I know people talk about how it's a great way to develop your, you know, your architecture and all this kind of stuff. I've just yet to really buy it, but I will say the thing that I do like doing is on problems that are hard to develop and have many moving constraints, I do like to co-create tests with it.

• 3:42Speaker 00

  Meaning that I come up with an idea, how do I want to use it? I program it out, then I go create a test, which is backwards from TDD, go, okay, this is what I want to see. Am I even correct on this? Can I get a fast iteration loop being like, no, no, no, yes, okay, got it. Next thing, I want to make sure it does this. No, no, no, yes, okay, got it. And so I do like the fast iteration cycle of getting tests in early, but few problems feel like I can actually have that. And I feel like this, all things need to be tested immediately and need to be written first. You just write dumb interfaces, everything becomes inversion of control, everything's an interface, and you just have this abstract mess at the end of it, because testing was the requirement, not, hey, use it when it's good.

• 4:26Speaker 00

  So that's my TDD experience, and I feel fairly strong about that. And if you say TDD's good, I feel like I'm gonna jump kick you.

• 4:36Speaker 02

  I can respond to that, or Casey, if you have strong feelings.

• 4:39Speaker 01

  No, I think Trash, let's go to Trash take first. Yeah, no blockers on my end.

• 4:49Speaker 02

  First, you couldn't like deter me from doing any task than to tell me I need to like start off by writing tests first off. Second, I kind of had a very similar experience recently building like a CL.

• 4:58Speaker 00

  Hold on, can you rewind for a second? What, what did you just say for your first point?

• 5:03Speaker 02

  I said, "You couldn't deter me from doing work if you let off with me having to write tests first."

• 5:10Speaker 00

  I don't even know what that means.

• 5:12Speaker 01

  You mean you couldn't deter me anymore? Like that's the most deterring thing or?

• 5:17Speaker 00

  Man, am I stupid?

• 5:19Speaker 02

  Can you try using a different word to describe what the issue is? Okay, let's, let's reword this. Flip, please take this out.

• 5:26Speaker 01

  Flip, leave it in.

• 5:27Speaker 02

  I don't like test-driven development, however.

• 5:29Speaker 01

  Okay, got it.

• 5:30Speaker 02

  However, however, I just built something recently at work, which was very kind of similar to what Prime was saying.

• 5:38Speaker 02

  I ended up doing a little bit of test-driven development, but

• 5:44Speaker 02

  you wouldn't notice this, but I'm actually really kind of passionate about testing. So I'm gonna kind of like, when I test, I would say

• 5:53Speaker 02

  2% of the time, I actually end up with test-driven development and the rest is usually post-implementation because one,

• 6:01Speaker 02

  requirements and all these things change afterwards, right? So it's like, why write tests for things that are potentially gonna change and you don't really know what's gonna happen? But I do write tests for very granular things that I think are gonna be used all the time. So for instance, like if I wrote a parser for like my CLI, the parser's gonna be reused by multiple commands, probably never gonna change. So I typically will write that and I'll also also write tests for things that take multiple steps to test. So like, you know, people argue for end-to-end tests because you test just like a user. So that's kind of how I like to develop. I like to use it like I would use it for anybody else. But if I gotta do like three steps to get to the point I want to test, I'm like, alright, screw this. I'm just gonna write a test specifically for this like instance and then I save time that way.

• 6:46Speaker 02

  So that's usually the time I arrive at test-driven development is specifically in those scenarios. Otherwise, it's purely just like, you know, uh, what do you call it? Integration testing and stuff like that because you end up with stuff like, like Prime said, like you, when you write tests, you write everything so granular and everything works so modular that you just don't even know what it's like when you actually put them together. So that's why, that's kind of like the way I approach things. That being said, like people that kind of like

• 7:12Speaker 02

  Live and Die by Test Driven Development? I don't, I don't get it. But I was hoping someone would be pro TDD in this group, so I could argue with them.

• 7:22Speaker 02

  That's all I got for right now.

• 7:25Speaker 01

  I'm, I'm, uh, pretty much right there with trash on this one. My, my feeling on TDD is pretty much the same as my feeling on OOP, right? The problem is not if you end up with an object that, or something that looks like an object in your code in OOP, that's fine. The problem is the object oriented part, the like thinking in terms of that wastes a lot of your time and leads you to bad architecture, I think. TDD is the same way. I think testing is very good, but test driven development, meaning forcing the programmer to think in terms of tests during development, like that is what your primary thing is, is I'm making tests to drive the development, the TD part. That's actually the problem. I think the reason for that's pretty straightforward.

• 8:08Speaker 01

  Uh, and it's kind of weird because usually the people who advocate for TDD are the same people who would poo poo any similar suggestion, like any suggestion that like, oh, we should measure performance during development. They'll be like, that's ridiculous. You're wasting so much program time. Like, what about the 8,000 tests that no one ever needed because we ended up deleting that system anyway? And they're like, well, you know, test driven development is very good. So I think like, the problem with all of those sorts of things, any of them, is just putting emphasis on something instead of talking about it like what it really is, which is a trade off. If you're spending time making tests or you're orienting things towards making tests, that is necessarily programming time that isn't being spent doing something else.

• 8:53Speaker 01

  Like, programming is zero sum. If you spend time doing one thing, you're not spending time doing another thing. And so, for example, if you end up spending a lot of your time making the interface revolve around tests, you weren't spending a lot of your time making the interface revolve around actual use cases, because actual use cases don't usually look like tests. Therefore, sometimes you get the exact situation that Prime was talking about, where you do test driven development and you end up with a completely unusable mess, which is actually very common in my experience, right?

• 9:20Speaker 00

  I wouldn't call it a mess. You know, I write pretty good. God, I mean, it was like pretty dizzy. I just want to say.

• 9:24Speaker 01

  No, the API, the API is a mess. Not the, not the code, right? Like, the code has been tested and it probably works. But the API is a mess, right? I mean, that's what happens. Uh, because it wasn't designed specifically for the kinds of use cases it was meant to tackle. And so you end up with that problem. And again, it was shifting the developer's focus was all it took to make that happen. Good APIs are hard to make in the first place, right? So if you're adding more complexity to the programmer's workflow when they're trying to make an API that's good for the, you know, the actual use case, you're just going to get a worse result. Again, zero sum. So what I would say is like, testing is better something that you do as the thing takes shape. When you're like, okay, we now have figured out like how this system works properly.

• 10:09Speaker 01

  We're really happy with it. The API is working nicely. We checked the performance and we think we've got like reasonable, we've got a reasonable path towards having this perform really well. Now let's start seeing what are the, what is the overlay? Like, what are the parts we can now start to add some little substrate to, to test that we think are going to have bugs that will be hard to find? Because that's usually the way I look at like, what are the parts of this system whose bugs will be either very hard to find or really catastrophic? Like, they'll cost us a lot of money or they'll crash the user's computer or whatever, right? And then you put the tests for that, right? And that's how I've always approached it. And I think that works pretty well. And, uh, you, you do want to do that step. But I don't think you want test-driven development. It just seems like a really bad idea.

• 10:54Speaker 01

  And to be honest, when I use software that comes from people who are like, TDD, I believe this organization, CD, whatever, it's full of bugs all the time. Right? Like, I'm not getting these pristine software things. I mean, I'm, I would imagine that the people at YouTube do a lot of testing and code reviews. And yet for the past eight years, the play button cannot properly represent whether something is playing or not. For eight years, right? Sometimes it's paused and it has the play thing. And sometimes it's playing and it has the play thing. And, and sometimes both of those times it shows the pause thing. It's just, it doesn't seem like the testing is working, guys. So again, I'd say, you know, I think you have to kind of rethink how you're allocating that time. That's what I would say.

• 11:38Speaker 00

  Can we get TJ in here? Can, can, uh, Playboy TJ? We don't know this one.

• 11:42Speaker 03

  Can we? Am I, am I coming in loud and clear?

• 11:45Speaker 00

  Yeah, your voice sounds good. Your voice sounds good. You're looking good. The brass looks good.

• 11:51Speaker 03

  Thank you, thank you. So I would, I'm, I'm, uh, I think I agree mostly with what everybody is saying, although I would say for a few kinds of problems. Uh, the problem is, I, I, I think people, when you say TDD, they just mean 10,000 different things. So this is like one of the problems, right? There's like people who are like, TDD is only if you do red, green cycle before you write the code. You have to write the tests and like, you can't write any code till you've written a failing test and blah, blah, blah. Which like I get, uh, that, that maybe they're the people who invented the word so they can define it that way. Uh, but like, I find that not, not very useful for some stuff.

• 12:30Speaker 03

  But, uh, there are like testing or like projects where I think I test like basically everything. And there's just ways that you can do it that are like more effective versus less effective. Like my favorite kind of testing is if you can find a way to do like snapshot or like golden testing, whatever, you know, people have different names for it. But where you can basically just write a test and then say, assert that the output is the same as it used to be. Right? Uh, like, so in you, I usually I've seen those called snapshot tests or expect tests. I find that those are like super.

• 13:05Speaker 00

  Often.

• 13:06Speaker 03

  Yeah, yeah. Or like people will use them in regression, but like the kinds that I've done it before, like, uh, maybe I'm building up a complex data structure to like represent the state of like a bunch of users, or maybe it's the parse tree of something, or maybe it's the type system, like whatever it is. If you can find a way to represent that in like a human readable way, usually just by like printing it in tests that I've done. You can just like compare the diff between this one and before you made the changes. And they're, they should be fast at least. If they're not fast, then that's a separate problem. Like your tests have to be fast for people to run them, otherwise they don't run them.

• 13:44Speaker 03

  Then then those kinds of things tend to be, I find like very effective and give you a lot of confidence that you haven't like scuffed anything throughout the system with your changes. Right. Because you're sort of like asserting the entire state of something. And if something changes, it's really easy to update because you just say, oh, snapshot, update this. Cool. Done. One second, all your tests are updated. You walk through them and change them. You get some other side benefits too, like in code review, you literally see the diff of the snapshot and you can be like, um, that looks wrong. That used to say false and now it says true. Those things that shouldn't have been flipped between those. So there are like certain problems where I've employed like

• 14:26Speaker 03

  nearly TDD in this snapshot style, in the sense that the snapshot always fails on the first time you run it. Because there's nothing to expect. And then you accept that snapshot if it looks good or keep changing the code until it does look good. And then you like move on to the next thing. So but that but it's kind of like a unique case. Like I wouldn't want to snapshot test a like

• 14:51Speaker 03

  HTML page, right? It would just like, there's no way that they're reproducible, it feels like anyways. Uh, but like, so there's, there's, it's only for like certain subsets of problems which have been more effective like for me because I've made a lot of dev tools in my career and stuff like that. So you want to do like verify that all of the references of something look like XYZ. Okay, very easy to do with snapshots.

• 15:18Speaker 03

  That's so that's kind of the main, the, the one thing that I wanted to throw out there is like that style of testing I find really powerful. And I've used it a lot, like across projects you wouldn't really expect as well.

• 15:28Speaker 02

  I've always had the opposite experience with snapshot tests. I, I feel like it's okay, maybe if like it's you and maybe some other people that actually know the expected output of a snapshot. My experience is It changes and they're like, oh, let's just update it. And then the test pass again and then you're kind of just left with this snapshot and you're just like, alright. So it's to the point like where it's like, alright, let's just get rid of them because they're just changing every time requirements change and no one actually knows what the expected output is. One more point I want to make, someone said something about like test helper refactoring. I think that's true after a product is actually mature. Because you're just going to be, it's kind of like a snapshot test where you're just going to keep updating your tests every time something changes

• 16:12Speaker 02

  to the point where it's kind of pointless. So I always find when tests are like that, I usually just delete them immediately. Okay, go ahead, Trash.

• 16:19Speaker 00

  Alright, alright.

• 16:20Speaker 02

  I called you Trash.

• 16:20Speaker 00

  Yeah, I know you call me Trash.

• 16:21Speaker 01

  Your, your comment filled Prime with so much joy, it translated into physical up and down motion.

• 16:29Speaker 00

  Yeah.

• 16:29Speaker 02

  I, I was literally watching him as I was talking and I was like, let's just see how hype we can get Prime.

• 16:35Speaker 00

  Yeah, dude, I was high.

• 16:37Speaker 03

  I wish I could see you guys. I see nothing.

• 16:41Speaker 00

  Yeah, yeah, we see you. We see you guys. Yeah, we could, we could see you, DJ, enjoying grass.

• 16:46Speaker 01

  TJ, if you see a strange mushroom in the grass, do not eat it.

• 16:50Speaker 02

  Oh yeah, don't eat it.

• 16:51Speaker 00

  Do not eat the strange mushroom, DJ. When you're mowing, there's a strange mushroom.

• 16:55Speaker 03

  Guys, do I, I don't have my AI glasses. Do I eat this or not? Do I eat it or not, guys?

• 17:02Speaker 00

  No, don't eat the mushroom.

• 17:04Speaker 03

  Eat the mushroom, that's all I heard. Okay, I'm going for it. Prime, say your thing.

• 17:11Speaker 00

  Alright, so the reason why I actually agree with Trash a lot on this one is that I've done a lot of snapshot testing and I think there's probably does exist a world where this makes a lot of sense. It's just all the times I've ever done it. Whenever a requirement change or I have to make a change that warrants changing the snapshot. I've also had to make some decent, some non minor change also to some logic throughout the program. So when my snapshot breaks, I also don't know if I got it right. So now I'm like trying to hand eyeball through this, this thing like one at a line like, okay, is this, is this really what I meant it for it to be in this moment? How do I know I should change this from true to false? Like I have to re, I have to re reason about everything and I find that true snapshot testing on big things is, is, feels very difficult.

• 17:56Speaker 00

  Though I love it, like I want, like ultimately, that's what I always want. I fall into the hey, let's just make a golden trap constantly. Like that is like my default go to is just like, it's golden time. And then I always get sad. As far as like the whole um, unit test breaking and mature product thing, I just find that testing generally, I do not think testing helps for refactoring unless it's a true end to end test. Like you use the product from the very tippy top and you only have the very bottom of the output. You have nothing else you look at. But often I don't like, that's really hard to test because I'll write a function that I know is very difficult to get right. And I just want to test those five things. So I'm gonna make a, I'm gonna make a test that's highly coupled to that

• 18:39Speaker 00

  piece. And if you decide to refactor it, yeah, the tests all break. Like that's just a part of it. I wrote it so that this one thing works this one way. And I wrote a test that runs that one thing to make sure you didn't screw it up if you had to make a minor bug fix to it. I don't try to do that whole um, I just really think this whole idea of unit testing and oh, it will prevent you from having headaches while refactoring, has just been alive from the beginning. And then what you'll hear is people do the exact same thing every time. Oh, that's because you're testing at the wrong level. It's just like, I'm just testing the things that I want to test. I don't test all the things. And what happens if you're testing on the wrong level just simply means that you broke a whole bunch of tests, but then some of your tests didn't. And that gave you the confidence that you did the right thing. And I just don't, I just am not going to

• 19:23Speaker 00

  test the universe. I'm gonna have a few end to end. I'm gonna have the hard parts and that's that. No more.

• 19:29Speaker 01

  I also think like, I mean, when I'm thinking back to times when I've, when I've done testing, uh, like, a lot of times when I, when I will break out testing, it's for things that I know aren't really all that refactorable or changeable anyway. It's like, okay, I, you know, I made this memory allocator. And we pretty much know what it does, right? You have these like two ways you can allocate memory from it and you have one way to free or something like that. And then I just have these tests to make sure that, you know, it doesn't end up in bad states or whatever it is. And, and that's a really useful thing, right? I've definitely done that. I do that for math libraries. Like, we kind of all know what the sign of something should return.

• 20:10Speaker 01

  So if I'm making a math library that implements the sign function, that's not like, I don't have to think too hard about refactoring sign because math has been around for like, you know, hundreds of years and isn't going to change tomorrow. Like, hell, bad news, guys, like React updated and now the definition, the mathematical definition of sign is different today. So I'm so happy that's not even a possibility, that React does stuff with math because we'd be screwed.

• 20:34Speaker 03

  You know, guys, there's still time. There's still time.

• 20:37Speaker 01

  Yeah. So, uh, so I do, I do like them for that. And, and actually, like, there was a, I can think of a specific time, uh, when I was shipping game libraries, when, uh, we had a really bad bug, uh, that caused a problem for one particular customer that took forever to find. And it was because it was in a thing that I had added to the library just for that customer as like a backdoor, right? They, they like, we need this one thing really quick. And I was like, okay. And I put it in there and it had a bug in it. And of course it wasn't in any of my standard testing or anything. It wasn't even really in anyone's use case, right? And, uh, and so like, you know, that was a, to me, I was like, okay, that's a pretty,

• 21:21Speaker 01

  that's pretty good evidence that like the testing that I was doing before was good and like wasn't a waste of time because like the one time it didn't happen, it caused a significant problem, right? And so, you know, presumably I would have had more of those in actual other customers had the main, had all the stuff that normally shifts to the library not been tested in the way that I thought it should have been. And, you know, from then I was like, so like, you know, don't, never do a one-off ad. Like, if I'm going to add something for a customer, I'll add it like the right way and actually put in the testing for it, or I'll just tell them, I'm sorry, I can't do that right now because, you know, I'm too busy or whatever. And so I do think like, yeah, when you can actually talk about this is isolated, this is what this thing does, I'm pretty sure it's not going to change.

• 22:06Speaker 01

  I think the tests are great, but before then, yeah, it's just, and also. This was just kind of going off on a random anecdote, but just to circle back. The thing that Prime was saying is absolutely true. It's like refactor testing? Not really. Like, that's only... You can only really get like a very high level confidence from that. You can't really know. There could be all sorts of things that aren't really working. Because whole system testing like that is gonna miss... It's gonna miss like weird edge cases that can overlap. And the only way you catch those with test-driven development is knowing about those edge cases and making tests to kind of target them. And you can't do that if you just refactor everything, because now the edge cases are different, right? So, yeah, I think TDD people overstate the degree to which that really works.

• 22:50Speaker 00

  That's why I'm a huge fan of PDD, prog-driven development. I think that every time you release and you have a bunch of people that use your product, you should do a slow release, you should measure the outcome, because it's just gonna test all the weird things you're never gonna be able to think of or set up. And if it starts breaking, you know that you screwed up. Previous version worked better than the new version. You can't release this new version. You gotta back it up. Not P Diddy, PDD, okay? Stop saying P... There's no baby oil in this situation, okay? But I mean, to me it's like-

• 23:24Speaker 01

  Shots fired at Sonos, or Sonus, or whatever that company is called.

• 23:28Speaker 00

  I don't know what they did.

• 23:30Speaker 01

  I don't know what they did.

• 23:31Speaker 00

  Yeah.

• 23:32Speaker 01

  You guys are supposed to be the web dev people. What's going on?

• 23:37Speaker 00

  What the hell are you talking about, Casey?

• 23:39Speaker 01

  Sonos, the company that makes the speakers.

• 23:44Speaker 01

  Nobody? Boz? Are you kidding me? You guys are the web dev guys!

• 23:49Speaker 02

  Beats by Dre?

• 23:51Speaker 01

  No, this is legendary. Like, everyone knew about this. So, there's this company called Sonos. They make like the most popular internet-powered speakers in the world. Like, everyone has these things. I mean, not me, because there's no way in hell I'm connecting speakers to the internet. That's just a way to make them not work, which Sonos went ahead and demonstrated for me. They had a product that everyone liked. They were extremely popular speaker. They then did a big software update where like they were like, "We're rolling out our new system." Like, you know, kinda like how Google does every couple of years. They're like, "Hey, guess what? Gmail's gonna suck way harder than before today." Like, you have five, you know, you'll get five months of being able to switch between them and then you're on to the new bad one. They did that, only there was no five months. They just basically said like, "Here's

• 24:34Speaker 01

  the new rollout." And it was awful. Like, tons of feature regressions. It didn't really work. Customers couldn't do most of the things they used to. And it like tanked the company. Like, it was just like, immediately, everyone hated it and nobody wanted to buy their stuff and they had to do this walk back. And they couldn't roll back because they had redeployed all their servers and I guess they didn't know how to like, undeploy them. So they couldn't, you couldn't go back to the old app. It was, "Go read about this. I can't believe no one knows about this but me." It was a massive failure.

• 25:06Speaker 03

  Casey, I know it. I know it. Never heard of them. I heard it. Casey, I don't know if you guys can hear me, but I heard that. Okay.

• 25:12Speaker 00

  We can hear you. We can hear you, TJ. We can't really see you. You have a very few expressions.

• 25:17Speaker 03

  That's fine.

• 25:18Speaker 00

  But I do, okay. Your point is that hardware is definitely different than software. You know, if Twitter has a small breaking thing because they made something, Twitter, all these major sites, they break all the time, right? I have much more forgiveness for that. This is, you know, that's obviously different than, you know, you gotta play the field a little bit different. When I say break production, if I'm building a game that only, you know, so many people play, you don't want it just crashing non-stop while you're testing features. There's probably some levels. But if you're Twitter and all of a sudden you can't post for five minutes and only .001% of the people couldn't post for five minutes, it's just not gonna make a huge splash and it's probably not gonna deter them from actually using the product again.

• 25:58Speaker 01

  Well, I was actually, no, I was agreeing with you with the Sonos thing. What I was saying was, they should have done exactly what you're talking about. They should have And deployed a small server bank for what they were doing.

• 26:09Speaker 00

  Oh, I see.

• 26:09Speaker 01

  Given it to 1% of their users and their users would be like, this is utter garbage and I do not want it. And they would have been like, okay, pull that on back. Nobody saw it. We're fine. Right? So exactly what you're saying, I was totally agreeing with you. I'm like, so no, if they had done that, they would have totally saved their company, literally like six months of hell. Had they just given a slice of users this thing for a while, they would have been, you know, some significant percentage above beta test, right? Like, so like you said, you know, 5, 10% of the people got this. They would all been like, nope, nope, nope, nope, nope. And then they would have like, you know, been like, okay, okay. So.

• 26:46Speaker 01

  That's all.

• 26:46Speaker 03

  I do want to, can I, can I say one thing just about the testing part though? Is that okay? From before?

• 26:53Speaker 00

  Maybe.

• 26:53Speaker 02

  Sure. That's what the podcast is about. Exactly.

• 26:56Speaker 00

  I like this. I like this. Go for it, TJ.

• 26:59Speaker 03

  My favorite project that I've ever worked on, NeoVim, by the way.

• 27:04Speaker 00

  Can you go back to the grass? You were much more like better in the grass.

• 27:08Speaker 03

  Really?

• 27:08Speaker 02

  Your house is blocking the towers.

• 27:10Speaker 03

  I got a notification that said, your phone is overheating because I was in the sunshine.

• 27:15Speaker 02

  I was actually gonna bring that up in the beginning.

• 27:18Speaker 03

  Like it literally said, your phone is overheating.

• 27:20Speaker 00

  Your body's the one covering. Okay. All right. Sit over, create a shadow TJ. So they can't, the sun can't get your phone.

• 27:28Speaker 02

  You gotta plank, plank over your phone.

• 27:30Speaker 03

  Hey, what's up, boys? Hey, yo, what's up, boys?

• 27:32Speaker 01

  This is like getting into like TJ profile pic territory where it's like on the dating app. Can you tell us some of your pet peeves, TJ? Do you like long walks on the beach?

• 27:42Speaker 03

  I'm a, I'm a long walks enjoyer. I'm really not a fan of short conversations. I want to really find out about the real you. Um, I don't know what else.

• 27:53Speaker 03

  I don't even know if any of my words are coming through.

• 27:55Speaker 00

  TJ is actually just describing himself right now. That's all TJ is doing.

• 28:00Speaker 03

  I love laughing. I love having fun. I like to make you laugh. This is just me to prime. This is my pitch for getting on the, on the podcast.

• 28:10Speaker 00

  I'll make fun of trash.

• 28:13Speaker 03

  Yo, if, if there's a guy named Trash on the podcast, I will make sure to make fun of him every episode. He uses one password. Are you joking?

• 28:24Speaker 00

  Alright, alright, alright, alright. TJ, what's your take?

• 28:26Speaker 03

  My take is my favorite project that I've worked on and like definitely the one with the most weird edge cases and probably users, Neo Vim. Like, the test sweep for Neo Vim is amazing. We have tons of, we have different like, styles of tests, we have different ways to run them, we have all this different stuff. Tons of work went into making it. And maybe that's just partially because Neo Vim is like, trying to maintain compatibility with Vim on some stuff like literally forever, right? So there's some other requirements that are different. But man, every time I did a PR and a random test broke, it was my fault. It was not the other tests, it was not the snapshot tests, it was nothing else. Like, and like random

• 29:10Speaker 03

  stuff. I mean, maybe it's just because Neo Vim's also like a really old C project. So you like, change something and then a random global is destroyed. I don't know, you know, it's like, whatever, it's maybe not always the best, but like, so I don't, in, in my experience where people like actually cared about it and like cared to maintain snapshots and make those happen, like, no one's even getting paid to work on Neo Vim. You know what I'm saying? And like, the snapshots were valuable, the unit tests are valuable, the functional tests were valuable. And I used them in a ton of different PRs and across like really large refactors, adding Lua to auto commands, like doing lots of different stuff where I had to change tons and tons of the code. I was just wrong every time until the test went green.

• 29:54Speaker 00

  I think that probably goes to show that tests are as valuable as the people who wrote them. I was literally gonna say that.

• 30:01Speaker 03

  Yeah, that's what I've been waiting to say too, is, I just wanted to say, but since I've been lagging, I didn't want to yell. I just wanted to yell "skill issue" to everything Trash has been saying.

• 30:10Speaker 01

  I would, I would disagree, actually. I will defend Trash.

• 30:14Speaker 03

  Yes!

• 30:14Speaker 01

  Actually.

• 30:15Speaker 03

  Casey's smarter than you.

• 30:16Speaker 01

  Because here's the thing.

• 30:18Speaker 03

  That's true, dang it!

• 30:20Speaker 01

  I suspect, I suspect that some of what you're seeing, though, is not so much the skill of the people who did it, although they may have been very skilled. It's more just about when the tests were added. So, at least for me, a lot of times what I'll do is I'll go, okay, if I, like, let's say I have a bug that I actually work on for a day. Like, like, this is, because most bugs are like, oh yeah, I know what it is, and you just fix it, right? But you get one and you're just like, what the hell is happening? And you actually, it's like that once every few months where you're just like, okay, this is really weird and I'm not sure what it is. When I find those, I typically try to add a test, or assertions, or whatever I think would have caught it.

• 31:05Speaker 01

  Right? Because I'm like, okay, this one's nasty, and I wasn't expecting it. What can I add that will trip this kind of thing up? And I'll add it. And I suspect that much of what gets seen in like, mature products like NeoVim are just like, hey, if you've got some post-test discipline, like, after we made the product, when we're finding bugs, we add the test that would have caught the bugs. Those are often very good. Because they come out of, what are the things that are likely to break when we do stuff? And we put in a net there to catch them. And so, those are like, very good. And they're the opposite of TDD. They're like, oh, they're TDD! They're test-driven debugging. Uh, maybe you could say, or, or something like that.

• 31:49Speaker 01

  But, but, uh, so, so I do think there's something to that, uh, that, that's a lot more. I had a couple other hot takes that I'll get to in a second, but I just wanted to leave it, I wanted to say that one, based on what Tej said. I don't know if that's true about NeoVim, but I, I might suspect it, right?

• 32:02Speaker 03

  It definitely is. The thing, though, that like, I was more pushing back, I was like, I find the snapshot tests in NeoVim invaluable. Like, I, they're amazing. I find a bunch of these good end-to-end tests. Truly end-to-end, like, literally, it runs a test and does user, I don't know if I just dropped, uh, it does like, user input, and then asserts the outputs. Right? Like... Though I found those super valuable in Neovim, probably just dropped.

• 32:29Speaker 01

  Yeah.

• 32:30Speaker 00

  I also, no, you didn't, you didn't drop. We saw, we heard your point. I think also to put on top of it is think about the, the horizon of Neovim versus you starting up a new dev tool at work. And maybe there, maybe there's something to be said about when are the Goldens introduced during exploration phase or like Neovim is just not going to be changing. My assumption is that a lot of Neovim doesn't change super hard. It's not like, hey, guys, let's do a whole new data model today.

• 32:57Speaker 03

  Wrong.

• 32:58Speaker 00

  Changes all the time.

• 32:58Speaker 03

  All the ones, that's all the ones that ever, everything you guys have poo-pooed on that I found them the most valuable. When I changed literally the entire data model and how all auto clans work all inside of Neovim, it was the most useful to have the snapshot tests then.

• 33:12Speaker 02

  I mean, your testing is going to depend greatly on what you're building 100%.

• 33:17Speaker 03

  Sure. Yeah. Yeah. But I'm just saying like, we were fundamentally changing the data models.

• 33:21Speaker 02

  Yeah.

• 33:22Speaker 03

  Yeah. I'm just saying, that's what happened.

• 33:26Speaker 02

  I thought it was a pretty smart statement. Come on.

• 33:30Speaker 03

  You're really smart, Trash.

• 33:31Speaker 01

  I wouldn't poo-poo that. I wouldn't poo-poo that at all. I, I, you've noticed I have not poo-pooed snapshot testing.

• 33:37Speaker 03

  True. Good job, Casey. I love you. You're smarter than Trash.

• 33:44Speaker 01

  I love that I'm now getting gung-hoed on it. Can we start rearranging the windows based on who Tej thinks is smartest at any given time? And I'm guessing that it has a very high correlation to who is agreeing with him. It's, it's like I'm getting that sense.

• 33:57Speaker 03

  No, I'll just always put Trash at the end. It's fine, Casey. You don't have to worry.

• 34:01Speaker 02

  Today I'm going to tweet that Casey protected me from Tej.

• 34:04Speaker 01

  Yes.

• 34:04Speaker 02

  This is a pinnacle moment of my career.

• 34:07Speaker 03

  I can't even see your guys' videos, so I have no idea what's happening. I'm talking to a black screen right now.

• 34:15Speaker 02

  I've been just cracking up at your feed this whole time. I just can't handle.

• 34:19Speaker 01

  It's pretty good. It's very good. Like the result is, is top notch of the feed.

• 34:24Speaker 00

  Unfortunately, I think this will be a stream only feature. I don't know if TJ will actually have this effect on the video because Riverside will be uploading it.

• 34:31Speaker 01

  That's true. Yeah, that's true. He'll still be in, in weird like, uh, it is, uh, profile, is dating site profile pick phase, but, but it will be much smoother. Yeah. So, so anyway, uh, I, I wanted to add, I, I, I would agree that snapshot testing can be good. It depends, it just depends on whether those snapshots are going to be stable. I mean, that's the bottom line. If you have. If you're, I would imagine what Prime and Trash were talking about is if you're trying to do snapshot testing at a level below something that's actually consistent across the refactor, obviously that's not going to work, right? And so if your snapshot testing is like someone mentioned a parse tree, well, if it's a parse of something that could be parsed different ways depending on how you chose

• 35:14Speaker 01

  to implement that subsystem, those snapshots are useless. If the parse always has to be the same because it's like parsing for the C language specification and it can't change because like, you know, it's hardcoded, so if the thing's in C99 mode, then it had better produce a exact parse tree, that's a great snapshot test, right? And so I think the difference boils down to those things. It has to be about something that actually is, you know, not going to change, item potent to the, to the release of the project, otherwise it's a limited time test. But, you know, that's just how it is, so.

• 35:45Speaker 00

  Probably also the scope of the breaking, right? If only 10% of your golden is invalid because you made this change, that's a lot different than 100% of your goldens are invalid and you have to like rework all of them because of big kind of refactors. That was my last, my last experience is that I had to do a bunch of testing against this games reporting thing at Netflix. And to be able to do this autodoc and to be able to do it, I had to like create the data model of the incoming events. And then they were changing the incoming events like once a week because it was still in exploration phase. So it was just like, well, not only is my input incorrect, now my output's incorrect. So I have to like change both sides. So I'm just like, I guess I'm just redoing all the things again. Here we go.

• 36:24Speaker 01

  Yep.

• 36:24Speaker 00

  And so then it just became super hard to know if I was actually right or if I was breaking things.

• 36:29Speaker 01

  So I, I think at the outset, I made it pretty clear, I don't like test-driven development as an idea. I don't like focusing the developer's attention on tests. I think they should understand tests and know how to deploy them smartly, but I don't think, you're right. I do want to say there is a place where I do recommend literal test-driven development. And it is as follows. That is, if the programmer is finding that they are low productivity on a particular task and that task does not provide feedback. So, for example, if you have to implement some low-level feature component that has no visual output, no auditory output, no anything,

• 37:12Speaker 01

  it's just like an abstract thing that happens inside the computer, right? Then sometimes programmers, like, they get bored working on these sorts of things. But a lot of programmers will like focus on things like numbers that come out of a thing. Like if a thing, if you run a thing and the number like, you know, 37% pops out, suddenly the programmer's like super motivated to get it up to like 45% and 50%, and so, right? So sometimes it's like, oh, okay, if you can turn this problem, if you can make a test that turns whatever you're working on into something with a concrete output that you feel some reward about when it goes up or improves,

• 37:57Speaker 01

  that can be a valuable use of test-driven development. And we do this in performance a lot. We'll put, make a little test around something that outputs things like, "How many cache misses did it have?", or "How many, you know, cycles did it take to complete?", or whatever. And it's like, "Ooh, now I'm like trying to get that number down", and that's a lot more fun than I'm just trying to make the system faster, or whatever. So like, a lot of times you can get a, you can make it so that those things go together, and then you get the test, or the good statistics gathering, or whatever it is that you actually wanted anyway, and you get this motivation for the programmer, and it, they all kind of work together in this nice way. So I just wanted to put that out there. There's one place where I think it does kind of work to drive development by a test of some kind.

• 38:42Speaker 02

  First thing that comes to my mind is like code coverage reports. And you see that number really low, and I have the opposite reaction where I just don't care, and I want to do it.

• 38:54Speaker 02

  I don't think I've ever saw like a low code coverage report and I was like, you know what? Today, we're making that number higher. I'm kind of just like, guys, we suck and I'm not gonna help us get better. There's no way.

• 39:07Speaker 01

  Oh, no. Alright.

• 39:08Speaker 00

  I, I mean, to be fair, if I go to a place and it's like 100% code coverage, I also go, uh-oh. Right? Like, I'm also kind of worried about the 100% code coverage, uh, code cases as well.

• 39:19Speaker 02

  I'm not gonna say the company I was at. I've been at many.

• 39:21Speaker 00

  Netflix.

• 39:22Speaker 02

  And there was a team that required 100% test coverage to the point where people were just like writing the most worthless tests just to get like this like PR bot to just shut up and was

• 39:39Speaker 02

  and to make it even like funnier, there was like bugs every day, like in production. It's just like, I was just like, what's happening here? I don't know if anyone's ever experienced like 100% like test coverage quota by their team.

• 39:50Speaker 00

  Yes, I did it with Falcor when I was at Netflix. 100% code coverage, bugs every day of my life.

• 39:54Speaker 03

  Oh, for Falcor, the one line production takedown?

• 40:02Speaker 01

  Wait, Fa- do you say Falcor?

• 40:05Speaker 02

  Uh huh.

• 40:05Speaker 01

  Like the luck dragon from the never ending story?

• 40:08Speaker 02

  Yeah.

• 40:10Speaker 01

  What? Why was it called that?

• 40:12Speaker 02

  You don't know what Falcor is, bro?

• 40:13Speaker 01

  I do know what Falcor is. It's the luck dragon for the never ending story.

• 40:17Speaker 00

  Yes, it was named after the luck dragon from the never ending story.

• 40:20Speaker 03

  Tell the story, Brian. Tell the story. Tell it all so Casey can hear and then we'll be done.

• 40:24Speaker 00

  Alright, I'll do a quick one. Casey, Casey Snow, how, how it works is that you would provide effectively an array with keys in it as the path into your data model in the backend. So say you want videos, some ID, title. It would return to you the title in that kind of as a tree. Videos or as a graph, or maybe as a forest is a better way to say it. Videos, ID, title. So if you wanted, say, a list, I want list 0 through 10, title, it would go list 0 through 10, which would be IDs, which would then link to videos by ID, title. So it's kind of like a graph-like data structure that makes it so that if you make duplicate requests, it's able to go, oh, okay, I already have videos, whatever. I already have list item 0. I can kind of not request so much from the back. The problem was is that what happened if list seven doesn't exist?

• 41:09Speaker 00

  You don't know, so we have to return something back to you that says it does not exist. So we returned something called like a, it's a boxed value. It's an atom of undefined. It lets you know, hey, there is a value there and the value there is nothing. You don't need to request it again. It's not going to be something. So we created something called materialize. On the backend, that means you need to materialize missing values. Well, what happens if you request list zero through 10,000, zero through 10,000, zero through 10,000 title? Well, you just created a gigantic number that were going to materialize everything for you. And so with one simple line of code, a while loop in bash, you can DOS or at one point 2016, you could have DOSed down and took down Netflix permanently. There was no rolling back. It was in production for like two years.

• 41:54Speaker 00

  You could do it for all endpoints, for all devices, for everything and just it would never run again. And that's it. It was a simple while loop. And I wrote a lot of beautiful code about it and I did a lot of beautiful stuff with it and it was very nice code. And then I also discovered it and reported it and then later on it got named the repulsive grizzly attack.

• 42:12Speaker 01

  Repulsive grizzly?

• 42:14Speaker 00

  Yeah.

• 42:16Speaker 01

  So has no one at Netflix watched the never ending story? Shouldn't it be called the Gomork or something like it's a movie. It's probably on Netflix.

• 42:25Speaker 00

  Okay, that, no, no, this wasn't some, this wasn't some 9,000. It was a completely separate team that when they found out about it, they had to do a bunch of work and and it ended up being, you know, a fix on the Zul gateway and all this kind of stuff.

• 42:36Speaker 01

  Alright, because there are no grizzly bears in the never ending story. I didn't read the original book because I think it's in German or something. I don't know.

• 42:42Speaker 00

  Yeah, yeah, but the nice news is I originally didn't come up with the materialize idea. But I did do a lot of refactoring that while refactoring and recreating everything, I stopped and went, wait a second. Something seems funny here. So then I trashed staging for a day where no one could use staging for a day. And then that's when I realized we made a mistake.

• 43:05Speaker 02

  When I say we, I mean me.

• 43:10Speaker 01

  And that's it, and he's off.

• 43:12Speaker 02

  He's like.

• 43:13Speaker 01

  I told my story and I'm not participating in this podcast anymore.

• 43:17Speaker 02

  The transitions at the end of this podcast are just getting worse and worse. He's end stream, he walks off frame.

• 43:24Speaker 00

  End stream?

• 43:25Speaker 02

  What's happening?

• 43:26Speaker 00

  All right, TJ, do you have anything you want to add to this all?

• 43:29Speaker 02

  He's frozen with a smile.

• 43:30Speaker 00

  Is stream done? Are we still live? Yeah. We're still live?

• 43:31Speaker 00

  TJ's smile, dude, find a friend that smiles at you like TJ smiles at us. I think today's big takeaway is don't bring Trash Dev into a project that needs help. He'll just quit.

• 43:43Speaker 02

  Yeah. No, actually, I will help your, your product get better.

• 43:47Speaker 00

  Bye.

• 43:48Speaker 01

  He'll drive your code coverage metrics down to zero.

• 43:51Speaker 00

  Yeah.

• 43:52Speaker 02

  Code coverage is a lie anyways.

• 43:54Speaker 00

  Are we live? And then quit. Oh, TJ's back again.

• 43:59Speaker 00

  All right. Well.

• 44:01Speaker 02

  What is happening?

• 44:03Speaker 01

  Don't know.

• 44:04Speaker 00

  See you later.

• 44:08Speaker 00

  No blockers on my end.

• 44:12Speaker 01

  I think we had a really productive meeting today, everyone.

• 44:15Speaker 00

  Yeah.

• 44:15Speaker 02

  Everybody enjoy your weekend.

• 44:17Speaker 01

  Next quarter is gonna be big.

• 44:19Speaker 00

  It's gonna be the biggest quarter of our lifetime.

• 44:21Speaker 01

  Absolutely.

• 44:22Speaker 02

  We gotta hit our OKRs.

• 44:23Speaker 01

  Yep.

• 44:26Speaker 00

  Well, okay then.

• 44:29Speaker 00

  Bye.

• 44:30Speaker 01

  Another professional podcast from the stand up.

• 44:33Speaker 00

  All right, I'm ending it. All right, hold on one second, let me, I'm gonna, I'm gonna end the stream. Thanks, everybody for being around here. TJ is back, big, whatever TJ is.