A casual conversation concerning causal inference

Lucy D’Agostino McGowan, cohost of the Casual Inference Podcast and a professor at Wake Forest University, joins Daniel and Chris for a deep dive into causal inference. Referring to current events (e.g. misreporting of COVID-19 data in Georgia) as examples, they explore how we interact with, analyze, trust, and interpret data - addressing underlying assumptions, counterfactual frameworks, and unmeasured confounders (Chris’s next Halloween costume). Sponsors: Linode https://linode.com/changelog – Get $100 in free credit to get started on Linode – our cloud of choice and the home of Changelog.com. Head to linode.com/changelog https://linode.com/changelogChangelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. LaunchDarkly https://launchdarkly.com/ – Power experimentation at any scale. Fast and reliable feature management for the modern enterprise. Featuring: Lucy D'Agostino McGowan – Website https://www.lucymcgowan.com, LinkedIn https://www.linkedin.com/in/ldmcgowan, X https://x.com/lucystatsChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Casual Inference Podcast https://casualinfer.libsyn.comCasual Inference Podcast | Twitter https://twitter.com/casualinferCommunicating Complex Statistics (video) https://youtu.be/-W2ku4-fp10Communicating Complex Statistics (slides) https://docs.google.com/presentation/d/1fO4vZn-dNII4UOd1itjomZW1oBcv51jDv7r35RGKc3M/edit?usp=sharingPractical AI is a “Media Sponsor” of the R Conference | Government & Public Sector https://rstats.ai/gov, where Lucy D’Agostino McGowan is giving the talk with Malcolm Barrett called “Causal Inference in R”, as well as a workshop with the same title. This will be the first ever R Conference focused on data science work in government, defense, and the public sector. Practical AI listeners get a special discount code valid for 20% off all ticket types, General & Academic Admission and workshops: PRACTICALAI20 Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The big picture with causal inference, essentially, it's exactly as it sounds. I guess I'm going to use the word in the definition, which I know you're not supposed to do. But if you're trying to answer a causal question, so does something cause something else? Which often, most of the questions we're interested in are in that framework. Although, it is distinct from something like prediction, which would be a different kind of space. But in the inference space, when we're trying to determine the relationship between different factors or different variables, a lot of times, we talk about associations as a way to sort of describe relationships that we know maybe are correlated, but we don't want to go as far as to say they're causal. But I would argue, in most cases, the human instinct is to want to talk about things causally. That's most of the time when we're studying relationships between variables. It's often because we want to know if there's a causal connection.

• 0:45Speaker 00

  So in randomized trials, like the ones that we're talking about with Pfizer and Moderna, looking at different vaccines, they're not just interested in whether or not getting a vaccine is somehow related to whether or not you get COVID. They want to know if getting the vaccine will actually cause you to not get COVID. So this is sort of the relationship that we are focused on in causal inference.

• 1:06Speaker 01

  Bama for Change Log is provided by Fastly. Learn more at fastly.com. Our feature flags are powered by LaunchDarkly. Check them out at launchdarkly.com. And we're hosted on Lino cloud servers. Get $100 in hosting credit at linode.com/changelog. What up, friends? You might not be aware, but we've been partnering with Lino since 2016. That's a long time ago. Way back when we first launched our open source platform that you now see at changelog.com, Lino was there to help us, and we are so grateful. Fast forward several years now, and Lino is still in our corner, behind the scenes, helping us to ensure we're running on the very best cloud infrastructure out there. We trust Lino.

• 1:49Speaker 01

  They keep it fast, and they keep it simple. Get $100 in free credit at linode.com/changelog. Again, $100 in free credit at linode.com/changelog.

• 2:17Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at change.com/community and follow us on Twitter. We're at Practical AI FM.

• 2:43Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International, and I'm joined, as always, by my co-host Chris Benson, who is a Principal Emerging Technology Strategist at Lockheed Martin. How are you doing, Chris?

• 3:00Chris Benson?

  I am doing very well, Daniel. How are you today?

• 3:03Daniel Whitenack?

  I can't complain. No complaints from this end. I had a good restful weekend. As listeners might remember, I recently downloaded the new Tony Hawk Pro Skater for Xbox, and that's been filling my waking moments as I try to increase my combo scores to ridiculous numbers.

• 3:25Chris Benson?

  I'm so jealous. You had a much better weekend than me. I had a weekend full of honey-do lists and stuff. I mean, I was exhausted and happy to return to work on Monday morning.

• 3:35Daniel Whitenack?

  I've got a pretty cool wife. She played with me. So there are things on my honey-do list, but it was okay because we were spending that quality Tony Hawk time together.

• 3:45Chris Benson?

  Yeah, I have like, I kept hammering my thumb and things like that. It's just terrible. It was horrible.

• 3:52Daniel Whitenack?

  That's rough, man.

• 3:53Chris Benson?

  I'm just wounded. I'm the walking wounded now, man. That's terrible. Anyway, go ahead. You keep talking.

• 3:59Daniel Whitenack?

  Yeah, well, I won't, you know, go into the the cause of you hitting your thumb with the with a hammer, but we are going to talk a lot about causes today and causal thinking. Today, we're really lucky to have with us Lucy D'Agostino McGowan, and she is an assistant professor in statistics at Wake Forest University. She's also very involved in the R conference and giving a talk at the upcoming Government and Public Sector R conference, which actually Practical AI is a media sponsor of. So, welcome Lucy. It's really great to have you here with us.

• 4:36Speaker 00

  Thanks. I'm excited to be here.

• 4:38Daniel Whitenack?

  Yeah, it's wonderful to collaborate with you on this. We should say, too, that you are also the co-host of a podcast. I think you're the co-host. There's another host of podcasts.

• 4:50Speaker 00

  Yes, Ellie Murray and I.

• 4:51Daniel Whitenack?

  Yes, of causal inference. So.

• 4:54Speaker 00

  Casual, casual.

• 4:55Daniel Whitenack?

  Casual inference. I'm sorry. I'm getting mixed up with our topic for the day. Good, good call.

• 5:02Daniel Whitenack?

  Casual inference. Yes.

• 5:03Speaker 00

  Yes, it's a pun. We gotta.

• 5:05Daniel Whitenack?

  Yeah, all good. I kept trying to pitch pun names for our podcast and none of them made it through the filter. So I don't know. I don't know what the reason is for that, but I'm all for pun names. So.

• 5:17Chris Benson?

  They just weren't very punny.

• 5:22Speaker 00

  Yeah, I love a good pun. We have some segments that have pun names as well. It's a good, a good time. Yeah.

• 5:27Daniel Whitenack?

  Yeah, yeah, for sure. So we will put a link to that in our show notes. Definitely I encourage our listeners to check out the podcast and see what they have. There's some really good content. But before we jump into all of the great things that you're involved with and that you're passionate about in terms of data science and R and causal inference and casual inference, if you could just let us know a little bit about your background, how you got interested in data science and R and other things and how you ended up where you're at right now and what you're working on.

• 5:59Speaker 00

  Yeah, so my background is in biostatistics. So I have a PhD in biostatistics from Vanderbilt. And so that's essentially statistics for the medical field. But while I was doing the work there, I had an internship with R Studio for six months. And so as part of that, I mean, biostatistics is a pretty applied statistical field. And so there's lots of kind of programming and data science like things involved with the training there. But then the software component was something that really interested me kind of further than even what we were doing already. And so I pursued it more. I got heavily involved with R Ladies and did this internship under Jenny Bryan at R Studio, which really kind of built up some of my coding chops and being able to actually

• 6:44Speaker 00

  do software development and things like that.

• 6:47Daniel Whitenack?

  And R Ladies, if you could describe that? That's what I was gonna ask. Yeah.

• 6:51Speaker 00

  Oh, yeah. Yeah, so R Ladies is basically an organization, a global organization, that there's the global kind of aspect of it, but then they're also, it's mostly run by local meetups in different communities. And the idea is to sort of increase the gender diversity in the R community. And so I had started one in Nashville, and they exist kind of all over the world. And they do really excellent work in trying to sort of increase that diversity. And it's been great. It's been actually really successful. There's some really fun plots that you can see as sort of the first couple. It started in San Francisco, and then it really has taken off since then in the past couple of years.

• 7:28Daniel Whitenack?

  Yeah, it's kind of all over the place now, isn't it?

• 7:31Speaker 00

  It is, yeah.

• 7:32Daniel Whitenack?

  Yeah, it's so wonderful.

• 7:33Speaker 00

  It's definitely international. So that was excellent and really formative for my interest in R. I was interested in R kind of before that, but that really kind of solidified that this was a community that kind of, in addition to being, you know, a language that offers the type of statistical tooling that I'm gonna need, it also had that community aspect where there was a lot of support for learning and for developing and things like that. And so that kind of was some of what branched me into this interest in sort of data science more broadly. And then I went on to do a postdoc at Johns Hopkins, and Jeff Lee Lab. And so my dissertation is very causal inference related and a big arm of what I do is still causal inference.

• 8:16Speaker 00

  But then through my postdoc, I added a bit on data science pedagogy. And so thinking about how to teach data science and how that can integrate in with kind of medical applications in particular. And so I worked on some of that and that sort of branched into this thinking about human data interaction problem, which is another big arm of my research right now, where we think about kind of how people interact with data and how they conduct data analyses and sort of how we can potentially think about interventions to nudge people towards conducting a correct data analysis as opposed to one that maybe would be incorrect in different ways, or maybe getting kind of alignment between stakeholders and producers of data analyses and things like that. And so this was all kind of what brought me into the more data science type space.

• 8:59Daniel Whitenack?

  Yeah, I think that human data interaction is, yeah, it seems to me to be like the really the tough problems that data scientists deal with, kind of live at that boundary. With the tooling now, there's so much great tooling around and it's fairly, I mean, of course, you know, you need some context and some background knowledge and some domain knowledge and all that to solve problems. But in tooling wise, it's fairly easy to actually create somewhat sophisticated systems. But that communication part and that interaction, the human data interaction, all is for naught if that's not properly taken into account and managed.

• 9:41Speaker 00

  Right.

• 9:42Chris Benson?

  It's such a big deal, too. As you were talking about that a moment ago, every single day in my job, I run into issues where there are challenges between how people are interacting with data and stuff. So I'm very excited to hear what you have to teach us in the minutes ahead.

• 9:56Speaker 00

  Yeah, no, it's a huge, it's, it's kind of neat because it builds on, I think, the kind of foundation for this field of human data interaction, you know, it kind of builds on what people would talk about, you know, before they were talking about human computer interaction. And I feel like in some ways we've mastered that. In a lot of ways, there's still a lot to learn, but I think, you know, that was happening a lot when. Things like Apple were coming about and there is and Google and sort of thinking about how we can help people interact with computing systems to make it that they're, you know, for some people, using a terminal is like a great way to interact. And I personally love interacting with the terminal, but to kind of get like the internet and everything into the hands of the average consumer, someone had to figure out that, no, we need to have something that has pictures and words and people can click things and they

• 10:40Speaker 00

  don't have to be just sort of interacting via this text module. And so, you know, that really launched a huge revolution, I think, when we were able to bring what computing had to offer to the average person. And I think that this human data interaction is sort of the next step on that. So now we've got the computing and we know how to get that into the hands of people who, you know, maybe otherwise wouldn't have been able to implement these different things. But making sure that now that they can do it, that they're kind of doing it correctly, that they're interacting with the data the way that is going to get to kind of a result that is correct and also aligns with the incentives of everybody involved, I think is really crucial.

• 11:20Daniel Whitenack?

  Yeah, and I think it's not only that's deeper sort of data interaction, but there's, there's a lot of communication about data now in the media as well. And, you know, how people perceive that is very different depending on who you talk to. So I know even in the last week, I think it's happened all in one week, there's like two or three companies that announced the sort of statistics about how good their COVID vaccines are going to be. So this happened, you know, like days ago, or maybe it was even, I forget if it was it this morning, the last one came out or yesterday or something?

• 11:58Speaker 00

  I guess yesterday.

• 11:59Daniel Whitenack?

  Yesterday. Okay. Yeah.

• 12:00Chris Benson?

  Ninety-five, I think, was the thing they were throwing around in the news, ninety-five percent.

• 12:05Daniel Whitenack?

  Yeah, so I think one was ninety-five percent. And the

• 12:08Chris Benson?

  Yep.

• 12:09Daniel Whitenack?

  The way it was communicated on the other one that I noticed was like ninety percent or higher or something like that?

• 12:15Speaker 00

  Above ninety percent.

• 12:15Daniel Whitenack?

  Above ninety percent. Okay, yeah.

• 12:17Chris Benson?

  And just as a real quick clarification for listeners, that we, at the point where we're recording, we just heard the first two vaccine initial results and those and everyone's talking about that. So you're probably beyond us and you know more than we do by the time you hear this.

• 12:32Daniel Whitenack?

  Yeah, this is late November twenty twenty.

• 12:34Chris Benson?

  There you go.

• 12:35Speaker 00

  Yeah.

• 12:35Daniel Whitenack?

  When you hear that sort of communication come out, do you get the sense that people are?

• 12:42Daniel Whitenack?

  Consuming that in a way that they are understanding the implications of those types of numbers? Or what is your thought on that?

• 12:50Speaker 00

  Yeah. Oh, this is such a good question. And it actually like it bridges essentially all of my research interests.

• 12:55Daniel Whitenack?

  Perfect.

• 12:56Speaker 00

  Yes, because we're talking about to the Pfizer and Moderna, I think, are the two big ones that came out relatively recently. And these were both randomized controlled trials. And so obviously, the question of interest is a causal question. It definitely covers my causal inference kind of framing. And then we also it covers the human data interaction and also communication of statistical kind of concepts. And so it bridges it really nicely. But one thing to mention in all of this, well, first, I think you said that it was 95% for the for Moderna, which is what a lot of outlets are reporting. The actual interim analysis reported 94.5. And the thing that's interesting, I know. Also, what's very interesting about this is that there's been, you know, just from a communication perspective, even that has been a little bit controversial, because they're giving

• 13:41Speaker 00

  a sense of precision where there really isn't that kind of precision. So the number of participants in this trial that received the placebo that got sick, it was there were, I think there were 90 that received the placebo and five that received the treatment that ended up getting COVID. And so 94.5 implies that there's this real precision, but we actually don't even have 100 participants that have reached that end point of getting.

• 14:08Daniel Whitenack?

  Right. It makes you think that there's like 10s of thousands and they they were able to get like really, really granular data, right?

• 14:16Speaker 00

  Right. Yeah. And giving, I mean, I'm sure that there were, well, I shouldn't say I'm sure, I don't know, maybe there wasn't someone that thought about if they said 94.5 that everyone would round up to 95 and then that this would sort of give some kind of sense of things that maybe wasn't exactly what was being represented by the data. And the other thing with these that's really challenging to make sure is well understood is that both in both cases, these are interim analyses. And so they're not the final result. And we wouldn't, as you know, as a statistician involved on data safety monitoring boards or these different groups that actually come out with these estimates. First of all, it's not common. I mean, in other circumstances, often interim analyses don't get the same kind of press that these are getting. I mean, sometimes they will get a press release,

• 15:01Speaker 00

  and they will see a little bit of a market change, but only for people who are really, really paying attention. And of course, now it's very different because the whole world is paying attention. And so Kind of how we communicate it has to be even more clear. But you know, interim, it's not unusual to get a different result than an interim analysis as a final analysis. You know, certainly you would expect, especially in both cases, because they're closer to the kind of what their final endpoint is than they're closer to that than they are far away. So this isn't like a first look. It's, you know, they're closer to what the final result likely will be. So probably it's going to be in the ballpark of what we're seeing, but it probably won't be exactly 94.5% for example. It would be really surprising if it was.

• 15:42Speaker 00

  And some of the concern is when those kind of precise numbers are reported in the news and then, you know, it comes out that, oh, in the end, it was 80% effective, which is actually really great. People kind of can start getting concerned.

• 15:57Daniel Whitenack?

  Yeah. And then they can kind of come up with a storyline of their own and one that maybe they're not comfortable with taking the vaccine for whatever reasons, because these people can't get their numbers right or, you know, whatever it is. Yeah, which could cause actual health and safety issues.

• 16:15Chris Benson?

  So having said all that, it raises kind of an ethical question in my mind at least to, you know, clearly this is done for, you know, kind of the marketing benefit. You know, the first one comes out, the second one gets out there to persuade the public. And yet, you know, as you just instructed us, it might reasonably fall to 80 hypothetically, but that does change the way people are thinking about it. And so, you know, any thoughts on using data in that way? I mean, there's a certain amount of manipulation potentially involved in that.

• 16:47Speaker 00

  Yeah, for sure. And it's really challenging to know what the right thing to do is because, I mean, I think one thing that can certainly help, and I know some of the reporting that I've read has been, you know, they've been careful to include in their text, like I was reading the New York Times piece on Moderna yesterday, and, you know, they did include that this is an interim analysis and we expect these results to change. But unfortunately, and, you know, maybe to no fault of the author, but that ends up being like paragraph four and, you know, the headline says something very different and people. Tend not to read past headlines. And so it's kind of hard to figure out the right balance of getting people to be able to consume this information because it is good to sort of have people see a light at the end of the tunnel. I think there's definitely a benefit-

• 17:28Daniel Whitenack?

  It's a need for optimism.

• 17:30Speaker 00

  Yes, there's definitely a benefit to that, but then being able to balance it with the reality and kind of being able to articulate the uncertainty in such a way that when the numbers slightly change, it doesn't cause people to sort of lose trust. And I don't know the perfect answer to this. This has been something that my lab has been studying. I have a student that's been running a study that's been looking at kind of if we randomized people to see kind of health recommendations qualified with uncertainty versus ones that are sort of said definitively. When a follow-up recommendation is made that maybe reverses the previous one, but gives kind of the full context, are they more or less likely to trust it if they saw things with kind of the proper uncertainty the first time or

• 18:12Speaker 00

  kind of with just like a clear certain statement? And it turns out, we're still in the process of analyzing this data, but it's challenging because the first result, when you just look at people randomized to seeing a certain versus kind of qualified recommendation, on average, people do tend to prefer the certain recommendation. Like, they just want to be told what to do. They don't want to be told the like, you know, roundabout, this came from a small study and we aren't actually quite sure and this is the best that we know right now, but we're going to be taking in more information as it comes. And so if you knew for sure that the recommendation or the numbers wouldn't change, that that actually is the easier way to go to get people to do kind of what would be in their best interest

• 18:56Speaker 00

  from a public health perspective. But of course, we don't know for sure. If we only had a small study, we really don't know that that evidence isn't going to change. And so if you you have to think about kind of hedging against that possibility. And in the kind of secondary piece, if you've said something with certainty and then you turn back and say the opposite, the lack of public trust, I think, is at a real potential to be lost and that and that can really be negative. And so sort of thinking about this and in the long run as opposed to in the short run is also really important.

• 19:25Daniel Whitenack?

  So Lucy, I'm curious. We talked a little bit about these numbers that are thrown around and obviously coming from statistical analysis of some type. And what I was thinking in my mind was there have been those times in my data science career where I felt- Maybe a little bit uncomfortable with how some of the numbers that I've communicated kind of up the chain to stakeholders have been used. Like, you know, oh, maybe I have this new speech recognition model. And for this particular language, like it performs whatever, you know, 5% better than the Google one from.

• 20:06Daniel Whitenack?

  And then, you know, up the chain, maybe it starts to be like used in marketing that like our speech recognition is 5% better than, you know, Google's version. Or, you know, there's situations where there's just like, I never intended for my work to be sort of taken in that way. Do you have any general recommendations for data scientists and statisticians and analysts in terms of how they deal with those interactions where you've done some analysis and now you're communicating to a different sort of audience and it's maybe going up the chain? Any recommendations or general principles that you think should be kept in mind in that type of situation?

• 20:48Chris Benson?

  So we don't have to be hostage to the marketing machine?

• 20:52Speaker 00

  This is such a good question. And it's so hard because, you know, in your example, the marketing team is at least within your own company. So you have like kind of constant communication with them. And often, you know, what I see on the, on the medical side at least is like someone will, will publish something. And actually there's a, there are a lot of examples from this from early kind of COVID data where there were papers that were published in our very top medical journals that people, some of the reviewers missed things. And I think the scientific community came together and, and found them pretty quickly. And the, the articles were either updated or retracted and it, it worked, but it worked on the scientific side, but the media ran with some of that. And, you know, the, the articles that ran headlines based on the original data often

• 21:35Speaker 00

  would get much more press than the updated ones, for example. So I think that this problem is not unique to kind of the setting where, that you described, where you're trying to kind of help a marketing committee understand the numbers in such a way that they can market them accurately. But it also happens sometimes where it's like not even in your control in the sense that it's like someone outside, totally outside of your organization is the one kind of running with it. I have this talk that I've been working on for a while. Have some aspirations of sticking it in like a short course or something, but basically it's I think of statistical communication and probably just scientific communication in general as on this kind of two by two grid where you have like you could imagine

• 22:17Speaker 00

  this like a X Y axis where you have maybe on the Y axis whether or not something is true or not and on the X axis whether it's interesting or not. And so you have this quadrant of like interesting and true, which is kind of where we want all of our communication to be. So something that kind of is interesting to the marketing folks, but also conveys the actual truth. And kind of next to that, you have a quadrant that is not true, but still interesting, which sort of falls into what you were describing where you you saw that there was a 5% improvement in in your speech recognition versus this other one in in this one specific case. And the marketing committee kind of ran with it and sort of implied that that was Generalized.

• 22:59Speaker 00

  Exactly. Generalized it beyond it. Anyway, so the kind of picture that I try to paint is moving from interesting to or from not interesting to interesting is one kind of dimension that we want to move across. But then the other one that sort of describes what you're talking about is moving from not true to true and sort of what are the pieces there. And so the ones that I've sort of defined as on this like journey to truth would be that it has to be mathematically correct, marketed correctly, disseminated correctly, and the audience has to interpret it correctly. And so those are kind of the four main pieces that I think determine whether or not it's true. So that one that you just said there fits in my second category of being marketed correctly. And I guess from my perspective, even just naming that these are the things that need to happen with statistical

• 23:44Speaker 00

  communication is moving us forward because I think that I haven't answered your question on how to get people to market it correctly, but I've at least named that these are things that can go wrong when you're trying to communicate statistics. And I feel like that's Potentially a step in the right direction is just knowing that that could happen. And so being cognizant that even if the thing that you do is statistically correct, you need to make sure when you're passing it on to people and when you yourself are marketing it, that you're marketing it kind of in the right way.

• 24:12Chris Benson?

  Will you take a second and rename those four things? Cause you started down and I got caught on the mathematical and I missed the other three.

• 24:20Speaker 00

  Yes.

• 24:20Chris Benson?

  So I wanted to do that.

• 24:21Daniel Whitenack?

  Chris is always stuck in the math.

• 24:23Chris Benson?

  I constantly, it's terrible. And then I also wanted to see, I actually wanted to like follow up on Dan's thing. Cause I know you're gonna do it anyway, but can you take your grid and kind of give us an example of how to use it? Because selfishly, when we are done recording this podcast and this is gonna go out for release, I want to use this as a tool myself. So you've just given me hope.

• 24:44Speaker 00

  Yes.

• 24:45Chris Benson?

  So can you kind of walk us through like how you would use the tool itself, what the four principles are? And then like how we can use it day to day. So if someone can listen to this and then they can go off and actually use it. Because I have the same set of issues. I work for a big company and there's lots of different audiences and they may use information in all sorts of different ways, like any other company. So I am all ears.

• 25:06Speaker 00

  Yes. Okay. So, so the four that I had mentioned are that it's mathematically correct, that it's marketed correctly so that you have kind of given the correct marketing, that it's disseminated correctly. And so that's slightly different than the marketing. So you could sort of yourself market it correctly, but then the way that it gets kind of disseminated after you've provided that marketing could be.

• 25:29Chris Benson?

  Kind of delivery outward, you know, the mechanism?

• 25:31Speaker 00

  Exactly.

• 25:31Chris Benson?

  Okay.

• 25:32Speaker 00

  Yeah, exactly. And then the third part, or the fourth part, rather, which I think this is one of the hardest parts is that the audience interprets it correctly. And the example that I like to give for that is that, you know, just because you've done all these other parts, right? If your audience. Misinterprets your result, then maybe you really need to be thinking about communicating it differently, even if you've actually communicated it in a correct way. And so somewhat relevant example for that, that made some rounds, you know, it's hard to know, I think I'm in an insular Twitter sphere, so it's hard to know how much this made outside of my world.

• 26:07Chris Benson?

  Art wheel.

• 26:08Speaker 00

  Yes, but in my world, this made kind of a splash that back a while ago, back in July, there was a dashboard that was going around from Georgia where they plotted these figures, a map of the cases per 100,000 people.

• 26:22Chris Benson?

  Yeah, I'm in Georgia, trust me, that impacted us in a bit, that was what we were talking about.

• 26:26Speaker 00

  Okay, so it's not just my Twitter sphere, this is good to know. And so the intention behind this map was, it was looking at kind of which counties relative to the others are the worst. So that's what this map was trying to do, is one of these maps that had, you know, it highlighted basically the ones that were in the top percentile. But how audiences were using this, a lot of people were visiting this site every day and taking a screenshot of this map, and they were comparing this over time. And this particular visualization is really not meant to be compared over time, because the bins are going to change every time, because they're percentiles or quantiles. And so those bins are calculated relative to all the other ones. And so they're not telling, this map doesn't tell you if things are getting worse overall

• 27:09Speaker 00

  in the state, it just tells you basically which counties are getting worse or better by these different. And when I first saw this, people were a little bit outraged because they were like, "Look, they've changed the bins, and so, you know, in the past two months, it looks like we're the same, but we're actually way worse, even though these two plots look the same." And when I first saw this, I'm like, "Yeah, but they did, the people that made those maps did the right thing, that's how you make those maps. How could you possibly guess what bins to use two months in the future? You know, how could you guess how many cases there might be so that you could set the colors, you know, in the past, based on what you might see in the future, so that you could be showing this graph over time, unless you retrospectively did it?" How could anybody do that?

• 27:51Speaker 00

  But the fact that so many people, the way they were consuming this information was via looking at this once a day, taking a screenshot and comparing over time, you know, that was an important piece of information.

• 28:03Chris Benson?

  Yeah, would it be fair to say it wasn't being interpreted correctly because it wasn't being disseminated with enough communication of how it should be consumed to ensure that interpretation was occurring correctly?

• 28:18Speaker 00

  Exactly, yeah, and I think that the solution here as like an analyst is that either you need to very explicitly say these maps are not meant to be looked at over time and, you know, somewhere where it's like in the screenshot that will end up on the screen. There was also a problem with if you were viewing it on mobile or a desktop, if it was on a desktop, it was a little bit clearer that these weren't meant to be viewed over time. If you're viewing on mobile, there was a line that said you can look at things over time that was corresponding to a different plot that ends up not being in view on mobile. So there were other pieces communication wise that made this challenging. But what I saw from this was that either you need to make that explicit or you need to just take away this map and use a different form of trying to show this result because

• 29:00Speaker 00

  people are going to be doing what they're do- I mean, people are going to consume information as they consume it. And, you know, I think it's outside of Georgia, I've heard from lots of people that the way that they've been consuming information from like the Johns Hopkins maps and things is that they take screenshots and record this over time because there's some kind of fear that we're going to lose information and so they need to be holding it. And so this is like, it's sort of a sociological kind of study, but just looking at this is how people are consuming this information that we're putting out. And it's not necessarily how the people who are putting out the information would have consumed it themselves, but they need to recognize that if the masses are consuming it like this, we need to be adjusting because the whole point of putting out these dashboards is to make them consumable, so.

• 29:42Chris Benson?

  So we totally need to get you on a consulting job as a side gig

• 29:46Daniel Whitenack?

  In Georgia?

• 29:46Chris Benson?

  For the Georgia Department of Public Health, because we have 12 million Georgians who are, and now we're doubting that. So we just need you there. We can fix Georgia.

• 29:56Speaker 00

  I know. It's so, it's a hard problem. I think it's not just Georgia.

• 29:59Chris Benson?

  Get ready to come down here.

• 30:01Speaker 00

  I'm not too far in North Carolina, so.

• 30:03Chris Benson?

  There you go.

• 30:04Daniel Whitenack?

  It is interesting what you said about like, people don't trust that they'll be able to, like, they think that they're gonna lose this data or something, or they're gonna lose this information. It's like, this thing was generated by some experts somewhere, and who knows how long this website's gonna be around or whatever, so I wanna, they're basically trying to do some data gathering of their own and come up with their own sort of self-service, you know, data dashboard or something that they think that they need. Yeah, I mean, that's a really interesting psychological thing. I don't know what the solution to that is in terms of giving people more trust that they'll have access to things.

• 30:48Daniel Whitenack?

  If it's like giving people the ability to, you know, self-serve themselves data more frequently, or I don't know.

• 30:55Chris Benson?

  So it's funny, I'll just note that in this particular example that we're discussing, that has happened, that that lack of trust has happened where like lots of local media stations and other organizations in Georgia have been pulling data from like Johns Hopkins, because they weren't trusting the Georgia data, and doing a whole bunch of graphing instead of pointing people at the institution that's supposed to be doing that, you know, there have been a whole bunch of surrogates out there. So it's been interesting to see how trust affects that in a pretty, pretty big way.

• 31:28Speaker 00

  Yeah, well, and I know, I mean, I'm not sure about in Georgia, but I know in other states, there have been cases where things were on a dashboard one day and then they were not made available the next day, and obviously

• 31:38Chris Benson?

  Yes, here.

• 31:39Speaker 00

  That kind of, yeah, so I guess it's happened there too. And so I think that that kind of thing happening also can make the trust kind of, that can erode it a little bit, too.

• 31:57Speaker 02

  ChangeLog++ is the best way for you to directly support Practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of Practical AI into the future. Simply follow the ChangeLog++ link in your show notes, or point your favorite web browser to changelog.com/plusplus. Once again, that's changelog.com/plusplus.

• 32:48Daniel Whitenack?

  All right, so we started talking into these COVID-related numbers with the vaccines, and then also talked a lot about data communication. All of that's super useful. But I do want to get a chance to talk a little bit more about that kind of third piece that you mentioned was mixed into that original problem we talked about of the vaccine numbers, which was causal inference. And I know that at the upcoming R conference, this is the first R conference that's going to be focused on government and public sector, which is super exciting. Chris and I are going to moderate a panel there, which will be a lot of fun for us to join. But you're giving a workshop there on causal inference. Of course, that name has also inspired your podcast,

• 33:32Daniel Whitenack?

  and I see that sprinkled around throughout your web presence. So could you just give us a little bit of context for what causal inference means? Why it's different than some of the other types of inference or prediction that we might perform as data scientists? I'd love to hear that because I definitely think that we have not had that specific conversation on this podcast as of yet.

• 33:57Speaker 00

  Yeah, great. I love talking about causal inference. So the the big picture with causal inference, essentially, it's exactly as it sounds. I guess I'm going to use the word and the definition, which I know you're not supposed to do, but if you're trying to answer a causal question, so does something cause something else, which often, most of the questions we're interested in are in that framework, although it is distinct from something like prediction, which would be a different kind of space. But in the inference space, when we're trying to determine the relationship between different factors or different variables, a lot of times, we talk about associations as a way to sort of describe relationships that we know maybe are correlated, but we don't want to go as far as to say they're causal. But I would argue, in most cases, the kind of human instinct is to want to talk about things

• 34:41Speaker 00

  causally. That's most of the time when we're studying relationships between variables. It's often because we want to know if there's a causal connection. So, you know, in randomized trials, like the ones that we're talking about with Pfizer and Moderna, looking at different vaccines, they're not just interested in whether or not getting a vaccine is somehow related to whether or not you get COVID. They want to know if getting the vaccine will actually cause you to not get COVID. So this is sort of the relationship that we are focused on in causal inference. And so randomized trials kind of often are the what people think of as the gold standard, although there are several ways that causal estimates can get skewed or biased in a randomized trial as well. So there are ways that you

• 35:25Speaker 00

  can actually need to do some more sophisticated analyses to get at a causal effect even in the randomized setting. But then where my work mostly is, is more in observational data, where we don't have a formal randomized trial, we're just sort of observing things, for example, in electronic health records, or you can think of all different types of data sets that have already been collected, but you want to try to determine if there's some kind of causal effect between. Different elements. And so to do so, you kind of have to build this framework that involves both kind of statistical modeling, but then also a lot of assumptions. And so a lot of times, like the way that we build these kind of causal, we build up this causality is being able to kind of state assumptions about our data

• 36:09Speaker 00

  that we're making. And should these assumptions be true, then we can assume that the effect that we're seeing is actually a causal effect and not just kind of an association between two things. Does that kind of answer your Brad question about causal inference?

• 36:21Daniel Whitenack?

  Yeah, yeah, it definitely does. It's very useful because I do think that it is the natural human reaction when we're doing any sort of modeling to assume like if these features that are fed into our model allow us to predict, you know, whatever it is, Y1, Y2, whatever those things out, then then those things are somehow causing that response or those labels or whatever it is. It's a very natural reaction to think that. One of the things I'm curious about is, so you mentioned this sort of process of defining your assumptions, being very careful about how you do that. I'm sure there's a number

• 37:06Daniel Whitenack?

  of things, but what are some of the kind of common tools that people use in causal inference? That maybe are, are there kind of gold standard tools or very common tools that people use in this case?

• 37:20Speaker 00

  Yeah, so there's a lot of kind of underlying assumptions and there's ways that people try to kind of help get at them. So my work uses something often called propensity scores. And so what that basically means, it's in the kind of observational setting where we don't have a randomization to an exposure or to some treatment. And so you try to kind of construct what we call a counterfactual framework. Like, all I know is that you got, for example, if I'm looking at diabetes drugs and heart disease, all I know is that you received diabetes drug X. And I don't know what would have happened if you had received diabetes drug Y, for example, but I could try to construct what I think may have happened. And so I could basically look at all of the different, your baseline characteristics,

• 38:05Speaker 00

  I could adjust for all of those and then kind of adjust for other people who have the same baseline characteristics you could imagine on the other drug and sort of assume that those two, if you were to measure All potential characteristics, so there's nothing unmeasured that might be confounded. Then you could compare those two kind of groups, or maybe those two groups on average to each other to sort of be able to build that counterfactual that we couldn't actually observe. And so in the randomized setting, because we're randomly assigning you to one or the other, the counterfactual is much easier to deal with because we assume that all of those baseline characteristics kind of on average are going to end up being just, they're going to be balanced because we end up kind of randomly assigning to one group or another.

• 38:49Speaker 00

  But in the observational space, you don't have that luxury. So actually kind of constructing something that can help you achieve that balance between the two via something like propensity score is the tool that I use most often. And that's essentially, it's just a summary score of your baseline characteristics. And so you essentially are estimating the propensity that you would get one treatment versus the other. And then you can use that as an adjustment tool in various ways, like weighting or matching, to be able to get comparable groups. And then once you have comparable groups, then you can start making some more causal assumptions. But of course, the big piece here is that there's, you can't have anything unmeasured, which in a randomized setting, you're less worried about because

• 39:30Speaker 00

  we assume that things are going to kind of be balanced in the long run. In an observational setting, you have to either feel very certain you don't have anything unmeasured, or do some sensitivity analyses to see kind of how bad things would be if you were missing an important variable.

• 39:45Chris Benson?

  Where do most people go wrong with this? Maybe they don't, aren't thinking explicitly enough about it. They might be an experienced data scientist but aren't really focused on, you know, like implementing a counterfactual framework. That's not part of their thinking. Where do you see people go wrong where it kind of takes that process off the rails a little bit?

• 40:07Speaker 00

  I think the first place where people kind of, it depends, because people kind of of all levels of experience go wrong on this. And so it's not even just beginners. But I think that the folks on the beginner's side, it tends to be this unmeasured confounding piece and thinking about the plausibility of it. I think it's really easy to do kind of a sophisticated analysis that adjusts for many things and assume that you're really capturing all of the variability. And you see this a lot with electronic health records. That's where, because I do bias statistics, that's kind of the data source that I tend to be thinking about. But you've got tons of information in electronic health records, but that doesn't mean you have everything that's important. And it also doesn't mean that you're going to end up with an unbiased

• 40:49Speaker 00

  result. And so I think that there can sometimes be confusion between I adjusted for lots and lots and lots of things and I have everything that's important. Like, those are two kind of potentially distinct pieces of information. So, the example that we talk about in healthcare a lot is, for a long time, there were studies that looked at hormone replacement therapy and heart disease. And it used to be recommended that hormone replacement therapy was actually protective against heart disease. And this was based on several large observational studies that were all kind of conducted around the same time and that did adjust for several things. And they were seeing kind of these consistent results. And so you could do a meta-analysis across all of these studies, and they all were sort of showing that it looked like it was probably protective. And then a randomized trial came about, and it

• 41:34Speaker 00

  showed that it wasn't really protective. And in fact, there was a chance it could have even been harmful. And so this sort of threw a bit of a wrench into things, and more studies were done. And it turned out, there's some nice plots that you can sort of look at for this. But the observational studies that adjusted for socioeconomic status were showing null effects or even potentially harmful effects, and the ones that didn't were showing protective effects. And so essentially, the whole effect of, or the large part of the effect of this hormone replacement therapy on heart disease that was being thought to be protective, was mostly driven by socioeconomic status, which just wasn't adjusted for in these original models. So while they adjusted for other things, they didn't adjust for that important variable. And it turned out that was actually a huge driver. So, women who had access to, kind of, they were from kind of higher socioeconomic

• 42:19Speaker 00

  categories, they had access to different healthcare, and so they were less likely to get heart attacks or have these cardiovascular events, that it didn't have anything to do with the hormone replacement therapy itself. And so, sort of this, I think that type of example is what the first, you know, relying on previous knowledge in an area doesn't always save you from this unmeasured confounder piece. And I think that people who are first new to a Kind of discipline, your default might be, okay, well, everybody that's fit this model before has included these variables and this is what we do and it looks like we're getting the same kind of effect that people see, so it must be right. And I think that's kind of not always the case.

• 42:56Chris Benson?

  I like the term for that, too, unmeasured confounder.

• 42:59Speaker 00

  Yes, unmeasured confounder, I know.

• 43:02Daniel Whitenack?

  That'd be a good podcast name if anyone else is wanting to start it.

• 43:06Chris Benson?

  That's what I'm gonna be next Halloween. I'm gonna be an unmeasured confounder for Halloween.

• 43:09Daniel Whitenack?

  Yeah, that would be a great Halloween, yeah.

• 43:13Daniel Whitenack?

  On that note, not the Halloween costume, but the confounder note, I'm curious, I could see myself getting into a state where I'm like, I'm a little bit gun shy in the sense of like, oh, I'm like in this situation, I'm trying to do some causal inference, but I'm like, how do I know when I've, you know, I could see myself always thinking there's gonna be another confounder out there. How do I know when I'm ready to pull the trigger and like actually give some results to someone? There's so many different, you know, things that play in here in this situation and I'm just, you know, I think I've accounted for everything, but I don't know if I've accounted for everything. How do I know when I've accounted

• 43:55Daniel Whitenack?

  for everything, or at least enough things to where I can have some confidence? You have any thoughts there?

• 44:02Speaker 00

  Yeah, it's a great question. And unfortunately, there's not like a test or something for have you accounted for all that. You know, unmeasured confounding is one that's specifically onerous because there's not a way to know for sure. One of my dissertation papers was on building tipping point sensitivity analyses for unmeasured confounding. And so the idea is that you can do your analysis and you do the best you can. You state the assumptions that you're making. And then at the end, you can do one of these tipping point analyses that we basically show mathematically, it's a pretty simple formulation. There's an R package called Tipper that can do it for you. But essentially, what it will do is it'll take your effect that you're assuming to be causal and it'll tell you the size of an unmeasured confounder that would be needed to tip that analysis. And so I think that

• 44:47Speaker 00

  that's kind of the best case scenario for what you can do. If you've done all that you can to account for what What you have, then you can just explicitly state that, you know, we think that this effect, we feel like we've done what we can. These are the assumptions that we've made. If there were an unmeasured confounder out there like this that, you know, was related to the exposure and outcome in this manner, it would make our result no longer significant. It would make it, it would nullify our result or make it inconclusive. And so I think that's kind of the best that people can do. And I think just doing that would really move the field forward in a lot of ways. You know, one thing that that can do is, once that's been stated explicitly, kind of in a paper or write-up or whatever it is that you're doing about this causal estimate that you're

• 45:28Speaker 00

  working on, then content matter experts that come in, you know, from all different areas can see that. And some may say, hey, that actually is plausible because I actually have seen that this particular variable that you didn't account for can have that kind of an impact on, on your exposure or have that impact on your outcome that, that really would tip your analysis. And so sort of doing that work for them where they can see what the, what a confounder like that would need to look like. And then they could sort of map that back to their own content expertise, I think is what can help with this. And so that's my best recommendation on that front.

• 46:05Daniel Whitenack?

  It's very helpful.

• 46:06Chris Benson?

  It's pretty good. Excellent. So I guess as we wind up, what are things that you're excited about? You know, it could be an R, it could be about causal inference, it could be trends that you're seeing. What are the things that you're looking forward to doing over the next couple of years versus the stuff that you have been working on?

• 46:23Speaker 00

  Let's see, I'm excited about so many things.

• 46:25Chris Benson?

  That's fine. Let's hear it. Go for it.

• 46:28Speaker 00

  I think, well, so I think on the causal side, I'm excited that this has been getting a lot of attention recently. I think that we had Roger Pang, he hosts a podcast called Not So Standard Deviations with Hillary Parker. He also hosts one called The Alpha Report with Elizabeth Matsui. And I had him on our podcast a little bit ago to kind of talking about his thoughts on causal inference. And he talked about how he sort of At one point, you'd imply that maybe it's a fad, but essentially that like people are interested in this. And I think that it's true. I think that it's something that people are gaining more and more interest in, in terms of understanding the methods as opposed to just trying to make causal claims. I think people, like as we've discussed, human instinct is to want to make causal claims. And so that's something that people have always been interested in. But the interest in sort of incorporating the more rigorous methods has been going up.

• 47:11Speaker 00

  So I'm really excited for that to continue. I think in that space, I think we're starting to see better and better kind of introductory level information on how to conduct causal type analyses. And I think the gap still is sort of in that intermediate spot. I think that we have lots of people who are very competent on the heavy, heavy methods and lots of people who are working on the introductory. And I think we've got this nice middle spot that has a lot to be left to be contributed. So I'm excited about that because I think that there's potential there. And then kind of going back to the science communication piece that we were talking about before, I'm also excited. I think that a lot of the scientific process has been brought to the forefront and with just the pandemic response

• 47:53Speaker 00

  and sort of thinking about how this was done and how it was communicated. And so I think that we have a lot of data and information now on kind of how people have tried to communicate things and where that has potentially failed or maybe where it succeeded. And so I think going forward, we have a good horizon for being able to sort of improve on how we're communicating results, which I think is only going to be something that's better for everyone, both on the scientist's side and on the general public's side. So those are two things I'm excited about. Could probably come up with others.

• 48:24Daniel Whitenack?

  That's good. You've made me definitely excited about those things as well. And if our listeners are, I'm sure they are, also excited about causal inference and these things that we've talked about, I would encourage you very much to check out the upcoming R conference. You can go to rstats.ai.gov and find out all of the info there. Lucy's giving a workshop there on causal inference. So it's a really great opportunity to dig deeper than we can during this period on the subject. The conference is December 2nd through the 4th. The workshops are on the 2nd and the conference is December 3rd through the 4th. And our listeners have a special discount code.

• 49:09Daniel Whitenack?

  So make sure you use Practical AI 20 is the discount code Practical AI 20. And you'll get 20% off all of the ticket types, including the workshop that Lucy's giving. So make sure and check that out. Chris and I will be there as well, moderating a panel. So it's going to be a great time. I would encourage everyone that our community is so welcoming and awesome. And I really encourage people to check that out and hopefully see Lucy and both of us there. Thank you so much, Lucy, for joining. It's been a pleasure.

• 49:41Speaker 00

  Yes, thank you. It's been great. I I'm so excited that you mentioned that workshop. The propensity scores that we talked about are going to you'll learn how to fit those types of models. You'll also learn about counterfactuals and things. And so exactly the methods that we talked about today, you'll be able to actually implement from the comfort of your R consoles.

• 49:57Daniel Whitenack?

  Awesome. That's so perfect. Yeah, great, great timing. It was great timing, this discussion, because you know, those numbers came out just then for the for the vaccines just a couple of days ago. And then we've got the R conference coming up and people can just follow this whole story arc and get get trained up in causal inference and go do all of these exciting things that we talked about. So thank you so much, Lucy.

• 50:22Speaker 00

  Thanks for having me.

• 50:28Speaker 02

  Come hang out with Daniel, Chris, and hundreds of other AI practitioners in our community Slack. It's a cool place to be, not a lot of noise, some great signal, and best of all, it's totally free. Check it out at changelog.com/community. And don't forget to follow the show on Twitter for AI news and links, highlights from past episodes, and more. We are at Practical AI FM. We love to have you following along. Thanks to Daniel and Chris for hosting Practical AI, week in and week out, to the mysterious Breakmaster Cylinder for the excellent beats you hear on all Changelog podcasts, to our sponsors who have our back, Fastly, Linode, and Launch Darkly, and to you for listening. We appreciate your time and attention. That's all for now.

• 51:09Speaker 02

  On the next episode, the guys chat with the team at Unsplash all about their huge open data release. So stay tuned for that one next week.