a16z Podcast: A Guide to Making Data-Based Decisions in Health, Parenting... and Life

with Emily Oster (@ProfEmilyOster) and Hanne Tidnam (@omnivorousread) Are chia seeds actually that good for you? Will Vitamin E keep you healthy? Will breastfeeding babies make them smarter? There’s maybe no other arena where understanding what the ...

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Hanne Tidnam?

  Hi and welcome to the A16Z Podcast. I'm Hannah. Good data, bad data, there's maybe no other area where understanding what the evidence actually tells us is harder than in health and parenting. In this episode, economics professor Emily Oster, author of Expecting Better and the recently released Crib Sheet: A Data-Driven Guide to Better, More Relaxed Parenting, does just that, looking at the science and the data behind the studies we hear about and make decisions based on in those worlds. From whether to breastfeed your child, to screen time, to sleep training. We talk about what it means to make data-based decisions in these settings, in diet and in health and in life, like whether chia seeds are actually good for you, and how we can tell what's real and what's not.

• 0:42Hanne Tidnam?

  We also talk about how guidelines and advice like this gets formalized and accepted, for better or for worse, and how they can or can't be changed, and finally, how the course of science itself can be changed by how these studies are done. You describe yourself as teasing out causality in health economics. Can you give us a little primer on what exactly that means and how you start going about doing that?

• 1:02Speaker 01

  So there are a lot of settings in health, and in all of those settings, we have to figure out what does the evidence say? And I think about some of them in this context of parenting, but you can think about even questions like, is it a good idea to eat eggs? Or is it a good idea to take vitamins? Other kinds of health decisions. And you can sort of think about there being kind of two types of data you could bring to that. One would be randomized data, so you could run a randomized trial in which half of the people got eggs and half of the people didn't, and you followed them for 50 years and you saw which of them died. And that would be very compelling and convincing, and when we have data like that, it's really great.

• 1:37Hanne Tidnam?

  I mean, I kind of think of that as being the default, no? Is that not at all the standard?

• 1:42Speaker 01

  That is, that's the gold standard, but it is not the default. So many of the kinds of recommendations that I look at in parenting, but that you look at in general in health, are based on observational data. To the other kind, where we compare people who do one thing to people who do another thing and we look at their outcomes. And the, one of the ways in which the people differ is on the thing that you're studying. But of course, there are other ways that they may, they may differ also.

• 2:05Hanne Tidnam?

  A million other ways.

• 2:06Speaker 01

  A million other ways, yes. And data like that is really subject to these kinds of biases that the kind of people who make one choice are different from the kind of people who make another choice. One of the things that's very frustrating in a lot of the health literature is that there isn't always that much effort to improve the conclusions that we draw from those kinds of data.

• 2:25Hanne Tidnam?

  And we're drawing, we're using that kind of approach because of the inability to have long longitudinal studies, or it does tend to be a shortcut.

• 2:33Speaker 01

  So I think it is, it is both things. So it is much easier, faster to write papers, to produce research about that. And it can be really useful for developing hypotheses.

• 2:42Hanne Tidnam?

  So it's like a scratch pad almost.

• 2:43Speaker 01

  In the best case scenario, it'd be like a scratch pad. Like, let's just look in, let's look, look in the data and see what kinds of things are associated with good health or associated with good outcomes for kids. And then we could imagine a next step where you would, where you would analyze more rigorous goal standard. And sometimes that happens. So there's one really nice example of the book where this happens exactly like you would hope, which is in studying the impact of peanut exposure on peanut allergies.

• 3:07Hanne Tidnam?

  Right.

• 3:08Speaker 01

  So the, the first paper on that is written by a guy and what he did was he just compared Jewish kids in the US, or sorry, Jewish kids in the UK to Jewish kids in Israel. And he saw that the kids in Israel were less likely to be allergic to peanuts. And he said, that's because they eat this peanut snack when they're

• 3:22Hanne Tidnam?

  Bombas, right?

• 3:23Speaker 01

  Yeah, the bomba. And so, so then that's, that's like the hypothesis generation. And then he went and did the thing you would really like, would just say, okay, let's run a randomized trial and let's randomly give some kids early peanuts and some kids not. And indeed, like, he found that he was, he was right. So that's like a great example of like how you would hope that literature would evolve. But in many of the kinds of health settings we're interested in, that you can't do that, or it is hard, much harder to do that because the outcomes would take a long time to realize, or it's expensive, or it's hard to manipulate what people are doing. And then we often end up relying on these more biased sources of data to draw our conclusions, not just as a scratch pad. And I think that's where We encounter problems.

• 4:06Hanne Tidnam?

  That's where it gets murky and we never know whether we should eat eggs or not. Yeah, and that's exactly the area that you tend to focus on.

• 4:13Speaker 01

  Yeah, exactly. I try to first see, are there good pieces of data that we can, that we can use? And then if we're stuck with the data that isn't good, trying to figure out which of the murky studies are better than, than others. And, and you know, what would you mean by better? Well, you know, how, it's roughly like, how good is this study at, at controlling or adjusting for the differences across?

• 4:34Hanne Tidnam?

  Right.

• 4:34Speaker 01

  Across people.

• 4:36Hanne Tidnam?

  So you talk about kind of breaking it down into both, into the relationship between data and preference. How do you factor in that in the healthcare system where it's so diverse, where preference has such an incredible effect and puts you into so many different possibilities?

• 4:51Speaker 01

  I think this is why in these spaces, decision making should be so personal. We often run up in, in health and also in, in parenting and all of these spaces into a place where we're telling people like, there's a right thing, there's a right thing to do.

• 5:05Hanne Tidnam?

  Yeah.

• 5:06Speaker 01

  And I think that, that that can be, that can be problematic because it doesn't recognize this, this difference in preferences across people.

• 5:14Hanne Tidnam?

  You have to basically accept the variety in the system and then give a space for preference in the decision making.

• 5:18Speaker 01

  Right, yeah, but I think it's exactly these preferences that of course make it hard to, to learn about these relationships in the data. Cause once you, once you recognize that a lot of the reason that some people choose to eat eggs and some people choose to eat Cocoa Crispies is that some people really like Cocoa Crispies.

• 5:31Hanne Tidnam?

  Right.

• 5:31Speaker 01

  Some people really like eggs. How can you ever learn about the impact of eggs? Because we know there must be differences across people. And I think that that becomes even more extreme when we think about really important decisions that people are making, like the kinds of choices they make in parenting or also in their diets.

• 5:45Hanne Tidnam?

  So can you walk us through one example like that of where it was a really kind of murky gray area and how you pull out the causality?

• 5:53Speaker 01

  The best example of this in the data, in the parenting space is probably in breastfeeding. Let's say you want to know the impact of breastfeeding on obesity in kids.

• 6:00Hanne Tidnam?

  Right.

• 6:00Speaker 01

  That's a thing which you hear a lot. That breastfeeding is a way to make your kid skinny and, and so on. And so the basic way you would, you might analyze that is to compare kids who are breastfed to kids who are not and look at their obesity when they're say, seven or eight. And indeed, if you do that, you will find that the kids who are breastfed are less likely to be obese than the kids who are not. But you will also find that they, that the, there's all kinds of relationships between obesity and income and obesity, your mother's income and mother's education and other things about the family. And, and those things correlate with breastfeeding and they also correlate. So you can't really pull apart this web. So it's hard to pull apart the web. So I would say this is an example where like the data is suggestive. Like it would certainly be consistent with an effect of breastfeeding on obesity, but

• 6:45Speaker 01

  I, I think it doesn't prove an effect. And then you can sort of take the next step and say, okay, well, do we have any data that's better? And in that example, you know, we do have one kind of randomized data, but again, we run up against the limits of all kinds of evidence. So the randomized data on this question is from a randomized trial that was run in Belarus in the 1990s. Oh. They randomly encouraged some moms to breastfeed and some moms not. And so there's a lot of good things that we can learn from that. But such a specific place in time. Exactly. It's so specific. And you said like, well, you know, how, how do I take that result to, you know, the Bay Area in 2019? Right. It's a challenge. Okay, well, is there any other, within the space of, you know, not randomized data is or anything that's better?

• 7:28Speaker 01

  And in that case, there is, you know, there are some studies that like compare siblings. Oh, okay. Where you look at two kids, you know, born to the same siblings, born to the same mom, one of whom was breastfed and one of whom was not. And then look at their obesity rates. And when you do that, you find there's basically no impact. So then you're kind of holding constant, like who's the mom? So if your worry was that there are differences across parents in their choices to breastfeed, well, now you're looking at the same parent. Right. You're normalizing. You're normalizing. And so you may think, oh, that's great. Perfect. I'm totally done. But of course, you're not, this isn't perfect because why did the mom choose to breastfeed one kid and not the other? People are not choosing randomly. You had a C-section one time, you didn't another time. If that were the reason, that would be great. Right? If the reason were just like, it kind of worked one time, it didn't work the other

• 8:11Speaker 01

  time, you know, if there were something that was effectively a little bit random. Uh huh. Then that would be exactly what, that would be exactly the kind of variation you'd want to use. Okay. But the thing you worry about is like, one kid is not doing well, is unhealthy. So the mom chooses not to breastfeed or chooses to breastfeed to try to make them healthier. Those are the kind of things where there's some other reason that they're choosing differences in breastfeeding, which has its own effect on on the kids outcomes. So you kind of like, some of what I'm I try to do in the book is sort of like, put all of these pieces together and kind of like, like, look at them and and think about them all as a as a sort of totality of evidence and just think like, how compelling is this altogether?

• 8:52Hanne Tidnam?

  It sounds almost like sifting, like using a sifter. You take all this very murky data, very variable from all sorts of different contexts, and like, put it through the sifter of like, this kind of data, this kind of data, and then match it all up and say, okay, what do we have left? And then therefore, and then hand that over and say, and now you make the decision.

• 9:09Speaker 01

  Yeah.

• 9:10Hanne Tidnam?

  Based on this.

• 9:11Speaker 01

  Based on this. Right. Here's kind of what we can be more or less or less sure about.

• 9:15Hanne Tidnam?

  You talk a little bit about the idea of constrained optimization as being very important. Can you explain what that means and how that plays out?

• 9:23Speaker 01

  In economics, we we think about people optimizing their utility function. The idea is that you have a bunch of things that make you happy. That's your utility. They produce your utility. And you want to make the choices that are going to optimize your utility, that are going to give you the most amount of happiness points. Yoodles, call them yoodles.

• 9:42Speaker 01

  It's really, it's a very warm and fuzzy discipline.

• 9:45Hanne Tidnam?

  I'm gonna go home and use that with my kids. You gave me some yoodles today.

• 9:49Speaker 01

  But we also recognize that people have constraints. In the absence of constraints, like having money to buy things or time to time to do them, people would just have an infinite amount of stuff. That's the thing that would make them the most happy. And so, but when you're actually making choices, you're constrained by either money or or time. And in the book, I talk a lot about this in the context of time, that, you know, you're you're, as a parent, you're making your choices. And you have some preferences, some things you would like to do, but you are also facing some some constraints.

• 10:19Hanne Tidnam?

  But is there is information flow kind of what and the data itself a constraint in that regard? Is that a because it's so piecemeal, the information you get, that feels almost totally random. Like some media story picks up on something, you tend, you know, some tidbit, you hear someone. As you're like, systemically studying a graduate seminar on parenting, which none of us do, you know, then it is random.

• 10:41Speaker 01

  Yeah. Yeah. And I think we wouldn't necessarily think of that as in constraints, because of course, in our models, people are fully informed about everything all the time.

• 10:49Hanne Tidnam?

  Right.

• 10:49Speaker 01

  That's, that's one of the great things about the models.

• 10:52Hanne Tidnam?

  But in real life?

• 10:52Speaker 01

  But in real life, yeah. I think people face, face constraints associated with just not having all the, the information and, and you know, also the fact that this, these kind of information, like whips, whipsaws over time.

• 11:03Hanne Tidnam?

  Yeah.

• 11:04Speaker 01

  That, you know, you get one piece and then, and then you kind of, the next day, there's a, there's a different piece of, of information and we have a tendency to kind of glom onto whatever is the most recent thing that we have seen about this, as opposed to what is the whole literature over this whole period of time say?

• 11:21Hanne Tidnam?

  Right. You say, you have a great quote where you say, "In confronting the questions here, we also have to confront the limits of the data and the limits of all data. There's no perfect studies, so there will always be some uncertainty about conclusions. The only data we have will be problematic. There will be a single not very good study and all we can say is that this study doesn't report, support a relationship." So it feels kind of hopeless. I loved when you talked about the first three days of when you brought Penelope home and it really brought that back for me. It's just this dark room that you're kind of alone, making these decisions. How do you even begin to see this data, you know, as a decision-making practice? Like, how does that, how does that translate?

• 12:02Speaker 01

  There are pieces where it's, where it's easier, where the data is better and makes it, is clearer about what you need to do or what the choices are. You will be making many choices without the benefit of evidence or data or very good data. I think part of what makes some of this parenting so hard is that for those of us who like, you know, evidence and facts and it's hard to accept, I'm just going to have to make this decision, basically based on what I think is a good idea.

• 12:29Hanne Tidnam?

  Based on my gut.

• 12:30Speaker 01

  Based on my gut.

• 12:31Hanne Tidnam?

  Yeah.

• 12:32Speaker 01

  And, uh, and you know, maybe based on my mom.

• 12:34Hanne Tidnam?

  Yeah.

• 12:35Speaker 01

  And, you know, and.

• 12:36Hanne Tidnam?

  Yeah. Which is a sample size of one.

• 12:37Speaker 01

  Sample size of one. Um, and you know, maybe if you have like a mother-in-law, father-in-law, it's like a sample size of two. Um, but, but that's kind of, that's kind of it. And I think that that's, that's really scary, especially when the choices seem so important.

• 12:50Hanne Tidnam?

  Yeah, I mean, but it feels like, you know, that's kind of at heart what you're trying to do, right, is like to translate and to give tools in this decision-making place. So how would you begin to systematize that? I mean, is there a way to bridge that gap better in the system?

• 13:06Speaker 01

  I think that it would be helpful if more information was shared. So I think in a lot of these things, there is a lot of information that is contained in people's experiences that we are not using in our evidence production. So in the book, I talk about like the sleep schedule.

• 13:26Hanne Tidnam?

  Right.

• 13:27Speaker 01

  So you're sort of told as a parent, like, oh, you know, this is kind of roughly like around six weeks, your kid will start sleeping like longer at night. But there's no, the information that's sort of typically conveyed to people is not, is not a range. It's just like around, around six, six weeks-ish. You know, that'll start to happen. But the truth is like, yeah, that's, that's kind of right. But it's, it's, if you look at data on when that actually happens, it's pretty, it's a pretty wide range. And I think part of what is so stressful about this early, these like early parts of parenting are that, that it's very hard to understand whether what you are experiencing is like normal. And I think if, if you could understand like, yeah, most kids don't do this thing at this

• 14:10Speaker 01

  time, or most, most parents have this experience, or...

• 14:13Hanne Tidnam?

  The way the graph plots, kind of, a little more broadly.

• 14:16Speaker 01

  Yeah, what does the graph look? Exactly. What is, I think that would be, that would be super helpful. And that's a place where I can imagine, you know, data collection helping, right? That, you know, we have a much more of an ability at this point to, to like get information about what is happening with our kids, what's happening with, you know, with our health. There is a sense in which that could be helpful in, in just setting some norms for the normal, the standard variation. Across people.

• 14:42Hanne Tidnam?

  So looking at the variation and providing that as like a piece of the information.

• 14:46Speaker 01

  As a piece of the information.

• 14:47Hanne Tidnam?

  Here's also the variation on that. Yeah.

• 14:49Speaker 01

  Yeah. And I think that's, that is kind of part of like generating the uncertainty and sort of showing people like, what are the limits of the data?

• 14:55Hanne Tidnam?

  Yeah.

• 14:55Speaker 01

  Right? That how sure are you that this should happen at this time?

• 14:58Hanne Tidnam?

  Not just how sure, but like, what are some of the like other ends of the curve? I mean, that's just information you just don't get.

• 15:04Speaker 01

  Yeah.

• 15:04Hanne Tidnam?

  Right. So let's zoom out a little bit as somebody who lives in the world deeply of data in the health system. We're in a time of enormous shift, right, for data. Does the improvement, does our kind of the sea of data and like better data, cleaner data, more granular data, all that help this at all, this question?

• 15:23Speaker 01

  Yeah. And, you know, I think we are collecting so much data on people. Both sort of individual people are collecting a lot of data about themselves. Health systems are collecting a lot of data about people. This data is like underutilized, I think. We're amassing pools of it, but not in ways that are especially helpful. So, you know, in my, in, when I go to conferences with people who work on healthcare, like, there's a tremendous amount of data that's being used on health claims.

• 15:50Hanne Tidnam?

  Okay.

• 15:50Speaker 01

  So if you sort of think about like, what are some kinds of data that we have? We have like health claims data, like payments, everything that where there's an individual payment for it, we'll like see, we'll see it. There's almost no work with medical records, even though every hospital, everybody's using Epic. You would think that that would make it straightforward to have that data in a usable form. But it's not.

• 16:11Hanne Tidnam?

  Yeah.

• 16:11Speaker 01

  And, but, you know, at the same time, the, the potential for sort of going beyond like, here is all the tests that you ordered into actually like what happened with those

• 16:20Hanne Tidnam?

  Yeah.

• 16:21Speaker 01

  tests, and then what happened to this person later? Like, that, that data is not being mined in the way that, that we could to try to look at some, you know, at some of the kinds of outcomes that are, that are a result.

• 16:33Hanne Tidnam?

  The causality that you would pull out afterwards.

• 16:36Speaker 01

  Yeah, absolutely. You know, how can we improve our, our causality? More data is helpful, more information about people is helpful, being able to look at, you know, the timing relationship between some treatment and some outcome. Those are all the kinds of things that, you know, having better data would help us, would help us do.

• 16:51Hanne Tidnam?

  Are there other areas where you start, you are starting to see the data coalesce in a way where you're able to pull meaningful insights from it?

• 16:59Speaker 01

  So I think, yes, you know, when we have better data, we can use better tools, even if we don't have randomization. A classic example in health is looking at the impacts of like a really advanced neonatal care. Like, how cost effective is it to have like, you know, kids in, in sort of getting like really extensive NICU care?

• 17:17Hanne Tidnam?

  Right.

• 17:17Speaker 01

  Like, how, like, how effective is that in terms of improving survival? And how much does it cost?

• 17:22Hanne Tidnam?

  No, such a basic question.

• 17:24Speaker 01

  Such a basic question and super hard to imagine analyzing because of course, you know, babies that are very small and are sick cost more, but also have worse outcomes.

• 17:32Hanne Tidnam?

  Yeah.

• 17:32Speaker 01

  And so what you, if you sort of looked at that, you would be like, well, actually, like, spending more, we're not getting anything because those babies are more likely to die than babies that are spending less. We define very low birth weight babies as less than 1,500 grams.

• 17:45Hanne Tidnam?

  Okay.

• 17:46Speaker 01

  Which means that the treatment that you get, if you're a baby at 1,503 grams, is very different than the treatment that you get as a baby at 1,497 grams, which is completely arbitrary.

• 17:56Hanne Tidnam?

  Yeah.

• 17:56Speaker 01

  I mean, the, the choice of 1,500 grams has nothing to do with science.

• 17:59Hanne Tidnam?

  It's like this line in the sand.

• 18:00Speaker 01

  That's not a good way to set policy. However, having set the policy like that, you can then say, okay, well, now we have some babies that are almost exactly the same, but the babies that are a little bit lighter, that are like 1,497 grams, get all kinds of additional interventions relative to the babies that are 1,503 grams. And when people have done that, they see actually the babies at 1,497 grams do better.

• 18:21Hanne Tidnam?

  So the line actually is beneficial in that way because you're defining these two groups very closely. Oh, interesting.

• 18:27Speaker 01

  So the line, setting this line in this arbitrary way lets you get at some causality.

• 18:31Hanne Tidnam?

  Even though not good for the babies.

• 18:32Speaker 01

  So having done it, good for, good for.

• 18:34Hanne Tidnam?

  Good for information.

• 18:35Speaker 01

  Yeah.

• 18:35Hanne Tidnam?

  Interesting. What are some of the other tools? Are there others in that list?

• 18:39Speaker 01

  So that's, that's, that is, that's an example of regression discontinuity, that there's some discontinuous change in policy on either side of, of a, of a cutoff. And that has become a sort of part of a big toolkit of things people are, people are using more of. The other is to look at sort of sharp changes in policies at a time, at like a moment in time.

• 19:01Hanne Tidnam?

  Oh, so the same thing, but in time.

• 19:03Speaker 01

  Then there's that. And then there's, you know, looking across when different policies change differently for different groups. So, so all of these things have become easier with more data and become more possible with more data. And I think that that has improved our inference in some, in some of these settings.

• 19:19Hanne Tidnam?

  I love that you talked a little bit about the experience of doing your own data collection, kind of in the wild with this spreadsheet after Penelope was born, which made me laugh so much because it was so much like my spreadsheet. It was just so sad to think of like all these moms alone in their bedrooms at night.

• 19:35Speaker 01

  I know.

• 19:35Hanne Tidnam?

  With their like Google Drive.

• 19:38Speaker 01

  I know.

• 19:39Hanne Tidnam?

  I think there's been a lot more apps since like, since that.

• 19:42Speaker 01

  We had kids.

• 19:43Hanne Tidnam?

  That helped, yeah. But still, how, I love that you said, it gives the illusion of control, not control. And in that particular, in these kinds of like data vacuums, like if we're not good at statistical analysis, or like pulling out causality from these murky areas, like if we're not Emily Oster, basically, how do you like, or even if you are, how do you kind of stay on that line of like the illusion of control versus like actual knowledge that like impacts real decision making?

• 20:13Speaker 01

  No, I think it's super hard because the thing is the illusion of control is a very powerful illusion.

• 20:18Hanne Tidnam?

  Very.

• 20:19Speaker 01

  And both empowering and dangerous in health, in health context, I think. Exactly. Like we would, you know, you would sort of, we like people to feel like they're in, they're in control. Some of the message of this book, I think people have taken, not quite right, but to say like, well, it doesn't really matter what choices you make, like all choices are good choices.

• 20:35Hanne Tidnam?

  Well.

• 20:36Speaker 01

  I think there's, that's not quite the right, it's not quite the message.

• 20:39Hanne Tidnam?

  I'm surprised that that's the message that people take from it.

• 20:42Speaker 01

  Occasionally. There are a lot of different good choices that you could make about parenting. And so I think that there, there is a piece that like, we maybe don't need to be so obsessive about all of these.

• 20:55Hanne Tidnam?

  About one of those.

• 20:55Speaker 01

  About one of those, any one of those, of those choices.

• 20:58Hanne Tidnam?

  Well, it's your point about range. It's like, well, let's give, let's educate a little bit more about like the spectrum of possibilities.

• 21:04Speaker 01

  The spectrum of good, of good choices.

• 21:05Hanne Tidnam?

  Yeah. Another area I feel like where every other day, there's a new study that says something different, and it feels like there's a plethora of studies, is screen time. I'm just, I'm gonna put that out there right now. I'm sorry, everybody, we're gonna touch that third rail.

• 21:21Speaker 01

  Great job.

• 21:22Hanne Tidnam?

  So can you walk us through, like, can you help guide us through some of that maze?

• 21:27Speaker 01

  So when I Looked into screen time, I had always thought about like, screen time is like, bad. Like, it's like, the question is, is it bad or not?

• 21:34Hanne Tidnam?

  Right.

• 21:35Speaker 01

  But actually, there's like, a whole other side of this, which is, some people like, screen time is the way to make your kid smart. Like, you can, like, your baby can learn from that.

• 21:42Hanne Tidnam?

  Okay, so, point number one is, what does screen time actually mean?

• 21:44Speaker 01

  Right.

• 21:45Hanne Tidnam?

  Which is, a bunch of different stuff.

• 21:46Speaker 01

  Yeah, and I think that's part of the, like, that's part of the problem with this, is like, when you say, "screen time," like, what do you mean?

• 21:51Hanne Tidnam?

  Yeah.

• 21:51Speaker 01

  Do you mean, like, you know, educational apps?

• 21:54Hanne Tidnam?

  Yeah.

• 21:54Speaker 01

  Do you mean, uh, FaceTiming with grandma?

• 21:57Hanne Tidnam?

  Do you like, jump in the shower?

• 21:58Speaker 01

  Right, exactly, and, or, yeah, and that point, like, do, while you jump in the shower?

• 22:01Hanne Tidnam?

  Yeah.

• 22:02Speaker 01

  Like, what is the other thing you're gonna be doing with your time?

• 22:04Hanne Tidnam?

  Yeah.

• 22:04Speaker 01

  I think this is where, all of these recommendations seem to assume, that the alternative use of your, of, like, if your kid wasn't watching Sesame Street, you would be like, on the floor, like, playing puzzles with them, and like, super engaged with them, which, like, maybe is true.

• 22:19Hanne Tidnam?

  Taking them to the zoo, and like, having them touch different textures of animal skins, or whatever, like, sensory development, yeah.

• 22:25Speaker 01

  Yeah, which, like, is great stuff that you should definitely do with your kid, but some of the time, when, you know, when our kids are, are watching TV, it's 'cause, like, we, it's, it's, that maybe isn't the thing that you would otherwise be doing.

• 22:36Hanne Tidnam?

  Yeah, where you could be, like, pureeing healthy vegetables, to like, feed them well.

• 22:40Speaker 01

  Yeah, exactly. I'm sure that's what we're all doing.

• 22:45Hanne Tidnam?

  Or maybe, watching a little reality TV for five minutes while you fold laundry.

• 22:49Speaker 01

  Like, a little bit of Call the Midwife, you know, got a little bit of, um, yeah. The problem with screen time is that the evidence is very, is very poor.

• 22:56Hanne Tidnam?

  Can you just break up, like, why the evidence is so poor? Because this does seem like an area where there should have been time for that kind of gold standard randomized study, that, to develop, no? What is the evidence problem?

• 23:07Speaker 01

  So, the, I think the evidence problem is twofold. Um, one, uh, it's actually not a super easy thing to run a randomized trial on, um, because these are choices that people are thinking a lot about.

• 23:17Hanne Tidnam?

  Mm-hmm.

• 23:17Speaker 01

  And, you know, think about something like an iPad, like, do you wanna be involved in a randomized trial of whether your kid?

• 23:22Hanne Tidnam?

  Oh, there's too much intention, too much at stake, too much lifestyle.

• 23:26Speaker 01

  Exactly, too much, like, lifestyle stuff. Some people have been able to use, like, the introduction of TV, which was sort of, had some random features to, like, look at the impacts of TV, and that evidence is sort of reassuring, and suggests that TV is okay, but of course, it's very old. It's like from the 50s.

• 23:40Hanne Tidnam?

  Whole different way of consuming.

• 23:41Speaker 01

  Exactly.

• 23:42Hanne Tidnam?

  Everything. Yeah.

• 23:43Speaker 01

  And I think the other thing is, the other problem with the sort of current, answering the current questions people want, like, what about iPads? What about apps? You know, is that they just haven't been around long enough. And so a lot of the kinds of outcomes you would want to know that even things like short run, like test scores. You know, I got the first iPad when my daughter was born. Like, that was like one, and I remember getting in being like, this is never going to catch on. This is why I'm not in tech. I was like, who would use this?

• 24:10Hanne Tidnam?

  Meanwhile, your daughter's like swiping.

• 24:12Speaker 01

  Meanwhile, yeah. I was like, okay. But, you know, now she's, she's in second grade. Like, that's kind of the earliest that you could kind of imagine getting some kind of, I don't know, what we'd measure test scores or something like that. But even, you know, she didn't use the iPad anywhere near as, in, near as like facile away as my, my four-year-old, right? This is evolving so quickly that any kind of even slightly longer-term outcomes are, are really hard to imagine measuring. Let alone, you know, absent a randomized trial, the, if you, like, if you weren't able to randomize this, which I think you won't be able to. The, the amount of time kids spend on these screens is really wrapped up with other features of their, of their household.

• 24:51Hanne Tidnam?

  Yeah. Okay. So you have the definitions, you have the time and the speed at which things are changing. And then you have the willingness for people to actually like engage in change or doing things differently. And then, so all of that leads to what kind of, so what the, what do the studies actually tend to look like in this space that we draw conclusions from?

• 25:12Speaker 01

  So actually, there's almost nothing about iPads or phones. Almost nothing.

• 25:16Hanne Tidnam?

  That seems so contrary to like what the media is saying every five minutes.

• 25:20Speaker 01

  Yeah. So there's tons of studies on TV, which compare kids who watch more and less TV. And, you know, you can, but most of that, again, is sort of studies that are like based on data where before the, before people were watching TV on these screens. You know, maybe TV is, is TV and, and, you know, you can imagine that that would be kind of similar. But, but things like these absolutely, just like no studies.

• 25:39Hanne Tidnam?

  Mm-hmm.

• 25:39Speaker 01

  You know, or there'll be, there's like, I think there's one like abstract from a conference. This is not a paper, it's like an abstract conference where it was just like, we have some kids. And we like compare the kids who like spend like the babies who spend more time watching their parents' phones and then they like do worse. They're like,

• 25:57Hanne Tidnam?

  Oh my gosh.

• 25:57Speaker 01

  But it's like, this is not, I mean, this is like terrible.

• 25:59Hanne Tidnam?

  It's pathetic. It's sad.

• 26:00Speaker 01

  This is a terrible piece of evidence.

• 26:01Hanne Tidnam?

  So is this an area in which you just go with your gut?

• 26:04Speaker 01

  I mean, I try to, I try to generate a fancy version of go with your gut, which is called Bayesian updating. And so, so I basically try to say, look, you know, I mean, we want to step back and think about what are the places of uncertainty? Logic would tell you, you know, your kid is awake for what it is like 13 hours a day, 12 hours a day. If your two-year-old is spending seven of those 12 hours playing on the iPad, then there is a lot of things that they are not doing.

• 26:31Hanne Tidnam?

  Right.

• 26:31Speaker 01

  That's probably not good.

• 26:32Hanne Tidnam?

  Right.

• 26:32Speaker 01

  On the other hand, you know, if your kid is spending 20 minutes every three days, it's very hard to imagine how that could be bad.

• 26:41Hanne Tidnam?

  So just thinking about it purely in terms of like time allotted to any one activity, basically.

• 26:46Speaker 01

  And then I think once you do that, then you're sort of like, okay, but you know, there are things that we're uncertain about, you know, what if my kid watches an hour of TV every day? Or spends an hour on a screen every day? Like, is that too, is that too much? Is the limit, if we sort of accept like five minutes a day is fine.

• 26:58Hanne Tidnam?

  Yeah.

• 26:59Speaker 01

  Seven hours a day is too much? Like, is the limit at an hour? Is the limit at two hours? You know, and I, I think the truth is what we will, we will find if we end up doing any studies like this, is that it depends a lot what other things they would be doing.

• 27:11Hanne Tidnam?

  Yeah.

• 27:12Speaker 01

  With their, with their time.

• 27:13Hanne Tidnam?

  Wouldn't it also depend so much on the child? Some children need, you know, learn in a kind of way that lends itself to this technology. Some children need other kinds of learning, you know, that

• 27:24Speaker 01

  Yeah.

• 27:24Hanne Tidnam?

  It's highly individual.

• 27:25Speaker 01

  Yeah. I mean, I think, I think this gets into the problem with studying older kids in general, that just like there's so much, there's so many differences across kids. It's hard to even think about how you would structure a study to learn about them, never mind actually like using evidence that exists.

• 27:37Hanne Tidnam?

  It's really interesting because the last time we went to take my daughter for her annual checkup, or maybe it was my son, I can't even remember it. It's so different from the first days of those early spreadsheet. We're now like, um, did I even get it?

• 27:48Speaker 01

  Which one is that?

• 27:49Hanne Tidnam?

  Yeah, exactly. Anyways, the doctor said very concretely, two hours, two hours max within any day. But it was really interesting to me that That it was such a specific line in the sand. And now I'm thinking about how that information would even get into that, to percolate down to that level of like the system and get kind of fossilized into the system so that that recommendation is being passed on to parents. Like, how does that happen with these studies? How do they translate to that level of advice?

• 28:18Speaker 01

  Yeah, I mean, I think what happens, the, what happens is like organizations like the American Academy of Pediatrics, they bring people together to basically talk about the, the conversation we just had, which was like, well, it's, you know, okay, let's agree, sort of like, we don't know that much about this, like, five minutes seems fine, seven hours is too much. These are like, smart people who see kids a lot, who presumably are using some knowledge that they have about kids to pick some number. But the answer is like, it, like, you could pick a lot of different numbers. We sort of say this, and then it becomes like this rule.

• 28:46Hanne Tidnam?

  Yeah.

• 28:47Speaker 01

  And people have some impression that it comes from some piece of evidence as opposed to sort of like, you know, a

• 28:54Hanne Tidnam?

  Some gold standard study.

• 28:55Speaker 01

  Synthesis of expert opinion or something.

• 28:57Hanne Tidnam?

  Yeah.

• 28:57Speaker 01

  Um, which is really what it's, um, what it's, what it's from.

• 29:01Hanne Tidnam?

  You also work specifically on the, on certain health recommendations. So how they change over time and how we stick to them. You wrote a paper on behavioral feedback. And then you talk about how those individual choices might in fact be changing the science itself. Can you talk about what that means and how that might be happening?

• 29:18Speaker 01

  I was thinking about exactly this issue of like, okay, we just make some recommendation and sometimes those recommendations are kind of arbitrary.

• 29:23Hanne Tidnam?

  Yeah.

• 29:24Speaker 01

  But then they go out in the world and like, people, people respond to them.

• 29:27Hanne Tidnam?

  Take on lives of their own.

• 29:27Speaker 01

  Exactly. And so, so like, a sort of, a good example, vitamin E is supplements, like in the early 90s, there were a couple of studies which suggested that like, they are good for your health, that like, prevent cancer.

• 29:38Hanne Tidnam?

  Okay.

• 29:38Speaker 01

  And so then there was like, a recommendation, like, people should take vitamin E. And, you know, then we had to ask the question, like, what, like, who takes vitamin E after that? And one of the, the concerns is the kind of people who would adopt these new recommendations, like, who, who listens to their doctor? It is people who are, you know, probably, maybe they're more educated, maybe they're richer, but, but like, above all, they are interested in, in their health. Right? So they are, they are taking vitamin E so they avoid cancer, but they're also exercising so they avoid cancer and they're eating vegetables so they avoid cancer. We use, call them selected. These people are like positively selected on other health things. And so indeed, you can see in the data that the people who start taking vitamin E after this recommendation changes are

• 30:23Speaker 01

  kind of also exercising and not smoking and, and doing all kinds of other stuff. Well, why is that like, you know, interesting or problematic? Well, later, we're gonna, we're gonna go back to the data because that's like the way science works. But now, the people who take vitamin E are even more different than they were before.

• 30:41Hanne Tidnam?

  Right.

• 30:42Speaker 01

  So now these people are like-

• 30:43Hanne Tidnam?

  So you've added another layer.

• 30:44Speaker 01

  Added another layer. So in fact, you can see that in the, in the data. You can see that basically before these recommendations changed, there was sort of a, a small relationship between taking vitamin E and like subsequent mortality rates.

• 30:55Hanne Tidnam?

  Okay.

• 30:56Speaker 01

  But after the recommendations change, you see like a very large, very large relationship between vitamin E and mortality rates. And so it looks like, basically ends up looking like vitamin E like is really great for you.

• 31:08Hanne Tidnam?

  Has this big impact.

• 31:09Speaker 01

  But of course, that's because, at least it seems like it must be, at least in part, because the people who adopt vitamin E are the people who are, uh, who are also doing these other things.

• 31:21Hanne Tidnam?

  So what does that mean then? It feels like such a loss. Like, how, how does one ever?

• 31:26Speaker 01

  It's so depressing.

• 31:27Hanne Tidnam?

  How would one ever develop like a recommendation based on what we think we know?

• 31:33Speaker 01

  I know.

• 31:33Hanne Tidnam?

  And untangle it from like-

• 31:36Speaker 01

  So this paper is very, is very destructive in some sense. So I'm like, other than saying like, it probably doesn't matter if you take vitamin E. So that's like, that's like news you can use, you can take at home with you. But I mean, I think it, it does, it, it does, yeah, more or less just highlight some of the, the inherent and very deep limitations with our ability to learn about some of these, some of these effects, particularly when they're small.

• 31:59Hanne Tidnam?

  Is this basically part of the sort of crisis of reproducibility?

• 32:02Speaker 01

  I think it's, I think it's not unrelated. Yeah. So I often think about this idea of P-hacking, which refers to the idea that you like, keep, you can like, you keep running your studies until you like, get a significant result. There are a bunch of, like, there's a bunch of people interested in this process of like how science, how like science evolves and the, the ways in which the like, the evolution of science influences the, the science itself or the incentives for, for research influence how, how science works. And I think it's, it's particularly hard- To draw conclusions in these spaces like diet.

• 32:37Hanne Tidnam?

  Yeah.

• 32:37Speaker 01

  Or these, these health behaviors where the honest truth is probably a lot of these effects are very small. So if you ask the question, like, what is the effect of like chia seeds on your health? My dad is like really into chia seeds.

• 32:49Hanne Tidnam?

  Yeah, that was a thing. There was a moment.

• 32:51Speaker 01

  Yeah, that's a thing. Well, he's still in that moment. He's still in there. And, and you know, like, what is the effect of those on your health? The actual effect is probably about zero. Maybe it's not exactly zero, but it's almost certainly about zero.

• 33:01Hanne Tidnam?

  But are there sometimes secret sleeper, like, whoa, there actually might? You know, the only way to find out is, is to do, yeah.

• 33:08Speaker 01

  Yeah, yeah. And so maybe there are some secret sleeper things. Maybe it is, but it's probably not. You know, I don't, I, you know, I spend a lot of time with these diet data and, you know, there's, there's this sort of like dietary patterns, like the Mediterranean diet, which do seem to have some sort of vague, you know, support in the, in the evidence. But I would be extremely surprised if we ever turn up like the magical food.

• 33:34Hanne Tidnam?

  So the point is, it's the pattern, not the food.

• 33:37Speaker 01

  And, and it's all the other things that you're doing.

• 33:38Hanne Tidnam?

  Yeah.

• 33:39Speaker 01

  If you smoke three packs a day and you never exercise, but you eat some kale, that's not going to help you.

• 33:45Hanne Tidnam?

  Yeah, yeah.

• 33:46Speaker 01

  The kale's not going to help.

• 33:47Hanne Tidnam?

  What about when you really do need to affect change? What are the ways in which these guidelines can shift over time with kind of new sources of information or data and statistics? Like, what's the positive? How does that actually play out in a, in the right manner?

• 34:02Speaker 01

  Yeah. So I think there, there are times in which we, the, the change in evidence is so, is so big and so like compelling that we can get changes. Best practices in, in obstetrics, like how do you deliver a breach baby?

• 34:18Speaker 01

  As an example, they change, like those changed over time because there was like one very big, well-recognized study that everybody agreed, like, this is now the state of the art.

• 34:28Hanne Tidnam?

  And it happens fast at that point?

• 34:29Speaker 01

  And then it happens pretty fast. It doesn't happen immediately. Like, you might've thought that those kinds of changes could be like immediately affected. And I think that they're, that they're not, but they do happen over time. Those examples really rely on there being like a. A cohort of sort of like experts who are all reading the guidelines and sort of seeing that they changed and then themselves are sort of doing this all the time. I think part of what's hard in the broader health behavior space where it's people who need to make the choices, not physicians.

• 35:02Hanne Tidnam?

  Yes, when it's in the home, in those dark bedrooms.

• 35:05Speaker 01

  Right. It's like, you know, that's much harder to get people to change their behavior in those spaces.

• 35:10Hanne Tidnam?

  It's not these pediatric guidelines. Those are not effective.

• 35:13Speaker 01

  Yeah, I do not think those are effective. Or I think we don't see any evidence in the data that those are effective at moving these, at least in these kind of spaces.

• 35:20Hanne Tidnam?

  So what's the answer? How do we positively affect change and gather these insights and have smart people making good recommendations?

• 35:26Speaker 01

  So I mean, I think one answer is media attention. The kind of few times when we see very large spikes and changes, they actually seem to correspond with some media coverage. On the flip side, like, media can often be very bad. Some of these big changes in these expert things were kind of resulted from media coverage, which was really like sensationalist and like totally inappropriate.

• 35:47Hanne Tidnam?

  Yeah.

• 35:48Speaker 01

  And, you know, it wasn't like a very nice, like, New York Times story about some study. It was like a sensationalist 2020.

• 35:56Hanne Tidnam?

  About what?

• 35:57Speaker 01

  About, this is about vacuum extraction, which is a way of pulling the baby out and has gone down a lot over time. And it was like this sort of sensationalist, like, John Stossel 2020 episode about how it could hurt your baby, which caused like big reductions.

• 36:11Hanne Tidnam?

  Interesting.

• 36:12Speaker 01

  But, you know, that like...

• 36:14Hanne Tidnam?

  But the science was there.

• 36:15Speaker 01

  The science, yeah, was. I mean, he overstated the science. But it was, it was probably there.

• 36:21Hanne Tidnam?

  So it's almost like a random confluence of like, when the science is there and the media hits it the right way, and then we see change?

• 36:27Speaker 01

  Yeah.

• 36:27Hanne Tidnam?

  Which is...

• 36:28Speaker 01

  It's something to hope for.

• 36:30Hanne Tidnam?

  Yeah, that doesn't feel like we can plan so much for that.

• 36:34Speaker 01

  No.

• 36:34Hanne Tidnam?

  You also study when we are resistant to change. You looked specifically at diabetes, people I think who had been diagnosed with diabetes, and then whether or not their behavior changed, even given a certain amount of information. So what do you see there about our resistance to change even with the right kinds of information?

• 36:52Speaker 01

  Yeah. I mean, I think one of the big challenges in the health space at the moment is that like so much of the, so many of the health problems that we have in the US are like problems associated with behavior. Just the fundamental fact that like people do not eat great and we have a lot of... Morbidity and expense associated with that. And I think there is often a lot of emphasis on the idea like if we just get the information out, if people just understood vegetables were good for them, they would eat vegetables. Doesn't happen. That's not true. Yeah. I think, and so, you know, this paper is about sort of looking at something where kind of a pretty extreme thing happens to people, like they are diagnosed with diabetes and we can see what happens to their diet. And the answer is, you know, it improves a tiny amount. Even with a real come to Jesus moment. Exactly.

• 37:37Speaker 01

  And a lot of new information. Right. Like, you know, and monitoring, follow up, right? I mean, you're diagnosed with diabetes, like, you know, you have to take medicine every day, you've got to go to the doctor, like, get a test your, you know, test your insulin, at least for some period of time. So this isn't like something where you can just forget that it, that it happened. And even then, the changes in diet, you know, they're there, but they're really small. They're like, you know, one, like one less soda a week or something like, really, like really small. And how are you noticing these? We're inferring information on diagnosis from people's purchases of testing products and then following their grocery purchases. So this is like an example of using, you know, a different kind of data. Yeah. So not health data in this case, it's actually like, like Nielsen data, so Nielsen data on, on what people buy.

• 38:21Speaker 01

  But then, you know, using some like machine learning techniques to try to figure out from the kinds of things people buy, when were they diagnosed with diabetes, and then looking at their diets over time. So is the answer that there has to be some sensational story that talks about? I mean, I think there, I'm not even sure that would help. I think part of the problem is people really like the diets that they're comfortable with. Like diet is like such a habit formation thing and, and, you know, people are, are willing to, to make important health sacrifices to maintain the diet that they, that they like. We get into some of these questions of preferences, like, and, you know, if people, if that is the choice that people want to make, like, should, should we be trying to intervene with, with policy? Like, let's say everybody had all the information, they knew that they shouldn't drink so much

• 39:06Speaker 01

  soda and that they should lose weight, but they still chose not to. Like, do we want to develop policies that affect that? I'm not sure.

• 39:13Hanne Tidnam?

  Yeah, or maybe that's just free will.

• 39:15Speaker 01

  Yeah, maybe that's just free will.

• 39:16Hanne Tidnam?

  Yeah.

• 39:16Speaker 01

  And it comes up in the parenting stuff, too. Like, you know, how much do we want to be externally controlling the choices people make with their, with their kids, even if we don't think that they're the right choices?

• 39:24Hanne Tidnam?

  Mm hmm. But I do think there's a segment of people who want to make the change, but the gravity, you know, because of the information, but the gravity of the habit is so much that it's hard to know where to go about it. I guess I would say, where do you see this data going? Like, if you had your fantasy for where you want the kind of data and the way that we see this data evolving and the way that you see that kind of percolating out to the public? I mean, in terms of being sort of a translator and providing people the tools, like, what do you want to see in terms of the way the system responds to or integrates this data in the future?

• 39:57Speaker 01

  Yeah, I mean, I think the, the big message of the book is, is in some sense that, like, you, you should use the data to make yourself, like, confident and happy in your, in your choices. I think so much of what is hard about parenting is that in the, in the moment, you are not often confident in your choices. And then when somebody asks you, like, why did you do that? Then you feel bad.

• 40:19Hanne Tidnam?

  Yeah.

• 40:19Speaker 01

  Right? And I, I think that there's, there's a sense in which sort of looking at the data, but then confronting, like, well, we don't know, but you'd be like, okay, I made this choice. You know, I decided to let my kids watch an hour of TV every day.

• 40:29Hanne Tidnam?

  Yeah.

• 40:30Speaker 01

  Because, like, I thought about it and I thought there wasn't any data and, like, that's the choice that, that I made. That sort of that confidence is, like, important for, for being happy. And if we could sort of, like, move in that direction, I think that would be, that would be good.

• 40:44Hanne Tidnam?

  It reminds me a lot of what one of my good friends, Branny, said to me when I was in the trenches of, like, babyhood and having a lot of anxieties around all these hot button issues, breastfeeding, sleep time, like, all of it. She had been through it, her kids were in college, and she was like, let me give you a piece of advice. Be wrong, but be wrong with confidence.

• 40:59Speaker 01

  Yes.

• 40:59Hanne Tidnam?

  Just be wrong with confidence. That's all that matters.

• 41:02Speaker 01

  Yeah. No, exactly. Be wrong with confidence. I love that. Yes, I am wrong with confidence so frequently.

• 41:06Hanne Tidnam?

  And actually turns out to be right.

• 41:08Speaker 01

  It's fine.

• 41:09Hanne Tidnam?

  The truth is, there's a lot of good options.

• 41:10Speaker 01

  A lot of good options.

• 41:11Hanne Tidnam?

  Yeah. Thank you so much for joining us on the A16Z Podcast.

• 41:14Speaker 01

  Thank you for having me.