Revisiting the China Shock (with Joseph Francis)

What if the results of one of the most influential economics papers of the past 25 years are unreliable? The "China shock" narrative--that a surge of Chinese imports destroyed American manufacturing jobs and the communities that depended on them--became conventional wisdom in Washington and beyond. But economic historian Joseph Francis argues that the claim is much weaker than it looks. Listen as he tells EconTalk's Russ Roberts how a 2022 correction to the paper’s research design cut its estimate of lost jobs in half and erased its findings on wages and living standards. He and Russ then look at what this affair reveals about economics as a field, including the incentives that reward dramatic results, and the problems with peer review.

• 0:02Speaker 00

  Welcome to Econ Talk, Conversations for the Curious, part of the Library of Economics and Liberty. I'm your host, Russ Roberts of Shalem College in Jerusalem and Stanford University's Hoover Institution. Go to econTalk.org where you can subscribe, comment on this episode, and find links to other information related to today's conversation. You'll also find our archives with every episode we've done going back to 2006. Our email address is [email protected]. We'd love to hear from you. Today is October 13, 2026, and my guest is economic historian Joseph Francis. He is

• 0:42Speaker 00

  an honorary research fellow in the Department of Economics at the University of Birmingham. Our conversation is going to discuss his recent article in the September 2026 Econ Journal Watch. The title of that piece was "Whimsy and Fragility in the China Shock Narrative." In short, we're going to be talking about a very, very famous paper, an influential paper on the impact of Chinese manufacturing on employment, manufacturing, standard of living in the United States, and some of its implications. Joe, welcome to Econ Talk.

• 1:21Speaker 01

  Hello, welcome.

• 1:22Speaker 00

  And I also want to add that I interviewed David Autor on this program back in 2016 about his, that research, and we'll link to that. So, that body of work, and there were a number of papers on it, but they were all looking, a lot of them were looking at the impact of Chinese imports into the United States, was correlated, not necessarily causally, but correlated with a terrible time for manufacturing employment in the United States. So, between 2000 and 2010, the United States lost about 6 million jobs in manufacturing,

• 2:05Speaker 00

  which is about a decline of about a third, which is quite dramatic in such a short period of time. And it coincided, perhaps causally, when imports from China, in nominal terms, went from, in that 10 years, 100 billion, which is already at a high level, 100 billion seems like a lot of money, although as a percentage GDP, it's not as big as you might think, it's about 1%. But it went to 365 billion, which is about 2.5%. So, it's a dramatic increase. And many, many people argued that that was due, the loss of manufacturing employment was due to Chinese imports. And what did, let's start with what Autor, Dorn,

• 2:47Speaker 00

  and Hanson, sometimes called ADH, but I'll try to say their names because listeners will get perhaps confused. Autor, Dorn, and Hanson tried to tease out, measure, whether this was causal, whether the increase in Chinese imports to the United States, their exports are United States imports, whether that had destroyed manufacturing in many, many places, or dealt it a grievous blow. What were they, what were they trying to, how were they trying to measure that?

• 3:20Speaker 01

  So, they were trying to, they were using imports from China, but they didn't use imports from China into the United States because they were endogenous. So, imports from China into the United States responded to US demand, whereas they were interested in the supply shock, the supply shock from China. So, they were, their framing is that they want to look at the, this one-off kind of world historical transition from communism to capitalism in China, and to see what those effects were in the United States. But they can't do that just by looking at the imports into the United States, because that also

• 4:02Speaker 01

  responds to demand in the United States. So, instead, they used an instrumental variable based on Chinese imports into, I think, seven or eight other OECD countries in Europe and Japan. And they used those imports as an exogenous measure of the supply shock in China.

• 4:30Speaker 00

  And we should just, we should just clarify one thing, which is, at the same time, China's trade status with the United States had changed, right? So, China's transitioning from communism to capitalism, sort of, that's a crude statement, but suddenly China was able to produce a great deal more than they had before, and they were able to sell to the United States at better terms. So, the United States definitely bought a lot more stuff, whether that was because of Chinese productivity, that transition, or whether it's because, as you point out, that Americans wanted to buy more manufacturing goods, is, is the question. But it's also, there are many, many, many, many other things happening

• 5:14Speaker 00

  at the same time. And the challenge of econometrics or statistics is to try to isolate the impact of China from those other things. For example, the world was getting more productive, there's technological change, there's innovation in producing, in manufacturing generally, with being able to produce things with fewer people. So, we want to blame, I put that in quotes, because I don't really believe that's the right word, but if we want to, I'll make a more neutral term, if we want to attribute the collapse in manufacturing in the early 2000s to China, you have to be able to hold those other things constant using statistical techniques.

• 5:53Speaker 01

  Yeah, they were, they were trying to get the causal effect of China's transition from communism to capitalism. So, Autor et al. were trying to, they were trying to look at the effect of China's, the Chinese miracle, if you like, China's transition, by separating out the, the effects of the increased supply, and the great productivity growth in China, from demand factors in the United States. So, whatever might be causing an increased demand for imports in the United States. And with this statistical technique,

• 6:32Speaker 01

  they, they claimed to have found that imports from China had this big causal effect, that it caused a decrease in the manufacturing share of employment. And it also had a lot of negative effects on living standards in the United States, things like unemployment, wages, household income, welfare transfers from the state. That was in their 2013 paper. Then they published other papers afterwards, which said that there was a negative impact

• 7:10Speaker 01

  of Chinese imports on things like mortality, and the marriage market, and politics. They said it, it drove a lot of the vote for Republicans, and had an impact on the 2016 election. And what I found, and I'm really the messenger at this point, I'm not actually an econometrician, I am an economic historian. And I got into a habit of doing what you call replications, or post-publication reviews of published papers. And I did various reasonably well-known

• 7:52Speaker 01

  articles. And then I said, well, what's the most famous, most influential economics article published in the past 25 years? And that's where I found this Autor et al. for China Syndrome. And I spent two or three months looking at it, trying to figure it out, trying to understand it with my, with my quite limited understanding of econometrics, using AI and large language models to help me to explain it to you repeatedly, as if I was a five-year-old, which they very patiently did. And I focused exclusively on the manufacturing share of employment, because that's what everyone else is focused on. This article by Autor et al., it's had a lot of discussion, debate, and everyone's about the effect

• 8:37Speaker 01

  on the manufacturing share of employment. And I did all the kind of robustness tests I could to see whether it was robust. And I ended up with the same conclusion as these three reasonably young econometricians, who obviously know what they're talking about, called Kerol Borosyak, Peter Hull, and Xavier Haravel. And they published an article in 2022 in the Review of Economic Studies, in which they, they found some problems in Autor et al.'s study. They found that Autor et al. hadn't distinguished very well between

• 9:21Speaker 01

  the general decline of manufacturing towns in the United States, and the specific decline caused by China.

• 9:30Speaker 00

  Meaning that if, if your town produced something that China didn't produce, and it also declined, you wouldn't want to mistakenly attribute that to the China shock. But their technique was not able to separate those very effectively, right? Is that a good summary?

• 9:48Speaker 01

  Yeah, yeah. So, Borosyak et al., they found that a lot of the effect on the manufacturing share of employment that Autor et al. had found was basically a comparison between manufacturing towns and towns without any manufacturing, or with very little manufacturing, with just service towns. And that meant that, that they couldn't really distinguish between what China had caused from 1990 onwards, and what had been a very long-term, from the 1970s onwards, decline of manufacturing in the United States. So, to fix that, Borosyak

• 10:28Speaker 01

  et al. introduced a new controls into the regression, which meant that they were comparing manufacturing, it's not, it's like commuting zones in the United States, so these, these kind of areas which are all linked around a town. They were comparing commuting zones with similar levels of manufacturing. So, what really mattered was how exposed they were to Chinese imports within the same level of manufacturing.

• 11:02Speaker 01

  And what they found was that the effect on the manufacturing share of employment, once they introduced these new controls, fell by about half. It fell from about 0.6 to 0.3. I can't quite remember what the units of the coefficients are in. So it fell by about half.

• 11:19Speaker 00

  Yeah, and the original, when I interviewed David Autor in 2016, it was, at that point, he and his co-authors were saying that about a quarter, 25% of the drop between 1990 and 2007, which is a different time period than we've talked about so far, but it's before the 2008 recession. So that's an argument for leaving out the later couple of years of, say, to 2010. But from 90 to 2007, about 25% of the drop was due to China and about 45% of the drop

• 11:58Speaker 00

  from 2000 to 2007, which is maybe 1.5 to 2 million jobs. So it might only be, once that correction is made, it might only be 700,000 to 1 million, rather than 1.5 to 2 million, once that correction's made. I would just point out that, at that point, a rough number of jobs in the United States is about 140 to 150 million over this, the time period, around 135 to 140 million over the time period we're talking about. So you can debate whether that's a large number or a small number, but it was treated like a very large number. And of course, for the

• 12:41Speaker 00

  people who lost their jobs, who may not have easily found new jobs, which was part of the claim, it was very brutal and it was very localized, which, meaning it was very, put in various places in the United States, which could suggest very powerful political changes were put in motion because a large group of people in a small area like a state were giving a big impact. And so it was still taff, but it's still a big impact.

• 13:10Speaker 01

  Yeah, the period in all this is 1990 to 2007. So it ends in 2007. The 45% figure that's in the abstract, and I know that he said on your podcast as well, that's actually a different issue, which probably comes on to later. Their regressions can't actually produce that figure. And that was something else I found independently of the Borussia Kettl corrections. Their corrections were to the kind of the raw regression coefficient, which is, I can't

• 13:55Speaker 01

  remember, I honestly can't remember what the units are. It's per $1,000 of exposure to Chinese imports, something like that. And then the effect fell by about half. And that meant that people looked at the Borussia Kettl article, which was 2022, and they said, okay. So they introduced some controls, the effect survived, so Autoratals research design survived. And as I was doing this long process of reviewing it, I was like, oh yeah, okay, well, that's fair enough. But then eventually I had this kind of eureka moment where I said, well, what about all the other effects that they report, which is actually what their abstract's really about, all the stuff on unemployment, about wages, about all the

• 14:35Speaker 01

  living standard stuff. What effects does the corrections, the Borussia Kettl corrections have on those results? So I started to look, and I found it on page 45 of their online statistical appendix, of Borussia Kettl's statistical appendix, where they actually showed that their corrections kind of eliminated all those effects, all those negative effects on living standards. But they never mentioned it in the actual paper, in the actual article itself. They just said, there's a footnote in which they say, "We also look at the effects on living standards in table C.1 in the online appendix."

• 15:18Speaker 01

  But these were actually kind of the main results. If you look at Autoratals' abstract to their original article, the abstract is mainly about the living standards effects. And the Borussia Kettl corrections really shown that those effects didn't survive once you adjust the regressions to correct for the research design errors.

• 15:48Speaker 00

  So the reason this is important is that it's not surprising that if the United States is importing a lot more from China, that it's possible that it will take away manufacturing jobs in the United States. But there was another effect, and this, of course, when I interviewed David back in 2016, David Autor, we talked about this at some length. Economists have known this forever, that that's not interesting. What's interesting is what happens next. In theory, if America's importing an enormous amount of less expensive goods from China, there will be resources available to spend on other things. And the

• 16:29Speaker 00

  things that expand will increase the demand for jobs. The standard of living, in theory, should go up, although their adjustment costs, it's a question of how long it takes for adjustments to take place. But the idea would be that you can't, certainly don't want to evaluate the expansion of Chinese imports based on what happens to US manufacturing in areas where factories are competing with China. You want to see what the full effects are, and they don't pretend to measure those. But when you look at the, on the employment side, but when you look at the wages and standard of living, the fact that this methodological

• 17:08Speaker 00

  correction in appendix C, in the online, unpublished but online available adjustment, shows that it's actually no impact on wages and standard of living, it is kind of shocking. Why do you think they kept that quiet in the online appendix? It's there. I looked at it. I checked before we did this, because it's shocking. If I may use a bad pun unintentionally. It's very surprising that this very technical paper, which, as you say, left the impact at half of what it had been before, but still a sizable impact, that the other rest of the effects, they just thought, eh, we'll just not mention them.

• 17:51Speaker 01

  I can only speculate on that, Russ. I think the way that the, the way the academic publishing system works, especially in economics, young researchers especially are under a lot of pressure to get publications in top five economics journals. That's what they have to do to get tenure. And, and you might not want to upset senior economists if you want to get published in top five journals, would be my speculation.

• 18:34Speaker 00

  That's a very cynical one, but it could be true.

• 18:41Speaker 01

  Yeah, I'd say it's realistic, but it's cynical.

• 18:44Speaker 00

  One man's cynical is that other man's realist.

• 18:47Speaker 01

  The way, the way that economics is currently, and the institutions and incentives in economics, I think, is, produces that kind of result. And in this case, it meant that we've had three or four years, four years, say, without anyone actually noticing that those results were there tucked away in the appendix.

• 19:15Speaker 00

  But to be fair to Autor at all, they might not, have they responded to that paper that critiqued their assumptions?

• 19:26Speaker 01

  I don't think so. I think that they published, they published a chapter in one of the, one of the economics handbooks last year, on which they did incorporate, they incorporated some of Borussia Kettl's corrections. It's not entirely clear how many of them. They might not have, and, but they didn't, certainly didn't say anything about the broader effect of those corrections. Possibly they're not aware of them. I mean, they might not have got to page 45.

• 20:01Speaker 00

  And that, that...

• 20:05Speaker 00

  That paper, the 2013 and later papers, they've been cited, it's not like, you called it the most influential paper of the last 25 years. It's been cited, I checked, thousands of times, mostly just in passing, not critically. And you could have it, we could have a conversation, I don't know if it'd be fruitful, but you could argue it mainly had an impact on the economics profession. Certainly, it was used in the phrase China shock, which also I checked on Pangram, had an enormous rise in usage after the 2016 paper.

• 20:50Speaker 00

  It was invoked by politicians and pundits, for sure. But the correlation between imports from China and manufacturing employment, most people don't care whether it's real or not, it's just something they used. And I think many, many political people I've spoken to in the United States, people involved in politics, you know, treat this like an obvious fact, that America's globalization, that its willingness to import willy-nilly from China has destroyed the American economy. And it's one of the reasons I mentioned that, you know, it's a small portion of the total employment, again, the human costs are not small for the people who suffered them if

• 21:35Speaker 00

  they did not find other work. But this became, certainly Otter et al's work, Otter Dorn and Hanson's work, was used, whether it was persuasive, I'm suggesting is not so clear, was definitely used to prop up a variety of policy decisions made over the last 10 years that have really changed the US perspective on foreign trade. Comment on that, if you would.

• 22:04Speaker 01

  Yeah, I think, I've done a little bit of research on this. The other day, I was looking at it, and it was interesting, I looked back at 2005, and I found a New York Times article by Nick Kristof, a commentator in the New York Times, who, he said that American elites were using China as a scapegoat, basically, to distract attention from their own shortcomings. And my opinion, which is just my opinion, is that that process has kind of continued.

• 22:47Speaker 01

  And some of the people who responded to Nick Kristof 20 years ago were the same people who are still saying it's the China shock now. And it's interesting, because back then, it was Democrats as well, it was the Democrat politicians who were saying it was China, China. And it's almost like Trump has brought the Republicans on board with that Democrat position. And now, it's kind of a bipartisan consensus there, which is quite unusual in the US, I believe.

• 23:28Speaker 01

  And my opinion would be that it's a way to avoid having debates on other issues that you really need to have debates on over there. And it's a way of avoiding kind of other questions, which it's much easier to blame foreigners. And it's a long tradition in the United States as well, especially blaming China. I mean, I'm a historian of the 19th century mainly, and there was a lot of kind of anti-China rhetoric used then as well. Yeah.

• 24:05Speaker 00

  Yeah, blaming foreigners is, goes back a lot more than the 19th century, but...

• 24:11Speaker 01

  And it's not just Americans as well, to be clear.

• 24:14Speaker 00

  For sure. And I just, I want to put the alternative hypothesis on the table, and then we'll talk about the reliability of econometric work generally, and why you titled your paper, how you titled it. The alternative hypothesis, certainly, the acceptance of China into the World Trade Organization increased imports from China into the United States. And it's certainly true that the transition from China, in China, from a more state-controlled, top-down system to a more capitalist, one, increased the amount of the productivity

• 24:58Speaker 00

  of China, increased their standard of living, had extraordinary transformation of their economy from agricultural to urban, enormous migrations of hundreds of millions. It's one of the most dramatic economic events of all time, in terms of magnitude. So, that's going to happen, that happened, that's not up for discussion. What's up for discussion is, how responsive is the US economy, and especially the labor market, in response to that challenge? That's one question. The ability and willingness of people to move toward economic activity if their local activity is being hurt by, say, imports.

• 25:40Speaker 00

  But the third thing that's going on, that I think is extremely important, is the revolution of, the application of technology to manufacturing that has reduced the employment of workers there in any one factory, but at the same time, an enormous expansion that's statistically challenging, you have to be careful, but a lot of expansion of manufacturing, whether it's computers only and others, we've had episodes on that. But the point is, is that the United States economy is not being hollowed out, it's not

• 26:16Speaker 00

  somehow being destroyed, and that these trends that have become so politically salient, started, you said, I think, I can't remember what year you started, but they really go back to 1945. At the end of World War II, going to the present, proportion of the US economy manufacturing has fallen steadily. That's not a coincidence, it's not an external effect, it's not a measure of how poorly the US economy has done, it's a tremendous tribute to human creativity in figuring how to make more stuff with fewer people using technology and innovation.

• 26:58Speaker 00

  Now, we could make a decision as a nation, the United States could, to not let that happen, it'd be really hard to implement, but it's really important whether that transformation of the manufacturing sector in employment is caused by foreign imports versus internal innovation and the application of technology. Because in one case, you can justify to protect local people from change, some kind of perhaps tariff policy, but otherwise, you're barking up the wrong tree there. So, it's, for those who are listening, we've had a number of episodes on trade over the

• 27:39Speaker 00

  years, and we'll link to them, but it's important to remember that a lot of things happen at the same time, and to blame the one that is convenient is a very dangerous policy, not just because it's an error and it's a mistake, it can lead to very bad decisions in terms of policy.

• 27:57Speaker 01

  Policy and geopolitics and all sorts of things. And, I mean, my prior as an economic historian is that countries tend to industrialize, and then they tend to deindustrialize, because as they become wealthier, people don't want to work in manufacturing anymore, and it's basically a wealth effect. And that's facilitated by globalization. And if it hadn't have been China, it probably would have been another country supplying those goods. Perhaps China might have accelerated the process slightly, but it was a long-term, ongoing process that was then accelerated by China, maybe.

• 28:37Speaker 01

  But it might have been Japan, it might have been Malaysia, it might have been India supplying the American market, if China had stayed a communist basket case. That's my opinion, more than a reasoned or researched position.

• 29:01Speaker 01

  And I think, I think the question for the United States is how it adjusts to that. And in international trade, there's always, it's always has different regional effects. That's been known since Olin in the, in the 1930s, I think.

• 29:17Speaker 01

  So if you're going to be involved in international trade, you have to accept that, especially in a big country like the United States. So you could have regional programs to help, help support places that were losing out and things like that.

• 29:32Speaker 01

  Or there might be other solutions. You, you and I might disagree about that, which is, which is why you really need to have a debate about it. Rather than just saying it's China.

• 29:44Speaker 01

  Because blame in China and trying to reverse it could, could have adverse impacts. I'm also an economist who's doing a bit of Argentina.

• 30:00Speaker 01

  So I, I know a bit about protectionism and industrial policy, how it can go wrong.

• 30:09Speaker 01

  I'm actually on the left myself, so I'm, I'm a kind of, I'm a leftist, but I, I do kind of disagree with some of my, my leftist fellow travelers about, about the benefits of industrial policy and protectionism. Because from, from studying Argentina, I do know that it's not as easy as it sounds.

• 30:35Speaker 01

  And, yeah.

• 30:39Speaker 01

  And that's also why Otter Atal is so important, because they did seem to have find this big causal effect of China, which just, just might not have been there.

• 30:50Speaker 00

  And, and to be maybe clarify what I was saying earlier, you know, between 1945 and 1990 or 2000,

• 31:00Speaker 00

  manufacturing as a source of employment became less and less important in the United States economy. As a source of output was very, very healthy, but certainly as a source of employment, it was less so. And it was not a crisis for the United States that manufacturer employment was falling. People who normally would have taken those jobs found other jobs. There was still most years full employment.

• 31:26Speaker 00

  If it, if it wasn't, it wasn't because of manufacturing becoming less active, because, because of other macroeconomic effects. So what was the, the claim here that was just, that was, this was different because it was so large and so sudden. And it was hard for the US economy to adjust, the labor market to respond, perhaps as it had in the past. So that's part of the conversation. That's why I mentioned, you know, the ability to move to where economic opportunity might be in a different part of the country. If housing markets are not particularly healthy, it can be very expensive to move to the places where the best jobs are.

• 32:06Speaker 00

  And there's a related question, which we haven't mentioned, which is, you know, a lot of the people who work in manufacturing, not all, but some of them are relatively low levels of education and, and may have trouble finding a job that's equally attractive, given their levels of skill and education, if manufacturing is no longer thriving and healthy. And that's a serious problem for a country. And I, I think, you know, I've always felt that the last 20 years, that is certainly part of what has produced a lot of anger in the United States. And a feeling that the elites don't care about the less educated, say.

• 32:44Speaker 00

  But, but to blame that on China is a special case that economists have, have accepted. And it's very much a change in their perspective on trade generally. In the past, they were much more comfortable and confident that while trade, of course, hurts certain people, while the general country might thrive, that this was such of, of such a magnitude that we, that we needed to do something about it.

• 33:14Speaker 01

  Yeah, I, I think it's, I think it has been a big change. I think it's odd because,

• 33:20Speaker 01

  because it's, it's been known for a long time that globalization produces winners and losers. And, and in, in relative terms. And really, if you're going to embrace globalization, you need to do something to compensate the losers. Because otherwise they get angry and vote for politicians who, who can then do unwise things.

• 33:45Speaker 01

  And that's quite sensible. And it's kind of, it's kind of obvious. And it's been just known for a very long time. So some people, some people who responded to my, my critique of Otter Atal, they said, okay, okay. So the economics wasn't great and the effect wasn't so big, didn't have these effects on living standards. But they, they at least made it really clear that international trade does have these different effects on different people and different localities. And, but I think economists have known this for, for, for a very long time. And,

• 34:25Speaker 01

  and possibly it's because they, they, they've become much more beholden to kind of simple causal stories in, in recent years, that they, they like this kind of China caused it story.

• 34:41Speaker 01

  But I think it, it probably says something about economics that it has taken hold in this way. I'd say this is a good thing about econometricians, because econometricians have looked very closely at Otter Atal, like Borrese Akatal. And there are others before them who I discuss in, in the article as well. And they, they really looked at this result in, in, in the original 2013 article. And they put a lot of time and effort into, into picking it apart, I suppose. And, and I'm really just the messenger who's bringing what, what they found to more kind of work wider notice, maybe. Yeah.

• 35:22Speaker 00

  Yeah, it's, um,

• 35:25Speaker 00

  I mean, there's a moment, let me tell the story more accurately because, or more, in a more interesting way. Because I don't know if I mentioned it when I interviewed Otter back in 2016. I was at a seminar where an economist was making a very dramatic claim

• 35:45Speaker 00

  about some impact of something or other. I don't remember what it was now. And I asked a question and I have to confess, this is maybe, I don't know, 10 years ago. So I've been, at that point, I've been an economist for about 35 years. And I asked a question I've never heard anyone ever ask at a seminar. I asked it with some trepidation. I said, how many regressions, regression being a statistical attempt to estimate something using a statistical package typically. I said, how many regressions did you run?

• 36:16Speaker 00

  And the reason I asked was because you can always run some more, try something different, add a different variable, change the way you, the instrumental variable you alluded to earlier, which is a way to control for correlation and, and try to isolate causation more effectively. You can always run some more. And many people have suggested, we'll talk about that in more detail in a minute. But many people suggested that this is really not a healthy way to look for the truth. But it is the way our profession, this is, this is what you'll see in the kitchen. And I've, I've been in the kitchen, like you, I'm not an econometrician per se, but I've done empirical work. And you can always do some more until you find a result that maybe is a little more comforting.

• 37:00Speaker 00

  Now, it might take a lot of torturing of the data, that so much so that people say, you know, that's not a reliable finding. But in the more complex world that we're talking about, it's very hard to know how, what it means to do something that's justifiable or true. You can always do something one more time. So I asked him, how many regressions did you run? He didn't know what I meant. He literally looked at me, what? He, he couldn't answer the question. And I said, no, because I'm sure you're, I said, I'm sure you ran a lot because that's how the world works. And I'm curious to know how many of times in all those regressions did your finding stand up, hold up, stand out?

• 37:45Speaker 00

  And it was, you know, it's a rhetorical question. So I asked David Otter that question. And I didn't let him answer because I knew it was, it's not a meaningful question. It's just a, it's a cheap shot in a way. It says, look, I know how the business, how we do our work.

• 37:58Speaker 00

  We run a lot of regressions till we find something that's publishable or that we think we can defend. And that's the wrong way to do econometrics in my view, but it's the way it's done.

• 38:10Speaker 01

  Yeah, I, I think, I think it is. I think, I think what happened, there was, there was what they call the credibility revolution in, in economics. So if I go back to when I, when I was younger, say, 20 years ago, my, my critique of, of economics was it was all too theoretical. Like economists, they were just kind of in this unreal world of pure theory.

• 38:33Speaker 01

  And then suddenly all that changed. Economists, they all started doing lots of empirical work in what they called the credibility revolution. And they introduced, they started using these econometric techniques.

• 38:47Speaker 01

  And I think the econometricians of that generation, they were very rigorous, but they also, they also kind of had this attitude of, oh, here's my shiny toy. Would you like to play with it? And, and the economists went away and played with these shiny toys and, and produced all these papers. Said, oh, we've, we've proved that this, that X caused is why.

• 39:09Speaker 01

  But now I feel with the econometricians of today, the younger people, that they're kind of pushing back a bit on that, that they're saying, well, actually, maybe we shouldn't just play with shiny toys. Maybe we should be slightly more serious and actually kind of make things a bit more robust. And I've been studying economics, kind of as an outsider of economics now, for about a year, I suppose, doing a lot of these reviews of different articles, different papers. And I've been very impressed by the work of econometricians about trying

• 39:46Speaker 01

  to make it more robust. And I think this is one example of how a research community of econometricians have made autore tells work more robust and actually changed the results in doing so, changed what the research design actually shows. And, yeah.

• 40:16Speaker 00

  Well, you called your paper, your article, "Whimsy and Fragility in the China Shock Narrative", which is a whimsical title. But you're making a reference to Ed Lemer, who passed away recently. He was a guest on the program many times. He's one of my favorite economists. You mentioned in passing recently that it's a shame he wasn't winning a Nobel Prize, which was announced recently, because he's gone. But he wrote a very provocative, non-influential article called "Taking the Con Out of Econometrics". And what he was arguing for there was a more honest look at what empirical

• 41:00Speaker 00

  work actually was doing, along the lines I was just mentioning. And why did you, in a way, this is an homage to him?

• 41:11Speaker 01

  Yeah, I really love Lemer's work and his attitude as well. Lots of people didn't like his attitude. They said he was too aggressive, too mean. I think people say the same thing about me sometimes. But I think honestly, I think you need to be a bit aggressive and mean in intellectual debates, otherwise you don't really get anywhere. And he said that economics, it was fragile because of this process that you said of where people run regressions, they try different things until they kind of get a result that they think can be published.

• 41:51Speaker 01

  Whereas if you change a few things, then that result disappears or changes, as in this case. And then he said it was also very whimsical, because there's a lot of these kind of assumptions baked into a lot of economics research. And actually, if you pick at those assumptions, it doesn't really hold up very well. I live in Wales, and there's a lot of old farmhouses in Wales. And if you start picking away a bit of plaster on the wall, you soon discover that there's a massive great big hole, because they're all just completely damp and falling apart. And I've found a lot of economics papers are like that as well. So, yeah, Ed Lemer's

• 42:34Speaker 01

  kind of a hero to me. And I re-listened to your episodes that you did with him to prepare for this. And what he said was like, oh, yeah, that completely chimes with what I say, that economics should actually be more about telling stories and about finding patterns in data. And I think a lot of the problems with economics now is that they want to use these econometric techniques, which are very good at precisely saying that X caused Y. But in the real world, we have multiple causality. There's always multiple factors causing things. So if you look at the China shock, or, sorry, if you look at deindustrialization, you know, China might have had a role, but it's probably quite a small role. And there's all these other things going on that if you want

• 43:19Speaker 01

  to have a proper analysis of deindustrialization, what caused it in the United States, you really need a much, much broader analysis, which you're not probably going to be able to do in a regression, which is fundamentally based on X causes Y. There's all these other factors that you need to take into account. And econometrics might not be able to do that very well.

• 43:44Speaker 00

  And of course, the data you have don't exactly match what you're trying to measure off. And but you quote, "Do the best you can" because it's better than doing nothing. And the answer is no, it's not. If it misleads or runs you, turns you in the wrong direction. You mentioned the credibility revolution. It's been followed, of course, by the replication crisis, which is that a lot of the papers, certainly in psychology, and I think increasingly in economics, have not held up when people take an independent look. And that's, I want to salute you for working on that. It's generally not a rewarding strategy to try to show that something that we thought was true actually isn't true. Just like if you find no effect, instead of announcing it and

• 44:29Speaker 00

  publishing it, you just throw it away. It's called the file drawer effect, or the garbage can effect. And so, we only see the results that look dramatic and large. And if you've only, if you ignored all the other ones, you're not going to get very far. Would you expand on what you said a minute ago about storytelling and Ed Lemer and economic history? Because economic historians, you know, over the last 50 years, have turned to statistics with great excitement because it's prestigious and it looks scientific, whether it is or not, doesn't matter. What do you think economic historians ought to be doing to understand something like the deindustrialization, or the reduction in manufacturing in the United States as a source of employment?

• 45:18Speaker 01

  So I'm a very old fashioned economic historian. Like, my style of economic history goes back to, say, the 1950s, 1960s. And I see economic history as having been in a very long term decline. And a lot of what gets kind of described as economic history now, I refuse to recognize it as economic history. What I would do, if I was going to do this properly, I would do a lot of time series analysis. I would try, maybe do kind of econometric things on small scale,

• 46:00Speaker 01

  kind of sub-questions. I think econometric techniques can be very good for small questions, which then if you build them all together, you could answer the bigger questions. Like, so you could try and measure the effect of Chinese imports, other countries' imports, you could try and measure the effect of automation. But then ultimately, you need to kind of put them together into a more holistic analysis that kind of makes sense. And is kind of sophisticated

• 46:39Speaker 01

  beyond this kind of X causes Y approach that tends to dominate. I confess that I haven't looked at deindustrialization in the United States beyond this paper. So I'd need to sit down and really think about, I think I'd also go back to a lot of the old debates. Look at how people were trying to understand it back in the 1980s, in the 1990s, when the process was still quite young. Because I find a lot of value in kind of people at the time, how

• 47:15Speaker 01

  they were trying to understand things as well. Yeah. And a lot more description, and just kind of describing what happened, I think. And because actually, it's through description that you can get a much better understanding of what happened. And that kind of leads you to why it happened as well.

• 47:40Speaker 00

  Yeah, sometimes I think that's the best you can do. And that's too depressing for most people. It doesn't get you on the cover of the New York Times, or even on the editorial page. You know, I think about the changes that have happened in the United States over my lifetime. I now live in Israel, but, you know, I lived in the United States from, until five years ago, and grew up in the 60s and 70s. And there's an enormous number of things that changed. If you're trying to answer even a small, simple question akin to what we talked about recently with Jesus Fernandez Villaverde, you know, why are Americans having fewer children? Oh,

• 48:25Speaker 00

  because fewer people are getting married. Why are a few people getting married? Well, that's because technology and the workforce, and there's 3,000 interesting possible, it's a smartphone, you know, fill in the blank. It's the decline of the US education system, especially in the urban areas. I mean, there's just so much change. And the challenge, of course, of being a good historian, is to tell a story that certainly has some complexity to it, rather than X causes Y. But all the rewards go in the other direction, right? All the incentives go the other direction. So it's the rare person who can honor the complexity of the world,

• 49:10Speaker 00

  even though it's not always going to be the most personally successful approach.

• 49:17Speaker 01

  Yeah. I mean, I hope by doing this kind of study, like what I've just published in Econ Journal Watch, and I've published other ones before, I also hope to do a little bit to kind of change the incentives. So that actually, my kind of economic history, which I should have also said should also be theory informed. I think that's very important. Like, in the case of deindustrialization, we really need to think about whether deindustrialization is kind of inevitable as a country gets richer. And I'm not sure whether that debate's really happening. So, but through my Econ Journal Watch articles and other work, I hope to be able to do a little

• 50:00Speaker 01

  bit to, and my blog as well, I hope to be able to do a little bit to change the incentives in economics by showing that a lot of what's getting published, and a lot of what young researchers are being pushed to do especially, isn't very robust. And maybe the emperor doesn't have any clothes. So that's kind of my end goal of this kind of paper.

• 50:29Speaker 00

  If you look at...

• 50:33Speaker 00

  If you look at your paper, of course, we'll link to it and we'll link to the Econ Journal Watch paper, we'll link to your blog. You know, there's a lot of data in the paper, there's a lot of tables, and you look at it, if you're a non-academic person, a non-trained economist or econometrician, and it looks, you know, it looks thorough, it looks like there's a lot of work there that you're citing of these, of Otter, Dorn and Hanson, and the people who revise their methodology to get a different number. But there's a natural impulse of the public to think, well, these people are at first-rate

• 51:17Speaker 00

  universities, and look how much technique they've applied. And even I, who am a skeptic about China, about the China shock, and I'll confess, part of it purely out of ideological reasons, or I don't know if that's the right word, it's not the right word, I don't know how you would describe it. I like trade, I think trade's good for the, in general, there are exceptions, of course, but in general, trade is a good policy, having more trade rather than less, and I feel like being involved with the world rather than less, and I think it's good to be outward-facing rather than just inward-facing for a nation. And it's a thousand things that I like about trade that have nothing to do with the narrow

• 51:58Speaker 00

  question of whether it makes the pie bigger, I think it's about much more than that. So, when a person reads, and this, forget me, so now you're just a smart person who doesn't have a PhD in economics, and you're trying to stay up with public policy and be informed, and you read on the front page of the New York Times or the Wall Street Journal, that a study has come out of people from world-class universities saying that China cost America millions of jobs, it's hard not to take it seriously. You know, there's a certain, you know, I've always wanted, you know, the incentives, we've talked about a little bit, but what if your life mattered?

• 52:42Speaker 00

  What if your future well-being mattered, your econometric results being, standing up to various robustness checks? Would you say what you say? And I don't, I'm not suggesting there's anything untoward about econometricians and their claims, but in general, they seem to be more confident than they, than their, than the work merits, it's a difficult field. But you get famous by, you know, finding something dramatic, and so, which is a good idea, it's the way the world works, it should work that way, you should try to find something dramatic, but it's sad how many results, not just in economics, goes in every field, or challenge when they're looked at more closely, it's really not, it's alarming.

• 53:27Speaker 01

  Yeah, yeah, I agree, and it's not just economics as well, I think that there's, I think there's big problems in the knowledge production system as a whole, I'm mainly looking at economics because I find it very interesting, and it's adjacent to my own speciality as an economic historian. And I think it is very disturbing, and it is, it would be nice to believe that academia was producing reliable knowledge that we could depend upon, and if it's not, then it's problematic,

• 54:08Speaker 01

  because it suggests that there's big institutional problems that really ought to be addressed. And some disciplines have gone through that, I think psychology's gone through that. I, I'm curious to see how much difference it's made to psychology, is the new stuff any better than the old stuff? I don't know, I haven't looked at it at all, I would hope so. Economics is a very curious one, because economists obviously pride themselves on being very robust, the economics journals, they make all these replication packages available, but my impression is that no one looks at them. And then, if you write to, if you write to an economics journal and say, look, I found

• 54:51Speaker 01

  this problem in this article, they're like, I don't really care, send it to Econ Journal Watch. I've literally had that response, well, it wasn't, it was a bit more sophisticated than that, they told me to send my complaints to Econ Journal Watch. On the other hand, other disciplines don't have an institution like Econ Journal Watch. So, something about economics has produced that, which is, which I think is also, says something good about the discipline.

• 55:18Speaker 00

  Yeah, Econ Journal Watch, I think was started by Dan Klein, my former colleague at George Mason and past Econ Talk guest, and it's a watchdog, it tries to give a forum for people like yourself who find things that are somewhat, perhaps alarming. I would just say that I remember having a conversation with an astronomer, who, you know, brilliant person, who was obsessed with the reality, I'm sure, or at least his perception of reality, that the empirical work that justified some claim in astronomy was shoddy.

• 55:59Speaker 00

  And we have this certain, you know, romance, you know, we're only in the social sciences, and of course, there's all kinds of problems we have with our data, and the fact that human beings have their own sense of agency, and life's much more difficult for us, but we assume that in the sciences, I mean, that's a whole different thing, it's open and shut, they just find truth, or they don't find, in which case, they put it down, but it was basically a claim that the same things were happening in astronomy that were happening in economics. The use of advanced techniques, the ability to tease out certain really nuanced, small things from data that were noisy, et cetera.

• 56:41Speaker 01

  I think, yeah, I think each discipline has a different variety of problems. I think what may be different in STEM, in science, in the sciences, is that they've got a real culture of post-publication peer review. I've seen scientists descend on a shoddy Nature article, like a pack of kind of rabid dogs, and tear it apart on social media, in just a way that, I think within about a week or two, Nature put out an expression of concern, and they were just savage. Whereas in economics, when something gets into a journal, it tends to be taken as kind of, as law.

• 57:26Speaker 01

  And that's, you know, each discipline has a different culture, and it has its different issues, and I'm not saying that economics has any more issues than any other one. A sociologist wrote to me, actually, and I asked him what sociology's like, and he said, well, we don't accept, like, these results so easily if something's passed these statistical tests, on the other hand, no one ever wants to make their data available because, or the qualitative people would just, like, see this as a very dangerous thing to do, because then people might start poking around too much, and, so, yeah, each discipline has its issues.

• 58:08Speaker 00

  Have you heard from either Otter, Dorn and Hanson, or the, and I apologize, I'm not, I can't pronounce the, I don't have it in front of me, or, give me the other trio who challenged them.

• 58:25Speaker 01

  Borosyak, Borosyak, Hull and Havel, I'm not sure if I'm pronouncing all those correct either.

• 58:31Speaker 00

  And have you heard from any of them after you published your paper?

• 58:34Speaker 01

  So, not since I published a paper, I sent a work in paper version to Otter, Dorn and Hanson, and David Otter responded to say thank you for sending it to me. I've had various interactions with Peter Hull, mainly on social media, and my impression is that Peter hasn't found anything wrong with what I've written, he's kind of, so, that's good for me. I've had some, I've had some interaction with Kilborosyak, another of the econometricians.

• 59:13Speaker 01

  I released a statistical package for our statistical software, based on some of the research I've done with him, and Kilborosyak wrote to me and said, this is rubbish. So, I retracted that, I retracted that rapidly, but that was a part, that was a part from the, a part from the article, that was a separate thing. So, as far as I'm aware, I haven't made any econometric errors in the article, yeah.

• 59:43Speaker 00

  You know, I interviewed Adam Astriani, the psychologist who wrote a shocking article, you know.

• 59:53Speaker 00

  You get disillusioned. It just takes a long time. You think everything that's peer reviewed is true, and then you realize it's not. But you assume if it's really good economists, it's true. And then you realize, maybe not. And what Adam wrote, partly in response to the replication crisis in his field of psychology, was that peer review is awful. It's not like, "Well, it's flawed. It's imperfect." No, it's a bad system. And I remember reading that and thinking, "Well, that's a ridiculous idea." But then I read his article, and I realized, "Oh, my gosh, I think he's right." And it doesn't mean that you can publish whatever

• 1:00:33Speaker 00

  you want. It means there's a vigorous, rigorous, ideally, post-publication process, the way science used to be before there were formal, an industry in both journals and in academic life. What do you think about that?

• 1:00:51Speaker 01

  I think peer review is completely necessary, but a lot of the actually existing versions of peer review are not very good. I'd say, in economics, it's a complete black box.

• 1:01:08Speaker 00

  Explain. What do you mean?

• 1:01:11Speaker 01

  Well, you see the work in paper released at some point, and then however many years later, you see the article released, and you don't really know what happens in between. I know people have written to me and said, "Yeah, you know, I reviewed that article, and I said it was rubbish, and it just got published anyway." Because an editor might decide that actually, they want to publish it anyway. The editor just overrides the reviewers. The editor might be friends with the author, et cetera, et cetera.

• 1:01:49Speaker 00

  And note to self, says the editor, "Don't say anything to that person again." So the whole process has a certain indigeneity to it that's not attractive.

• 1:01:59Speaker 01

  It's quite corrupt, because it's not open. So that corruption is quite easy to get away with. If you look at the sciences, so if you look at Nature, for instance, they often publish their peer review reports. Sometimes the peer reviewers are named, sometimes not. And you can then really kind of track what happens with the paper. And of course, there is this culture of post-publication peer review where an article... The one I'm really thinking of, it was an article where it was written by quite famous scientists, and they said

• 1:02:36Speaker 01

  that the time of day when you took a particular treatment for cancer had a big effect on how effective the treatment was. And the research had all been done in China, and the results, it was all a bit kind of shady. And these other, a whole load of scientists just kind of jumped in there and said, "No, they found all these irregularities in the data," and all sorts of stuff. And you just don't see that culture at all in economics. And for all the economists claim to be very robust and have this very robust seminar culture, I like to think that if my articles contribute, I think it's to show that even though they're

• 1:03:21Speaker 01

  very proud of that robustness, it's not really doing its job very well.

• 1:03:24Speaker 00

  My guest today has been Joseph Francis. Joe, thanks for being part of Econ Talk.

• 1:03:30Speaker 01

  You're very welcome. Thank you very much for having me.

• 1:03:39Speaker 00

  This is Econ Talk, part of the Library of Economics and Liberty. For more Econ Talk, go to EconTalk.org, where you can also comment on today's podcast and find links and readings related to today's conversation. The sound engineer for Econ Talk is Rich Goyette. I'm your host, Russ Roberts. Thanks for listening. Talk to you on Monday.