How does scientific peer review work?

Daniel and Kelly shine a light on the inner workings of one of the crucial parts of the scientific process: peer review. How does it work? Who are these peers? Can peer-reviewed papers be wrong, and does that mean that all of science is questionable? See omnystudio.com/listener https://omnystudio.com/listener for privacy information.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Daniel Whiteson?

  This is an iHeart Podcast. Guaranteed Human.

• 0:04Speaker 01

  Small businesses are the pulse of every community. They bring people together, create opportunities, and drive growth. Chase for Business helps business owners like you with personalized guidance and convenient digital tools all in one place. With that guidance and your determination, you can take your business further and help build a brighter future for your community. Learn more at chase.com/business. Chase for Business. Make more of what's yours. The Chase mobile app is available for select mobile devices. Message and data rates may apply. JPMorgan Chase Bank, N.A., member FDIC. Copyright 2026 JPMorgan Chase & Company.

• 0:54Kelly Weinersmith?

  Science is supposed to be our best way of revealing the truth about the universe. But science is constantly being updated and corrected. And sometimes we learn after the fact that a study was flawed, or even was shoddy to begin with. So how do scientists decide whether a new result is robust or not? And how does the general public know when the science is settled or about to be upended? No system is perfect, but it's important that the process is transparent. So today, we're going to shine a light on the inner workings of one of the crucial parts of the scientific process: peer review. How does it work? Who are these peers? Can peer-reviewed papers be wrong? And does that mean that all of science is questionable? Welcome to Daniel and Kelly's Peer- reviewed Universe.

• 1:52Daniel Whiteson?

  Hi, I'm Daniel. I'm a particle physicist, and I've published over a thousand papers, but haven't read most of them.

• 2:00Kelly Weinersmith?

  Hi, I'm Kelly Wiener-Smith, and I'm confused. What does that mean, Daniel? How have you published over a thousand papers? At the LHC, does everybody...

• 2:10Kelly Weinersmith?

  Who is re- oh, that's cheating, man. That's cheating.

• 2:13Daniel Whiteson?

  Well, the way the collaboration works is everybody who's contributed to the project is an author. And when you're an author, you get included on every paper that comes out during your authorship period. And Atlas is a huge collaboration of lots of clever people. We put out about 100 papers a year. Every single one has my name on it. And I haven't read most of them. I couldn't even explain the titles of some of them to you.

• 2:37Kelly Weinersmith?

  So in my, so fields have their own cultures. In my field, in my culture, that would ethically not be okay. You should be responsible for what is in every paper that your name is on, in my opinion.

• 2:52Daniel Whiteson?

  And I was in another collaboration previously where things worked differently, where you weren't an author on a paper unless you asked to be. And nobody questioned it. If you had the right to be an author, if you were on the list, you could be, but it was an opt in. And I thought that was much better. The lengths of all the lists were shorter and authorship meant something. But here's, you can ask to be removed, but you have to go through this process. And it would be like, I'd be doing it twice a week, every week.

• 3:18Kelly Weinersmith?

  Wow. So what, does your CV include all of those papers? Or just the ones that you feel you've actually contributed to?

• 3:25Daniel Whiteson?

  So that depends. The CV I make and share with people only includes papers that I actually did anything on or had ideas for or really contributed to. But sometimes an institution wants a list of your official papers. And then I have to like include all, I don't even know the number, 1,000 plus papers at this point.

• 3:43Kelly Weinersmith?

  Oh, poor Daniel.

• 3:44Daniel Whiteson?

  No, I know.

• 3:49Kelly Weinersmith?

  How do you get them all? Do you just copy them from Google Scholar?

• 3:52Daniel Whiteson?

  Yeah, Particle Physics is pretty good at automation and stuff like this. So we have our own databases of papers and it's pretty easy to download that kind of stuff. But yeah, it's pretty tough to have so many papers.

• 4:04Kelly Weinersmith?

  Oh, I know. So many pages you have to print out. The file for the PDF for your resume is so big.

• 4:10Daniel Whiteson?

  How about you, Kelly? How many papers do you have? And have you finished that paper from your thesis yet?

• 4:14Kelly Weinersmith?

  Oh, I'm feeling angry now, Daniel. I have not. I've been busy with other things, including my new ducks and geese, but

• 4:24Daniel Whiteson?

  And goats.

• 4:25Kelly Weinersmith?

  And goats, which I won't mention of my co-authors.

• 4:27Daniel Whiteson?

  Do your ducks and geese and goats go on your CV? That's my question.

• 4:30Kelly Weinersmith?

  No.

• 4:31Daniel Whiteson?

  Students supervised. Come on.

• 4:33Kelly Weinersmith?

  Oh, yeah, no, I've got that list, but it doesn't-

• 4:35Daniel Whiteson?

  Kevin the Great.

• 4:38Kelly Weinersmith?

  The Great Goat. I think I've got, I haven't, I haven't counted. I think I've got something like 30 that my pace has slowed since I started writing pop sci. But, you know, I still get a couple in there every year.

• 4:50Daniel Whiteson?

  My CV has something whimsical and unserious on it. Yours is totally professional?

• 4:58Kelly Weinersmith?

  Yes.

• 5:01Kelly Weinersmith?

  What whimsical thing do you have on yours?

• 5:03Daniel Whiteson?

  Oh, I'm just gonna leave that as an Easter eggs. I put it in there to see if anybody actually reads my CV, because if they don't respond to that, I'm like, hmm, you didn't really read it.

• 5:11Kelly Weinersmith?

  Okay.

• 5:12Daniel Whiteson?

  Yeah, check it out.

• 5:13Kelly Weinersmith?

  All right. Well, I'll be looking up your CV. Do you have it updated? Here's the question. Most scholars don't update their CV, but like, once every half decade. Is your online CV updated?

• 5:25Daniel Whiteson?

  Yeah, I think I updated it a couple of months ago.

• 5:27Kelly Weinersmith?

  Oh, all right. Bravo. Bravo. All right. Well, today, we're not talking about the thousands of publications that Daniel has, but we're all very proud of Daniel for all of his hard work.

• 5:37Daniel Whiteson?

  Or the tens of geese and ducks that Kelly is currently raising, though we're very proud of her and her literally extended family.

• 5:45Kelly Weinersmith?

  Well, I, I don't have that many, and I wouldn't claim other people's ducks that I didn't help out with as my own ducks.

• 5:51Daniel Whiteson?

  Oh, snap. Snap.

• 5:55Kelly Weinersmith?

  But anyway, today, we are answering a question from a super fan, and we're gonna go into a bit more detail. This question inspired us to make a whole episode about the process of peer review. So let's go ahead and hear what Stephen from British Columbia has to say.

• 6:12Speaker 03

  Hi, Daniel and Kelly. This is show super fan Stephen from British Columbia, Canada. My question today has to do with how scientific studies are peer reviewed. Is it me? Or is science denial spreading these days? And it came to my attention recently that not all peer reviewed studies are replicated.

• 6:37Speaker 03

  And then it occurred to me, doesn't this automatically cast doubt on a discovery? If a study is not replicated, but is still considered peer reviewed? I would love to hear an explanation about how the peer reviewing process works? And how scientists determine when a fact is a fact, a finding is a finding? And how the community comes to a consensus on new discoveries? I also think your audience could use some tools for how to determine what studies are legitimate? And how to spot a bad study?

• 7:18Speaker 03

  How should one respond to someone who claims a scientific study is bogus? Thank you, Daniel and Kelly for this great podcast and for explaining science topics to the masses.

• 7:33Daniel Whiteson?

  Thank you, Stephen, for asking this question. I think it's really the right moment to dig into this. You know, when we see institutions being attacked, when we see science being denied? When we see the whole process of science being questioned? I think it's important to shine some light on how does this all work? What does it mean? Especially for those folks who are not scientists, who don't know? What does it mean for a paper to be peer reviewed? What is bad science? Thank you, Stephen, for asking this question and giving us the chance to dig into all of this.

• 8:02Kelly Weinersmith?

  That's right. And so the question in particular is about peer review for scientific manuscripts. But peer review actually happens at multiple stages in the process of doing science. And so we thought we would take a step back and talk about peer review from the very beginning, which is when your idea becomes a grant. And this is one of the less fun parts about science is writing this grant, because peer review for grants is harsh.

• 8:27Daniel Whiteson?

  This is like the science version of when a bill becomes a law. We should have written a song for this.

• 8:31Kelly Weinersmith?

  No, well, we still can, you know? We could write it and then we could add it to the top.

• 8:37Kelly Weinersmith?

  No, we won't do that to everyone.

• 8:39Daniel Whiteson?

  So Kelly, tell me, what is your process of grant writing? First of all, who do you submit your grants to? And what's it like for you to prepare that grant?

• 8:46Kelly Weinersmith?

  So I submit my grants to the National Science Foundation. And actually, it's possible that in the last six months, the process has changed a bit because I know the new administration is shaking things up.

• 8:58Daniel Whiteson?

  The agency formerly known as the National Science Foundation?

• 9:01Kelly Weinersmith?

  No, what is it called now?

• 9:02Daniel Whiteson?

  I'm joking, I'm joking.

• 9:03Kelly Weinersmith?

  Oh, God. Anything is believable, Daniel, at this point. All right. The institution's still known as the NSF or the National Science Foundation. So for this, you write a grant. And there's different programs that accept different kinds of grants, but often the grant is like 15 pages where you go into extreme detail in what we know about the question you want to ask, based on literature that's already out there, extreme detail in the experimental design, and then a lot of detail about what you think the implications will be of your results.

• 9:36Daniel Whiteson?

  So you talk about why this question is interesting. You argue that your work will help answer it. And then you lay out in detail how you're going to spend every dollar, who's going to do what and when. You've got like a whole grant chart for like when everything is going to happen.

• 9:50Kelly Weinersmith?

  Yep.

• 9:50Daniel Whiteson?

  And then you have to lay out also like the products for the grant, right?

• 9:53Kelly Weinersmith?

  Yeah, yeah, the budgets. And so you're supposed to, there's a section called Intellectual Merit, where you talk about how many papers you think you're going to end up producing, what big questions you're going to answer. And then there's also a section called Broader Impacts, where you talk about how your work is going to benefit society writ large. And, you know, for my grants, in the past, it's been things like, you know, my information is going to be helpful to fisheries managers who are managing fish populations, which is like a literally a multi-billion dollar industry for like people who want to go out fishing and stuff. So most of my research has been on fish and their parasites. Also, Intellectual Merit will include things like how many students you're going to train and what skills you're going to give them that they can then use when they go out in the workforce. And depending on the field you're in, sometimes your broader impacts will be things like,

• 10:38Kelly Weinersmith?

  hey, this could be what we learn about the brains of this fish could one day help us produce a new medication for anxiety or something like that. So you talk about how your stuff would impact the broader society.

• 10:50Daniel Whiteson?

  And so for those folks who might think, hey, scientists are just cashing in on the government gravy train, tell us, like, is this the kind of thing you throw together in an afternoon and then can confidently think it will be funded?

• 11:02Kelly Weinersmith?

  Oh, my God. Okay, so one of the reasons, to be honest, one of the reasons I sort of transitioned away from academia in the most purist sense, so I'm like academic adjacent now, but was because I spent too many Christmas Eves working on my grants because they were due like sometime in January and and they just, they were like so many, I would say months of work in a lot of cases, if it's a new idea, if you're just cleaning up an old grant, then it can be months of work. And then so many great grants are submitted. And when they go to review, you have three peers of yours who are in closely related fields review it. And then there's a

• 11:43Kelly Weinersmith?

  panel discussion where they talk about their reviews of your grant in front of a bunch of other experts who can also weigh in. And then all those reviews go to program directors who pick out of like, the 20 best grants, you know, maybe they can only fund 15 or something. And then they pick some on the East Coast, some on the West Coast, some in the center of the country, some from major research institutions, some from institutions that focus a lot on undergraduate research, and they have research programs as well. And so so many really great grants don't make it into the pile that get funded.

• 12:19Daniel Whiteson?

  Meaning that people have read them have said, this is excellent. The science is good. It's interesting. It's important. We would benefit as a society if we did this. It's all well thought through, but no.

• 12:30Kelly Weinersmith?

  That is exactly right. Yes. And it is soul crushing. I know so many people who are like, I gave up Christmas Eve on this grant, and this is the third time I've submitted it. And it just never makes it quite high enough, even though everybody thinks it's amazing. And then quickly, just to mention for follow up, if you do get a grant, every year, you have to write a progress report saying what you've done, have you stuck to your timeline? How have you spent the money that you've been given? Are you gonna hit your timelines? And what have you done that impacts society? And so every year, you have to report on that. And then at the end, you have to write a like, bigger report. So you need to be justifying what you're doing with the money every step of the way.

• 13:07Daniel Whiteson?

  And before we turn it around and ask me questions, who are the peers here? Who are folks who are reading your grants and commenting on it?

• 13:15Kelly Weinersmith?

  There are folks in the same discipline at other universities. So you also have to fill out a detailed Excel sheet where you talk about who in the field who does similar work has been your student, or your postdoc, or your mentor, or a co-author.

• 13:29Daniel Whiteson?

  You might have a conflict of interest.

• 13:30Kelly Weinersmith?

  And anyone who might like you for you, and who might be too nice to you when they review the grant. So it has to be only people that you haven't worked with, or haven't co-authored a paper with for something like five to 10 years. I guess they assume that the liking of someone wears off in about half a decade. But it can be tough, because these fields are kind of small. But usually, it's other people at research institutions who do similar work, are familiar with the literature, and can tell if what you're proposing makes sense or not, and is good or not.

• 14:01Daniel Whiteson?

  And so the thing to take away from this is getting a grant funded is hard. Not only is it a huge amount of work, but you have to survive an excruciating process where really only the best are selected. It's sort of like watching the Olympics, and you're wondering like, oh, here's somebody from this country, and somebody from that country. And you know that each person has already won some sort of like, really competitive national competition, just to even be there at the Olympics representing their country. And so like, every grant that's funded, even if it seems silly, and it's about like, the sex lives of ducks, you know that it's been like, excruciatingly written, reviewed in detail, questioned by experts without conflicts of interest, and found to be excellent, beaten out lots of other grants.

• 14:46Daniel Whiteson?

  These are not slush funds shoveled to scientists to just like, do whatever they want with. These are hard-won funds to do science that the agency thinks are a good idea.

• 14:58Kelly Weinersmith?

  Yeah, it is absolutely excruciating to do these things. It feels great when you get it funded. And I said I hate writing grants. Sometimes I enjoy the process of like, figuring out just the right experimental design for a question. That's like, fun for me. But in general, when the grant doesn't get funded, it sucks. But anyway, all right. So what's your experience?

• 15:17Daniel Whiteson?

  Yeah, my experience is very similar. You know, we come up with an idea, we spend a lot of time polishing it, often doing a lot of... A lot of the work in advance that we need to demonstrate that the work described is reasonable, right? If it's too far forward thinking, then you won't get the money because they're like, well, that might work, but can you really prove it? It's too much of a risk. If you've already done too much of the work, then they're like, oh, this is already done. Why would we fund it? So it's really sort of on the edge there of like, you've done the initial work for free, right? Or on some other grant or whatever. So you prove that the idea is valid and can fly, but not so much that they're like, why would we fund this? You've already done it, which is

• 15:58Daniel Whiteson?

  often a delicate balance. And my feeling with grants is submit it and forget it because such a tiny fraction ever come back with money that you just gotta like, let it go. Like, hey, I submitted it and I never expect to hear back, you know, anything positive. And so I just sort of like, give up emotionally on everyone because otherwise it's too hard, you know? It's too hard to deal with. So my process is very similar to yours with a couple of exceptions. Sometimes I submit to private foundations, like I recently found a private foundation that likes to fund projects that are too blue sky that were rejected by the NSF for being like way too out there.

• 16:39Kelly Weinersmith?

  Oh, wow.

• 16:40Daniel Whiteson?

  And that submission process was like, write a paragraph, send us your rejected NSF grant with the reviews and that's it.

• 16:47Kelly Weinersmith?

  Whoa.

• 16:48Daniel Whiteson?

  And that one actually just came back and they just gave me some money to build my Cosmic Ray smartphone telescope down here in Irvine. So yeah, that was actually a really positive experience.

• 16:58Kelly Weinersmith?

  Congratulations. Yay. That's awesome.

• 17:00Daniel Whiteson?

  Thank you. Yeah, exactly. Because, you know, private foundations can do whatever they like with your money. What we talked about previously was mostly the process of applying to government institutions, the NIH, the NSF. Most of my funding comes from Department of Energy Office of Science, which funds particle physics and lots of other stuff in the United States. But private foundations can do anything. Like the Bill Gates Foundation, you can't even apply to. They have to like, reach out to you.

• 17:25Kelly Weinersmith?

  Wow.

• 17:26Daniel Whiteson?

  And, you know, Mackenzie Scott just like, gives money to people that don't even apply. They just get a phone call saying like, by the way, here comes a million dollars or more.

• 17:34Kelly Weinersmith?

  Whoa.

• 17:34Daniel Whiteson?

  So yeah, private foundations are weird.

• 17:37Kelly Weinersmith?

  Yeah, so I'm sure that they make solid choices about who they're going to give their money to, but I guess one way to sort of check how much you can trust an article is you can look in the acknowledgement section at the end and say, oh, this was funded by the National Science Foundation. And then you know, this went through rigorous peer review. And just because something doesn't go through rigorous peer review at the grant stage, doesn't mean it's bad science. But at least, you know, if it went through a government agency, it has been sort of gone over with a fine tooth comb.

• 18:05Daniel Whiteson?

  Yeah. And the thing that's frustrating to me is this process is so inefficient. Scientists spend so much of their time preparing grant proposals and having them rejected. And, you know, each of these is a great idea. Each of them, if we did it, would benefit society in terms of sheer knowledge, or, you know, new technology, or something. We have all these smart people constantly pitching great ideas to the government, the government saying, yeah, that's awesome. We'd love to do it, but we can't. Like, why don't we just double, triple funding for science? It would be like, better for us. Every dollar we spend, we know comes back two, threefold in terms of like, economic output. It just seems crazy to me to reject all of these excellent ideas.

• 18:50Kelly Weinersmith?

  I mean, I gotta say, not every idea I've reviewed on a panel has been excellent.

• 18:55Daniel Whiteson?

  No, but there are plenty above threshold.

• 18:57Kelly Weinersmith?

  Yes, absolutely, absolutely. There's always a grant every round where my heart breaks a little that it didn't get funded. Because I thought, oh, that is so cool.

• 19:04Daniel Whiteson?

  Right.

• 19:05Kelly Weinersmith?

  But there wasn't enough money to cover it. And so, yes, there's a lot of good work that's not getting done.

• 19:09Daniel Whiteson?

  And I agree. I'm a grant reviewer often. And I see grants, I'm like, no, this is not well thought out, or there's a flaw here, or this isn't cutting edge. Somebody did this last year, and it's important that this stuff gets reviewed and gets reviewed in a fair way. But there's so many that are above threshold, and only a fraction of those get funded. And it just seems to me to be a waste. But whatever. I'm not a political person. I don't understand these things. But I want folks out there to realize that every grant you've seen that's been funded, has been reviewed in excruciating detail before dollar one was even sent to the institution.

• 19:41Kelly Weinersmith?

  That's right. I don't think we need to get into lots of detail about this, but every once in a while, my husband and I will have chats about What would be a better system for funding grants? Like, maybe every four years, you get a chance to submit. And that way, you don't have to write grants the other three years. And there's fewer people in the pool and you're more likely to get it. I think that's not the answer. But I wonder if there's some way we could save scientists from spending all of this time on grants that don't get funded. Or maybe we should just throw 20 times as much money at the scientific community. There's our solution.

• 20:12Daniel Whiteson?

  Well, you know, it used to be 50 years ago that the philosophy is a little bit different, that the government funded people rather than projects. And they were like, oh, scientist X at this university, you're a smart lady. You've done good work. We'll just keep giving you money and it'll just go for a while. As long as you keep doing something, we'll keep giving you money. And I think that we used to be the major model. And it's just not that way anymore. Now it's projects. And so if you have a lab at a university, it's sort of like running a small business. You know, you have to constantly be pitching grants to get funds. You know, you're like running a store at the mall, Katrina likes to say. And you're constantly looking for new ways to contribute. The one holdover that I'm aware of is actually experimental particle physics is still a little bit of the older model.

• 20:57Daniel Whiteson?

  They run competitions for junior faculty. And if you win one of these awards, like I won a outstanding junior investigator award when I was a very young professor, then you sort of get in the club and then they mostly fund you. And your funding can go up if you do great and go down if you're less productive. But it's much more stable than typically. I think because these particle physics projects last like 20 years. You know, we build a collider, we expect to use it for 25 years. That's my thinking for why they still do it in the sort of older model. But it means I don't have to write as many grants because I do have one more stable source of funding.

• 21:32Kelly Weinersmith?

  Wow. So even to this day, you still get a chunk of money every year?

• 21:35Daniel Whiteson?

  I have to submit a grant every three years to propose new work and to tell them what I did in the last three years. And so far, every time, my grant has been renewed and continued. So yeah, I've been funded from the Department of Energy for a couple of decades now, which is very nice. And I'm very grateful to the Department of Energy. Thank you very much. And to all the taxpayers who support them.

• 21:54Kelly Weinersmith?

  Oh, you government shill.

• 21:58Kelly Weinersmith?

  I'm just kidding. I am so happy for you. That's awesome. I know there's a couple of grants for people in the medical fields that work that way also, where they fund your lab and just trust that you're doing awesome stuff and you will continue to do awesome stuff with that money. And that sounds pretty sweet.

• 22:12Daniel Whiteson?

  Yeah, it's pretty nice. Yeah. I think you're talking about like the Howard Hughes Awards, for example.

• 22:16Kelly Weinersmith?

  That sounds right. Yeah, yeah. Yeah. All right. Well, my jealousy aside, let's all take a break. And when we get back, we're going to talk about the next phase when peer review is done. And that is when you're about to start your experiments.

• 22:37Speaker 04

  Cash flow crunch? OnDeck's small business line of credit gives your business immediate access to funds up to $200,000 right when you need it. Cover seasonal dips, manage payroll, restock inventory, or tackle unexpected expenses without missing a beat. With flexible draws, transparent pricing, and control over repayment, get funded quickly and confidently. Apply today at ondeck.com. Funds could be available as soon as tomorrow. Depending on certain loan attributes, your business loan may be issued by OnDeck or Celtic Bank. OnDeck does not lend in North Dakota. All loans and amounts subject to lender approval.

• 23:08Speaker 05

  Right now, you can save 70% on SimpliSafe home security. When an intruder approaches a home, they expect a camera. What they don't expect...

• 23:16Speaker 06

  Stop. This is SimpliSafe. You're being monitored and recorded. I'm requesting police dispatch.

• 23:21Speaker 05

  Unlike traditional alarm systems, SimpliSafe uses smart AI cameras that spot potential threats and alert live US-based agents to help stop a break-in before it starts. Our agents intervene in under 30 seconds to deter intruders while they're still outside, preventing them from getting close to your family. Plus, you control your privacy and recordings. And with a 60-day money-back guarantee, you can try us risk-free. Monitoring and deterrence plans start around $1 a day. Order today to get 70% off a new system with professional monitoring. Save now at simplisafe.com/iheart. SimpliSafe, ahead of crime in real time.

• 24:00Speaker 06

  30 seconds response time requires monitoring plan with outdoor protection starting at $49.99 a month. Visit simplisafe.com/licenses for alarm license information.

• 24:12Speaker 04

  She'd throw things, wander, and started hoarding. Mom's Alzheimer's was already so hard. But then we found out she had agitation that may happen with dementia due to Alzheimer's disease. And that was a different kind of difficult. So we asked her doctor for more help.

• 24:27Speaker 07

  Seeing symptoms like these in a loved one? It could be time to ask their doctor about Rexulti, Rexiprazole 2mg, the first FDA-approved treatment for this condition. Rexulti is proven to reduce agitation symptoms. Rexulti should not be used as an as-needed treatment. Elderly people with dementia-related psychosis have increased risk of death or stroke. Report fever, stiff muscles, and confusion, which can be life-threatening, or uncontrolled muscle movements, which may be permanent. High blood sugar can lead to coma or death. Weight gain, increased cholesterol, unusual urges, dizziness on standing, falls, seizures, trouble swallowing, or sleepiness may occur. Learn more about these and other side effects at rexulti.com. Tap ad for PI.

• 25:02Speaker 04

  I'm glad I asked for help.

• 25:04Speaker 07

  Ask your loved one's doctor about Rexulti. Your business operates under pressure, and you need control of your data. But IBM Research shows only 9% of companies have real oversight of their technology. IBM's Open Hybrid Cloud approach is designed to give you visibility with built-in governance tools, data sovereignty, and compliance reporting. It's another way IBM helps you stay in control of your data. Let's create smarter business. IBM. Visit ibm.com to learn more.

• 25:50Daniel Whiteson?

  All right, we are back, and we are talking about peer review, thanks to a question from a listener. And we dug into the process of peer review for grant proposals and grant writing. And now let's talk about what it's like to review an experiment while it's running, before the paper is even sent to the journal. Kelly, I mostly do research on particles that don't have rights and don't have institutional review boards protecting them. What's it like to do an experiment on living creatures with emotions that can feel pain and have people looking over your shoulder?

• 26:22Kelly Weinersmith?

  Oh, it can be pretty stressful. So, so say you get that National Science Foundation grant, they won't actually release the funds to you until you've shown that you've acquired certain permits and protocols. So your institution is giving you permission to do the research. So my PhD work required collecting some fish out of estuaries in California. So first I had to talk to the state government and fill out permits to get permission to go collect those fish. So I had to convince the California government that I wasn't going to take too many, that there were plenty of these fish out there, that what I was going to do to them was asking a worthwhile scientific question.

• 26:58Daniel Whiteson?

  Kelly, what are you gonna do to them? That's so aggressive.

• 27:02Kelly Weinersmith?

  Well, the questions I'm going to ask of them, that they will be contributing in a meaningful way to science. And so first you have to get that permission to take the animals out of the wild. And then you need to fill out a protocol through the Institutional Animal Care and Use Committee, which is IACUC. It includes five people and they include folks like veterinarians and outside members. So like somebody who's part of the community who is just sort of going to weigh in on what the general public feels about the work that's being done. Sometimes you'll also have like an ethicist or a lawyer in there. And often you'll have like a faculty member on there also.

• 27:38Daniel Whiteson?

  And what do these people want? Do they want to say yes? Do they want to say no? Are they just totally disinterested? Like, what are their motivations? Why is like a rando from the public doing this anyway?

• 27:48Kelly Weinersmith?

  So if I'm being completely honest, my field in the past did some uncomfortable things to animals. And, you know, they didn't use, for example, anesthetic when they were doing some surgeries and they should have, things like that. And so this committee is trying to make sure that you are treating the animals as nice as possible and using the fewest animals that you need to get answers to your questions. So you need to convince them that you have read up on the most effective anesthetics for whatever animal it is that you're using. You also often have to take a bunch of online training courses where you show I am well trained and the best way to treat these animals and I know how these anesthetics work. You often have to

• 28:32Kelly Weinersmith?

  Prove to them that you have teamed up with someone who can make sure you're doing the process correctly. And you need to convince them that you've thought really hard about the exact number of animals that you're going to use for these experiments. Their job isn't to stop science and they're not necessarily trying to figure out if the question that's being asked is good or not. They assume that if you have funding from the National Science Foundation, that has already happened. But their goal is to just make sure that nothing unethical happens and that the animals who are contributing their lives to science have the best life possible.

• 29:01Daniel Whiteson?

  Oh, that sounds nice.

• 29:02Kelly Weinersmith?

  It is.

• 29:03Daniel Whiteson?

  And it's nice to see the process sort of self correcting.

• 29:06Kelly Weinersmith?

  Like, yeah.

• 29:06Daniel Whiteson?

  Okay, we trust scientists. Actually, maybe they need some more eyeballs on them because they have conflicts of interest. And so it's good to have other people who don't have those conflicts looking over your shoulder and making sure you're doing things the right way. That's nice.

• 29:20Kelly Weinersmith?

  It is. And I also appreciate that there's a veterinarian on there. The veterinarian will like check in pretty regularly. I have training in parasitology, but I don't necessarily have training in care of fish or something. And I've done a bunch of reading, but it's nice to have a veterinarian check in every once in a while and offer their expertise to make sure that these animals really are being treated as well as you can in a lab setting. And I have never worked on humans, but if you are doing things with humans, including just like sending out a survey to humans that might ask questions that would make people feel uncomfortable, you have to pass those procedures through an institutional review board. So there's a similar procedure for working through humans. So I guess you don't have anything like that for particle physics, because we don't care what you do to the particles.

• 30:03Daniel Whiteson?

  No, we don't. But I did one time come home from work to find Katrina collecting samples for an experiment from our children.

• 30:10Kelly Weinersmith?

  Oh.

• 30:11Daniel Whiteson?

  And I was like, hmm, shouldn't you be asking the other parent for approval before experimenting on your children? Like, you have a conflict of interest here. And she was like, it's just a saliva sample. And I'm like, the children can't consent. So we have a lot of fun joking about that.

• 30:26Kelly Weinersmith?

  Did you give permission? That's good. That's good. So Daniel, there's this new thing that's getting sort of big in science that my friends have been talking about. I haven't had a chance to do this yet, but preregistering a scientific study. Have you done this yet?

• 30:40Daniel Whiteson?

  We don't actually do this in particle physics because this is another way that particle physics is weird, is that we publish our studies even when we get a negative result.

• 30:48Kelly Weinersmith?

  Nice.

• 30:49Daniel Whiteson?

  Like, in lots of fields of science, you might say, I have an idea for how we might learn something, and you do the study and then it didn't work. Or you learned that there's nothing new there. Like, okay, bears eat salmon, we already knew that, you know, like, or whatever. And often, those studies don't get published if the result isn't statistically significant, or you didn't learn something new. And there's a statistical issue there, which is that this can lead to a bias in our data and our understanding. The effect is called p-hacking, and it means that sometimes things that are not real can appear to be significant just due to random fluctuation. Say, for example, I'm testing a bunch of coins to see if they're fair. I flip each one 100 times. Then, like, most of the time,

• 31:32Daniel Whiteson?

  I'm going to get 50 heads. But occasionally, just due to random fluctuations, I'm going to get one that gets like 70 heads, right? And the more I do this experiment, the more likely I am to have one that's a fluctuation. And so if I only publish the ones that seem to be significant, that I cross some statistical threshold to be weird, it's going to look like these coins are not fair, when in reality, they are. That's what p-hacking is. P refers to the probability for the alternative hypothesis that the coin is fair to fluctuate randomly to look like it's not fair. And so, the way to counteract this is to publish negative results, to say, look, I did all these experiments, and the coin came out fair, we already knew that, yawn. But it's important that we include this context, so that if one in a thousand

• 32:17Daniel Whiteson?

  studies says, look, I saw a one in a thousand times effect, you know what it means. And often, we do this by preregistering studies, by saying, I'm going to go out and do this study, and I don't know the results yet, but I'm going to publish it either way, right? And so, that's a way to counteract p-hacking.

• 32:34Kelly Weinersmith?

  So, in my field, what you've just described, I think we would call the file drawer effect. And so, the- The idea here is that, yeah, when you get a null result or a result that's less interesting, you might try to publish it somewhere, but you're less likely to put in the effort to try to publish it in a lower tier journal that you're going to get less credit from your institution for having published in.

• 32:53Daniel Whiteson?

  Yeah.

• 32:54Kelly Weinersmith?

  And that could make it look like an effect is there because when you randomly get a positive effect, it gets published and otherwise it doesn't.

• 32:59Daniel Whiteson?

  Yeah.

• 32:59Kelly Weinersmith?

  For us, P-hacking is when you didn't get the result that you wanted, but you're looking at your data and you're like, oh, you know, I wasn't actually asking a question about this other thing, but if I look at my data, there's actually a statistically significant result there. And so I'm going to publish on that.

• 33:16Daniel Whiteson?

  Hmm.

• 33:16Kelly Weinersmith?

  And, you know, maybe I'll mention that this wasn't the initial thing, but I found this, you know, significant relationship that's positive. So I'll publish on that. And the, the deal there is, you know, like you said, randomly, you would expect to get results that look like something is really going on. And if you just are searching through your data set, you're more likely to find something that's significant. And that wasn't the question you were originally asking. And so the reason we do preregistering a scientific study is you'll say ahead of time, I am specifically looking at the relationship between, I don't know, parasites and how often a fish darts. And so if I go on and I publish a paper about parasites and how active a fish is, you could say, hey, did you just find that result and that you changed your paper to be about that because it looked

• 34:02Kelly Weinersmith?

  interesting? And so anyway, this is how we make sure that you're not looking for significant results and just publishing whatever you find.

• 34:08Daniel Whiteson?

  And in particle physics, we're lucky enough that we always publish because a negative result is still interesting. Like if you look for a particle because you think it's there and the universe would make more sense for it to exist, like the Higgs boson, then if you see it, great, you publish it. If you don't, that's still interesting. You still want to know, oh, there isn't a Higgs boson. Because if we smash particles together often enough and we don't see it, we can say something about the likelihood of it not existing. Because if it existed, we would have seen it. Now, we still are susceptible to P-hacking because we will sometimes see fluctuations, like data will just like look like a new particle sometimes. And to counteract that, we have a very stringent criteria. It's called this five.

• 34:50Daniel Whiteson?

  Sigma threshold, which means that we only claim a particle is discovered if it's very, very, very unlikely to come from a fluctuation. But we still, in principle, could get fooled. And that's why we always have duplicate experiments. Like at the Large Hadron Collider, we have two of these big detectors, and they're totally independent. And that's why you expect to see the same physics in both. And if you don't, you know, something is squirrely.

• 35:12Kelly Weinersmith?

  Yeah. And in my field, we're getting better, where if you get no result or negative results, as long as you can show that you did a sound experiment, and the answer is no, then you can still publish it somewhere. And so, Public Library of Science, or PLOS, is a journal that encourages folks to submit their null results, as long as the science was good, and you can defend what you did, to try to make sure that you're not getting this file drawer effect stuff. So you get the no answers just as often as the yes answers.

• 35:39Daniel Whiteson?

  And that's helpful, right? It saves people time. Because if Kelly had a great idea, and it turned out the answer is no, and then she just throws it in the file drawer, then Sam, across the country, is going to try the same thing someday and waste their time. So it's, it's good to get this stuff out there.

• 35:53Kelly Weinersmith?

  Right. And you know, in my field, wasting time also often means that some animals may have died in the process. And so, to me, I feel like there's this additional ethical requirement, that you get anything that you learned from these animals out, so that nobody has to go and repeat it. And it, you know, more animals might have to pass away for science. So, anyway, good to get it out there.

• 36:13Daniel Whiteson?

  All right. So then, let's talk more about what happens when you think you have an interesting result, and you've written it up, and you want to publish it.

• 36:20Kelly Weinersmith?

  And we're gonna leave everybody on a ledge for just a second. And when we get back from the break, we'll talk more about the fascinating world of peer review.

• 36:36Speaker 07

  OnDeck is built to back small businesses like yours. Whether you're buying equipment, expanding your team, or bridging cash flow gaps, OnDeck's loans up to $400,000 make it happen fast. Rated A+ by the Better Business Bureau, and earning thousands of five-star TrustPilot reviews, OnDeck delivers funding you can count on. Apply in minutes at ondeck.com. Depending on certain loan attributes, your business loan may be issued by OnDeck or Celtic Bank.

• 37:01Speaker 05

  Does not lend in North Dakota. All loans in amounts subject to lender approval. Right now, you can save 70% on Simply Safe home security. Because what a break in really costs isn't your home or your valuables. It's the feeling that your home stopped being safe. That's why Simply Safe doesn't wait for danger to reach your family. We help stop it before it gets close with not just one, but three layers of protection. One, preventing threats outside with smart AI cameras and live US-based agents who intervene to deter intruders before they break in. Two, sounding the alarm at points of entry. And three, agents requesting emergency response when needed. Three layers, all working together to keep your loved ones safe. Plus, you control your privacy and recordings. And with the 60-day money-back guarantee, you can try us risk-free.

• 37:46Speaker 05

  Monitoring a deterrence plan start around $1 a day. Order today to get 70% off a new system with professional monitoring. Save now at simplysafe.com/iHeart. Simply Safe, ahead of crime, in real time.

• 37:59Speaker 06

  Outdoor deterrence plans start at $49.99 a month. Visit simplysafe.com/licenses for alarm license information.

• 38:08Speaker 04

  She'd throw things, wander, and started hoarding. Mom's Alzheimer's was already so hard. But then we found out she had agitation that may happen with dementia due to Alzheimer's disease. And that was a different kind of difficult. So we asked her doctor for more help.

• 38:23Speaker 07

  Seeing symptoms like these in a loved one? It could be time to ask their doctor about Rexulti, Rexiprazole 2mg. The first FDA-approved treatment for this condition. Rexulti is proven to reduce agitation symptoms. Rexulti should not be used as an as-needed treatment. Elderly people with dementia-related psychosis have increased risk of death or stroke. Report fever, stiff muscles, and confusion, which can be life-threatening, or uncontrolled muscle movements, which may be permanent. High blood sugar can lead to coma or death. Weight gain, increased cholesterol, unusual urges, dizziness on standing, falls, seizures, trouble swallowing, or sleepiness may occur. Learn more about these and other side effects at rexulti.com. Tap ad for PI.

• 38:58Speaker 04

  I'm glad I asked for help.

• 39:00Speaker 07

  Ask your loved one's doctor about...

• 39:06Speaker 07

  Your team is on a deadline, but moving all your enterprise data will take time your business simply doesn't have. IBM Technology lets AI work with your data where it already lives, with an open hybrid cloud approach designed to build on your existing investments. It's another way IBM helps you put your data to work. Let's create smarter business. IBM. Visit ibm.com to learn more.

• 39:45Kelly Weinersmith?

  All right, Daniel, you have a significant result. You've got a result that looks good, or at least even if it's no, you're sure it was no because the experiment was done well. What do you do next?

• 39:54Daniel Whiteson?

  So we write it up. And when all of our co-authors are happy with it, the first thing we do is we post it online. So particle physics is a very, very online science. And we invented this thing called the archive, where scientists post their papers. And it's called a pre-print because we post it before we submit it to the journal. So for example, I finished a paper this week and yesterday it appeared on the archive. And nobody's reviewed it. Nobody's confirmed it. It's just out there for other scientists to look at. And in a week or so, if we don't get like screaming fits from somebody saying, "You stole my idea", or "This is deeply wrong", then we'll submit it to a journal. So the process for particle physics is first post it online, then submit it to a journal

• 40:39Daniel Whiteson?

  where it's going to get reviewed. And that's going to take months and months. But it's so slow that particle physicists just read the pre-print server and read those papers. And nobody really cares when it comes out in the journal later because we already have seen the paper months earlier.

• 40:53Kelly Weinersmith?

  But what if the peer-reviewed version catches an error? Do you go back and update the archive version?

• 40:58Daniel Whiteson?

  Absolutely. You can update it. And usually papers in the archive are like version 3, version 4. And there's comments in between. You can compare them. You can see the mistakes getting fixed. Absolutely. And some papers appear in the archive and then are never published. And that's fine. I actually had one paper like 10 years ago put on the Archive, it took like more than a year to review. And then they rejected it because the paper already had like 30 citations. And so they were like, why publish this? It's already out there. People are reading it and using it.

• 41:27Kelly Weinersmith?

  That's such a stupid reason. Yeah.

• 41:30Daniel Whiteson?

  It was really dumb, but then we just gave up. And now it's like one of my more cited papers and it was never published. The system is silly.

• 41:37Kelly Weinersmith?

  It is silly.

• 41:38Daniel Whiteson?

  But I know that in other fields it's different. Like my friends in astronomy, they never put papers on the archive until after they've been reviewed. And they think that if they send it to a journal, it's already out in the archive. Those journal referees are going to find that disrespectful, like that they didn't wait for them to chime in. So it's definitely a field by field culture kind of thing. How does it work for you guys?

• 41:59Kelly Weinersmith?

  Yeah. So y'all were definitely the trailblazers. We subsequently came up with Bio Archive. And I remember when I was initially wanting to submit to Bio Archive, I was hesitant to do it because some of our journals did say, uh, your paper should not be available anywhere else because we want to be the only place where you can find it. And that's because journals make money off of people who download their articles. And so they didn't want someone else to be able to get another version for free somewhere else. And on Bio Archive or Archive, all of the articles are free. But it seems like it's now become acceptable that you can put papers in these pre-print servers like Bio Archive. And so now a lot of people will do that and they do it for a variety of reasons. Some of it is they want to get feedback early.

• 42:44Kelly Weinersmith?

  They want to get the results out there early. But sometimes it's also like if you've got a PhD student who wants to go looking for a job. On their resume, if they finished a study, they'll write in preparation next to the manuscript. But in preparation could mean I've got the data and I still need to do all the stats and I haven't even started writing. Or it could mean I'm about to submit it to a journal. And if you've put it on Bio Archive, you're saying, look, it's done. I just need to start the peer review process. Or even it's in peer review because that process can take a long time. So anyway, it's becoming a lot more common to put your papers on Bio Archive just to make them easier to access for other people and just to show that you really have actually finished that study finally after a decade. Or however long it took.

• 43:28Daniel Whiteson?

  The clock is still ticking on your thesis there, Kelly. We'll see.

• 43:31Kelly Weinersmith?

  I'll get it.

• 43:31Daniel Whiteson?

  In preparation.

• 43:33Kelly Weinersmith?

  In preparation.

• 43:34Daniel Whiteson?

  So then you send it to a journal.

• 43:36Kelly Weinersmith?

  Yeah.

• 43:36Daniel Whiteson?

  And you pick a journal, like if you think it's a really exciting paper, you send it to Nature or Science. If you think it's less exciting, you send it to a journal with a smaller impact factor. So in particle physics, we often publish in like Physical Review or in Journal of High Energy Physics or these kinds of journals. And it goes to an editor and an editor finds people to review it. These are the peers and they reach out to folks they know who are experts in the field, who aren't your supervisor or your brother or this kind of thing, and ask them to review these things. And Katrina is a senior editor on a journal, so I see this process all the time. And she has to read the paper, think about who might know about it, and then find people to review it. And usually you have one to three people

• 44:17Daniel Whiteson?

  read this paper and give comments on it. But finding reviewers can be tough.

• 44:22Kelly Weinersmith?

  So for our grants that we talked about earlier, you have to make sure that the person reviewing the grant doesn't have a conflict of interest. They're not your brother or your supervisor or something like that. And in my field, the same holds true for when you're reviewing manuscripts. Is that true for your field as well?

• 44:36Daniel Whiteson?

  Yeah, absolutely.

• 44:37Kelly Weinersmith?

  Yeah.

• 44:38Daniel Whiteson?

  You have to say, oh, I can't read this one because that was my student or something like that.

• 44:43Kelly Weinersmith?

  Yeah. How long does review usually take for your field?

• 44:45Daniel Whiteson?

  Yeah, I'd say it's, you know, like four to eight weeks or something like that.

• 44:49Kelly Weinersmith?

  Wow.

• 44:50Daniel Whiteson?

  How about you guys?

• 44:50Kelly Weinersmith?

  I'd be happy four to eight weeks. That sounds good, but it can be sometimes six months. And at six months, you start writing the editor, being like, come on, man. This is crazy. But yeah, it can be hard to find reviewers.

• 45:03Daniel Whiteson?

  Because reviewers are just other scientists busy writing their own grants and writing their own papers and doing all their work and picking up their kids from daycare and stuff like this and trying to get through that mountain of laundry. You know, I think that people forget that like, science is just people, right? And people are busy.

• 45:18Kelly Weinersmith?

  Yeah.

• 45:18Daniel Whiteson?

  And if you're out there and you've written some treatise on the fundamental physics of the universe and nobody has read it, it's not because we're gatekeeping, it's just because we're busy, you know, and there's lots of those. And I often say no to editors who ask me to review stuff because I'm just too busy to do it in a timely manner.

• 45:33Kelly Weinersmith?

  Yeah, no, me as well. And I think it's worth noting that while we're doing this review work, we're doing it for free.

• 45:39Daniel Whiteson?

  Yeah.

• 45:39Kelly Weinersmith?

  Like, when you're on a grant review panel, you can lose weeks of time to reviewing these things, and you don't get reimbursed for that time. And the same thing goes. So when I review a manuscript, I read it for one day, takes me a couple hours to read carefully, and then I sit on it for a few days.

• 45:55Daniel Whiteson?

  Yeah.

• 45:55Kelly Weinersmith?

  And that's the main thing that is occupying my brain in the background for those days. And that could be, like, I could have been thinking about the introduction to my next book or something, but instead, I'm thinking about the methods to make sure it made sense. And then I spend another three to four hours reading it again and writing in detail my comments. And if I think there's some literature they missed, I go and I find it. And like, it's a multi-day process when I commit to reviewing a paper. And reviewers don't get paid for that, nor do the editors. How long does it take you?

• 46:22Daniel Whiteson?

  Well, I wish that all of my papers were reviewed by somebody as thorough as you. But my process is similar, like, an initial read, some thoughts, and then let it sit. Sometimes I'll discuss it with people in my group, we'll read it together, and then I read it again in detail. And then, especially if the review is negative, I wait, and I sit on it, and I come back later, I'm like, was this fair? And also, most importantly, was I nice enough? Are my comments constructive and helpful and not, like, just negative or harsh in any way? I'd, like, try to take a sandpaper to anything that's negative and smooth it over. Because I've been on the other end of harsh reviews, and it's, it doesn't help anybody to get zingers in there. And like, especially because a lot

• 47:07Daniel Whiteson?

  of my papers are written by young people, and it's often their first paper. And then I have to show them this review that's like, this guy's a jerk.

• 47:12Kelly Weinersmith?

  Yeah.

• 47:13Daniel Whiteson?

  This isn't necessary. I have to explain to them that the anonymity here is protecting them. But, you know, most reviewers are fair and are thoughtful and are courteous. And so often, we get, like, helpful feedback, like, what about this? Or have you thought about this question? Or what would you say to this concern? Do you find your feedback to be mostly useful?

• 47:31Kelly Weinersmith?

  Yeah, but most of the time, the feedback I get improves the paper. I did have one reviewer when it was one of my first papers who told me they were very disappointed in me. And it's like, You're a jerk.

• 47:42Daniel Whiteson?

  Disappointed in you.

• 47:45Daniel Whiteson?

  Thanks, Dad.

• 47:47Kelly Weinersmith?

  And that was because my supplemental materials were too long. I was like, God, give me a break. But anyway, so in my field, we're moving towards double-blind review. So the reviewers are not supposed to know the names of the authors and the authors are not supposed to know who reviewed their paper. And that's a really great idea so that you feel like you can honestly tell someone if their paper was good or not. You don't have to worry about someone getting mad at you in particular. Though in practice, there's usually like four people who could review your paper. So you know, like, you know, it was Joe or Francine. And Francine tends to be meaner. And so, but anyway, so yeah. What about in your field? Is it blind or a double-blind?

• 48:25Daniel Whiteson?

  It's only blind in one direction. The review is anonymous, but you see who the authors are. I know in some other fields, like in top computer science conferences, for example, it's double-blind and that's helpful, but also it's not that hard to figure it out. If you wanted to know who these people are, you could figure it out.

• 48:41Kelly Weinersmith?

  Yeah.

• 48:42Daniel Whiteson?

  But you might also be wondering like, what is the job of the reviewer exactly? Does reviewer have to make sure the experiment was right? Like if I'm checking Kelly's work, I'm the reviewer, do I have to go out and do the experiment and make sure she was right? And that's the thing is the peer review is not replication. The job of the reviewer is to ask, do the claims of the paper, are they supported by the evidence provided? Is there logic there? Is there mathematical tissue between the work that's been done and the conclusions that are being claimed? And also, is it well-written in the citations? And finally, is it interesting? Is it important? Does it play a role in the scientific conversation? Which is, you know, a little bit subjective, of course, but science is by the people and for the people. But if you're working on something nobody's interested in, and nobody's going to read

• 49:27Daniel Whiteson?

  your paper, then a journal might say, this is solid work, but nobody cares. It's a boring question, or you didn't learn anything interesting. The reviewer's job is not to say, hey, I did this experiment also, and I know that this is a feature of the universe. That's not the task of the reviewer. It's not their job to do that for you. It's also not the task of the reviewer to say like, hey, this is a cool idea. I would have done this differently. And so you now need to go back and do all these additional studies that I think are important. And you see this a lot in peer review, that reviews come back and say, this is cool, but also do X, Y, Z. And like, that's frustrating to me because that's not the job of the reviewer.

• 50:05Kelly Weinersmith?

  Yes. Yep. Nope. That's super frustrating. And I think it's worth noting that sometimes bad papers get through this procedure. And I, one of the ways that bad papers get through is that reviewers aren't required, or, and this makes sense, but they're not responsible for going through, for example, the codes for the statistical models that you ran. And so if somebody, you know, checked their codes a bunch of times, but they forgot a minus sign somewhere, the result could be wrong. And maybe nobody knows. And I, and I have known a couple of people who afterwards have like gone to use those models again for some other question and caught their mistake. And then a good scientist will contact that journal and say, I have to retract my paper, or I have to submit an errata and let everybody

• 50:48Kelly Weinersmith?

  know, I forgot the minus sign. These, it's a different result. And that's really painful, but at least they're being honest. And that's great. And I think that's really important.

• 50:57Daniel Whiteson?

  Yeah.

• 50:58Kelly Weinersmith?

  Or, some people are dishonest. And there's no way for the reviewer to know that. My field had somebody where they were one of the biggest names in the field. And there's this new requirement in our field. Well, it's a couple of decades old now. I'm old. But where you have to put the data that you use, that you collected as part of the experiment online somewhere, in a public place where people can download your data. And again, you know, in a field where animals are being used to collect data, it's great to know that those data could be used by other people to ask questions. You can get more information out of them. But it also means that if someone gets suspect results, you can pull their data and rerun the models. And when somebody was looking at the data, it was clear that

• 51:41Kelly Weinersmith?

  the numbers just didn't make sense. Like, one column was always the Prior column times three. And it anyway, after more scrutiny, it became clear that this person was making up their data. But you know, we have this new check where your data have to be available to everyone else. And that has helped us sort of tease out people who are being dishonest. So the system is evolving and getting better over time. But still, sometimes you get stuff through.

• 52:04Daniel Whiteson?

  You certainly do. And there are folks out there who are like combing through papers to find this stuff. And there's a scientist, for example, her name is Elizabeth Bick, and this is her passion. She combs through old papers and finds evidence of like photoshopping in biology. You know, you take a picture of your gel, and it's supposed to have this blob, and she finds, oh, this blob is the same as that blob, or it's been reverted, or whatever. And so there are definitely lots of ways that you could find this stuff now that we couldn't have done beforehand. But peer review is not a guarantee that this hasn't been done. Like reviewers should look for this stuff and call it out if you see it. But stuff definitely can get through. It's not a perfect system. You know, it's one of these like, it's a terrible system, but it's also the best that we have so far.

• 52:48Kelly Weinersmith?

  Just an aside here, I am a massive wimp. If I like purposefully fabricated data in this era where everything is like public and people can peek behind the curtain, I would never sleep again. I'd be like, someone's gonna find me out. And it would be like, I would be miserable the rest, the rest of my life. I'd rather have a bunch of low, low impact papers than worry that I was gonna get called out for lying. But anyway, I am a wimp.

• 53:13Daniel Whiteson?

  Yeah. And so the highest standard really is not just that something has been peer reviewed, but that something has been independently replicated. Like, you might have a scientist who's doing totally solid work and see some effect in their lab, but doesn't realize that it's an artifact due to some condition, the humidity, or you know, the local gravity, or the trains or something are causing some influence on their experiments. And so you want people on the other side of the world who built it differently, who made different assumptions, who sensitive different stuff to reproduce it. You might remember this excitement about high temperature superconductors a couple of years ago, LK99, Korean scientists claimed to have created this room temperature cheap superconductor, which would revolutionize

• 53:55Daniel Whiteson?

  Is the industry. And so very quickly, people were out there trying to replicate it. And people were excited, but until another independent group built it and showed that it was real, nobody really accepted it and thought we were in a new era. And personally, for example, when we were discovering the Higgs boson, I saw the data accumulating around a mass of 125. We were looking at it constantly. It was building up and up and up. But until I heard that my colleagues around the ring also saw an effect at the same place, I didn't personally think, okay, yeah, we found this thing. And so this sort of independent replication is really sort of the highest standard. What do you think, Kelly?

• 54:32Kelly Weinersmith?

  I think so. And to me, this is one of the current weaknesses in science, at least in my field. So replicating data is absolutely critical. But if you write to the National Science Foundation, for example, and you're like, oh, I just want to do the same experiment that Maria did, but I'm going to do it on another continent. It's going to be hard to get money for that because it's not a new idea. And it's also probably not going to get published in a top tier journal. And so if you are training PhD students who are going to want to get jobs and they're just quote unquote, just replicating somebody else's work, that's not going to help them get a job as much as following up on their own exciting new idea. And so we've got this incentive structure that's actually not super great for encouraging

• 55:16Kelly Weinersmith?

  people to replicate each other's studies. But I do think it's absolutely critical and I would love to see us sort of work on that incentive structure to make replication just as important as the initial thing that got done.

• 55:28Daniel Whiteson?

  That's interesting. In particle physics, there's maybe a slightly healthier environment. We have a few signals of new physics that we've seen in experiments that we're all curious about, but nobody really accepts because it doesn't quite make sense. And then there's been another generation of experiments to follow up on those. So for example, there's a very significant signal of dark matter, an experiment in Italy called Dama. And nobody really believes it because we've never seen it in another experiment. And people have set up other experiments very similar to Dama, but in another part of the world with different conditions and a different cave, for example, and not seeing the same effects. And so those were experiments definitely inspired. By this signal to test this in other conditions.

• 56:09Daniel Whiteson?

  And in the last few years, there was this quote unquote discovery of a fifth force in this experiment in Hungary. And folks in Berkeley are trying to replicate it. And there's an experiment in Italy trying to probe it as well. And so there's definitely like follow up work. But usually those follow up experiments are a little bit broader and they try to not just check this one new result, but also learn something else along the way to make it so they're also covering new ground. But I agree, we should definitely have replication, but it comes back to funding, right? Like if you're a reviewer, what would you rather fund? Like, let's check Kelly study or let's do this brand new thing that could tell us something new about the universe.

• 56:46Kelly Weinersmith?

  Yeah, and when I write a grant, I try to work my own replication in there. Like as part of asking a new question, I'm going to repeat the experiment, but maybe put a little tweak. But then I can at least make sure that I'm still getting the same results in, you know, a different space or with slightly different lighting or something like that. And so, and you know, especially when, to defend my field a little bit, especially when animals need to be euthanized as part of the experiment. You know, you, you might be a little bit hesitant to be like, well, this is just for replication. Or if you already got an answer, do animals need to? And so anyway, so we do also try to ask additional questions while trying to replicate. But it would be nice if we had more incentive for that.

• 57:22Daniel Whiteson?

  And there's folks out there just doing this, like you might've heard of the replication crisis. That comes out of people finding papers in the literature and saying like, all right, let's reproduce this. Let's see if it holds up. And this is one reason that like P hacking is a thing people talk about because we've discovered that some of the results in literature are just statistical artifacts that were selected in order to get a paper out there. And so I think what you're seeing in a broader sense is science is self correcting. Just like we saw that we needed to add external reviewers and members of the public. When we're talking about the experiments you do on animals. Now we see like, okay, we need some sort of way to protect against this kind of abuse as well. And so, you know, the process is a living thing.

• 58:06Daniel Whiteson?

  It's not like science has a crisp philosophy. What we call science has changed over the last 10 years, 50 years, 100 years. And it will continue to evolve and I hope keep delivering great truths about the universe.

• 58:18Kelly Weinersmith?

  Yeah, me too. And so Stephen asked, how can you know if something is good science or not? I feel like that was the, the big push behind the question. And so what do you think? When you read a paper for the first time, what do you look for?

• 58:32Daniel Whiteson?

  Well, you know, science has to stand on its own. So the thing I look for is like, does the paper make sense? Does it lay out an argument? Do the conclusions follow from the evidence presented? That's the most important thing to me. But before I'm going to personally believe that something is real part of the universe, yeah, I need to see it reproduced by another group. I've often reproduced papers myself, especially if they're like statistical, just to make sure I understand like, how is this calculation being done exactly? How do you get from step three to four? Because I really want to understand what are the assumptions being made? And think about whether those are broad enough. So I would say that yes, peer review papers can be wrong, but that's part of science. And the highest standard, I think, is independent replication. What do you think, Kelly?

• 59:14Kelly Weinersmith?

  Yeah, no, I agree. I mean, when I read a paper, I'll look to see, you know, what journal is it in? So lately, there in this day and age, there are some new, what we call predatory journals, where they will encourage people to submit, but they publish just about anything they get. And peer review isn't so much about peers having a chance to turn down a study, but they'll like, give a little bit of input, but the paper gets published one way or another. So you need to look to see what kind of journal it was published in. And then, you know, if it's a field I know about, I'll look to see, did you cite the papers that I would expect you to be citing? Was your literature search deep enough? And were your experiments designed well? What else could those results have meant? And then who funded the study? And stuff like that.

• 59:57Kelly Weinersmith?

  But you know, if you are a member of the general public and you can't do all of that, and you're just reading a pop sci summary, I would look for, again, like, where is this being published in? You know, like if it's The Atlantic by Ed Yong, it's probably great. And if the science journalist had other scientists weigh in on what might have been wrong about the study, that's a good sign. And so you look for where you're getting the information, how critical they seem to be, and stuff like that. What do you look for in a pop sci article?

• 1:00:24Daniel Whiteson?

  Yeah, in the Popsly article, I definitely look to see whether they have talked to people who are not authors, you know, other people in the field who know this stuff. And is it just a press release from the university? Or is it journalists who's actually thought about this stuff and written something up? Those are definitely the things I look for in Popsly press releases because there's the temptation in the sort of marketplace of ideas to overinflate the meaning of an incremental study.

• 1:00:50Kelly Weinersmith?

  Yep, yep. All right. So in general, I would say that, you know, we have this process in place to try to make sure that the best ideas are the ones that move forward and that we're all checking to make sure no one missed anything important along the way. Some stuff gets through either by mistake or because some people are unscrupulous, but hopefully that doesn't happen that often. But, you know, the system is evolving at all times. And if you have a study that you're interested in, but you don't know if you can trust the press release or whatever, you can send it to us. And if it happens to be in our wheelhouse, we're happy to weigh in and tell you what sort of, you know, set off our alarm bells in our heads or what we liked about the study. And we're, we're happy to help people figure out what, what was done well and what was just squeaking through.

• 1:01:34Daniel Whiteson?

  All right. Thanks very much, Stephen, for asking this question and for shining a light on the inner workings of science.

• 1:01:41Kelly Weinersmith?

  And if you'd like to reach out to us, send us an email at [email protected]. And let's see what Stephen had to say about our answer.

• 1:01:49Speaker 03

  Hi, Daniel and Kelly. It's Steve here. Thanks so much for answering my question. I think you shed a lot of light on some topics that most of the general public are not aware of, and I really appreciate that. It's actually really fascinating to learn that just because something is peer reviewed doesn't mean it's 100% fact. And it's definitely a lot to take away here. So I appreciate you guys diving into the topic and looking forward to the next episode.

• 1:02:25Kelly Weinersmith?

  Daniel and Kelly's Extraordinary Universe is produced by iHeartRadio. We would love to hear from you.

• 1:02:31Daniel Whiteson?

  We really would. We want to know what questions you have about this extraordinary universe.

• 1:02:37Kelly Weinersmith?

  We want to know your thoughts on recent shows, suggestions for future shows. If you contact us, we will get back to you.

• 1:02:43Daniel Whiteson?

  We really mean it. We answer every message. Email us at [email protected].

• 1:02:50Kelly Weinersmith?

  Or you can find us on social media. We have accounts on X, Instagram, Blue Sky. And on all of those platforms, you can find us at D and K Universe.

• 1:02:59Daniel Whiteson?

  Don't be shy. Write to us.

• 1:03:03Speaker 04

  OnDeck is built to back small businesses like yours. Whether you're buying equipment, expanding your team, or bridging cash flow gaps, OnDeck's loans up to $400,000 help make it happen fast. Rated A+ by the Better Business Bureau and earning thousands of five-star Trust Pilot reviews, OnDeck delivers funding you can count on. Apply in minutes at ondeck.com. Depending on certain loan attributes, your business loan may be issued by OnDeck or Celtic Bank. OnDeck does not lend in North Dakota. All loans and amounts subject to lender approval.

• 1:03:32Speaker 07

  With ChatGPT, you can do more than get answers. You can get things done. Whether you're leading a big project, planning an elaborate trip, or trying to balance work and life, ChatGPT can help. Give it a goal, plus things like files, notes, or project plans. Develop a trip itinerary with daily updates or create a prioritized schedule. So you can spend less energy sorting out the details and more on bringing your biggest ideas to life. Get started at chatgpt.com.

• 1:04:01Speaker 07

  What's that? My fresh detector. Uh, your what? There it is. What is it detecting? Fresh produce, fresh meat. Okay, what happens if it detects something really fresh?

• 1:04:15Speaker 04

  Apparently we celebrate. At Food Lion, you can count on fresh food at fresher prices when your fresh detector is going off. Head to your local Food Lion or shop at foodlion.com. Sponsored by GSK, the makers of Extenture, Depamocumab ULAA. If your asthma keeps interrupting, it may be time to try something else. Extenture is an ultra-long-acting prescription-maintenance treatment for severe asthma, eosinophilic type, ages 12 and older. It isn't for sudden breathing problems. Extenture is the first and only add-on treatment that can help deliver protection from asthma attacks with two doses a year. Just one dose every six months. That's a lot of time for everything in between.

• 1:04:58Speaker 04

  Serious allergic reactions can occur. Get help right away for swelling of face, mouth, tongue, or trouble breathing. Don't change or stop medicines, including steroids, unless told by your doctor. Report if you have a parasitic infection, common cold, sinus infection, inflammation in the nose, flu or flulike symptoms, joint pain, or sore throat may occur. Carry on with fewer attacks and ask your doctor about Extenture. Learn more at extenture.com or call 1-833-393-3678. This is an iHeart Podcast. Guaranteed Human.