Why Peer-Reviewed Doesn’t Mean Perfect

A published study is a starting point, not a final word

Jun 18, 2025

One recurring theme on Beyond the Abstract is the value of examining peer-reviewed studies with a healthy dose of skepticism — not to tear them down, but to make science and medicine better. But that raises an important question: if all the research we're critiquing has already been peer-reviewed, why are there so many issues with it? Is it contrarian — or even anti-science — to question peer-reviewed research?

That’s exactly what we’ll explore this week. Ultimately, I’ll introduce a simple analogy that helps frame how I think about peer-reviewed papers: a peer-reviewed article is like a bachelor’s degree. It's meaningful, but not final. Let's walk through why.

The Simplistic View

When a study is described as “peer-reviewed,” it often signals the end of the conversation. For many, the label conveys a kind of finality — evidence has been vetted, experts have weighed in, and the result is something we can trust. Questioning a peer-reviewed study can feel, in some circles, like questioning science itself.

But this view, while understandable, oversimplifies what peer review really is. My goal here isn’t to dismiss the process or undermine the value of scientific publishing. It’s to examine what peer review actually entails — what it does well, what it doesn’t do, and why understanding its limitations is essential for anyone who truly wants to engage with science in an informed, thoughtful way. Healthy skepticism doesn’t reject science — it respects it enough to look closely.

Misperceptions of Peer Review

Not long ago, I asked a group of new graduate students to estimate how many reviewers typically assess a peer-reviewed paper before publication. One question was simple: “How many people usually review a manuscript?” About half the respondents believed it was at least 20. A few thought more than 75. That kind of scrutiny might sound ideal — but the reality is far more modest.

In most journals, a manuscript is reviewed by two to three people. Occasionally, it may be just one. In rare cases — especially at high-profile journals or for controversial submissions—it might be reviewed by six or more (thank you, The BMJ). The exact number depends on journal policy, editorial discretion, reviewer availability, and the complexity of the topic. But it's rarely more than a handful.

So who are these reviewers? Typically, they’re other researchers — experts doing their best to provide constructive feedback, often anonymously and almost always without compensation. Their job is to help editors decide whether a paper is publishable and, ideally, offer suggestions to improve it. But they’re also human. They’re juggling lab meetings, teaching, deadlines, and their own research. They bring their own perspectives, preferences, and sometimes, misunderstandings. That doesn’t mean peer review is broken or untrustworthy — it just means, like every human system, it’s imperfect.

When It Works, It Works Well

When peer review functions as intended, it improves the paper. I’ve published over 80 peer-reviewed papers and served as a reviewer for more than 60 journals. In my experience, the process usually works — even if some of the feedback is frustrating and revisions require considerable effort. But, in my experience, reviewer feedback ultimately makes for a better published product.

Reviewers provide insight, flag gaps, and help clarify ideas. Their comments may range from helpful suggestions or true mandates — but it’s up to the authors to respond, revise, or respectfully disagree… and then up to the reviewers and editor(s) to determine if the paper is worthy of acceptance.

Still, peer review is thin. Ultimately, a small number of people — sometimes just two or three — read the paper and weigh in. That’s the whole process.

A Revealing Example

To see how human this process really is, let me share one example. I once reviewed a paper alongside two other reviewers. Reviewer #1 wrote a single sentence: “This is a great paper and worthwhile contribution to the literature, and should be accepted as is.” Reviewer #2 offered a short paragraph of mild praise and also recommended acceptance. Then there was me. I thought the paper had merit, but clearly needed substantial revision.

The most striking issue? The paper repeatedly referred to Figure 4, a key part of the results. But Figure 4 wasn’t there. Figures 3 and 5 were present, but 4 had been omitted entirely. It turned out the authors had simply forgotten to upload it, and they corrected it in the revised version. Still, the question lingered: how did two reviewers recommend acceptance without noticing that a central figure was missing?

This is why we use multiple peer reviewers — to catch different things. But it also shows how much can be missed. If something that obvious can slip through, what about more subtle issues? Peer review isn’t a flawless filter. It’s a human process, vulnerable to distraction, fatigue, and assumption. And yet, we often treat it like a stamp of scientific perfection.

The College Graduate Analogy

To put this in perspective, consider how we evaluate a college graduate. Over four years, a student is assessed across dozens of courses, usually 30 or more, graded by faculty through exams, papers, labs, and projects. This is a longitudinal, multilayered vetting process involving dozens of content experts over thousands of hours.

We all understand that earning a degree doesn’t mean someone got every question right. I certainly didn’t ace every exam. And in over 15 years of teaching, even my most brilliant students have never earned a perfect score for an entire semester. A degree signals competence — not perfection. It reflects that a student met the standard.

And yet we know that having a degree doesn’t make someone’s knowledge infallible. I’m reminded of the scene from Tommy Boy, where Chris Farley’s character, newly graduated after seven years, proudly declares that “Herbie Hancock” signed the Declaration of Independence. He’s passed the system. He has the degree. But that doesn’t mean we should trust his historical interpretation.

The same applies to peer-reviewed papers. They’ve passed a threshold. They’ve been reviewed by a few experts. But we shouldn't treat them as flawless or final. Like a college degree, a peer-reviewed paper is a meaningful step — not a guarantee of perfection.

What Peer Review Really Tells Us

Whether assessing students or scientific papers, reviewers are human. They may provide thoughtful, detailed feedback — or miss something entirely. It’s not that peer review is meaningless. It’s that we should be careful not to treat it as more rigorous, comprehensive, or exhaustive than it really is.

Yet in public discourse, “peer-reviewed” is often invoked as though it signals complete validation — like a study has been dissected by a panel of experts over months. In reality, it may have been reviewed by two people, neither of whom saw the raw data, and both of whom may have overlooked key issues. That’s not a flaw of science. That’s how the system works. And if we want to engage with science honestly, we need to understand that.

What This Means for Us

This matters — especially for clinicians, journalists, policymakers, and anyone trying to make sense of new research. A peer-reviewed paper is a signal, not a verdict. It deserves careful attention, not blind trust. Instead of asking, “Is this study true?”, we’re better served by asking, “How strong is the evidence? How well does it apply here? What other research supports or contradicts it?”

Just as we wouldn’t give someone the keys to a hospital simply because they have a diploma, we shouldn’t hand over the reins of clinical decision-making — or public policy — based solely on one published paper. Evidence isn’t something you follow blindly. It’s something you interpret carefully.

The Takeaway

Getting a degree doesn’t mean you’re right about everything in your field — it means you’ve started the journey. Likewise, a peer-reviewed article isn’t the final word. It’s an entry point into the broader scientific conversation. The real work of science begins after publication — through replication, debate, reanalysis, and integration into a larger body of evidence. Trustworthy knowledge is rarely the result of a single paper. It’s what remains standing after a whole stack of them has been rigorously tested over time.

Recognizing the limitations of peer review isn’t anti-science. It’s pro-science. It means respecting science enough to be honest about how it works. Science isn’t trustworthy because it’s perfect — it’s trustworthy because it welcomes correction. Peer-reviewed research is essential, but it’s also fallible, provisional, and shaped by human judgment.

So the next time someone cites a peer-reviewed paper as if it settles the debate, remember: it’s more like a degree than a decree. It means someone did something hard. It means they passed an evaluation. But it’s just the beginning of what we still need to learn.

If you found this helpful, consider sharing it with a colleague, student, or friend who’s trying to make sense of research and wants to ask better questions. And if you haven’t already, I’d love for you to subscribe to Beyond the Abstract — and invite others to do the same. I write this newsletter to help people better understand science and academia, and every subscription is a signal that the work resonates.