The Study of the Week Looks Back and Asks One Question

What is going on with medical science?

I am worried about the state of medical science and our journals.

At the risk of sounding naive, I want to believe that people who choose academic medicine do so to answer important scientific questions.

I’ve traveled the world and met academics. Many of them impress me as serious people trying to advance the field.

But that observation is hard to square with some of the studies that make it into big journals—and then into the lay press.

Today let’s do a bit of meta-analysis of recent examples.

Example 1:

Last week, Professor James McCormack described a subgroup analysis of a trial studying a new cholesterol-lowering drug called bempedoic acid.

A subgroup analysis of a trial involves comparing two types of patients within the main trial group—for instance, diabetic/non-diabetic, male/female, older than 55/younger than 55, etc.

JAMA published the subgroup analysis of primary/secondary prevention patients. But. But. They only presented the data for the primary prevention arm—which happened to have the stronger results.

Here is the thing: everyone knew this was a problem. The authors of the paper knew it. The peer reviewers and editors and statisticians also knew it. But they did it anyways. And more than 80 media outlets covered the study. One small caveat: JAMA editors at least allowed a critical editorial.

Example 2

I wrote yesterday about the MONITOR HF trial of an expensive device placed in the pulmonary artery to help manage patients with heart failure.

The trial was designed without a proper placebo arm. As an open-label trial, one group knew that they got the procedure and fancy device and the other group knew that they did not. The primary endpoint was subjective: how patients score their health on a questionnaire.

Here is the thing: everyone knew this was wrong. The authors knew that a subjective endpoint required proper blinding. The editors and peer reviewers also knew it. But there it is, published in the top journal, The Lancet.

Along with that fatally flawed study, was a glowing editorial from an author who declared strong financial relationships with the company that makes the device.

Example 3

This winter, I wrote about the ATLAS trial comparing the subcutaneous ICD to a regular transvenous ICD. The S-ICD is a more expensive device that is heavily marketed to doctors in my field.

Get this. To compare these two devices, the authors chose an endpoint with 5 bad outcomes from the implant. But. But. Four of the five endpoints could only occur in the transvenous ICD arm.

When the positive results came out, so did the marketing materials.

Here is the thing: the authors of this study come from one of the world’s top research organizations. They knew. The editors and peer reviewers of the Annals of Internal Medicine also knew the fatal design flaw. But there it is, published in a prominent journal.

Example 4

Of all journals, we expect the utmost from the New England Journal of Medicine. They hold the loftiest reputation. One reason is that they rarely publish observational studies that attempt to make causal claims.

But in November of 2022, the NEJM published an observational comparison of masking in schools. This was an attempt to compare—in retrospect without randomization—two groups of individuals. I have written often about the problems with this technique, namely, without randomization, you can’t tell if the groups were similar.

Here at Sensible Medicine, Tracy Beth Hoeg, MD, PhD, offered a point-by-point rebuttal of the limitations of this paper. Instead of this sort of critical appraisal, NEJM published a supportive editorial.

In this case, it is possible that the authors actually thought their analysis was correct. But surely the neutral scientists at NEJM knew otherwise. And if so why did they publish such a flawed analysis?

---

I could go on with more examples. Let me close with two big thoughts.

First is that NEJM, JAMA, The Lancet and the Annals of Internal Medicine are the top journals in all of medicine.

If such biased and flawed papers can not only reach this level, but do so with outsized conclusions and glowing editorials, what are consumers of medical evidence to think of the adjudication process of science?

We all agree that someone or some process has to judge science. But if this can happen, how can we trust all the other papers in these journals? The system seems broken.

The second thought is worse.

In most (if not all) of these examples, the authors knew the fatal flaw in their study but ignored it.

I may be wrong; I hope I am wrong, but if they knew the design and analysis was flawed and passed it off anyways, then…whoa. I don’t want to go there. It makes me extremely sad.

---

I am happy for anyone in the comments to help me feel better about our current climate of medical evidence. Or suggest a better way.

I would also urge you to think past pure financial conflicts of interest. Money is obviously a driver of conflicts, but it can’t be the only issue.