Sensible Medicine

Share this post
How Sure Should We Be About Healthcare Interventions?
sensiblemed.substack.com

How Sure Should We Be About Healthcare Interventions?

The Study of the Week should reset our prior beliefs about the success of healthcare interventions

John Mandrola
Oct 31, 2022
233
4
Share this post
How Sure Should We Be About Healthcare Interventions?
sensiblemed.substack.com

A sobering paper about the success of healthcare interventions flew under the radar of attention this Spring.

Professor Jeremy Howick from the University of Oxford had the idea to estimate the fraction of healthcare interventions that are shown to be effective by high-quality evidence.

(Before you read any further, pause and make a guess in your head.)

Along with a group of colleagues they looked at a random sample of nearly 2500 Cochrane Reviews published in the last 15 years. Most experts consider Cochrane Reviews the gold standard of medical evidence

Howick and colleagues studied interventions that were compared with placebo or no treatment and whose evidence quality were rated using something called the GRADE standard. GRADE specifies four levels of the certainty for a body of evidence for a given outcome: high, moderate, low and very low.

Then they calculated the fraction of interventions that met three criteria: a) efficacy based on high-quality evidence; b) statistically significant positive effects and c) were judged as beneficial by the Cochrane reviewers. They also looked at the fraction of interventions in which harms were studied.

Sit down for the results:

  • Only 6% of eligible interventions (n = 1567) met all three criteria of high-quality evidence and statistically significant positive effects enough to convince reviewers that it was beneficial.

  • Harms were only measured in about a third of interventions, of which 8% had statistically significant evidence of harm.

  • Among the interventions that did not have any outcomes with high quality evidence, the highest-GRADE rating was moderate in 30%, low in 34.0%, and very low in 24%.

Comments:

When you go to the doctor and undergo a test or procedure, or take a new medication, most of us expect that there’s darn good support for it.

This paper suggests that it’s not true.

When I started practice, this data would have surprised me. But as I’ve gathered experience in both practice and in the review of medical evidence, I am no longer surprised.

Everyone is far too confident.

Doctors prescribe a drug, and the patient improves. We ascribe the improvement to the drug. But we don’t know if the patient improved because of the prescription or despite it. When it’s an antibiotic, drawing such a conclusion is fair, but most drugs are unlike antibiotics.

Procedures are even more likely to fool our cause-seeking brains. A doctor does a procedure that induces a change in something that can be seen—a stent makes an angiogram look smoother; an ablation vanquishes a cardiac signal.

The visual feedback makes us feel successful; we get a rush of dopamine. If the patient improves, we make a causal connection between the visual feedback and the success. But we don’t know if the improvement is because of the procedure or despite it.

That’s where evidence comes in. If an intervention truly causes improvement, evidence bears it out. Many therapies we thought were successful ended up as mirages in our brains.


Here are some pictures to show why our prior beliefs matter. Think of trials as medical tests: they almost never give a definitive answer. Rather, trials modify our prior beliefs.

A typical “positive” cardiology trial often finds a 10-20% improvement in a new therapy.

The first picture below shows the before and after beliefs if we start with a gullible prior of more than 75% chance of things working.

See: if you start with overly optimistic priors, you end up cocksure that healthcare interventions most always work.

Now look what happens if you use the evidence from the Howick et al paper, where less than 10% of interventions have high-quality, statistically robust findings of benefit.

The *same* trial has a much different effect.

Conclusion:

I love being a doctor. I believe we help people. But one of our greatest foes is over-confidence.

Healthcare is hard. The human body is complex. Most stuff doesn’t work.

Patients and doctors alike would do better to be far more skeptical. Not cynical, just skeptical.

This would lead us to want stronger evidence for our interventions—especially when there is harm or high cost involved.

Then we would be less likely to adopt therapies that don’t work, or worse, harm people we are supposed to help.

4
Share this post
How Sure Should We Be About Healthcare Interventions?
sensiblemed.substack.com
4 Comments
Martin Derrow MD FACP
Nov 3, 2022

As a recently retired physician, age and experience provide a little perspective. The old adage, half of what you learn in medical school will turn out to be wrong, and we don't know which half, seems often true. Over the decades, so many therapeutic assumptions, interventions, at one time state of the art, subsequently thrown into question. We've all seen it repeatedly, cherished rock solid ideas upended with better data. Healthy informed skepticism is worth cultivating as an aspect of critical thinking and not just in medicine.

Expand full comment
ReplyCollapse
HardeeHo
Nov 2, 2022

I greatly appreciate the minimalist doctors that I now use. OTOH I am always checking advice. I did get the first two but only because I trusted the officials, my doctor and wished to sidestep my worries about exposure. I did try to validate the data. As time progressed and the booster was offered the data said no despite my age, 83, and various conditions post cancer issues. Still I worried but decided that I was still going to go shopping and remain active. I found the recommendations used by days of old - simple mouth/nose sanitation. I watch others around me get infected more than once. I pass on what I'm doing that seems to work, knock on something. Yet our officials say nothing. My doctor just nods OK by him. He also sanitizes. I haven't asked if he advises others. OTOH, I do my very best to avoid my doctors except on the routine schedules.

Expand full comment
ReplyCollapse
2 more comments…
TopNewCommunity

No posts

Ready for more?

© 2023 Editors
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great writing