The Study of the Week explores a provocative trial presented at the recent American Heart Association meeting. The message goes to the norms of reviewing science
I’m a doctoral student in a political science program and for my quantitative analysis 2 course we need to find a paper that uses a non-linear regression model, critique the paper and extend the analysis.
I have been searching for DAYS for papers on my topic that have published data and replication files (also limited by our use of Stata and not R or other programs).
My professor has been adamant that transparency regarding data and methods is paramount for academics but this clearly isn’t the norm.
Thanks for writing about this! I’m learning a lot (slowly but surely) of great context on interpreting scientific papers/results/studies!
Please keep up your writing JOHN MANDROLA. I truly appreciate your point-of-view so far and am out of the loop in medicine, being 79...but still interested. Thank you.
So I’m a mid level- FNP, as a nurse we always ask to see some results. It is almost never called source data, but we’ll say how many clients have you treated with X?, or what were the goals you met with it? Who else is using X that I can speak with. Bringing some polite skepticism into our practice is a safety check we all require. In my mid level opinion.
Yes, I've work in Computer Science / Bioinformatics / Complex Systems / Simulations, and they do not ask for the code either. The reviewers only check if the results are consistent with what they already believe to be true or reasonable, or in line with the mainstream ideology in the field. My PhD advisor used to say: "We have to write our results in a way that we can sell the idea". "Selling the idea???" No wonder why I am an independent researcher now, I don't sell BS.
In any case, no one would be able to review all the data, codes and everything, even though it should be submitted and stored, in case someone in the future wants to check. But then you have issues with proprietary code, methodologies... which is kind of ridiculous. This whole issue happens because research, within Academia and within Industry, uses a business model, where data and source materials and methods must be kept secret. The whole foundation of publication, peer-review process and authorship is completely unscientific in my opinion.
It is glaring— the double standards in medical science. Scientists should accept every experiment with an open mind. If other experiments were accepted without data source, why shouldn’t it be thesame for the Chinese “Tongxinluo”?
As an African, seeing how a traditional Chinese physician— Tu Youyou persevered in bringing global attention to the Chinese herb: “Artemisinin”, which still remains the messiah in preventing so many malaria attributable deaths in my continent, I think every potential cure should be given a chance.
I’m not excusing data source presentation, but if data source is gonna be asked, it should be for all experiments!
With what we saw with the certification of this COVID “vaccine” by the FDA every potential drug no matter how its describe could get certified for use...if they called it HCQ or ivermectin they would be delayed forever. The entire credibility of medical science has been turned upside down...those who believe they are “scientist” could be ascribed to my youngest daughter who has little interest in science.
Thank you for this article. The provision of data should indeed be mandatory. They might not prevent all fraud as even raw data can be defrauded, but that is another problem and requires "excellent forgery work" which I hope is much rarer. This could perhaps also sometimes allow us to look at the facts from another angle, for example by analyzing more precisely groups at risk or more specific (example: age), to identify biases that have gone unnoticed or to redo the study with a another primary end point😀. I often wondered during this period, what would have given the analysis of the data (and their interpretation) with a researcher with another angle or point of view. The conclusion might have been radically different?
Judging by the AZ mask study, and NEJM "mask is better-pay no attention to confounding variables" study, no, it doesn't surprise me. It makes me sad that we are in a place where"science" "scientist" and "trust" are orthogonal. When Fauci uttered the "I represent science" but didn't elaborate on any possible financial entanglement.....well what are users who don't have a Fauci bobblehead doll (to borrow Dr. Prasad's expression) to exepct? But here we are. I think Sensible Medicine substack is a way of bridging the gap. As for is this because the "Chinese did it?" response? I think that's right but for the wrong reasons. What do I mean? China is a country that has no transparency. Whether that's finance, housing, intellectual property, human health etc., there isn't much to trust. So it's an understandable knee-jerk reaction. But as the article pointed out, the irony is that no one asks for it anyway. So really, we're not talking about accuracy of data, we're talking about complacency of trust. And that there is the whole ball of wax. It's the one thing I'm grateful for the pandemic. As stupid as that sounds, it truly opened my eyes to how much garbage plays in to "science" Color me completely and utterly jaded.
As an author of 100+ refereed publications and and editor for several journals over the years, I have been bleating about this for decades. If one is publishing something with results and conclusions, then I believe it is imperative that all of the data be made available.
How this is even a point for discussion I still find completely perplexing. The fact that this is now coming up as a novel idea in this article is bemusing to say the least. It should be the core of every refereed publication.
As a reviewer on many papers, I ALWAYS insist on having access to the raw data or I will not review. It has also always amazed me that other reviewers on the same paper do not do so.
I had no idea that source data wasn’t regularly shared. What, in effect, are their peers reviewing, then? Solely methodology (gender and professional status)?
"The sudden desire to review source data".... tres riche and a critical reason why trust for Pfizicians has all but evaporated for millions of people worldwide. You guys really need to pick a lane. When it comes to the C 19 vaccines (which killed my friend's 17 year old son who got sick immediately after his first shot, and was dead a month later) no questions dared be asked- especially not by patients, who were shamed for doing so. Some docs even encouraged that they be ostracized from society and that medical treatment be withheld from them for the crime of choosing not to be vaccinated with a product which you did NOT review the source data on! To add insult to literal injury, when those same patients become injured, you continue to deny that vaccines are at fault, or admit that they are- but refuse to 'go on the record'. The only thing that separates you from used car salesman at this point is that used car salesman ACKNOWLEDGE that they are conflicted because they put their own interests first.
Lo and behold though, now 'suddenly', you've discovered you need to 'review source data' before accepting the manufacturer's claims about a novel pharma product. Honestly- get stuffed- until you:
1. APOLOGIZE. Admit the C19 vaccines cause injury to a significant number of recipients and that you
you didn't do your due diligence before recommending that any and all be vaccinated. FFS, it's not like the info on mRNA technology 'toxicity' and 'dangerous side effects" wasn't there. Moderna and BioNTech execs themselves told us this verbatim as far back as 2016:
They even warned that because of the dangers inherent in the LNP, mRNA based pharmaceuticals *should not be used in repeat doses over months and years- but here we are, with pfizicians worldwide still recommending that we do precisely that.
2. ADMIT that you rolled over for your licensing body pharma whores, neglected your duty to your patients to 'do no harm', and that you put yourselves and your interests, first.
3. SUPPORT your patients and get behind them in holding your overlords responsible for this fraudulent negligence. Then disband and replace them with licensing bodies beholden to the people you serve- your patients.
4. HELP the vaccine injured! Accept that you caused their injuries because you neglected to provide information which would have informed their consent, help them receive compensation, and address their injuries.
5. THEN, promise a us you'll NEVER let this happened again, because you have new-found and 'sudden desire to review source data' .
I read Rigor Mortis by Richard Harris at the end of 2019. Since then I've been able to watch scientific (or "scientific") discussions and developments practically in real time. This is, unfortunately, old news.
What stuns me is when thinking people in the field are not appalled and sickened by the state of affairs. That they don't demand change - now, if not yesterday. That we're all just sitting here like [insert local wildlife] in the headlights waiting to be the next roadkill
It’s utterly absurd that source data is not reviewed. When financial gain is on the line, humans will do anything to watch out for their own interests even if it means fudging the numbers a little. Trust but verify
In psychology, where I publish, I preregister most of my studies. I also upload data for every study I have permission to. This is becoming routine. OSF is an essential repository for this. I'm surprised at the degree to which this is not standard practice in medicine.
In the summer of 2020, a particularly obnoxious Facebook acquaintance - in the spirit of 'I told you so' - shared an Lancet abstract from a 'definitive' a study that had purportedly enrolled tens of thousands of people worldwide to study the benefit of hydroxychloroquine for Covid-19.
Yes, that study. The one that blew up in my acquaintance's face 2 weeks later when it was retracted for being completely fraudulent.
I had replied that, as far as I could tell from the abstract, that HCQ had in most cases been administered way too late, often to people already hospitalized. I did not mention, but I quietly wondered, how the authors had been able to recruit such a huge body of subjects, on relatively short notice, without anybody who was following the issue even being aware that such a study was even underway.
About the same time, a study out of Brazil purportedly proved HCQ - a drug used safely for decades to fight malaria - was in fact quite deadly, likely to kill you via cardiac dysrhythmias. Mind you, those who died had received 4 or 5 times the dose recommended by physicians who had experience treating Covid with the drug.
So the good news is, a lot of us out here have understood for a couple of years now that the pharmaceuticals companies have completely captured, not just the regulatory apparatus, FDA CDC NIH et al, but also the medical establishment itself and the journals that they publish.
So unless and until sharing source data with the public becomes the norm, those journals, the researchers, and the pharmaceuticals companies that often dictate their results can all go straight to hell.
I’m a doctoral student in a political science program and for my quantitative analysis 2 course we need to find a paper that uses a non-linear regression model, critique the paper and extend the analysis.
I have been searching for DAYS for papers on my topic that have published data and replication files (also limited by our use of Stata and not R or other programs).
My professor has been adamant that transparency regarding data and methods is paramount for academics but this clearly isn’t the norm.
Thanks for writing about this! I’m learning a lot (slowly but surely) of great context on interpreting scientific papers/results/studies!
Please keep up your writing JOHN MANDROLA. I truly appreciate your point-of-view so far and am out of the loop in medicine, being 79...but still interested. Thank you.
So I’m a mid level- FNP, as a nurse we always ask to see some results. It is almost never called source data, but we’ll say how many clients have you treated with X?, or what were the goals you met with it? Who else is using X that I can speak with. Bringing some polite skepticism into our practice is a safety check we all require. In my mid level opinion.
Yes, I've work in Computer Science / Bioinformatics / Complex Systems / Simulations, and they do not ask for the code either. The reviewers only check if the results are consistent with what they already believe to be true or reasonable, or in line with the mainstream ideology in the field. My PhD advisor used to say: "We have to write our results in a way that we can sell the idea". "Selling the idea???" No wonder why I am an independent researcher now, I don't sell BS.
In any case, no one would be able to review all the data, codes and everything, even though it should be submitted and stored, in case someone in the future wants to check. But then you have issues with proprietary code, methodologies... which is kind of ridiculous. This whole issue happens because research, within Academia and within Industry, uses a business model, where data and source materials and methods must be kept secret. The whole foundation of publication, peer-review process and authorship is completely unscientific in my opinion.
It is glaring— the double standards in medical science. Scientists should accept every experiment with an open mind. If other experiments were accepted without data source, why shouldn’t it be thesame for the Chinese “Tongxinluo”?
As an African, seeing how a traditional Chinese physician— Tu Youyou persevered in bringing global attention to the Chinese herb: “Artemisinin”, which still remains the messiah in preventing so many malaria attributable deaths in my continent, I think every potential cure should be given a chance.
I’m not excusing data source presentation, but if data source is gonna be asked, it should be for all experiments!
With what we saw with the certification of this COVID “vaccine” by the FDA every potential drug no matter how its describe could get certified for use...if they called it HCQ or ivermectin they would be delayed forever. The entire credibility of medical science has been turned upside down...those who believe they are “scientist” could be ascribed to my youngest daughter who has little interest in science.
Thank you for this article. The provision of data should indeed be mandatory. They might not prevent all fraud as even raw data can be defrauded, but that is another problem and requires "excellent forgery work" which I hope is much rarer. This could perhaps also sometimes allow us to look at the facts from another angle, for example by analyzing more precisely groups at risk or more specific (example: age), to identify biases that have gone unnoticed or to redo the study with a another primary end point😀. I often wondered during this period, what would have given the analysis of the data (and their interpretation) with a researcher with another angle or point of view. The conclusion might have been radically different?
Judging by the AZ mask study, and NEJM "mask is better-pay no attention to confounding variables" study, no, it doesn't surprise me. It makes me sad that we are in a place where"science" "scientist" and "trust" are orthogonal. When Fauci uttered the "I represent science" but didn't elaborate on any possible financial entanglement.....well what are users who don't have a Fauci bobblehead doll (to borrow Dr. Prasad's expression) to exepct? But here we are. I think Sensible Medicine substack is a way of bridging the gap. As for is this because the "Chinese did it?" response? I think that's right but for the wrong reasons. What do I mean? China is a country that has no transparency. Whether that's finance, housing, intellectual property, human health etc., there isn't much to trust. So it's an understandable knee-jerk reaction. But as the article pointed out, the irony is that no one asks for it anyway. So really, we're not talking about accuracy of data, we're talking about complacency of trust. And that there is the whole ball of wax. It's the one thing I'm grateful for the pandemic. As stupid as that sounds, it truly opened my eyes to how much garbage plays in to "science" Color me completely and utterly jaded.
As an author of 100+ refereed publications and and editor for several journals over the years, I have been bleating about this for decades. If one is publishing something with results and conclusions, then I believe it is imperative that all of the data be made available.
How this is even a point for discussion I still find completely perplexing. The fact that this is now coming up as a novel idea in this article is bemusing to say the least. It should be the core of every refereed publication.
As a reviewer on many papers, I ALWAYS insist on having access to the raw data or I will not review. It has also always amazed me that other reviewers on the same paper do not do so.
Keep up the good work!
I had no idea that source data wasn’t regularly shared. What, in effect, are their peers reviewing, then? Solely methodology (gender and professional status)?
Bluntly: Holy cow. You’re saying we typically ( ever??) didn’t* get data to review???
😮
Naive old nurse
"The sudden desire to review source data".... tres riche and a critical reason why trust for Pfizicians has all but evaporated for millions of people worldwide. You guys really need to pick a lane. When it comes to the C 19 vaccines (which killed my friend's 17 year old son who got sick immediately after his first shot, and was dead a month later) no questions dared be asked- especially not by patients, who were shamed for doing so. Some docs even encouraged that they be ostracized from society and that medical treatment be withheld from them for the crime of choosing not to be vaccinated with a product which you did NOT review the source data on! To add insult to literal injury, when those same patients become injured, you continue to deny that vaccines are at fault, or admit that they are- but refuse to 'go on the record'. The only thing that separates you from used car salesman at this point is that used car salesman ACKNOWLEDGE that they are conflicted because they put their own interests first.
Lo and behold though, now 'suddenly', you've discovered you need to 'review source data' before accepting the manufacturer's claims about a novel pharma product. Honestly- get stuffed- until you:
1. APOLOGIZE. Admit the C19 vaccines cause injury to a significant number of recipients and that you
you didn't do your due diligence before recommending that any and all be vaccinated. FFS, it's not like the info on mRNA technology 'toxicity' and 'dangerous side effects" wasn't there. Moderna and BioNTech execs themselves told us this verbatim as far back as 2016:
https://www.statnews.com/2016/09/13/moderna-therapeutics-biotech-mrna/
They even warned that because of the dangers inherent in the LNP, mRNA based pharmaceuticals *should not be used in repeat doses over months and years- but here we are, with pfizicians worldwide still recommending that we do precisely that.
2. ADMIT that you rolled over for your licensing body pharma whores, neglected your duty to your patients to 'do no harm', and that you put yourselves and your interests, first.
3. SUPPORT your patients and get behind them in holding your overlords responsible for this fraudulent negligence. Then disband and replace them with licensing bodies beholden to the people you serve- your patients.
4. HELP the vaccine injured! Accept that you caused their injuries because you neglected to provide information which would have informed their consent, help them receive compensation, and address their injuries.
5. THEN, promise a us you'll NEVER let this happened again, because you have new-found and 'sudden desire to review source data' .
I read Rigor Mortis by Richard Harris at the end of 2019. Since then I've been able to watch scientific (or "scientific") discussions and developments practically in real time. This is, unfortunately, old news.
What stuns me is when thinking people in the field are not appalled and sickened by the state of affairs. That they don't demand change - now, if not yesterday. That we're all just sitting here like [insert local wildlife] in the headlights waiting to be the next roadkill
It’s utterly absurd that source data is not reviewed. When financial gain is on the line, humans will do anything to watch out for their own interests even if it means fudging the numbers a little. Trust but verify
In psychology, where I publish, I preregister most of my studies. I also upload data for every study I have permission to. This is becoming routine. OSF is an essential repository for this. I'm surprised at the degree to which this is not standard practice in medicine.
In the summer of 2020, a particularly obnoxious Facebook acquaintance - in the spirit of 'I told you so' - shared an Lancet abstract from a 'definitive' a study that had purportedly enrolled tens of thousands of people worldwide to study the benefit of hydroxychloroquine for Covid-19.
Yes, that study. The one that blew up in my acquaintance's face 2 weeks later when it was retracted for being completely fraudulent.
I had replied that, as far as I could tell from the abstract, that HCQ had in most cases been administered way too late, often to people already hospitalized. I did not mention, but I quietly wondered, how the authors had been able to recruit such a huge body of subjects, on relatively short notice, without anybody who was following the issue even being aware that such a study was even underway.
About the same time, a study out of Brazil purportedly proved HCQ - a drug used safely for decades to fight malaria - was in fact quite deadly, likely to kill you via cardiac dysrhythmias. Mind you, those who died had received 4 or 5 times the dose recommended by physicians who had experience treating Covid with the drug.
So the good news is, a lot of us out here have understood for a couple of years now that the pharmaceuticals companies have completely captured, not just the regulatory apparatus, FDA CDC NIH et al, but also the medical establishment itself and the journals that they publish.
So unless and until sharing source data with the public becomes the norm, those journals, the researchers, and the pharmaceuticals companies that often dictate their results can all go straight to hell.