The Power of Post Publication Review, A Case Study

Pic of the day - Libraries

There are many discussions and examples of post-publication review as an alternative to the currently more common peer-review model. While this comes up fairly regularly in my Twitter stream, I don’t think I’ve done more than hint at it within the blogposts here. I’ve also been watching (but neglecting to mention here) the emergence of data journalists and data journalism as a field, or perhaps perhaps I should say co-emergence, since it seems to be tightly coupled with shifts in the field of science communication and communicating risk to the public. Obviously, these all tie in tightly with the ethical constructs of informed consent and shared decisionmaking in healthcare (the phrase from the 1980s) which is now more often called participatory medicine.

That is quite a lot of jargon stuffed into one small paragraph. I could stuff it equally densely with citations to sources on these topics, definitions, and debates. Instead, for today, I’d like to give a brief overview of a case I’ve been privileged to observe unfolding over the weekend. If you want to see it directly, you’ll have to join the email list where this took place.

Part One: Publication

Last week, a new article on hormone replacement therapy (HRT) was published in the British Medical Journal (BMJ).

Schierbeck LL, Rejnmark L, Tofteng CL, Stilgren L, Eiken P, Mosekilde L, Køber L, Jensen JEB. Effect of hormone replacement therapy on cardiovascular events in recently postmenopausal women: randomised trial. BMJ 2012;345:e6409 doi: (Published 9 October 2012)

The article was on the outcomes from a clinical trial which includes more information in the trial registry.

Danish Osteoporosis Prevention Study

Two days later, a message was posted to an evidence-based health care email list ( [EBH]), asking for discussion of the article.

The same day, a Rapid Response was published by BMJ criticizing the article.

Mascitelli L, Goldstein MR. The flawed beneficial effects of hormone replacement therapy. BMJ.

The Rapid Response closed with this delightful witticism.

“If you torture numbers enough they will say anything you want.”

Part Two: Discussion

Meanwhile, on the EBH list, the conversation was going fast and furious. I’m not going to quote individuals, but I would like to collate an overview of the topics covered.

– blinding (it wasn’t)
– placebo-controlled (nope)
– 8% of eligible patients recruited
– sample size (small, compared to the Women’s Health Initiative (WHI) study)
– age confounding of participants

Ethics / Bias:
– Funding (pharma)
– Authors linked to pharma

– incomplete?
– does it include the most important portions of the relevant evidence base?
– specifically lacking core references on the “age hypothesis”

– Were they the right folk? (Yes, the list was assured by a BMJ editor)
– Did they read the article critically and review it thoroughly, including the bibliography?

– implications for future practice
– placing this article appropriately in the context of the larger body of evidence
– implications for participatory medicine, informed consent, shared decisionmaking, and how to inform the public about risk for personal decisionmaking

Recommendations for future analysis:
– pool with similar data from other studies
– include in systematic review or meta-analysis
– strategic genomic analysis (NOTE: this was not available in 1993 when the study started)

– apparent publication delay (data collection first completed in 2003, then later followup in 2008, published in 2012)
– ghostwriting (specifically the history of it related to HRT)
– ‘System I’ thinking (gut feelings) vs ‘System II’ thinking (transparent methodological approach to decisionmaking)
– “science by sound-bite”

I’m not equipped to judge the article on any of these points. I did find it extremely interesting that the discussants included faculty and faculty emeritus from major universities both in the UK and the US, patient advocates, medical and health librarians, experts in evidence-based health care methodologies, and an editor of the journal which published the article.

Part Three: The Press

Of course, the press jumped all over this, in part because the BMJ press release directing attention to this study.

HRT taken for 10 years significantly reduces risk of heart failure and heart attack. BMJ Press Releases, Wednesday, October 10, 2012 – 08:37.

There are a lot of articles out there now in the popular press. Notice the type of language used.

BBC News: HRT reduces risk of heart attack, study suggests:

Guardian: : HRT can cut heart attack risk, study shows:

Telegraph: HRT is safe and cuts heart deaths, ‘significant study’ finds:

Time: Heart Benefits from Hormone Replacement Therapy?:

US News: Health Buzz: HRT May Be More Than Safe, Study Says:

Kind of makes you want to run out and get pills, doesn’t it? This one is not from a major popular press venue, but it has some interesting aspects. Again, look at the language used in the headline.

MedPage: HRT Helps Heart with No Cancer, Clot Risks; By Charles Bankhead, Staff Writer, MedPage Today:

This one is from a medical news service, and was published the same day as the original article, before even the BMJ press release. What is really interesting is that it says the article was reviewed prior to publication by an MD and medical faculty member.

Published: October 09, 2012
Reviewed by Robert Jasmer, MD; Associate Clinical Professor of Medicine, University of California, San Francisco and Dorothy Caputo, MA, BSN, RN, Nurse Planner

That’s awfully fast. I know they need to be fast, it’s news, but it seems to me that it would be hard to have enough time read carefully or think about the implications and context; to have time to do much more than think, “Hmmm, BMJ, a good journal. This article says what the BMJ article said. OK.” I’m not saying that’s what the doc did who signed off on it, I’m just saying that the process and speed lend themselves to flaws.

The authors of the article and the BMJ editor both emphasized that this is exceptionally unique data. Because of the WHI study, there is virtually no chance of generating this type of data again. That the Danish study findings run so contrary to the findings of the WHI study is shocking and noteworthy. Why? Is this significant enough to reopen the question of HRT risks? What does this mean for individual patients and clinicians attempting to make treatment plans and decisions?

Obviously, it is not as simple as the press would make it seem. Open access makes the article accessible, but without open post publication peer review, the CONTEXT is not made accessible. Open access can only go so far in supporting personal decisionmaking.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s