Systematic Reviews: Methodology, Overview, Sharing

In my previous job, a big part of what I did was systematic review searching. This is still going on as I am on a Cochrane team and a few grants. Technically, this isn’t probably emerging technologies. On the other hand I’ve found that with many “new” technologies it is all about who thinks it is new. For example, I’ve been teaching classes on using Delicious for at least five years, and there are still many people I work with now who have never heard of it. For them, Delicious, now approaching its 6th birthday, is an emerging technology.

Despite the popularity of systematic reviews in the medical literature, it is just now starting to really make the leap from being something a bit on the obscure side to something everyone has heard of. I think what is making the difference has been the push in many schools to include evidence based health care in the training of new clinicians, and to encourage graduate students to do systematic reviews as their research project.

My personal experience with systematic reviews goes back to roughly 1999 or 2000, when I was contracted by NIH to do systematic review searches for 13 of the teams preparing for the first Consensus Development Conference to require use of systematic review methodologies for all presenters.

NIH: Consensus Development Conference on Diagnosis and Management of Dental Caries Throughout Life (2001): http://consensus.nih.gov/2001/2001DentalCaries115html.htm

NIH: Consensus Development Conference on Diagnosis and Management of Dental Caries Throughout Life (2001) (University of Michigan, University Libraries): http://www.lib.umich.edu/health-sciences-libraries/nih-consensus-development-conference-diagnosis-and-management-dental-carie

I’ve lost track of how many systematic reviews I’ve worked on since that time, but it is a lot, ranging from graduate student projects to Cochrane teams, searching on topics with large research bases to those with almost no research. The techniques appropriate for searching in these various areas are very different. My particular area of expertise is searching in topics where there is a very small research base, meaning instead of seeking clinical significance the researchers are doing a preliminary project to define the levels and quality of evidence available on that topic. What I’ve noticed for many years now is that many of the articles published as systematic reviews do not actually follow an appropriate methodology for the topic being investigated. Another challenge is that many of the published systematic reviews do not include enough information about the search strategy for it to be replicated or verified by other research teams. I’m far from being the only person who has noticed this! Here is the most recent article I’ve seen examining the quality of published systematic reviews.

Song F, Loke YK, Walsh T, Glenny AM, Eastwood AJ, Altman DG. Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews. BMJ. 2009 Apr 3;338:b1147. doi: 10.1136/bmj.b1147. PMID: 19346285

The methodological “Bible” for doing a systematic review to determine clinical effectiveness is the Cochrane Handbook.

Cochrane Handbook: http://www.cochrane-handbook.org/ OR http://www.cochrane.org/resources/handbook/

Like the actual Bible, it takes a long time and a lot of work to actually read the Cochrane Handbook, making it impractical for many of the busy clinical researchers who are exploring the idea of publishing a systematic review. On many of the Cochrane systematic review teams the primary role of the librarian is to provide the data being analyzed. I have been finding that my experience having worked on so many teams and systematic review projects has placed me also in the role of providing assistance and insight to new systematic review team leaders with respect to the process and methodologies that should be followed. This actually makes a lot of sense, since both the statistician and the librarian on these projects are more likely to participate in large numbers of reviews, across a variety of topics. It also means that I have a large personal collection of search strategies I’ve developed for other reviews that can be quickly tapped or reworked, making the time required for the search process sometimes much faster.

Increasingly, when I work on a systematic review team I ask permission for the search strategy to be archived (post-publication of course) on a public web site to make it easier for other librarians to find and use. The NIH CDC on Caries was the first time we did that.

EBHC Strategies: http://ebhcstrategies.wetpaint.com/

A more fully fleshed out wiki is here:
EBM Librarian: http://ebmlibrarian.wetpaint.com/

EBHC is focused on creating a collection of search strategies to share
among other librarians so folks don’t need to reinvent the wheel; EBM
Librarian is more a tutorial, a detailed how-to as a collaboration of several of the most expert EBHC librarians and teachers.

I like the idea of both using social media and technologies (my ‘new’ job) to help share information both on methodologies and findings from systematic reviews (my ‘old’ job). The wikis are a great start, and I very much hope more medical librarians get involved with both of them. I’d like to see a more active online community of people working in evidence based health care, as well as placing findings more aggressively in social media spaces. Again, others thought of this first. 🙂

Cochrane Collaboration:
– Facebook: http://www.facebook.com/group.php?gid=63721740498
– Twitter: http://twitter.com/CochraneCollab

You can find out more about Cochrane’s social media presence and strategy in these slides from Chris Mavergame at Cochrane.

I’m actually writing this blogpost as a sequel to a guest lecture I recently gave for Tiffany Veinot’s class on medical librarianship at the University of Michigan School of Information. To put my money (time) where my mouth is, I’d like to share those slides also. Creative Commons 3.0 licensing applies, so feel free to download and share.