Systematic Reviews: Methodology, Overview, Sharing

In my previous job, a big part of what I did was systematic review searching. This is still going on as I am on a Cochrane team and a few grants. Technically, this isn’t probably emerging technologies. On the other hand I’ve found that with many “new” technologies it is all about who thinks it is new. For example, I’ve been teaching classes on using Delicious for at least five years, and there are still many people I work with now who have never heard of it. For them, Delicious, now approaching its 6th birthday, is an emerging technology.

Despite the popularity of systematic reviews in the medical literature, it is just now starting to really make the leap from being something a bit on the obscure side to something everyone has heard of. I think what is making the difference has been the push in many schools to include evidence based health care in the training of new clinicians, and to encourage graduate students to do systematic reviews as their research project.

My personal experience with systematic reviews goes back to roughly 1999 or 2000, when I was contracted by NIH to do systematic review searches for 13 of the teams preparing for the first Consensus Development Conference to require use of systematic review methodologies for all presenters.

NIH: Consensus Development Conference on Diagnosis and Management of Dental Caries Throughout Life (2001):

NIH: Consensus Development Conference on Diagnosis and Management of Dental Caries Throughout Life (2001) (University of Michigan, University Libraries):

I’ve lost track of how many systematic reviews I’ve worked on since that time, but it is a lot, ranging from graduate student projects to Cochrane teams, searching on topics with large research bases to those with almost no research. The techniques appropriate for searching in these various areas are very different. My particular area of expertise is searching in topics where there is a very small research base, meaning instead of seeking clinical significance the researchers are doing a preliminary project to define the levels and quality of evidence available on that topic. What I’ve noticed for many years now is that many of the articles published as systematic reviews do not actually follow an appropriate methodology for the topic being investigated. Another challenge is that many of the published systematic reviews do not include enough information about the search strategy for it to be replicated or verified by other research teams. I’m far from being the only person who has noticed this! Here is the most recent article I’ve seen examining the quality of published systematic reviews.

Song F, Loke YK, Walsh T, Glenny AM, Eastwood AJ, Altman DG. Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews. BMJ. 2009 Apr 3;338:b1147. doi: 10.1136/bmj.b1147. PMID: 19346285

The methodological “Bible” for doing a systematic review to determine clinical effectiveness is the Cochrane Handbook.

Cochrane Handbook: OR

Like the actual Bible, it takes a long time and a lot of work to actually read the Cochrane Handbook, making it impractical for many of the busy clinical researchers who are exploring the idea of publishing a systematic review. On many of the Cochrane systematic review teams the primary role of the librarian is to provide the data being analyzed. I have been finding that my experience having worked on so many teams and systematic review projects has placed me also in the role of providing assistance and insight to new systematic review team leaders with respect to the process and methodologies that should be followed. This actually makes a lot of sense, since both the statistician and the librarian on these projects are more likely to participate in large numbers of reviews, across a variety of topics. It also means that I have a large personal collection of search strategies I’ve developed for other reviews that can be quickly tapped or reworked, making the time required for the search process sometimes much faster.

Increasingly, when I work on a systematic review team I ask permission for the search strategy to be archived (post-publication of course) on a public web site to make it easier for other librarians to find and use. The NIH CDC on Caries was the first time we did that.

EBHC Strategies:

A more fully fleshed out wiki is here:
EBM Librarian:

EBHC is focused on creating a collection of search strategies to share
among other librarians so folks don’t need to reinvent the wheel; EBM
Librarian is more a tutorial, a detailed how-to as a collaboration of several of the most expert EBHC librarians and teachers.

I like the idea of both using social media and technologies (my ‘new’ job) to help share information both on methodologies and findings from systematic reviews (my ‘old’ job). The wikis are a great start, and I very much hope more medical librarians get involved with both of them. I’d like to see a more active online community of people working in evidence based health care, as well as placing findings more aggressively in social media spaces. Again, others thought of this first. 🙂

Cochrane Collaboration:
– Facebook:
– Twitter:

You can find out more about Cochrane’s social media presence and strategy in these slides from Chris Mavergame at Cochrane.

I’m actually writing this blogpost as a sequel to a guest lecture I recently gave for Tiffany Veinot’s class on medical librarianship at the University of Michigan School of Information. To put my money (time) where my mouth is, I’d like to share those slides also. Creative Commons 3.0 licensing applies, so feel free to download and share.

6 responses to “Systematic Reviews: Methodology, Overview, Sharing

  1. Thank you for these insights as I mainly do searches and haven’t participated in a full-fledged systematic review. The FRIAR/SECT approach is useful. I agree that the quality of most systematic reviews is wanting. My experience with literature searching is that when you seek to make an evidence-based recommendation, as you start excluding the studies that fail to meet the criteria, the sound evidence can be woefully limited. This is a real barrier to evidence-based practice.

    As you observe, web 2.0 technologies are not diffusing nearly as quickly as expected. I just used delicious to create and update a list of health literacy resources at work – an excellent way to bypass the bureaucracy of manual updating – and that seemed ingenious to my colleagues. The concept of RSS feeds remains obscure to most people and even when I have taught it, uptake has been limited. I continue setting up e-mail alerts because people fail to check the feeds.


  2. Pingback: Finding Credible Health Information Online: MedLibs Round 1.8 | Highlight HEALTH

  3. Hi there,

    I see that you have been involved in systematic reviews fora while now. I am a research librarian with ARCHE recently hired and working primarily on systematic reviews. i was looking around for some library discussions on the librarian role and searching and stumbled upon your blog. I was wondering if there is any consensus out there as whether or not to include brand names in search strategies for systematic reviews that involve drugs?


    • I don’t know about a consensus, but I am happy to share with you what I do. It depends on the size and history of the research domain. By this I mean when you search the topic, does it retrieve a large number of results? Has the drug been in use for a long time? Is the drug in widespread use? If the answer to all three of those questions is yes, I will tend to not be as aggressive about using freetext search terms to expand the MeSH terms. Also, it would depend on if the drug was a major concept in the search, a core central topic, or more peripheral. If the answer to all of those questions is no — the drug is new, not used extensively, and either has no MeSH term or doesn’t retrieve much with the MeSH term — then absolutely I would work hard to find equivalent terms, including brand names and drug family or related drugs. I would also want to know with what drugs it is often compared, and then search for comparative studies for that drug as well, at least within the timeframe that the target drug existed. I have an example of a search I did on articaine that might be an example? It isn’t in press yet but has been submitted.


  4. Pingback: 100,000? 100,000! | Emerging Technologies Librarian

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s