Informed Consent in a New Era

Informed Consent copy

I’m a big fan of John Wilbanks’ work in the area of open personal health data and informed consent, and have blogged about that here before. Briefly, my awareness of John’s work began with “We Consent” which has now transformed into Sage’s “Participant Centered Consent Toolkit.”

Cool Toys Pic of the day - We Consent
Sage: Participant-Centered Consent Toolkit (E-Consent)

Recently someone asked me a question about “online informed consent.” I think they were remembering my having mentioned John Wilkin’s stuff, a.k.a “portable legal consent” or “portable informed consent.” These and “online informed consent” are … related concepts, but perhaps not as closely related as some might think. Just to complicate matters, people are also using jargon like “dynamic consent” and “broad consent” to mean things related to both of these, but which are not quite the same. There are also people trying to get the phrase “informed consent” converted to “educated consent” as possibly being more meaningful. In this post, I will try to sort some of this out, but I’m no kind of expert in consent, and this is complicated, really REALLY complicated.

First, the short-short explanation. Portable informed consent (PIC) usually is part of online informed consent, but online informed consent (OIC) is rarely portable. Riiiight. OK, a step backwards.

PORTABLE CONSENT

The idea of portable informed consent is (in my mind, at least) analogous to Creative Commons licensing for your own creative works, except that it applies to your own health data. Actually, the idea of this really came from people wanting to share genomic data. You walk through an online informed consent process, agree to which version of a license you are comfortable with, and then when you share your data in a secure repository, that license or consent agreement is attached. People who want to use your data, must agree to follow those predetermined restrictions. Researchers who don’t agree, aren’t allowed to see your data, only data from other folk who agree to whatever guidelines they need for their project. Researchers who don’t follow the rules will be denied access to all of the data.

Personal Genomics

Genomics is basically mapping the genome. Personal genomics is doing this for a person in particular, rather than a species or condition or other collective group. Some people get involved in exploring personal genomics because of simple curiosity, but many are driven by long standing medical challenges without any easily identifiable solution. Some people are terrified at the idea of what they might find out. Others are concerned that the data will result in problems with jobs or insurance. Those urgently seeking help for health problems often want to share and find others who might have insights into their problem. OpenSNP and the Personal Genome Project are two examples of places where people share their genomic data. By making their data public and consenting to its use by researchers, they are hoping to support solutions not only for themselves but for others like them. Making sure that consent is LEGAL is essential for supporting future research. One great example of this is Jay Lake, who contributed his whole DNA sequencing data and that of his tumor, making possible research on new treatments that came too late for him. It’s a powerful story.

ONLINE INFORMED CONSENT

Online informed consent is a great deal simpler, in that it mostly takes the usual informed consent process (reading forms, signing forms, filing forms) and puts it all into an online web-based interface in a secure system. But, PIC gets more buzz in the popular press and media, while OIC gets more attention from within the hallways of day-to-day research communities. PIC grew out of work with personal genomics and is designed to make data sharing simpler, research more open, and problem solving more dynamic, all while still being responsive to issues of privacy and ethics. OIC is a tool designed to make the IRB management simpler for researchers.

DYNAMIC CONSENT

Dynamic consent is closer to portable consent, but grew more out of tissue and biobanking contexts, rather than data or genomics. Dynamic consent has a lot of nitpicky little options, and allows you to change your mind over time. That’s why it’s dynamic — things keep changing. Right now, dynamic consent is used primarily for what happens to parts of your body that are removed from your body while you are alive, and used for various medical purposes. Sometimes those purposes involved throwing what wasn’t used in the nearest incinerator, but sometimes there is something interesting and the doctors or researchers want to keep a sample for future use.

Biobanking

Now, remember, I’m drastically oversimplifying here. There are many more situations and options that come into play. Healthcare researchers have come to realize that we often don’t know where the next interesting possibility will come from, which is part of why biobanking is becoming more important. A biobank is sort of a library of tissues (meaning parts of human or animals or plants). Biobanks are often focused on a certain type of tissue or condition. Many biobanks collect tissues for a particular kind of cancer, or conditions like Parkinson’s, Alzheimer, autism, etc. Others may focus on a particular organ, like brains, breast tissue, lungs, or genome. In book and journal libraries, the librarians have traditionally spent a lot of time trying to select just the most important material on their special topics, but over generations, we’ve found the most desired content is as often as not the parts that were considered cheap and unimportant at the time, which are now expensive and hard to find, because no one kept them. Some of the same issues are coming up with biobanking, but complicated by the challenge of each and every sample being unique (although there might be copies of cell lines). At least with books, if one library lost theirs, another library might have a copy. Part of the idea of all these different kinds of consent is to try to maximise the number and diversity of samples that can be preserved and made accessible to future researchers.

PRESUMED CONSENT

Presumed consent also related to tissues, actually organ donation, but after you are no longer alive or aware enough to give or change your consent. Where I live, you have to register as an organ donor. If you don’t, and are in a fatal accident, no one is allowed to use your organs as transplants to save the lives of other folk who need new organs to survive. That isn’t how it works in all countries, though. In some countries they have “presumed consent,” where the assumption is that organ donation is fine with you as long as you don’t say NO beforehand. So, opt-in vs. opt-out. That’s the main difference. Sounds simple, doesn’t it? But people have incredibly strong feelings about both of these options.

BROAD CONSENT

Broad consent is probably the messiest of all of these. Just look at these article titles!

Can Broad Consent be Informed Consent?

Broad consent is informed consent

Broad consent versus dynamic consent in biobank research: Is passive participation an ethical problem?

Broad Consent Versus Dynamic Consent: Pros And Cons For Biobankers To Consider

Broad Consent in Biobanking: Reflections on Seemingly Insurmountable Dilemmas

Should donors be allowed to give broad consent to future biobank research?

You can just feel the tensions rising as you read through the list. It is obvious that this is not an area of consensus. And what can it possibly mean to consent mean when there isn’t an agreement about what consent is?

“Broad consents are not open nor are blanket consents. To give a broad consent means consenting to a framework for future research of certain types.” Steinsbekk KS, Myskja BK, Solberg B. Broad consent versus dynamic consent in biobank research: Is passive participation an ethical problem? European Journal of Human Genetics (2013) 21:897–902.

Broad consent attempts to make a best guess of what might be needed by the researcher of the future, and to try to get the individual to agree to a flexible use and reuse of tissues, samples, or data. As you can tell from the titles above, “broad consent” tends to refer to tissues rather than data, but when you get down to brass tacks, all of these could theoretically apply to a wide variety of donated content.

CLOSING THOUGHTS

The idea behind all of these myriad forms of consent is knotted into the dynamic between the rights of the individual and the needs of the community. Without research, we stagnate and die, literally, since solutions cannot be discovered for the aches and pains and problems that lead to increased mortality and reduced longevity. As a community, as a species, we don’t make progress without sharing. At the same time, the goal is to reduce harm to individuals, and forcing people to ‘consent’ against their will causes harm. I’ve known people who practically had a nervous breakdown at the idea of becoming an organ donor, the idea of part of them living on in someone else distressed them that deeply. I know others who fear what could happen to them if their genetic data fell into the “wrong hands.” I’m not one of them. I’m a registered organ donor, and I donated my genomic data to OpenSNP. But I still respect the emotional pain that would be caused by forcing consent. It’s an ethical dilemma which our society is obviously still working to solve. While looking at background material for this post I stumbled across two phrases that seemed to express some of the challenges well: “From Informed Consent to No Consent?” “Open Consent for Closed Minds.”

“I’m proposing … that we reach into our bodies and we grab the genotype, and we reach into the medical system and we grab our records, and we use it to build something together.” “I hate [the] word ‘patient.’ I don’t like being patient when … health care is broken.” John Wilbanks

Personalized Medicine, Biosensors, Mobile Medical Apps, and More

At the Quantified Self Meetup, someone was praising the Rock Health slides. Of course, I had to go explore and see what was so great. These are my favorites.

About FDA’s Guidance for Mobile Medical Apps

FDA 101: A guide to the FDA for digital health entrepreneurs by @Rock_Health: http://www.slideshare.net/RockHealth/fda-101-a-guide-to-the-fda-for-digital-health-entrepreneurs

I especially took note of slide 10, where they describe things I would think of as an app, but which do not qualify as such for FDA regulation. This is an important distinction I hadn’t previously considered. Slide 12 takes it further by describing the categories of regulation as based on risk to patients, with good clear examples. Slie 21 on “pro tips” would have really benefitted companies like 23andMe (even though that isn’t actually a mobile medical app, the pro tips still apply, and in spades).

Biosensing Wearable Tech

The Future of Biosensing Wearables by @Rock_Health http://www.slideshare.net/RockHealth/the-future-of-biosensing-wearables-by-rockhealth

This one definitely gets into topics relevant to the quantified self movement and self-tracking. Slide six emphasizes the shift from the low hanging fruit (fitness, pulse, sleep) to the long tail — more targeted solutions for specific challenges (hydration, glucose, salinity, skin conductance, posture, oxygenation, heart rhythm, respiration, eyetracking, brain activity, etc.). That’s really quite interesting, and it gives examples of companies working in each space.

Slides 19-24 get into several of the areas our own local meetup defined as challenges to success for companies working in this space and for the future success of the entire area — it has to work, easily, and dependably. Slides 27-30 extrapolate these challenges into the transition into healthcare environments.

Personalized Medicine

The Future of Personalized Health Care: Predictive Analytics by @Rock_Health http://www.slideshare.net/RockHealth/the-future-of-personalized-health-care-predictive-analytics-press Video https://www.youtube.com/watch?v=UJak41hIDWc

SLIDES

VIDEO

It’s probably safe to say that most individuals working in the quantified self / self-tracking space eventually end up struggling with the issue of how to use their data to anticipate avoidable problems. This idea can be translated into the jargon phrase of “predictive analytics.” Slide 11 does a nice job of lining this up with how traditional healthcare is practiced, which is very useful. Slide 12 places this in the context of big data resources, databases, and tools, listing several of the main players. This context is essential for making personal data relevant beyond the drawn out process of n=1 studies. Slide 14 identifies the BIG problem of how companies working in this space largely focus on hospitals and health care providers, and seem to have entirely missed the idea that patients are deeply and actively engaged in this space. And, frankly, there are more of us than them (even if our pockets aren’t as deep). I love the phrase on slide 18, “Symptom calculators are the “recommendation engines” of health care.” Most of the rest of the deck identifies challenges and opportunities, which I hope any entrepreneurial types would examine closely. Do notice that there is a video with this one. You can hear the entire webinar as well as reviewing the slides.

Quantified Self Meetup, Ann Arbor

Cool Toys, Devices, Quantified Self

Last week, I felt really lucky that I was able to make it to the first Quantified Self Meetup of the New Year (thanks to Nancy Gilby for the ride!). This session was held at the UMSI Entrepreneurship Center. Roughly ten folk came, and I’m not sharing names even though they said I could because I’m not sure I got the names down right. The group included a wide range of types of people: corporate folk, students, entrepreneurs, faculty, alumni, and independents. The conversation was fast, dynamic, and overlapping, so I couldn’t catch everything. I will talk about what I did catch of the IDEAS and the GADGETS. That’s what’s really fun, eh?

INTERESTS

What the Meetup group page SAYS they are interested in (as a sampling) is pretty extensive.

“Aging in Place Technology • Behavior change and monitoring • Caregiving of digital patients • Chemical Body Load Counts • Citizen science• Digitizing Body Info • Medical Self-Diagnostics • Lifelogging• Location tracking • Non-invasive Probes• Mindfulness and wisdom tracking • Parenting through monitoring/ tracking • Personal Genome Sequencing • Psychological Self-Assessments • Risks/Legal Rights/Duties • Self Experimentation • Sharing Health Records • Wearable Sensemaking”

What’s even more interesting is what people said they were interested in as they went around the table.

  • aging population
  • big data
  • biohacking
  • data visualization
  • diabetes
  • epigenetics
  • fitness
  • geofencing
  • legal advice
  • patient communities
  • personal genomics
  • sleep tracking
  • telehealth

The “legal advice” bit? That was from someone planning a wearable tech start up. They got some interesting answers on that point: Scott Olson, of UM’s Pediatric Device Consortium; SPARK; Medical Innovation Center, Fast Forward Medical Innovation, and (depending on your UM affiliation) possibly the Student Legal Services, UM’s Startup Law Clinic (Twitter), Zell Lurie Institute.

For the personal genomics, it was a great surprise to me to meet another person who knows their MTHFR status (and who also has two defective copies of the gene, AND is working on problem solving as hard as I am)! We were swapping info, apps, diet tips and tricks, formulations of supplements, and more. There just wasn’t enough time to dig as deeply into this as I wished. I did get to do my now normal rant, “23andMe was NOT killed off!”

ISSUES

After introductions, we just had an open conversation, much of which touched on challenges in quantified self tools. This was what had the meeting stretching WAY past the planned time!

  • QS devices are not being designed for longevity, but for rapid failure
  • QS devices are not being designed to actually work, by and large, which is frustrating to folk buying them early, and an argument for doing QS with low-tech self-hacked solutions
  • to integrate into personal healthcare solutions, there is a need for calibration with official medical devices
  • how are data measurements defined? it. “sleep” cycles based on movement, rather than REM cycles.
  • desperate need for standards of measurement, to empower folk wanting to discover trends and patterns across tools, data sources, and apps
  • who is funding these?
  • data visualization for self-discovery; “correlation” vs aggregator apps; challenges of meaningful analysis
  • HIPAA and QS: patient self-reporting data as an FDA loophole; PHI – Personal Health Information (personal sharing loophole)
  • requirements for insurance coverage – need doctor’s prescription for some very useful medical devices; reimbursement codes can be tricky
  • reverse innovation
  • risk science, risk of failure, costs of failure
  • when designing a device, think about how will it fail?
    design for how to make it work or how to make it fail?
  • how can small companies compete? “innovative/unique, protected, acquired”
  • security, open data, hack into someone else’s data, ownership of data

Any one of these could easily be a devoted session, presentation, or series of blogposts. The bit about failure especially interested me. The idea was that these devices seem to be being designed to fail, as is pretty standard for tech in general these days. But what happens to the end user if they get to the point where they trust the wearable tech device, trust its data, and can’t tell that it has stopped working properly or is on the verge of failure? The FDA keeps tabs on what happens with medical device failures in their MAUDE database. The problem is that this only applies to devices that go through FDA approval, and most of the wearable tech devices folk use for biohacking or self-tracking personal health information, well, they are not FDA approved. People were talking about how much risk is there, impacts, and devices that are low risk. I shared a story of a time when a blood pressure cuff lead to a fatality some decades ago. That was pretty shocking to them, because we tend to think of blood pressure cuffs as being pretty innocuous. How did it happen? It failed during surgery, and kept giving normal readings when the patient was actually having trouble. The idea was that even simple tech can have serious impacts when the stakes are high and people are depending on it.

DEVICES, SERVICES, APPS, & MORE

Of course, we all had to talk about our toys, how we like them or don’t, what we’d change, what we’re thinking about buying, our experiences with customer service from the different companies, companies that are failing or expanding, new releases, etc. I tried to keep a list of devices mentioned or waved around (not all of which were pertinent to QS), but I’m pretty sure I missed a few. The same is true of services, apps, and such, but I’ll give links for the ones I caught.

DEVICES

While most of the gadgets mentioned were in the room and functional, that wasn’t true across the board. Some of these were mentioned as warnings (“a glorified pedometer” “gave me headaches” “out of business”), so please don’t take this list as an endorsement.

SERVICES

I know there was another few genetic analysis tools mentioned that I can’t remember, and I’m really frustrated that I can’t remember. Later, trying to prod my memory, I found this great list (“What else can I do with my DNA test results?“) but I’m still hoping that the person who mentioned the other tools will comment on this post with what I missed.

APPS / SOFTWARE

The apps here include tools for mobile and desktop, for data analysis, self-tracking, behavior modification, communities, and time management / lifehacking. What isn’t included is the conversation about low-tech alternatives, such as replacing calorie counting apps with photos of what you ate, or using notebooks instead of tracking apps. Quantified self doesn’t have to take a lot of money and gadgets (but perhaps that should be a separate post).

RESOURCES

Please note that this is NOT a collection of the best ever anywhere resources on Quantified Self, but rather (as with all the other lists in this post) a collection of what was mentioned during the meeting.

Last but not least, I collected a whole bunch of links I stumbled on during the meeting in one large “OneTab” collection. It includes 76 web pages that I wanted to come back to, reflecting more details or random conversation digressions. You can find it here: http://www.one-tab.com/page/EKdC99v0Q2-nZYfOm41lOw.

Big Beautiful Questions (A HOTW post from #hcldr)

Guy with questions 8

The other blog for which I was writing the “Hashtag of the Week (HOTW)” posts has changed focus, so I am no longer doing them weekly, but I am still doing them when available time and something amazing both intersect. The something amazing part happens ALL THE TIME, and if that was the only factor, I could do these daily! But this time, the conversation was so relevant and useful that I would feel like I wasn’t doing my job if I didn’t share it.

Yesterday evening, the Healthcare Leadership group had a conversation about the role of questions and questioning in healthcare. The conversation was lead by Bernadette Keefe, MD, and was triggered by Warren Berger’s work in the area of “beautiful questions.” He wrote a book, but you can find a short intro to the core ideas in his New Year’s article, “Forget Resolutions.” To help people ask better questions, more answerable questions, questions that have a higher potential for leading to positive change in their life, Bernadette pointed out the tips from the “Right Question Institute“, and I pointed out the “Question Prompts List” strategy.

Right Question Institute Question Prompt Lists

The real value of the #hcldr conversation, however, came from the questions. The questions posed for the group, and the questions posted as answers. My favorite of the questions posed was, “What are we not asking?” Keep that in mind as you read the following selections from the questions given as answers to the prompts.


T1 In our sizable efforts to make healthcare more efficient, accurate and safe,as well as less costly, what are the questions weʼre not asking?


T2 As you experience healthcare delivery today – is questioning valued?


T3 What are questions you, personally, would like to ask of your healthcare provider, medical insurance company, or hospital?


T4 How could the value of questioning be incorporated into healthcare delivery in an efficient and effective way? Programs etc?


Closing Thoughts

Fair Use & Figures: When Is It OK? (Part Two: Fair Use)

Fair Use?

Point 2: Is it fair use?

In part one of this post on using research tables and figures in social media, the focus was on, “Let’s find something that isn’t copyrighted, and just use that, because we know that will be safe” (sort of). But sometimes what you want is new, not old, and from a copyrighted publication, not something open access. When that happens, what do you do? Can you still use it?

It started with Andrew’s question, but he sure isn’t the only person asking!

The answer is (again) it depends. And, to be honest about it, the “it depends” is a whole lot messier. The idea of “fair use” makes the idea of “copyright” look like child’s play. As usual, I am not a lawyer (IANAL), so take the information here at your own peril, and don’t go quoting me to your own lawyers. That said, I’ll be giving references to a lot of information that IS from lawyers, so refer people to the original sources whenever possible!

This is a question that comes up a lot. Even more often, I suspect, it doesn’t come up at all when it ought to, because someone assumed it was alright and didn’t think about it more deeply. I may have even done that myself (and I’m afraid to go back and look too closely at my blogposts).

WHY ARE FIGURES DIFFERENT?

The shortest version I’ve found explaining why “fair use” for research figures from articles are DIFFERENT from other images is this snippet from Scholarly Publishing @ MIT Libraries:

“Note: an image or figure would commonly be considered a work in and of itself, weighing against fair use; or could summarize the key point of an article, also weighing against fair use.” Reuse of figures, images, and other content in theses http://libraries.mit.edu/scholarly/publishing/copyright-publishing-guide-for-students/reuse-of-figures-images-and-other-content-in-theses/

That’s the gist of it. Applying “fair use” to reusing content from a larger work depends on four factors:

1. The purpose and character of the use, including whether such use is of commercial nature or is for nonprofit educational purposes
2. The nature of the copyrighted work
3. The amount and substantiality of the portion used in relation to the copyrighted work as a whole
4. The effect of the use upon the potential market for, or value of, the copyrighted work
US Copyright Office: Copyright: Fair Use: http://www.copyright.gov/fls/fl102.html

The point from MIT regards the 3rd factor, “amount and substantiality.” If the image is considered a work independent of the article, then copying it is copying a complete work instead of a percentage, and that is not OK. If the figure is considered a fair distillation of the complete article, then that could be considered equivalent to copying the whole article. To be allowed to use something under “Fair Use” claims, it is recommended to meet all FOUR of the four factors: no money, factual or data-focused content, a tiny bit of it (insignificant amount), and little to no impact on sales of the original. If any of those are iffy, you’ve got a problem.

In a recently published article in the research journal CIRCULATION, the editors of the journal attempted a randomized controlled trial to test the success or failure of social media in promoting readership of articles. How did they do this? Well, they posted randomly selected articles to Facebook and Twitter, but a bit part of how they posted included (you guessed it!) FIGURES. For exactly the same reasons we are NOT supposed to use them, technically speaking.

“However, our social media postings were comprehensive in that they focused on the main message of the article and included a key figure from the article. Thus, it is possible that social media users did not find it necessary to access the full article and therefore experienced increased awareness of the article but not online access of the primary source. This raises the possible concern that social media could reduce the potential reach of original published research as demonstrated by altmetrics.” Fox CS, Bonaca MA, Ryan JJ, Massaro JM, Barry K, Loscalzo J. A Randomized Trial of Social Media From Circulation. Circulation 2015; 131: 28-33. http://circ.ahajournals.org/content/131/1/28.full

In that test, it was the editors of the journal posting the figures, and even then, they questioned the utility and benefit. A big part of the question, which they highlight in that snippet, was whether it is better to have awareness of the article and its findings, or actual readers. This assumes that people who click through to the article actually read it, which is itself questionable.

So, that’s the short view. Just to round it out, here are a few more pieces relating to this topic.

WHAT IF I DRAW MY OWN FIGURE?

A common recommendation is to redraw the figure. Annoying, time-consuming, but supposedly legal, most of the time, as long as the content is primarily data (since data can’t be copyrighted). Indeed, this is recommended by many experts as THE solution to this problem.

Q: I want to use a figure from another thesis or dissertation from my group. Do I need to ask permission?
A: “Usually. The student who wrote the thesis or dissertation owns the copyright and must be asked for permission. Figures are generally considered works in and of themselves and do not usually constitute a small portion of the work. See “How to Use Copyrighted Materials” for more information. If, however, the figure is a simple representation of data, you may not need permission. Data cannot be copyrighted, so non-creative ways of representing the data are generally considered fair use.” Copyright Frequently Asked Questions | Michigan Tech Graduate School http://www.mtu.edu/gradschool/administration/academics/thesis-dissertation/copyright/faq/

It depends on who you ask, though. I don’t know the court law on this, but I’ve found recommendations on both sides. Overwhelmingly, what I’ve found it people recommending to redraw the figure. And then I found this, from the Association for Computing Machinery (ACM).

“Redrawing a figure does not change the Fair Use analysis; the figure cannot be used without permission just because it is redrawn.” ACM Guidance for Authors on Fair Use http://www.acm.org/publications/guidance-for-authors-on-fair-use

WHAT IF I JUST GO AHEAD AND USE IT? NO ONE WILL CARE, RIGHT?

Another common approach (probably the one I am most guilty of myself) is to assume that you aren’t important enough to bother with, or that no one will notice you did it, or that if they do notice all that will happen is they ask you to take it out. I haven’t even attempted to figure out how often this happens. The gist of the idea is that, while it may not be strictly legal, it IS fairly common practice, and can be a win-win for each side (as long as you are saying nice things, anyway).

The “Win-Win” Argument:

“Most authors will probably be happy that their results are disseminated, and reuse is likely to lead to more people reading the full paper and citing the work.”
Martin Fenner. Why can’t I reuse these tables and figures? Gobbledygook (PLoS Blogs) 2010.
http://blogs.plos.org/mfenner/2010/09/30/why-cant-i-reuse-these-tables-and-figures/

The “Standard Practice” Argument:

“Posting figures from papers “was something we all did, all the science blogs, and I had been doing it since I started the blog. I always thought I was doing a public service, I wasn’t plagiarizing or claiming it was my own data.”” Andrea Gawrylewski. For blogger: A threat, then an apology. The Scientist May 2, 2007. http://www.the-scientist.com/?articles.view/articleNo/25066/title/For-blogger–A-threat–then-an-apology/

So, it’s fine, right? Uh, not so much. If you are doing this on a work related blog or for your organization, it is awfully risky, since they can be held liable, which is a real can of worms. If you are only doing this on your personal blog, and it is made clear that these are your words and actions and not to reflect on your employer, … it’s your decision.

WHAT IF THEY PULL OUT THE BIG GUNS?

There are publishers who have (or will) pull out the big guns. Yes, usually, even if they notice, they’ll just ask you to take it down, to “cease and desist,” but there is no guarantee. The best known case in science blogging of use of a figure gone wrong actually centered around a PhD student here at the University of Michigan, Shelley Batts, back in 2007.

“When Shelley Batts wrote up a report on an article about antioxidants in fruits, she never expected to get contacted by the copyright police, but that’s exactly what happened.” Munger, Dave. Is reprinting a figure fair use? http://scienceblogs.com/cognitivedaily/2007/04/26/is-reprinting-a-figure-fair-us/

In Shelley’s words:

“In short, I was threatened with legal action if I didn’t take it down immediately. I used a panel a figure, and a chart, from over 10+ figures in the paper. I cited and reported everything straight forwardly. I would think they’d be happy to get the press. But alas, no.” Batts, Shelley. When Fair Use Isn’t Fair. http://scienceblogs.com/retrospectacle/2007/04/25/when-fair-use-isnt-fair-1/

The blogosphere went WILD. Some of the issues brought up included that this was government funded research paid for by taxes; the “amount used” argument (since it was such a small fraction, not even a complete image); the stifling impact on science discourse; the chilling of public awareness and policy discussions; and heavily, the critical distinction between “fair use” and being granted permission.

“First, the blogger Batts did not go through the usual process to request permissions to use published material in advance. It is not clear that had this been done that she would have been refused and indeed permission may eventually be extended. … Of course, once the appropriately senior person at Wiley was involved, the situation was resolved. This is not a “win”. This is a “loss” …” DrugMonkey. “This figure is reproduced with permission of the publisher.” https://drugmonkey.wordpress.com/2007/04/26/this-figure-is-reproduced-with-permission-of-the-publisher/

It is easy to find posts about this, but the one that seems to have really tipped the balance was when BoingBoing stepped in.

“This is, of course, bullshit. Reproducing part of a figure in a critical, scholarly essay is so obviously fair use that it hardly bears discussion. Wiley’s lawyers know this. You and I know it too.” Doctorow, Cory. Wiley threatens scientists with copyright law – UPDATED. http://boingboing.net/2007/04/26/wiley-threatens-scie.html

Eventually, it all calmed down, when Shelley was “granted permission” to use the image by the publisher. Many expressed concern that this ended up muddying the waters, that this was and should have been respected as fair use, that the publisher should have acknowledged this was fair use, and that granting permission implies that permission was theirs to grant (permission that is irrelevant in the case of fair use). Shelley was happy – she wasn’t getting sued, and her original blogpost was able to stay (although I can’t find it now, and I don’t know why).

“I was so surprised that anyone would think I was doing science a disservice. Science blogs bring pedantic ‘ivory tower’ knowledge to a completely new audience that would probably never hear about it otherwise. But in the end, I’m glad it happened — and that the entire blogosphere howled.”
Moments in Medicine at Michigan, Summer 2007. http://medicineatmichigan.org/magazine/2007/summer/moments

But the University of Michigan lawyers stated that she had done nothing wrong to begin with. So. Clear as mud, eh?

“Maybe if we weren’t so worried about copyright, we’d be able to report on more research.” Dave Munger, op cit.

Systematic Reviews 101

Systematic!!!

This morning in the Emergent Research Series, my colleagues Whitney Townsend and Mark MacEachern presented to a mix of mostly faculty and other librarians about how medical librarians use the systematic review methodology. They did a brilliant job! Very nicely structured, great sources and examples, excellent Q&A session afterwards. They had planned for some activities, but it ended up there wasn’t time. I’d like to know more about what they had planned!

I was one of the folk livetweeting. According to my Twitter metrics, this was a popular topic. I assembled a Storify from the Tweets and related content. I thought it would be of interest to people here.

Storify: PF Anderson: Systematic Reviews 101: https://storify.com/pfanderson/systematic-reviews-101

On Validating Search Strategies

validation

This question came up because of this:

Varela-Lema L, Punal-Riobóo J, Acción BC, Ruano-Ravina A, García ML. Making processes reliable: a validated pubmed search strategy for identifying new or emerging technologies. Int J Technol Assess Health Care. 2012 Oct;28(4):452-9. http://www.ncbi.nlm.nih.gov/pubmed/22995101

What did they mean when they said “a validated PubMed search strategy”? Our MLA systematic review team that is working on search strategies for emerging technologies identification was, shall we say, curious. For this article, it meant that they tested the search results against the next best method previously used (handsearching). The topic was emerging technologies, and what they did was select influential journals and scanned the TOCs manually (which actually means by using their own eyeballs). The journals they scanned were: Science, JAMA, Lancet, Annals of Internal Medicine, Archives of Internal Medicine, BMJ, Annals of Surgery, Am J Transplantation, Endoscopy, J Neurology Neurosurgery Psych, Archives of Surgery, Annals of Surgical Oncology, British Journal of Surgery, and Am J Surg Path. Of the 35 articles that qualified from these journals, the search strategy accounted for 29. The ‘missing’ articles lacked appropriate title words relating to the novelty of the concept, OR used text words that had been removed from the search strategy to improve specificity (reduce total numbers retrieved).

Is that an appropriate way to validate a search strategy? Probably a pretty fair approach for this one, IMHO, especially since they did such a good job of reporting the specific calculations and details of the actual findings of the searches. Is that how most search strategies are validated? Well, perhaps not.

What I’ve been doing to validate search strategies for systematic reviews is to test and compare the search results to a defined set of sentinel articles. The sentinel articles are selected by the team’s subject experts as being good examples of articles that should be retrieved by a search on the defined question. The requirements beyond topic are that each of the sentinel articles should be older than two years, newer than 1990 (this can be flexible, depending on the topic), and must meet all of the defined inclusion criteria for the review. I usually recommend that the pool of selected sentinel articles include no fewer than 3 and no more than 10 citations. This is to make it possible to achieve complete success, as with each added citation, inclusion of all of them becomes more difficult. I also emphasize that the articles do not need to be excellent or required articles on the topic (ie. “gold standard” articles), but that it is, in my opinion, actually more effective for testing if the articles are a selection of relevant, but not necessarily the best ever written on the topic.

Draft versions of the search are tested against this set of articles, and if any “drop out” (are not included) we need to then figure out why, and determine whether to revise the search to include them, or justify the exclusion, or request NLM to correct the coding error in that article’s record. In these last two cases, the exclusion must be reported in the methods. Ideally, one would also describe the strengths, weaknesses, and/or limitations of the search strategy.

Here are some citations to other ways in which searches are validated.

Hausner E, Waffenschmidt S, Kaiser T, Simon M. Routine development of objectively derived search strategies. Systematic Reviews 2012 1:19. http://www.systematicreviewsjournal.com/content/1/1/19
NOTE: This is basically the same “sentinel articles” approach described above.

Hausner E, Guddat C, Hermanns T, Lampert U, Waffenschmidt S. Development of search strategies for systematic reviews: validation showed the noninferiority of the objective approach. J Clin Epid Feb 2015 68(2):191-199. http://www.sciencedirect.com/science/article/pii/S0895435614003874
NOTE: Interesting article tests the reproducibility of Cochrane reviews and their reported search strategies. The emphasis is on the need for objective and reproducible search strategies in systematic review publications.

Van Walraven C, Bennett C, Forster AJ. Derivation and validation of a MEDLINE search strategy for research studies that use administrative data. Health Serv Res. 2010 Dec;45(6 Pt 1):1836-45. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3026961/
NOTE: Compared to handsearching.

Walsh ES, Peterson JJ, Judkins DZ. Searching for disability in electronic databases of published literature. Disability and Health Journal Jan 2014 7(1):114-118. http://www.sciencedirect.com/science/article/pii/S1936657413001647
NOTE: Very interesting two part test to manage the quality control of the search strategy. First, they used the method described above (compared to sentinel articles), then, because the search excluded specific topic terms in favor of broad keyword searching, they validated by comparing retrieval to the results of a known topic search.

Hempel S, Rubenstein LV, Shanman RM, Foy R, Golder S, Danz M, Shekelle PG. Identifying quality improvement intervention publications–a comparison of electronic search strategies. Implement Sci. 2011 Aug 1;6:85. doi: 10.1186/1748-5908-6-85. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3170235/
NOTE: Compared relevance and quality of search strategies by results being reviewed for relevance by independent experts. My personal misgivings about this method for validating a search is that it cannot test for what is missed that you don’t know about.

Tanon AA, Champagne F, Contandriopoulos AP, Pomey MP, Vadeboncoeur A, Nguyen H. Patient safety and systematic reviews: finding papers indexed in MEDLINE, EMBASE and CINAHL. Qual Saf Health Care. 2010 Oct;19(5):452-61. http://www.ncbi.nlm.nih.gov/pubmed/20457733
NOTE: Compared sensitivity & specificity for new search strategies in comparison to previously published search strategies on the same topic. Validated by comparing to a large selection of sentinel articles. Very difficult to achieve, and am ambitious strategy!

Brown L, Carne A, Bywood P, McIntyre E, Damarell R, Lawrence M, Tieman J. Facilitating access to evidence: Primary Health Care Search Filter. Health Info Libr J. 2014 Dec;31(4):293-302. http://www.ncbi.nlm.nih.gov/pubmed/25411047
NOTE: Interesting strategy that first created and validated a search strategy in OVID for quality control over the search development process, and then converted the strategy to PUBMED and validated it again. The validation was again through the selection of a set of sentinel citations, but they explicitly selected for the best quality articles in the topic and referred to the set as the “gold standard.”

Damarell RA, Tieman JJ, Sladek RM. OvidSP Medline-to-PubMed search filter translation: a methodology for extending search filter range to include PubMed’s unique content. BMC Med Res Methodol. 2013 Jul 2;13:86. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3700762/
NOTE: Same strategy as the article by Brown, Carne…Tieman above, but a different topic.

You can find more articles on this topic by exploring the following search results:

(validated OR validation OR “quality control” OR “quality assessment”) search strategy review http://www.ncbi.nlm.nih.gov/pubmed/?term=(validated+OR+validation+OR+%22quality+control%22+OR+%22quality+assessment%22)+search+strategy+review