“Kitemarking” Quality & Credibility in Healthcare?

Good Housekeeping, 1950's Good Housekeeping, 2010
Good Housekeeping, 2010 Good Housekeeping, 2010


This morning, catching up on my Linked In groups, I spotted this article.

Miller, Jennifer E. Bioethical accreditation or rating needed to restore trust in pharma. Nature Medicine 19, 261 (2013) doi:10.1038/nm0313-261 http://www.nature.com/nm/journal/v19/n3/full/nm0313-261.html

It’s a good article, worth a read, and well intentioned, as is the case with most proposals of rating and ranking systems.

I’ve encountered many kitemarking or accreditation systems (“Good Housekeeping Seal of Approval” type of systems) in the area of online health information. I’ve been watching how they’ve worked in the online healthcare environment literally since the birth of the Web. In my opinion, there is not one I trust. Why? Many reasons.

1) They are typically extremely expensive projects to run in terms of time and costs of analysis.

2) Because of the expense, many marks of approval end up charging for their review time. This backfires because it excludes small groups without a lot of money (this might not apply in pharma.) It also gives the impression that the “mark” or seal of approval has been purchased, thus devaluing its credibility.

3) They are slow. The initial review is slow and time consuming. Then they are slow to respond to changes in the organization or their information. Better Business Bureau reviews of charitable organizations has a good way of handling this, with expiration dates, and transparency about when their reviews were run, with the date of approval, and date the company should be re-examined.

4) These programs are often biased, or ill managed. Almost always they are one of these, if not both. Ill managed usually means a volunteer organization that is understaffed or underfunded. They mean well, they try hard, but it is too big a project.

5) Policing is a problem. Some organizations (especially snake oil salesmen) “steal” the mark and place it on their site without the right to do so, and without following appropriate guidelines for the mark’s use.

I don’t name names in public on these issues, because some of the most offensive stories I know of kitemarking abuse in healthcare involve individuals or organizations that are highly litigious, and sadly also highly trusted. They have lots of money or lots of lawyers or both. I don’t. So for years, even decades in some cases, I sit back, silently simmering over trusted sources of online health information that were trusted mostly because they said you should trust them, loudly and frequently. What I usually end up saying in public is, “Who watches the watchers?”, coupled with a recommendation to ALWAYS question healthcare information, both the information and who says it is good or bad, even if it is peer reviewed. You’ve probably already seen RetractionWatch, and are aware of the increasing rate of retractions in science, which is only one indicator that the current peer review system isn’t accomplishing quite what it was intended to do.

What frustrates me, having seen these systems either fail or mislead over and over again, is the conversations still happening NOW, with good-hearted, well-intentioned people proposing these are needed. I’ve been in several of these conversations lately. Sometimes, people didn’t know any kitemarking systems already existed. Sometimes they knew, but were aware the old systems had failed and thought they could make a new one that works. There is a lot of the “If we build it, they will come” mentality that surrounds these. That has NOT worked so far. The librarians, bless their souls, have tried hard to adopt and embrace quality indicators for online health information. They’ve tried to spread the word and inform the public. Librarians have tried to build their own guidelines for vetting online health information. Enormous manhours have gone into developing tools to help with this. Many of them are excellent, but … now well known. And regarding the semi-official kitemarking system, I’ve found even librarians to be mislead by some of these. I’ve had a number of quiet, offline, private conversations when I was particularly deeply worried by enthusiastic endorsement of some of these.

So, what is it that made the Good Housekeeping Seal of Approval the standard to which all other quality ranking / rating systems aspire? The accountability. Good Housekeeping, to this day, says that if a you buy a product they’ve approved and it doesn’t meet the standards, THEY WILL REFUND YOUR MONEY.

Good Housekeeping Seal List: http://www.goodhousekeeping.com/product-reviews/seal/

About the Good Housekeeping Seal: http://www.goodhousekeeping.com/product-reviews/history/about-good-housekeeping-seal

Oh. Now, how would that fly in healthcare? Obviously, it can’t. You can’t refund someone’s limb, or pain, or life. How would it fly with information? Again, it can’t. It isn’t a standardized product with measurable units and production. You can’t guarantee that the person consuming the information digests it in the same way as the next person. There is no typical information physiology. Information consumption is too subtle, too personal, too individual of an experience, too filtered by local circumstance and prior experience and expectation.

Frankly, kitemarking without accountability is rather meaningless. If we can’t have accountability behind systems that endorse online health information, what you have is simply opinion. Some opinions are worth more than others. Some people like my opinions, but I make mistakes, too, and would not want to be held accountable legally for some quick blogpost I whipped up in a few minutes or a couple hours. I want people to question what they read, NO MATTER WHO SAID IT OR WHERE. Even if it’s me. And I want people to help each other, to talk with others about the things that matter, to build understanding as a community. I don’t want to put this process in the hands of some official organization. Not for health information. Build communities, not rubber stamps.

Good Housekeeping Seal of Approval, 1930's



2 responses to ““Kitemarking” Quality & Credibility in Healthcare?

  1. Thank you!
    I enjoy reading your blogs!


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s