Tell the FDA the whole story, please

Susannah fox

I scan menus for keywords (fig, parsnips, salmon…) and it turns out I scan Twitter the same way, looking for anyone who is talking about my favorite topics (data, consumers, information quality…)

So when I saw Jonathan Richman‘s tweet the other night, I couldn’t resist it:

Anyone ever seen data on the overall accuracy of medical information found online? Need help for some final stats for #fdasm

Short answer: No. Long answer:

The Pew Research Center’s Internet & American Life Project has been reporting on the social impact of the internet since 2000, when “information quality” on health websites was a big part of the conversation. It was the era of wagging fingers, scolding patients for straying too far outside their boundaries, and Pew Internet data was ammunition.

We released our first report about the internet’s impact on health & health care in November 2000. The Medical Library Association (MLA) contacted us, asking for research looking at how consumers decide which sites/sources to trust. With their help we created a set of questions asking first if respondents went online for health info, then asking if they look for the source and date of the info they find (the two key quality indicators according to the MLA).

Our 2002 report, Vital Decisions, reported the findings:

Only about one quarter of health seekers follow the recommended protocol on thoroughly checking the source and timeliness of information and are vigilant about verifying a site’s information every time they search for health information.  Another quarter of health seekers check a site’s information “most of the time.”  Half of all health seekers search for medical advice and “only sometimes,” “hardly ever,” or “never” check the source or date of the information they read online.

We repeated the source/date questions in August 2006 and reported on the “demand side” of health information in our report, Online Health Search 2006:

Three-quarters of health seekers do not consistently check the source and date of the health information they find online.

Unbeknownst to us, the U.S. Department of Health and Human Services was in the field with their study about the “supply side” of health info: Estimating the Proportion of Health Related Websites Disclosing Information That Can Be Used to Assess Their Quality.

I summarized their findings in our report, hoping to soften the impact of the ammunition we were handing out:

Health seekers might be forgiven if they give up what at times is a search for a needle in a haystack. A recent study commissioned by the U.S. Department of Health and Human Services (HHS) finds that a tiny percentage of health sites display the source and date of the information on their pages.

After reading the HHS study, looking at our data, and scanning the changing landscape (ie, consumers diversifying their online research to include user-generated content), I decided that trying to measure the quality of health info had run its course as a research question.

Pew Internet was also gathering evidence that consumers were not being harmed by inaccurate information. In a 2008 survey we conducted in partnership with the California HealthCare Foundation, just 3% of e-patients said they or someone they know has been harmed by following medical advice or health information found on the internet, a number that has remained stable since 2006. Meantime, 60% of e-patients (or 42% of all adults) said they or someone they know has been helped by following medical advice or health information found on the internet. That’s an increase from 2006 when 31% of e-patients (25% of all adults) said that.

After exchanging a few tweets and then sending a long email, I was pleased to see our data used in Jonathan’s upcoming presentation to the U.S. Food and Drug Administration‘s hearing on social media. But I worry that only the “three-quarters don’t check” data made it in to his slides, not the needle-in-a-haystack caveat provided by the HHS study.

Are we really going back to the era of scolding patients for not checking the source and date of health information they find online? If so, it’s my duty as the purveyor of that ammunition data point to speak up.

To everyone who is providing insights to the FDA: Tell the whole story. Help the FDA see not only the reality of today’s information marketplace, but tomorrow’s, which is increasingly mobile and social.

Livongo’s Post Ad Banner 728*90

Categories: Uncategorized

Tagged as: ,

10
Leave a Reply

10 Comment threads
0 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
5 Comment authors
Bill SilbergRobertSusannah FoxKorkut VataJoshua Seidman Recent comment authors
newest oldest most voted
Korkut Vata
Guest

I believe what we see on social media about health care today reflects the perception from public point of view. And It is the same debate before internet era. Public will get what doctors and media tell them. Big pharma and ignorant doctors have a tendency to blurr the social media. http://www.americangenes.com/node/250 I wish money was not a part of the picture. Anyway, My suggestion is to make PubMed really public. While keeping the high quality editorial restrictions we can add “share-this icons” under the abstracts and people can circulate the original articles on social media. (Aallthough the circulation would… Read more »

Bill Silberg
Guest

As far as I can tell, Susannah, the full-text of the editorial is indeed behind JAMA’s subscription wall, so all I could manage was a link to the open-access excerpt. I did check PubMedCentral, but it doesn’t appear to be deposited there.

Susannah Fox
Guest

There’s a parallel conversation happening around this post on e-patients.net, including a debate around the quality of information on Wikipedia, Medpedia, and in the Cochrane Reviews:
http://e-patients.net/archives/2009/11/tell-the-fda-the-whole-story-please.html#comments

Susannah Fox
Guest

Thanks, Bill!
One of my favorite moments at the Informatics for Consumer Health summit last week was David Ahern’s Friday opener, honoring the health IT heroes of the 1990s (Mary Jo Deering, Gary Kreps, Tom Ferguson…). Your remembrance of things past means we are (hopefully) not condemned to repeat mistakes.
David’s remarks and the rest of the videos will soon be up:
http://www.consumerhealthinformatics.org/default.asp
May I ask you to correct what I see as a mistake (a subscription wall blocking our view of the JAMA editorial from 1997 – is such material a profit center??). Could you post more of the text on your blog?

Bill Silberg
Guest

Word of this week’s FDA hearing on the use of the internet and social media to promote regulated medical products gave me a sense of déjà vu, a similar hearing (minus the social media bit) having been held in 1996 (http://bsilberg.wordpress.com/2009/11/06/the-more-things-change). I get the same feeling from the related and very interesting discussion of the quality of internet medical information, a conversation that dates back at least to the 1995 establishment of the Health on the Net Foundation (http://www.hon.ch). I wrote about it with colleagues in a 1997 JAMA editorial (http://jama.ama-assn.org/cgi/content/summary/277/15/1244?maxtoshow=&HITS=10&hits=10&RESULTFORMAT=&fulltext=caveant&searchid=1&FIRSTINDEX=0&resourcetype=HWCIT). Apologies for the long URL. Since then, we’ve seen… Read more »

Robert
Guest

I love the posts on personal SEO-optimized sites that claim to be from “Dr. Such-and-Such” with some super cutting-edge advice and, of course, a convenient little link to their own fancy multivitamin product. If you need advice, stick to your doctor or, at the very least, WebMD.

Susannah Fox
Guest

I wanted to expand on the “tiny percentage” quote a bit since it’s worth knowing the full story: From Pew Internet’s Online Health Search 2006 report: HHS’s Office of Disease Prevention and Health Promotion, working with industry experts, identified six types of information that should be publicly disclosed to health seekers: the identity of the site’s sponsors, the site’s purpose, the source of the information provided, privacy policies to protect users’ personal information, how users can provide feedback, and how the content is updated. Of the 102 websites reviewed for the report, none met all six of the disclosure criteria… Read more »

Susannah Fox
Guest

Josh, thank you so much for linking to your excellent paper and for the reminder that JMIR has been at the forefront of the information-quality debate for many years. That research is yet more evidence that nobody ever figured out how to codify “health information quality” even before social media took off and changed the whole conversation. I definitely don’t have the answers (and agree with Korkut Vata not to blame the FDA or anyone else for also not yet having the answers). But I do want us to keep focused on the right questions. Asking why consumers aren’t checking… Read more »

Korkut Vata
Guest

Diseases are social events. Some of them are contiguous, jumping from one individual to another individual. Just like swine flu.
So is social networking. Today what matters is whether information spreads faster than the viral pandemics or not. I am not blaming FDA and on old academics in health care industry for not following up so far. In one year all the system will be in gear. I hope it won’t be used for the dark side of the power, for big pharma accomplishments.
http://www.americangenes.com/node/253

Joshua Seidman
Guest

Efforts to measure Internet health information quality have been going on for more than a decade. Most efforts to examine Web site quality have examined structural measures of information quality (e.g., MLA guidelines) and sponsorship characteristics of sites (for-profit vs. not-for-profit, advertising vs. none, etc.). Unfortunately, systematic research has demonstrated that these methods don’t have much relationship to the actual accuracy and comprehensiveness of the content (see http://ixcenter.org/publications/documents/e0035.pdf and references there to more methodological papers published in the Journal of Medical Internet Research). That’s not to say that there’s no value in knowing these things, but rather to ensure that… Read more »