Tell the FDA the whole story, please

Susannah fox

I scan menus for keywords (fig, parsnips, salmon…) and it turns out I scan Twitter the same way, looking for anyone who is talking about my favorite topics (data, consumers, information quality…)

So when I saw Jonathan Richman‘s tweet the other night, I couldn’t resist it:

Anyone ever seen data on the overall accuracy of medical information found online? Need help for some final stats for #fdasm

Short answer: No. Long answer:

The Pew Research Center’s Internet & American Life Project has been reporting on the social impact of the internet since 2000, when “information quality” on health websites was a big part of the conversation. It was the era of wagging fingers, scolding patients for straying too far outside their boundaries, and Pew Internet data was ammunition.

We released our first report about the internet’s impact on health & health care in November 2000. The Medical Library Association (MLA) contacted us, asking for research looking at how consumers decide which sites/sources to trust. With their help we created a set of questions asking first if respondents went online for health info, then asking if they look for the source and date of the info they find (the two key quality indicators according to the MLA).

Our 2002 report, Vital Decisions, reported the findings:

Only about one quarter of health seekers follow the recommended protocol on thoroughly checking the source and timeliness of information and are vigilant about verifying a site’s information every time they search for health information.  Another quarter of health seekers check a site’s information “most of the time.”  Half of all health seekers search for medical advice and “only sometimes,” “hardly ever,” or “never” check the source or date of the information they read online.

We repeated the source/date questions in August 2006 and reported on the “demand side” of health information in our report, Online Health Search 2006:

Three-quarters of health seekers do not consistently check the source and date of the health information they find online.

Unbeknownst to us, the U.S. Department of Health and Human Services was in the field with their study about the “supply side” of health info: Estimating the Proportion of Health Related Websites Disclosing Information That Can Be Used to Assess Their Quality.

I summarized their findings in our report, hoping to soften the impact of the ammunition we were handing out:

Health seekers might be forgiven if they give up what at times is a search for a needle in a haystack. A recent study commissioned by the U.S. Department of Health and Human Services (HHS) finds that a tiny percentage of health sites display the source and date of the information on their pages.

After reading the HHS study, looking at our data, and scanning the changing landscape (ie, consumers diversifying their online research to include user-generated content), I decided that trying to measure the quality of health info had run its course as a research question.

Pew Internet was also gathering evidence that consumers were not being harmed by inaccurate information. In a 2008 survey we conducted in partnership with the California HealthCare Foundation, just 3% of e-patients said they or someone they know has been harmed by following medical advice or health information found on the internet, a number that has remained stable since 2006. Meantime, 60% of e-patients (or 42% of all adults) said they or someone they know has been helped by following medical advice or health information found on the internet. That’s an increase from 2006 when 31% of e-patients (25% of all adults) said that.

After exchanging a few tweets and then sending a long email, I was pleased to see our data used in Jonathan’s upcoming presentation to the U.S. Food and Drug Administration‘s hearing on social media. But I worry that only the “three-quarters don’t check” data made it in to his slides, not the needle-in-a-haystack caveat provided by the HHS study.

Are we really going back to the era of scolding patients for not checking the source and date of health information they find online? If so, it’s my duty as the purveyor of that ammunition data point to speak up.

To everyone who is providing insights to the FDA: Tell the whole story. Help the FDA see not only the reality of today’s information marketplace, but tomorrow’s, which is increasingly mobile and social.

Categories: Uncategorized

Tagged as: ,

10 replies »

  1. I believe what we see on social media about health care today reflects the perception from public point of view. And It is the same debate before internet era. Public will get what doctors and media tell them.
    Big pharma and ignorant doctors have a tendency to blurr the social media.
    I wish money was not a part of the picture. Anyway,
    My suggestion is to make PubMed really public. While keeping the high quality editorial restrictions we can add “share-this icons” under the abstracts and people can circulate the original articles on social media. (Aallthough the circulation would be low).
    Example: People should read original pubmed articles on statins, rather than advertisements. Name and phone # of the authors should circulate in social media.

  2. As far as I can tell, Susannah, the full-text of the editorial is indeed behind JAMA’s subscription wall, so all I could manage was a link to the open-access excerpt. I did check PubMedCentral, but it doesn’t appear to be deposited there.

  3. Thanks, Bill!
    One of my favorite moments at the Informatics for Consumer Health summit last week was David Ahern’s Friday opener, honoring the health IT heroes of the 1990s (Mary Jo Deering, Gary Kreps, Tom Ferguson…). Your remembrance of things past means we are (hopefully) not condemned to repeat mistakes.
    David’s remarks and the rest of the videos will soon be up:
    May I ask you to correct what I see as a mistake (a subscription wall blocking our view of the JAMA editorial from 1997 – is such material a profit center??). Could you post more of the text on your blog?

  4. Word of this week’s FDA hearing on the use of the internet and social media to promote regulated medical products gave me a sense of déjà vu, a similar hearing (minus the social media bit) having been held in 1996 (
    I get the same feeling from the related and very interesting discussion of the quality of internet medical information, a conversation that dates back at least to the 1995 establishment of the Health on the Net Foundation ( I wrote about it with colleagues in a 1997 JAMA editorial ( Apologies for the long URL.
    Since then, we’ve seen seals of approval, codes of conduct, guidelines and all manner of caveats come and go, and the establishment of a growing body of valuable research (some cited here). Yet not only do connected consumers embrace medical information sources as never before, but patients and families regularly visit vibrant online communities that can provide sophisticated information and much-needed social support.
    Should we stop worrying about whether the internet is a source of misleading or even harmful information? No. Should we pay close attention to ongoing research and see that it informs policy? Sure. Should we continue to emphasize best practices in terms of listing authorship, attribution, disclosure and other clues to the validity of postings? Absolutely, although as know, even the most rigorous peer review at major journals, much less consumer web sites, can’t guarantee quality.
    But as this discussion continues, what fascinates me most is that the “market” – consumer and, to a lesser but growing degree, professional – steadily moves ahead in adopting available technologies in ways that meet each audience’s needs of the moment. I, for one, am interested in learning what such user behavior tells us about how we can improve communications among and between medical professionals and consumers, in hopes of connecting the two more effectively.

  5. I love the posts on personal SEO-optimized sites that claim to be from “Dr. Such-and-Such” with some super cutting-edge advice and, of course, a convenient little link to their own fancy multivitamin product. If you need advice, stick to your doctor or, at the very least, WebMD.

  6. I wanted to expand on the “tiny percentage” quote a bit since it’s worth knowing the full story:
    From Pew Internet’s Online Health Search 2006 report:
    HHS’s Office of Disease Prevention and Health Promotion, working with industry experts, identified six types of information that should be publicly disclosed to health seekers: the identity of the site’s sponsors, the site’s purpose, the source of the information provided, privacy policies to protect users’ personal information, how users can provide feedback, and how the content is updated. Of the 102 websites reviewed for the report, none met all six of the disclosure criteria and only six complied with more than three criteria. Just 4% of “frequently visited” health websites disclosed the source of the information on their pages and 2% disclosed how the content is updated. Less-popular health sites fared even worse: 0.3% of these sites listed their content’s source and only 0.1% disclosed how the content is updated.

  7. Josh, thank you so much for linking to your excellent paper and for the reminder that JMIR has been at the forefront of the information-quality debate for many years.
    That research is yet more evidence that nobody ever figured out how to codify “health information quality” even before social media took off and changed the whole conversation.
    I definitely don’t have the answers (and agree with Korkut Vata not to blame the FDA or anyone else for also not yet having the answers). But I do want us to keep focused on the right questions. Asking why consumers aren’t checking the source and date of the health info they find online? I’d nominate that as among the most useless questions in the era of Health 2.0.

  8. Diseases are social events. Some of them are contiguous, jumping from one individual to another individual. Just like swine flu.
    So is social networking. Today what matters is whether information spreads faster than the viral pandemics or not. I am not blaming FDA and on old academics in health care industry for not following up so far. In one year all the system will be in gear. I hope it won’t be used for the dark side of the power, for big pharma accomplishments.

  9. Efforts to measure Internet health information quality have been going on for more than a decade. Most efforts to examine Web site quality have examined structural measures of information quality (e.g., MLA guidelines) and sponsorship characteristics of sites (for-profit vs. not-for-profit, advertising vs. none, etc.).
    Unfortunately, systematic research has demonstrated that these methods don’t have much relationship to the actual accuracy and comprehensiveness of the content (see and references there to more methodological papers published in the Journal of Medical Internet Research). That’s not to say that there’s no value in knowing these things, but rather to ensure that we understand the limitations.
    These threshold criteria essentially are akin to licensure of physicians — we certainly want our doctors to be licensed, but that doesn’t necessarily mean that they practice high-quality medicine. Or, to put it another way, structural measures like the MLA guidelines are perhaps a necessary but not sufficient condition.
    Susannah’s point about telling the whole story is also certainly on target. But perhaps part of the reason consumers aren’t bothering to check for things like date & source is because those criteria aren’t necessarily going to lead them to accurate & comprehensive content.