TECH: Manhattan on PHRs. by Erica Fishman with UPDATE from Matthew

Erika Fishman from Manhattan Research, who wrote the report that I mentioned yesterday in my piece on "PHRs, EMRs and pretty much useless surveys", is rightfully a little grumpy about her survey being called useless. She very kindly wrote a very detailed reply to the questions I asked her. Here’s her answer. I’ll be back with my response to it later today (when I have a couple of deadlines out the way), but note carefully her explanation of why surveys are different and how they produce different answers.

I’ve now moved this up to Friday (originally posted on Thursday) with my comments after Erika’s in bold italics

I read your blog “PHRs, EMRs and pretty much useless surveys.”  As you can imagine, I do not agree that these surveys are useless (and I believe our loyal client base of leading global health and pharmaceutical companies would agree with me on that point). Furthermore, I believe the basis for your comparison is not entirely sound (given difference in approach, questions, methodology and sample).

Erika is right to defend surveys in general (and not just because she, Mark Bard and the crew at Manhattan make their living off them). Surveys are very, very good at explaining what is going on in the trenches, rather than what "accepted wisdom" says about something, and much better than the anecdotes that actually "inform" our debate.They are also good at picking up shifts in consumer and business/organization behavior and thinking. However, they are less good at understanding consumers’ future intentions, as opposed to business people’s likely decisions. And yeah, calling them "useless" is blogger’s hyperbole — no well researched data is useless. I’ve sold (and bought) several surveys in the past and will in the future, so no arguments from me — even if clients aren’t always sure exactly what they’re buying them for, nor are the answers about what to do with the results always that obvious.  However, better than tossing money at management consultants I say!

As you know, methodology is critical to the validity of results. Our survey sample size included more than 4,000 adults and was conducted via random digit-dialed telephone methodology-– providing for a representative mix of online and offline consumers. The 2005 Markle study used a sample of 800 registered voters (each for the two-separate sub-studies)- introducing potential selection bias due to demographics of registered voters. Markle’s 2003 study was out of 1,246 online consumers who were solicited via email. Already those consumers are more tech savvy than the general population base we surveyed, especially since they are “interactive” by taking online surveys.

Erika’s point here is very important, and unfortunately gets completely lost in the way surveys are reported. When the press says "a survey said X or Y" no one ever bothers to check the exact language, let alone the methodology. I worked for IFTF and Harris for years tracking the growth of the Internet and health care users of it, and our data and Cyberdialogue’s data consistently disagreed. (Harris always showed more users) I never found out what their questions were and what the discrepancy resulted from, but I only ever had ONE person (the very smart Sam Karp at CHCF) ask me why they were different, even though plenty of companies were buying both surveys and the results were widely reported.

And the inside baseball discussion about online versus telephone surveys is so deep in the mire that I can’t believe you’re not already asleep reading this.  Suffice it to say that the online-only crowd has a ton of data about how their surveys correct for the online/offline distinction. And somehow the registered versus likely voter discussion from last year gets picked up in this mess too because of the choices made by Markle’s pollsters (probably because they were combining polling questions with some policy work)! One thing we do know is that very, very close elections (less than 2-3% differences) cannot get picked up by surveys which have built-in margins of error. And exit-polls (questions of fact) are more reliable than pre-election surveys (questions of intention), even if the voting machines or other disqualification techniques screw-over the connection to the end result (as in Ohio in 2004 and Florida 2000).

The additional point, and here Manhattan seems to have done a better job than Markle’s survey guys, the Republican-affiliated Public Opinion Strategies (and why Markle’s using them rather than a "neutral" firm is  a little curious), is that it’s very hard to determine a consumer’s intention to use something that they’ve never used before or even seen, as it depends (as I said in my last piece) on how well it’s marketed to them, and how it fits into their daily workflow.  Most consumers cannot say and can’t really imagine what technologies they will be using in the future — after all if we all knew the iPod was going to be a big deal we’d have invented it ourselves! So when they say that 15% of adults are likely to use a PHR my guess is that it could be 5% and it could be 30%, but it won’t be 60% (which is what Markle said).

I am also not keen on the comparison because there are different meanings behind the words used in each study. While Markle uses “favor” and “want”, our question asks: “How likely are you to use a place to access/update your personal medical records online? I may still favor personal health records but not be likely to use them. In fact, that is how I personally feel about PHRs currently.

To be fair to me, I was in my piece contrasting the difference in numbers between the two surveys (the 15% and the 60%) and suggesting that Manhattan’s much lower number was more likely to be right based on the slow take up of PHRs and people emailing their physician.  Of course I’m much more interested in the numbers people report of what they are  actually doing (and of doctors reporting of what technologies they are using) because you can then see if a trend is taking off or not.  And it seems that EMR/PHR trend is taking off much more slowly than optimists from a few years back (who included me) guessed.

Our press releases are not chock-full of data because we do not want people/companies taking the data and re-purposing it as their own-– a problem we encounter frequently. Instead, we prefer to invite interested parties for more controlled webinar and multimedia presentations.  Please let me know if you are interested in accessing the presentation for this module, and I will arrange it.

Here I must disagree a little. The important part of the survey data for a client is not the top-line stuff that Harris puts in their press releases, it’s in the tabs–the information sub-dividing the studied population by their attributes and their intended actions. And it’s in the advice from the survey company about what to do with that information. Of course Manhattan and all survey companies are struggling to get people to come back to buy their surveys, and they don’t want their data stolen and re-purposed by other companies without giving them credit (and money!), but there is something to be said for letting a little more of it get out there in order to stop observers like me being forced to read into the tea-leaves. It certainly doesn’t seem to have hurt Harris, who’s healthcare business is much bigger now than when I was there, and which is putting out great information in its newsletters and in its Wall Street Journal articles. I suspect what they’re gaining in publicity vastly exceeds the sales that they’re losing by clients on the margin.

The 7.6 million emailers would include you-– we ask: “Have you ever emailed with a physician office?”

The 29.8 million includes only consumers age 18 and older.

I still don’t understand, even given that the margin of error in a 4,000 person survey is relatively low, why they don’t just say "about 30 million."

Formerly, we asked consumers if they were interested in using personal health records, but we have changed it to likelihood to use. I do not think these are necessarily comparable questions.

And there is the bane of the surveyer’s life. You want a trend but you asked the slightly wrong question way back when, when you weren’t quite sure what you were asking about. Now you’ve changed the phrasing to get it right and blown your trend data. Not much you can do about this other than invent a time machine.

Another factor here is space (or time) on a survey. In the physician computing survey I did at Harris in 1999 & 2000 I skated around EMR use by asking about what technologies doctors used for certain functions. In future surveys (after I’d left) they changed all that detail to simply "Do you use an EMR?"– a simpler but much less helpful question.  Unless of course you know exactly what an EMR is and you’re sure that all doctors share your exact opinion. I suspect the change was made to fit in other questions, but it goes to show the complexity of what you have to juggle when you’re designing a survey — and, no, it’s not easy.

I do agree with you that consumers will not be using PHRs until they “get used to it,” as you put it. This is the main point of our Consumer Health Interactivity module. If consumers are not yet adopting online interactive tools and programs, PHRs will not be a reality for a long time. Furthermore, taking advantage of highly interactive features, such as email with a physician, can help prime consumers for future PHR use. In fact, our study reveals that consumers with chronic health conditions who are currently emailing with their physicians are 230% times as likely to be interested in transmitting personal health data online as online consumers with chronic conditions who have no interest in emailing with physicians.

The example of chronically ill emailers is a very juicy tidbit of information which is more likely to get me interested if I’m a prospective client, and it reveals that Manhattan (and Erika) understands that this is a much more complex consumer market than is suggested by some of the "80% of consumers say they like EMRs"-type surveys that we’ve seen.

I still maintain that surveys of what consumers actually do, which technologies doctors actually use, and what technologies health care organizations are actually installing, are the most helpful. Perhaps HHS agrees with me, as (apparently frustrated by the numerous surveys out there on health care IT use) they have commissioned a series of their own.  How knows if they’ll get that right, and I hope that they ask Manhattan, Harris and me for advice.

But I stand by my final point. Irrespective of what they think they may or may not do in the future, consumers will use PHRs if they are provided and marketed to them in a logical, constructive way that fits into their use of the health care system, and connects them with their providers. And unfortunately that depends on their providers having their patients data in a useful and complete format. That in turn suggests that something like the RHIO and complete inter-operability will be needed before the PHR becomes absolutely complete, and for that we’ll have to wait a while.

Livongo’s Post Ad Banner 728*90

Categories: Uncategorized

Tagged as:

Leave a Reply

8 Comment threads
0 Thread replies
Most reacted comment
Hottest comment thread
4 Comment authors
Matthew HoltCraigRon Greinerkarl s. Recent comment authors
newest oldest most voted
Ron Greiner
Ron Greiner

Matthew, May I request a topic that the media refuses to discuss. We all know that when a hurricaine wipes out a small business all employees and the owners lose their health insurance because of government mandates. Another hurricaine is expected to hit Florida and this time it may hit Tampa Bay. My wife has purchased all of the hurricaine equipment for the windows and doors but my neighbor, a Brit, has not. Be the first one to report that employers can’t purchase the security of low cost individual insurance on employees but instead must purchase dangerous over-priced group health… Read more »

Matthew Holt

You have lost the plot completely, and are making my comments unreadable. If you dont stick to the point, I will lose my patience and ban you.

Ron Greiner
Ron Greiner

The Blue Cross people are way into politics. MI Blues want Liberal Debbie Stabenow (D-MI) to win in 2006 for U.S. Senate. So they are rounding up some spending cash for her.
Screaming Howard Dean is coming to help her on 11/15/2005. EMILY’s List said Michigan is their most important state in ’06. EMILY’s List is a cash cow and a sexist organization that won’t support men.
Sometimes health care reform can be fun.
I predict Liberal Stabenow will spend $50 million and lose. So much for big money in politics.

Ron Greiner
Ron Greiner

THCB – best news first. 3 minutes ago the wires report: Idaho new low cost health insurance option. When KY let insurance companies back in we were the first ones there. If Governor Mitt Romney (R-MA) opens up MA to competition to Blue Cross of MA I’m sure a press release like this will pop up in Boston too. Romney says he is trying to get as many Beantowners on Medicaid as he can to funnel Federal taxpayer dollars into the Commonwealth. I bet Ted Kennedy won’t like Romney anymore if he even thinks about creating any competition to Blue… Read more »


It is intersting that people dismiss surveys so readily – after all that is what a political opinion poll is also – and we all know how much politicians care about opinion polls.
It is the design of the statistical analysis and the margin of error that gives the greatest clue as to relevance of survey findings
Some thoughts from Craig at
The Natural Choice Store

Ron Greiner
Ron Greiner

Health Care Reform – Calling All Futurists Maybe Jib will start a blog and talk about Health Care Reform. We need one. Both the Liberals and Conservatives are singing from the same sheet of music and bashing employer-based coverage. The Boston Globe today has “Why Is Health Insurance Tied To The Workplace?” Governor Mit Romney (R-MA) and Newt are being quoted. Romney would really like to be President and thinks there should be affordable insurance options for individuals to purchase. Romney is probably bought off by Blue Cross of MA. and is helping them to increase their Monopoly, exactly like… Read more »

Ron Greiner
Ron Greiner

Our tax-free HSA keeps online Personal Medical Records for future IRS verification and it is free. Plus, it pays 6% interest and includes a mutual fund option. Newt says a US citizen can get $250,000 in an HSA by the TIME they become 65 years of age. You didn’t send Newt that commercial I emailed you, did you Matthew? It says, “Will you need $250,000 at 65 years of age just for retirement health care? Will a couple need $500,000? It’s TIME to save in the smartest way. President Bush says, HSAs have tax-free deposits, growth and withdrawals. Blah, blah,… Read more »

karl s.

Nice. I appreciated your article and have recently started turning my blog toward the subject of PHRs/EMRs because I am interested in the way they will be adopted and because I am wondering how the technology will overcome the privacy concerns. For that reason, I thought the Markle survey was more interesting. I *somewaht* agree with you on the surveys being useless – you would know better because you worked at Harris – but all surveys need to be taken with a grain of salt, right? But the the idea that individuals are open to the idea of PHRs is… Read more »