Erika Fishman from Manhattan Research, who wrote the report that I mentioned yesterday in my piece on "PHRs, EMRs and pretty much useless surveys", is rightfully a little grumpy about her survey being called useless. She very kindly wrote a very detailed reply to the questions I asked her. Here’s her answer. I’ll be back with my response to it later today (when I have a couple of deadlines out the way), but note carefully her explanation of why surveys are different and how they produce different answers.
I’ve now moved this up to Friday (originally posted on Thursday) with my comments after Erika’s in bold italics
I read your blog “PHRs, EMRs and pretty much useless surveys.” As you can imagine, I do not agree that these surveys are useless (and I believe our loyal client base of leading global health and pharmaceutical companies would agree with me on that point). Furthermore, I believe the basis for your comparison is not entirely sound (given difference in approach, questions, methodology and sample).
Erika is right to defend surveys in general (and not just because she, Mark Bard and the crew at Manhattan make their living off them). Surveys are very, very good at explaining what is going on in the trenches, rather than what "accepted wisdom" says about something, and much better than the anecdotes that actually "inform" our debate.They are also good at picking up shifts in consumer and business/organization behavior and thinking. However, they are less good at understanding consumers’ future intentions, as opposed to business people’s likely decisions. And yeah, calling them "useless" is blogger’s hyperbole — no well researched data is useless. I’ve sold (and bought) several surveys in the past and will in the future, so no arguments from me — even if clients aren’t always sure exactly what they’re buying them for, nor are the answers about what to do with the results always that obvious. However, better than tossing money at management consultants I say!
As you know, methodology is critical to the validity of results. Our survey sample size included more than 4,000 adults and was conducted via random digit-dialed telephone methodology-– providing for a representative mix of online and offline consumers. The 2005 Markle study used a sample of 800 registered voters (each for the two-separate sub-studies)- introducing potential selection bias due to demographics of registered voters. Markle’s 2003 study was out of 1,246 online consumers who were solicited via email. Already those consumers are more tech savvy than the general population base we surveyed, especially since they are “interactive” by taking online surveys.
Erika’s point here is very important, and unfortunately gets completely lost in the way surveys are reported. When the press says "a survey said X or Y" no one ever bothers to check the exact language, let alone the methodology. I worked for IFTF and Harris for years tracking the growth of the Internet and health care users of it, and our data and Cyberdialogue’s data consistently disagreed. (Harris always showed more users) I never found out what their questions were and what the discrepancy resulted from, but I only ever had ONE person (the very smart Sam Karp at CHCF) ask me why they were different, even though plenty of companies were buying both surveys and the results were widely reported.
And the inside baseball discussion about online versus telephone surveys is so deep in the mire that I can’t believe you’re not already asleep reading this. Suffice it to say that the online-only crowd has a ton of data about how their surveys correct for the online/offline distinction. And somehow the registered versus likely voter discussion from last year gets picked up in this mess too because of the choices made by Markle’s pollsters (probably because they were combining polling questions with some policy work)! One thing we do know is that very, very close elections (less than 2-3% differences) cannot get picked up by surveys which have built-in margins of error. And exit-polls (questions of fact) are more reliable than pre-election surveys (questions of intention), even if the voting machines or other disqualification techniques screw-over the connection to the end result (as in Ohio in 2004 and Florida 2000).
The additional point, and here Manhattan seems to have done a better job than Markle’s survey guys, the Republican-affiliated Public Opinion Strategies (and why Markle’s using them rather than a "neutral" firm is a little curious), is that it’s very hard to determine a consumer’s intention to use something that they’ve never used before or even seen, as it depends (as I said in my last piece) on how well it’s marketed to them, and how it fits into their daily workflow. Most consumers cannot say and can’t really imagine what technologies they will be using in the future — after all if we all knew the iPod was going to be a big deal we’d have invented it ourselves! So when they say that 15% of adults are likely to use a PHR my guess is that it could be 5% and it could be 30%, but it won’t be 60% (which is what Markle said).
I am also not keen on the comparison because there are different meanings behind the words used in each study. While Markle uses “favor” and “want”, our question asks: “How likely are you to use a place to access/update your personal medical records online? I may still favor personal health records but not be likely to use them. In fact, that is how I personally feel about PHRs currently.
To be fair to me, I was in my piece contrasting the difference in numbers between the two surveys (the 15% and the 60%) and suggesting that Manhattan’s much lower number was more likely to be right based on the slow take up of PHRs and people emailing their physician. Of course I’m much more interested in the numbers people report of what they are actually doing (and of doctors reporting of what technologies they are using) because you can then see if a trend is taking off or not. And it seems that EMR/PHR trend is taking off much more slowly than optimists from a few years back (who included me) guessed.
Our press releases are not chock-full of data because we do not want people/companies taking the data and re-purposing it as their own-– a problem we encounter frequently. Instead, we prefer to invite interested parties for more controlled webinar and multimedia presentations. Please let me know if you are interested in accessing the presentation for this module, and I will arrange it.
Here I must disagree a little. The important part of the survey data for a client is not the top-line stuff that Harris puts in their press releases, it’s in the tabs–the information sub-dividing the studied population by their attributes and their intended actions. And it’s in the advice from the survey company about what to do with that information. Of course Manhattan and all survey companies are struggling to get people to come back to buy their surveys, and they don’t want their data stolen and re-purposed by other companies without giving them credit (and money!), but there is something to be said for letting a little more of it get out there in order to stop observers like me being forced to read into the tea-leaves. It certainly doesn’t seem to have hurt Harris, who’s healthcare business is much bigger now than when I was there, and which is putting out great information in its newsletters and in its Wall Street Journal articles. I suspect what they’re gaining in publicity vastly exceeds the sales that they’re losing by clients on the margin.
The 7.6 million emailers would include you-– we ask: “Have you ever emailed with a physician office?”
The 29.8 million includes only consumers age 18 and older.
I still don’t understand, even given that the margin of error in a 4,000 person survey is relatively low, why they don’t just say "about 30 million."
Formerly, we asked consumers if they were interested in using personal health records, but we have changed it to likelihood to use. I do not think these are necessarily comparable questions.
And there is the bane of the surveyer’s life. You want a trend but you asked the slightly wrong question way back when, when you weren’t quite sure what you were asking about. Now you’ve changed the phrasing to get it right and blown your trend data. Not much you can do about this other than invent a time machine.
Another factor here is space (or time) on a survey. In the physician computing survey I did at Harris in 1999 & 2000 I skated around EMR use by asking about what technologies doctors used for certain functions. In future surveys (after I’d left) they changed all that detail to simply "Do you use an EMR?"– a simpler but much less helpful question. Unless of course you know exactly what an EMR is and you’re sure that all doctors share your exact opinion. I suspect the change was made to fit in other questions, but it goes to show the complexity of what you have to juggle when you’re designing a survey — and, no, it’s not easy.
I do agree with you that consumers will not be using PHRs until they “get used to it,” as you put it. This is the main point of our Consumer Health Interactivity module. If consumers are not yet adopting online interactive tools and programs, PHRs will not be a reality for a long time. Furthermore, taking advantage of highly interactive features, such as email with a physician, can help prime consumers for future PHR use. In fact, our study reveals that consumers with chronic health conditions who are currently emailing with their physicians are 230% times as likely to be interested in transmitting personal health data online as online consumers with chronic conditions who have no interest in emailing with physicians.
The example of chronically ill emailers is a very juicy tidbit of information which is more likely to get me interested if I’m a prospective client, and it reveals that Manhattan (and Erika) understands that this is a much more complex consumer market than is suggested by some of the "80% of consumers say they like EMRs"-type surveys that we’ve seen.
I still maintain that surveys of what consumers actually do, which technologies doctors actually use, and what technologies health care organizations are actually installing, are the most helpful. Perhaps HHS agrees with me, as (apparently frustrated by the numerous surveys out there on health care IT use) they have commissioned a series of their own. How knows if they’ll get that right, and I hope that they ask Manhattan, Harris and me for advice.
But I stand by my final point. Irrespective of what they think they may or may not do in the future, consumers will use PHRs if they are provided and marketed to them in a logical, constructive way that fits into their use of the health care system, and connects them with their providers. And unfortunately that depends on their providers having their patients data in a useful and complete format. That in turn suggests that something like the RHIO and complete inter-operability will be needed before the PHR becomes absolutely complete, and for that we’ll have to wait a while.
Categories: Uncategorized
Matthew,
May I request a topic that the media refuses to discuss. We all know that when a hurricaine wipes out a small business all employees and the owners lose their health insurance because of government mandates. Another hurricaine is expected to hit Florida and this time it may hit Tampa Bay. My wife has purchased all of the hurricaine equipment for the windows and doors but my neighbor, a Brit, has not.
Be the first one to report that employers can’t purchase the security of low cost individual insurance on employees but instead must purchase dangerous over-priced group health employer-based coverage that everybody loses when Katrina blows a business away.
I’m sure you can fit this into your agenda driven topics. Somehow you can blame this nightmare of hundreds of thousands of Americans losing their employer-based insurance on President Bush, even if it isn’t true.
By the way, I meet with a big shot from 7-Eleven tomorrow. I enrolled this guy in 1997 into a tax free MSA. He is working on the national computer system so 7-Eleven can make HSA deposits into their employees’ HSAs. He says it’s just a software problem. I know you say that the 7-Eleven HSA program is no big deal but it is the oldest HSA program in America and started in January of 1997, the very first month.
I will try and limit my freedom of speech to only those topics that you discuss even if they are really boring like this one. Even in your post you are amazed that anybody is reading it. Please make sure all other commentors are restricted just like me because I don’t want to feel like I have been singled out and discriminated against just because I’m not a far left wing Liberal like the majority of your commentors.
Anytime you want to do a pod-cast with the first person in America that enrolled a tax free HSA just let me know. I really excell when I can use my tone of voice because I’m a salesperson. I’m sure you would do just fine Matthew in a pod-cast with me. I know Eric has a radio show but I have to admit his tone of voice is a little weak. He is a nice guy though.
Let employees have low cost HSA qualifying individual insurance so when Katrina comes your health insurance “Termination Of Coverage” doesn’t come with it. Come on Matthew this is the 3rd TIME I have asked you to report on this Politician Created Disaster (PCD).
Ron
You have lost the plot completely, and are making my comments unreadable. If you dont stick to the point, I will lose my patience and ban you.
Matthew
The Blue Cross people are way into politics. MI Blues want Liberal Debbie Stabenow (D-MI) to win in 2006 for U.S. Senate. So they are rounding up some spending cash for her.
http://www.crainsdetroit.com/cgi-bin/article.pl?articleId=27857
Screaming Howard Dean is coming to help her on 11/15/2005. EMILY’s List said Michigan is their most important state in ’06. EMILY’s List is a cash cow and a sexist organization that won’t support men.
Sometimes health care reform can be fun.
I predict Liberal Stabenow will spend $50 million and lose. So much for big money in politics.
THCB – best news first. 3 minutes ago the wires report: Idaho new low cost health insurance option. When KY let insurance companies back in we were the first ones there. If Governor Mitt Romney (R-MA) opens up MA to competition to Blue Cross of MA I’m sure a press release like this will pop up in Boston too.
Romney says he is trying to get as many Beantowners on Medicaid as he can to funnel Federal taxpayer dollars into the Commonwealth. I bet Ted Kennedy won’t like Romney anymore if he even thinks about creating any competition to Blue Cross. If Mitt wants to open up free markets he should call these people. Mitt says that there are a ton of young healthy males that are not giving Blue Cross money now and that has to change. In Ames, home of the first debate for President, a 25 year old male can get HSA qualifying health insurance for $29 a month. Matt should give his citizens the same insurance options that Iowa and Florida does with Governor Jeb Bush.
http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104&STORY=/www/story/10-17-2005/0004170579&EDATE=
It is intersting that people dismiss surveys so readily – after all that is what a political opinion poll is also – and we all know how much politicians care about opinion polls.
It is the design of the statistical analysis and the margin of error that gives the greatest clue as to relevance of survey findings
Some thoughts from Craig at
The Natural Choice Store
Health Care Reform – Calling All Futurists
Maybe Jib will start a blog and talk about Health Care Reform. We need one. Both the Liberals and Conservatives are singing from the same sheet of music and bashing employer-based coverage. The Boston Globe today has “Why Is Health Insurance Tied To The Workplace?” Governor Mit Romney (R-MA) and Newt are being quoted. Romney would really like to be President and thinks there should be affordable insurance options for individuals to purchase. Romney is probably bought off by Blue Cross of MA. and is helping them to increase their Monopoly, exactly like John Kerry. Trust me, that won’t play well in Ames at the first vote. I can just see Mit Romney say, “A 30 year old male can get HSA qualifying individual health insurance here in Ames for $35 a month but we make those companies illegal in MA and I promise I will make them illegal in all states if I’m elected President.” Check out the ehealthinsurance.com website with the Boston zip code of 02101. You get nothing. What kind of free and open market is Mit Romney running there? Then go to ehealthinsurance’s “Student Insurance” in Boston and only one company comes up – US, ha ha. ehealthinsurance even has our proper logo there, about TIME. I know ehealthinsurance.com won’t list us in Iowa’s 50301 zip code either so I checked our software and do confirm that we don’t market “Individual Medical” IM there in MA. Darn those bought off Politicians. Romney should say, “Hey, we have banned all the low cost insurance companies here in MA and I want to let in the same insurance companies that Governor Jeb Bush has in Florida.”
They want reform in WI too. Today this guy hits the HSA nail right on the head. He says that agents can’t figure out how to make any money on HSAs so none of them take the TIME to understand them. Then the goofball says don’t get a plan that combines the insurance and HSA together because they cost too much, fool. We are the only insurance company with the insurance and HSA and our HSA is free and pays 6% interest. This idiot got Blue Cross coverage then paid for an HSA at HSA Bank.
http://www.greenbaypressgazette.com/news/archive/biz_22610465.shtml
The big news is back in MOtown with these Michiganiacs. Steve Miller, CEO of broke Delphi, is in Newsweek’s 10/24/2005, “Motown Mechanic.” He was on Frank Beckman’s show on Friday on WJR. Beckman didn’t ask one question about health insurance benefits. Of course GM and Delphi are going down the tubes because of health insurance but these guys discussed Governor Granholm’s ideas about better educations, how disgusting and boring. These Michigan educators copied our commercial that says, “Massive Lay Offs, Highest Unemployement.” Another reason all the WJR adds are sounding like our HSA ads here in the Motor City.
What is really disgusting is Steve Miller saying he talked with Hillary about the Federal taxpayers paying for Delphi employees health insurance. Hillary won’t comment on this it is reported by Newsweek.
Michigan Governor Granholm is going down in flames even before Billionaire Devos spends a dime on advertising. Like an idiot she blasted Steve Miller and Delphi one day then was down on her knees to him the next. Granholm is finding out that the old Liberal propaganda isn’t working. It’s too funny.
http://www.detnews.com/2005/editorial/0510/16/A19-349690.htm
http://www.metrowestdailynews.com/columnists/view.bg?articleid=111786
Senator Debbie Stabenow replacement is being narrowed to just two canadates. If Governor Granholm is thrown out of office Stabenow will be dumped as well. These two Liberal babes will go down together. Michigan is so broke that the only thing Granholm can come up with in health care reform is the taxpayers paying for the health insurance of gay state employees’ lovers.
http://www.detnews.com/2005/editorial/0510/16/A23-349704.htm
Our tax-free HSA keeps online Personal Medical Records for future IRS verification and it is free. Plus, it pays 6% interest and includes a mutual fund option. Newt says a US citizen can get $250,000 in an HSA by the TIME they become 65 years of age. You didn’t send Newt that commercial I emailed you, did you Matthew? It says, “Will you need $250,000 at 65 years of age just for retirement health care? Will a couple need $500,000? It’s TIME to save in the smartest way. President Bush says, HSAs have tax-free deposits, growth and withdrawals. Blah, blah, blah.”
Newt says he might have to go to Ames, Iowa and run for President in 2008 if some of these other guys running for President don’t start talking about Republican health care reform and the tax free HSA. Newt says that if 5 or 6 other guys running “STEAL” his ideas he can quit and go play golf. I know how Newt operates and he is just selling himself to political candidates and he charges a bundle. Trust me, I’m much smarter than Newt on the tax free HSA, get real. Plus, when I go back to Ames I stay at my Mum’s house or one of my brothers. The big money is here in Michigan because some of these Republicans are loaded. Devos is running for Governor and he is a billionaire. Butler is running for the U.S. Senate and his church gives him 2 jets to get around in. So Governor Granholm and Senator Stabenow are pretty worried. Lucky Stabenow has $4.7 million banked so this could get pretty ugly.
I’m with you Matthew, these costly surveys are useless. Like Jimmy Buffett says, “the answers are the easy part, it’s the questions that raises a doubt.” I have a question Erika could ask and I can predict the results: //If your spouse got cancer and was terminated from their employer-based health insurance after a short COBRA, would you prefer your current Personal Medical Records emailed to you in the same electronic notification of your “Termination of Coverage” or would you prefer a additional email communication?//
Trust me, you can’t hide from your Personal Medical Records. I leave in 20 minutes to drive 2 hours to enroll some broke 26 year old guy who has already been declined by another insurance company because of Michigan law that makes their citizens uninsurable. I will spend all that TIME and money just to “not make one penny” because we will have to decline him too.
Every other radio commercial on WJR is starting to sound like our commercials, I should get a commission. Friday our HSA commercial ran and was followed by Blue Cross of Michigan’s HSA commercial. Then M-Care is running HSA commercials too. These local guys buy a ton more commercials but ours are a lot better. When I talk to my wife on my cell phone (when I’m driving to appointments) these HSA commercials are always playing. Pam said, “These guys are not to confident that their clients will save premiums with their HSA product.” I told her, “Thank you for being so much better than the voices these rookie local yokels use to promote their HSAs.”
President Bush said, “Get low cost HSA health insurance and combine that with a tax free HSA and you’ve got yourself affordable health care.”
Is there any doubt in your mind Matthew that we won’t find your Personal Medical Records and your “bum knee” if you try and get individual health insurance? Under Michigan law, if you lived here, you would probably be declined, sorry Matthew. I tell clients that everything is in the big computer in the sky, that works pretty good. Newt won’t touch the subject with a ten foot pole, the slacker. Blue Cross is getting all these calls from individual self employed people trying to save money on health insurance and go tax free with an HSA. I called and their guy told me, “If you want an HSA you have to get a job.” I thought wow, these guys may be rich but they certainly need better HSA consultants. You aught to call them Matthew and cash in on the exploding HSA business.
Newt should know that Personal Medical Records are extemely boring. I liked your comments on Erika Matthew. She should start asking better questions.
Nice. I appreciated your article and have recently started turning my blog toward the subject of PHRs/EMRs because I am interested in the way they will be adopted and because I am wondering how the technology will overcome the privacy concerns. For that reason, I thought the Markle survey was more interesting. I *somewaht* agree with you on the surveys being useless – you would know better because you worked at Harris – but all surveys need to be taken with a grain of salt, right? But the the idea that individuals are open to the idea of PHRs is interesting. And the fact that the dialogue on this subject is evolving says to me something will come of this in the near future. Even beyond what the President is proposing. One interesting item that came from the Markle conference was Gingrich’s (I think) comments that we should have PHRs by December 2006 – that is what? 10 years faster than HHS is proposing?