I am a clinician and a clinical trialist. Medical research in some form or another (performing it, consuming it, reviewing it, editing it, etc.) occupies much of my time. Therefore, you can imagine my excitement while watching Apple’s product announcement yesterday when they introduced a new open source software platform called ResearchKit. Apple states ResearchKit could:
“revolutionize medical studies, potentially transforming medicine forever”
ResearchKit allows clinical researchers to have data about various diseases collected directly from a study participant’s iPhone (and perhaps other devices in the future — see below). The software is introduced as a solution to several important problems with current clinical studies, such as:
- limited participation (the software allows everyone to participate; anyone with an iPhone can download a specific app for every study they want to participate in)
- frequent data entry (patients can enter data as often as required/desired, rather than only at limited opportunities such as hospital or clinic visits)
- data fidelity (currently-used paper patient “diaries” are prone to entering implausible or impossible values — the iPhone can limit the range of data entered)
Specifically, the website states:
ResearchKit simplifies recruiting and makes it easy for people to sign up for a study no matter where they live in the world. The end result? A much larger and more varied study group, which provides a more useful representation of the population.
This is a bold claim. We’ll see below that it doesn’t yet ring true.
How does it work?
(I describe the process here for an iPhone asthma study. I live in Canada, and am ineligible to participate because I’m not American, so I entered in data right up until they asked me to sign the consent form, which I did not do, so I was never enrolled in the study. You will note that my experience contradicts Apple’s statement above about “… no matter where they live in the world.” Most people in the world cannot participate in the initial iPhone research studies since the Institutional Review Board did not approve the study worldwide.)
Once you’ve downloaded the app for a specific study, you fire up the program, and it goes through a few questions to ensure you are eligible to participate (inclusion criteria and exclusion criteria).
Then, a formal consent process occurs, whereby you are made aware of the potential benefits and risks of participation. This is essentially a “Letter of Information Lite”. You are tested on your knowledge of the consent process with a brief quiz. The correct answers of the quiz are extremely easy to answer (even without reading any of the consent information) if you have any clinical trial experience.
You are able to read the full consent form from the start of the signup process. The full consent form is exactly like all of the Letters of Information a typical Research Ethics Board (REB) requires (since it has been passed by a real REB). The whole process to sign up only takes a minute or two.
The potential for smartphone-based research is enormous, since people can enter data by typing (i.e. blood pressure, heart rate), using sliders (i.e. for a visual analog scale measuring pain intensity), or other iPhone controls, and they can do it often. Thus, lots of data at many time points (a researcher’s dream). But, the real power of ResearchKit comes by integration with the touch screen and accelerometers built into the iPhone. This allows actualmeasurement of certain clinically-important parameters.
For instance, the mPower app (Parkinson’s disease) measures the number of two-finger taps done in a limited time to assess fine motor control. mPower also uses the microphone to assess various voice parameters which may portend a worse prognosis, and it uses the accelerometers in the iPhone to assess the stability of the patient’s gait and their balance. This is all potentially cool stuff. Other ResearchKit apps will include the ability to directly interface with Bluetooth medical devices, such as BP cuffs and spirometers. An app called Share the Journey, meant for women who have had breast cancer, permits researchers to better understand the long-term effects of chemotherapy on patients’ moods, energy, and cognitive abilities.
The researchers on Apple’s introductory video were (of course) quite impressed with the ability to quickly amass tons of data on the conditions they study.
This all sounds great, so what’s the problem?
Big data, Big problems
There are many problems with the current iteration of ResearchKit. Below, I will use some well-known types of bias in clinical trials as a framework to discuss why I think ResearchKit has a long way to go before it really becomes useful for believeable clinical research:
The sample recruited into a ResearchKit-based clinical trial is not necessarily representative of the population of interest. iPhone users are more likely to be affluent and educated, and many minority groups may well be under-represented. Plus, people using these apps are, by design, interested in their health. Much research suggests that people with a keen interest in their well-being tend to do better on average than those who are not so attuned to their health. So, the question arises: “Can the results of the research be extrapolated to the whole population”? Currently, I would strongly argue “No”.
Another problem involving selection bias is the fact that no verification ofany of the information provided by the user is possible. The breast cancer app is only supposed to be done in women. However, I easily signed up for the breast cancer trial by misrepresenting myself as an American woman (again, I didn’t complete the process, so I’m not actually a participant). This is a major problem, as the data could be easily corrupted by people who purposefully enter false data. The same thing goes for people under 18 who are not eligible for many studies. Who’s to know how much false data there will be in these studies’ datasets?
At inception, ResearchKit is only available for iPhones. No Android, Windows Phone, BlackBerry, etc. Do we want our clinical research to only apply to iPhone users? No, because they may in fact differ from other smartphone users in systematically important ways. Although Apple has made ResearchKit open source, this is not tantamount to allowing all smartphone users to participate. It is incumbent upon the researchers themselves to ensure the app for each platform is built, tested, and deployed. All of these things will be the easiest of course on Apple’s iOS. For many researchers with limited resources, alternative platforms may simply never happen. This will limit the inferential power of ResearchKit medical studies.
The study apps state you can leave the study at any time. Although this is also true for all clinical research, because of the ease of enrolling into the study and the novelty of doing “research” on your iPhone, I suspect many people will start out and then never finish. In other words, only a small fraction of the eligible sample will finish the trial. This will inexorably bias the results in favour of those who stay in the study, which may skew results in favour of better outcomes.
Observer (ascertainment) bias
Observer bias occurs when people enter incorrect data, perhaps because they want or hope the results to be better than they actually are. This is a significant risk for these apps. Early adopters of these apps are likely to want to show that the technology “works” and therefore may be at risk of entering falsely optimistic data. In addition, anybody might enter data (even if they are not the patient) if they have access to the iPhone and if they know the 4-digit code needed to unlock the app. There is a major potential for inaccurate (or fraudulent) information to creep into the dataset. Again, this could greatly limit inferential power.
Big data is a really sexy topic these days. It seems to be mentioned everywhere. ResearchKit will definitely be big data. The apps could potentially have hundreds of thousands (even millions) of study participants. However, I am deeply concerned that this data, although “big”, is going to be filled with incorrect or intentionally corrupted data points. The ability to make inferences applicable to the general population will be extremely limited, and potentially highly biased.
I can just imagine the negative reaction of medical Journal Clubs when the first paper coming from ResearchKit gets published. Knowing that men could be actively entering data, for long periods of time, into a breast cancer study meant only for women will be enough to make most readers simply ignore the results. Similarly, knowing that people without Parkinson’s disease can bias the mPower dataset with bogus information will cause readers to pass on to “regular” clinical research.
This is surely not what Apple intended, but it will be the result unless major changes are made. You cannot simply “crowdsource” medical research. There must be some minimal checks and balances, and those checks and balances are precisely what ResearchKit is currently missing.
How could they make this better?
This is a difficult question, since, naturally, as soon as one tries to more tightly control who is enrolled in the study or tries to verify inclusion criteria (sex, age, co-morbidities, etc.), recruitment into the study will necessarily drop. However, verifying some basic information would go a long way towards ResearchKit study legitimacy, and it does not mean that recruitment will go down to the low levels traditionally seen in “standard” clinical trials.
As an example of how this could work, let’s consider a study in primary care. Family doctors following up patients treated for depression could give out (via a paper registration card, or electronically via email) an invitation that would be coded with a unique number. Only those participants with a valid invitation would be allowed to enter the study. This would ensure that, at the very least, a trusted individual has confirmed the inclusion and exclusion criteria have been met. This manoeuvre would go a long way to increasing our trust in the studies emanating from data collected from ResearchKit.
Clinical research did not become bloated and highly controlled for no reason. Although I’d be the first to state that the red tape surrounding clinical trials is now creating a situation whereby ensuring safety and ethical behaviour (the original intent of clinical trial bureaucracy) is being overshadowed by barriers which limit research (because most clinicians don’t want the hassle), the answer is not to simply have a free-for-all of uncontrolled, biased research. However, currently, studies enabled by Apple’s ResearchKit are not the answer for most clinical trials, since the information garnered from them will not be credible nor generalizable.
However, the future is bright! I have no doubt significant improvements will be made that will enable high quality, high fidelity research to be done using smartphones and ResearchKit. But, we are not there yet.
If Apple truly wants to “revolutionize” clinical research, it has got to do better. I am confident that it will.
Philip Jones is Associate Professor at the University of Western Ontario in the Departments of Anesthesia & Perioperative Medicine, Medicine (Program in Critical Care), Epidemiology & Biostatistics. He is a fellowship-trained cardiac anesthesiologist with certification by the NBE in transesophageal echocardiography. He is Associate Editor at the Canadian Journal of Anesthesia. His blog posts do not reflect the views of any of his professional affiliations.
I am worried of listening to that one more application is everything we need to comprehend the rot of the human condition. I think we would be vastly improved off in the event that we disposed of our doltish telephones and began having genuine connections in stead of these pseudosocial media companions.
This article looks like a surfacing of the classic divide between standard researchers, with their double-blind checks and balances, and big data enthusiasts who believe that good statistical data cleaning will find the truth in even very dirty data. This philosophical disagreement is not a simple one to resolve, but I believe that ResearchKit could be useful if data sets are big enough and the truth is close enough to the surface to discover, given sophistication about the potential sources of error.
From the Apple site:
“Numbers are everything. The more people who contribute their data, the bigger the numbers, the truer the representation of a population, and the more powerful the results. A research platform that allows large amounts of data to be collected and shared — that can only be a positive thing for medical research.”
– Dr. Eduardo Sanchez, American Heart Association
Numbers are “everything”?
I disagree. Especially where it comes to numbers of uneven data quality and research rigor.
More broadly, are we assuming that case narratives have become irrelevant? The patients’ stories, and their physicians’ narrative interpretations of them are irrelevant in the face of “big data”? (I call coding and numeric data “lossy compression”)
Some of this goes to purpose. e.g., from my blog:
“I worked in subprime credit risk modeling from 2000 to 2005 … We could be “wrong” 99% of the time as long as the 1% we got “right” paid for everything and turned a profit (which they did; the bank set new profitability records year after year across my entire tenure).
We bought pre-selected direct marketing prospect lists and launched massive direct mail, internet, and phone campaigns. We got about a 5% response rate (or, equivalently, a 95% “error” rate). We then culled people not making the initial booking criteria cut (more “errors”), and subsequently booked those passing muster. Many of those would go delinquent and eventually “charge off” (yet again more “errors”). The tiny minority who proved profitable paid for the entire operation…”
Maybe heterogeneous healthcare “big data” arriving from millions of iPhone tributaries will help move the population health needles in the aggregate. Whether they will benefit individual patients near-term is a wholly different question.
OK, academics are going to access faster our data, partner with pharma, publish articles, apply for IP, make drugs, and make tons of $$$ while, we, the patients will still pay tons of $$$ our treatments.
So, please, when you talk about Big Data concerns, try not forgetting that medical data is a big asset. It has a tremendous value today.
Why the patients should give away their data for free, when the pharma industry and the big data industry still making huge profit?
This is still wrong with the ResearchKit or without !!!
I didn’t make the clinical assessment, but the technical assessment was disingenuous – at best. Apple intentionally used the word “open source” – and misled many into thinking that (through some Apple magic) the app could be used on other platforms like Android or MSFT.
Combined with their arrogance (basically a middle finger to everyone on the livestreaming of the event), I think what we’re seeing isn’t real leadership – but the terms of Apple’s participation in clinical research. Healthkit (the companion “framework”) has been relegated to fitness and wellness functions – so we’re a long way from any real value to either “framework.” Again, that’s not leadership and that – on the part of the worlds largest tech company – is disappointing.
Apple Could Lead In Healthcare. Here’s Why They Won’t.
Great article and thanks for elucidating the biases.
Regarding the primary care research issue, Healthloop is doing exactly that. We have doctors Rx the Healthloop app, which represents a longitudinal journey of ‘what to do’ and ‘how are you’ through Healthloop and HealthKit.
We are collecting amazing data and research ability – not using ResearchKit…yet.
I am so tired of hearing that one more app is all we need to solve the decay of the human condition. I think we would be much better off if we got rid of our stupid phones and started having actual relationships in stead of these pseudosocial media friends.