By PHILIP JONES, MD

I am a clinician and a clinical trialist. Medical research in some form or another (performing it, consuming it, reviewing it, editing it, etc.) occupies much of my time. Therefore, you can imagine my excitement while watching Apple’s product announcement yesterday when they introduced a new open source software platform called ResearchKit. Apple states ResearchKit could:

“revolutionize medical studies, potentially transforming medicine forever”

ResearchKit allows clinical researchers to have data about various diseases collected directly from a study participant’s iPhone (and perhaps other devices in the future — see below). The software is introduced as a solution to several important problems with current clinical studies, such as:

• limited participation (the software allows everyone to participate; anyone with an iPhone can download a specific app for every study they want to participate in)
• frequent data entry (patients can enter data as often as required/desired, rather than only at limited opportunities such as hospital or clinic visits)
• data fidelity (currently-used paper patient “diaries” are prone to entering implausible or impossible values — the iPhone can limit the range of data entered)

Specifically, the website states:

ResearchKit simplifies recruiting and makes it easy for people to sign up for a study no matter where they live in the world. The end result? A much larger and more varied study group, which provides a more useful representation of the population.

This is a bold claim. We’ll see below that it doesn’t yet ring true.

How does it work?

(I describe the process here for an iPhone asthma study. I live in Canada, and am ineligible to participate because I’m not American, so I entered in data right up until they asked me to sign the consent form, which I did not do, so I was never enrolled in the study. You will note that my experience contradicts Apple’s statement above about “… no matter where they live in the world.” Most people in the world cannot participate in the initial iPhone research studies since the Institutional Review Board did not approve the study worldwide.)

Once you’ve downloaded the app for a specific study, you fire up the program, and it goes through a few questions to ensure you are eligible to participate (inclusion criteria and exclusion criteria).

Then, a formal consent process occurs, whereby you are made aware of the potential benefits and risks of participation. This is essentially a “Letter of Information Lite”. You are tested on your knowledge of the consent process with a brief quiz. The correct answers of the quiz are extremely easy to answer (even without reading any of the consent information) if you have any clinical trial experience.

You are able to read the full consent form from the start of the signup process. The full consent form is exactly like all of the Letters of Information a typical Research Ethics Board (REB) requires (since it has been passed by a real REB). The whole process to sign up only takes a minute or two.

The potential

The potential for smartphone-based research is enormous, since people can enter data by typing (i.e. blood pressure, heart rate), using sliders (i.e. for a visual analog scale measuring pain intensity), or other iPhone controls, and they can do it often. Thus, lots of data at many time points (a researcher’s dream). But, the real power of ResearchKit comes by integration with the touch screen and accelerometers built into the iPhone. This allows actualmeasurement of certain clinically-important parameters.

For instance, the mPower app (Parkinson’s disease) measures the number of two-finger taps done in a limited time to assess fine motor control. mPower also uses the microphone to assess various voice parameters which may portend a worse prognosis, and it uses the accelerometers in the iPhone to assess the stability of the patient’s gait and their balance. This is all potentially cool stuff. Other ResearchKit apps will include the ability to directly interface with Bluetooth medical devices, such as BP cuffs and spirometers. An app called Share the Journey, meant for women who have had breast cancer, permits researchers to better understand the long-term effects of chemotherapy on patients’ moods, energy, and cognitive abilities.

The researchers on Apple’s introductory video were (of course) quite impressed with the ability to quickly amass tons of data on the conditions they study.

This all sounds great, so what’s the problem?

Big data, Big problems

There are many problems with the current iteration of ResearchKit. Below, I will use some well-known types of bias in clinical trials as a framework to discuss why I think ResearchKit has a long way to go before it really becomes useful for believeable clinical research:

Selection bias

The sample recruited into a ResearchKit-based clinical trial is not necessarily representative of the population of interest. iPhone users are more likely to be affluent and educated, and many minority groups may well be under-represented. Plus, people using these apps are, by design, interested in their health. Much research suggests that people with a keen interest in their well-being tend to do better on average than those who are not so attuned to their health. So, the question arises: “Can the results of the research be extrapolated to the whole population”? Currently, I would strongly argue “No”.

Another problem involving selection bias is the fact that no verification ofany of the information provided by the user is possible. The breast cancer app is only supposed to be done in women. However, I easily signed up for the breast cancer trial by misrepresenting myself as an American woman (again, I didn’t complete the process, so I’m not actually a participant). This is a major problem, as the data could be easily corrupted by people who purposefully enter false data. The same thing goes for people under 18 who are not eligible for many studies. Who’s to know how much false data there will be in these studies’ datasets?

At inception, ResearchKit is only available for iPhones. No Android, Windows Phone, BlackBerry, etc. Do we want our clinical research to only apply to iPhone users? No, because they may in fact differ from other smartphone users in systematically important ways. Although Apple has made ResearchKit open source, this is not tantamount to allowing all smartphone users to participate. It is incumbent upon the researchers themselves to ensure the app for each platform is built, tested, and deployed. All of these things will be the easiest of course on Apple’s iOS. For many researchers with limited resources, alternative platforms may simply never happen. This will limit the inferential power of ResearchKit medical studies.

Attrition bias

The study apps state you can leave the study at any time. Although this is also true for all clinical research, because of the ease of enrolling into the study and the novelty of doing “research” on your iPhone, I suspect many people will start out and then never finish. In other words, only a small fraction of the eligible sample will finish the trial. This will inexorably bias the results in favour of those who stay in the study, which may skew results in favour of better outcomes.

Observer (ascertainment) bias

Observer bias occurs when people enter incorrect data, perhaps because they want or hope the results to be better than they actually are. This is a significant risk for these apps. Early adopters of these apps are likely to want to show that the technology “works” and therefore may be at risk of entering falsely optimistic data. In addition, anybody might enter data (even if they are not the patient) if they have access to the iPhone and if they know the 4-digit code needed to unlock the app. There is a major potential for inaccurate (or fraudulent) information to creep into the dataset. Again, this could greatly limit inferential power.

Big data

Big data is a really sexy topic these days. It seems to be mentioned everywhere. ResearchKit will definitely be big data. The apps could potentially have hundreds of thousands (even millions) of study participants. However, I am deeply concerned that this data, although “big”, is going to be filled with incorrect or intentionally corrupted data points. The ability to make inferences applicable to the general population will be extremely limited, and potentially highly biased.

I can just imagine the negative reaction of medical Journal Clubs when the first paper coming from ResearchKit gets published. Knowing that men could be actively entering data, for long periods of time, into a breast cancer study meant only for women will be enough to make most readers simply ignore the results. Similarly, knowing that people without Parkinson’s disease can bias the mPower dataset with bogus information will cause readers to pass on to “regular” clinical research.

This is surely not what Apple intended, but it will be the result unless major changes are made. You cannot simply “crowdsource” medical research. There must be some minimal checks and balances, and those checks and balances are precisely what ResearchKit is currently missing.

How could they make this better?

This is a difficult question, since, naturally, as soon as one tries to more tightly control who is enrolled in the study or tries to verify inclusion criteria (sex, age, co-morbidities, etc.), recruitment into the study will necessarily drop. However, verifying some basic information would go a long way towards ResearchKit study legitimacy, and it does not mean that recruitment will go down to the low levels traditionally seen in “standard” clinical trials.

As an example of how this could work, let’s consider a study in primary care. Family doctors following up patients treated for depression could give out (via a paper registration card, or electronically via email) an invitation that would be coded with a unique number. Only those participants with a valid invitation would be allowed to enter the study. This would ensure that, at the very least, a trusted individual has confirmed the inclusion and exclusion criteria have been met. This manoeuvre would go a long way to increasing our trust in the studies emanating from data collected from ResearchKit.

Conclusions

Clinical research did not become bloated and highly controlled for no reason. Although I’d be the first to state that the red tape surrounding clinical trials is now creating a situation whereby ensuring safety and ethical behaviour (the original intent of clinical trial bureaucracy) is being overshadowed by barriers which limit research (because most clinicians don’t want the hassle), the answer is not to simply have a free-for-all of uncontrolled, biased research. However, currently, studies enabled by Apple’s ResearchKit are not the answer for most clinical trials, since the information garnered from them will not be credible nor generalizable.

However, the future is bright! I have no doubt significant improvements will be made that will enable high quality, high fidelity research to be done using smartphones and ResearchKit. But, we are not there yet.

If Apple truly wants to “revolutionize” clinical research, it has got to do better. I am confident that it will.

Philip Jones is Associate Professor at the University of Western Ontario in the Departments of Anesthesia & Perioperative Medicine, Medicine (Program in Critical Care), Epidemiology & Biostatistics. He is a fellowship-trained cardiac anesthesiologist with certification by the NBE in transesophageal echocardiography. He is Associate Editor at the Canadian Journal of Anesthesia. His blog posts do not reflect the views of any of his professional affiliations.