By Dan Diamond

Since CMS’s Center for Medicare and Medicaid Innovation launched three years ago, its staff have been frequently hailed for undertaking an ambitious research agenda.

But a New York Times story this week was eye-catching for a different reason: author Gina Kolata mostly assailed Medicare’s researchers for how they’re choosing to do that research.

“Experts say the center is now squandering a crucial opportunity,” Kolata wrote in a front-page article. “Many researchers and economists are disturbed that [CMMI] is not using randomized clinical trials, the rigorous method that is widely considered the gold standard in medical and social science research.”

But many researchers and economists that I talked to at this week’s Academy Health conference say that’s not the case at all. (And some were disturbed to learn that they were supposed to be disturbed.)

“RCTs are helpful in answering narrowly tailored questions,” Harvard’s Ashish Jha told me. “Something like—does aspirin reduce 30-day mortality rates for heart attack patients.”

“However, for many interventions, RCTs may be either not feasible or practical.”

“While RCTs may be the gold standard for testing some hypotheses, it is not necessarily the most effective or desirable model for testing all hypotheses,” agrees Piper Su, the Advisory Board’s vice president of health policy.

CMMI’s ambitious goals

On its surface, Kolata’s article is built around a reasonable conclusion: RCTs offer plenty of value in health care, and we’d benefit from more of them.

• As Jha alludes to, think of a double-blinded pharmaceutical study where half the participants randomly get a new drug and the other half get a placebo; that’s an RCT.
• The famous RAND study that found having health insurance changes patients’ behavior: An RCT.
• The ongoing Oregon Health Insurance Experiment: Also, an RCT.

And it’s fair to examine how CMMI is pursuing its research, too.

Congress entrusted considerable authority to CMS’s innovation center, which was created under the Affordable Care Act and designed to fast-track ideas and health reforms. If the center successfully uncovers a promising payment or delivery reform, an HHS secretary can scale it up nationally without having to wait on an act of Congress.

For now, CMMI has focused on launching about three dozen demonstration projects, like its Pioneer Accountable Care Organization program. A demonstration project is a better way to see how an intervention would fare in the real world, center director Patrick Conway told the Times. It also allows the center to make real-time changes, Conway pointed out. And “if it does not look like [a project] is working, we can stop.”

Would RCTs even work for CMMI?

CMMI’s approach isn’t totally above reproach; the data that the center is seeing from its pilots could be confused by secular trends, like changes in population, practices, and so on. That’s why, Harvard’s Jha acknowledged, it’s important to design studies with a contemporary control group and statistical testing.

But under CMMI’s ambitious charter, researchers are attempting to track a range of payment and delivery reforms. And it’s hard to think of how the center could use an RCT for some of its projects.

For example, I asked a half-dozen different researchers to construct a hypothetical RCT to test how accountable care organizations would work. All were stumped.

“You’d have three patients, focus on one intervention, and track them for six months,” one joked to me.

What randomized controlled tests can (and can’t) offer

Researchers are careful to stress that RCTs can be very valuable in health policy. The reaction to new data coming out of the Oregon Health Insurance Experiment is an example of that; the OHIE has been closely scrutinized because there’s so much interest in figuring out how the Affordable Care Act’s coverage expansion will affect patient behavior and health.

But the effort to conduct an RCT can be prohibitive. If RAND attempted its seminal experiment today, it would cost half a billion dollars, says Adrianna McIntyre, managing editor of The Incidental Economist blog.

And CMMI’s just not equipped for that kind of work, McIntyre added; its $10 billion, 10-year budget may seem huge, but that’s still just 0.1% of Medicare and Medicaid spending over the next decade.

“RCTs attempt to control for all unplanned variables to the greatest extent possible,” Su points out, “and that is nearly impossible to do when you consider the scale and diversity of models that CMMI is pursuing…multi-year, multi-stage models that may not lend themselves to a traditional RCT timeline.”

Dan Diamond (@ddiamond) is Managing Editor of the Daily Briefing, a California Healthline columnist, and a Forbes contributor. This  post originally appeared in The Daily Briefing Blog.