By Joshua Seidman

Jon Gabel from the National Opinion Research Center has an excellent op-ed piece in today’s New York Times. The basic argument is summarized in his conclusion:

“The Congressional Budget Office’s integrity is beyond questioning. But the record shows that it has substantially overestimated the cost of health care reform three times out of three. As Congress now works on its greatest push for reform in generations, the budget office needs to revise the methods it uses to make predictions about costs.”

Far from being an arcane methodological debate, CBO’s approach has profound consequences for health care reform and for the long-term health and economic conditions of the country. As Gabel puts it:

“The budget office’s cautious methods may have unintended consequences in the current health care reform effort. By underestimating the savings that can come from improved Medicare payment procedures and other cost-control initiatives, the budget office leads Congress to think that politically unpopular cost-cutting initiatives will have, at best, only modest effects. This, in turn, forces Congress to believe it can pay for reform only by raising taxes, which then makes reform legislation more difficult to pass.”

The reason that CBO has underestimated savings from past reforms of Medicare is that it makes the assumption that — without convincing empirical evidence of an initiative’s cost impact — it basically “scores” it as delivering zero savings. No doubt that CBO is consistent and conservative, but that doesn’t necessarily produce the most accurate budgetary forecast.

Perhaps more so than any other area in the federal budget, there are an enormous number of unknowns in health care. CBO has historically built its model on the premise that absence of evidence equates with evidence of absence.

But there is a major distinction. “Evidence of absence” means that we have an empirical reason to believe that there is no effect of an intervention (in this case on cost). In that case, it makes sense to score zero savings.

In contrast, “absence of evidence” simply means that we do not have sufficient evidence that an intervention produces any effect.  The problem is that, by definition, any true “innovation” (defined by Merriam-Webster as “the introduction of something new”) has no evidence. Which is to say: CBO has effectively ruled out scoring savings for true innovation.

Perhaps some would argue that’s an overstatement in that we certainly commonly use the term innovation to describe something that has been around long enough to be tested. Yes and no. There’s no doubt that new and innovation are relative terms, but there are still important reasons why that approach for CBO remains flawed.

First, evaluation takes time. To design a study, appropriately manage it, collect and analyze data, submit to peer review, and publish often takes many years.

Second, the level of evidence that CBO typically requires takes A LOT of time.

Third, innovation often comes from combining different initiatives and strategies that create a combined effect greater than the sum of their parts. Information therapy, patient decision aids, comparative effectiveness research, and other delivery system reforms may have a powerful impact when thoughtfully and appropriately combined together.

Fourth, the pace of innovation and the greatest innovative impacts can be dramatically robust. There is no way, in its current model, for CBO to capture those things that will have the most important effects on the federal health budget.

Like Jon Gabel, I don’t question the CBO’s integrity or analytical capacity, but I do believe that its methodological approach requires amendment. As I have written before, we — as health services researchers (and I admit to being one myself) — need to maintain our analytical rigor while being as creative in our research methods as the innovators are at innovating.

We should not shy away from the empirical idiosyncrasies that innovative care delivery initiatives create. Rather, we should rise to the challenge by employing a broader set of research and analytical skills to tackle these compelling research questions about new innovations. Indeed, the new care delivery strategies create opportunities for health services researchers to develop their own innovative research techniques.

I hope that health services researchers out there are up to that challenge.

If we aren’t, we will continue to create perverse public policy incentives.

Joshua Seidman is the president of of the Center for Information Therapy that aims to provide the timely prescription and availability of evidence-based health information to meet individuals’ specific needs and support sound decision making.  He frequently blogs for THCB and the Center for Information Therapy Blog, where this post first appeared.

SHARETHIS.addEntry({ title: “Innovation and Absence of Evidence vs. Evidence of Absence”, url: “http://ixcenterblog.org/archives/737” });