Tech

Healthcare.Gov and the Gulf Between Planning and Reality

Back in the mid-1990s, I did a lot of web work for traditional media. That often meant figuring out what the client was already doing on the web, and how it was going, so I’d find the techies in the company, and ask them what they were doing, and how it was going. Then I’d tell management what I’d learned. This always struck me as a waste of my time and their money; I was like an overpaid bike messenger, moving information from one part of the firm to another. I didn’t understand the job I was doing until one meeting at a magazine company.

The thing that made this meeting unusual was that one of their programmers had been invited to attend, so management could outline their web strategy to him. After the executives thanked me for explaining what I’d learned from log files given me by their own employees just days before, the programmer leaned forward and said “You know, we have all that information downstairs, but nobody’s ever asked us for it.”

I remember thinking “Oh, finally!” I figured the executives would be relieved this information was in-house, delighted that their own people were on it, maybe even mad at me for charging an exorbitant markup on local knowledge. Then I saw the look on their faces as they considered the programmer’s offer. The look wasn’t delight, or even relief, but contempt. The situation suddenly came clear: I was getting paid to save management from the distasteful act of listening to their own employees.

In the early days of print, you had to understand the tech to run the organization. (Ben Franklin, the man who made America a media hothouse, called himself Printer.) But in the 19th century, the printing press became domesticated.

Printers were no longer senior figures — they became blue-collar workers. And the executive suite no longer interacted with them much, except during contract negotiations.

This might have been nothing more than a previously hard job becoming easier, Hallelujah. But most print companies took it further. Talking to the people who understood the technology became demeaning, something to be avoided. Information was to move from management to workers, not vice-versa (a pattern that later came to other kinds of media businesses as well.) By the time the web came around and understanding the technology mattered again, many media executives hadn’t just lost the habit of talking with their own technically adept employees, they’d actively suppressed it.

I’d long forgotten about that meeting and those looks of contempt (I stopped building websites before most people started) until the launch of Healthcare.gov…

For the first couple of weeks after the launch, I assumed any difficulties in the Federal insurance market were caused by unexpected early interest, and that once the initial crush ebbed, all would be well. The sinking feeling that all would not be well started with this disillusioning paragraph about what had happened when a staff member at the Centers for Medicare & Medicaid Services, the department responsible for Healthcare.gov, warned about difficulties with the site back in March.In response, his superiors told him…

[…] in effect, that failure was not an option, according to people who have spoken with him. Nor was rolling out the system in stages or on a smaller scale, as companies like Google typically do so that problems can more easily and quietly be fixed. Former government officials say the White House, which was calling the shots, feared that any backtracking would further embolden Republican critics who were trying to repeal the health care law.

The idea that “failure is not an option” is a fantasy version of how non-engineers should motivate engineers. That sentiment was invented by a screenwriter, riffing on an after-the-fact observation about Apollo 13;no one said it at the time. (If you ever say it, wash your mouth out with soap. If anyone ever says it to you, run.) Even NASA’s vaunted moonshot, so often referred to as the best of government innovation, tested with dozens of unmanned missions first, several of which failed outright.

Failure is always an option. Engineers work as hard as they do because they understand the risk of failure. And for anything it might have meant in its screenplay version, here that sentiment means the opposite; the unnamed executives were saying “Addressing the possibility of failure is not an option.”

The management question, when trying anything new, is “When does reality trump planning?” For the officials overseeing Healthcare.gov, the preferred answer was “Never.” Every time there was a chance to create some sort of public experimentation, or even just some clarity about its methods and goals, the imperative was to avoid giving the opposition anything to criticize.

At the time, this probably seemed like a way of avoiding early failures. But the project’s managers weren’t avoiding those failures. They were saving them up. The actual site is worse—far worse—for not having early and aggressive testing. Even accepting the crassest possible political rationale for denying opponents a target, avoiding all public review before launch has given those opponents more to complain about than any amount of ongoing trial and error would have.

In his most recent press conference about the problems with the site, the President ruefully compared his campaigns’ use of technology with Healthcare.gov:

And I think it’s fair to say that we have a pretty good track record of working with folks on technology and IT from our campaign, where, both in 2008 and 2012, we did a pretty darn good job on that. […] If you’re doing it at the federal government level, you know, you’re going through, you know, 40 pages of specs and this and that and the other and there’s all kinds of law involved. And it makes it more difficult — it’s part of the reason why chronically federal IT programs are over budget, behind schedule.

It’s certainly true that Federal IT is chronically challenged by its own processes. But the biggest problem with Healthcare.gov was not timeline or budget. The biggest problem was that the site did not work, and the administration decided to launch it anyway.

This is not just a hiring problem, or a procurement problem. This is a management problem, and a cultural problem. The preferred method for implementing large technology projects in Washington is to write the plans up front, break them into increasingly detailed specifications, then build what the specifications call for. It’s often called the waterfall method, because on a timeline the project cascades from planning, at the top left of the chart, down to implementation, on the bottom right.

Like all organizational models, waterfall is mainly a theory of collaboration. By putting the most serious planning at the beginning, with subsequent work derived from the plan, the waterfall method amounts to a pledge by all parties not to learn anything while doing the actual work. Instead, waterfall insists that the participants will understand best how things should work before accumulating any real-world experience, and that planners will always know more than workers.

This is a perfect fit for a culture that communicates in the deontic language of legislation. It is also a dreadful way to make new technology. If there is no room for learning by doing, early mistakes will resist correction. If the people with real technical knowledge can’t deliver bad news up the chain, potential failures get embedded rather than uprooted as the work goes on.

At the same press conference, the President also noted the degree to which he had been kept in the dark:

OK. On the website, I was not informed directly that the website would not be working the way it was supposed to. Had I been informed, I wouldn’t be going out saying “Boy, this is going to be great.” You know, I’m accused of a lot of things, but I don’t think I’m stupid enough to go around saying, this is going to be like shopping on Amazon or Travelocity, a week before the website opens, if I thought that it wasn’t going to work.

Healthcare.gov is a half-billion dollar site that was unable to complete even a thousand enrollments a day at launch, and for weeks afterwards. As we now know, programmers, stakeholders, and testers all expressed reservations about Healthcare.gov’s ability to do what it was supposed to do. Yet no one who understood the problems was able to tell the President. Worse, every senior political figure—every one—who could have bridged the gap between knowledgeable employees and the President decided not to.

And so it was that, even on launch day, the President was allowed to make things worse for himself and his signature program by bragging about the already-failing site and inviting people to log in and use something that mostly wouldn’t work. Whatever happens to government procurement or hiring (and we should all hope those things get better) a culture that prefers deluding the boss over delivering bad news isn’t well equipped to try new things.

With a site this complex, things were never going to work perfectly the first day, whatever management thought they were procuring. Yet none of the engineers with a grasp of this particular reality could successfully convince the political appointees to adopt the obvious response: “Since the site won’t work for everyone anyway, let’s decide what tests to run on the initial uses we can support, and use what we learn to improve.”

In this context, testing does not just mean “Checking to see what works and what doesn’t.” Even the Healthcare.gov team did some testing; it was late and desultory, but at least it was there. (The testers recommended delaying launch until the problems were fixed. This did not happen.) Testing means seeing what works and what doesn’t, and acting on that knowledge, even if that means contradicting management’s deeply held assumptions or goals. In well run organizations, information runs from the top down and from the bottom up.

One of the great descriptions of what real testing looks like comes from Valve software, in a piece detailing the making of its game Half-Life. After designing a game that was only sort of good, the team at Valve revamped its process, including constant testing:

This [testing] was also a sure way to settle any design arguments. It became obvious that any personal opinion you had given really didn’t mean anything, at least not until the next test. Just because you were sure something was going to be fun didn’t make it so; the testers could still show up and demonstrate just how wrong you really were.

“Any personal opinion you had given really didn’t mean anything.” So it is in the government; any insistence that something must work is worthless if it actually doesn’t.

An effective test is an exercise in humility; it’s only useful in a culture where desirability is not confused with likelihood. For a test to change things, everyone has to understand that their opinion, and their boss’s opinion, matters less than what actually works and what doesn’t. (An organization that isn’t learning from its users has decided it doesn’t want to learn from its users.)

Given comparisons with technological success from private organizations, a common response is that the government has special constraints, and thus cannot develop projects piecemeal, test with citizens, or learn from its mistakes in public. I was up at the Kennedy School a month after the launch, talking about technical leadership and Healthcare.gov, when one of the audience members made just this point, proposing that the difficult launch was unavoidable, because the government simply couldn’t have tested bits of the project over time.

That observation illustrates the gulf between planning and reality in political circles. It is hard for policy people to imagine that Healthcare.gov could have had a phased rollout, even while it is having one.

At launch, on October 1, only a tiny fraction of potential users could actually try the service. They generated concrete errors. Those errors were handed to a team whose job was to improve the site, already public but only partially working. The resulting improvements are incremental, and put in place over a period of months. That is a phased rollout, just one conducted in the worst possible way.

The vision of “technology” as something you can buy according to a plan, then have delivered as if it were coming off a truck, flatters and relieves managers who have no idea and no interest in how this stuff works, but it’s also a breeding ground for disaster. The mismatch between technical competence and executive authority is at least as bad in government now as it was in media companies in the 1990s, but with much more at stake.

Tom Steinberg, in his remembrance of his brilliant colleague Chris Lightfoot, said this about Lightfoot’s view of government and technology:

[W]hat he fundamentally had right was the understanding that you could no longer run a country properly if the elites don’t understand technology in the same way they grasp economics or ideology or propaganda. His analysis and predictions about what would happens if elites couldn’t learn were savage and depressingly accurate.

Now, and from now on, government will interact with its citizens via the internet, in increasingly important ways. This is a non-partisan issue; whichever party is in the White House will build and launch new forms of public service online. Unfortunately for us, our senior political figures have little habit of talking to their own technically adept employees.

If I had to design a litmus test for whether our political class grasps the internet, I would look for just one signal: Can anyone with authority over a new project articulate the tradeoff between features, quality, and time?

When a project cannot meet all three goals—a situation Healthcare.gov was clearly in by March—something will give. If you want certain features at a certain level of quality, you’d better be able to move the deadline. If you want overall quality by a certain deadline, you’d better be able to simplify, delay, or drop features. And if you have a fixed feature list and deadline, quality will suffer.

Intoning “Failure is not an option” will be at best useless, and at worst harmful. There is no “Suddenly Go Faster” button, no way you can throw in money or additional developers as a late-stage accelerant; money is not directly tradable for either quality or speed, and adding more programmers to a late project makes it later. You can slip deadlines, reduce features, or, as a last resort, just launch and see what breaks.

Denying this tradeoff doesn’t prevent it from happening. If no one with authority over the project understands that, the tradeoff is likely to mean sacrificing quality by default. That just happened to this administration’s signature policy goal. It will happen again, as long politicians can be allowed to imagine that if you just plan hard enough, you can ignore reality. It will happen again, as long as department heads imagine that complex technology can be procured like pencils. It will happen again as long as management regards listening to the people who understand the technology as a distasteful act.

Clay Shirky is teacher, writer and consultant on the social and cultural effects of the internet and mobile phones, particularly where they allow for amateur access to the public sphere and easy coordination for group action. He is a professor of new media at New York University. He is most recently the author of Cognitive Surplus: Creativity and Generosity in a Connected Age. This post originally appeared on his blog, shirky.com.

Livongo’s Post Ad Banner 728*90

Categories: Tech, THCB

Tagged as: , , , ,

29
Leave a Reply

11 Comment threads
18 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
11 Comment authors
www.weddingwire.comFedBCRobert Rowley MDCatherine Recent comment authors
newest oldest most voted
www.weddingwire.com
Guest

I’m not sure why but this weblog is loading very slow for me.

Is anyone else having this issue or is it a problem on my end?
I’ll check back later and see if the problem still exists.

my blog; Utah wedding photography – http://www.weddingwire.com,

Fed
Guest
Fed

I think that some are falsely assuming that use of waterfall development methods is required in software developed for the federal government.

It isn’t.

Robert Rowley MD
Guest

I thought this was a good review of the process-failures that resulted in an embarrassingly-bad rollout. The top-down waterfall approach to technology development, which is characteristic of federal processes, is rarely used in industry. Its more modern alternative, the agile sprint, iterative process has become the norm in every IT business I am familiar with – identify pieces of the roadmap (yes, under a global vision), build with teams that provide two-way feedback, test and retool depending on the results for subsequent sprints going forward. You start small and build up to an eventually-massive project. Very different than how the… Read more »

Catherine
Guest
Catherine

Finally – someone who knows what they are talking about. I’ve been IT sys admin for over 20 years, have participated and headed numerous roll outs of hardware and software. It is exactly like that. If I had a nickle for every time some managerial id10t told me “failure is not an option,” I would not have as many stress related health problems. “I was getting paid to save management from the distasteful act of listening to their own employees.” Exactly.

archon41
Guest
archon41

Michelle has just suggested 14 “talking points” for your Thanksgiving repast, with the admonition “Be persistent.”

archon41
Guest
archon41

Never underestimate the insidious reach of the Republican Black Hand.

Peter1
Guest
Peter1

“Even as President Obama has issued a constant refrain of how upset he is that his Obamacare website doesn’t work and promises that he’s on top of the fix, three 20-year-old website designers in San Francisco made a working Obamacare website using Healthcare.gov’s own code. They did it in only three days.”

Now, can it really be that tough?

Robert
Guest
Robert

Incorrect. You’re referring to healthsherpa.com and it’s NOT and nothing like a what a working Obamacare website needs to be, despite what some lazy reporters have said. At healthsherpa.co you enter your zip code and it tells you what insurance plans are available for sale in your area and tells you the phone number of the insurance agencies. It also crashes if more than 1,000 people try to use it at once. In contrast, healthcare.gov must simultaneously interact with not only the customer, but the IRS, Immigration, 35 state Medicaid depts., credit bureaus, etc. and also determine if you’re eligible… Read more »

Peter1
Guest
Peter1

“You’re referring to healthsherpa.com and it’s NOT and nothing like a what a working Obamacare website needs to be”

Well it does one thing the Healthcare.gov site did not do and that is to allow shopping and comparing by age and income without registering or buying.

The Gov site only gave two price estimates, below 49 and above 50, but used best assumption on age to give false estimate – teaser rate, or bait and switch.

Robert
Guest
Robert

There’s no such thing as a “false estimate,” since estimate means preliminary guess. From Webster’s: “to judge tentatively or approximately the value, worth, or significance of; to determine roughly the size, extent, or nature of; to produce a statement of the approximate cost of.” It’s not bait and switch, either, because that’s, “a selling method in which a customer is attracted by the advertisement of a low-priced product but then is encouraged to buy a more expensive one.” At healthcare.gov there are very visible notices on each page that clearly state these are estimates and actual cost could vary. So… Read more »

Peter1
Guest
Peter1

“There’s no such thing as a “false estimate,” since estimate means preliminary guess.” Where’s the “guess”, there’s no guess or estimate involved – but there is deception. They don’t say “all rate estimates below 49 based on 27 year old” , or “all rate estimates above 50 based on 50 year old”. – do they? They’re deliberately misleading all buyers above age 27 and 50. That’s what you’d expect from car dealers not from a government department. “healthcare.gov isn’t advertising or encouraging you to buy a particular plan. It’s simply presenting a list of all of insurance plans that you… Read more »

Robert
Guest
Robert

Peter1, in response to your comments below (the system is not allowing me to reply directly below your comment) the Affordable Care Act absolutely is reform and DOES lower the cost of healthcare. Please see details regarding: “Since 2010, when the act was passed, real health spending per capita — that is, total spending adjusted for overall inflation and population growth — has risen less than a third as rapidly as its long-term average. Real spending per Medicare recipient hasn’t risen at all; real spending per Medicaid beneficiary has actually fallen slightly.” at: http://www.nytimes.com/2013/11/29/opinion/krugman-obamacares-secret-success.html?ref=opinion&_r=0 Also, how in the world can… Read more »

Peter1
Guest
Peter1

““Since 2010, when the act was passed, real health spending per capita — that is, total spending adjusted for overall inflation and population growth — has risen less than a third as rapidly as its long-term average.” Not because of the ACA but because of the economic crash. Health spending is not cost reduction. People simply could not afford elective treatment or going to the doctor as much, or they lost their coverage due to job loss. The ACA has not cut one price, and that’s why we spend double what other industrial countries do. “how in the world can… Read more »

Peter1
Guest
Peter1

Robert you may also want to consider this;

http://www.washingtonpost.com/blogs/wonkblog/wp/2013/04/26/will-obamacare-end-medical-bankruptcies-probably-not/

Being “covered” is not necessarily affordable.

Robert
Guest
Robert

Peter1, no offence, but what you’re saying is refutable by facts, but you simply dismiss the facts and offer opinion. You certainly have a right to your opinion, but opinion is not fact. Health insurance can be expensive, no doubt, but with the ACA you have more lower cost options than ever before. Due to a pre-existing condition I was uninsurable until the ACA banned such discrimination. Due to the cost of treatment for me, treatment would have quickly maxed out due to dollar limits before the ACA. In Cincinnati where I live, I can now choose from 72 different… Read more »

Peter1
Guest
Peter1

“but with the ACA you have more lower cost options than ever before.” A lower monthly premium means higher co-pays, deductibles and co-insurance. Robert, the risk did not suddenly change with the ACA. Prove to me the options are lower than before. Certainly those with cancelled policies don’t think it’s lower than before. “thanks directly to competition resulting from the ACA.” Robert, if you believe that insurance companies were padding their profit before the ACA and now they are competing by taking less profit and that’s the reason the plans are “less” you would be wrong. The plans cannot be… Read more »

Robert
Guest
Robert

Peter1, the overall cost of healthcare did change, and it’s lower, that’s a fact. It’s also lower for those without a subsidy, as I told you about my friend who’s Blue Cross rate dropped for $1,700 to $550 per month, without any subsidy. You also know that most of those cancelled policies covered barely anything. You can argue you all you want, even whine as you are doing, but it doesn’t change facts. Is the ACA perfect? Nothing is perfect. Is it a vast improvement? Yes, and any objective analysis proves that. There’s no sense continuing our discussion, because in… Read more »

Peter1
Guest
Peter1

“My friend in Kentucky is now saving over $1,000 per month on premiums from Blue Cross thanks directly to competition resulting from the ACA.” “as I told you about my friend who’s Blue Cross rate dropped for $1,700 to $550 per month, without any subsidy.” Only if he had a preexisting condition prior to ACA. However, others are paying more because of that. “You also know that most of those cancelled policies covered barely anything.” T o some extent you are right, but they also covered what those people needed with good coverage. The administration did not forewarn us this… Read more »

Robert
Guest
Robert

Peter1, so now you’re resorting to name calling? Calling people who disagree with you “idiots” and “trolls.” Don’t you realize how name calling and using rhetoric instead of facts, and making sweeping generalizations, dimninishes any points you try to make? Arguing that way might cause some people to think you’re a GOP troll on this blog. However, I’ll give you the benefit of the doubt and say you just like I complain. I suspect I’m not the first person to ever tell you that, either. Talking with you further is pointless. Goodbye.

Peter1
Guest
Peter1

I didn’t see the word “idiots” in my answer – how did you?

Posters on this blog know me as anything but GOP – I’ve been here since it started. But you continue to confirm you are an administration blogger troll by your hysterics and misrepresentations of what counts as fact.

Saurabh Jha
Guest
Saurabh Jha

Excellent piece.

Healthcare.gov could have learned from their cousins across the pond. The giant IT investment in the NHS was a gargantuan failure. It had more planners than doers.

Robert
Guest
Robert

If President Obama has never been involved in a tech rollout, which he hasn’t, then you can’t fault him for what he didn’t know he didn’t know. It’s also been painfully obvious during these oversight hearings that congressmen barely know what the internet is, either. These are people that are used to telling staff what to do and then expecting it to be done. Ultimately, Obama is responsible because he is the President, but those who knew the disaster that was coming and chose not to get word to the President humiliated him. He’s not the first CEO to learn… Read more »

BC
Guest
BC

“The GOP adopted a comprehensive, deliberate strategy to make the federal website fail, and they succeeded.”

I agree they attacked the bill nonstop, but don’t understand the above comment.

Robert
Guest
Robert

I meant simply that in addition to the challenges of a tech implementation (something that few politicians have any prior experience in), Obama has had to deal with half of the company (Congress) actively working to destroy the program and cripple it’s implementation. That’s something that no CEO would have to deal with, and few politicians. If he can successfully repair the failed implementation and get enrollment up to where it needs to be (turn the program around) then that will demonstrate he’s ultimately a great leader.

BC
Guest
BC

Robert: Thanks for the clarification. I would agree with you that in some ways that being president is like being a CEO; however, in other ways it is quite different. Had Perot gotten elected in 92 I think he would have had great difficulty in governing the nation because he was a CEO used to getting his way. Bloomberg on the other hand made the transition quite nicely. CEO and senior management all have one goal – earnings per share and in this regard they are united. Arguments arise from differences in opinion as to how this needs to happen.… Read more »

userlogin
Editor

I get what Dont Go Back to Rockville is saying. Anyone who’s dealt with technology developers knows there’s a little bit of a shell game. Overpromise, underdeliver, charge for the services on the front end, push back the expected ready date, then charge on the back end to fix the problem. And all this with the arrogance that only a technology person can have when dealing with non-tech people. But that doesn’t absolve the administration. The two key points of the article are exactly correct. There are multiple reports that have now been disclosed that made clear that the developers,… Read more »

Bobby Gladd
Guest

“I still support the law, but am disappointed in who this president is turning out to be.” __ He’s been criticized as a detached policy dilettante, an eloquent empty suit. He was certainly detached from the legislative development of the PPACA. I followed every draft as the bill wound its way through Congress. I knew what was in it. Looks like he didn’t. I voted for him twice. There was no choice, given the absurd alternatives. And, I’m sure the GOP will give yet another round of Clown Car Candidates in 2016. I guess I should just be grateful that… Read more »

Robert
Guest
Robert

I totally agree with your last sentence. However, I don’t agree he’s detached from the details of the PPACA. He was detached from the implementation details, yes, but not the law. Now he’s in deep in the implementation details. Check this out: http://www.bloomberg.com/news/2013-11-26/obama-nightly-reading-reveals-angst-about-health-site-fix.html

userlogin
Editor

Nice essay but not sure I agree with your takeaway.

We’ve got to do better than “the user just isn’t listening” – “they just don’t get it”

Communication is a two way street – or a bidirectional form of data transfer in your lingo. Personally, I’m getting kinda tired of being told “I just don’t get it.”

I get it.

These are some smart people. I know many of them personally. If they didn’t get it, it’s pretty unlikely they would have tried to build a government that runs like Amazon.