Much has been made of David Lazer’s finding that Google’s Flu Trends tracker seriously missed the mark in its measurement of flu activity for 2012-2013—and in previous years, too.
For those who don’t know, Flu Trends monitors Google search behaviors to identify regions where searches related to flu-like symptoms are spiking.In spite of Flu Trend’s notable misstep, Lazer still believes in the power of marrying health and social data.
In discussing the results of his study, he has maintained Google Flu is “a terrific” idea—one that just needs some refining. I agree.And, earlier this month, Nicholas Christakis, several other colleagues, and I—with funding from the Robert Wood Johnson Foundation—published a new method offering one such refinement.
Our paper shows that, in a given social network (in this study’s case, Twitter), a sample of its most connected, central individuals can hold significant predictive power.
We call this potentially powerful group of individuals a “sensor group.”
By finding and monitoring the tweets of a sensor group, we can catch—and sometimes even predict—the outbreak of contagious information early on. That detection edge could improve how we track the outbreak of disease epidemics, the rise of certain terms or phrases, or shifts in political sentiment.
Whereas Flu Trends relies on a relatively static, proprietary “dictionary” of flu-related search terms based on average Google search habits, the sensor method taps into what is really happening in social networks in real time.
By drawing on language being used by a sensor group—such as mentions of an emergent symptom or a popular newly coined name for a disease—Google could gain insight into what their dictionary might be missing.Sampling both the average Googler’s behavior and that of the exceptionally connected social network user can paint a much fuller picture of whatever landscape we are interested in tracking. We can more accurately see how it looks now—and how it could look in the near future.
In their best-selling 2013 book Big Data: A Revolution That Will Transform How We Live, Work and Think, authors Viktor Mayer-Schönberger and Kenneth Cukier selected Google Flu Trends (GFT) as the lede of chapter one.
They explained how Google’s algorithm mined five years of web logs, containing hundreds of billions of searches, and created a predictive model utilizing 45 search terms that “proved to be a more useful and timely indicator [of flu] than government statistics with their natural reporting lags.”
Unfortunately, no. The first sign of trouble emerged in 2009, shortly after GFT launched, when it completely missed the swine flu pandemic. Last year, Nature reported that Flu Trends overestimated by 50% the peak Christmas season flu of 2012. Last week came the most damning evaluation yet.
In Science, a team of Harvard-affiliated researchers published their findings that GFT has over-estimated the prevalence of flu for 100 out of the last 108 weeks; it’s been wrong since August 2011.
The Science article further points out that a simplistic forecasting model—a model as basic as one that predicts the temperature by looking at recent-past temperatures—would have forecasted flu better than GFT.
In short, you wouldn’t have needed big data at all to do better than Google Flu Trends. Ouch.
In fact, GFT’s poor track record is hardly a secret to big data and GFT followers like me, and it points to a little bit of a big problem in the big data business that many of us have been discussing: Data validity is being consistently overstated.
As the Harvard researchers warn: “The core challenge is that most big data that have received popular attention are not the output of instruments designed to produce valid and reliable data amenable for scientific analysis.”
The amount of data still tends to dominate discussion of big data’s value. But more data in itself does not lead to better analysis, as amply demonstrated with Flu Trends. Large datasets don’t guarantee valid datasets. That’s a bad assumption, but one that’s used all the time to justify the use of and results from big data projects.