Tom Davenport, research fellow at the MIT Center for Digital Business, makes a confession on page two of his new book Big Data At Work. He was a big data skeptic. A doubter. In fact, he says, he considered taking his previous books and just doing a global replace of analytics with big data and publishing the result. Davenport says he’s kidding about that last bit, but one of those books was actually called Analytics At Work, so it can’t be too far off the mark.
But this begs a larger question — why? Why would a man who preaches the gospel of analytics for a living doubt big data? Davenport blames technology vendors and their puffery. He says that big data evangelists (full disclosure: that’s what I do for a living) have oversold the idea’s potential.
This may be true. Any industry with winner-take-all economics is constantly tempted to promise too much. Let me pause for a moment of soul-searching.
Ok, done. I am squarely in the big-data-is-a-big-deal camp. But let me make my argument plain so you can judge it for yourself:
1. The scientific method is a good way of dealing with a complicated world. Systematically observing the world, analyzing what you saw, and applying what you learned helps you make decisions and take actions that leave you better off.
2. The cheaper something is, the more of it you get. As the cost to observe the world, analyze the observations, and use what you learned falls, more people will do this more often.
3. Digital technology makes the scientific method orders of magnitude cheaper than it used to be, encouraging its use in orders of magnitude more situations.
4. Therefore, the world will make a digital copy of itself and infuse data-driven decisions and actions into countless daily activities in commercial, civic, and private life. This is a change of such degree that it’s a change in kind, and we call it big data.
There you have it –the big data creed. It’s not about data formats, technologies, or statistics. It’s about a disruptive drop in the time, cost and effort to know your world empirically.
And Davenport uses another 200 pages to show his orthodoxy.
Data Capital 