Nate Silver and the sordid history of data
He was one of Time magazine’s 2009 most influential people, he was a 2012 Rolling Stone magazine “game changer,” and in 2013 he was anointed the #1 most creative person in business by Fast Company magazine.
All hail the mighty Nate Silver.
The king of predictive modeling was invited to Ventyx World to talk Big Data, and he provided an interesting perspective by going back to the beginning. In the case of data, that would be the advent of the printing press in the 15th century. To call it a paradigm shift would probably be underselling it, but Silver pointed out something I’d never considered before.
Prior to Gutenberg, Silver explained, it cost the equivalent of $25,000 in today’s dollars to produce one copy of a book. The printing press cut that cost by several orders of magnitude, and the technology spread across Europe within 50 years. The result was the proliferation of the ability to share information and it happened almost overnight in historical terms.
The impact can be seen in global GDP, which Silver noted remained essentially stagnant from Roman times up to the Industrial Revolution when it hit an inflection point and hasn’t looked back, barring the occasional financial crisis.
But there is about a 150 year gap here between the game-changing technology and the full economic impact, so what happened? In a word, war. Silver said the period of the late 15th through the 16th century was the bloodiest on record until the advent of mechanized warfare in the 20th. It simply took a while for the value of the printing press to be fully realized.
Jumping to the present era, Silver pointed to computers as an example of another lag between the introduction of a technology and its widespread adoption. Most companies were using them to some degree in the 1970s, but it wasn’t until the early 90s before PCs were on everyone’s desks. It’s probably not a coincidence that around that same time a protocol known as hypertext markup language started to be used to make information available to any user regardless of what operating system their machine was running.
Now, the web has become nearly as ubiquitous as the telephone and the buzz is all about “big” data. But it may be instructive to look to the past and realize that not every data mining project will yield the expected results. In fact, there is a very real risk that faulty models could produce some very dire unintended consequences.
This, Silver said, is a high-stakes game—high risks, high rewards. The recent financial crisis provides an object lesson. The financial models underpinning the mortgage-backed securities at the core of the crisis were based on data that only went back to 1985. The models, in short, had never been exposed to a nation-wide housing bubble like the one that popped in 2008.
So, a note of caution to go with your enthusiasm over the potential of Big Data to revolutionize every business on the planet. As enticing as it is, we might have to fight a few wars, lose our economic shirt or simply reset our expectations before we realize Big Data’s full potential.
Caveat emptor.