Nate Silver: 4 steps for turning big data into big progress
Nate Silver says there’s a gap between the promise and the reality of “big data,” and on Thursday the uber statistician-slash-journalist offered some suggestions for how people can get smarter about using data to make better business decisions.
Silver, known for his on-the-spot predictions for the last two presidential elections via his FiveThirtyEight blog, spent the last three years at the New York Times, where he honed both his blog and his personal brand before ESPN hired him last month “to extend ESPN’s leadership in using data and analytics in its cross-platform storytelling.”
1. Think probabilistically. “The quest for certainty can be the enemy of progress,” Silver said. In other words, it’s important to avoid absolutes when making predictions. The further in the future you’re looking, the more uncertain the outcomes are likely to be.
2. Know where you’re coming from. When working with data, it’s important to be aware of your weaknesses, blind spots, and biases. Silver compared data analysis to a security infrastructure – both are defined by their weakest links.
3. Survey the data landscape. Silver cited three key components of rich data – a term he prefers over big data. The first is quantity – a large volume of observations. The second is quality – the inputs must be reliable. The third is variety – observations should be collected “under a wide variety of conditions.” It’s not enough to have one or two of these components – you need all three to be effective.
4. Try and err. A willingness to experiment is very important. “Until you test a hypothesis, it’s not very valuable,” Silver said. Particularly in competitive industries, a steady stream of incremental improvements can help companies make profits around the margin. Silver cited Google as a prime example of this willingness to constantly experiment. “Google always sees its products as a work in progress,” he said.