The Signal to Noise Ratio
Does more data lead to better decisions?
In order to answer that question, let’s go to the horse races for the afternoon.
In a famous study, Professor Paul Slovic gave a group of professional gamblers 5 more datapoints for each successive race – starting with 0 datapoints for the first race and ending with 40 datapoints for the final race -- and he tracked how accurate the bets were.
What Professor Slovic found was extremely interesting:
• First, having 5 datapoints was better than having no datapoints. The gamblers were more accurate -- and confident -- with some datapoints.
• Second, as the gamblers got more data, they became more confident in their accuracy, but their bets did not, in fact, get more accurate
• Third, the gamblers’ confidence increased every time they were given more datapoints – they were twice as confident with 40 datapoints as they were with 5 datapoints – but they were no more accurate with 40 datapoints than 5 datapoints.
As this study makes clear, more data often leads to worse decisions, not better decisions. And because people tend to make bigger bets when they’re more confident, “big data” can lead to big losses.
The idea that “big data” can lead to worse decisions is called “Signal To Noise Ratio.”
STNR is particularly relevant in digital marketing, where all marketers increasingly have access to big data. Unfortunately, most of the big data used to optimize and measure digital marketing campaigns is just “noise.”