The Signal to Noise Ratio
Article by:
Does more data lead to better decisions?
In order to answer that question, let’s go to the horse races for the afternoon.
In a famous study, Professor Paul Slovic gave a group of professional gamblers 5 more datapoints for each successive race – starting with 0 datapoints for the first race and ending with 40 datapoints for the final race -- and he tracked how accurate the bets were.
What Professor Slovic found was extremely interesting:
• First, having 5 datapoints was better than having no datapoints. The gamblers were more accurate -- and confident -- with some datapoints.
• Second, as the gamblers got more data, they became more confident in their accuracy, but their bets did not, in fact, get more accurate
• Third, the gamblers’ confidence increased every time they were given more datapoints – they were twice as confident with 40 datapoints as they were with 5 datapoints – but they were no more accurate with 40 datapoints than 5 datapoints.
As this study makes clear, more data often leads to worse decisions, not better decisions. And because people tend to make bigger bets when they’re more confident, “big data” can lead to big losses.
The idea that “big data” can lead to worse decisions is called “Signal To Noise Ratio.”
STNR is particularly relevant in digital marketing, where all marketers increasingly have access to big data. Unfortunately, most of the big data used to optimize and measure digital marketing campaigns is just “noise.”
An important question becomes how to identify “signal” and ignore “noise”?
One approach is to track “long data” not “big data.”
To explain what I mean by “long data” let’s talk about Nassim Taleb for a moment, who is fond of the phrase “time is the best judge of success.” Taleb reasons that no company, strategy or tactic can fool us over a long period of time. He believes, simply put, that long periods of time reveal the separate signal from noise.The example below explains what I mean.
An investment fund might get lucky and perform well over a short period – say less than a year – but very few funds will be lucky enough to perform well over a 10-year period. So, if you track performance over a long enough period of time, you will separate a good fund from a bad fund. The strategy to track “long data” is common in finance, where most funds don’t even report performance on a timeframe less than 1 month in marketing materials because they’d be laughed out of the room. Contrast that with digital marketing, where marketers judge performance based on what’s happening in real-time.
Sadly, marketing today does not think, measure, or invest in “long data.”
Marketing effectiveness researchers, Les Binet & Peter Field, have been pointing out that marketing has become increasingly short-term for more than a decade. One the great contributions Binet & Field made to advertising is to explain there’s two types of marketing that work on two different timeframes:
1. Short-term sales activation
2. Long-term brand building
Most digital marketing today is short-term, with more than 55% of campaigns looking to make an immediate sale. However, many marketers will know the Binet & Field analysis of over 1,500 IPA Databank case studies shows that brand building is “the main driver of profit growth when it comes to marketing.” Furthermore, Binet & Field recommend spending 60% of budget on brand building and 40% on sales activation.
How is it that marketers spend so much more than is optimal on short-term, sales activation?
The answer, in part, has to do with the fact that short-term, sales activation can be measure in real-time, while brand building cannot be fully measured for 6-9 months, at earliest. However, if you think about most of the greatest advertising campaigns of all-time – A Diamond Is Forever, Just Do It, Priceless – they’ve been run for decades. Even though common sense and econometric analysis show that brand building is more profitable, the seductive nature of real-time metrics lures marketers into a short-term trap.
Don’t believe me that real-time metrics are seductive?
Then let’s look at the data from a recent survey where we asked 4,000+ marketers two questions:
Question 1: “do you measure marketing on longer than a 6-month timeframe?
Answer 1: Somewhat shockingly, only 4% of marketers surveyed measure their marketing over a timeframe longer than 6 months.
As I just mentioned, it is difficult to see the effect of brand building on a timeframe shorter than 6 months, so it is, perhaps, no surprise companies choose not to invest as much money in brand building. They’re measuring campaign performance over too short a timeframe!
Now let’s ask the second question:
Question 2: “how many marketers are optimizing campaigns in two weeks or less?”
Answer 2: The survey answer confirmed what we all anecdotally know – marketing is increasingly short-term – by showing 75% of marketers optimize campaigns in real-time.
Measurement is clearly a problem in marketing, and has been since 19th century retailer, John Wanamaker, famously said “half the money I spend on advertising is wasted; the trouble is I don't know which half.”
While marketing measurement is a thorny issue, I do think it’s fair to say marketers need to think, measure, and report in a more financial way. The inability to explain marketing impact in more financial terms is part of the reason marketers make up only 2.6% of the members of Fortune 1500 board in America.
How might we explain marketing in more financial terms?
By listening to Craig Griffin, a brilliant marketer for Macquarie Bank in Australia.
Craig is also a proponent of the idea that there’s two different types of marketing that work on two different timeframes: sales activation and brand building.
Craig says that when he’s talking to his CFO or his finance team, he explains that lead generation is like an income statement. The numbers on an income statement – like gross sales – change regularly and, thus, can be reported on a weekly, monthly, or quarterly basis.
However, Craig says that brand building is more like a balance sheet. The numbers on a balance sheet – think real estate like The Empire State Building – change slowly, if at all, and only need need to be reported on an annual basis.
That’s how Craig Griffin talks framing marketing measurement and reporting for each of the two types of marketing in financial terms.
In summary, marketers need do a better job focusing on the long-term metrics that correlate to company profitability. By taking a longer-term view, marketers will separate signal from noise and make the right investments to profit not just this quarter, but for all quarters to come.
Signal To Noise