Written by Will Jeffreys on 05 Dec 2016
It’s funny how the things you hear can trigger things in your brain.
I was listening to a thing about critical thinking yesterday (on one of those pop psychology type podcasts) and something called the Texas Sharpshooter Fallacy came up.
If you haven’t heard it, its name comes from the joke about a Texan who fires some gunshots at the side of a barn, then paints a target around the biggest cluster of hits and claims to be a sharpshooter.
When it comes to data, it means that you stress the importance of one data cluster while ignoring all the others that don’t prove what you want.
Sometimes this is done nefariously, and you know exactly what you’re doing – twisting the facts to fit your hypothesis. Sometimes it isn’t. Given that the brain is programmed to see patterns and organise order out of chaos.
It struck me that I may have done it myself the other week.
We interviewed people as part of a brief. Of the 12 people we interviewed, 8 said that they participated in some form of sports activity (mountain biking, running, road cycling). We concluded that, if the company that we were working for wanted more people like this to join, they should look at targeting sites associated with these sports.
But is that right?
I mean, yes, there were quite a lot of people in our set who said they were into sports. But there are quite a lot of things to factor in:
Are the people who agree to be interviewed more likely to do that sort of pastime because they like saying “yes” to stuff?
Can you extrapolate from such a small data set?
And if our insight is true, what about the third of our respondents who didn’t profess a love of running or cycling? What about them?
There’s a real tendency in our industry to want absolutes.
We research because we, and more importantly, our clients demand definite answers.
Should we have a red button or a blue button?
Do millennials value training and development over salary?
Are carousels a no-no on my website?
The ugly secret is that research won’t give us that definite answer. Because, what it boils down to is the fact that it’s not supposed to. As advertising agencies, all it can do is point us towards something.
It might give us a hypothesis that we can use. Or it can give us an angle to hang our creative idea on. But saying a definite yes or no? I don’t think so.
So let this be a warning. When you’re told that people don’t click below the fold or that they always click below the fold or that Chinese people hate the colour red or curvy button corners are better than pointy button corners AND there are statistics to back it up, there are two things to remember.
1. Even if the statistics point to that conclusion, it doesn’t mean it will be the same when you’re trying to sell YOUR product or advertise YOUR job to YOUR target audience. These will be different EVERY TIME THE TEST IS DONE.
2. Remember the other people that DIDN’T follow that pattern for those statistics.
Because for every Tom, Dick and Harriet who clicked on a curvy button there’s a Dwayne and Jenny who didn’t.
And what if they’re the ones you’re after?