Obviously, it’s becoming increasingly important to be able to critically assess research and data for decisions – both to make decisions in the first place, and to evaluate the decisions we’ve made. But that certainly doesn’t mean we’re all great at it, nor does it mean we develop and sustain good practices around it. And sometimes the drive to either use research for marketing purposes, or to push research out the door to get to the next client or project, seriously undermines the process.
This study, for instance, has a very positive-sounding headline – “Three years of consecutive growth” sounds great, right? But when I dig into it, I find an industry growing more slowly than overall GDP, the number of companies participating going down, and slower revenue growth than last year. These may not be disastrous statistics, but I don’t know if they’re quite as sunny as the headline would have us believe. I’d want to find out more.
I recently read a report from a survey company where they seemed to have plugged the results of the study into a boilerplate report and fed that back to the client we had in common. I saw some really disturbing trends in the study, but they were buried in the blandness of the writing. The step of actually thinking about the implications for the client was entirely absent.
Neither of these approaches gets us where we need to be to make and evaluate our decisions. So, what to do?
Suggestions for when we’re consumers of data for decisions:
- Never just assume it’s good data. How was it gathered? What was the methodology? Was there adequate sample size?
- Look at the source data, and insist on having as much of it as possible provided to you in raw form. Anonymity may need to be preserved, but there are ways for you to get the raw data without violating any confidentiality. I’m often amazed to hear about clients doing research for years and not having the underlying data because the vendors haven’t provided it.
- Read reporting critically. Are the verbal conclusions really supported by the underlying data? Are they telling you the negatives as well as the positives?
- Translate the results. Is your provider tailoring the report specifically to you and the findings from this particular study? Are they making connections between topics? Are they finding – and explaining – paradoxes based on their knowledge of your organization or business or industry? If they aren’t, and frequently they don’t, then you’ll have to do that yourself. (And if they really can’t, consider finding another vendor who can.)
- Come at it from different angles. Share the data amongst yourselves, invite others (from other parts of your organization, or even from outside) in to take a look at it, kick the tires, and think about what it might mean.
Suggestions for when we’re providers of data for decisions:
(And note, these hold whether you’re internal to the organization or a third party.)
- Don’t overpromise. Don’t claim you can get samples you can’t, validity that’s not going to be possible, and so on. Be honest about how far the budget or methodology will take you.
- Don’t oversell what the data is actually telling you. It’s OK to put some constraints on the conclusions you can draw from the data. If it’s only directional or even not that, for crying out loud, say so. If people want a service, that doesn’t mean it would be successful – unless you also did analysis on the cost to provide the service (and not just the expressed interest in it), you can’t comment on whether it should be provided.
- Don’t stop with merely reporting the numbers. Help your client translate the results to their own situation. Use what you know about them to help them find the useful points.
- Embrace the notion that any clarity is a good thing, even if it tells a client what they don’t want to hear. If people hate us, we need to know. If the people who love us only love us for one thing, we need to know that, too. It’s the basis of what you’re getting paid for – again, whether you’re internal or external. Don’t be timid.
- Make it relevant. Do the research on your client to understand at least at a high level what they will and won’t be surprised by. Don’t force every report into a boilerplate.
If there are ways we can help you collect or make sense of data to make good decisions, please get in touch.
Photo by Rohit Mattoo via flickr, used via Creative Commons license.
