Perhaps you’re a classically trained researcher who’s had it hammered into you that a question’s wording, format, context and placement can impact its result. Thus you vigorously guard the sanctity of your tracking instrument, modifying only the “special issues” section toward the back. Or maybe you flex to the pressures of day-to-day business and make other, earlier changes… then pray.
Perhaps you understand that different respondent sources, even when demographically matched, can yield different results. Thus you carefully avoid sampling fluctuations over time. (Of course, as sources themselves change their business models and sourcing strategies, sometimes you SHOULD adjust… carefully).
But did you realize that keeping your long-standing tracker unchanged may, in fact, be causing the results to shift for methodological reasons, misleading you into thinking that your brand or category is shifting (or staying stable) when in fact it’s not?
This may seem impossible, but it’s true, and deep down you already know why… who is doing surveys and how they are doing them is changing as the world continues its meteoric and inexorable march toward the dominance of mobile devices.
If you’re an industry veteran, like me, maybe you’re feeling a sense of déjà vu … that you’ve been through this before, having been forced to make the painful choice of abandoning years of landline telephone history to shift to the emerging online data collection platform.
The pain of that transition was mitigated by the silver linings… lower sample costs, the ability to get more answers in the same amount of respondent time, and the opportunity to improve the quality of recognition and recall measures through visual stimuli like logos, packages and ads.
Unfortunately, the next wave of method shifts toward self-administered smartphone surveys will be accompanied by a swing backward toward the constraints of interviewer-administered telephone surveys… a need to ask fewer, simpler questions. I think it is these additional trade-offs that have led the industry to be slow to act in response to this latest shift in respondent behavior.
While mobile-only research has many exciting new capabilities (location-sensitivity, passive metering, camera, etc.), the world hasn’t swung THAT far in the direction of smartphone adoption….yet. We’re in that awkward in-between stage right now. But just like teens grow into adulthood, this transition is SURE to occur. Just as you would be crazy to launch a new tracker that is not smartphone-friendly, you run tremendous risks if you do not convert your existing trackers into smartphone-compatible ones.
If your tracker is several years old, is too long or complex, or is being run by a company that lacks the tools to auto-adjust survey display to the respondent’s device, you are almost certainly experiencing shifts in the sample composition of your surveys (not on demographics, which can be easily controlled, but on more crucial psychographic and behavioral traits less easy to detect). The problem is that your well-established, multi-year tracking survey probably can’t be taken on smartphones (or shouldn’t be).
so what?® It may be time to invest in a side-by-side transition study during which you reimagine your instrument for the modern world and wean yourself off the old instrument that isn’t as trendable as you thought it was. After all, though a historical view is nice to have, getting an accurate current view is much more crucial to business success. In 2016, it is the rare product category that can reflect its current marketplace without smartphone-reliant respondents in the study.
I’d be happy to help you with this difficult transition.
You must be logged in to post a comment.
Hilary, very smart observations as usual. LRW is leading the charge to bring trackers into the 21st century. Trackers must evolve or face the usual consequences of not keeping up with the times.