What Political Polling Doesn’t Say About Market Research

Just as in 2016, political polls largely swung at the blue, red, and orange-colored ball and missed. Just before the election, Biden led the polling averages by 8.4 points in Wisconsin and 7.9 points in Michigan. As we now know, he ended up winning both states, but by margins closer to 0.5% and 1% respectively. 

How could they have been so wrong again, especially when they had four years to adjust weighting, rework algorithms, and revamp data collection efforts? One oversimplified answer: the polls failed to account for the complexity of a rapidly changing electorate. They potentially underestimated voter turnout, which was at its highest in over a century, and failed to translate static polling data into accurate forecasting.

In the market research industry, we are again hearing cries of doubt over the utility of survey-based research writ large. Insights teams are no doubt feeling the same skepticism, along with pressure to prove their value. 

Let’s be clear. The doubts many people express with regard to quantitative surveys (i.e., “polls” in the political world) are understandable. But just as misinformation spreads by deliberately conflating and combining the similar (mistaking sample ballots burning for the real thing, for example), the same is true as it relates to insights. 

That’s why I want to dispel some understandable confusion between several related, but distinct entities: a poll, a forecast, and an insight. 

The critical difference between polling and forecasting

A poll is a politically relevant label for what is essentially a survey of claimed sentiment and/or anticipated behavior among a sample of people selected (and often weighted) to represent a particular population (e.g., eligible likely voters; TV streaming subscribers; those with kids in the household under age 12; etc). 

At the most fundamental level, the result of a poll is descriptive. Of course, we expect a lot from these descriptions. Rarely is a survey’s reason for being merely to describe a sentiment. And more often, our reason for gathering such descriptions is to facilitate a prediction of the future.

A forecast is not a description of a present sentiment. It is a claim about what is likely to happen in the future based upon information we know in the present — information that often includes descriptive survey results, but often also includes many other types of information. Forecasts come from a model for the future based on a variety of information known in the present.  

Why forecasting is so much harder than polling

Because polls are often conducted to facilitate a forecast, and because forecasts often rely on polls as one of their essential ingredients, they are easily confused — but they are worlds apart.  

Consider what that world looks like between the incidence of people described as “intending to vote for candidate X” in a United States presidential election at one moment of time, and whether candidate X is elected at the future point in time. What are all the other factors a forecast may need to incorporate beyond the mere “intend to vote” descriptive? Just a few examples:

• The likelihood that intender will be distracted from voting by some unforeseen personal issue (e.g., flat tire, sick child to attend to, the lines at the voting booth are too long, etc.)
• The likelihood that their choice of voting method will be officially tabulated in the election (e.g., due to post office problems, making a mistake on their ballot, legal challenges, etc.)
• The presence and severity of a public health crisis (like COVID-19) and an intender’s perceptions of the crisis, which may or may not be related to their political preferences
• Something happens between the survey and election to change their mind (e.g., a conversation with a friend or campaign volunteer, exposure to an ad, an intimidation faced at the voting place, etc.)
• Whether the answer provided on a survey was intentionally a lie, due either to annoyance at being surveyed or a politically-motivated desire to mislead (e.g., “troll,” debase the accuracy of politically inconvenient polling expectations, etc.)
• The likelihood that a non-intender will become motivated to vote for any combination of the above factors and/or other reasons
• A “herd effect” causing some people to act differently upon their perceptions of how others will act, possibly due to their exposure to published findings of a poll!

Any of these and other elements may or may not be factored into an election forecast model to varying degrees, even interacted with one another, along with various assumptions and randomness. A poll descriptive is merely one of those many elements. Heck, a poll may be one of many polls used as just one element in a model with many dynamically varying elements.  

Even more confusing, the output of a forecast is often given in probabilistic terms which, being percentages, are optically identical to poll results. For example, candidate X has a 27% chance of winning the election vs. 52% of eligible likely voters say they will vote for candidate X. These are apples and oranges, except the oranges are the same color as the apples.

For such models to work “well,” all the appropriate factors, interactions, assumptions, and processes need to be taken into account and finely tuned — beyond just the quality of the polling ingredient. When a celebrity forecaster or news organization’s probabilistic claim of election victory doesn’t “come true,” it’s not that the polling ingredients were necessarily “wrong” (though a particular cook may be working with some ingredients of poor quality), it’s that ingredients weren’t used in the right way along with all the other ingredients.  

How insights sheds light on the complexity of human behavior

Where polls (descriptive surveys) and forecasts (predictive models) are ingredients, insights are the flavors and textures that result from their skilled use. For those of us working in the modern era of marketing research and marketing services, gone are the days of merely providing a descriptive result of sentiment or behavior, or providing just a forecast of future behavior or sentiment. These are merely a few of the many sources of information we have for making sense of the world and helping others better navigate that world to achieve their goals. 

In a resource constrained world, no one — no company, no country, no political candidate — has unlimited money, attention, energy, or time. But we exist in a world in which we have agency — a sense of purpose that we are free to determine, which corresponds with the decisions we make about how to use the limited time, energy, attention, and money we have. But as we know, no one makes decisions absent an array of rich inputs, psychological predictors, and environmental factors that affect human behavior. 

People, organizations, countries are free to go at it alone, in a vacuum of descriptive information about the state of the world in which they exist, in a vacuum of expectation of how the world may be in the future, but why would they? The business of insights is about so much more than describing a sentiment or predicting an outcome. It’s about helping people make sense of a complex world and make better decisions toward achieving their purpose.

Yes, the polls “got it wrong,” and methodologists across the political spectrum are surely scrambling to address the inaccuracy of their predictions. But that’s not the entire story. Insights is the set of tools that helps us unpack the human factors behind that story to help guide how we, as individuals and organizations, make decisions in an increasingly complex world.