Surprise! Survey Respondents Don't Always Tell the Truth

 "People don't think how they feel, they don't say what they think, and they don't do what they say."

David Ogilvy, Founder of Ogilvy & Mather

Research surveys have become a prominent feature of the B2B marketing landscape in recent years. The growing need to produce effective thought leadership content, and easy access to free or inexpensive survey technologies have led many B2B marketers to make surveys an integral part of their marketing efforts.

Today's survey technology tools make it relatively easy to create survey instruments and conduct online surveys. But these tools don't eliminate the challenges involved in designing surveys that produce accurate results. In reality, survey research is a complex topic, and expertise is required to do it well. The validity of survey data can be affected by numerous factors, many of which are not intuitively obvious.

This is my third post dealing with a range of issues relating to research surveys and survey reports. The first two posts in the series are available here and here.

In this post, I'll focus on a group of issues that can affect the accuracy of survey responses. These issues are by no means all of the factors that can impair the validity of survey data, but they are important for marketers to understand both as producers and consumers of survey-based content.

Response Biases

Experienced researchers have long been aware that survey respondents don't always answer survey questions accurately or truthfully. Response bias is the general term used to describe several tendencies of survey participants to respond inaccurately to survey questions. These biases can be conscious or subconscious, and they can have a significant impact on the validity of survey data.

Response biases can occur for a variety of reasons, and social scientists have defined several forms of response bias. Here are some of the more common forms.

Acquiescence bias (sometimes called agreement bias or yea-saying) - Acquiescence bias refers to the tendency of some survey participants to select a "positive" response option (e.g. agree rather than disagree) regardless of their actual preference or opinion. The bias is more likely to exist when a survey question asks for an opinion and presents two opposite choices, such as agree/disagree or true/false.

Demand characteristics - This term refers to a type of response bias where survey participants alter their responses simply because they are taking a survey. Social scientists argue this bias arises because some survey participants will try to determine what the purpose of a survey is and then subconsciously change their responses to fit the perceived purpose of the survey.

Question order bias - This term refers to the tendency of some survey participants to respond differently to questions based on the order in which the questions are asked. For example, research has found if survey participants are first asked about their general interest in a given subject, their responses will indicate a greater interest than if they are first asked a specific or technical question about the same subject.

Extreme response bias - Extreme response bias refers to the tendency of some survey participants to only choose the most extreme response options available for survey questions. For example, if a survey provides a series of statements and asks participants to rate their agreement or disagreement with each statement using a 5-point scale, some participants will only give ones or fives as answers.

There are several steps survey designers can take to mitigate the impact of response biases, but none of these techniques is likely to be 100% effective.

The Social Desirability Bias

One type of response bias has become especially relevant for marketers in today's business environment. The social desirability bias refers to the tendency of survey participants to answer survey questions in a manner they believe will be viewed favorably by others versus what they actually think, feel or do.

This bias is more likely to exist when survey questions relate to sensitive personal topics - such as alcohol consumption or drug use - or "hot button" social topics - such  as climate change or racial diversity. The social desirability bias can lead to the over-reporting of "good" or socially acceptable attitudes and behaviors and the under-reporting of attitudes and behaviors survey participants perceive will be viewed as "bad" or socially unacceptable.

It's important for marketers to be aware of the social desirability bias because a growing number of pundits are now contending that corporate social responsibility has become an important aspect of customer buying decisions, and that therefore companies should include "social purpose" messaging in their marketing programs.

At first glance, this argument appears to be supported by a significant body of research. Numerous surveys conducted over the past several years have found that many people - particularly younger people - have become more socially conscious, and that they increasingly expect business organizations to take a more active role in addressing social problems. Here are a few example of this research:

• In Edelman's Trust Barometer 2021 survey of consumers in 27 countries, 86% of the respondents said they expect the brands they buy to take one or more of several actions. These actions included address social challenges, create positive change in society, address political issues, and make our culture more accepting.
• In Accenture Strategy's 2019 Global Consumer Pulse survey of consumers in 36 countries, 74% of younger consumers (basically millennials and Generation Z) said they want companies to take a stand on issues "close to their hearts," and more than 50% said they have shifted a portion of their spend away from their current service provider when a company disappointed them due to its words or actions on a social issue.
• In a 2017 survey of consumers in 16 countries by BBMG and GlobalScan, nearly two-thirds (65%) of the respondents said they want to support companies with a strong purpose.

Social purpose marketing is a controversial subject, and my intent here is not to take either side of the debate. But we know the social desirability bias exists, and therefore some surveys may overstate the importance or value of social purpose marketing. In addition, we also know there is often a discrepancy between what survey participants say they believe or will do and their actual behaviors. That phenomenon is called the "value-action gap," but that's a topic for another blog post.

The real takeaway here is that marketers should be cautious about relying on survey research that is susceptible to the social desirability bias.

Image courtesy of Chris Short via Flickr (CC).