8 examples of biased survey questions

Customer Experience

background_image background_image

Whether you intend to or not, you may be introducing bias in your customer surveys. Writing your survey in a way that only accepts certain answers or responses can provide inaccurate data, which can lead to bad decisions. 

Biased survey questions don’t provide the honest and genuine data your business needs. You end up with inaccurate customer feedback that does not consider all reasonable opinions. If you make a core change to your business as a result of collecting feedback on a biased survey, it may not meet the needs of your customers. 

In this article, we’re looking at the different types of biased survey questions so you can identify biased questions before they affect your data and analysis. We’ll discuss the following:

  • What is a biased survey question?
  • What is a biased answer?
  • 8 Examples of biased questions
  • How to avoid survey bias
  • Idiomatic for analyzing survey data

What is a biased survey question?

A biased survey question is one where the respondent is led or forced to a specific outcome. It doesn’t allow for all topics and expressions of feedback. It could be that it’s not flexible enough to cover all responses (with limited answer options for multiple-choice questions) or that the way survey questions are written encourages specific responses or perspectives. 

Biased survey questions will often:

  • not help you get accurate answers and results
  • have a higher drop-out rate
  • won’t give you the actionable data you need to make changes to increase customer satisfaction.

Read these examples of good survey questions: 130+ effective survey question examples

What is a biased answer? 

A biased answer is when the answer is not truthful, intentionally or inadvertently. 

When your survey was not conducted ethically and with neutrality, you’re likely to get inaccurate responses that won’t represent the actual customer experience. A good survey leaves no room for response bias, so that you can better interpret and analyze your survey results

8 Examples of biased questions

How you write your survey can introduce intentional or unintentional survey bias. Here are some types and examples of biased questions to help you avoid these in your next customer survey:

1. Acquiescence and agreement bias

Not everyone is excited to fill in a survey. Whatever their reason, some may rush through the survey and not give honest answers, known as having an acquiescence (or agreement) bias. They are more likely to choose the more positive-sounding answer in these cases.

Acquiescence bias example: These biases come into play when the customer is “forced” to complete the survey before moving forward with the next desired action, if they are answering to win a prize, or if the survey is boring and they just want it done.  

How to avoid acquiescence bias:

  • Never force someone to complete a survey.
  • Keep your survey short.
  • Vary your question types, so people don’t get bored. 

2. Loaded question or assumptive bias

A loaded question is based on assumptions without supporting data to back up the assumption. You are forcing someone to answer a biased question that they may or may not actually be qualified to answer or that is untrue, as in our example below.

Loaded question examples:

How happy were you with the event?

How to avoid loaded questions: There are a few ways to fix a loaded question. One is to ask a clarifying question first, and if they choose a specific answer, they are shown a follow-up question to get more details. In this case, the question is no longer assumptive because they made their opinion known in the previous question. This second survey question would not appear to anyone for whom the survey hasn’t already confirmed the assumption first. Below are two ways to fix this biased question example:

Question 1: Did you enjoy the event?

Question 2 if respondent answers yes to #1: Rate your satisfaction with this event

[Rating scale from 1-5]

Or

Question 1: Rate your experience at the event.

[Rating scale from 1-5]

3. Leading question bias

Leading questions use language or placement that consciously or subconsciously pushes the respondent towards a specific, certain answer. For example, these might be questions where all the possible responses are positive responses. In this case, someone with negative feedback has nowhere to respond. The analyzed data for these questions will look perfect because there is no possibility of a negative response.

Leading questions can also be written using language meant to influence survey respondents to answer a certain way. Examples of both are below. 

Leading question examples:

Were you satisfied with this new feature?

  • Yes
  • Somewhat
  • Usually
  • Often

Don’t you think this was a positive addition to the platform?

  • Yes
  • No

How to avoid leading question bias: It can be tricky to identify leading questions in your own work due to your own bias to collect as much positive feedback as possible. Have a colleague help you fix leading questions by reviewing your work. Or, you can review each survey question to ensure you have an even mix of positive and negative response options. 

Were you satisfied with this new feature?

  • Yes
  • No
  • Undecided/Not Applicable

How useful did you find this new addition to the platform?

[Rating scale 1-10]

4. Dichotomous question bias

This question type (also known as absolute questions) is not helpful as the answers may only account for some possible answers or interpretations. Dichotomous questions usually have only two possible answer options (commonly yes/no, or true/false). 

These absolute questions can skew your data. You can often identify an absolute question as one that uses words like: all, must, every time, always, and never.

Dichotomous question example:

Do you know all about SEO?

  • Yes
  • No

I always find logging into my account easy.

  • True
  • False

How to avoid dichotomous question bias: Consider rephrasing the absolute question or offering a rating scale or additional options to cover other scenarios. Sometimes adding a “does not apply” or “other” response option is necessary:

How confident are you in your SEO knowledge?

[Scale from 1-10]

How satisfied are you with the process of logging on to the platform?

  • Dissatisfied
  • Somewhat dissatisfied
  • Indifferent
  • Somewhat satisfied
  • Satisfied
  • Does not apply

5. Double-barreled question bias

A double-barreled question confuses your respondents. These questions try to solicit two unrelated responses within one question. Putting these “combined” questions in surveys to shorten the survey length can be tempting, but they introduce inconsistent interpretations.

Double barreled question example:

Are you satisfied with the product pricing and features?

[Text Field]

How to avoid double barreled questions: Separate the questions. Focus on one aspect or feature in each question:

Are you satisfied with the product pricing?

[Text Field]

Are you satisfied with our product’s feature set?

[Text Field]

6. Negative and double negative question bias

These biased survey questions also confuse the respondent. Negative biases are when they are forced to respond “No” to mean affirmative and “Yes” as the negative answer. Similarly, introducing double negatives also confuses them as they spend more time untwisting the double negatives and trying to figure out what their response is. 

Negative and double negative question examples 

I don’t think this feature should be free.

  • Yes
  • No

Do you disagree with not allowing users to edit their profile name?

  • Yes
  • No

How to avoid negative questions biases: Keep the questions brief and focused on the positive to avoid confusion and bias:

Should this feature be free?

  • Yes
  • No

Should users be allowed to edit their profile name?

  • Yes
  • No

 7. Open-ended question bias

Some might consider an open-ended question a bias because the answers may be interpreted differently. These types of survey questions are essential and valuable when used in the right way. 

However, consider whether the response is better in a different format (like multiple choice answers or rating scale) for easier analysis. Some respondents may not want or have the time to think of an answer and write it down, so you may get a lower response rate to open-ended questions.

You can also use a survey analysis platform like Idiomatic to help you analyze and use machine learning to read and pull commonalities in open-ended survey question data.

Open-ended question example: 

Tell us about your experience with our customer service team today.

[Open ended text box]

How to avoid open-ended question bias: We’re not suggesting to avoid this question type altogether. First, determine if the question could be better analyzed with a different question type. If not, include open-ended questions, but try not to overwhelm your survey with them.

Did your customer service rep solve your problem today?

  • Yes
  • No
  • Not sure yet

If you answered no or “not sure” please provide more details:

[Open Ended text box]

8. Vague question bias

The easier and quicker you make it for someone to complete a survey, the higher your response rate will likely be. If you have questions that people have to think about too much before answering, they’re not likely to respond. This includes if you write biased survey questions that don’t provide enough detail, where the respondent may have to make an assumption to answer. These vague questions may not have a focus or use too much jargon.

Vague question examples 

Are we better than other companies in our niche?

  • Yes
  • No

Do you think your family members would like this product?

  • Yes
  • No

How to fix vague questions: Try breaking the question into multiple, more focused questions. Or, re-word questions to ask a more specific (less general) question:

Is our customer service team easier to contact than other software companies you’ve worked with?

  • Yes
  • No
  • Doesn’t apply

How likely are you to recommend our product to a family member?

[NPS scale]

How to avoid survey bias 

As a survey creator, are you doing everything possible to remove bias from your customer surveys? Ensuring your surveys have a neutral focus and no vague or ambiguous questions helps you get a more honest response that accurately represents the customer experience.

Avoid survey response bias in your entire survey by keeping each question as focused and clear as possible and using neutral language. This will help you minimize survey dropout and better understand the customer. 

Bias in surveys is often unintentional. If you’ve spotted bias in old surveys or been called out on it, don’t fret. Taking intentional action to ensure it doesn’t happen again is well worth the effort. With the right diversity and inclusion training for your team, your surveys can have more neutrality to them and allow respondents to focus on their answers rather than issues with the questions.

When written with neutrality in mind, open-ended questions can help you minimize the chance of receiving biased responses because respondents offer answers in their own voice and words, not yours. 

Idiomatic for analyzing survey data

Collecting more open-ended questions may mean more effort to summarize and analyze for commonalities in responses. With manual, human-powered analysis, this is true. But, if you use machine learning algorithms like Idiomatic to collect and analyze your data, you get faster, more accurate feedback survey results.

Idiomatic can analyze large amounts of data, even open-ended questions, faster and more accurately than manual analysis. Contact our team today for a customizeddemo to see how Idiomatic can help you gather valuable insights from your survey data without bias. 

Request a demo