Why you should avoid double barreled questions in your surveys

Double barreled questions undermine surveys and ruin your data, yet I still see them a lot — why?

Because they’re quite tempting, and also easy to accidentally use.

Designing effective surveys can be more complicated than it might appear.

One of the most common mistakes that I see folks repeatedly making is including double-barreled questions when trying to get broad feedback.

Double-barreled questions occur when you stack two questions in one, usually with an “and” or an “or” in the middle.

These double-barreled questions confuse users, dilute the accuracy and response rate of your surveys, and can cause major issues in your data.

Let's talk about how to avoid double-barreled questions and how to better structure your surveys for maximum response and accuracy.

What is a double-barreled question?

A double-barreled question is when you include two questions within a single sentence.

You can often identify a double-barreled question as one that has an AND or an OR in the middle.

Examples of the impact of double-barreling

When you're beginning to write your survey, you might be inclined or tempted to double-barrel some of your questions because it seems quite natural.

For example, you might ask a new hire: how was your onboarding process and our onboarding documentation?

In your mind, you're asking them about how their onboarding went.

But in reality, what you're asking is two different questions.

First, how was your onboarding overall?

And second, how were our onboarding documents?

The reality is that the onboarding process could have been fairly good overall, but your onboarding documents themselves could have been a bit more challenging to read and understand.

While maybe the candidate found working with the team and using other resources much easier, combining these into a double-barreled question creates confusion.

Make sure that each question is really only addressing a certain aspect of your business or product.

They can be subtle at times.

For example, you might ask someone, how satisfied are you with our product and our service? In reality, you're asking two very different questions. Your SaaS product might be great, but your customer service might be weak!

More examples of double barreled questions

In another example, you might ask someone: do you enjoy our product and UI?

While these might seem fairly innocuous at first, you're really asking two fundamentally different questions.

You're really asking for their opinion on two different topics. First, your product as a whole. Second, on your UI in particular. They might like both, or they might dislike both, but there's also a world in which they have a unique and differentiated take on each element.

Another example is asking: what's your favorite feature and why?

In this second example, the issue is even more subtle, and has more to do with your response rate and data structure. While the question revolves around their feelings on a particular feature, you risk reducing your response rate by lumping in the ”item selection” of a specific feature with the qualitative query of “why?”

Plus, in your survey results it'll take a little bit more work to segment out the answers.

It’d be better to have these as two distinct questions, so that you can have a single query which collects the specific feature, and a follow-on that collects qualitative text-based feedback

In general, questions might not only have different answers, but also different scales like a 1 to 10, intensity of support, or specific unique feedback. It's important to have single item focused questions with the same response criteria/scale.

The negative impacts of double-barreled questions

You can see how these questions can be a bit problematic. Let me outline a couple of the major concerns that they can cause.

User confusion

First of all, they can confuse users by obfuscating the core question at hand. You are asking two in one, which can make users have to work harder to understand what they need to respond to.

Weakened response rate and data accuracy

This confusion can reduce response rate and answer quality. You want to have users put forth minimal effort to answer the question. Any reduction here jeopardizes your data quality.

Data analysis challenges

Having lumped DBQs can make it more difficult to actually analyze your data. Because you've got basically dual responses within each response box, it's harder to separate out context, trends, and turn the data into actionable insight.

Avoiding double-barreled questions

There are easy ways to avoid double barrel questions and ensure your surveys are clean and easy to answer.

Focusing on single ideas

Each question should focus on one clear idea or aspect. It should not have multiple layers or any ambiguity or confusion-causing elements. When you begin to structure your questions, you should have a single query and goal in mind.

Planning and testing

In your survey structure document, you should have a clear reason you're asking each question AND the exact metric or feedback lever that you're measuring. This ensures that you have clear questions that users will understand quickly and respond to easily — no double barrel questions with multiple goals

Building good habits + practical heuristics

Avoiding double-barrel questions is a practice and a habit that you have to build. But after you've built it, it's fairly easy to follow. Here are some good heuristics to live by.

1. When I write my survey questions, I never include “and” or “or” unless absolutely necessary and with justification.
2. My questions are as short as they can possibly be to get maximal content across.
3. I have clear metrics that I measure with each question and I can easily turn those responses into actionable organized data.

If you don't answer yes to all of these criteria, you're at risk of including double-barreled ambiguous questions in your surveys and weakening your data overall.

Conclusion

It's important to make sure that you're turning all of your survey feedback into actionable insight. With double-barrel questions, your job becomes that much harder. Avoid them at all costs and start getting clearer and more accurate survey data immediately.