Avoiding 5 Survey Design Traps
Mike thrives on delivering data-driven recommendations and action plans to solve complex business problems. His international research experiences and consultative engagements typically include a mix of qualitative and quantitative techniques. As a well-versed moderator, Mike is focused on better understanding the B2B decision making process, mostly among high tech, telecom, and business service verticals. His expertise often finds him speaking at industry-related events and conferences on topics such as how to conduct research to support the creation of thought leadership content and how emotions influence B2B decision making. Mike's out-of-office passions include global travel, improving his golf game, and sampling Syracuse's rising food scene - which is better than you might think!
Stating the obvious, survey design is key to obtaining quality survey findings. Yet we as market researchers can fall into traps that result in survey designs that are not as high quality as we desire and intend. These traps come in many forms, several of which are described here, together with our suggestions on steps to avoid these traps.
Trap 1: It would be nice to know
One of the most common traps that we can fall into in survey design, is including lines of questioning that are not directly related to the objectives of the research. This can lead to excessively long questionnaires which risk losing client engagement.
To limit this trap:
- At the beginning of the design phase, ask your clients what decisions they are expecting to make based on this research. Then, as the questionnaire is under construction, understand how lines of questioning contribute to these decisions.
- Push back when a questionnaire is becoming too long. We are the survey experts and we need to let our clients know that they will not get quality information if they lose respondent engagement…a frequent problem with lengthy questionnaires.
- If the questionnaire is just too long, see if the sample base can be randomly divided with only some “nice to know” lines of questioning given to each subsample.
Trap 2: Everyone knows what this means
This is a trap that we can run into particularly when working with product managers. It is not uncommon for product managers to believe that the respondent base is familiar with the terms related to their products. However, our experience suggests that even in B2B research with decision-makers, there is often no consistent understanding of terms.
To limit this trap:
- Identify the terms that you believe may not be understood by all. (You may be the best person to pose these questions, particularly if you are not the product expert.)
- Ask clients to define these terms. (We often find that we hear different definitions from members of the same client team.)
- For terms for which a definition is not obvious, define the term in the questionnaire to make sure that all respondents have a common definition.
Trap 3: Let’s let respondents fill in the blank
This is a trap that is particularly common in pricing research. Clients are often hesitant to define a price range to test, concerned that they will either set the range too high or not high enough.
To limit this trap:
- Push clients to identify their expected price point or price range, either based on competitive or substitute offers, costs associated with the offer, or some other way.
- If clients are insistent in understanding respondents’ top-of-mind willingness to pay, follow up by testing a specific price point or price range to get a common reaction to these prices across respondents.
Trap 4: Respondents will remember
This trap can come in at least 2 forms, believing that:
- Respondents will remember a definition or explanation after reading it once.
- Respondents will remember responses that they gave previously.
As researchers, we need to remember that respondents may not be giving their full attention to the survey that they are completing. Therefore, recall of previous information or ratings may not be complete. Even the most diligent responder is not likely to remember the exact rating provided on an earlier question, particularly if the rating scale is 7 or more points. This provides challenges in pre/post measuring of product interest, appeal, likelihood-to-pay, etc.
To limit this trap:
- In response to the first trap, include mouse-overs in online surveys when using terms that were previously defined and may not be obvious by their name. For phone surveys, define the term at least the first two times it is used.
- For the second trap, if you are asking a question pre/post, here are a few options:
- remind respondents of their previous rating
- use an anchored scale which is 5-points or less
- If using an unanchored or larger scale, combine ratings before analyzing (e.g., top 2/bottom 2; or top 3/bottom 3)
Trap 5: We’ll develop the analysis plan after we field
This is a trap that is easy to fall into, especially given the frequent time pressure that is placed on researchers to get projects underway so that findings can be available quickly. Often in the press of time, we think that we’ve covered everything and we will finalize the analysis plan after we field. However, understanding the analysis plan before finalizing the questionnaire should limit the “I wish I had asked this” comment when you go to actually
analyze the data.
To limit this trap:
- Sketch out the analysis plan based on the final or near final questionnaire design.
- Before fielding, review with your clients the outputs of the analysis plan to confirm that it will meet their needs.
While there is much more to questionnaire design than what has been discussed here, being mindful of these 5 common traps will place researchers well on the path to having a high quality design.
