Conducting an online survey? How to make sure you don’t get bad data

Skipped questions. Contradictory responses. Irrelevant answers.

These kinds of responses are bad for business. They skew survey results, so companies can’t get the answers they need. And, unfortunately, survey responses like these are becoming increasingly common as online surveys gain in popularity.

This doesn’t mean you should stop conducting online surveys, but it does mean that market researchers need to take extra steps to ensure data integrity.

When data is bad, either the survey results need to be thrown out, or companies mistakenly use the results and make misinformed business decisions. Either way, getting bad data is a waste of time, energy and money.

Online surveys: Great for market research, but added challenges

The ability to conduct online surveys has been a major advancement in market research. Quickly reaching thousands of respondents, data is easily collected and tabulated. Surveys can be tailored automatically based on real-time responses. Cost and time savings are immediate.

However, market researchers do miss out on some of the advantages of the “old days,” particularly the ability to directly observe and react to the respondent. For example, during a telephone interview, if a respondent was having trouble understanding a question, the interviewer could immediately step in to clarify. The researchers could also monitor calls and identify any issues with the survey questions.

With online surveys, it’s much more difficult to get that real-time interaction. The respondent opens up the survey, alone, on a computer monitor, while he/she is at home or at work. Perhaps there is an acronym the survey taker doesn’t know, or a product description isn’t clear enough for the respondent to provide meaningful feedback. He/she attempts to answer nonetheless, which often leads to nonresponsive answers. In addition, it’s more difficult to screen respondents when they’re recruited online. Some examples of survey takers who can provide bad data include:

• Fraudulent respondents – misrepresent themselves and provide inaccurate information, especially demographic information
• Inattentive respondents – not attentive when taking the survey; do not provide thoughtful answers to survey questions 
• Professional survey takers – may belong to more than one panel or take numerous surveys each month to collect incentives

Ensuring data integrity

Given these challenges, how do market researchers make sure the data is accurate? Here are three methods that KS&R uses to ensure that our clients get accurate data from their online surveys. 

1. Conduct a Pre-Test 

Pre-tests are invaluable. While they require extra time and money, the costs of not pre-testing can be much greater. To pre-test, the survey is given to a small group of respondents (in person or on the phone) while researchers observe and follow-up to uncover any issues. Did they encounter any problems? What was unclear, ambiguous or confusing? How can the survey be improved? Using that feedback, we then re-write survey directions and questions to ensure that it’s intuitive and relevant to respondents.

2. Set-up Traps 

By asking innovative questions, we can catch the “bad” survey takers (those fraudulent, inattentive and/or professional respondents). Here are a few examples:

• Ask demographic questions twice. Collecting demographic information at the beginning and again at the end of the survey gives fraudulent respondents an opportunity to trip up.
• Add “red herring” answer choices. Including answer choices that are not legitimate options – i.e., non-existent products, retailers, medical conditions – is very effective in identifying “bad” survey takers. For example:

Which of the following IT vendors provides your company with Intel-based servers? 

1. HP
2. Technology Frontier
3. IBM 
4. Dell

In this example, Technology Frontier does not exist. Respondents who select it should be considered for removal.

• Tell the respondent to pick a certain answer. To help detect inattentive respondents, a simple question included more than halfway through the survey can help flag them. For example:

Please select the number three in the list below. 

1. 1 
2. 2
3. 3

Respondents who do not select 3 should be considered for removal.

3. Establish Data Collection Red Flags

Because online surveys can be programmed and quickly analyzed, it’s a great opportunity to identify “red flag” data. Some examples:

• Quick takers: Using a time stamp, it is easy to see if respondents sped through the survey. Survey takers who complete a survey in less than 1/3 of the average time typically provide unusable data.
• Duplicate entries: Collecting an IP address from respondents identifies respondents who took the survey twice.
• Straight-lining: When a respondent selects the same response (i.e., all A’s, all “very satisfied”) for many questions right in a row, it might indicate a fraudulent survey taker. While the responses can be valid, a more detailed examination can identify respondents who are either fraudulent or simply rushing through the survey.
• Inconsistent or contradictory responses: When respondents consistently fall for the “trap” questions listed above; provide open-ended responses that are not meaningful; and have outlying answers, it’s important to analyze their completed surveys for possible data contamination.

Better data is worth the effort

Setting up pre-tests, adding trap questions, and screening and analyzing for red flags are just a few of the methods we use to identify data that could muddy your survey results. A clean data set is a critical starting point for the information and insights you need to move your company forward.

Do you have questions about preserving the data integrity for your online surveys? Contact KS&R to learn more.