Blogger’s Disclosure: the following blog entry may be considered "old school" or even "outdated" to some – I beg to differ!
When I first started out in market research 20+ years ago, we regularly pre-tested quantitative surveys with a smaller set of respondents before starting data collection to ensure the questionnaire was well understood, engaging, and promoted thoughtful responses from participants. We ascribed to the adage "if you're going to do something, you might as well do it right". In our experience, pre-tests are a rarity in today's fast-paced, I've got to have the answer yesterday mentality where budgets are tight and patience is thin.
Let's face it, pre-testing is not an enjoyable experience for most since you're really looking for what's wrong with your survey (although what's right about it is just as important). I mean, would Picasso step away from what he thought was a completed painting and embrace the opportunity for other people to find flaws in it. Perhaps the analogy is a bit overdone, but researchers tend to find perfection rather than imperfections in their surveys even though our job is to get at the "truth" through a crisp, well though-out, and highly engaging survey. Dare we think our survey doesn't pass muster on any of these accounts.
There is also a certain out of sight, out of mind mentality to survey research nowadays, particularly as we continue to move more and more to online data collection. We can't be in the minds of all the respondents as they complete a survey, but since it is presumably so well written (see pre-testing isn't fun above), then we take a leap of faith that everything is "fine", and of course respondents have complete clarity about what they're being asked and are highly engaged throughout the entire survey.
Finally, and perhaps more to the point, pre-testing is no longer in vogue (dare I say it's "uncool"). Who wants to take a stand and say we need to slow down (just a little) to make sure we've got this one right because too much is at stake if it isn't. I've silenced an entire room taking that stand on various occasions over the last few years" (for the record, also "not fun").
These are all fair points, or at least honest ones. And I've certainly missed an opportunity to pre-test a survey at different times in my career when my brain and my gut told me to insist on doing so – we all have! However, and it's a BIG however, we as researchers need to bring pre-testing back into the mainstream as a standard practice, at least in the following situations:
Although there are honest and practical reasons that pre-testing has fallen out of favor over the past several years, the truth is the downside risk of not executing a well thought out pre-test in certain situations is too great to dismiss. If there isn't a little extra time and budget to pre-test, an even more honest question might be whether the research is worth doing in the first place.
Note: in a future blog, I'll provide suggestions to execute effective pre-tests when time and money is tight (as it usually is!)