Most of us love our mothers dearly. But that doesn’t mean we go to them for advice every time we need answers to some important problem in our lives. Sure, it would be easy, quick, and cheap to get a few words of wisdom, but it is just not realistic to expect them to be objective. Thousands of bitterly disappointed American Idol contestants learnt this fact the hard way.
And so it is with some online surveys. It is now easy to throw together a web-based questionnaire, get it sent to a bunch of people, and have answers back all within a fortnight. But if the people who received the survey are skewed on some important dimension (e.g., technologically literate, mostly young, mostly employed, etc.) you can’t expect the results to accurately reflect the opinions or likely behaviours of a more diverse group. The technical term for this sort of bias is coverage error and it is one of the key reasons to think carefully about how you select the sample for a survey.
There are two very general categories of survey sampling techniques:
- Non-probability sampling: You don’t specifically go out to get a random selection of people from your target group. Instead, you let allcomers complete your survey. Perhaps you send an invitation out to your friends and ask them to invite their friends, etc. Or perhaps you advertise the survey and let anyone who happens to see the ad fill in a questionnaire. These surveys have all the objectivity of talkback radio. They might be entertaining, but you wouldn’t usually base a policy or business decision on them.
- Probability sampling: You make an attempt to get a random selection of people from your target group completing your survey. In the ‘holy grail’ version of this approach, you’d have a list of all the people in your target group, take a simple random sample from the list, send the invitations to the sampled people and then follow up to get as many of them answering as possible. Survey researchers and statisticians have developed lots of variations on this theme to take account of practical issues, but the aim is always to get a wide mix of people from the target group responding. Although your results under this approach won’t be perfectly accurate, you can be confident that you’ll come close to reflecting the opinions and behaviours of the full group.
Sounds clear enough, doesn’t it?
And it is. Until we enter the wild world of internet survey respondent panels. You see, it is possible to order up a random selection of people from a panel that makes you feel like you are taking a probability sample when really your results may be subject to the sorts of coverage errors inherent in a non-probability sample. This is because many panel providers build up their lists of eager members by non-probability methods. Few providers source members via a random (or pseudo-random) process like Random Digit Dialling or Address Based Sampling because it is so expensive to do so. Even fewer provide internet access to those households who don’t have it. Knowledge Networks is one company that does these things.
Predictably, a recent study titled Study Finds Trouble for Internet Surveys highlights the differences in accuracy that arise from the different panel recruitment approaches (probability vs non-probability). Here are some selected excerpts:
In the most extensive such analysis to date, David Yeager and Prof. Jon Krosnick compared seven non-random internet surveys with two others based instead on random or so-called probability samples. The non-probability internet surveys were less accurate, and customary adjustments did not uniformly improve them.
While the random-sample surveys were “consistently highly accurate,” the internet surveys based on self-selected or “opt-in” panels “were always less accurate, on average, than probability sample surveys, and were less consistent in their level of accuracy,” the researchers said. Further, they said, adjusting these samples to known population values had no effect on accuracy (and in one case even worsened it) as often as that process, known as weighting, improved it.
In the most extensive such analysis to date, David Yeager and Prof. Jon Krosnick compared seven non-random internet surveys with two others based instead on random or so-called probability samples. The non-probability internet surveys were less accurate, and customary adjustments did not uniformly improve them.
While the random-sample surveys were “consistently highly accurate,” the internet surveys based on self-selected or “opt-in” panels “were always less accurate, on average, than probability sample surveys, and were less consistent in their level of accuracy,” the researchers said. Further, they said, adjusting these samples to known population values had no effect on accuracy (and in one case even worsened it) as often as that process, known as weighting, improved it.
So, be wary when purchasing a “random” or “representative” sample from an opt-in panel provider. Such a sample might be fine for your particular purpose or target group, but you need to at least consider the risks of coverage error you are taking. And don’t expect weighting to magically solve any coverage error you do have!
_____
Short URL for this post: http://wp.me/pnqr9-1t
davidwallacefleming 9:00 on Friday, November 11, 2011 Permalink
Valuable information to stay appraised of. Thank you. I hope this does not get implemented.