Tagged: Survey Methodology Toggle Comment Threads | Keyboard Shortcuts

  • Ben 8:23 on Friday, November 11, 2011 Permalink | Reply
    Tags: , , , , Survey Methodology, Survey Walls   

    Looks like Google is getting into the Survey Business 

    From Neiman Journalism Lab:

    Google appears to be experimenting with a new paywall-esque content roadblock for publishers, and it’s not One Pass. For lack of a better name, let’s call it a “survey wall,” because instead of dollars the system asks readers a question before they can move on to continue reading what they like.

    This could get interesting.  Instead of a standard paywall, people may be able to ‘pay’ for content by answering survey questions.  The publisher gets valuable information it can on-sell to advertisers, and Google dulls the old-media knives that are increasingly aimed at its vital organs. A natural extension of this would be that the publisher would become a survey panel provider of sorts.  Survey companies would be able to buy access to the survey-wall to ask their own questions for a fee-per-answer.  There is also no reason why independent panel companies could attempt to step into the role Google appears to be playing as the third-party technology provider.

    Of course, there are big questions about the quality of data that may come from these distributed surveys.

    • Would people answer honestly?
    • What can reasonably be done with one or two answers from each visitor? (e.g., it would be difficult to examine relationships between more than a couple of variables)
    • Why would we expect people who visit survey-wall sites to be representative of a given population?
    These, and other questions, will keep survey methodologists in business for a while :)
     

    • davidwallacefleming 9:00 on Friday, November 11, 2011 Permalink

      Valuable information to stay appraised of. Thank you. I hope this does not get implemented.

  • Ben 13:57 on Sunday, October 4, 2009 Permalink | Reply
    Tags: , , K.I.S.S., Survey Completion, Survey Methodology   

    Proof that People Appreciate Good Survey Design 

    I’m a huge advocate of simplicity in survey design, especially when a survey is to be delivered online. Yet, when I talk to people about cutting out questions, simplifying response tasks, and minimizing the use of various presentational options (e.g., AJAX), I sometimes get the sense I’m viewed as a spoil-sport. Fortunately, people don’t have to take my word for it. A wealth of methodological research shows that completions and data quality suffer as you stray from following the Keep It Simple, Stupid (K.I.S.S.) principle in survey design (see the links at the bottom of this page).

    Some respondents will also tell you how well (or badly) you’ve structured your questionnaire, although waiting until you get this to do anything is probably leaving things a little too late! Respondents can give feedback on a survey in a couple of ways: by dropping out if they are having difficulty with it, or by mentioning their experience at the end (assuming you give them an opportunity to do so). Here are some examples pulled from three surveys I’ve been involved with over the past 12 months. These went out to general population samples provided by a well-known consumer panel. The topics differed, but the surveys were similar in length – about 35 questions over 15 pages. Two of them involved presenting choice sets as part of a stated choice modelling experiment.

    My intent here is not to take the glory for the results I’m about to present; I took care of the online delivery in these surveys, but the questions and structure were mainly developed by others. I’m using them because I do think the questionnaires were generally well designed. Questions were kept to a minimum, pre-testing was done, and the technology used was as simple as possible.

    First, some selected respondent comments taken from across the three surveys. Many other comments echoed the same general sentiment:

    “Very good survey, was easy to follow and understand.”
    “Thoroughly enjoyed that survey is all I can say.”
    “Clear and simple, well worded – well done, whomever designed it.”
    “It was more interesting than the usual surveys :o)”
    “It was a very simple, well put together survey that was easy to understand. Well done.”
    “I enjoyed doing it :)”
    “This was a great survey, thank you!”
    “I really enjoyed this survey. Very easy to follow.”
    “Great Survey, easy to do and no dumb questions.”
    “Wish they were all this easy to complete.”

    My key take-out points are that a) it is actually possible for people to enjoy completing a questionnaire and b) many of the online surveys people are sent appear to be complex, hard to follow, and sprinkled with “dumb questions”.  Although I’m speculating, I think the “dumb question” comment refers to those that are ambiguous, overly complicated or repetitive (e.g., matrix-style questions) or attempt to psychoanalyze the respondent (e.g., brand ‘personality’ items).

    However, most people won’t take the time to leave a comment.  In fact, if your survey suffers from particularly bad design, many won’t even stick around to get to the last page.  So, you should pay attention to the second (silent) respondent feedback mechanism: completion rates.   Here are the completion rates for the three surveys mentioned above. These show the proportion of people who started the survey that went on to complete it. A low completion rate is a key signal of problems with your survey design because it means many people dropped-out.

    Survey 1: 79%
    Survey 2: 71%
    Survey 3: 81%

    These are pretty good completion rates, but you’ll notice that one is about 10 percentage points lower than the others (Survey 2).  We knew download times were going to be an issue for that survey because it contained several large images and used a JavaScript library.  The images were large because they contained complex backgrounds (i.e., they weren’t simple!). Despite doing all we could to optimize and pre-load the images, use the bare-minimum JavaScript library and warn respondents, this issue clearly led to increased drop-out rates.   It is also worth noting that, although 1,100 people started the survey, only 8 of the 800 who completed it  mentioned the slow load times at the end.

    So, it really is worth keeping things as simple as possible in your survey design.  Respondents can tell when you are asking them flaky, ill-prepared questions and many won’t stick around if your questionnaire causes them frustration.
    _____

    Short URL for this post: http://wp.me/pnqr9-1K

     

  • Ben 12:21 on Thursday, September 17, 2009 Permalink | Reply
    Tags: , Free Stuff, , Survey Methodology   

    How to Run a Great Online Survey 

    Back in 2007 I wrote a guide to running online surveys.  I’ve been updating it over the past couple of months and the newly minted version 1.0 is now ready to see the light of day (the 2007 version was 0.5!).  Jump over to the Free Guide to Online Surveys page to grab a copy.

    Suggestions for improvement are welcome.

    _____
    ShortURL for this post: http://wp.me/pnqr9-2e

     

  • Ben 22:57 on Tuesday, September 15, 2009 Permalink | Reply
    Tags: , Coverage Error, , , Survey Methodology, Your Mother   

    Why some Internet Surveys are like your Mother 

    Most of us love our mothers dearly.  But that doesn’t mean we go to them for advice every time we need answers to some important problem in our lives.  Sure, it would be easy, quick, and cheap to get a few words of wisdom, but it is just not realistic to expect them to be objective.  Thousands of bitterly disappointed American Idol contestants learnt this fact the hard way.

    And so it is with some online surveys.  It is now easy to throw together a web-based questionnaire, get it sent to a bunch of people, and have answers back all within a fortnight.  But if the people who received the survey are skewed on some important dimension (e.g., technologically literate, mostly young, mostly employed, etc.) you can’t expect the results to accurately reflect the opinions or likely behaviours of a more diverse group.  The technical term for this sort of bias is coverage error and it is one of the key reasons to think carefully about how you select the sample for a survey.

    There are two very general categories of survey sampling techniques:

    1. Non-probability sampling: You don’t specifically go out to get a random selection of people from your target group.  Instead, you let allcomers complete your survey.  Perhaps you send an invitation out to your friends and ask them to invite their friends, etc.  Or perhaps you advertise the survey and let anyone who happens to see the ad fill in a questionnaire.  These surveys have all the objectivity of talkback radio.  They might be entertaining, but you wouldn’t usually base a policy or business decision on them.
    2. Probability sampling: You make an attempt to get a random selection of people from your target group completing your survey.  In the ‘holy grail’ version of this approach, you’d have a list of all the people in your target group, take a simple random sample from the list, send the invitations to the sampled people and then follow up to get as many of them answering as possible.  Survey researchers and statisticians have developed lots of variations on this theme to take account of practical issues, but the aim is always to get a wide mix of people from the target group responding.  Although your results under this approach won’t be perfectly accurate, you can be confident that you’ll come close to reflecting the opinions and behaviours of the full group.

    Sounds clear enough, doesn’t it?

    And it is.  Until we enter the wild world of internet survey respondent panels.  You see, it is possible to order up a random selection of people from a panel that makes you feel like you are taking a probability sample when really your results may be subject to the sorts of coverage errors inherent in a non-probability sample.  This is because many panel providers build up their lists of eager members by non-probability methods.  Few providers source members via a random (or pseudo-random) process like Random Digit Dialling or Address Based Sampling because it is so expensive to do so.  Even fewer provide internet access to those households who don’t have it.  Knowledge Networks is one company that does these things.

    Predictably, a recent study titled Study Finds Trouble for Internet Surveys highlights the differences in accuracy that arise from the different panel recruitment approaches (probability vs non-probability).  Here are some selected excerpts:

    In the most extensive such analysis to date, David Yeager and Prof. Jon Krosnick compared seven non-random internet surveys with two others based instead on random or so-called probability samples. The non-probability internet surveys were less accurate, and customary adjustments did not uniformly improve them.
    While the random-sample surveys were “consistently highly accurate,” the internet surveys based on self-selected or “opt-in” panels “were always less accurate, on average, than probability sample surveys, and were less consistent in their level of accuracy,” the researchers said. Further, they said, adjusting these samples to known population values had no effect on accuracy (and in one case even worsened it) as often as that process, known as weighting, improved it.

    In the most extensive such analysis to date, David Yeager and Prof. Jon Krosnick compared seven non-random internet surveys with two others based instead on random or so-called probability samples. The non-probability internet surveys were less accurate, and customary adjustments did not uniformly improve them.

    While the random-sample surveys were “consistently highly accurate,” the internet surveys based on self-selected or “opt-in” panels “were always less accurate, on average, than probability sample surveys, and were less consistent in their level of accuracy,” the researchers said. Further, they said, adjusting these samples to known population values had no effect on accuracy (and in one case even worsened it) as often as that process, known as weighting, improved it.

    So, be wary when purchasing a “random” or “representative” sample from an opt-in panel provider.  Such a sample might be fine for your particular purpose or target group, but you need to at least consider the risks of coverage error you are taking.  And don’t expect weighting to magically solve any coverage error you do have!

    _____

    Short URL for this post:  http://wp.me/pnqr9-1t

     

c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
shift + esc
cancel
Follow

Get every new post delivered to your Inbox.