A great write-up on determining sample sizes for, and avoiding common traps in, split testing. Yet another good testing post from the folks at 37 signals. R code and discussion of power calcs included. http://37signals.com/svn/posts/3004-ab-testing-tech-note-determining-sample-size
Tagged: Data-Driven Design RSS Toggle Comment Threads | Keyboard Shortcuts
-
Ben
-
Ben
Links: Using SQL ‘With’ statements, and a great example of A/B Testing
Two links worth keeping:
- Michael Berry at Data Miners Blog describes how to use SQL common table expressions (i.e., WITH statements) to simplify complex queries by creating on-the-fly temp tables (named subqueries) prior to the full query definition.
- Designer Andrei from 37Signals describes the outcomes of a number of A/B tests they did on their Highrise signup page, along with the variants they tested.
-
Ben
How one extra word in an email subject line improved end point conversions by 279%. This stuff never ceases to amaze me. http://whichtestwon.com/archives/7353
-
Ben
Google’s User Interface Design and Decision Process
Here is a link worth keeping. Google recently updated the look and feel of its search user interface. This article describes the behind-the-scenes process Googlers followed to get to the end point we are all seeing today. Unsurprisingly, they followed a thorough research process, incorporating extensive qualitative and quantitative feedback before settling on an optimal solution.
-
Ben
Online Experimentation at Microsoft
Over the last three years Microsoft embraced experimentation as a mechanism for testing changes to their various online products. That they are only recently formally adopting a data-driven approach to their design was a little surprising to me, but it is certainly better late than never!
As part of the process of making the shift away from simply following the Highest Paid Person’s Opinion (HiPPO) to actually testing the ROI of different ideas, the team in charge of experimentation has been disseminating some of their experiences. You can see a recent talk on the topic, presented at a September meeting of Seattle Tech Startups, at the URL below (sorry, the quality isn’t great and I can’t embed because of WordPress.com restrictions). Alternatively, go to the Microsoft experimentation portal to see other work from this group.
http://www.ustream.tv/flash/video/2134721
The talk presents a number of interesting insights, ranging from the results of some tests (winning versions are often different to what you’d think) through to the cultural hurdles arising from an increased reliance on data for decision making (e.g., people with strong opinions get their egos bruised).
Amazon.com is also mentioned a couple of times. I think a few of the current Microsoft team originally cut their teeth there, so those of you interested in this topic might also like to see this eMetrics Summit 2004 presentation (pdf). It showcases the Amazonian approach to deciding on site changes and resolving bitter political disputes over whose pet area should get highly coveted slots on the home page. Interesting stuff that more and more organisations are going to have to grapple with as their products and services become increasingly digitized.
Ben
Proof that People Appreciate Good Survey Design
I’m a huge advocate of simplicity in survey design, especially when a survey is to be delivered online. Yet, when I talk to people about cutting out questions, simplifying response tasks, and minimizing the use of various presentational options (e.g., AJAX), I sometimes get the sense I’m viewed as a spoil-sport. Fortunately, people don’t have to take my word for it. A wealth of methodological research shows that completions and data quality suffer as you stray from following the Keep It Simple, Stupid (K.I.S.S.) principle in survey design (see the links at the bottom of this page).
Some respondents will also tell you how well (or badly) you’ve structured your questionnaire, although waiting until you get this to do anything is probably leaving things a little too late! Respondents can give feedback on a survey in a couple of ways: by dropping out if they are having difficulty with it, or by mentioning their experience at the end (assuming you give them an opportunity to do so). Here are some examples pulled from three surveys I’ve been involved with over the past 12 months. These went out to general population samples provided by a well-known consumer panel. The topics differed, but the surveys were similar in length – about 35 questions over 15 pages. Two of them involved presenting choice sets as part of a stated choice modelling experiment.
My intent here is not to take the glory for the results I’m about to present; I took care of the online delivery in these surveys, but the questions and structure were mainly developed by others. I’m using them because I do think the questionnaires were generally well designed. Questions were kept to a minimum, pre-testing was done, and the technology used was as simple as possible.
First, some selected respondent comments taken from across the three surveys. Many other comments echoed the same general sentiment:
“Very good survey, was easy to follow and understand.”
“Thoroughly enjoyed that survey is all I can say.”
“Clear and simple, well worded – well done, whomever designed it.”
“It was more interesting than the usual surveys :o)”
“It was a very simple, well put together survey that was easy to understand. Well done.”
“I enjoyed doing it :)”
“This was a great survey, thank you!”
“I really enjoyed this survey. Very easy to follow.”
“Great Survey, easy to do and no dumb questions.”
“Wish they were all this easy to complete.”
My key take-out points are that a) it is actually possible for people to enjoy completing a questionnaire and b) many of the online surveys people are sent appear to be complex, hard to follow, and sprinkled with “dumb questions”. Although I’m speculating, I think the “dumb question” comment refers to those that are ambiguous, overly complicated or repetitive (e.g., matrix-style questions) or attempt to psychoanalyze the respondent (e.g., brand ‘personality’ items).
However, most people won’t take the time to leave a comment. In fact, if your survey suffers from particularly bad design, many won’t even stick around to get to the last page. So, you should pay attention to the second (silent) respondent feedback mechanism: completion rates. Here are the completion rates for the three surveys mentioned above. These show the proportion of people who started the survey that went on to complete it. A low completion rate is a key signal of problems with your survey design because it means many people dropped-out.
Survey 1: 79%
Survey 2: 71%
Survey 3: 81%
These are pretty good completion rates, but you’ll notice that one is about 10 percentage points lower than the others (Survey 2). We knew download times were going to be an issue for that survey because it contained several large images and used a JavaScript library. The images were large because they contained complex backgrounds (i.e., they weren’t simple!). Despite doing all we could to optimize and pre-load the images, use the bare-minimum JavaScript library and warn respondents, this issue clearly led to increased drop-out rates. It is also worth noting that, although 1,100 people started the survey, only 8 of the 800 who completed it mentioned the slow load times at the end.
So, it really is worth keeping things as simple as possible in your survey design. Respondents can tell when you are asking them flaky, ill-prepared questions and many won’t stick around if your questionnaire causes them frustration.
_____
Short URL for this post: http://wp.me/pnqr9-1K
Ben
Not All Conversions are Created Equal
Tim Ferris posted a Google Website Optimizer Case Study the other day showing how data-based design tweaks at Gyminee (now Daily Burn) helped them increase conversions by 20% and then another 16% on top of that. The post presents a really nice example of how simple it is to use free tools along with good landing page design principles to generate improvements in site goal performance. That said, I’d add a couple of things to round out the article:
- The performance improvement was measured in number of free trial sign-ups. There is nothing wrong with that if Daily Burn has free sign ups as a key goal. However, it is worth noting that the improvements in free sign-ups may have had the opposite effect on conversions to paid accounts. One reason for this is that by reducing the possible actions on the page to one (sign up for a free trial) in the second set of changes, Daily Burn may be seeing an increase in sign-ups from tire kickers who just want to see what the ap looks like. In the past visitors could click the ‘tour’ button to do this; now they have to go via the free trial route. If the requirement to sign up also puts some other potential purchasers off before they get a chance to see the product, the net effect of the change may be to decrease the proportion of free trialers that go on to paid subscriptions. One of the sites I read presented an example of exactly this issue a few weeks back; I think it was Marketing Experiments but now I can’t find the article (doh). [Update: here is a different example with a similar finding ]
- Here is a link to the Paradox of Choice concept Tim mentioned. I’m not so sure the original Gyminee page was overwhelming people with choice (causing choice paralysis) as much as providing too much of an opportunity to get distracted before clicking on the sign-up button. Ultimately it doesn’t matter; the effect of the modification was positive whatever the underlying reason for the change in behaviour!
- Tim didn’t specify the ‘conversion marketing best practices’ behind the design changes tested in the second half of the post. Going by the screenshots presented, these included the use of testimonials (social proof), awards (authority), and specificity (specific facts are more persuasive). Feel free to posts others if you spot them…
____
Short URL for this post: http://wp.me/pnqr9-13
