One for the Econ Geeks
Here’s something I never expected to see; Keynes and Hayek battle it out in a very nicely produced rap vid.
Recent Updates RSS Toggle Comment Threads | Keyboard Shortcuts
-
Ben
-
Ben
-
Ben
Can You Think of a Personalised Plate for this Ride?
Apparently the following picture is from the mid sixties. It would have been something to see Spock staring back at you from his Buick on the freeway.
-
Ben
How ‘Information’ is made out of Thin Air
Have you heard that a weekday edition of The New York Times contains more information than the average person was likely to come across in a lifetime in 17th century England?
If so, did you also know that this statement probably has no basis in fact?
I’ve come across the tidbit more than once over the last couple of months, in conversations and online. It struck me as being in the same general vein as another popular myth: that we only use 10% of our brains, so I did a little digging. Here’s what I found:
- This viral YouTube video contains the snippet (at 3:20) and may be the key source of its popularity at the moment. No sources are cited for any of the information presented. Not even in fine print. The video is one of a number of versions of this particular montage of ‘facts’ which, together, have been seen over 10 million times.
- The snippet has popped up in a number of places around the web, some more reputable than others. It also appears in numerous books. Like the YouTube vid, most don’t make any effort to verify the assertion. The context also changes from repeat to repeat; sometimes it is the Sunday edition and 19th century citizens, sometimes it is exposure to information in a day and ‘our ancestors’. So, that lesson you learnt playing Chinese whispers as a kid still holds; people are prone to error when retelling a story. It’s worth being wary of this when retelling something yourself or hearing a startling ’truth’ from someone else.
- Where sources are cited, they generally lead to a statement made by Richard Saul Wurman in his 1989 book titled Information Anxiety (page 32). Unfortunately, I don’t have access to the book, so I can’t see whether he presents any evidence to support the claim. Thankfully, Geoffrey Nunberg at UC Berkeley managed to find a copy and took a look for himself… apparently the book ”asserts the fact without offering a source or explanation” (p9 of a pdf by Nunberg that in part examines the likely veracity and lack of meaning of the snippet).
So, after making a reasonable effort aimed at finding evidence to support the claim, I’ve come up empty handed. Others have had the same experience. My conclusion is that it is probably nothing more than an statement without foundation that has made its way into popular consciousness by virtue of it being superficially plausible and sufficiently repeated without critical thought. Fascinating stuff.
This particular example is innocuous, but sometimes these ‘facts from thin air’ can make their way into places where they might have more impact on policy or business decisions. For instance, in Damn Lies and Statistics, Joel Best recounts that a published Journal article he once read contained the statement “Every year since 1950, the number of American children gunned down has doubled”. Go here if you are interested to see why the statement is so absurd, along with the history of how this ‘mutant statistic’ came to be.
_____
ShortURL for this post: http://wp.me/pnqr9-4j
-
Ben
Rant: How to Encourage Piracy
Imagine you’ve just done something of benefit to someone else. Lets say you’ve put out the trash for your flat. Which of the following would you like to hear from one of your appreciative flatmates?
- “Hey, thanks for putting the trash out. It’s great you remember to do this even though we’ve all got busy Monday mornings. I got a voucher for a free coffee last night – you want it?”
- “Phew. If you hadn’t put the trash out this morning me and the others were going to spit in your coffee and bad-mouth you behind your back. Remember this next time you consider not putting the trash out. “
Unless you are a masochist, I’d guess option 1 is both more attractive and most likely to reinforce your behaviour. Under option 2, putting the trash out becomes even more of a chore. You’d quickly begin to resent doing it and the incentive to continue would reduce to nil if you suddenly found out your flatmates would have no way of knowing whether you had done it or not.
So what is it with the movie and software anti-piracy groups? Do they not understand this? Do they not care? Or are they simply under the undead control of Sauron-the-lawyer?
Here is a painful clip from 1992 titled “don’t copy that floppy”
(In 2009, an even more painful sequel was released targeting college kids. Apparently you can make anything cool with ‘the kids’ if you rap it.)
And here is a current clip playing on DVDs in the UK and Australasia. I think the man featured is supposed to look like Satan.
Both use fear appeals (watch out, terrorists will eat your babies and we’ll take your mom off to prison) and the assumption that people are too stupid to realise piracy only rarely results in punishment for minor offenders. What is worse is that these appeals are forced on an audience that is already compliant. It is essentially the same as rewarding an as-yet-undetermined flatmate for putting the trash out by attaching a note to the bag that looks something like option 2.
I’m quite happy to pay for quality content, but it infuriates me every time I have to sit through one of these anti-piracy propaganda pieces. In fact, on more than one occasion the thought has gone through my mind that it would be easier to go the piracy route – at least that way I wouldn’t have to watch the anti-piracy dross.
It seems I’m not the only one.
Anti-piracy organisations should be reinforcing purchasing customers, not punishing them.
End of rant.
_____
ShortURL to this post: http://wp.me/pnqr9-2U
-
Ben
Anything that doesn’t take years of your life and drive you to suicide hardly seems worth doing.http://bit.ly/2vbTI3
Cormac McCarthy (Author of ‘No Country for Old Men) -
Ben
Have You Fallen Prey to Simpson’s Paradox?
In a previous post on experimentation at Microsoft I linked to a recent presentation by Ron Kohavi (GM of their experimentation platform). One point he raised was that you can actually get the wrong answers from split tests because of a phenomenon called Simpson’s Paradox. You read right; your test might tell you that version A is the best bet when in reality the better performing version is B.
That should send a shiver down the spine of anyone tasked with improving a website’s ROI.
Simpson’s paradox can occur in any setting where the proportion of people allocated to split groups (e.g., control and test) varies according to some important attribute in the study. It is easiest to understand the paradox by example. Thankfully, the Wall Street Journal presented one a couple of days ago in an article on the Flaw of Averages. Essentially, it showed that although current aggregate unemployment rates in the US (expressed as % jobless) don’t appear as bad as they were during the 80s recession, they are actually consistently worse when the figures are examined by educational subgroup. This is because the proportion of people in each educational subgroup has shifted between the 1980s and now, and each subgroup has a different susceptibility to unemployment.
The WSJ article also presents two other examples (U of C Berkeley admissions gender bias and Kidney stone treatment efficacy). If you are still scratching your head after reading through the narrative explanations, try having a look at a the data-based explanations of the same examples on this Wikipedia entry.
Turning to a web-based scenario, in a recent paper outlining pitfalls to avoid in online experimentation, the folks at Microsoft showed how Simpson’s Paradox can occur when a test is ‘ramped up’ over time. Their example involves a page design test run over two days, with a 1% sample of users assigned to the test group on the first day (Friday) and then a 50% sample assigned to the test group on the second day (Saturday). Here is the data from the paper:
(Note: The percentage in the version B ‘total’ cell is different here due to an error in the original)
On both test days ‘B’ was the winning version. However, the result is reversed in the aggregated total; Version A is the winner. This is essentially because both the test split allocations and response levels varied by day.
Test ‘ramp ups’ are quite common. It is good practice to do a pilot of the test on a small sample to make sure everything is working OK before unleashing it on a larger sample. So, the potential for Simpson’s Paradox to occur is very real. If you are analysing split test results, you can make sure your analysis avoids the problem by re-weighting the results from periods with different allocation procedures or by simply discarding the results from the pilot phase.
_____
ShortURL for this post: http://wp.me/pnqr9-3X
-
Ben
Online Experimentation at Microsoft
Over the last three years Microsoft embraced experimentation as a mechanism for testing changes to their various online products. That they are only recently formally adopting a data-driven approach to their design was a little surprising to me, but it is certainly better late than never!
As part of the process of making the shift away from simply following the Highest Paid Person’s Opinion (HiPPO) to actually testing the ROI of different ideas, the team in charge of experimentation has been disseminating some of their experiences. You can see a recent talk on the topic, presented at a September meeting of Seattle Tech Startups, at the URL below (sorry, the quality isn’t great and I can’t embed because of WordPress.com restrictions). Alternatively, go to the Microsoft experimentation portal to see other work from this group.
http://www.ustream.tv/flash/video/2134721
The talk presents a number of interesting insights, ranging from the results of some tests (winning versions are often different to what you’d think) through to the cultural hurdles arising from an increased reliance on data for decision making (e.g., people with strong opinions get their egos bruised).
Amazon.com is also mentioned a couple of times. I think a few of the current Microsoft team originally cut their teeth there, so those of you interested in this topic might also like to see this eMetrics Summit 2004 presentation (pdf). It showcases the Amazonian approach to deciding on site changes and resolving bitter political disputes over whose pet area should get highly coveted slots on the home page. Interesting stuff that more and more organisations are going to have to grapple with as their products and services become increasingly digitized.
Ben
Evolution of a Form
I’m not sure how long they have had it posted, but I came across Statistics New Zealand’s archive of historic census forms a few weeks ago. Starting with the household form from 1906, it covers every census up to 2006 and provides a window into the social changes the country has gone through as well as the evolution of our understanding of good form design. It is fascinating to see how our language, norms, and data collection technologies are captured for posterity in Stats NZ’s version of a family photograph album.
I’ve only had a quick glance through the archive, but took note of a range of interesting things in a short space of time…
About the information collected:
- In the 1916 questionnaire, (1mb pdf) the ‘Infirmity‘ section asked how many “imbeciles and feeble minded” people resided in the household. Instructions under the ‘Particulars as to Marriage‘ section also dictated that “A Chinese is to be set down as ‘never married’ unless he has or has had a wife in Australia” (wtf?)
- Beyond 1916, questions relating to disability disappeared. It wasn’t until 1996 (83k pdf) that a question on disability was included again.
- Understandably, the impact of the world war II can be seen in the 1945 (381kb pdf) questions.
- We were a nation obsessed with fowl. From the first form in the historic series right through to the questionnaire in 1966 (353kb), there was a section devoted to the ‘Census of Poultry‘, asking things like the gender, age and number of live fowl kept. In fact, even the 1971 form (261kb) asked how many ‘fowls‘ the household owned (there was also a question asking whether the household had a flush toilet!).
About the question structure and overall form layout:
- The early forms were verbose in their instructions and short on space. This was probably partially due to the cost of printing and an assumption that there would be a ‘learned’ person in the household to complete the form. The questionnaires were also mostly open-ended (i.e., no check boxes). All collation of figures would have been done by hand (ouch!), so there may not have been much incentive to have the data highly structured on the form itself at that point.
- As time progresses, the instructions become clearer, shorter, and better distributed throughout the questionnaire. The shift from a columnar format to a vertical format also happened fairly quickly. This makes the forms much easier to read and follow.
- In the 70’s, checkboxes and specific navigational elements like arrows were introduced. I wonder how much of this was due to advancing computer technology (it is easier for data entry operators to work with more structured forms) and how much was due to advances in our understanding of good form design.
- 2001 (484kb) saw a change to the way people were asked to indicate a response. Instead of using a tick, they were asked to fill in the circle with a horizontal line. This was no doubt required so that optical scanners could read the millions of forms that were returned, but it actually goes against people’s learned behaviour (we like to use ticks because we’ve been doing so all our lives). It would be interesting to know how many 2001 forms were returned incorrectly completed.
There are bound to be other interesting insights to be unearthed. If you come across any, please drop a note in the comments (hit the ‘reply’ link at the top right hand side of the post).
_____
ShortURL for this post: http://wp.me/pnqr9-3A
Ben
Why You Should Read Some Cialdini
Recently, Carsonified posted this video of a Kevin Rose (Digg) speech at the Future of Web Apps conference in London. The talk, titled “Taking your Site from One to One Million Users”, covers a range of methods for growing web traffic and engaging users.
Although many of his examples are interesting in themselves, it is worth noting that a number draw upon core psychological principles first put together 25 years ago by Robert Cialdini in a book titled Influence. Cialdini studied how social norms and ingrained behaviours have a massive influence on our responses to the various stimuli we are presented with on a daily basis. He grouped them into six general categories:
- Reciprocity
- Commitment and Consistency
- Social Proof
- Authority
- Liking
- Scarcity
You can read a little more about each of these here, but it really is worth taking the time to read the book (most libraries will have it). It presents both the underlying motivators along with specific examples of their expression in practice, so it is more useful than hearing only the examples themselves, which don’t necessarily lend themselves to application in a range of different domains.
Cialdini also recently co-authored a book titled Yes!, which presents summaries for over 50 scientific studies relating more-or-less to the six categories above. It is a fascinating read. Here are a couple of the findings:
- Tips increase by up to 23% when waiters give mints to their customers directly, rather than letting them pick the mints up from a bowl at the counter. [an example of reciprocity]
- A hotel increased guest towel reuse rates (an environmentally friendly action) by 33% simply by changing the wording of the call-to-action message on a sign in the bathroom. The most successful message suggested that others who had stayed in the same room recently had reused their towels. [an example of social proof, and also liking - we are influenced more by those who share attributes with us]
- Asking people to predict whether or not they will go out and vote actually increases the chance they will go out and vote (by about 25% points in the study cited). [an example of social desirability bias combined with commitment and consistency]
There are many other examples in Yes! that translate the six general principles into practice, so you are bound to find some inspiration for mixing it up and applying the principles yourself.
_____
ShortURL for this post: http://wp.me/pnqr9-2R
