Two things struck me when I saw this AdSense optimization ad…
First, the age profile of the first two cases was toward the high end. Google is probably trying to target older website owners, but it is still worth remembering that the urge to test empirically does not necessarily discriminate by age. They did manage to find two cases for their ad, after all :)
Second, the fact that Google is running ads focussed on getting people to try out different ad formats suggests they are pretty sure a large proportion of current placements are sub-optimal. More testing and optimisation of AdSense placements by website owners means more revenue for both parties. Unless, of course, the website owner has other revenue streams that may be cannibalised by changes to the ad formats they have on their site…
It is not uncommon to see news stories celebrating the success of some initiative or individual as being due to some bright idea or moment of inspiration. This phenomenon is not new; every child is taught that Archimedes had his ‘Eureka moment’ and can recite the story of Netwon’s falling apple. It is these flashes of insight that we remember and strive to emulate.
However, the focus on creativity is unfortunate because it only paints half the picture. For instance, the ‘file drawer problem’ means we see those flashes of inspiration that led to success, rather than the countless others that didn’t. And, it is easy to forget that people like Archimedes and Newton were old-school split testers. They subjected idea after idea to the brutal scientific method and learned from the many failures they no doubt had. It is their perseverance and commitment to testing, not just their creativity, that we should remember them for.
Fortunately, more news is starting to bubble to the surface about the interplay between the creative and scientific processes. For instance, this wired story shows how the gaming industry (typically considered a bastion of creativity and design) is embracing split testing to drive development decisions. I also recently saw the following talk shared widely on Twitter about the testing that went into the success of Obama’s 2008 fundraising campaigns.
These stories highlight the fact that creative ideas are like the random mutations that drive the evolutionary process. They are necessary, but certainly not sufficient, for progress to occur. And another interesting recurrent theme is that the mental models underlying our creativity – the source of our ‘gut feelings’ about what will work – are often wrong. Indeed, testing is essential to updating these models and is an under appreciated input to the creative process. Together, they form an iterative learning cycle.
This interplay has implications for organisational and personal development in that as much effort should be put into developing the testing and learning process as goes into supporting the creative process.
A great write-up on determining sample sizes for, and avoiding common traps in, split testing. Yet another good testing post from the folks at 37 signals. R code and discussion of power calcs included. http://37signals.com/svn/posts/3004-ab-testing-tech-note-determining-sample-size
How one extra word in an email subject line improved end point conversions by 279%. This stuff never ceases to amaze me. http://whichtestwon.com/archives/7353
Here is a link worth keeping. Google recently updated the look and feel of its search user interface. This article describes the behind-the-scenes process Googlers followed to get to the end point we are all seeing today. Unsurprisingly, they followed a thorough research process, incorporating extensive qualitative and quantitative feedback before settling on an optimal solution.
How Google got its New Look.