Over the last three years Microsoft embraced experimentation as a mechanism for testing changes to their various online products. That they are only recently formally adopting a data-driven approach to their design was a little surprising to me, but it is certainly better late than never!
As part of the process of making the shift away from simply following the Highest Paid Person’s Opinion (HiPPO) to actually testing the ROI of different ideas, the team in charge of experimentation has been disseminating some of their experiences. You can see a recent talk on the topic, presented at a September meeting of Seattle Tech Startups, at the URL below (sorry, the quality isn’t great and I can’t embed because of WordPress.com restrictions). Alternatively, go to the Microsoft experimentation portal to see other work from this group.
The talk presents a number of interesting insights, ranging from the results of some tests (winning versions are often different to what you’d think) through to the cultural hurdles arising from an increased reliance on data for decision making (e.g., people with strong opinions get their egos bruised).
Amazon.com is also mentioned a couple of times. I think a few of the current Microsoft team originally cut their teeth there, so those of you interested in this topic might also like to see this eMetrics Summit 2004 presentation (pdf). It showcases the Amazonian approach to deciding on site changes and resolving bitter political disputes over whose pet area should get highly coveted slots on the home page. Interesting stuff that more and more organisations are going to have to grapple with as their products and services become increasingly digitized.