Failing at Testing

The use of A-B testing and bucket testing for optimizing tasks can provide a clean, programatic way to make many design decisions. It’s an invaluable technique for comparing two similar but incompatible choices, and the results are impossible to have much debate about. Because of the urge to move fast, some groups have been tempted to use A-B testing to create artificial certainty about qualitative questions, however:
Testing can’t give insight into what the product package should be; it will never tell you why the results are what they are. Trying to test features individually and then assemble them into a coherent package will not give insight into the success or failure of the entire package, nor provide a sense of what the right product is. At best, it results in a sort of “drunken sailor’s walk,” as each new incremental test takes the product in a different direction.
Testing won’t tell you which product will be more successful; direct comparison is seldom the way that people decide what to buy (Pepsi beats Coke in taste tests year after year, yet Coke has 44% of the market and Pepsi is stuck at 32%). In fact, attempting to compare a new products to each other this way can result in audiences simply choosing what’s familiar. This can be seen in the heavily AB test driven designs of the search results pages of Google, Yahoo, MSN, and Ask, which are all very similar.
A great deal of research has been done into the product choices people make, and it consistently points to the formation of a “story” around a product being the most important factor in whether it succeeds or fails (the most eloquent person on this topic is Seth Godin, ex-Yahoo! employee). This is not the marketing at all, but how the actual product makes overall sense to people who use it, surprises them with something new and better, or disappoints for some reason. There is no way to create a story out of a list of features or parts, instead it’s the combination that either works, or it doesn’t.

Post a Comment

Your email is never shared. Required fields are marked *