from www.codinghorror.com
programming and human factors
by Jeff Atwood
On a recent airplane flight, I happened to catch the movie Groundhog Day. Again.
If you aren't familiar with this classic film, the premise is simple: Bill Murray, somehow, gets stuck reliving the same day over and over.
Phil doesn't just go on one date with Rita, he goes on thousands of dates. During each date, he makes note of what she likes and responds to, and drops everything she doesn't. At the end he arrives at -- quite literally -- the perfect date. Everything that happens is the most ideal, most desirable version of all possible outcomes on that date on that particular day. Such are the luxuries afforded to a man repeating the same day forever.
This is the purest form of A/B testing imaginable. Given two choices, pick the one that "wins", and keep repeating this ad infinitum until you arrive at the ultimate, most scientifically desirable choice. Your marketing weasels would probably collapse in an ecstatic, religious fervor if they could achieve anything even remotely close to the level of perfect A/B testing depicted in Groundhog Day.
But at the end of this perfect date, something impossible happens: Rita rejects Phil.
The next time you reach for A/B testing tools, remember what happened to Phil. You can achieve a shallow local maximum with A/B testing -- but you'll never win hearts and minds. If you, or anyone on your team, is still having trouble figuring that out, well, the solution is simple.
Read more at www.codinghorror.com
Wednesday, July 28, 2010
An argument against A B split testing
Subscribe to:
Post Comments (Atom)
Rubbish
"If an opinion contrary to your own makes you angry, that is a sign that you are subconsciously aware of having no good reason for thi...
-
I listen to arguments about global warming and climate change; about the effects or non-effects of human intervention and the tipping point...
-
I wonder what 'hardened' means? Does it mean encasing the electronic switchgear in some pulse-resistant material to prevent th...
-
Italian: piazza Area libera, limitata in tutto o in parte da costruzioni, con varia funzione urbanistica, all'incrocio di più strade o l...
No comments:
Post a Comment
Post a comment and start a conversation...