An Optimization Mix-Tape

By Jason Carmel | 3 Comments | Posted in in Optimization | Permalink

I grew up making mix-tapes (note: for those of you unfamiliar with the concept of a mix-tape, think "playlist," but with far, far crappier technology than you are using right now to listen to far, far crappier music). I am not the first person to wax eloquent on the poetry of the mix-tape, so I will spare you my own musings in deference to those better written. There is no more holistic way to describe a circumstance, event, or task than through a lovingly-crafted compilation of songs. Normally, the songs speak for themselves. I don't need to tell my ex-girlfriend, for example, that the third song on side two of my 8th Grade Summer Mix ("Baby Bitch" by Ween) was pointed directly at her. She knows, man. Used properly, the mix-tape is a perfect medium.

This is a mix-tape for optimization. Ten life lessons about how and why you should be testing your websites, all neatly wrapped in a delicious candy-coating of RAWK:

1. "I Might Be Wrong" - Radiohead (Amnesiac) - This is why we test in the first place, so it's a good one to start off with. You never know whether a new initiative is helping or hurting relative to what was up there before (and by how much) unless you run a test to prove it. As this theme could be our anthem, it deserves pole position. Plus, every mix-tape since Pablo Honey came out had to include a Radiohead song by law, so it's good to get that out of the way (I'd usually cram mine in between "Forever Young" by Alphaville, and whatever Depeche Mode song I was listening to at the time). If those two weren't reason enough, I also have it on good authority that this is one of our Project Manager's (Ariel) favorite songs. First rule of test execution: do whatever it takes to get on your PM's good side.

2. "If I Had $1,000,000" - The Barenaked Ladies (Gordon) - If you had a million dollars (or, as is often the case, far less), where would you apply those resources? Would you test your homepage? The landing page for search results? Your credit card form? Any optimization program worth its salt will dynamically prioritize tests according to return on investment. The song is goofy, but the concept is deadly serious. Live by it.

3. "Minus Two" - Fishboy (Albatross) - Know your math. Love your stats calculators. Revel in Excel. Running a test is a lovely thing, but if you can't determine whether the results are statistically valid, you're in a bad place. In truth, the song doesn't have a lot to do with math (other than the name), but  Fishboy is one of our Web Analytics expert's (Halee) favorite bands, so it's in. Second rule of test execution: do whatever it takes to get on your Analyst's good side.

4. "Invisible Ink" - Aimee Mann (Lost in Space) - Aimee (who has a little crush on me) reminds us to test hiding things in addition to adding things. As everyone fights for real estate on a homepage (for example), customers may get lost with all the choices in front of them. Try hiding a few things (do people really need to sign up for a newsletter on the first visit to a homepage?) and see if that doesn't boost engagement or clickthrough.

5. "Tracks of My Tears" - Billy Bragg (Talking With The Taxman About Poetry) - Covers are wonderful things, as they remind us how a different voice can transform a message. Here we have a smooth, Motown production by Smokey revamped entirely using a single electric guitar and a very bitter-sounding bloke from Essex. What Billy teaches us is that we should consider testing new voices for our messages to see if they strike a chord (proverbially speaking) with our audience as a whole, or various subsets thereof.

6. "Lola" - The Kinks (Lola versus Powerman and the Moneygoround, Part One) - Ah, here we have the classic tune about unexpected results reminding us to watch out for the same when we are testing. You expected conversion rates to go up, and they did, which is great. Did you also expect your churn to skyrocket? A 20% off coupon you tested increased purchases, which is great. But did you expect your average order value to drop through the floor? Much like transvestites in SoHo clubs, unanticipated results aren't necessarily a "bad" thing, but the more work you can do to spot them in advance will save you the occasional awkward moment after launch.

7. "All My Mistakes" - The Avett Brothers (Emotionalism) - Not all of your tests will make things better. Some will have no effect at all, and some (gasp) will demonstrate that the control version is the best one out there at the moment. These are not bad outcomes. As long as you have entered an experiment with a hypothesis, you are proving or disproving something that can lead to a better understanding of how your customers interact with your site, and (just as important) a list of future experiments to consider executing.

8. "Know Your Onion" - The Shins (Oh, Inverted World) - This is my song for segmentation and behavioral targeting. Justin, my colleague from UX, always tells clients that a site should get better at addressing the needs of customers as it acquires more information and adjusts content accordingly. He refers to this as peeling layers of an onion with each customer interaction. So, music fans, I challenge you to know your onion, and to know how to peel those layers off for your customers to target them with relevant content as they demonstrate specific behaviors on your site.

9. "Do What You Want" - Bad Religion (Suffer) - A punk staple, in my humble opinion. Many herald the song as a generalized call to anarchy. While I wasn't in the room when the band wrote the song, I believe that they were referring specifically to the sense of freedom you get by testing your site (some may point to the album's 1988 release date- some five years before the first web browser- as evidence to the contrary. I just think of it as yet another demonstration of how ahead of its time Bad Religion was). "Break all the @$%&ing rules," says singer, Greg Graffin, and I agree. Remove "we've always done it this way" from your vocabulary and challenge your site's most sacred of cows to see if they prove their merit against other alternatives.

10. "Automatic" - Less Than Jake (Losing Streak) - It took a courageous ska-punk band from Gainesville to remind us how we should include optimization automatically in all site releases. Got a new feature going out? Test it. New product offering? Test it. Campaign? Testamundo. You get the idea. Include optimization automatically into your production and release schedule. Your site will be better for it.

Enjoy the music, folks.

[Author's Note: Incidentally, I don't think Chris Kerns could come up with a mix-tape for analytics. I just don't think he has it in him.]