Primacy: When Good Redesigns Do Bad Things

By Jason Carmel | 1 Comments | Posted in in Optimization | Permalink

One of my most favorite websites in the world, eMusic, just got a facelift. I love eMusic because they have a great selection of indie music, they are cheaper than most of the competition, and the music you get is legal and DRM free, as it should be. The reviews of the redesign have been generally positive: now eMusic sports some fancy AJAX-y interface bits, a wealth of third party content (youtube videos, reviews, etc), and a new music recommendation and personalization engine, among other things.

So why am I, a loyal eMusic customer, not happy with what are clearly improvements to the site?

The Primacy Factor

I've been a member of eMusic for almost three years now, and I visit the site like 27 times a day. I don't know why I visit the site so often. Maybe I'm afraid of getting in a rut, which at my age, can quickly lead to a well documented form of musical taste fossilization- the kind where I sit in a corner yelling at pesky kids to "turn down that noise," while I lament the crapification of music these days, and pine exclusively for Hair Metal, Grunge, Disco, or some other fixed aural remnant of a bygone and best-forgotten era.  Before you know it, I'm eating dinner at 4:30, I have a regular bingo night, and I can quote Matlock extensively.

When you visit a site so many times, you get comfortable using it. All of the rough user interfaces are smoothed over by the calm running waters of routine. Any glitches and inconveniences morph into idiosyncrasies and "power user" workarounds as you become increasingly efficient at dealing with the user experience. This is precisely what happened to me relative to eMusic- I adjusted how I browse and purchase music to conform to the old eMusic interface (warts and all) and did such a good job of changing my behavior over time, that this new, improved interface actually reduced my comfort level.

In the world of site-side optimization, we refer to this phenomenon as Primacy. If a customer routinely uses the same feature or traverses the same path on a site, then any change- even one that improves an experience- will negatively impact that customer (who now has to re-learn an entirely new method of doing things). Re-learning anything is annoying at first. As such, the short term effects of an altered experience could make a great improvement erroneously appear like it is underperforming relative to the old version, as these veterans struggle to adjust to the new (improved) world.

For eMusic, I've been biased by almost three years of the old site, and so I will need a lot of time to adjust before I become an advocate. Of course this is dangerous ground to tread for any site, since the people most at risk of defection following a significant change are the heaviest users.

How to address the Primacy factor

It is important for sites that experiment on areas with significant return traffic to not be discouraged by early results of those tests. As a test designer, your options for dealing with the effects of Primacy are to let the test run long enough for the veterans to get used to the new idea, or (even better) to segment the test by new and returning customer groups so you can see whether the design has a positive net impact for visitors who have not been biased by a previous experience. eMusic should definitely be segmenting here to reduce the risk of alienating hardcore users and testing to quantify the results of their new design.

I'm pretty sure the new design is better. It just might take me a while to get used to it.