July 21, 2009 - Digital Marketing Summit at ZAAZ

 

Space is limited for this event! If you would like to join us, please RSVP to emoffet@zaaz.com

How to Design the Best Variants for an Optimization Test

Variant design in Optimization is important, and it deserves more care and attention from Optimization test designers.  In the current practice of test design, much attention is paid to analytics numbers when deciding the hypothesis and direction of a test.  However, numbers by themselves only tell you where the problem is, they do not tell you what alternatives to try.  At ZAAZ, we have been combining User Research methods with our Optimization practice and have found a new piece to this puzzle: User Research in the process of variant design.

So, how can you incorporate UR to design the best variants you can?   You need to…

Stop, collaborate, and listen.   There are several ways to use your friendly local usability/UR practitioner to help generate a hypothesis and variants to test that hypothesis.  Which method is best in which case depends on the nature of the page, problem, and budget in question.  The rule of thumb here is something is always much better than nothing.

Heuristic Review:  When budget and timelines are mercilessly tight, as they often are, a heuristic review can turn up a lot of insight in a little time.  These employ the best practices in user experience as a template to which the site is compared.  The findings from an heuristic review can do two things for a test designer: 

1)    Identify which usability best practices are being compromised on under-performing pages.  This lends itself easily to hypothesis creation.  Test variants can then be alternate ways to implement the best practice on the page.

2)    Uncover places to test on the site that are not immediately obvious through an analytics lens, but that need to be improved.  

Usability Test:  This is my best-practice recommendation when deciding what to test.  In concert with analytics data, usability tests become invaluable.  Usability tests are best introduced when a test designer sees analytics data he or she cannot explain.  For example:

Recently, we at ZAAZ had created a test that was meant to increase the number of purchases of an add-on offer in a purchase path.  We created our variants carefully and lovingly - even bringing UX in for a consult - remaking the underperforming page into a clear, concise, value-communicating standout.  And yet, the ultimate conversion rate for users who saw ANY of our variants was much lower than the control.  We thought:  What gives?  So, we conducted a usability test on the control and on the variants.  In the course of the usability test we found that the promise made by the page we were testing was not clearly reflected further down the purchase path.  Our variants that made the initial promise more explicit led to confusion and abandonment further down the line.  This lack of continuity was subtle – but it was affecting user confidence in the process.  Introducing a usability test was key to getting in touch with our user’s qualitative impressions, which were driving their ultimate decisions in the purchase path.  Put another way, usability tests uncover the root cause of a problem.  Root cause understanding can show you both where you really should be testing, and what your hypothesis should be.

Usability studies can also be used to vet variants before they are launched.  This is a particular advantage with high-traffic tests where any underperforming variant can be costly. 

Eye Tracking:  Eye tracking is a great tool, if used wisely.  It can be used both to define hypotheses and to evaluate variants that have already been made.  That is because eye tracking shows you what users actually notice on the page – where their eyes go.   As a method of inquiry, eye tracking can satisfy what can otherwise be exhausting and unproductive internal debates about what the user is actually noticing on the page, or what is easily ‘discoverable.’  If the name of the game in a particular test is to get the user to notice a key piece of content, eye tracking is your new best friend. 

Other:  The most exciting thing about the intersection of UX and Optimization is that it is new.  We are still discovering new ways to put these disciplines together. 

Rachel Elkington is an Online Test Designer in the Optimization group at ZAAZ.  Previously, she worked in ZAAZ’s User Experience Group.  This combination of disciplines means she gets to have lots of fun scheming up new ways to put qualitative and quantitative methods of inquiry together.  In her spare time she heads up the Pacific Northwest chapter of the American Society of Information Science and Technology, and co-produces InfoCamp – an annual unconference for the user-centered information industry.  She has an MS in Information Management from the University of Washington, and a BA in a liberal arts discipline that people told her wouldn’t get her a real job.

You're Invited to a Social Media / Beer Event at ZAAZ: Thursday June 18th

ZAAZ is hosting the latest in our social media events series next Thursday, June 18th, at ZAAZ hq in downtown Seattle. Join us from 6 - 9 for a panel discussion on agency / client relationships in social media. Panelists will include folks from the agency side who specialize in digital, brand, and PR; along with client-side web and community managers. Here's the announcement:

How do agencies help clients develop richer, more valuable online relationships with customers? Should agencies moderate communities on behalf of their clients? How will budgets adjust to accommodate ongoing, as opposed to project-based, work? What skillsets are best suited for in-house groups? Which agency type owns social media strategy? Should corporations create social media teams, or distribute the work across groups?

Confirmed panelists so far are:

Dave Ballantine, DNA

Ted Zahn, Creative Director, ZAAZ

Jordan Williams, Online Community Manager, REI

We're still confirming another 2 or 3 panelists, and we're hoping for a very lively discussion around the future of agencies--I'll be moderating, so if you have thoughts about topics, please drop them in the comments below.

We'll have a Facebook event up soon, I'll update this post with the link when we do.

I hope to see you there--come say hello!

How Twitter Promotes Quantity to the Detriment of Quality, And Why Twitter Matters Nonetheless

(Cross-posted from Web Social Architecture.)

My Twitter experiment was a “successful failure.”

I started an experiment a few weeks ago, basically using Twitter differently than I had previously. Until then, I'd only followed people I know personally--friends, family, colleagues, and professional acquaintances. But a couple weeks ago, I decided to try following everyone who follows me.

This is not a large number of people by Twitter standards (a couple hundred), but I was immediately annoyed with Twitter, and I stopped looking at it for a while. The people I was using Twitter to keep tabs on were simply drowned out by the deluge of incoming tweets, which I found almost entirely uninteresting and not at all useful.

I didn't give up. I created a group in TweetDeck for people I know personally, so I could follow them separately from my full collection of tweeters. That kind of worked on my computer, but it didn’t work on my phone--I never used poor, neglected Twidroid any more. I had to turn off the updates.

I started sort of hating Twitter. What I thought was interesting, though, was people sending me messages thanking me for following them. "You're... uh... welcome...?"

That was what raised the question:

Why do people on Twitter care how many people are following them?

The answer is, because Twitter wants it that way. The affordance for social capital on Twitter is all about quantity: How many following, how many followers, and how many updates posted. The measure of a person, in Twitter, is all about the total amounts accumulated:

The display of these measures represents just a fraction of the text on a Twitter page, just a few pixels of space—but they’re critical, because they are the attributes attached to the representation of my identity. Effectively, they are me, and I am valued on the basis of their accumulation. Other people, at a glance, get a sense of who I am that’s based on the amount of stuff I have, and not what kind of stuff, and not on the value of that stuff.

This is a bad thing. As Twitter’s massive increase in number of users the last few months has started to illustrate, an increase in users also increases the breadth of topics, decreasing the signal to noise ratio. Six months ago, when Twitter was populated mainly by a relatively narrow group of social software and Web aficionados, there was a stronger sense of Twitter being a community. Today there’s much less topical focus, and Twitter’s limitations as a social tool have become much more readily apparent.

The fix: Build quality-based social capital.

What if, instead of showing, underneath my photo and name, the number of people I follow, the number of people following me, and the number of times I’ve posted, Twitter showed data that reflected the value of my activity? For example, what if my Twitter identity were associated with the percentage of my posts replied to, favorited, and re-tweeted by others?

I’m going to go ahead and suggest that this tweak would substantially alter the way people use Twitter—for the better. Because the measure of a person would be their ability to consistently create value, people would be encouraged to be more interesting. But notice that the measures I’m suggesting don’t reflect the number of people finding any particular tweet valuable, only the percentage of tweets found interesting by someone. That’s an important difference, because it says you don’t need to play to the crowd—that one-to-one value counts the same as one-to-many value. So you don’t have to be interesting to lots and lots of people, only interesting to a single person.

And there’s more here yet: Because the social capital of the system would be tied to recognition by others, the system would encourage connections that were active, and based on mutual attention and interest. “Capital” here works in the sense of “currency.” We’d pass it back and forth—if I like your tweet, I reward you with a reply, a retweet, or a favorite. My doing so counts for you. You look for an opportunity to reciprocate.

Quality and value, when rewarded, are fundamentally self-reinforcing. So if Twitter supported quality and value, great ideas would be passed along more, reaching more people. Reputation and credibility would flow from meaningful contribution. Relationships grounded in mutual helpfulness would flourish.

Now, as Twitterers like the prolific Nancy White illustrate, quantity and quantity are not mutually exclusive. And it’s certainly better to have more good stuff than it is to have only some good stuff. So a shortcoming of my suggestion is that it doesn’t reward the consistent creation of value over time, or put another way, the creation not only of consistent value but of a LOT of it, consistently.

Batting average only counts beyond a threshold of at-bats. Total home runs matters only relative to years played. If Ichiro goes 4-for-4 in the season opener, he’s batting a thousand, but that doesn’t yet qualify his season for any discussion of the all-time best. So the real answer, if we’re going to get serious about it, is for Twitter to create a quality algorithm and use that as the measure of an individual’s contribution.

Twanalyst is a gesture in the right direction. It’s a service that analyzes Twitter accounts across a number of quality-focused dimensions, resulting in a (sort of cheesy) personality test: I am very pleased to tell you I am a “chatty coherent guru.” Yes!

Kind of fun, but what I actually really like about Twanalyst is that it’s looking primarily at the nature of what you’re doing on Twitter, rather than the amount of it. That, I believe, is the future on an increasingly noisy web.

Speaking of the future, what about Twitter’s future?

At this point, it’s kind of common knowledge that Twitter has passed a… can I say… Twipping point. As Twitter gets huge, and as the purchase offers get bigger, which direction it takes—toward signal or noise—will help determine its fate as a social application.

Will Twitter end up yet another spammy channel, a trivialized feature of a larger social network, under pressure to monetize? Or will it add a valuable layer of ambient awareness to rich, multi-channel online relationships? I’m rooting for the latter, not just because I’m a fan of Twitter (I am), but because I want to live in a digital future where quality, meaning, order, and value trump unwanted noise.

 

But what I’ve said here is also kind of wrong. The story (like stories always are) is more complicated.

Despite all that—despite the shortcomings of a system that rewards meaningless connections, high volumes of worthless posts, and claiming to listen to more people than anyone possibly could—the truth is we can’t ignore Twitter.

And there’s a reason it’s hard to get comfortable with that truth. Twitter, in the midst of its deluge of valueless tweets, also produces a kind of magic: The collective value of all the noise has a tremendous potential to add up to something meaningful—something with a value beyond what any particular contributor can produce. As the Moldova uprising most recently illustrated, and as did the presidential election and Katrina did before it, Twitter, at its best, is a powerful venue for the expression of real-time collective intelligence. And that’s where businesses need to stand up and pay attention.

Straw Horse: An Enterprise Social Media Platform Feature List

(Cross-posted from Web Social Architecture.)

We did an internal exercise recently that produced a list of the advanced features we think are crucial for a successful enterprise social media platform. The idea is that functionality for user participation across every owned venue should draw upon a central system, enabling a multifaceted approach to CRM, data analysis, reporting—and ultimately leveraging distributed corporate efforts to generate enterprise business intelligence.

I’ll share the results of that exercise here, with the caveat that this is undoubtedly a partial list only. Your comments and suggestions are welcome, of course!

I helped think through this some of this stuff, but the bulk of the credit (including for the writing) goes to my ZAAZ colleague Ariel van Spronsen. We also got some help from another longtime friend and colleague of mine, Gary Carlson, an expert on enterprise metadata management. Enjoy:

 

Reputation (authority systems)

When properly implemented, reputation systems are excellent for creating trust and motivating users to participate at greater and greater levels of engagement. When implemented poorly, High volume of participation or seniority are rewarded—the key to getting reputation right is to focus on the quality of the contribution, not the amount of it.

In implementations where credibility matters, reputation is critical. See this great presentation from Bryce Glass of Yahoo for more detail.

User management

The platform provides the opportunity for centralized management of user data and permissions, including authentication, account management, personalization, segmentation, and behavioral targeting.

User data can provide the connective link among multiple social networking implementations (personal, business group). Content owned by a user can be shared among these via permissions or syndication.

Identity services

A unified data repository means an individual user can centrally manage her public-facing identity, and also create a more robust data picture for the business .

Interfaces from the platform access custom degrees of information contained in the central identity.

Quality algorithms

User-generated ratings have important utility, but translating them directly to measure “quality” is fallible. Ratings are opinion-driven and the ability to control input is minimal. However, combined with analytics data using weighted algorithms quality becomes a more stable and useful metric that both users and business can trust.

Recommendation engine

An important use for user-generated data and analytics is the ability to enrich experiences with recommendations, prompting discovery and deeper engagement. A centralized social networking platform is primed to leverage this functionality.

Taxonomy-driven folksonomy

Tags are a powerful way to augment search and increase information “find-ability”. They also give the business a powerful view into how people are thinking about the tagged content.

A purely user-generated tag set (a “folksonomy “) has issues such as misspellings, tense shifts, and count (singular vs.. plural). A taxonomy-driven folksonomy maps user tags to a controlled vocabulary authority to allow for specific schema analysis.

Video, audio, and photo streams

A significant part of the communication among social networks will be in multimedia forms. Easy uploading, tagging, and sharing features will create a robust social media environment, greater user satisfaction, and increased engagement.

Mobile

The demand for social media in the mobile space is undeniable. Application development for a new breed of smartphones is rapidly increasing as the ability to manage social networking functions becomes a key differentiator for users. The platform should provide for mobile implementation as well as web and API calls, and it should support both content consumption and content production via mobile.

Custom syndication

Custom syndication allows users to filter and process feeds in ways that are meaningful to their specific information needs. Yahoo! Pipes is an example of a custom syndication mechanism.

Custom syndication can augment other elements of a social networking system, especially for a user group that is highly specialized in goal and purpose.

Social bookmarking

Social bookmarking functions promote the development of shared information collections among networked groups.

Collaborative filtration

Collaborative filtration gives users the ability to vote submissions (bookmarks, feeds, entries, etc.) up or down. A popular feed-based example of this is Digg. In the marketing realm, Dell’s IdeaStorm lets users identify the best ideas for product development.

Private groups

Ad hoc, user-created groups for sharing or collaboration can support communities of practice and leverage user data management features.

Microblogging

Twitter is perhaps the most ubiquitous example of microblogging, which invites low-threshold, stream-of-thought information sharing and ambient connection among networked groups. Link sharing, whether to photos or other assets, is pervasive in microblogging, creating connections that can be used in many ways.

Marketplace

A social networking platform could provide functionality for connecting people to products or services, offered by the company or by one another. Examples are Craigslist, eBay, and Xbox Live Marketplace. Marketplace connections give a strong view into communities’ product needs, and they also support, to varying degrees, the purchase process itself.

Chat

Instant communication among community members creates a synchronous communication layer that can be particularly useful within a collaborative environment for communities of practice.

Moderation Tools

Property owners need tools to support management of their users and communities, along with the structures to support governance and workflow at distributed and global levels.

 

I’m sure there are other ideas out there. and for that matter lots of ways to slice and dice what constitutes a “feature.” For example, is blogging a feature, or is discussion? Or are those both high-order uses supported by features like WYSIWYG publishing, commenting, etc?

I don’t really want to get into an argument about that stuff, but I am very interested in what kind of emerging capabilities corporations need to support in order to realize the full promise of engaging with their constituencies online.

Do share!

Unique Visitors...A new definition?

I WISH!
Well, if the goal was to spark a discussion and fuel up the fireplace...mission accomplished.
Eric Peterson's post last week on "Unique Visitors Only Come In One Size has has done just that...
(http://blog.webanalyticsdemystified.com/weblog/2009/03/unique-visitors-only-come-in-one-size.html)
It's needless to say that Unique Visitors has been a top subject matter on many posts.

Over the years we have all discussed its drawbacks, using a weighted average, how to improve it, and in some ways have found ways around the metric. Plus personally, how many unique visitors I get adds no value to my ongoing analysis. I'm interested in behavior or better yet, whether they are going to accomplish what I desire and/or what the user wants...I know I am the first to use authenticated users and visits before visitors.

But this is not about me...

As someone who participates in the process, fact still remains that the individuals of the Standards Committee have taken these and a bunch of other scenarios into account.
Though I clearly see the IAB's point, a new term is in order. But to say that we are going to deny the definition that we've all used and 'grown' to know, is not going to happen.
I think that we need to validate any new proposal.
It’s not that I disagree with the IAB, it's just unrealistic at this point and at this time we are attempting to establish a common language for 'right now.'
I think its good to expect more and move the industry forward.
Do I think the two should be named differently, of course. Do I believe the industry deserves better measurement, YEP! Is it a good debate, no doubt. But boy do I have bigger fish to fry.
When we get a better metric we will use it and guess what? We would call it something else!
Why? Because even before this, the reality is that it was already confusing...

Among many parts within the post, I found the following to be interesting...
And Joe did clarify for me what a “measurement organization” is … he just didn't directly clarify the impact on web analytics vendors.

HMMM...I recall a conversation between the IAB and the Standards Committee where it was stated by the IAB that it would affect analytics vendors....perhaps I completely misunderstood.

Also, last time I checked the word Panel (used in the IAB definition) in 'our' world, it did not mean population. So...there are negatives on both ends...Plus, where's the algorithm? Let's get that going before we start calling things out. Or perhaps it exists? Be sure to let us all know. Perhaps this may be in the works?

Oh and as someone who worked for DoubleClick, Inc during the early days, I can add the important issues we have with privacy behind identity but I won't...
However, somehow it was forgotten!

There's so much work that goes into all of this and unfortunately this all has spilled over to becoming personal. I am glad to see it has tapered off, because this is all far from personal.
All I can say is, let's embrace what we do have, strive for better, enhance relationships, lead and honor those who volunteer their time just for the love of it.

 

 

 

How to Develop a Consistent Targeting Strategy

Targeting, a hot-topic marketing practice, deserves digital marketers' attention. Targeting can help marketers deliver the right content or message to the right person, at the right time and in the right space. But targeting can be an ambiguous concept. Often it means different things to different people.

Most digital marketers divide their efforts between offsite and onsite targeting. Offsite targeting comprises advertising and demand-generation activities like search, display, and e-mail. They drive performance by sending traffic to your online real estate. Successful marketers recognize that traffic by itself won't guarantee success, so they apply site-side strategies to maximize performance once a visitor lands on the site.

Today's holy grail for targeting requires the integration of off- and onsite targeting efforts. This happens too rarely, however. Ask yourself these questions: How often does your paid-search team seek data and insights from the person who manages Web site analytics? How often do your site-optimization efforts employ data and insights from the teams that manage search and display media?

Below are the most typical and effective forms of off- and onsite targeting, as well as tips for how businesses can successfully bridge the divide so targeting represents one consistent marketing strategy, rather than separate parts of a disjointed and sometimes ineffective effort.

Offsite Targeting

Most ads are served contextually, meaning that if you visit auto site Edmunds.com, you'll probably see a lot of ads related to cars. But offsite targeting is more precise. Here you rely on a huge network, or networks, to provide different ways to serve traffic based on user behaviors or profiling. The targeting can happen by any number of factors. For example:

• Geotargeting. Only shows ads in states or regions where a business has retail stores. This is common for both display and paid search campaigns.
• Interest group. Only shows an ad to someone who has visited a Web site on a specific topic (such as autos) in the last month.
• Prior view. Shows someone who viewed but didn't click on the first ad a second ad with a free-shipping coupon.
• Demographic targeting. Loosely targets display ads to network sites that fit a general demographic profile. Paid-search platforms like Microsoft adCenter offer user-based targeting and allow for incremental bidding where demographic criteria like gender and age can be identified.
• Day-part targeting. Serves an ad during specific times of the day and days of the week.

While results vary greatly depending on product, business, season, and so forth, the industry has pretty much proven that targeted ads consistently outperform their generic and even contextual counterparts.

A popular form of offsite behavioral targeting is search re-targeting. Here, an ad-serving platform can identify a person who has performed specific searches in the past. A corresponding display or text ad can then be served to that person across select network sites they visit. So if you've searched for "new cars" or "hybrid cars" on Google, don't be surprised to see display ads for the Ford Fusion Hybrid or Toyota Prius when you start browsing other sites later in the day. (You might want to think twice about what you're searching for, because if your spouse reads this column and sees a bunch of unsavory display ads on your home computer...well, you get the picture.)

Onsite Targeting

Onsite targeting relies on a more focused array of characteristics to align a Web site to what a visitor should experience. In addition to offsite parameters (geography, interest group, etc.), as well as other general parameters, such as referring source, you can use any authenticated data (i.e., information kept behind a login) to provide a personalized site engagement for each visitor. As with offsite targeting, there's no industry average for how a targeted Web site improves business. But anecdotally it isn't hard to see how a customer will more likely return if the Web experience closely aligns to what she's clicked on, viewed, or purchased before. At my agency, our site-side targeting test programs are among the most effective ROI (define) proven services we provide for clients.

Amazon and eBay are industry leaders of onsite targeting. Each individual eBay or Amazon user could experience a completely unique site, depending on their past behavior and profile. If you've been looking at auctions for Xbox consoles, then the next time you visit eBay, you'll probably experience a site that features Xbox games, Xbox accessories, and related products. Whether you sell or buy, how much you pay, what types of products you buy, and what time of day you tend to buy certain products are factors that drive onsite targeting.

Fortunately you don't need to be an eBay or an Amazon to take advantage of site-side targeting. Tools like Omniture's Test & Target and Optimost allow companies to deliver targeted site experiences.

Bridging the Divide

Bridging the divide between off- and onsite targeting efforts will be unique for every business, but here are some recommendations:

• Define success. You may think this is obvious, but at my agency we consistently find that separate teams and stakeholders have different definitions of success. It's critical that your teams work together and toward the same goals.
• Understand targeting mechanisms. Different ad-serving technologies offer different types of targeting -- behavioral, geographic, demographic, and so on. Make sure you understand their capabilities. Test and use them. I'm particularly excited by new targeting technologies that allow for search re-targeting.
• Use one consistent targeting strategy. Even though this sounds logical, it's more difficult than it sounds. Onsite and offsite folks don't talk to each other often. You will need to establish a process to ensure this happens.
• Share data. Once you get your teams and stakeholders playing for the same goal, encourage them to share data. And not arbitrarily. At my agency, our search team consistently delivers keyword trend and performance data to our onsite optimization team. This allows our optimization team to design tests and targeting strategy based on offsite opportunity. Likewise, our analytics team shares behavioral and attitudinal data with our media teams. Everyone has a consistent standard for what success looks like. There is no divide between offsite and onsite.
• Test, measure, execute. Repeat. Don't assume you're getting it right. Validate your strategy with consistent testing. For onsite, A/B and multivariate testing are excellent programs. Don't stop when you get promising results. Execute, measure, and repeat the process.

 

Analytics spawned yawning among analysts?...is that possible?

I never thought I'd see the day that I would yawn during an analytics discussion.  But it happened.  I always thought it was my job to motivate people, show them the value and they will move forward.  I get little butterflies as I pull and integrate numbers and find a story to share.  I often feel like the journalist, breaking news.  But this time, I felt like I had been transported to 2004 and I though that was a good year for me, I was not pleased.

The lack of analytics mobility is starting to get boring. Apparently, I am not alone.   After so many years, let's move on folks!  I keep hearing things like, 'analytics is our focus, we need to act'...okay so why do you shy away from tracking based on your goals, not just the 'data' you are able to get at this time?  Why when you are given insights you don't act on them.  Segmenting is a bad word and you still find geography 'views' valuable.  This is just all very lame.

Now, during this economic turmoil, more than ever we need to stop making excuses for why analytics funding is not a priority and why you cannot act on customer requests.  Don't get me wrong, we have worked with so many clients that have grown and are now data driven businesses. But far too many are stagnant.  Ensure that you are not on that list. 

For 2009, you have already asked yourself, what am I spending money on?  Now, ask yourself what are you spending time on?  What are key stakeholders focusing on?  How is that growing or even maintaining your business?  Look at your analytics maturity level and if you see yourself having the same discussion you had in 2004. Stop.  Start the roadmap on moving forward and monetizing your business so you can optimize. 

Of course I know you are still thinking about costs so...put together a cost benefit worksheet (yes, it takes time and you do need to understand what you are doing) and among the obvious ensure to include:

1. Speed - what customer driven projects can we quickly turnaround that is going to influence return?

2. Better Results - improvements in results because actions were taken - savings included -

3. Shorten the meetings and discussions on items you have action plans for or documented a roadmap...and use them already...you'll be surprised how much time and money you'll save.

You can adjust things as you go...but take the step.

Remember that yawning is contagious.  Don't put yourself in a position where your analyst' yawn, de-motivate your stakeholders and it all transcends to consumer behavior.

Primacy: When Good Redesigns Do Bad Things

One of my most favorite websites in the world, eMusic, just got a facelift. I love eMusic because they have a great selection of indie music, they are cheaper than most of the competition, and the music you get is legal and DRM free, as it should be. The reviews of the redesign have been generally positive: now eMusic sports some fancy AJAX-y interface bits, a wealth of third party content (youtube videos, reviews, etc), and a new music recommendation and personalization engine, among other things.

So why am I, a loyal eMusic customer, not happy with what are clearly improvements to the site?

The Primacy Factor

I've been a member of eMusic for almost three years now, and I visit the site like 27 times a day. I don't know why I visit the site so often. Maybe I'm afraid of getting in a rut, which at my age, can quickly lead to a well documented form of musical taste fossilization- the kind where I sit in a corner yelling at pesky kids to "turn down that noise," while I lament the crapification of music these days, and pine exclusively for Hair Metal, Grunge, Disco, or some other fixed aural remnant of a bygone and best-forgotten era.  Before you know it, I'm eating dinner at 4:30, I have a regular bingo night, and I can quote Matlock extensively.

When you visit a site so many times, you get comfortable using it. All of the rough user interfaces are smoothed over by the calm running waters of routine. Any glitches and inconveniences morph into idiosyncrasies and "power user" workarounds as you become increasingly efficient at dealing with the user experience. This is precisely what happened to me relative to eMusic- I adjusted how I browse and purchase music to conform to the old eMusic interface (warts and all) and did such a good job of changing my behavior over time, that this new, improved interface actually reduced my comfort level.

In the world of site-side optimization, we refer to this phenomenon as Primacy. If a customer routinely uses the same feature or traverses the same path on a site, then any change- even one that improves an experience- will negatively impact that customer (who now has to re-learn an entirely new method of doing things). Re-learning anything is annoying at first. As such, the short term effects of an altered experience could make a great improvement erroneously appear like it is underperforming relative to the old version, as these veterans struggle to adjust to the new (improved) world.

For eMusic, I've been biased by almost three years of the old site, and so I will need a lot of time to adjust before I become an advocate. Of course this is dangerous ground to tread for any site, since the people most at risk of defection following a significant change are the heaviest users.

How to address the Primacy factor

It is important for sites that experiment on areas with significant return traffic to not be discouraged by early results of those tests. As a test designer, your options for dealing with the effects of Primacy are to let the test run long enough for the veterans to get used to the new idea, or (even better) to segment the test by new and returning customer groups so you can see whether the design has a positive net impact for visitors who have not been biased by a previous experience. eMusic should definitely be segmenting here to reduce the risk of alienating hardcore users and testing to quantify the results of their new design.

I'm pretty sure the new design is better. It just might take me a while to get used to it.

Site Optimization: Lipstick on a Pig?, Or, Bacon and the Theory of Local Maximum

Cross-posted from Web Social Architecture.

 

Jason Carmel is a colleague of mine I learn a ton from. His expertise is in web site optimization--running experiments where he tests versions of web pages against each other to see which performs best. (Not to be confused with search engine optimization, improving a site's visibillity in search engines.)

Jason is a fairly unflappable guy. Nonetheless I recently started making an effort to get his goat. He gives me just enough encouragement that I keep going. The gist of my teasing is that optimization is nothing but a mechanical exercise to determine whether a red button works better than a blue button. "Glad to hear that red button worked out better by 2.84 percent Jason. The sum of your creative energy has produced yet another quarter million in revenue. You must really love your life, man. Hey, have you thought of trying one of those animated GIFs instead of a regular button?"

Fortunately, Jason is twice as nice as me, as well as twice as smart. He takes my ribbing well--and responds thoughtfully to the serious question underlying my teasing: We know optimization can move big numbers in terms of revenue, but can it do more than simply tweak pages to bump up conversion? Can it vet creative concepts? Can it maximize the creation of mutual value between businesses and customers? Can it help create more engaging experiences?

(image credit, found via Lee)

The short answer here, according to Jason, is that it depends, partly on what you're trying to achieve. If all you're focused on is moving business value measures, you're probably putting lipstick on a pig. But testing against value creation has the potential to uncover game-changing opportunities.

Here's an email exchange between Jason and me, in which he explains in a little more detail:

 

RYAN:

That whole web site optimization thing—isn’t it really just putting lipstick on a pig?

 

JASON:

I think “I hate you so much” might be a succinct way of responding, but I'll provide a little more detail:

Web Site Optimization is exactly like putting lipstick on a pig, but only if you start out with a pig. And if you are starting out with a pig, your opportunities for improving things are limited, and you’d be using the wrong tool to fix the problem. We are talking here about the concept of a “local maximum” which is a fancy, math term applied to mean “the best something can be within a limited dynamic.” Consider the aforementioned pig’s ability to fly, which, metaphorically speaking, is not particularly developed. We could take a pig and genetically modify it to be more aerodynamic. We could investigate building pig hang-gliders and attempt to train the smartest pigs to use them. But even in the best case, with the most aerodynamic pig, benefiting from the best training, and using the best pig flying technology, it will never fly as well as a bird. The best case flying scenario for a pig (the pig’s local maximum as far as flight is concerned) is nowhere near as effective as a bird’s. In that scenario, you’d be better off exchanging the pig for a bird at the start, rather than waste any time or effort teaching a pig to fly better.

Applied to the web: if a site sucks so much- if the goals and purpose are unclear, if the information architecture looks like my desk (at the moment), if the navigation is counterintuitive and the messaging has absolutely no intersect with the audience, then no amount of optimization in the world will make it right. The local maximum of that crappy site is too low for any optimization to matter. Or (even worse) you’d need the infinite number of monkeys to stop typing Shakespeare and to start applying experiments to your site to get the right combination where testing would make a real difference. Neither is very efficient. If your site is the pixilated equivalent of a pig, you need much more elemental help from a user experience expert first (know any?). Until you fix the fatal flaw(s) in a site, anything else you do will be throwing good money after bad.

Site Side Optimization works well in circumstances where the local maximum is high, but for some reason, the site is not achieving it. This can be due to single points of failure on the site, like a specific conversion path or page underperforming, or because the audience needs to be targeted more specifically, or because the existing content is stale/irrelevant. In each of these cases, experimental testing can make a huge difference. Optimization also works exceptionally well (and this is far more interesting to me) when applied as a method of trying out a new (and potentially risky) idea that could radically change and significantly improve an experience.  In both of these examples, the basic site is healthy, and the optimization program serves as a tool to reach its fullest potential.

 

RYAN:

But what I keep looking for is the way to test birds against pigs, not in the sense of which flies better, because as a user experience expert I do have the capability to predict the winner of that contest—but when I don’t have a clear sense of the best conceptual solution. For example, maybe I just can’t decide between eggs and bacon. Can optimization help design a better breakfast, or only decide between pulp and no-pulp in the OJ?

 

JASON:

Optimization can test more conceptual ideas, but it will be really hard to unpack the WHY after we determine which one wins. Most sites aren’t deciding between bacon and eggs, but rather between the bacon, eggs and hashbrowns with coffee or the granola, fruit and yogurt with yerba matte. If the former wins in a test, I don’t know whether it’s because of the bacon (which will usually win over everything) or the coffee, or because the person deciding had granola for the past three days, and would have taken ANYTHING other than more granola.

The other trick about testing high concepts in a website format is that you would have to build each solution to test them, which is usually more expensive than testing out wireframes or front-end prototypes in front of a more controlled audience.

 

RYAN:

First, you seem to be suggesting that a test win for the bacon breakfast might not imply extensibility for bacon breakfasts in general: That because, lacking control, the results might be idiosyncratic, they might not therefore apply broadly. Next week you might get a different result. My question is, why does that matter? And why does “knowing why” the bacon breakfast worked matter, as long as you know it worked.

 

JASON:

It is definitely a question I get from clients a lot. Why do I care about the individual elements of a variant- if the variant as a whole makes us more money, let’s just launch it and move on. I can’t fault the sentiment, but knowing why the bacon worked could lead to better tests, more focused messaging and (even more) cash money. I want to know that it’s the bacon by itself that is the motivating factor outside of all the other influences. Let’s say that we ran a test breakfast against a bowl of Total cereal and we tested bacon with powdered eggs as the experimental variant. Now let’s assume that bacon and powdered eggs lost to the control by 1%. You could take the position that we would do better to serve Total because we want to avoid losing that 1%, and you would be right. But what if you knew that the bacon by itself actually improved the breakfast by 15%, but the powdered eggs were so crappy that they hurt the breakfast by 16%, so you netted the 1% loss? If I controlled for all the variables in my breakfast, that knowledge would a) help me make a better breakfast overall (just serve bacon), and b) will also prevent me from throwing away a positive variable simply because it was paired with a really negative one.

I imagine you run into that a lot with prototyping and wonder how you deal with it in the UX world.  If a subject totally fails at a task, are you ever afraid of overcorrecting a prototype to account for it? Do you ever throw the baby out with the bathwater? How do you control for that?

 

RYAN:

A great question. One of the answers is that in usability testing, you're looking for usability problems. So as long as your test participants are representative of your users as a whole, major failures are, practically speaking, never anomalous. If your user population is one million, and one of the eight people participating in your test has trouble understanding some aspect of the interface, what are the odds they're the only person who's going to have that problem?

The other thing I wonder about is what happens when what you’re trying to accomplish is harder to measure than conversion (e.g. brand lift) or if you want to measure it over time (e.g. engagement). Especially in social media, it’s quality that matters, not quantity. You want to know how valuable your user-generated videos are more than you want to know how many of them you have. Can web site optimization help you get to answers?

 

JASON:

Ah, you and your social media. When are you going to come to terms with the fact that this whole thing is a fad? The future is in email, Ryan, and lots of ‘em. Mark my words.

Absent a more qualitative tie-in with optimization (surveys, satisfaction scores, etc.) you will be hard-pressed to get good data about branding or the impact of social media. But I’m not saying you shouldn’t optimize for branding or social media. I’m saying you need to get that qualitative kicker. I’ve done a few branding tests, and I think they provide some interesting feedback. But I’ve never optimized where a KPI has to be judged on quality (e.g., good comments vs. troll comments) or off the site entirely (e.g., buzz in the blogosphere). Sounds fun.