Search Engines and Brand Lift

By Anders Rosenquist | Feb 24, 2010 3:50:07 PM

Adweek recently published the results of a new study conducted by ZAAZ, Wunderman, BrandAsset® Valuator, and Compete that explores the relationship between the search engine that consumers use to find a brand’s website and the consumer’s perception of that brand. The study found that loyal users of Bing, Yahoo!, and Google have distinct characteristics that benefit some brands more than others.

“This research demonstrates that marketers have a real choice to make when formulating search strategies,” said Shane Atchison, CEO of ZAAZ. “The search engine acts as a kind of ‘train’ on the Internet. Each train provides a different set of unique results or ‘destinations.’ Consumer preference for a specific train demonstrates a unique demographic and psychographic profile.” That preference, the study found, can have enormous impact on any brand a consumer searches for.
 
ZAAZ led the analysis portion of this pioneering research study, which combined attitudinal and behavioral data to draw a more complete picture of consumer search behavior.  In this two-part post, we’ll first discuss our methodology;   in the second post, we’ll discuss the findings and how marketers could put these findings to use.  (Note: this research was independently conducted, and was not funded by any of our clients.  You can view a version of the results presentation here: Search Engine Research (pdf, 850 kb)
 
Our team combined two extensive databases from BrandAsset Valuator (BAV) and Compete. Both BAV and Compete own the largest databases in their areas of expertise.
 
BAV’s databases contain a range of attitudinal information – focusing on 72 brand metrics – and use the world's most comprehensive database of brands. Its BrandAsset Valuator model has measured brands since 1993; today, over 35,000 brands have been evaluated, and over 600,000 respondents in over 50 countries have been surveyed. To find out more about BAV data, try comparing a few of your favorite brands using the ZAAZ-built interactive tool: http://thebrandbubble.com/explore/.
 
We also used behavioral data provided by Compete. Compete combines site and search analytics and surveys on a diverse sample of more than 2 million users in the U.S. to understand what people are clicking on and why. Compete manages the largest pool of online consumer behavior data in the industry.
 
The datasets from BAV and Compete  both contained information on the same brands across verticals including retail, travel, wireless, and automotive.  Additionally, both data sets contained a field with information on a consumer’s primary search engine. Taken independently, each data set provides insights into search engine use.  However, when ZAAZ combined and mined the data sets, we found a much more powerful and complete story.
 
At ZAAZ, we live in a world of disparate data sources; we have data for Web analytics, social media, usability, optimization testing, surveys, and more.  Almost always, this data is supplied through different tools and in different formats.  Most online marketers will be familiar with this scenario: too much data, too few connections, limited insights and action.  We’re constantly looking for new and innovative ways to combine data and tell a true 360-degree story of the customer.
 
The research published by AdWeek is certainly not the first time we’ve been through this process.
 
Working with Ford we’ve joined attitudinal survey data to behavioral Web analytics data, and analyzed this data by unique user.  With this combined approach, we are able to map how different user journeys affect customer’s brand perception and likelihood to purchase a Ford vehicle.  As a result, we have quantitatively identified what areas of the Ford site and types of content are most effective at driving sales – an insight we never could have gained by analyzing the data sets in isolation.  Ford has used this data to channel media, redesign site architecture, and run a data-driven optimization program.
 
Working with Dell, we combined site optimization testing with in-person usability testing to understand why a particular creative treatment won. For a particular campaign, we found from our optimization tests which page variants generated the greatest interaction with the page. In addition, our analytics data showed that there were several key places in the purchase path where users were abandoning at an alarming rate. To gain insight into why this was happening, we conducted usability tests in our Seattle lab with Dell consumers, having them step through the purchase path and describe their process. We were able to uncover how participants’ attitudes toward the site and the brand changed through the process, why parts of the site became annoying, and why they decided to jump out of the purchase path—either to find a different product, or to go  a competitor’s site.  The insights gleaned from combining optimization and usability test data helped generate  site changes focused on improving user attitude and increasing conversion.
 
With the search engine research, the objective was to understand the relationship between a customer choice of search engine and the brand perception of products being searched.  Once we had the data sets combined we set to work looking for trends.  Using a series of aggregations, drilldowns, and data visualizations, we were able to observe trends in the data common across attitudinal and behavioral data sets. Once the trends in the data were clear, we pulled together stakeholders from BAV and Compete to build consensus on the interpretation of the data as well as actionable strategies.
 
Next week we’ll discuss some of the findings from the research and how to take action on these findings...

By Erik Koto and Anders Rosenquist

Crafting a Great Community / Social Media Policy

By Ryan Turner | Nov 21, 2009 10:31:08 AM

(Cross-posted from Web Social Architecture.)

I've said before that there are three crucial elements of online community policy: Legality, appropriateness, and relevance. I'm now thinking that there's actually a "hierarchy of needs" among those three, and that taken together and put into practice, they are much more than a set of rules for what's ok to do in the community. They’re more like a social contract, a creator of quality and value.

 

Legality isn’t exactly black and white, as you’d sort of want it to be. But it is quite straightforward in that you basically need to do what your lawyers tell you. In many cases there’s some education and negotiation, but there’s also a reason lawyers get paid a lot. In the end, you do what they say.

Likewise, appropriateness is pretty easy. You generally rule out profanity and abuse. From there you can dial the politeness requirements up or down to suit your needs. In fantasy football, for example, smack talk is part of the fun; in a forum for survivors of childhood sexual abuse, behavioral standards are much higher.

But deciding what's relevant and what isn't is a very different story. I consulted, for example, with REI on their online community strategy and governance, and there are a few great examples from that exercise of how challenging relevance is to define:

 

• It’s tempting for an outdoor retailer with a merchandising focus to align community discussions with its product taxonomy. But REI doesn’t sell photography stuff, and photography is a huge part of many people’s outdoor experience. They want to talk about it. So, this one isn’t too tough, right? How about:

 

• Would it be ok to talk about snowmobiling? To post photos and videos of snowmobiling? What about heli-skiing? REI’s focus on stewardship and sustainable enjoyment of the outdoors makes this tough, because most cross-country skiers despise snowmobiles, and heli-skiing is accessible only to the well-off. And then there’s the whole snowmobiles in Yellowstone thing. Yikes. But wait:

 

• In regions like Texas, among the outdoor activities most enjoyed by REI customers are hunting and fishing. Would it be ok on an REI web site to post a photo of yourself posing with the elk you just shot? The poor barefoot hikers in Portland would pass out! But the folks in Plano, or Missoula, would get it. So maybe the elk feels like an extreme example. What about a photo of your six-year-old proudly displaying the day’s catch of bluegill?

 

The thing is, these are brand questions. And when you involve customers in content creation, you expose yourself to all the variety of your customers’ senses of what your brand is all about, from regional variation to variation in taste and preference.

It’s messy. And yes, you have to exercise control. But a degree of openness is also critical, and not only because people won’t participate in a system they perceive as restricting their self-expression. Online community and social media represent a tremendous opportunity to give customers an emotional stake in the brand, a sense of ownership that will increase value at the far end of the funnel—increasing loyalty and generating word of mouth advocacy.

Craft your policies—both outward-facing in community settings and inward-facing for managing your own company’s social media activity—not with the mindset that you need to restrict activity or restrain creative self-expression, but with the mindset that you need to enable creativity and empower participants. I think of it as analogous to sports: The rules are there to make the game safe, fair, and fun—not to keep people from playing.

Where to Start? 20 Questions for Corporate Social Media

By Ryan Turner | Nov 19, 2009 3:46:28 PM

(Cross-posted from Web Social Architecture.)

Our social media engagements typically start with some kind of assessment, with varying degrees of formality and scope. We have an internal list of questions we use to plan these assessments, some of which are more relevant and important for a given engagement than others. Here they are:

 

1. Have you formalized the goals, KPIs, and reporting for your social media activities?

This gives us a sense of the degree to which social media efforts are aligned with the business, as well as the current state of listening, analysis, and reporting.

2. Do you know who’s talking about you online, what they’re saying, and the scope of their influence?

Most (though not all) companies I’ve worked with have a general sense of what’s being said about them online. Typically, the past year, this sense is mainly anecdotal. In the next year I expect to see much more systematic, sophisticated, and analytical listening. But if you’re not there yet, you’re not alone.

3. How effectively are you able to respond?

Yes, this begs the question of whether a business is responding at all. For those who are, the question of degree of effectiveness can be a stumper. The real question here is: How do you know how effective you are (see #1)?

4. What technology tools are you using to monitor social media activity around your brand / product / service?

People really are surprisingly resourceful when it comes to using free tools to listen online. Even for businesses without a sophisticated listening platform in place, a conversation about the tools they’re using tells us a lot about what they care about and are (or aren’t yet) able to measure.

5. Which groups and individuals are informally involved in social media activities?

Once you start walking around asking people, the variety here can be surprising. Typically corporate social media efforts emerge out of PR, Marketing, or Customer Service. But ad hoc efforts are very common, and there’s usually something important driving them. Building out a strong program requires accommodating, supporting, and enabling ad hoc efforts.

6. Whose job description includes it, and who has overall responsibility?

As you might guess, the answer here last year was very often “nobody.” Next year we’ll see a shift toward the guerilla social media people formalizing their roles and management recognizing the need for coordination and leadership. And yes, this question can set off turf wars. Tread lightly.

7. Have you defined a corporate policy for engaging with customers through social media?

If not, better get on it. Talking early to legal / brand / compliance, especially in regulated industries, always saves frustration later.

8. In what third-party venues do you have a presence?

This always yields surprises. “None…. Well, oh yeah, I guess we do have the Facebook thingie. And someone in marketing has been posting our ads to YouTube.” Or: “Marketing is in charge of our Twitter accounts. Except for the ones they use in customer service. And Dale down in R&D is a total Twitter fanatic.”

9. How well are those efforts coordinated?

Yes, more question-begging. Most often, efforts across social networks, blogs, and media sharing sites are not coordinated. Maybe, just maybe, they should be.

10. What is your brand’s online personality?

This one is a great conversation starter. It’s really about understanding how to show up in social media (hint: not with offers, and not with campaign messages). This topic is really about starting to think about how the people representing the brand should show up in social settings—authentically, as people, but as people not only representing but also enacting the brand and its character. I like to use the example of our client NAU. They make sustainably-developed clothing, and they blog not about their clothing products but about sustainability, outdoor recreation, and social action—the passions that are at the emotional core of their brand. A while back they posted, for example, a video of people moving an entire Portland, OR household by bicycle. Awesome. You want to subscribe, to follow, to befriend them.

11. How consistently do your social media efforts embody the character of the brand?

This is really a question about governance. How organized are you? Do you have a system in place to manage customer interaction across touch points? Is the system in use?

12. Where do your customers spend time online? What content do they create?

Market research typically tells us a lot about where customers spend time online. What it typically doesn’t tell us is very much about what they’re doing—So 40% of your customers check Facebook daily. That’s good to know, but to really drive action, you need to understand whether they’re there socially, professionally, or both. Whether they’re using it to market their services, keep in touch with Granny (oh yes, Granny is definitely on there), or what. They’re on Twitter, good—but what are they talking about? Whom are they following?

13. What are their preferred information sources, and how do they consume them?

What’s the information ecosystem your customers tap? Who are the influencers? What do they read? Blogs, newspapers, Digg? Are they looking at web pages, RSS feeds? Are they reading on mobile? Are they sharing things they find? Which things? With whom?

14. Where are their relationships?

Whom do your customers interact with online? Through what channels—IM, email, blog post commentary, Flickr photostreams? On social networks? Twitter? Do they use different channels for different kinds of relationships? Which ones, and what kinds?

15. What are you doing to enable customer participation on your own properties?

Do you have an email contact form buried in your footer? Or a p2p support forum? Corporate blogs? Can customers comment? Review? Rate? Can they interact with each other? Create content and add it? Suggest or vet ideas? Do they have a stake in your next version? What value can they create for each other, and how can you enable it?

16. How does your organization interact with customers online?

Can your customers contact you? How? Simply being reachable is a great first step. The next step is to proactively engage customers who need support, to reach out to your customers for feedback and ideas, and to create opportunities for customer collective intelligence to create business intelligence.

17. How do you capture business intelligence from those conversations?

Social media listening has a major difference from behavioral web analytics: It’s a two-way conversation, and it’s not just about what people do. It’s also about what they say, and how they feel.

18. What is the process for making your business intelligence actionable?

Intelligence is useless without action. But the challenges in actionablizing (ha!) business intelligence are often really substantial. How do you get the right bits and pieces to the people who can take action? This question is really about escalation, delegation, roles and responsibilities, and workflow. To make the most of what you know, you need definition around how you’re going to do something about it, who’s responsible, and how success gets measured and reported.

19. Have you monetized the value of your social media efforts?

I’ll be honest. The answer here is always no. Social media ROI is one thing, and monetized estimates of the impact of social media activities are another. ROI is great, and showing ROI in social media is absolutely possible to do. The problem is that a large portion of the payoff in social media happens over the long term and is measured in, for example, lifetime customer value and word of mouth—neither of which show up on your quarterly balance sheets.

20. Estimated the financial impact on lifetime customer value or word of mouth?

We do have a very advanced approach to this, but it’s a subject for another post. Essentially the idea is to be really smart about some monetized estimates of the value of certain measurable activities, then validate and refine those estimates over time.

 

Naturally, we don’t typically get these questions answered by sitting down with the marketing people for an hour and just asking. We basically never ask these questions in these words. A huge part of the assessment is getting time in conversation with the right people in the first place, and talking with them about their jobs, their goals, satisfactions, and frustrations. We use a combination of interviewing approaches including contextual inquiry and appreciative inquiry, and a fair amount of intuition and sneaking around. In other words, it’s not a mechanical process.

But I hope you (you few who’ve read all the way to the bottom of this post) find this list useful, and I’d really love to hear your thoughts about it. Anything missing? Anything off the mark?

Too Much Marketing

By Ryan Turner | Nov 6, 2009 4:37:11 PM

(Cross-posted from Web Social Architecture)

I really like David Armano’s Conversation Starter post from earlier this week with predicted trends in social media for next year. In particular, his first point hits on exactly what I think is going to be the key to success for marketers in social media:

1. Social media begins to look less social
With groups, lists and niche networks becoming more popular, networks could begin to feel more "exclusive." Not everyone can fit on someone's newly created Twitter list and as networks begin to fill with noise, it's likely that user behavior such as "hiding" the hyperactive updaters that appear in your Facebook news feed may become more common. Perhaps it's not actually less social, but it might seem that way as we all come to terms with getting value out of our networks — while filtering out the clutter.

OK, I actually think the header on Armano’s paragraph is, as he hints, actually the opposite of the truth—the way I’d say it is that “social media begins to look less anti-social!”

But the real point Armano is making is spot on. Social channels are too noisy, and as more and more marketers start to proactively reach out to consumers online next year, and as listening technologies are more widely adopted, we’ll see a dramatic increase in, well, spam.

I made a joke on Twitter recently about going to back to grad school. I was, I think, really obviously joking. But in minutes, I had four @replies from universities plugging their Master’s programs. Yikes!

We are fast approaching a time when public online social venues are saturated with listening marketers, responding to every mention of everything related to their brands, products, and services. It’s easy to anticipate Twitter’s @ channel, for example, becoming essentially useless.

Armano is right to predict an increase in user behaviors that enable them to filter the noise. But I also think we’ll see a rise in demand for and sophistication of privacy controls (and even products—like the “social router” I once pitched to Nancy White after her talk here at ZAAZ), as well as an increase in the importance of private networks and communities.

What’s a marketer to do?

Create value, that’s what. Next year will be a year for opt-in marketing—the focus will be on creating content and services people want to use. Heads up, folks, people don’t want you to “engage them” in “conversation.”

Social Media Teams: Different on the Agency Side than Internal

By Ryan Turner | Aug 10, 2009 7:27:51 PM

[Cross-posted from Web Social Architecture.]

One of the things I've been saying a lot lately is that over the next couple years, we can expect to see corporations adding dedicated internal social media teams. Does this sound like a statement of the obvious? Then maybe the corollary is more interesting: I think agencies should do the opposite.

I've seen a similar pattern in a dozen or more corporations in the past year: Responsibility for social media falls to the person or people who get interested and raise their hands. They come from PR, customer service, marketing--and sometimes just out of the woodwork. Good!

In other words, responsibility for social media tends to happen in an ad hoc way--and I actually suspect ad hoc is a perfect way to staff an emerging discipline. And staff it you must.

Social media requires a new combination of skill sets, traversing all the disciplines I just mentioned, but also including strong writing skills, a thick skin, and the social savvy to interact with customers through online media in ways that are on-brand, authentic, human, and bounded by corporate guidelines, policy, and politics. It's hard, but for the right people it's super fun.

For the corporate social media team, representing the brand is a full-time job, with its own discrete challenges and rewards. They need to focus on developing the skills and experience to do it well.

But on the agency side, the whole thing is different. Agencies need to bring to bear their full range of capabilities to support clients' social media efforts, and that includes all the "traditional" digital disciplines--at the agency where I work, we have a 10-person social media team that includes people from development, analytics, search, optimization, user experience, creative, and client services. It's a witchy brew. Our goal is to implode the whole idea of "social media" in the next 2 years.

So my message to agencies is: Adapt, or specialize yourself into an oblivion-vortex!

To be sure, social media specialty agencies (I have friends at several of them) provide a tremendously valuable service with their depth of expertise in social media. But my prediction is, as web marketing evolves further, the breath of expertise brought to bear by "traditional digital" agencies will pose a grave threat to the specialists. They simply won't match digital agencies' capabilities in development, analytics, creative, usability, planning, and optimization.

Seven years ago, we were explaining the idea of online community. Three years ago, we were selling the importance of Web 2.0. Today, we're answering ubiquitous demand from clients that "social media" be included as a component in all our web work. Give it another 2 years and clients will see "web" and "social" as synonymous--data, content, service, identity, content objects, and relationships as integral to a holistic web strategy.

So on the agency side, there should ultimately be no such thing as a "Social Media Team," only a company made of web-savvy, passion-driven professionals who can support all aspects of corporate social media efforts--from concept through implementation.

The nuts and bolts, and rubber meeting the road, and the delivery of the service belong on the client side; and the vision, concept, and creative / conceptual infrastructure are where agencies can help. All in all, I have to admit, it ends up looking a lot like Mad Men, which my 90 year old Grandmother, true story, described as "the most realistic show yet about the '50's."

"It shows," she says, "exactly how we lived."

Site Optimization: Lipstick on a Pig?, Or, Bacon and the Theory of Local Maximum

By Ryan Turner | Dec 5, 2008 12:34:33 PM

Cross-posted from Web Social Architecture.

 

Jason Carmel is a colleague of mine I learn a ton from. His expertise is in web site optimization--running experiments where he tests versions of web pages against each other to see which performs best. (Not to be confused with search engine optimization, improving a site's visibillity in search engines.)

Jason is a fairly unflappable guy. Nonetheless I recently started making an effort to get his goat. He gives me just enough encouragement that I keep going. The gist of my teasing is that optimization is nothing but a mechanical exercise to determine whether a red button works better than a blue button. "Glad to hear that red button worked out better by 2.84 percent Jason. The sum of your creative energy has produced yet another quarter million in revenue. You must really love your life, man. Hey, have you thought of trying one of those animated GIFs instead of a regular button?"

Fortunately, Jason is twice as nice as me, as well as twice as smart. He takes my ribbing well--and responds thoughtfully to the serious question underlying my teasing: We know optimization can move big numbers in terms of revenue, but can it do more than simply tweak pages to bump up conversion? Can it vet creative concepts? Can it maximize the creation of mutual value between businesses and customers? Can it help create more engaging experiences?

(image credit, found via Lee)

The short answer here, according to Jason, is that it depends, partly on what you're trying to achieve. If all you're focused on is moving business value measures, you're probably putting lipstick on a pig. But testing against value creation has the potential to uncover game-changing opportunities.

Here's an email exchange between Jason and me, in which he explains in a little more detail:

 

RYAN:

That whole web site optimization thing—isn’t it really just putting lipstick on a pig?

 

JASON:

I think “I hate you so much” might be a succinct way of responding, but I'll provide a little more detail:

Web Site Optimization is exactly like putting lipstick on a pig, but only if you start out with a pig. And if you are starting out with a pig, your opportunities for improving things are limited, and you’d be using the wrong tool to fix the problem. We are talking here about the concept of a “local maximum” which is a fancy, math term applied to mean “the best something can be within a limited dynamic.” Consider the aforementioned pig’s ability to fly, which, metaphorically speaking, is not particularly developed. We could take a pig and genetically modify it to be more aerodynamic. We could investigate building pig hang-gliders and attempt to train the smartest pigs to use them. But even in the best case, with the most aerodynamic pig, benefiting from the best training, and using the best pig flying technology, it will never fly as well as a bird. The best case flying scenario for a pig (the pig’s local maximum as far as flight is concerned) is nowhere near as effective as a bird’s. In that scenario, you’d be better off exchanging the pig for a bird at the start, rather than waste any time or effort teaching a pig to fly better.

Applied to the web: if a site sucks so much- if the goals and purpose are unclear, if the information architecture looks like my desk (at the moment), if the navigation is counterintuitive and the messaging has absolutely no intersect with the audience, then no amount of optimization in the world will make it right. The local maximum of that crappy site is too low for any optimization to matter. Or (even worse) you’d need the infinite number of monkeys to stop typing Shakespeare and to start applying experiments to your site to get the right combination where testing would make a real difference. Neither is very efficient. If your site is the pixilated equivalent of a pig, you need much more elemental help from a user experience expert first (know any?). Until you fix the fatal flaw(s) in a site, anything else you do will be throwing good money after bad.

Site Side Optimization works well in circumstances where the local maximum is high, but for some reason, the site is not achieving it. This can be due to single points of failure on the site, like a specific conversion path or page underperforming, or because the audience needs to be targeted more specifically, or because the existing content is stale/irrelevant. In each of these cases, experimental testing can make a huge difference. Optimization also works exceptionally well (and this is far more interesting to me) when applied as a method of trying out a new (and potentially risky) idea that could radically change and significantly improve an experience.  In both of these examples, the basic site is healthy, and the optimization program serves as a tool to reach its fullest potential.

 

RYAN:

But what I keep looking for is the way to test birds against pigs, not in the sense of which flies better, because as a user experience expert I do have the capability to predict the winner of that contest—but when I don’t have a clear sense of the best conceptual solution. For example, maybe I just can’t decide between eggs and bacon. Can optimization help design a better breakfast, or only decide between pulp and no-pulp in the OJ?

 

JASON:

Optimization can test more conceptual ideas, but it will be really hard to unpack the WHY after we determine which one wins. Most sites aren’t deciding between bacon and eggs, but rather between the bacon, eggs and hashbrowns with coffee or the granola, fruit and yogurt with yerba matte. If the former wins in a test, I don’t know whether it’s because of the bacon (which will usually win over everything) or the coffee, or because the person deciding had granola for the past three days, and would have taken ANYTHING other than more granola.

The other trick about testing high concepts in a website format is that you would have to build each solution to test them, which is usually more expensive than testing out wireframes or front-end prototypes in front of a more controlled audience.

 

RYAN:

First, you seem to be suggesting that a test win for the bacon breakfast might not imply extensibility for bacon breakfasts in general: That because, lacking control, the results might be idiosyncratic, they might not therefore apply broadly. Next week you might get a different result. My question is, why does that matter? And why does “knowing why” the bacon breakfast worked matter, as long as you know it worked.

 

JASON:

It is definitely a question I get from clients a lot. Why do I care about the individual elements of a variant- if the variant as a whole makes us more money, let’s just launch it and move on. I can’t fault the sentiment, but knowing why the bacon worked could lead to better tests, more focused messaging and (even more) cash money. I want to know that it’s the bacon by itself that is the motivating factor outside of all the other influences. Let’s say that we ran a test breakfast against a bowl of Total cereal and we tested bacon with powdered eggs as the experimental variant. Now let’s assume that bacon and powdered eggs lost to the control by 1%. You could take the position that we would do better to serve Total because we want to avoid losing that 1%, and you would be right. But what if you knew that the bacon by itself actually improved the breakfast by 15%, but the powdered eggs were so crappy that they hurt the breakfast by 16%, so you netted the 1% loss? If I controlled for all the variables in my breakfast, that knowledge would a) help me make a better breakfast overall (just serve bacon), and b) will also prevent me from throwing away a positive variable simply because it was paired with a really negative one.

I imagine you run into that a lot with prototyping and wonder how you deal with it in the UX world.  If a subject totally fails at a task, are you ever afraid of overcorrecting a prototype to account for it? Do you ever throw the baby out with the bathwater? How do you control for that?

 

RYAN:

A great question. One of the answers is that in usability testing, you're looking for usability problems. So as long as your test participants are representative of your users as a whole, major failures are, practically speaking, never anomalous. If your user population is one million, and one of the eight people participating in your test has trouble understanding some aspect of the interface, what are the odds they're the only person who's going to have that problem?

The other thing I wonder about is what happens when what you’re trying to accomplish is harder to measure than conversion (e.g. brand lift) or if you want to measure it over time (e.g. engagement). Especially in social media, it’s quality that matters, not quantity. You want to know how valuable your user-generated videos are more than you want to know how many of them you have. Can web site optimization help you get to answers?

 

JASON:

Ah, you and your social media. When are you going to come to terms with the fact that this whole thing is a fad? The future is in email, Ryan, and lots of ‘em. Mark my words.

Absent a more qualitative tie-in with optimization (surveys, satisfaction scores, etc.) you will be hard-pressed to get good data about branding or the impact of social media. But I’m not saying you shouldn’t optimize for branding or social media. I’m saying you need to get that qualitative kicker. I’ve done a few branding tests, and I think they provide some interesting feedback. But I’ve never optimized where a KPI has to be judged on quality (e.g., good comments vs. troll comments) or off the site entirely (e.g., buzz in the blogosphere). Sounds fun.

Site Optimization: Lipstick on a Pig?, Or, Bacon and the Theory of Local Maximum

By Ryan Turner | Dec 1, 2008 8:01:09 AM

(Cross-Posted from Web Social Architecture.)

Jason Carmel is a colleague of mine I learn a ton from. His expertise is in web site optimization--running experiments where he tests versions of web pages against each other to see which performs best. (Not to be confused with search engine optimization, improving a site's visibility in search engines.)

Jason is a fairly unflappable guy. Nonetheless I recently started making an effort to get his goat. He gives me just enough encouragement that I keep going. The gist of my teasing is that optimization is nothing but a mechanical exercise to determine whether a red button works better than a blue button. "Glad to hear that red button worked out better by 2.84 percent Jason. The sum of your creative energy has produced yet another quarter million in revenue. You must really love your life, man. Hey, have you thought of trying one of those animated GIFs instead of a regular button?"

Fortunately, Jason is twice as nice as me, as well as twice as smart. He takes my ribbing well--and responds thoughtfully to the serious question underlying my teasing: We know optimization can move big numbers in terms of revenue, but can it do more than simply tweak pages to bump up conversion? Can it vet creative concepts? Can it maximize the creation of mutual value between businesses and customers? Can it help create more engaging experiences?

(image credit, found via Lee)

The short answer here, according to Jason, is that it depends, partly on what you're trying to achieve. If all you're focused on is moving business value measures, you're probably putting lipstick on a pig. But testing against value creation has the potential to uncover game-changing opportunities.

Here's an email exchange between Jason and me, in which he explains in a little more detail:

 

RYAN:

That whole web site optimization thing—isn’t it really just putting lipstick on a pig?

 

JASON:

I think “I hate you so much” might be a succinct way of responding, but I'll provide a little more detail:

Web Site Optimization is exactly like putting lipstick on a pig, but only if you start out with a pig. And if you are starting out with a pig, your opportunities for improving things are limited, and you’d be using the wrong tool to fix the problem. We are talking here about the concept of a “local maximum” which is a fancy, math term applied to mean “the best something can be within a limited dynamic.” Consider the aforementioned pig’s ability to fly, which, metaphorically speaking, is not particularly developed. We could take a pig and genetically modify it to be more aerodynamic. We could investigate building pig hang-gliders and attempt to train the smartest pigs to use them. But even in the best case, with the most aerodynamic pig, benefiting from the best training, and using the best pig flying technology, it will never fly as well as a bird. The best case flying scenario for a pig (the pig’s local maximum as far as flight is concerned) is nowhere near as effective as a bird’s. In that scenario, you’d be better off exchanging the pig for a bird at the start, rather than waste any time or effort teaching a pig to fly better.

Applied to the web: if a site sucks so much- if the goals and purpose are unclear, if the information architecture looks like my desk (at the moment), if the navigation is counterintuitive and the messaging has absolutely no intersect with the audience, then no amount of optimization in the world will make it right. The local maximum of that crappy site is too low for any optimization to matter. Or (even worse) you’d need the infinite number of monkeys to stop typing Shakespeare and to start applying experiments to your site to get the right combination where testing would make a real difference. Neither is very efficient. If your site is the pixilated equivalent of a pig, you need much more elemental help from a user experience expert first (know any?). Until you fix the fatal flaw(s) in a site, anything else you do will be throwing good money after bad.

Site Side Optimization works well in circumstances where the local maximum is high, but for some reason, the site is not achieving it. This can be due to single points of failure on the site, like a specific conversion path or page underperforming, or because the audience needs to be targeted more specifically, or because the existing content is stale/irrelevant. In each of these cases, experimental testing can make a huge difference. Optimization also works exceptionally well (and this is far more interesting to me) when applied as a method of trying out a new (and potentially risky) idea that could radically change and significantly improve an experience.  In both of these examples, the basic site is healthy, and the optimization program serves as a tool to reach its fullest potential.

 

RYAN:

But what I keep looking for is the way to test birds against pigs, not in the sense of which flies better, because as a user experience expert I do have the capability to predict the winner of that contest—but when I don’t have a clear sense of the best conceptual solution. For example, maybe I just can’t decide between eggs and bacon. Can optimization help design a better breakfast, or only decide between pulp and no-pulp in the OJ?

 

JASON:

Optimization can test more conceptual ideas, but it will be really hard to unpack the WHY after we determine which one wins. Most sites aren’t deciding between bacon and eggs, but rather between the bacon, eggs and hashbrowns with coffee or the granola, fruit and yogurt with yerba matte. If the former wins in a test, I don’t know whether it’s because of the bacon (which will usually win over everything) or the coffee, or because the person deciding had granola for the past three days, and would have taken ANYTHING other than more granola.

The other trick about testing high concepts in a website format is that you would have to build each solution to test them, which is usually more expensive than testing out wireframes or front-end prototypes in front of a more controlled audience.

 

RYAN:

First, you seem to be suggesting that a test win for the bacon breakfast might not imply extensibility for bacon breakfasts in general: That because, lacking control, the results might be idiosyncratic, they might not therefore apply broadly. Next week you might get a different result. My question is, why does that matter? And why does “knowing why” the bacon breakfast worked matter, as long as you know it worked.

 

JASON:

It is definitely a question I get from clients a lot. Why do I care about the individual elements of a variant- if the variant as a whole makes us more money, let’s just launch it and move on. I can’t fault the sentiment, but knowing why the bacon worked could lead to better tests, more focused messaging and (even more) cash money. I want to know that it’s the bacon by itself that is the motivating factor outside of all the other influences. Let’s say that we ran a test breakfast against a bowl of Total cereal and we tested bacon with powdered eggs as the experimental variant. Now let’s assume that bacon and powdered eggs lost to the control by 1%. You could take the position that we would do better to serve Total because we want to avoid losing that 1%, and you would be right. But what if you knew that the bacon by itself actually improved the breakfast by 15%, but the powdered eggs were so crappy that they hurt the breakfast by 16%, so you netted the 1% loss? If I controlled for all the variables in my breakfast, that knowledge would a) help me make a better breakfast overall (just serve bacon), and b) will also prevent me from throwing away a positive variable simply because it was paired with a really negative one.

I imagine you run into that a lot with prototyping and wonder how you deal with it in the UX world.  If a subject totally fails at a task, are you ever afraid of overcorrecting a prototype to account for it? Do you ever throw the baby out with the bathwater? How do you control for that?

 

RYAN:

A great question. One of the answers is that in usability testing, you're looking for usability problems. So as long as your test participants are representative of your users as a whole, major failures are, practically speaking, never anomalous. If your user population is one million, and one of the eight people participating in your test has trouble understanding some aspect of the interface, what are the odds they're the only person who's going to have that problem?

The other thing I wonder about is what happens when what you’re trying to accomplish is harder to measure than conversion (e.g. brand lift) or if you want to measure it over time (e.g. engagement). Especially in social media, it’s quality that matters, not quantity. You want to know how valuable your user-generated videos are more than you want to know how many of them you have. Can web site optimization help you get to answers?

 

JASON:

Ah, you and your social media. When are you going to come to terms with the fact that this whole thing is a fad? The future is in email, Ryan, and lots of ‘em. Mark my words.

Absent a more qualitative tie-in with optimization (surveys, satisfaction scores, etc.) you will be hard-pressed to get good data about branding or the impact of social media. But I’m not saying you shouldn’t optimize for branding or social media. I’m saying you need to get that qualitative kicker. I’ve done a few branding tests, and I think they provide some interesting feedback. But I’ve never optimized where a KPI has to be judged on quality (e.g., good comments vs. troll comments) or off the site entirely (e.g., buzz in the blogosphere). Sounds fun.

Twitter: Trivial. Beautifully So.

By Ryan Turner | Nov 11, 2008 11:18:37 AM

I wrote a post a year and a half ago on Twitter in which I identified two types of use for it. This was at the height of the initial buzz about Twitter, and while some folks were really excited to finally see an HTML / SMS / IM social application, others worried aloud that Twitter, for the same reasons it offers such a low threshold of entry, would also tend to erode the quality of social relationships online. The thoughtful anti-Twitter point of view was that it encourages triviality, and the gist of my post was that while that may be true, there are times when trivialities are useful.

Since I wrote that post, I've become an avid Twitterer, and I have to say my perspective on Twitter, and indeed on triviality, has changed. I now see triviality as maybe the critical element of truly meaningful relationships, online and off, and Twitter has accordingly become one of the cornerstone services of my personal and professional social lives.

That sounds crazy, I know, and while it may be true that I am prone to crazy-sounding declaratives, I'm actually not kidding about this. I now use Twitter at work very frequently, mostly within my team, and it has improved our functioning and, dare I say it, made us closer, more personally connected.

Let me float an assertion: The deeper the relationship, the greater the proportion of it dedicated to triviality; and beyond, say, 90% triviality, the relationship isn't a relationship at all. And likewise with meaningful interactions--if everything is meaningful, it's not a relationship, it's therapy.

So here is my Bullseye Diagram of Love, illustrating the way I'm starting to envision online social systems supporting healthy relationships:

So I'm arguing for Twitter, or an analogous triviality service (ha!) as a supplement to existing relationships, not as a full-fledged social channel in and of itself. As a social network, I actually do think it's useless, or worse. And you can take that as a caveat.

But I am saying I think triviality in general and as supported by Twitter, in both personal and professional settings, is indispensable--that we can and should deliberately design it into social systems.

The Carl Lewis Rules of Marketing

By Leslie LaRue | Aug 27, 2008 3:23:32 PM

Remember Carl Lewis? The Olympic hero of the 80’s and 90’s? He won a lot of medals, set some world records and was in general, the type of athlete you marvel at. He had the potential to endorse multiple household brands and be a sports commenter for as long as Bob Costas. Until Michael Phelps came along, outside of Mary Lou Retton and Bruce Jenner, there were very few other Olympic athletes who were as widely recognized.

But, somewhere along the way, Carl Lewis decided, “I’d like to have a music career.”

Bad idea.

http://www.youtube.com/watch?v=jamJ4-C_TME

This video is quite likely, the most horrible thing I’ve ever seen. My perception of Carl is permanently damaged. It makes me never want to watch him give commentary on a sporting event ever again. Let alone, sing, dance, lift weights or blow bubbles. (?)

Maybe it’s the black unitard, the special effects, or just the song itself. But if there was ever one reason why he never became a household name this video might it. Part of the excitement in the world of marketing is trying new things. But we still have to remember, “Is this a good long-term decision?”

If you’re Carl Lewis, you’re probably asking yourself that very question. Because, whether you like it or not, when you or your company attains a certain level of awareness with the general public, you have to make some decisions on how to market yourself. I'm pretty sure that doesn't nvolve an old lady in yellow sunglasses chasing you around a gym.

So, here are some general rules-of-thumb for anyone trying to make that decision.

(Phelps, are you paying attention here? I don’t want to see anymore of these videos surface.)

1.       Be consistent. – Trying new things is good. But that crazy thing you cooked up late Friday night should support your brand, not define it.

2.       Pick things you’re good at. It’s OK to just be an amazing athlete. You don’t have to sing too. We don’t mind. It’s tempting to constantly widen your services, your product offerings, (your music career) but at what cost? The reason people were inspired by Carl Lewis in the first place was because he is a great athlete. Not a singer/dancer, bubble-blower.  I’m not suggesting we shouldn’t push ourselves to succeed in new ways. But sometimes, use what you’ve got.

3.       Be pro-active, not reactive. It’s very easy to get so busy, that we end up working within the realms of who is contacting us.  To sponsor an event, advertise in their publication, market their product, etc. But does it make sense for your own long-term vision and goal? Spend the extra time doing research on ideas or concepts that fit who you are, not who are filling up your In-Box. Maybe Carl got a great offer from a record company to make this video. But does that mean he should have done it?

4.       Follow through - I recently visited the official Carl Lewis website. He has a blog. He has not posted a blog since January of 2007 and it was a test to see if it was working. If you’re going to do something, do it. Otherwise your audience stops paying attention.

5.       Remember your roots. Success makes us feel good. But it shouldn’t cloud our decision-making. In other words, don’t get cocky and think that now that you’ve established your brand, you can go off the deep-end.  Carl Lewis actually had two endorsement deals, with Coke and Nike. . But he turned Coke down, confident he’d be worth more after the 1984 Olympic Games. Coke cancelled the offer. Nike had Lewis under contract for a few years, despite questions about how it affected his status as an ameteur. He was appearing on Nike television ads, in print and on billboards. After the Games and faced with Lewis’ new negative image, Nike dropped him.

(I think Nike saw the video. Truly, a lesson for all of us.)

My summer prediction: Microsoft will buy Yahoo by year end

By Nigel Morgan | Aug 19, 2008 2:09:24 PM

So it has been a long summer so far and I hope there are many more days of sunshine to go. It has had its ups, just look at Michael Phelps, and it has had its downs as we bid farewell to Isaac and Bernie. Of course for us online marketers there has been an ongoing story that has had both the ups and the downs. I am talking of course about the on again, off again, on again but now off again saga of the Microsoft takeover of Yahoo.

So while Jerry Yang is in China at the Olympics and the Microsoft upper management is enjoying a fine Pacific Northwest summer, I will throw in my opinions on where this story will end.

To me it is quite clear that this merger will happen; there are a number of reasons why this deal will go through for Microsoft and Yahoo. I list them here and ask you all to join the debate!

Market share of total online ad revenues.

This is the number one reason why this takeover will happen. Online advertising revenues are still growing very fast and will continue to do so according to every survey and expert opinion you read.

Today Google is firmly in the box seat owing to its market position as undisputed search engine leader and ownership of Doubleclick. Once Google fully optimizes it’s ability to understand the vast majority of internet behaviors through its search engine and Doubleclick tags, it will be able to target marketing messaging to daunting effect (read attain massive revenues from online advertisers). Although MSFT owns Atlas, the combination of Atlas and MSN will, most likely, never get MSFT to a market leadership position. If Microsoft doesn’t gain a huge increase in market share and soon, it will be virtually impossible for them to catch Google. How do you suddenly increase market share? Answer, buy a competitor.

“Owning” the vision of the PC is Microsoft’s heritage

Microsoft has continually innovated across the years and while they have some very successful products such as X-Box, Windows and Office to name a few, they have not truly come on from their computer on every desktop vision of 20 years ago.

When a computer was used mostly for work, Microsoft set the vision for the personal computer with its operating systems and productivity software. These products are still by far the biggest driver of the company’s profits. However thanks to internet applications PCs now are equally used for recreation and Microsoft has never managed to dominate the market in this reality. Microsoft’s online presence that gave them early market advantage in the form of Internet Explorer and MSN are becoming more strategically irrelevant every day and their market share continues to follow a long-slow downward trajectory. Efforts such as Silver light, .NET and other initiatives will not change the fortunes of Microsoft in this regards.

The people at Microsoft need to innovate

While Microsoft could happily continue to dominate the OS market, Bill Gates will not want to see his life’s work become a monopolistic commodity. Microsoft also has enormously talented employees and they will not go down this route without kicking and screaming.

It makes good old fashion financial sense.

Although MSN and Yahoo users will not like me saying it, they really are substitute products for each other. Taking best practices from each and merging them together will improve the product and dramatically reduce redundancy. It will also create a broad base of messenger and email users that will dwarf the number of users on gmail. This is one beachhead that a combined Yahoo/MSFT should exploit in their ability to compete with Google.

At the end of the day, it may be the only path ahead for Yahoo too

Unless Yahoo does things very differently, they will continue to lose market share in search queries. This will ultimately result in long-term financial disaster as their editorial teams and messenger and email products will be left to support site visits and therefore online ad revenues. Yahoo’s content is not differentiated enough from other content sources to survive in the long run.

So, a takeover makes a lot of sense, ironically more so for the one that is blocking it. Microsoft can have a future without Yahoo but I struggle to see where Yahoo fits into the online ecosystem long term. Therefore I predict that Yahoo will come to this conclusion, Microsoft will have another attempt at a takeover and this time it will go through.

So there you have my opinions. I’m off to see my stockbroker!