Your SEM Bid Management Solution: it’s a ‘Turk’

By Rich Devine | May 31, 2011 4:41:58 PM

 

Image via Wikipedia

In 1770, Johann Wolfgang Ritter von Kempelen invented an automaton chess-playing machine affectionately known in modern times as "The Turk". For more than 80 years the 'machine' toured Europe and the Americas, defeating most of its opponents -- including Napoleon and Benjamin Franklin.

In 1820, Surprise! As if anyone didn't know, the chess-playing Turk was exposed as a hoax, a parlor trick, powered by humans stuffed inside the machine -- not automated by the machine itself.

Bid Management software is a 'Turk'. It's not that Bid Management isn't necessary, or helpful, or effective -- but if an agency is trying to sell you based on how awesome their 'proprietary' bid management software is, or how they've cracked some special code and have super-top-secret algorithms that make their bid management software smarter than Ken Jennings, take a pass.

Bid Management is a Turk, only as effective as the human being who is inside the box pulling the levers.

Don't misunderstand me. I am a big believer in bid management software. I recommend it for most of our clients, and there are some really, really good solutions in the marketplace -- some of them almost as smart as Ken Jennings.

Bid management is attractive for two reasons:

First it has the capability of processing information really fast, smartly predicting bids, and executing real-time optimization decisions that can result in added efficacy for your SEM campaigns.

Second, it offers tremendous efficiency. Multiple publishers can be managed from one single platform. Thousands of keywords, budgets, and bid decisions can be automated based on configured rules and parameters.

Clients get excited by the prospect of added value to their business and increased ROI from their ad dollar. Ad agencies get excited because they can provide services at some fraction of the human resourcing cost that SEM would otherwise demand.

The problem is that bid management is really just a Turk. In order for Wolfgang's chess robot to beat most of it's opponents like a robot should, he had to find world class chess players to sit inside the machine. So it is with bid-management software: while useful and even superior to humans for many automated functions, unless it's powered by a skilled human operator -- it just doesn't work very well.

Sometimes (actually lots of times) agencies get a little over-enthused by the prospect of profit margin and they skimp on the quality of human resourcing necessary to make it work. They buy into the myth of the Turk.

So when you're ready to hire an agency here are some things to look for:

1. They should have a bid-management solution. If they don't, I'd move on. At ZAAZ, I've had my team build familiarity with multiple tools -- some of which are better solutions for some clients than others. I really believe that there's an 80/20 rule here: without bid management you spend 80% of your time doing admin functions for paid search, and only 20% of your time doing strategy. With bid management, it can be the reverse. But the operative word is 'can'.

2. They should talk about their people. They should spend more time talking about the value and intelligence of their people than the magical value of the machine. If it's all about automation, algorithms, and predictive intelligence -- I'd take a pass. I can't tell you how often I'm in a new pitch, and the incumbent agency has deployed a great bid management solution; but they've thrown in a low-cost intern or a project manager with no time to spare and no real search experience. It just doesn't matter how awesome that machine is, without a good human to operate it, the venture always turns into a big mess.

I have a basic rule that surprisingly isn't always obeyed by others: I don't hire people to play chess unless they know how to play chess. Yes it costs me more to hire good people, but I have significantly lower churn on my clients, and my clients are happy.

3. Pricing should be transparent. If you can't tell what you're paying for between human contribution and software contribution -- take a pass. Bid Management software is usually priced at a percent-of-spend fee. At ZAAZ I've elected to take whatever volume discount we earn as an agency from bid management partners and I pass-through the discounted fee directly to my clients. Then I'm extremely transparent about what kind and how much human effort is required to make that Turk work.

 

this blog post originally appeared at RichDevine.me

 

Monetize This

By Lindsay Hasz | Aug 25, 2009 3:05:41 PM

Building a monetization model is like traveling to a foreign country.  It starts out confusing, not knowing where to begin or even how to speak the language, but after some time, you broaden your horizon and learn a few things.  You start to wonder, ‘’Why isn’t everyone doing this?’’

Monetization is the method in which you convert a site’s activity into a monetized amount.  This can range from a button clicked to an application completed.  This is essential to one’s business because it helps you put a dollar value on each visitor’s movement.  By combing analytics, one can determine not only which area gets the most activity, but what’s the most valuable (and those are often not the same!)  For example, you may know that the billboard on the homepage gets the most clicks, but you may not know that by promoting one call-to-action over another in that space could double your monetized value.  Building a monetization model is a key example of how ZAAZ “meets at the intersection of creative and logic.”   

There are numerous benefits to building a monetization model.   

• Optimization analysis and reporting:  quick valuation on optimization requests, discover which key buying activities are worth more for testing purposes, etc.

• Gather important key findings such as value per visit/ value per application

• Time series data: analyze seasonality, utilize predictability of the model

• Dynamic prioritization: determine which initiatives have highest ROI potential

• Match-back to offline sales regression (full circle sales cycle)

• Correlate data to survey analysis with regards to brand opinion, etc.

• Monetize loss avoidance within optimization field

When building a monetization model, you must start with a few major questions.

1. Who are the stakeholders?  Is it the product team or marketing team?  They’ll probably want the model for completely different reasons…
2. What are the identified monetized transactions?  Is this readily accessible?  For example, can we get contribution margin per product?  Do we have the data to support separating visitors into different segments?
   
3. How will we use the results?  How granular will we need the data to be? 
4. What are the assumptions?  Does everyone agree?  How do we determine what’s a good benchmark for these areas of unknown?

Next, discuss these things with your analytics team.  How will each of these things be measured?  What needs to be tagged (method in which you track metrics)?  Can we pull historical data?  Begin to put together a data dictionary (an established set of rules to ensure conformance in the manner which data is gathered) and start to gather whatever metrics you can.

Common hurdles:

• Difficulty agreeing on strategy of model (gaining buy-in on a model that does not reflect actual revenue, but still carries tremendous value)

• Too much data… How do you decide what actually goes into the model?  How do you be sure you’re using deduped data so as not to double-count any activity/ revenue?

• Agreeing on the assumptions

Even with the challenges listed above, building a monetization model to represent your site is essential to the growth of one’s business.  It is a direct way to measure how successful a marketing campaign was, how to predict next year’s seasonality, which call-to-action is best used on what page and so forth.  ZAAZ has built these for multiple business models and believe it is completely worth whatever hurdles you may come across, just as is it to travel somewhere new. 

Lindsay is an Online Test Designer in the Optimization department at ZAAZ. 

Analytics spawned yawning among analysts?...is that possible?

By Judith Pascual | Dec 22, 2008 2:50:00 PM

I never thought I'd see the day that I would yawn during an analytics discussion.  But it happened.  I always thought it was my job to motivate people, show them the value and they will move forward.  I get little butterflies as I pull and integrate numbers and find a story to share.  I often feel like the journalist, breaking news.  But this time, I felt like I had been transported to 2004 and I though that was a good year for me, I was not pleased.

The lack of analytics mobility is starting to get boring. Apparently, I am not alone.   After so many years, let's move on folks!  I keep hearing things like, 'analytics is our focus, we need to act'...okay so why do you shy away from tracking based on your goals, not just the 'data' you are able to get at this time?  Why when you are given insights you don't act on them.  Segmenting is a bad word and you still find geography 'views' valuable.  This is just all very lame.

Now, during this economic turmoil, more than ever we need to stop making excuses for why analytics funding is not a priority and why you cannot act on customer requests.  Don't get me wrong, we have worked with so many clients that have grown and are now data driven businesses. But far too many are stagnant.  Ensure that you are not on that list. 

For 2009, you have already asked yourself, what am I spending money on?  Now, ask yourself what are you spending time on?  What are key stakeholders focusing on?  How is that growing or even maintaining your business?  Look at your analytics maturity level and if you see yourself having the same discussion you had in 2004. Stop.  Start the roadmap on moving forward and monetizing your business so you can optimize. 

Of course I know you are still thinking about costs so...put together a cost benefit worksheet (yes, it takes time and you do need to understand what you are doing) and among the obvious ensure to include:

1. Speed - what customer driven projects can we quickly turnaround that is going to influence return?

2. Better Results - improvements in results because actions were taken - savings included -

3. Shorten the meetings and discussions on items you have action plans for or documented a roadmap...and use them already...you'll be surprised how much time and money you'll save.

You can adjust things as you go...but take the step.

Remember that yawning is contagious.  Don't put yourself in a position where your analyst' yawn, de-motivate your stakeholders and it all transcends to consumer behavior.

Primacy: When Good Redesigns Do Bad Things

By Jason Carmel | Dec 9, 2008 10:53:56 AM

One of my most favorite websites in the world, eMusic, just got a facelift. I love eMusic because they have a great selection of indie music, they are cheaper than most of the competition, and the music you get is legal and DRM free, as it should be. The reviews of the redesign have been generally positive: now eMusic sports some fancy AJAX-y interface bits, a wealth of third party content (youtube videos, reviews, etc), and a new music recommendation and personalization engine, among other things.

So why am I, a loyal eMusic customer, not happy with what are clearly improvements to the site?

The Primacy Factor

I've been a member of eMusic for almost three years now, and I visit the site like 27 times a day. I don't know why I visit the site so often. Maybe I'm afraid of getting in a rut, which at my age, can quickly lead to a well documented form of musical taste fossilization- the kind where I sit in a corner yelling at pesky kids to "turn down that noise," while I lament the crapification of music these days, and pine exclusively for Hair Metal, Grunge, Disco, or some other fixed aural remnant of a bygone and best-forgotten era.  Before you know it, I'm eating dinner at 4:30, I have a regular bingo night, and I can quote Matlock extensively.

When you visit a site so many times, you get comfortable using it. All of the rough user interfaces are smoothed over by the calm running waters of routine. Any glitches and inconveniences morph into idiosyncrasies and "power user" workarounds as you become increasingly efficient at dealing with the user experience. This is precisely what happened to me relative to eMusic- I adjusted how I browse and purchase music to conform to the old eMusic interface (warts and all) and did such a good job of changing my behavior over time, that this new, improved interface actually reduced my comfort level.

In the world of site-side optimization, we refer to this phenomenon as Primacy. If a customer routinely uses the same feature or traverses the same path on a site, then any change- even one that improves an experience- will negatively impact that customer (who now has to re-learn an entirely new method of doing things). Re-learning anything is annoying at first. As such, the short term effects of an altered experience could make a great improvement erroneously appear like it is underperforming relative to the old version, as these veterans struggle to adjust to the new (improved) world.

For eMusic, I've been biased by almost three years of the old site, and so I will need a lot of time to adjust before I become an advocate. Of course this is dangerous ground to tread for any site, since the people most at risk of defection following a significant change are the heaviest users.

How to address the Primacy factor

It is important for sites that experiment on areas with significant return traffic to not be discouraged by early results of those tests. As a test designer, your options for dealing with the effects of Primacy are to let the test run long enough for the veterans to get used to the new idea, or (even better) to segment the test by new and returning customer groups so you can see whether the design has a positive net impact for visitors who have not been biased by a previous experience. eMusic should definitely be segmenting here to reduce the risk of alienating hardcore users and testing to quantify the results of their new design.

I'm pretty sure the new design is better. It just might take me a while to get used to it.

Site Optimization: Lipstick on a Pig?, Or, Bacon and the Theory of Local Maximum

By Ryan Turner | Dec 5, 2008 12:34:33 PM

Cross-posted from Web Social Architecture.

 

Jason Carmel is a colleague of mine I learn a ton from. His expertise is in web site optimization--running experiments where he tests versions of web pages against each other to see which performs best. (Not to be confused with search engine optimization, improving a site's visibillity in search engines.)

Jason is a fairly unflappable guy. Nonetheless I recently started making an effort to get his goat. He gives me just enough encouragement that I keep going. The gist of my teasing is that optimization is nothing but a mechanical exercise to determine whether a red button works better than a blue button. "Glad to hear that red button worked out better by 2.84 percent Jason. The sum of your creative energy has produced yet another quarter million in revenue. You must really love your life, man. Hey, have you thought of trying one of those animated GIFs instead of a regular button?"

Fortunately, Jason is twice as nice as me, as well as twice as smart. He takes my ribbing well--and responds thoughtfully to the serious question underlying my teasing: We know optimization can move big numbers in terms of revenue, but can it do more than simply tweak pages to bump up conversion? Can it vet creative concepts? Can it maximize the creation of mutual value between businesses and customers? Can it help create more engaging experiences?

(image credit, found via Lee)

The short answer here, according to Jason, is that it depends, partly on what you're trying to achieve. If all you're focused on is moving business value measures, you're probably putting lipstick on a pig. But testing against value creation has the potential to uncover game-changing opportunities.

Here's an email exchange between Jason and me, in which he explains in a little more detail:

 

RYAN:

That whole web site optimization thing—isn’t it really just putting lipstick on a pig?

 

JASON:

I think “I hate you so much” might be a succinct way of responding, but I'll provide a little more detail:

Web Site Optimization is exactly like putting lipstick on a pig, but only if you start out with a pig. And if you are starting out with a pig, your opportunities for improving things are limited, and you’d be using the wrong tool to fix the problem. We are talking here about the concept of a “local maximum” which is a fancy, math term applied to mean “the best something can be within a limited dynamic.” Consider the aforementioned pig’s ability to fly, which, metaphorically speaking, is not particularly developed. We could take a pig and genetically modify it to be more aerodynamic. We could investigate building pig hang-gliders and attempt to train the smartest pigs to use them. But even in the best case, with the most aerodynamic pig, benefiting from the best training, and using the best pig flying technology, it will never fly as well as a bird. The best case flying scenario for a pig (the pig’s local maximum as far as flight is concerned) is nowhere near as effective as a bird’s. In that scenario, you’d be better off exchanging the pig for a bird at the start, rather than waste any time or effort teaching a pig to fly better.

Applied to the web: if a site sucks so much- if the goals and purpose are unclear, if the information architecture looks like my desk (at the moment), if the navigation is counterintuitive and the messaging has absolutely no intersect with the audience, then no amount of optimization in the world will make it right. The local maximum of that crappy site is too low for any optimization to matter. Or (even worse) you’d need the infinite number of monkeys to stop typing Shakespeare and to start applying experiments to your site to get the right combination where testing would make a real difference. Neither is very efficient. If your site is the pixilated equivalent of a pig, you need much more elemental help from a user experience expert first (know any?). Until you fix the fatal flaw(s) in a site, anything else you do will be throwing good money after bad.

Site Side Optimization works well in circumstances where the local maximum is high, but for some reason, the site is not achieving it. This can be due to single points of failure on the site, like a specific conversion path or page underperforming, or because the audience needs to be targeted more specifically, or because the existing content is stale/irrelevant. In each of these cases, experimental testing can make a huge difference. Optimization also works exceptionally well (and this is far more interesting to me) when applied as a method of trying out a new (and potentially risky) idea that could radically change and significantly improve an experience.  In both of these examples, the basic site is healthy, and the optimization program serves as a tool to reach its fullest potential.

 

RYAN:

But what I keep looking for is the way to test birds against pigs, not in the sense of which flies better, because as a user experience expert I do have the capability to predict the winner of that contest—but when I don’t have a clear sense of the best conceptual solution. For example, maybe I just can’t decide between eggs and bacon. Can optimization help design a better breakfast, or only decide between pulp and no-pulp in the OJ?

 

JASON:

Optimization can test more conceptual ideas, but it will be really hard to unpack the WHY after we determine which one wins. Most sites aren’t deciding between bacon and eggs, but rather between the bacon, eggs and hashbrowns with coffee or the granola, fruit and yogurt with yerba matte. If the former wins in a test, I don’t know whether it’s because of the bacon (which will usually win over everything) or the coffee, or because the person deciding had granola for the past three days, and would have taken ANYTHING other than more granola.

The other trick about testing high concepts in a website format is that you would have to build each solution to test them, which is usually more expensive than testing out wireframes or front-end prototypes in front of a more controlled audience.

 

RYAN:

First, you seem to be suggesting that a test win for the bacon breakfast might not imply extensibility for bacon breakfasts in general: That because, lacking control, the results might be idiosyncratic, they might not therefore apply broadly. Next week you might get a different result. My question is, why does that matter? And why does “knowing why” the bacon breakfast worked matter, as long as you know it worked.

 

JASON:

It is definitely a question I get from clients a lot. Why do I care about the individual elements of a variant- if the variant as a whole makes us more money, let’s just launch it and move on. I can’t fault the sentiment, but knowing why the bacon worked could lead to better tests, more focused messaging and (even more) cash money. I want to know that it’s the bacon by itself that is the motivating factor outside of all the other influences. Let’s say that we ran a test breakfast against a bowl of Total cereal and we tested bacon with powdered eggs as the experimental variant. Now let’s assume that bacon and powdered eggs lost to the control by 1%. You could take the position that we would do better to serve Total because we want to avoid losing that 1%, and you would be right. But what if you knew that the bacon by itself actually improved the breakfast by 15%, but the powdered eggs were so crappy that they hurt the breakfast by 16%, so you netted the 1% loss? If I controlled for all the variables in my breakfast, that knowledge would a) help me make a better breakfast overall (just serve bacon), and b) will also prevent me from throwing away a positive variable simply because it was paired with a really negative one.

I imagine you run into that a lot with prototyping and wonder how you deal with it in the UX world.  If a subject totally fails at a task, are you ever afraid of overcorrecting a prototype to account for it? Do you ever throw the baby out with the bathwater? How do you control for that?

 

RYAN:

A great question. One of the answers is that in usability testing, you're looking for usability problems. So as long as your test participants are representative of your users as a whole, major failures are, practically speaking, never anomalous. If your user population is one million, and one of the eight people participating in your test has trouble understanding some aspect of the interface, what are the odds they're the only person who's going to have that problem?

The other thing I wonder about is what happens when what you’re trying to accomplish is harder to measure than conversion (e.g. brand lift) or if you want to measure it over time (e.g. engagement). Especially in social media, it’s quality that matters, not quantity. You want to know how valuable your user-generated videos are more than you want to know how many of them you have. Can web site optimization help you get to answers?

 

JASON:

Ah, you and your social media. When are you going to come to terms with the fact that this whole thing is a fad? The future is in email, Ryan, and lots of ‘em. Mark my words.

Absent a more qualitative tie-in with optimization (surveys, satisfaction scores, etc.) you will be hard-pressed to get good data about branding or the impact of social media. But I’m not saying you shouldn’t optimize for branding or social media. I’m saying you need to get that qualitative kicker. I’ve done a few branding tests, and I think they provide some interesting feedback. But I’ve never optimized where a KPI has to be judged on quality (e.g., good comments vs. troll comments) or off the site entirely (e.g., buzz in the blogosphere). Sounds fun.

Site Optimization: Lipstick on a Pig?, Or, Bacon and the Theory of Local Maximum

By Ryan Turner | Dec 1, 2008 8:01:09 AM

(Cross-Posted from Web Social Architecture.)

Jason Carmel is a colleague of mine I learn a ton from. His expertise is in web site optimization--running experiments where he tests versions of web pages against each other to see which performs best. (Not to be confused with search engine optimization, improving a site's visibility in search engines.)

Jason is a fairly unflappable guy. Nonetheless I recently started making an effort to get his goat. He gives me just enough encouragement that I keep going. The gist of my teasing is that optimization is nothing but a mechanical exercise to determine whether a red button works better than a blue button. "Glad to hear that red button worked out better by 2.84 percent Jason. The sum of your creative energy has produced yet another quarter million in revenue. You must really love your life, man. Hey, have you thought of trying one of those animated GIFs instead of a regular button?"

Fortunately, Jason is twice as nice as me, as well as twice as smart. He takes my ribbing well--and responds thoughtfully to the serious question underlying my teasing: We know optimization can move big numbers in terms of revenue, but can it do more than simply tweak pages to bump up conversion? Can it vet creative concepts? Can it maximize the creation of mutual value between businesses and customers? Can it help create more engaging experiences?

(image credit, found via Lee)

The short answer here, according to Jason, is that it depends, partly on what you're trying to achieve. If all you're focused on is moving business value measures, you're probably putting lipstick on a pig. But testing against value creation has the potential to uncover game-changing opportunities.

Here's an email exchange between Jason and me, in which he explains in a little more detail:

 

RYAN:

That whole web site optimization thing—isn’t it really just putting lipstick on a pig?

 

JASON:

I think “I hate you so much” might be a succinct way of responding, but I'll provide a little more detail:

Web Site Optimization is exactly like putting lipstick on a pig, but only if you start out with a pig. And if you are starting out with a pig, your opportunities for improving things are limited, and you’d be using the wrong tool to fix the problem. We are talking here about the concept of a “local maximum” which is a fancy, math term applied to mean “the best something can be within a limited dynamic.” Consider the aforementioned pig’s ability to fly, which, metaphorically speaking, is not particularly developed. We could take a pig and genetically modify it to be more aerodynamic. We could investigate building pig hang-gliders and attempt to train the smartest pigs to use them. But even in the best case, with the most aerodynamic pig, benefiting from the best training, and using the best pig flying technology, it will never fly as well as a bird. The best case flying scenario for a pig (the pig’s local maximum as far as flight is concerned) is nowhere near as effective as a bird’s. In that scenario, you’d be better off exchanging the pig for a bird at the start, rather than waste any time or effort teaching a pig to fly better.

Applied to the web: if a site sucks so much- if the goals and purpose are unclear, if the information architecture looks like my desk (at the moment), if the navigation is counterintuitive and the messaging has absolutely no intersect with the audience, then no amount of optimization in the world will make it right. The local maximum of that crappy site is too low for any optimization to matter. Or (even worse) you’d need the infinite number of monkeys to stop typing Shakespeare and to start applying experiments to your site to get the right combination where testing would make a real difference. Neither is very efficient. If your site is the pixilated equivalent of a pig, you need much more elemental help from a user experience expert first (know any?). Until you fix the fatal flaw(s) in a site, anything else you do will be throwing good money after bad.

Site Side Optimization works well in circumstances where the local maximum is high, but for some reason, the site is not achieving it. This can be due to single points of failure on the site, like a specific conversion path or page underperforming, or because the audience needs to be targeted more specifically, or because the existing content is stale/irrelevant. In each of these cases, experimental testing can make a huge difference. Optimization also works exceptionally well (and this is far more interesting to me) when applied as a method of trying out a new (and potentially risky) idea that could radically change and significantly improve an experience.  In both of these examples, the basic site is healthy, and the optimization program serves as a tool to reach its fullest potential.

 

RYAN:

But what I keep looking for is the way to test birds against pigs, not in the sense of which flies better, because as a user experience expert I do have the capability to predict the winner of that contest—but when I don’t have a clear sense of the best conceptual solution. For example, maybe I just can’t decide between eggs and bacon. Can optimization help design a better breakfast, or only decide between pulp and no-pulp in the OJ?

 

JASON:

Optimization can test more conceptual ideas, but it will be really hard to unpack the WHY after we determine which one wins. Most sites aren’t deciding between bacon and eggs, but rather between the bacon, eggs and hashbrowns with coffee or the granola, fruit and yogurt with yerba matte. If the former wins in a test, I don’t know whether it’s because of the bacon (which will usually win over everything) or the coffee, or because the person deciding had granola for the past three days, and would have taken ANYTHING other than more granola.

The other trick about testing high concepts in a website format is that you would have to build each solution to test them, which is usually more expensive than testing out wireframes or front-end prototypes in front of a more controlled audience.

 

RYAN:

First, you seem to be suggesting that a test win for the bacon breakfast might not imply extensibility for bacon breakfasts in general: That because, lacking control, the results might be idiosyncratic, they might not therefore apply broadly. Next week you might get a different result. My question is, why does that matter? And why does “knowing why” the bacon breakfast worked matter, as long as you know it worked.

 

JASON:

It is definitely a question I get from clients a lot. Why do I care about the individual elements of a variant- if the variant as a whole makes us more money, let’s just launch it and move on. I can’t fault the sentiment, but knowing why the bacon worked could lead to better tests, more focused messaging and (even more) cash money. I want to know that it’s the bacon by itself that is the motivating factor outside of all the other influences. Let’s say that we ran a test breakfast against a bowl of Total cereal and we tested bacon with powdered eggs as the experimental variant. Now let’s assume that bacon and powdered eggs lost to the control by 1%. You could take the position that we would do better to serve Total because we want to avoid losing that 1%, and you would be right. But what if you knew that the bacon by itself actually improved the breakfast by 15%, but the powdered eggs were so crappy that they hurt the breakfast by 16%, so you netted the 1% loss? If I controlled for all the variables in my breakfast, that knowledge would a) help me make a better breakfast overall (just serve bacon), and b) will also prevent me from throwing away a positive variable simply because it was paired with a really negative one.

I imagine you run into that a lot with prototyping and wonder how you deal with it in the UX world.  If a subject totally fails at a task, are you ever afraid of overcorrecting a prototype to account for it? Do you ever throw the baby out with the bathwater? How do you control for that?

 

RYAN:

A great question. One of the answers is that in usability testing, you're looking for usability problems. So as long as your test participants are representative of your users as a whole, major failures are, practically speaking, never anomalous. If your user population is one million, and one of the eight people participating in your test has trouble understanding some aspect of the interface, what are the odds they're the only person who's going to have that problem?

The other thing I wonder about is what happens when what you’re trying to accomplish is harder to measure than conversion (e.g. brand lift) or if you want to measure it over time (e.g. engagement). Especially in social media, it’s quality that matters, not quantity. You want to know how valuable your user-generated videos are more than you want to know how many of them you have. Can web site optimization help you get to answers?

 

JASON:

Ah, you and your social media. When are you going to come to terms with the fact that this whole thing is a fad? The future is in email, Ryan, and lots of ‘em. Mark my words.

Absent a more qualitative tie-in with optimization (surveys, satisfaction scores, etc.) you will be hard-pressed to get good data about branding or the impact of social media. But I’m not saying you shouldn’t optimize for branding or social media. I’m saying you need to get that qualitative kicker. I’ve done a few branding tests, and I think they provide some interesting feedback. But I’ve never optimized where a KPI has to be judged on quality (e.g., good comments vs. troll comments) or off the site entirely (e.g., buzz in the blogosphere). Sounds fun.

My summer prediction: Microsoft will buy Yahoo by year end

By Nigel Morgan | Aug 19, 2008 2:09:24 PM

So it has been a long summer so far and I hope there are many more days of sunshine to go. It has had its ups, just look at Michael Phelps, and it has had its downs as we bid farewell to Isaac and Bernie. Of course for us online marketers there has been an ongoing story that has had both the ups and the downs. I am talking of course about the on again, off again, on again but now off again saga of the Microsoft takeover of Yahoo.

So while Jerry Yang is in China at the Olympics and the Microsoft upper management is enjoying a fine Pacific Northwest summer, I will throw in my opinions on where this story will end.

To me it is quite clear that this merger will happen; there are a number of reasons why this deal will go through for Microsoft and Yahoo. I list them here and ask you all to join the debate!

Market share of total online ad revenues.

This is the number one reason why this takeover will happen. Online advertising revenues are still growing very fast and will continue to do so according to every survey and expert opinion you read.

Today Google is firmly in the box seat owing to its market position as undisputed search engine leader and ownership of Doubleclick. Once Google fully optimizes it’s ability to understand the vast majority of internet behaviors through its search engine and Doubleclick tags, it will be able to target marketing messaging to daunting effect (read attain massive revenues from online advertisers). Although MSFT owns Atlas, the combination of Atlas and MSN will, most likely, never get MSFT to a market leadership position. If Microsoft doesn’t gain a huge increase in market share and soon, it will be virtually impossible for them to catch Google. How do you suddenly increase market share? Answer, buy a competitor.

“Owning” the vision of the PC is Microsoft’s heritage

Microsoft has continually innovated across the years and while they have some very successful products such as X-Box, Windows and Office to name a few, they have not truly come on from their computer on every desktop vision of 20 years ago.

When a computer was used mostly for work, Microsoft set the vision for the personal computer with its operating systems and productivity software. These products are still by far the biggest driver of the company’s profits. However thanks to internet applications PCs now are equally used for recreation and Microsoft has never managed to dominate the market in this reality. Microsoft’s online presence that gave them early market advantage in the form of Internet Explorer and MSN are becoming more strategically irrelevant every day and their market share continues to follow a long-slow downward trajectory. Efforts such as Silver light, .NET and other initiatives will not change the fortunes of Microsoft in this regards.

The people at Microsoft need to innovate

While Microsoft could happily continue to dominate the OS market, Bill Gates will not want to see his life’s work become a monopolistic commodity. Microsoft also has enormously talented employees and they will not go down this route without kicking and screaming.

It makes good old fashion financial sense.

Although MSN and Yahoo users will not like me saying it, they really are substitute products for each other. Taking best practices from each and merging them together will improve the product and dramatically reduce redundancy. It will also create a broad base of messenger and email users that will dwarf the number of users on gmail. This is one beachhead that a combined Yahoo/MSFT should exploit in their ability to compete with Google.

At the end of the day, it may be the only path ahead for Yahoo too

Unless Yahoo does things very differently, they will continue to lose market share in search queries. This will ultimately result in long-term financial disaster as their editorial teams and messenger and email products will be left to support site visits and therefore online ad revenues. Yahoo’s content is not differentiated enough from other content sources to survive in the long run.

So, a takeover makes a lot of sense, ironically more so for the one that is blocking it. Microsoft can have a future without Yahoo but I struggle to see where Yahoo fits into the online ecosystem long term. Therefore I predict that Yahoo will come to this conclusion, Microsoft will have another attempt at a takeover and this time it will go through.

So there you have my opinions. I’m off to see my stockbroker!

An Optimization Mix-Tape

By Jason Carmel | Aug 5, 2008 4:53:40 PM

I grew up making mix-tapes (note: for those of you unfamiliar with the concept of a mix-tape, think "playlist," but with far, far crappier technology than you are using right now to listen to far, far crappier music). I am not the first person to wax eloquent on the poetry of the mix-tape, so I will spare you my own musings in deference to those better written. There is no more holistic way to describe a circumstance, event, or task than through a lovingly-crafted compilation of songs. Normally, the songs speak for themselves. I don't need to tell my ex-girlfriend, for example, that the third song on side two of my 8th Grade Summer Mix ("Baby Bitch" by Ween) was pointed directly at her. She knows, man. Used properly, the mix-tape is a perfect medium.

This is a mix-tape for optimization. Ten life lessons about how and why you should be testing your websites, all neatly wrapped in a delicious candy-coating of RAWK:

1. "I Might Be Wrong" - Radiohead (Amnesiac) - This is why we test in the first place, so it's a good one to start off with. You never know whether a new initiative is helping or hurting relative to what was up there before (and by how much) unless you run a test to prove it. As this theme could be our anthem, it deserves pole position. Plus, every mix-tape since Pablo Honey came out had to include a Radiohead song by law, so it's good to get that out of the way (I'd usually cram mine in between "Forever Young" by Alphaville, and whatever Depeche Mode song I was listening to at the time). If those two weren't reason enough, I also have it on good authority that this is one of our Project Manager's (Ariel) favorite songs. First rule of test execution: do whatever it takes to get on your PM's good side.

2. "If I Had $1,000,000" - The Barenaked Ladies (Gordon) - If you had a million dollars (or, as is often the case, far less), where would you apply those resources? Would you test your homepage? The landing page for search results? Your credit card form? Any optimization program worth its salt will dynamically prioritize tests according to return on investment. The song is goofy, but the concept is deadly serious. Live by it.

3. "Minus Two" - Fishboy (Albatross) - Know your math. Love your stats calculators. Revel in Excel. Running a test is a lovely thing, but if you can't determine whether the results are statistically valid, you're in a bad place. In truth, the song doesn't have a lot to do with math (other than the name), but  Fishboy is one of our Web Analytics expert's (Halee) favorite bands, so it's in. Second rule of test execution: do whatever it takes to get on your Analyst's good side.

4. "Invisible Ink" - Aimee Mann (Lost in Space) - Aimee (who has a little crush on me) reminds us to test hiding things in addition to adding things. As everyone fights for real estate on a homepage (for example), customers may get lost with all the choices in front of them. Try hiding a few things (do people really need to sign up for a newsletter on the first visit to a homepage?) and see if that doesn't boost engagement or clickthrough.

5. "Tracks of My Tears" - Billy Bragg (Talking With The Taxman About Poetry) - Covers are wonderful things, as they remind us how a different voice can transform a message. Here we have a smooth, Motown production by Smokey revamped entirely using a single electric guitar and a very bitter-sounding bloke from Essex. What Billy teaches us is that we should consider testing new voices for our messages to see if they strike a chord (proverbially speaking) with our audience as a whole, or various subsets thereof.

6. "Lola" - The Kinks (Lola versus Powerman and the Moneygoround, Part One) - Ah, here we have the classic tune about unexpected results reminding us to watch out for the same when we are testing. You expected conversion rates to go up, and they did, which is great. Did you also expect your churn to skyrocket? A 20% off coupon you tested increased purchases, which is great. But did you expect your average order value to drop through the floor? Much like transvestites in SoHo clubs, unanticipated results aren't necessarily a "bad" thing, but the more work you can do to spot them in advance will save you the occasional awkward moment after launch.

7. "All My Mistakes" - The Avett Brothers (Emotionalism) - Not all of your tests will make things better. Some will have no effect at all, and some (gasp) will demonstrate that the control version is the best one out there at the moment. These are not bad outcomes. As long as you have entered an experiment with a hypothesis, you are proving or disproving something that can lead to a better understanding of how your customers interact with your site, and (just as important) a list of future experiments to consider executing.

8. "Know Your Onion" - The Shins (Oh, Inverted World) - This is my song for segmentation and behavioral targeting. Justin, my colleague from UX, always tells clients that a site should get better at addressing the needs of customers as it acquires more information and adjusts content accordingly. He refers to this as peeling layers of an onion with each customer interaction. So, music fans, I challenge you to know your onion, and to know how to peel those layers off for your customers to target them with relevant content as they demonstrate specific behaviors on your site.

9. "Do What You Want" - Bad Religion (Suffer) - A punk staple, in my humble opinion. Many herald the song as a generalized call to anarchy. While I wasn't in the room when the band wrote the song, I believe that they were referring specifically to the sense of freedom you get by testing your site (some may point to the album's 1988 release date- some five years before the first web browser- as evidence to the contrary. I just think of it as yet another demonstration of how ahead of its time Bad Religion was). "Break all the @$%&ing rules," says singer, Greg Graffin, and I agree. Remove "we've always done it this way" from your vocabulary and challenge your site's most sacred of cows to see if they prove their merit against other alternatives.

10. "Automatic" - Less Than Jake (Losing Streak) - It took a courageous ska-punk band from Gainesville to remind us how we should include optimization automatically in all site releases. Got a new feature going out? Test it. New product offering? Test it. Campaign? Testamundo. You get the idea. Include optimization automatically into your production and release schedule. Your site will be better for it.

Enjoy the music, folks.

[Author's Note: Incidentally, I don't think Chris Kerns could come up with a mix-tape for analytics. I just don't think he has it in him.]

You're Invited: Social Media Event at ZAAZ Seattle July 8th

By Ryan Turner | Jun 30, 2008 10:58:45 AM

Cross-posted from Web Social Architecture.

People keep asking me when we're going to host another event at the ZAAZ office's notorious Z-Bar in Seattle. Well...

I'm super excited about the lineup for this event, which includes industry leaders from corporate, agency, and independent circles who share passion and expertise in the human dimensions of social computing. Knowing most of these folks pretty well, I think I can guarantee an evening of thought-provoking conversation.

The format for this event will be similar to the previous one I organized, which seemed to go pretty well. We'll have short talks from each speaker followed by a panel discussion. Plus, snacks and beer.

Here's the rundown on the speakers:

Brian Fling of Flingmedia is a full-fledged mobile design geek and human Swiss Army knife. His talk, "Mobile 2.0: Design and Develop for the iPhone and Beyond" will explore some of the social dimensions of the emerging mobile world.

My colleague Justin Marshall is behind some of the most exciting work we've done in social media. His take on social media for marketers, titled "Money, Media, and Your Mom's Peach Cobbler: Social Media Marketing Done Right," offers guidance for marketers looking to engage with customers online.

Samantha Starmer is a highly-respected thought leader in the local information architecture community. In addition to her work at REI, she co-teaches the UW Information School's Summer IA Institute. Her talk, "Single Athletic Female seeking Single Slender Male: The Marriage of Social Media and Metadata," promises to reveal the secrets of better online living through metadata.

Nancy White of Full Circle Associates has been doing online community since WAY before it was cool. Her broadly-ranging expertise includes online learning and facilitation, communities of practice, technology in the developing world, and social technology in general. Her talk is titled "Slow Community."

Wendy Chisholm is a former co-editor of the W3C's Web Content Accessibility Guidelines and a passionate advocate for universally-accessible design online and off. She's currently working on what promises to become "THE book" on accessibility for the emerging Web.

We're really lucky in the Seattle area to have access to such quality thinkers and experts. I hope you'll join us, and if you do plan to attend, drop me a note at ryant (at) zaaz dot com so I can get a rough headcount.

The Facebook event is here: http://www.facebook.com/event.php?eid=66791410200.

I hope to see you there!

A Movie Test

By Jason Carmel | May 27, 2008 5:33:42 PM

I was sick over the Memorial Day Weekend, not that you cared enough to call or anything.

When I'm sick, I like to watch movies in the theater (preferably), provided I'm not rampantly contagious or bleeding out of both ears. Fortunately, the three day kick-off to the summer blockbuster season affords a lot of movie-watching opportunity, and so I saw the new Indiana Jones flick and Prince Caspian. Before you ask, no, I don't have kids. I saw them because they were at the top of my list. Don't judge me. You're judging me. Stop judging me.

I will provide my review of the films in five words or fewer, and will then get to the point of this post:

Indiana Jones: Still good; please stop now.
Prince Caspian: Not Lord of the Rings.

One thing I noticed (maybe because I was sick and in a foul mood) is how soul-crushingly terrible the movie-going experience is. I feel like I'm getting screwed at every turn. Tickets are north of $10 each, which gives me the privilege of sitting in an uncomfortable, dirty, ice-chest of theater while I get skeezy, full-frontal advertisement until the movie starts. The food is about 4X what you would pay for it at a 7-11, and I saw a helpful woman behind the concession stand counter literally ask a gentleman six different times if he would prefer a combo. (Note: I can't blame the woman for being so effusive about the combo opportunities. There's a sign in front of each register that bribes patrons to narc on the concession folks if they fail to do so.)

I suspect it couldn't be too hard to make movie-going fun again. I can think of tons of things I would try if I owned a theater. Maybe free popcorn (if salted properly, a nice loss leader for soft drinks). Seat consoles that let you vote on what parts of the film you liked or didn't like. Participatory cinema, like the geniuses at the Alamo Drafthouse Cinema do in Austin, TX.

I'm probably not the first guy to come up with this and more, so I wondered (while waiting for the NyQuil to kick in) why only the random independant theaters ever do anything that attempts to re-inject the positive back into an experience that can only be defined (generously) as craptacular. It occurred to me that, in all likelihood, the Standard Corporate Theaters (SCTs), such as the ones I patronized this weekend, were capable of the same ideas, but were probably stuck in a rut looking at the wrong things to see real results. My suspicion is that the SCTs rely on a multiplier of concession sales, tickets sold per movie showing, and advertising income to determine revenue. Asking a customer six times if he would like a combo is a good thing, because a test of such would demonstrate an immediate X% lift in concession sales. Any cost (e.g., a bad taste in the mouth for most patrons) wouldn't hit the bottom line until long-after it could be attributable to ComboMania. Now contrast this to a theater owner who gets permission to run a more disruptive (in a good way) free-popcorn test- the test is almost always doomed to failure, since, quite obviously, concession sales will deteriorate significantly in the short term if you are giving away free product, and any positive impact from word of mouth will take many weeks to cultivate. If the SCTs never take a step back and let a deeper change happen, I think we can expect little in the way of real improvement (and, most likely, combo offers in the bathrooms).

Maybe in addition to measuring a radical new idea on the same success metrics, the SCTs could learn something by finding new things to measure, like relative customer satisfaction, propensity to return, etc. When you are looking to fundamentally disrupt the way your business operates, new measurements are generally a necessity. You never want to abandon the bread-and-butter metrics of your business, but it is unfair to hold tests that seek to go beyond the normal parameters of the business model to only those measurements collected for the old business model.