How to Design the Best Variants for an Optimization Test

By Rachel Elkington | 3 Comments | Posted in in User Experience | Permalink

Variant design in Optimization is important, and it deserves more care and attention from Optimization test designers.  In the current practice of test design, much attention is paid to analytics numbers when deciding the hypothesis and direction of a test.  However, numbers by themselves only tell you where the problem is, they do not tell you what alternatives to try.  At ZAAZ, we have been combining User Research methods with our Optimization practice and have found a new piece to this puzzle: User Research in the process of variant design.

So, how can you incorporate UR to design the best variants you can?   You need to…

Stop, collaborate, and listen.   There are several ways to use your friendly local usability/UR practitioner to help generate a hypothesis and variants to test that hypothesis.  Which method is best in which case depends on the nature of the page, problem, and budget in question.  The rule of thumb here is something is always much better than nothing.

Heuristic Review:  When budget and timelines are mercilessly tight, as they often are, a heuristic review can turn up a lot of insight in a little time.  These employ the best practices in user experience as a template to which the site is compared.  The findings from an heuristic review can do two things for a test designer: 

1)    Identify which usability best practices are being compromised on under-performing pages.  This lends itself easily to hypothesis creation.  Test variants can then be alternate ways to implement the best practice on the page.

2)    Uncover places to test on the site that are not immediately obvious through an analytics lens, but that need to be improved.  

Usability Test:  This is my best-practice recommendation when deciding what to test.  In concert with analytics data, usability tests become invaluable.  Usability tests are best introduced when a test designer sees analytics data he or she cannot explain.  For example:

Recently, we at ZAAZ had created a test that was meant to increase the number of purchases of an add-on offer in a purchase path.  We created our variants carefully and lovingly - even bringing UX in for a consult - remaking the underperforming page into a clear, concise, value-communicating standout.  And yet, the ultimate conversion rate for users who saw ANY of our variants was much lower than the control.  We thought:  What gives?  So, we conducted a usability test on the control and on the variants.  In the course of the usability test we found that the promise made by the page we were testing was not clearly reflected further down the purchase path.  Our variants that made the initial promise more explicit led to confusion and abandonment further down the line.  This lack of continuity was subtle – but it was affecting user confidence in the process.  Introducing a usability test was key to getting in touch with our user’s qualitative impressions, which were driving their ultimate decisions in the purchase path.  Put another way, usability tests uncover the root cause of a problem.  Root cause understanding can show you both where you really should be testing, and what your hypothesis should be.

Usability studies can also be used to vet variants before they are launched.  This is a particular advantage with high-traffic tests where any underperforming variant can be costly. 

Eye Tracking:  Eye tracking is a great tool, if used wisely.  It can be used both to define hypotheses and to evaluate variants that have already been made.  That is because eye tracking shows you what users actually notice on the page – where their eyes go.   As a method of inquiry, eye tracking can satisfy what can otherwise be exhausting and unproductive internal debates about what the user is actually noticing on the page, or what is easily ‘discoverable.’  If the name of the game in a particular test is to get the user to notice a key piece of content, eye tracking is your new best friend. 

Other:  The most exciting thing about the intersection of UX and Optimization is that it is new.  We are still discovering new ways to put these disciplines together. 

Rachel Elkington is an Online Test Designer in the Optimization group at ZAAZ.  Previously, she worked in ZAAZ’s User Experience Group.  This combination of disciplines means she gets to have lots of fun scheming up new ways to put qualitative and quantitative methods of inquiry together.  In her spare time she heads up the Pacific Northwest chapter of the American Society of Information Science and Technology, and co-produces InfoCamp – an annual unconference for the user-centered information industry.  She has an MS in Information Management from the University of Washington, and a BA in a liberal arts discipline that people told her wouldn’t get her a real job.