Kampyle is part of Medallia Inc. | Visit us at our new home.

Go To Medallia.com

The Boy Scout Approach to A/B Testing: CRO, One Step at a Time, Part 3

  • The Boy Scout Approach to A/B Testing:

Want to get the best results possible from your testing program, as quickly as possible?

Here’s what you need to do. And it’s not what you think.

In this third post of our Conversion Optimization, One Step at a Time Series, we focus on planning and interpreting tests. You might note that the obvious step between planning and interpretation — running the test—is missing. That’s because in a successful testing process, the actual test is almost the last thing you’ll do.

The best way to ensure that your test will yield valuable, applicable results is to take your cues from the Boy Scouts and be prepared. Invest time and effort in projecting possible findings to make sure that they can be interpreted unambiguously and will yield the insight you need.

Skip the uniform, the neck scarf and the badge for knot-tying, if you must. But the bottom line is that carefully thinking about the end results you’re aiming for, and making sure you have everything in place to get there, is the best way to get the answers you need.

Document your plan

What is clear to you on Monday can be maddeningly murky by Wednesday. Write down each hypothesis you wish to test. Document how you plan to test it. And then let it sit for a day or two. Look again and think again, or bounce it off a co-worker.

When planning a test, it is best to change only one element at a time, so you know exactly what is driving changes in customer behavior.

Include as many details as possible in your documentation. View it as a database that you can turn to in the future to increase your understanding of customer needs and behavior.

Play out the possibilities

List out possible test results, both positive and negative. And don’t forget potential ambiguous results. What does each indicate about your hypothesis? Does it prove it? Disprove it? Are there other possible interpretations? Confounding factors? And does the test address the issues that customers raised in their feedback?

Make feedback an intrinsic part of your test plan

Put a feedback solution in place to gather information about why visitors respond to the new element or process as they do. Invite visitors to leave feedback immediately following their interaction with the element being tested. Ensure that the feedback trigger is neutral and open-ended to avoid prejudicing customer responses and to gather feedback in the customers’ own words.

Design test elements

Once you have a crystal clear statement of the hypothesis you are testing, and have documented exactly how you will test it, design graphic, functional and/or test elements for the test. These might include alternate messages, designs, images and website elements as well as alternate service options, product offerings or processes.

Make sure that new elements work in the context of the sales funnel as well as in the context of the individual page. For instance, a link to an appealing visitor may please customers, but if it diverts them from the purchasing process, sales will suffer.

Consult with in-house personnel who are not involved in the testing process but do know a lot about your customers to help you create alternatives that are likely to be better, not just different.
Once all is in place, go for it!

Excellent preparation makes running and interpreting tests a snap. Check back soon for our next post on implementing A/B and multivariate testing and interpreting results. And while you’re waiting, you might want to check out the clove hitch knot after all!

About the Author: