There’s no S-T-R-A-T-E-G-Y in Testing Technologies

Last week, Google announced their webpage testing technology, Google Website Optimizer (GWO), has been released from beta and is available, at no cost, to the general population. The questions poured in asking if this would cause our clients to take page testing in-house rather than rely on agency offerings. Here are a few reasons why this free technology isn’t going to be an immediate game changer for Resolution Media:

1. Testing is not just about technology. GWO is an execution tool. Prior to even using the tool, the strategy must be developed, data analysis completed, benchmarks/KPIs established, hypothesis written, confidence intervals agreed upon, etc. GWO isn’t going to tell you to test page two of the checkout process vs. a category landing page vs. your product page template…much less, what the alternate page designs should look like.

2. Landing page testing is as much about gaining new traffic as it is about improving the experience for current visitors (OK, a close second). Besides crunching historic bounce rates and conversion rates of landing pages, SEO & PPC opportunity must be included in the page selection strategy. As both Natural and PPC rankings factor in page relevance, someone versed in traffic levels, competitive opportunity, and ranking algorithms certainly should be part of any page design. It’s important to test which headline visitors react to more, but you also have to weigh which one may bring your more visitors via Natural Search or lower your PPC CPCs/ROI.

3. Lack of experience, talent, and ownership of page testing. No matter how many flash demo or help pages exist, there is a barrier to entry on experience on running tests as well as talent to interpret the data. This won’t exist for long, but in the meantime the testing talent is found in many agencies and WA consultancies. In addition, who, on the client-side, would own a testing initiative? IT, Marketing, E-Commerce?

4. Limitations of GWO. Like GA (Google Analytics), GWO is built for the general masses and that will not suit advanced/custom experiments or data analysis. For example: editing tests mid-stream, throttling test delivery, conditional element testing, traffic segmentation, testing non-conversion events, and more. You’ll need someone experienced in test design to navigate the waters of technology selection as GWO isn’t right for some experiments.

5. Google paranoia. Be it justified or not, there is reluctance to share conversion data (and profitability) with a media vendor who sets pricing (i.e. minimum bids for AdWords placement).

I don’t want to rain on the parade of a great tool; I am a big fan and user of GWO myself. In fact, I am thrilled to see the shift from optimizing incoming traffic to including the website experience itself. However, GWO is not the full answer to webpage testing. A quote sums it up well: “Using technology without strategy is a waste of technology.”

Posted by: Jeff Campbell, VP Product Development & Innovation


Anonymous said...

Nice post Jeff. Just one thought I would add to this conversation.

There are three types of strategy needed for testing. One is the strategy needed for deploying the technology. This can actually be quite creative.

The second is the analytic strategies for segmentation, test design and results analysis.

The third is the creative/marketing strategies for the tested elements.

Thanks for shining a light on test strategy.

CJeffCampbell said...

Great addition and very true, thanks sir!

Anonymous said...

Great list! The importance of testing expertise cannot be understated. I find that clients often have most of the skills necessary but don't know how to piece them together properly for testing. Optimization experience and knowledge are key to getting high impact results and getting them quickly.

One thing that was not mentioned is the importance of creative and copy. Agency's often excel in providing materials that is "outside" the usual of what the company is already doing. The biggest lifts usually come from taking risks and going in a direction that would be hard to justify without proper testing.

Anonymous said...

Great post Jeff. Optimization expertise cannot be understated. I have seen many clients try to optimize their page themselves with little success, but a lot of effort. While some companies may have the resources and some of the skills to do testing, it takes an experienced testing professional to bring things together in a way that gets results and gets them quickly.

I would also add that an additional advantage of having an outside party do the optimization is the new creative and copy that is generated. Taking risks and trying new ideas get the biggest lifts in my experience. Having an outside expert and testing technology to justify those risks, allows for new content and creative to be used that may never have been implemented if only done internally.

Copyright © 2008 Resolution Media, Inc. All rights reserved.