Kinds of Split Testing Software

Split testing works. Use it well, and you will improve your conversion rates, and, thus, your profits. And, if you are advertising, or have significant product costs, raising your conversion rate just a little can raise your profits a lot. But there are many different kinds of split testing software out there.

There are many differences among the split testing software options on the market today. Let me give you a brief taxonomy of the subject matter.

First, there's a difference between multivariate and single variable testing.

Single variable split testing is very simple. You just try stuff and measure the results. And all you need is a tool that rotates and keeps score.

There are a few simple split testing tools available. Some are free and some cost money. Google offers a free tool, though you do have to have an Adwords, and an Analytics account to use it.

Single variable testing can be slow. How would you like to test 10 times as much with the same amount of traffic? Well, you can. You use a method called multi-variable, or multivariate testing. This allows you to test several elemens on your page simultaneously.

Among multiple variable software solutions, there are hosted plans and self-install plans. If you want to install software on your server (which can be faster and cheaper) you will have more control over your data. And you can usually find someone to install it for you (often the vendor provides this service for a nominal fee). If you want to go with a hosted plan expect to pay more.

Hosted solutions include Google's Website Optimizer, and Vertster. Installed solutions include the Split Test Accelerator and Kaizen Track.

The next distinction probably causes the most confusion among people looking into split testing for the first time. It's also more important than the last distinction, becuase it determines how fast and accurate your tests can be. The three kinds of testers are 1) random independent rotators, 2) Taguchi testers, and 3) full-factorial testers. Taguchi ranks first for speed and second for accuracy. Basically Taguchi testing involves using a marvel of mathematics known as "orthogonal arrays". These have special properties that allow you to test every combination in a space while running only a small fraction of the actual combinations. You have to be a little bit careful about interaction effects with Taguchi testing. The Split Test Accelerator and Kaizentrack are Taguchi testing tools.

If you want to get around interaction effects you can with a full factorial tool. It will take more traffic to avoid the interactions — a lot more — but with a FFT, like Google's Website Optimizer, you can just see how every combination plays out, and live with the interactions.

Some multivariate tools randomly serve the options for each factor independently from what is served for the other factors. This can lead to more flexible test design, but also to more interaction effects and other statistical noise, leading to longer tests.

There are other distinctions and features worth noting. And each solution has a different mix of pros and cons. But this should get your oriented a bit before you make your decision.