Web analytics: insights into the customer journey

THE POWER OF ONLINE STATISTICS

Statistics such as A/B testing, in which outcomes are compared for two different marketing options, present some special challenges for web analytics because of their real-time nature. Traditional statistical analyses work with a fixed set of data and a stated confidence level that an effect is real and not happening by chance. When data is coming in constantly and being analyzed in real time, the results may go above or below statistical significance as time goes by. Therefore it is difficult to say for sure whether an effect has been achieved.

A company wanting to determine the impact of different content strategies on conversion can estimate the sample size required for a statistically significant result given data about the current (baseline) conversion rate and the expected or hoped-for lift. That approach, however, may require a large sample size to detect a small effect. In addition, if the users are trying to detect a large effect, statistical significance may not be reached, so smaller but relevant effects from the experiment may be overlooked. Another pitfall is testing for multiple effects on the same sample. That action can produce a false positive because the likelihood of finding an effect from a set of tests increases when multiple variations are tested simultaneously.

Optimizely, which offers an experimentation platform, uses a technique called sequential sampling. Powered by Optimizely’s statistics engine, it provides real-time results without requiring setting a sample size or level of effect in advance. Statistical significance increases as more data is collected. Another technique used by the Optimizely platform corrects for the number of goals and variations being tested, reducing the likelihood of false positives.

Source: A Practical Guide to Statistics for Online Experiments, provided by Optimizely