13 Feb 12
10 Jun 12 6:05 pm
Sorry, I haven't watched that video. But I do have some general advice on A/B split testing: Unless you have a lot of traffic, A/B split tests take a very long time. So you want to test significant changes, even "radical redesigns", not trivial changes. Trivial changes usually show only small (or no) statistically meaningful results.
And please remember the true goal of split testing. You are NOT looking for nice numbers to justify your hunches. You are looking for "new learnings" about how your site works, which will lead you to meaningful actions which you previously would not have understood were necessary. This is why negative results can be even better than positive ones.
The test results are only the first step in the process of gaining new learnings. Next you must investigate "Why" those particular results happened and what "rules of thumb" or new "best practices" or other insights can be gained from the "how?, why?, what?, when?" about your results. It's a process of investigation and learning, not some kind of raw metrics to report to someone.
A/B split testing is a fundamental skill of "scientific marketing". It's actually a kind of scientific research.
Marketing hunches are great, but tested and understood hunches, leading to proven actions that advance your bottom line, are much better.
Hope this helps...