User and Social Experience Design
The important thing about A/B testing is that there is a single variable that changes across the test samples. Not a dozen.
A/B testing is a well known method for comparing different test samples against a baseline. The important part of A/B testing is that there is a single variable present in all samples that is varied for each sample. Just one.
A/B testing is typically used for search engine optimisation, particularly around tweaking online ad campaigns to determine which keywords or ad design performs better; ie results in higher conversation rates.
So if the performance factor for test sample A is 40 and the factor for test sample B is 20 then A performed better than B and thus conclude that the one variable that was different in A resulted in that performance increase, be it a keyword, a colour, a position etc.
A/B testing is also used in user testing to test user interfaces and determine if one feature performs better over another – ie provides a better user experience, is easier to understand and use, more accessible etc.
Once again the important thing is to change just one thing across the test samples.
What I sometimes see is people taking one variable from a design that varies in dozens of ways from another design and attempting to conclude that just that one variable is responsible for the better response from users as opposed to the other differentiations like colour, position, layout, navigation, instructional text, responsiveness etc.
Just try the algebra, it doesn’t work:
If ABC > XYZ, how can you conclude that A > X? It’s entirely possible that A < X and is just propped up by other variables that do much better.
I know this is a fundamental principle of integrity in research but I don’t think people really consider what we do as proper “serious” research.
// purecaffeine.com is a user interaction and UX design, social media and Government 2.0 blog run by professional Canberra, Australia web user interaction designer Nathanael Boehm, licensed under a Creative Commons Attribution-Noncommercial-Share Alike 2.5 Australia License.
