Good post and an interesting take on CRO.

You say: “As i pointed out earlier that the fundamental problem with ‘conversion rate is that it is a ratio metric. I can increase the conversion rate by decreasing the website traffic”.

As an analyst it is key to be able to place things into perspective. Just by looking at CRO from an holistic point of view might give you a wrong impression since the data in aggregate. Therefore I believe CRO works, but you should always try to segment your data into meaningful groups. For example: Non-members vs. Members, light/medium/heavy users, traffic source, etc.

Then again, I hear what you say and I agree that especially for e-commerce organizations you should not only focus on CRO. With one client we experienced just that: CRO increased but the EBITDA decreased because the AOV took a small punch in the face.

Keep in mind that with AOV you should keep on eye on AOV outliers as these can really impact your results. I guess there is a BUT behind every metric

p.s. I just found your blog today and look forward to reading and commenting more.

As i pointed out earlier that the fundamental problem with ‘conversion rate is that it is a ratio metric. So increase in conversion rate does not always means increase in transactions even if we keep AOV constant. I can increase the conversion rate by decreasing the website traffic.

AOV can be much more important than transaction volumes in increasing sales and gross profit. For example you can sell 10 items of $10 each and make $100 in revenue at the marketing cost of say $20 or you can sell 5 items of $100 each and make $500 in revenue at the same marketing cost of $20. In the end you can make more money by focusing on high price items depending upon your business model and profit margin.

Anyone who selects ‘conversion rate’ as a success metric for its test is knowingly or unknowingly advocating to look at conversion rate in isolation. In testing it is quite common to select a winner on the basis of conversion rate alone. Statisticians interpret ratios very carefully because they are well aware of their limitations. This is not always the case with marketers and analysts. There is no point focusing on improving the conversion rate if there is a negative correlation between conversion rate and AOV/transactions

You should watch conversion rates to make sure they’re not changing dramatically without reason, but otherwise trying too hard to overoptimize for CR can end up killing your brand equity (which drives long-term revenue). Good post.

When you run any split test there’s a chance that your challenger page won’t perform as well as the control – it’s the nature of experimenting based on hypotheses: you won’t always be right. What you’re suggesting requires me to convince each of my clients to expose my experiments/tests to AT LEAST half of their traffic, and if I wanted to test two challenger pages against a control simultaneously, splitting the traffic equally would mean that I’m exposing 66% of the traffic to challengers; this percentage obviously increases with the number of simultaneous tests. Whether your turning over a few hundred bucks or many millions in revenue, your suggestion carries too much risk.

Alternatively, I’d need to ask my clients to create a copy of the control page (or do it myself), and only run my experiments on the challenger(s) and the copy of the control.

Adopting a method of equal traffic to every test page just isn’t practical. If ratios are good enough for statisticians, then they’re good enough for me. CROs the world over are still capable of looking at metrics holistically and driving all important revenue increases without relabeling the discipline or changing it entirely. How many people advocate looking at conversion rate in isolation anyway?