—Mark Twin, quoting Benjamin Disraeli
Using AdWords or other search engine CPC advertising program, you could test two or more ad copies for every ad group. Keep the winning ad copy after seeing the CTRs or conversion rate of them.
Questions to be Solved1) On ad copy one what confidence do you have?
In mathematics words,what is the real CTR of copy one at a 95% confidence level?
2) Ad copy one is better than copy two, what confidence do you have on this?
Is CTR difference between copy one and copy two significant?
Ad Copy Testing in Statistical LanguageRandom variable: ad copy being clicked
Stochastical process: ad copy being shown to searchers
Distribution function: 0-1, clicked or not
According to law of large numbers and the central limit theorem, CTR estimate that we observe for an ad copy will look like a normal distribution.
Before We Start the Test1.Keep the Same Traffic Source During Test
Do not change keywords in the tested ad group. No adding, deleting at all.
Here's another question,how about just contain one keyword in a group? This could be the best situation for testing when the keyword brings enough visits.
2.How Many Impression is Enough
50 times,according to statistic analysis, is large enough for normal distribution.
You do not need to worry about this problem since we usually choose to test ad groups whose traffic is large enough to worth our testing.
3.Keep the Ad Position
Different ad position means different CTRs.Look at this heat map for search result page.
Better keep the ad position. Here are some conditions you need to consider during testing phrase:
1) Higher competitior's bid
Correspond: set preferred ad position, or adjust your bid
2) Do not change bid yourself to change your ad position
4. How Long Should the Test Run?
Audience of your ad copy behaves differently at times, like weekend or work day, in the morning or evening, date before salary day or after that day.
Learn about your sales cycle, test at least one or two cycles.If you can not get your sales cycle, try to test at least one week.
Data Collection PhraseSet the ad rotation method to "rotate evenly" in Google AdWords. The following data in chart is artificial.
Answer to Question OneData we collected are observed data, we do not the real click through rate of one ad copy. What we can do is determine the range of real CTR from observed data. This would involve range estimation in statistics math.
Confidence range: potential range of real CTR
Confidence level: the rate of existance of real CTR in the confidence range
By using Excel statistic analysis, it's easy to get these inferential statistics. You can find these data in the following sections.
Answer to Question TwoWhether CTR two versions of ad copies are the same or significantly different?
From collected data above, we can see version one is better than version two, but are we 95% sure about this? Or in another way, is version two 95 out of 100 times better than version two?
Use Excel to Analyze These Two QuestionsAbout how to use the "statistic analysis" tool in Excel, try to search it yourself.
1) Use "descriptive analysis" to analyze question one
The following image shows the result of descriptive analysis on data above. There are many items, but we just care about some of them.
confidence interval = [average-level data, average+level data]
For ad copy one, at 95% rate the real CTR lies in [1.93, 2.19]
2) Use "T-test analysis" to answer question two
At first, you need to determine whether these two version's variance are the same or not.
Use "F-test analysis", here is the result for our testing. From it, we know the variance of the two ad copy version are the same.
Other Rotation MethodsThere are many other ad rotation methods using Google AdWords or other search engines.
Conclusion1) Test at least one week
2) Do not mix up significance and importance
Two versions' CTR may have a lot difference, but you need to significantly make sure about this.