SEM Ad Copy Testing Statistic Analysis

“There are three kinds of numbers: lies, damned lies, and statistics”
—Mark Twin, quoting Benjamin Disraeli

Using AdWords or other search engine CPC advertising program, you could test two or more ad copies for every ad group. Keep the winning ad copy after seeing the CTRs or conversion rate of them.

Questions to be Solved

1) On ad copy one what confidence do you have?
In mathematics words,what is the real CTR of copy one at a 95% confidence level?

2) Ad copy one is better than copy two, what confidence do you have on this?
Is CTR difference between copy one and copy two significant?

Ad Copy Testing in Statistical Language

Random variable: ad copy being clicked
Stochastical process: ad copy being shown to searchers
Distribution function: 0-1, clicked or not

According to law of large numbers and the central limit theorem, CTR estimate that we observe for an ad copy will look like a normal distribution.

Before We Start the Test

1.Keep the Same Traffic Source During Test
Do not change keywords in the tested ad group. No adding, deleting at all.

Here's another question,how about just contain one keyword in a group? This could be the best situation for testing when the keyword brings enough visits.

2.How Many Impression is Enough
50 times,according to statistic analysis, is large enough for normal distribution.

You do not need to worry about this problem since we usually choose to test ad groups whose traffic is large enough to worth our testing.

3.Keep the Ad Position
Different ad position means different CTRs.Look at this heat map for search result page.

Better keep the ad position. Here are some conditions you need to consider during testing phrase:

1) Higher competitior's bid
Correspond: set preferred ad position, or adjust your bid

2) Do not change bid yourself to change your ad position

4. How Long Should the Test Run?
Audience of your ad copy behaves differently at times, like weekend or work day, in the morning or evening, date before salary day or after that day.

Learn about your sales cycle, test at least one or two cycles.If you can not get your sales cycle, try to test at least one week.

Data Collection Phrase

Set the ad rotation method to "rotate evenly" in Google AdWords. The following data in chart is artificial.

Answer to Question One

Data we collected are observed data, we do not the real click through rate of one ad copy. What we can do is determine the range of real CTR from observed data. This would involve range estimation in statistics math.

Confidence range: potential range of real CTR
Confidence level: the rate of existance of real CTR in the confidence range

By using Excel statistic analysis, it's easy to get these inferential statistics. You can find these data in the following sections.

Answer to Question Two

Whether CTR two versions of ad copies are the same or significantly different?

From collected data above, we can see version one is better than version two, but are we 95% sure about this? Or in another way, is version two 95 out of 100 times better than version two?

Use Excel to Analyze These Two Questions

About how to use the "statistic analysis" tool in Excel, try to search it yourself.

1) Use "descriptive analysis" to analyze question one

The following image shows the result of descriptive analysis on data above. There are many items, but we just care about some of them.

From data above, we can calculate the confidence range at 95% confidence level
confidence interval = [average-level data, average+level data]

For ad copy one, at 95% rate the real CTR lies in [1.93, 2.19]

2) Use "T-test analysis" to answer question two

At first, you need to determine whether these two version's variance are the same or not.

Use "F-test analysis", here is the result for our testing. From it, we know the variance of the two ad copy version are the same.

Then use "T-test analysis for identical variance" to find out whether CTR of the two ad copies are the same or significantly different.

Other Rotation Methods

There are many other ad rotation methods using Google AdWords or other search engines.

Google is using multi-armed bandit in conversion optimizer and Google analytics "content experiment". To learn more about multi-armed bandit, search for it or go to analytics blog

Conclusion

1) Test at least one week
2) Do not mix up significance and importance
Two versions' CTR may have a lot difference, but you need to significantly make sure about this.

Online Buy Funnel and Corresponding Ads

"All Life is Problem Solving"
--Karl Popper

PPC Ads is about Solving Searcher's Question

PPC Ads is literally advertising, and advertising is about understanding and connecting with people.

Online ads make knowing audience better possible. In traditional ads channels, advertisers push their ads to audience, shout out what they have, "look at we i can offer you". Online ads change this situation, when searcher search one keyword, we almost know what they are looking for, what questions they have in mind; when they are visiting one website, we may know what they are visiting for.

The more you know your audience, the better conversion you may get. After understanding your audience's intention, answer their questions, they are more likely to take actions you want them to do.

Search query gives us a good chance to understand the audience. A searcher who searches "which phone is better" definitely has different intention and question than a searcher who searches "iphone 5 price". The previous one may be thinking about features of phones, which feature i perfer most. The latter may want to know price of iphone 5 and where can i buy one at a relatively lower price.

In Buying Funnels, People is Visiting with Different Questions

Search queries could be allocated into different buying funnels. A complete buying funnels includes awareness, interest, desire, action, satisfaction phases(AIDAS). A search query may belong to one or more buying funnels. The search queries above, "which phone is better" is in the awareness phase, while "iphone 5 price" is in the desire phase.

Phase 1 Awareness

Search Query in this phase are product types and "what" question on products

Questions in searchers mind: what is a specific product
Ads you should offer: ads that answer searchers questions directly

Phase 2 Interests

Search Query in this phase: about product of one brand, features of the products

Questions in searchers mind: how well is product of that brand, features
Ads you should offer: ads shows features and benefits of your products

Phase 3 Desire

Search Query in this phase: about price and comparison of products of different brands

Questions in searchers mind: which brand is the best, good price, reviews of this brand
Ads you should offer: ads shows benefits, your advantages

Phase 4 Action

Search Query in this phase: about where and how to buy these products

Questions in searchers mind: where offers this product and is there any discount
Ads you should offer: featuers of your products and discount at this time

Phase 5 Satisfation

Search Query in this phase: about after-sales service, how to set up products

Questions in searchers mind: how can make this products better, how to fix it, what's their toll free number
Ads you should offer: ads shows your service and numbers or your official website

What Ads to Show on Display Network

Display network visitors are totally different than searcher. We don't know what they are searching for exactly. But with what they are visiting and seeing, we know the theme of the website and almost know what the visitors are looking for and what questions they have.

Visitors are in the awareness and interests phase. Most visitors on display network are in the first two buying funnel. So ads we show to them should be about our products' features and benefits. And this also explains why ads for action phase does not work well on display network.

Conclusion

We show ads to searchers in different buying funnels with respective content. The landing page in turn should show more and deeper information our audience want. Answer their question and solve their problem, audience becomes our customer without question.

Heuristic Evalution Usability Testing - Better Than Never

"Testing one use is 100% better than testing none; testing one user early in the project is better than testing 50 near the end."
--Steve Krug, Don't Make Me Think

A few days ago, when i'm searching about Bayes Theroem, i did a heuristic evalution on a website selling professional photo editing service.

Here i sumed it up to share with all of you.

Tested site: tucia.com
Tester: i myself.
Goal: place an order as a new visitor

Steps to Place the Order and Problems

Steps are not recorded in details. Major problems and steps are listed in order as below.

1.Home Page
Logo and slogan should tell visitors what this website is about and specialized in.
There are a lot of identical photo editing software, but this website provides manual editing service, which seems more professional and worth its price.

2. Click "Get Start", registration comes
All right, it's been a lot time since websites allow orders without registration.
"I have not known you and your service, why i have to register and give you my information". This is major problem.

3. After registration, enter the product page
A large paragraph of text. By the way the tabs are not designed good.
We don't read, we scan.

4. Click one of the pictures to enter the detail page
This step could be combined with step 3.

5. Click "i want it too" to enter Upload page
Another big text block.
After upload success, buttons should become to "add more" and "next".
Here comes $8=1 credit the first time, but it is not explained what the credit is used for.

6. Click "finish" to enter Note page
Here comes the "add" button,but it is not a formal button.
When click "edit", the "Order Service" disappears, but two "submit" buttons appear.

7. "Submit" and choose price
Another big problem. After i chose "i want it too", it asked me to choose price again. Am i authorized to price this product?

8. "Submit" again
A float window appears to ask me to change my delivery time. All right, i'm asked to define their product again. First price in step 7, then the delivery time.
Credit appears 2nd time, but still without any note about what it is for.

9. "Submit", then the payment method page
A table of what i choose should have been shown on this page.

10. Click to pay, then we treat it as a complete order.

If we were doing lab usability testing, you'd see and feel more frustration i had during this test.

Sum Up

Here listed major problems during this testing, in order of trouble degree.
a. Contraversy pricing. Visitors choose product and price in step 3 and again in step 7.
b. Visitors are asked to register before purchase. This is not a question few years ago.
c. Visitors are prevented to placing the order. Read big text blocks, add notes, choose delivery time.
d. Site navigation is mixed up with user tools. Upload, project should be included in My account category, not in the site navigation.
f. Button, tabs are not conventional ones.

From this casual testing, you see that doing a evalution is very useful. You may not have a regular testing schedule, but to ask your friend or staff to test your site at some times brings impact on your business.

Remember best practices and check out your site. Links, buttons, tabs, there are conventional styles for these elements. You do not need to invent wheel.

Make sure elements have consistent style. By doing this, you save visitors time to learn how your website work.


Conclusion:
Do heuristic evalution every month really help you out of usability problems.
Every page on your site should have enough reason to appear, if not, go delete them.