Domain name split testing with ads

Choosing a new domain is usually a gut-driven decision. Maybe you’ll spend a few hours getting domain suggestions, check out a few Sedo auctions, build a shortlist, ask some coworkers for opinions, then pick one and hope for the best.

It doesn’t have to be that way.

Last week I wrote about how running a domain split testing experiment for my RV rental weekend project led me to a domain that performed 127% better than my personal favorite. Several readers asked for more details about the methodology so I’ve described it in detail below.

Methodology

What we’re going to do is set up several domains, write identical ads on Google AdWords, and then measure click through rates.

It’s important to test domains that are significantly different from each other. For example, if you were testing FriedChicken.io vs FriedChicken.co, you’d expect to need a very, very large pool of data before reaching statistical significance. On the other hand, if you were to test FriedChicken.io vs ChickenWorld.co, you’d probably figure out the winning domain quite quickly.

Step 1. Pick a landing page

It really doesn’t matter what page you send traffic to because you’re not trying to measure the performance of the landing page. At this point we’re only concerned with domain name performance. It is only after you’ve determined which domain is optimal that you should begin worrying about landing page pairings.

Step 2. Duplicate your site

In Rails-based sites, you can automatically serve up a site under any number of domains without any modifications to your settings. If you have hard-coded URLs through the site or are using a different framework, just do enough work to get each landing page to load without redirecting. Everything after that is irrelevant because, again, we’re only measuring ad clickthroughs.

Step 3. Create the ads

Create an Ad Group for each domain. Then write an ad for one domain and clone it to each of the Ad Groups, changing only the domain. You must use different Ad Groups because Google only allows one domain per Ad Group.

Step 4. Configure settings

Make sure that you are not “optimizing for performance” and that “rotate ads evenly” is turned on.

Step 5. Start the ads

Start the ads and measure just one variable: click through rate.

Step 6. Manage bids

You may see that one domain gets a few clicks and then Google slows down impressions on the other one. If that happens, reduce the bids on the higher performing domain and raise bids on the lower performing domain. Your goal is to make sure that both ads are running at the same time and at the same position. You’ll need to repeat this process daily since you’re trying to force Google’s ad rotator to do something it doesn’t want to do: show ads of different qualities evenly.

Step 7. Reach statistical significance

Eventually, through your bidding manipulation you’ll end up with one domain that, even though it’s at the same average rank as it’s challenger, costs significantly less per click and has a significantly higher clickthrough rate. That’s the domain you want to keep.