When to declare a "dead heat" when testing ads?

So I do a bit of online advertising for different websites. I always alternate between two slightly different ads to see which on gets the higher clickthrough rate (# of times clicked divided by # times displayed).

I always test ads until the difference is 95% significant, but some times ads just perform about the same. I am trying to figure out after how many clicks you can say that the long term difference will be negible.

I want to come up with a formula that will tell me this: "If after X clicks the difference in CTR between both ad variations still hasn't become 95% significant, the long term difference will be y or less."

I took a bunch of statistics years ago, but haven't used it in a while, so I've been failing miserably with this. I write a blog about internet marketing and ad testing, and I will mention the results in the blog if I anyone can help me come up with one. I'll be happy to give credit in the blog to anyone who helps me answer this question, and I'll link back to this forum and to your website if you have one.

Being able to write an excel function to answer this question would save me a LOT of time :D


New Member
What test do you use to test the null hypothesis?
Are the samples independent (so that both ads never are displayed at the same time)?

Hey sorry it took me so long to reply, but it said I didn't have permission to post for whatever reason...

so the ads are never displayed at the same time. Each visitor will randomly see either ad A or ad B, and they get 'stamped' with a cookie, so every time they return they will see whatever ad was displayed the first time ( so a user who sees ad A can refresh the page and wills till only see ad A).

I use this free tool to test significance

Last edited: