Google Exposed! Don’t Trust Webmaster Tools Reporting!

January 25th, 2011 by Search Influence Alumni

"The Simpsons" © 20th Century Fox Television (Don't Sue Me!)

In the middle of December 2011, Google rolled out some new changes to the Webmaster Tools. This update included search queries with top pages, organic CTR, and organic imprressions, as well as, intergrated graphs ala Google Analytics.

Since this update went live, I’ve been wondering if the data that Google is providing is accurate and how does it stand up against Google Adwords Keyword Tool and actual Adwords data.

We have a client who is position 1 for a cosmetic surgery term in a major metropolitan area. So I pulled the two sets of Google data reported and the results were mystifying.

Adwords Keyword Tool:

Click to Enlarge

Click to Enlarge

Keyword Tool reports there are an average of 2,900 local monthly searches for the phrase geo-modified cosmetic surgery.

Google Webmaster Tools:

Click to Enlarge

Click to Enlarge

Webmaster tool reported a mere 140 impressions! That’s a difference of 27k searches! Granted this data represents the holiday season in the US but you can not convince me that 96% of the searches stop. 10 or 15% sure, people have other things to think about during the holidays than getting bigger breast, but 96%! No %&*#ing way. Especially for a client that is in position one for the phrase.

And yes before you say it is an “approximate 12-month average number of user queries”, that would mean two out of the remaining twelve months would have to had double the search queries of the Google Adwords Keyword Tool number (go do the math, I’ll wait…).

Oh and to convince myself this isn’t some isolated event in Google’s data reporting, I pulled some additional information on another client who ranked in position one and was running Google Adwords ads during the same time frame that Webmaster tool reported on.

Adwords Keyword Tool:

Click to Enlarge

Click to Enlarge

Google’s Keyword Tool reports that the local monthly searches are 1,300…

Google Webmaster Tools:

Click to Enlarge

Click to Enlarge

And Webmaster Tools reported 140 impressions from Dec 22 – Jan 22. A big difference than what the Adwords Keyword Tool is reporting. But wait! The data looks even more like jelly when you compare Adwords  to Google Webmaster Tool for the same date range (oh and to clarify this client has been strong at position 1 for over a year for this term).

Google Adwords:

Click to Enlarge

Click to Enlarge

For the same exact date range and same exact keyword running simultaneously for the same location (this is a geo-modified keyword for a national level audience) the data is off by almost 50%! That’s failing (terribly) where I come from.

Seeing this data together is actually disheartening.  If I was using Google tools to perform an  A/B test and got a 50% accuracy in the data I would throw away the test. I understand Google is giving away ( most of) the data for free but there really should be some quality assurance for the Google Adwords Keyword tool. This tool should be giving advertisers the most correct data possible in order to help them create more effective campaigns and better gauge potential market growth.

At the end of the day, I wouldn’t trust Google Webmaster Tools as an authoritative source of search volume for your site. While it is collecting great data about indexing issues and missing pages, it is doesn’t give you accurate enough information to make a decision. I guess in a way, the three sources are like credit scores. If you just look at the one, you miss some important knowledge that could either be a boon or a bust.