Googlebot mobile was the name of the crawler Google previously used to crawl and index mobile content on the web. This worked well, however, there were some issue with people’s understanding of how best to utilize the tool.
“We’ve seen cases where a webmaster inadvertently blocked smartphone crawling or indexing when they really meant to block just feature phone crawling or indexing. This ambiguity made it impossible for Google to index smartphone content of some sites, or for Google to recognize that these sites are smartphone-optimized.”
New Googlebot for Smartphones
Google wanted to clarify how this really worked, while at the same time giving webmasters a greater level of control. Because of this, Google will be putting down the good old Google-Mobile user agent in late February 2014.
“From then on, the user-agent for smartphones will identify itself simply as “Googlebot” but will still list “mobile” elsewhere in the user-agent string.”
Old and New User-Agents
The new Googlebot for smartphones user-agent:
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
The Googlebot-Mobile for smartphones user-agent we will be retiring soon:
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
It is important to reiterate that this change only influences Googlebot-Mobile for smartphones, Googlebot will not be affected.
“The remaining two Googlebot-Mobile crawlers will continue to refer to feature phone devices in their user-agent strings; for reference, these are:”
Regular Googlebot user-agent:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
The two Googlebot-Mobile user-agents for feature phones:
- SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
- DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Google notes that you can test your configuration by using the Fetch as Google feature in Webmaster Tools. They also continue to update their list of existing crawlers in the Help Center.
Summing it Up
So what does this mean for you? First, please note it is a little confusing I had to read the article Google put out a few times to understand it.
It means that Google has basically made this change so that they can index more of the web. This will allow them to start crawling more content that is intended for smart phones right away with their standard Googlebot. The Googlebot Mobile crawlers and user agent directives will still exist, but they will only be related to stupid phones, or feature rich phones. Ultimately, this is a good move because smart phones are increasing in their browsing capabilities and can generally do a decent job viewing content.
Google states that the new Googlebot crawler will follow direction from robots.txt, robots meta tag, and HTTP header directives for Googlebot instead of Googlebot-Mobile.
As an example, this will now block smartphone user agents and regular.
User-agent: Googlebot Disallow: /
And now this robots.txt directive will block crawling by Google’s feature phone crawlers:
User-agent: Googlebot-Mobile Disallow: /
This is an important change that will cause at least one of Ignite Visibility’s clients to change their configuration. Thanks for making more work for us Google 🙂
