Crawling and Indexing Location-Adaptive Pages for SEO

We’ve been seeing a lot of dramatic changes and updates to Google SEO lately, particularly with respect to location-based ranking and mobile results. And just recently, Google made another significant change by introducing improvements to Googlebot that enable crawling and indexing of location-adaptive pages for SEO. The update will come as particularly great news for businesses that cater to international audience or operate in multiple countries, and all without requiring changes to their websites.

Location-adaptive pages, also known as geotargeted sites, are designed to automatically adapt to a user’s specific location and make changes to the content based on the visitor’s geographic location and language settings. Geotargeting is especially useful for businesses looking to reach a global audience and improve their overall international customer experience. However, some of the ways in which location-adaptive pages are created can actually prevent search engines like Google from crawling and indexing a site, acting as a virtual barrier to visitors located outside of the United States. This unintended consequence has long proved an obstacle to sites looking to improve their organic search results and customer base beyond U.S. boarders, that is, until now.

Google’s recently introduced improvements seek to resolve this long-standing challenge by enabling Googlebot to detect when it is crawling these location-adaptive pages. The update allows Googlebot to understand this type of content in two ways:

• Geo-distributed crawling – This allows Googlebot to use IP addresses that are located both inside and outside of the United States (as opposed to ones that appear to be just in the U.S. as it previously did).
• Language dependent crawling – In this case Googlebot will start to crawl with an “Accept-Language” HTTP header in the request.

What this means is that if Googlebot detects one of the two configurations on a particular site it will be able to more thoroughly crawl and index that site. This clearly comes as excellent news to business owners who have been using location-adaptive pages, but finding that those particular pages weren’t ranking well, or for those who have considered using them, but were hesitant to implement them given the previous limitations. In either case, it’s important to check that your site configuration supports this location-aware crawling.

As of now, Googlebot considers a variety of signals and indicators to find out if a particular site uses location-specific content. Three of the ways are:

• Providing different content on the same URL as dictated by the visitor’s perceived geolocation.
• Offering different content on the same URL based upon the “Accept-Language” field set by the user’s language setting in the HTTP request header.
• When a website blocks access to requests from specific countries.

Many of the updates that Google has made recently have left many businesses and developers scrambling to adapt to new configurations or requirements. Fortunately, however, in this case, the changes will happen automatically and you won’t need to change your CMS configuration or server settings. These changes will likely prove beneficial to anyone looking to do more business internationally, reach a wider audience, and provide a better customer experience on their site.

The importance of still optimizing for location and language

While these changes are going to change how many sites are crawled and indexed, it’s important to realize that you should still be optimizing your site with language and location information. There are likely more related changes to come regarding geotargeting, but there’s still much that should still be to best position your site to be found and indexed. One consideration is that, so far, Google is the only search engine that has made these changes, so it’s still important to optimize for other engines as well. There’s also likely to be tweaks and improvements made to the new configurations, likely in the coming months.

Google still recommends that site owners should maintain separate URLs for different geographic locations and use rel-alternate-hreflang annotations for them. Google advises that this approach is best for the user, which is in turn what Google ultimately is looking for. As this is the early phase of this update, there is also no telling if every language or location that you are seeking to target will be covered by this nuanced Googlebot. Google and Bing also offer geographic targeting in their webmaster tools, which is an effective way to signal to their crawlers the language and location of a given site.

You can expect to learn more about this update in the coming month as more sites are indexed using the new configurations. There’s also a good chance adjustments may be made in the near future, so we’ll continue to keep an eye out and keep you updated. In the meantime, whether or not you’re using location-adaptive pages, it’s important to ensure that you’re optimizing your site with respect to Google’s most recent updates.

What has your experience been with location-adaptive pages? What techniques have you found to be most effective for reaching a global audience and international SEO? Let us know in the comment section below.

Sources

• “Crawling and indexing of locale-adaptive pages” (Google Webmaster Central Blog)
• “New: Google supports locale-aware content based on IP and Languages” (Search Engine Roundtable)
• “Google improves crawling and indexing of location-adaptive pages” (Search Engine Journal)
• “Google now supports crawling and indexing locale-adaptive webpages” (Search Engine Land)
• “Google improves crawling for geotargeted sites” (Practical Ecommerce)