Many websites are moving to CNDs or Content Delivery Networks. These CDNs can really help page speed, which is excellent, but what a lot of people don’t know is they can also kill your Google crawl rate and in some cases your sales.
What is Issue with CDNs?
Over the past month, I have had 3 clients switch from normal hosting to a CDN system. Each time, their Google crawl rate (the ability for Google to crawl and index the website) dropped significantly. In addition, they saw drops in sales. In all cases, the large websites reverted back to the original hosting.
Apparently, they are not alone. Here are a few excerpts taken from the Google Webmaster Forums.
“About a month ago my crawl rate fell off a cliff, however, I was using a CDN called CloudFlare. Once I removed my site from that CDN my crawl rate went back to normal. Unfortunately, I went back to CF a few weeks later thinking that they had fixed the issue and my crawl rate fell again. I simply removed CF again thinking my crawl rate would recover like it did the first time but no luck so far. Have you ever heard of content delivery networks causing crawler problems?”
“Uh oh. Another CloudFlare issue. Do a search for that term on this forum. There was an issue here and several people complained. CloudFlare seems to have troubles with Google. I believe one guy said it was after an update, but TBH, I did not follow the thread closely, because I don’t use them and don’t know much about them.”
Industry Authorities Agree on CDN Issues with Crawl Rate
Search Engine Roundtable also reported on this. The article is very old, but it is clear some of these crawl rate CDN issues are still present today. Search Engine Roundtable actually has a direct quote from Google on this.
I am not saying all CDNs are bad for crawl rate. I am just saying people have issues all the time. If you see a drop, then check with your webmaster to see if you recently made any significant hosting or content delivery changes.