Google just like any other search engine needs to send crawlers to your website in order to index the information.
Search engines can crawl your website too fast, and you can block search engines from crawling your website.
With Google, you typically want them to crawl your site, but at a reduced rate so they aren’t slamming your server.
If you don’t already have a Google account you should read our introduction to Google Webmaster Tools article.
Changing Google’s crawl rate in Webmaster Tools
Adjusting the crawl rate for Google won’t affect ranking, just how long it takes them to index your entire website.
If you had 1,000 pages on your website, a search engine could potentially index your entire site in a few minutes.
However this could cause high system resource usage with all of those pages loaded in a short time period.
A Crawl-delay: of 30 seconds would allow crawlers to index your entire 1,000 page website in just 8.3 hours
A Crawl-delay: of 500 seconds would allow crawlers to index your entire 1,000 page website in 5.8 days
Using the steps below you can adjust Google’s crawl rate for your website using Webmaster Tools:
Step 1
Log in to Google Webmaster Tools
Step 2
Click on the website you’d like to manage
Step 3
Click the top-right settings cog, then Site Settings
Step 4
Click on Limit Google’s maximum crawl rate
Adjust the crawl rate to at least 30 seconds apart.
Step 5
Crawl delay settings are only saved for 90 days.
Note down the date you’ll need to set them again
Now the next time Google sends their Googlebot crawler to index your website, they’ll be sure to space out their requests at least 30 seconds in-between each other.
Does it really affect the ranking?
This guide is not really for helping your site’s rank but to reduce the usage of your server. For example, if you have a large site and are receiving a large amount of traffic due to a promotion, increasing the crawl delay may reduce the resources your server is using and speed up your site.
Thank you,
John-Paul
Certainly useful post for crawl rate setting. Thanks
30 seconds per page x 1000 pages = 3000 seconds / 60 = 500 minutes /60 = 8.3 hours.
How to do this calculations?
A Crawl-delay: of 30 seconds would allow crawlers to index your entire 1,000 page website in just 8.3 hours
A Crawl-delay: of 500 seconds would allow crawlers to index your entire 1,000 page website in 5.8 days
bro, what do you sugesst, what will be the best crawl rate for a website?
Hello Suleman,
Thank you for contacting us. The best crawl rate will depend on several actors, such as the amount of resources your server has available, amount of pages/content on your site, frequency of website update.
I recommend trying different configurations, to see what is working for your specific sites.
Thank you,
John-Paul
Crawl-delay: 1
If your robots txt file shows this does it mean Google will not crawl the site?
Hello Brad,
The crawl-delay in robots.txt is pretty much ignored by Google. It does, however, obey the crawl delays set in the Webmaster tools.
Kindest Regards,
Scott M
The irony is that my site is hosted on Google App. Engine, so that’s money going for them too 🙂
Then again, most of my traffic comes from them as well.
What can we do against the almighty…
but bing have i think
I cannot change that setting; it says:
“Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate.”
On Feb 19 Google crawled more than 2 million pages in my site (taht cost more than $30) and now is starting to spike again.
What could I do?
Hello MdeA,
I’m sorry you’re getting that message. If you’re getting it and it’s coming from Google, you won’t be able to make a change unless you can contact them and find out how. We have no way to control what Google has decided to do for your crawl rate.
Regards,
Arnel C.
how can i set for this times ?
Hello Javadh,
You would not be able to enable the crawling during certain times of the day as Google does not function like that.
Best Regards
TJ Edens
i am posting every 30min from 13am to 23am and from 23am to 12 am have not post
Hello Javadh,
It should be crawling your website during those times.
TJ Edens
if i want google bot fast crawl my site how should i change setting ?
Hello Javadh,
If you do not want to limit Googles crawl rate then I would suggest selecting the optimal option and let Google automatically set its crawl rate.
Best Regards,
TJ Edens