Have You Bing Crawled?

A post on the Live blog goes into discussion of the MSNBot and crawl delay.

My professional opinion leans toward not setting a crawl rate, at all. SEs will gauge the ‘crawlability’ of your site and will crawl at a frequency that is best suited for that site.

However, I do find it rather useful to have dynamic pages, such as news, blogs, and the like, to a more frequent setting so that the SEs are aware of and can find new, fresh content.

Crawl delay and the Bing crawler, MSNBot

Should you set a crawl delay?

Many factors affect the crawling of a site, including (but not limited to):

* The total number of pages on a site (is the site small, large, or somewhere in-between?)

* The size of the content (PDFs and Microsoft Office files are typically much larger than regular HTML files)

* The freshness of the content (how often is content added/removed/changed?)

* The number of allowed concurrent connections (a function of the web server infrastructure)

* The bandwidth of the site (a function of the host’s service provider; the lower the bandwidth, the lower the server’s capacity to serve page requests)

* How highly does the site rank (content judged as not relevant won’t be crawled as often as highly relevant content)

The rate at which a site is crawled is an amalgam of all of those factors and more.

via Bing – Crawl delay and the Bing crawler, MSNBot – Webmaster Blog – Bing Community.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: