A post on the Live blog goes into discussion of the MSNBot and crawl delay.
My professional opinion leans toward not setting a crawl rate, at all. SEs will gauge the ‘crawlability’ of your site and will crawl at a frequency that is best suited for that site.
However, I do find it rather useful to have dynamic pages, such as news, blogs, and the like, to a more frequent setting so that the SEs are aware of and can find new, fresh content.
Crawl delay and the Bing crawler, MSNBot
Should you set a crawl delay?
Many factors affect the crawling of a site, including (but not limited to):
* The total number of pages on a site (is the site small, large, or somewhere in-between?)
* The size of the content (PDFs and Microsoft Office files are typically much larger than regular HTML files)
* The freshness of the content (how often is content added/removed/changed?)
* The number of allowed concurrent connections (a function of the web server infrastructure)
* The bandwidth of the site (a function of the host’s service provider; the lower the bandwidth, the lower the server’s capacity to serve page requests)
* How highly does the site rank (content judged as not relevant won’t be crawled as often as highly relevant content)
The rate at which a site is crawled is an amalgam of all of those factors and more.