...set the crawl-delay to specific user agents. For example, you may find that your site is frequently crawled by SEO tools, which could slow down your site. You may also wish to block them all together if you don’t feel they a... [article continues]
from
http://redirect.viglink.com?u=https%3A%2F%2Fwww.sitepoint.com%2Frobots-and-you-a-guide-to-robots-txt%2F&key=ddaed8f51db7bb1330a6f6de768a69b8
No comments:
Post a Comment