Description
If you visit your website root URL and add /robots.txt on the end, you'll find our default compilation of bot blocks and rate limiting. The goal with this default is to ensure bots don't crawl irrelevant dynamic URLs, thus ensuring resources like CPU and RAM are not wasted on bots crawling the site unnecessarily. Bot crawls are an extremely common source of resource overusage on other hosts, so this is one component to our bot protection that helps ensure bots aren't a problem for your sites' performance.
Details & Overriding
If you need to use your own custom robots.txt, you can simply create one in your website root directory and it will override ours.
Two things to note about this:
- If you're on shared hosting and we find that bots are accessing dynamic processed resources unnecessarily for your site with your custom robots.txt, which our robots.txt would have blocked, we may remove it in favour of ours. Therefore we strongly recommend copying the contents of our existing robots.txt into your replacement before then adding your own rules to it.
- You will not see an actual robots.txt file anywhere in your web root -- this 'document' is provided by the web server engine configuration rather than through an actual file.
---
