User Agent Strings
The full user agent strings we use for Merjbot are:
Mozilla/5.0: (compatible; Merjbot/1.0; +https://merj.com/bot)
Mozilla/5.0: (compatible; Googlebot/2.1; +https://merj.com/bot)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +https://merj.com/bot)
Like many web crawlers, Merjbot obeys robots.txt files, including disallow and allow rules, unless our research requires us not to.
Controlling Merjbot on your website can be done a variety of ways.
Blocking Merjbot from your site
If you would prefer that the Merjbot does not visit your site, you can block it by using your robot.txt file, by adding the following lines:
Although we respect robots.txt where possible, if you would like to stop Merjbot from crawling certain pages or areas of your site, you can tell Merjbot not to crawl pages or subdirectories. For example:
Disallow: /comments/ # Block all comments
Rate Limiting Merjbot
You can also slow down Merjbot so that it crawls your site at a slower rate. Control this by entering the Crawl-Delay rule. For example, here's what you'd use for a 15 second crawl speed:
Crawl-delay : 15 # seconds