Merjbot

Last updated: 20 April 2026

Why Merjbot may have crawled your site

We collect data for a variety of different reasons.

It might be that we’re on one of our quests to conduct original and thought-provoking research. We often do this in order to help find interesting solutions to some of the technical community’s most difficult and problematic issues.

Client

You are a client of ours and we are working with your internal teams.

Competitor intelligence

We are working for one of your competitors and are conducting competitive intelligence.

Data collection

We are collecting market data for our reports, legal cases, investor relations or due digital diligence.

User agent strings

The full user agent strings we use for Merjbot are:

1Mozilla/5.0 (compatible; Merjbot/1.0; +https://merj.com/bot)
2Mozilla/5.0 (compatible; Googlebot/2.1; +https://merj.com/bot)
3Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +https://merj.com/bot)

Like many web crawlers, Merjbot obeys robots.txt files, including disallow and allow rules, unless our research requires us not to.

Controlling Merjbot on your website can be done a variety of ways.

Blocking Merjbot from your site

If you would prefer that the Merjbot does not visit your site, you can block it by using your robot.txt file, by adding the following lines:

1User-agent: Merjbot
2Disallow: /

Bot rules

Although we respect robots.txt where possible, if you would like to stop Merjbot from crawling certain pages or areas of your site, you can tell Merjbot not to crawl pages or subdirectories. For example:

1User-agent: Merjbot
2Disallow: /comments/ # Block all comments

Rate limiting Merjbot

You can also slow down Merjbot so that it crawls your site at a slower rate. Control this by entering the Crawl-Delay rule. For example, here’s what you’d use for a 15 second crawl speed:

1User-agent: Merjbot
2Crawl-delay : 15 # seconds

How many web pages can Merjbot crawl?

Merjbot is built on our distributed scaling language, with the capacity to crawl and process approximately one billion web pages per day (11,500 requests per second). We scale our resources according to client or research requirements.

FAQs

Why is Merjbot crawling our sites?

We may be providing your company with various data insights. Alternatively, we could be running competitor analysis or industry research

Does Merjbot have a fixed IP address?

The crawlers use both fixed and floating IP addresses. We can direct fixed IP addresses at specific sites to help teams manage Merjbot through firewall routing and safelisting bot protection.