Join Merj - We're Hiring

Robots.txt Checker Tool

MERJ provides free tools to help monitor domains for changes that could affect visibility. Setup your domain monitoring and let it run in the background for ease of mind.

We have already checked robots.txt files.


Email & Slack Alerts

Get instant notifications to your Slack team whenever robots.txt changes are discovered. Set the checks every 1 hour, 6 hours, 12 hours or 24 hours.

Request rechecking at any time, using the refresh feature.


Archiving Robots.txt Checks

You can archive up to 1,000 robots.txt checks, so you can go back and compare them at any time.


We provide an easy way for you to see text changed between each check.


Competitors & Prospects

Don't just monitor your own domain and assets. You can easily monitor competitor domains and prospects to see if and when they change their robots.txt file.

On-demand Checks

Call the robot.txt checker API to automatically schedule a recheck when deploying new changes. Simply replace {{Site_ID}} with your Site ID:

curl -I{{Site_ID}}/robot-checks/recheck

Add any time delay to ensure the check appears after deployment has finished. Example of 20 second delay:

curl -I{{Site_ID}}/robot-checks/recheck?t=20

Monitor your robots.txt for free!

Sign up