Robots.txt Checker Tool

MERJ provides free tools to help monitor domains for changes that could affect visibility.

Setup your domain monitoring and let it run in the background for ease of mind.

Free Robots.txt Checker

We have already checked robots.txt files.

Slack

Email & Slack Alerts

Get instant notifications to your Slack team whenever robots.txt changes are discovered. Set the checks every 1 hour, 6 hours, 12 hours or 24 hours.

Request rechecking at any time, using the refresh feature.

Slack

Archiving Robots.txt Checks

You can archive up to 1,000 robots.txt checks, so you can go back and compare them at any time.

Timestamps

We provide an easy way for you to see text changed between each check.

Diff

Competitors & Prospects

Don't just monitor your own domain and assets. You can easily monitor competitor domains and prospects to see if and when they change their robots.txt file.

On-demand Checks

Developers can call the robot.txt checker API to automatically schedule a refresh when deploying new changes. Simply replace {{Site_ID}} with your Site ID:


curl -I https://merj.com/api/sites/{{Site_ID}}/robot-checks/refresh

Add any time delay to ensure the check appears after deployment has finished. Example of 20 second delay:


curl -I https://merj.com/api/sites/{{Site_ID}}/robot-checks/refresh?t=20

Monitor your robots.txt for free!

Join MERJ