Set crawl rules without blocking useful pages.
What it does
Produces recommended robots.txt rules and compliance checks for controlling crawler access.
When to use it
Use when the user wants to configure, audit, or optimize robots.txt.
How to use it
Give it a site URL and available context; it returns recommended robots.txt content, a compliance checklist, and current-state findings when auditing.
What you provide
- the site URL
Latest review
What you need · 0
No setup requirements listed.
Access · 1
This skill
AI crawler vendor documentation
Read
Checks AI crawler vendor docs