Robots.txt Tester
FreeTest robots.txt rules for specific user agents and URLs
/admin/users → ❌ BLOCKED Matched group: User-agent: * (no specific "Googlebot" group — using "*") Matched rule: Disallow: /admin
About Robots.txt Tester
A single misplaced Disallow line in robots.txt can deindex an entire section of a site, and the syntax is unforgiving about precedence and wildcards. Robots.txt Tester lets you check a specific URL against a specific user agent and tells you whether crawling is allowed or blocked, so you can confirm intent instead of guessing how the rules combine.
Provide the robots rules, a target path, and a user agent, and the tool evaluates the matching directives the way a crawler would, right in your browser. It is free, no signup needed, and gives an instant allow-or-block verdict. Use it to sanity-check changes before they ship, so you never accidentally hide pages from search engines.
Common uses
- Verify that your /admin or /api paths are blocked while public pages stay crawlable
- Test how Googlebot versus a generic bot is treated by your user-agent-specific rules
- Confirm a new Disallow rule doesn't accidentally catch URLs you want indexed
- Debug why a page isn't appearing in search by checking if robots.txt is blocking it
How to use
- 1
Configure options
Set your preferences using the options panel below.
- 2
Click Generate
Click the process button to generate your result instantly.
- 3
Copy or download
Copy the result to clipboard or download it directly to your device.
Why ToolHub?
- Instant SEO audit
- Social preview included
- Google-aligned checks
FAQ
Does it test the live robots.txt or rules I paste?
You evaluate a target URL against a given set of rules and user agent, so you can test both your current file and proposed changes.
Does robots.txt guarantee a page won't be indexed?
No. It controls crawling, not indexing. A blocked URL can still appear in results if linked elsewhere; use a noindex meta tag to prevent indexing.
How are wildcards and precedence handled?
It applies standard matching rules, evaluating the most specific applicable directive for the given user agent, mirroring how major crawlers interpret the file.
Can I use Robots.txt Tester on mobile?
Yes — ToolHub is fully responsive and works on iPhone, Android, tablets and all modern browsers. No app install needed.