Robots.txt Tester
Validate your crawling directives instantly
Crawler Control
Manage how search engine bots perceive and navigate your website structure.
Data Privacy
Identify and block sensitive directories (like /admin or /private) from appearing in search results.
Regex & Wildcards
Our tester supports advanced matching patterns including * (wildcard) and $ (end anchor).
Instant Feedback
Real-time validation of your rules without needing to upload the file to your server first.
Understanding Robots.txt Rules
The `robots.txt` file is the first thing a bot looks for when visiting your site. It contains a set of "directives" that tell specific crawlers which parts of your website they should not visit. While most bots follow these rules, they are not a security measure—they are more like a "No Trespassing" sign.
Using our Robots.txt Tester, you can debug complex rulesets and ensure that important pages aren't accidentally blocked from being indexed by Google or Bing.
Common Directives
- User-agent: Specifies which crawler the following rules apply to (e.g., `*` for all, `Googlebot`).
- Disallow: Specifies a path that should NOT be crawled.
- Allow: Specifically overrides a Disallow rule for a sub-path.
- Sitemap: Provides the full URL to your XML sitemap.
Frequently Asked Questions
- Paste the full contents of your robots.txt file into the text box, or use the sample rules already loaded.
- Enter the crawler you want to check, such as Googlebot or an asterisk to test all bots.
- Enter the URL path you want to verify, for example /blog/my-post.
- Click Test Directive to see whether that path is allowed or disallowed for that user-agent.
- Review the Matching Rule field to see exactly which line in your file produced the result.
A robots.txt file is a plain-text list of directives grouped under User-agent headers. When a crawler visits your site, it requests /robots.txt first and reads the group of rules that matches its own user-agent name, falling back to the wildcard group if no specific match exists. Within that group, the checker compares your requested path against every Allow and Disallow rule and applies the most specific (longest) matching path, using the length of the matched pattern to break ties. Wildcards like an asterisk and end-anchors like a dollar sign let a single directive match many URLs at once, which is why testing a rule directly is safer than reading it by eye.
Disallow tells a crawler not to request a matching path, while Allow is used to carve out an exception inside a broader Disallow rule, such as permitting one file inside an otherwise blocked folder. If a path matches both an Allow and a Disallow rule, most crawlers use whichever rule has the more specific (longer) path pattern, regardless of the order the lines appear in the file.
No. Robots.txt is a voluntary instruction for well-behaved crawlers, not an access control mechanism. It cannot password-protect a page, and malicious bots frequently ignore it entirely. Anyone can view your robots.txt file directly, so listing sensitive paths there can actually advertise their existence. Use authentication, a noindex meta tag, or server-level access rules to truly protect or hide content.
A broad rule like Disallow: /assets/ or Disallow: /*.js written to block old admin scripts can unintentionally catch the CSS and JavaScript files your pages need to render. When Google cannot fetch those resources, it may index a broken-looking version of your page and misjudge mobile usability. Always test key asset paths, such as /static/style.css, after editing your rules.
Robots.txt must be placed at the root of your domain, such as https://example.com/robots.txt, and it only applies to that exact host, protocol, and port. A file at /blog/robots.txt or on a different subdomain has no effect on the rest of your site. Each subdomain that needs crawl rules requires its own robots.txt file at its own root.