Robots.txt Tester
Test and validate robots.txt files for crawling permissions. Check if specific URLs are allowed or disallowed for search engine bots and web crawlers. Essential for SEO optimization.
Fetch Robots.txt from URL
Robots.txt Content
Test Configuration
Robots.txt Best Practices:
- Place robots.txt file in the root directory of your website (yoursite.com/robots.txt)
- Use specific user-agent names for better control over different bots
- Include sitemap URLs to help search engines discover your content
- Be careful with wildcard (*) rules as they apply to all unspecified bots
- Test your robots.txt file regularly, especially after website changes
- Remember that robots.txt is publicly accessible and not a security measure
- Use crawl-delay sparingly as it can slow down indexing
About Robots.txt Tester
A comprehensive robots.txt testing and validation tool that helps website owners, SEO professionals, and developers verify crawling permissions for search engine bots. Test specific URLs against robots.txt rules to ensure proper indexing and crawling behavior.
Why use a Robots.txt Tester?
Proper robots.txt configuration is crucial for SEO and website performance. This tool helps you validate that your robots.txt file correctly allows or blocks specific pages from being crawled by search engines, preventing indexing issues and ensuring optimal search engine visibility.
Who is it for?
Perfect for SEO specialists optimizing website crawling, web developers implementing robots.txt files, digital marketers managing search engine indexing, and website administrators ensuring proper bot access control. Ideal for anyone responsible for technical SEO and website crawling policies.
How to use the tool
Paste your robots.txt content or enter a website URL to fetch it automatically
Choose the user agent (search engine bot) you want to test against from the list
Enter the URL path or full URL you want to check for crawling permissions
View detailed analysis of whether the URL is allowed or disallowed
Review the syntax warnings and the parsed rules, sitemaps and crawl delay
Frequently Asked Questions
How do I test a robots.txt file?
Enter the site URL (the tool fetches `/robots.txt` from the domain) or paste the file's contents, then enter a path or full URL to test and pick a user agent. The tool parses the robots.txt syntax following RFC 9309 (longest matching rule wins, `*` and `$` wildcards) and reports: which user-agents are configured, which paths are blocked, whether your test path is blocked or allowed, the sitemap URL (if declared), and any syntax warnings. Useful for: pre-deployment validation, debugging unexpected crawl behavior, audit before launching a new site.
What is robots.txt?
`robots.txt` is a plain-text file at the root of your domain (`https://example.com/robots.txt`) that tells well-behaved web crawlers what to fetch and what to skip. It's defined by the Robots Exclusion Protocol (REP, now RFC 9309). Syntax: `User-agent: Googlebot` followed by `Disallow: /admin/` and `Allow: /admin/public/` rules. Also: `Sitemap: https://example.com/sitemap.xml`. Honored by Google, Bing, and most commercial crawlers; ignored by malicious bots. Place at the domain root; subpath robots.txt files are NOT used.
Is robots.txt a security mechanism?
**No — robots.txt is NOT security.** Three critical caveats. (1) **It's public**: anyone can read `/robots.txt`, so listing 'Disallow: /admin/' actually advertises that you have an /admin/. (2) **It's advisory only**: well-behaved crawlers respect it; malicious bots ignore it. (3) **Disallow ≠ noindex**: a page that's `Disallow`ed but linked from other indexed pages can still appear in search results (with no snippet). For real privacy, use authentication, IP allowlists, or `noindex` meta tags — never rely on robots.txt to hide sensitive paths.
Is the data sent to a server?
Only when you use the fetch button. Then the domain you entered is sent to Toolsana's API, which downloads the live robots.txt from that domain for you (browsers block most cross-origin fetches via CORS). Our server's request log records the request URL, which includes the address you typed. Pasted robots.txt content and the path tests are processed in your browser and are not sent anywhere. For checking your own robots.txt before publishing, you can also just open `https://yoursite.com/robots.txt` directly in a browser (since it's public), or paste the content into the box, which also works for staging files that are not online.
What's the difference between Disallow and noindex?
Different mechanisms with different effects. **`Disallow: /private/`** (robots.txt): tells crawlers not to FETCH the page — but the URL can still appear in search results if linked elsewhere (with no description, since the bot didn't read the content). **`<meta name='robots' content='noindex'>`** (in HTML): tells crawlers they CAN fetch but must NOT index. For ensuring a page never appears in search, use `noindex` (the crawler must be able to fetch the page to see the meta tag — so don't combine with Disallow). For hiding admin paths from crawl entirely (where you don't care if the URL leaks), Disallow works.
What are common robots.txt syntax mistakes?
(1) **Wrong location**: must be at `https://yoursite.com/robots.txt`, NOT `/site/robots.txt`. (2) **Case-sensitive paths**: `Disallow: /Admin/` doesn't block `/admin/`. (3) **Wildcard misuse**: `Disallow: /*` blocks everything; `Disallow: *.pdf` is non-standard (use `Disallow: /*.pdf$` if supported). (4) **Allow vs Disallow conflicts**: the longest (most specific) matching rule wins, and Allow wins a tie, per the spec, but some bots get this wrong. (5) **Blank Disallow**: `Disallow:` (empty value) matches nothing, so it allows everything — the opposite of what some authors expect. Test with this tool before deploying.
Should I disallow common paths like /wp-admin/?
**No** — for two reasons. (1) **Security through obscurity**: listing `/wp-admin/` advertises that you run WordPress; targeted attacks. (2) **Crawl budget**: Google doesn't significantly waste budget on routes that return 401/403/redirect, so explicit disallowing rarely helps SEO. For pages you actively don't want indexed (search results, filtered facet URLs, user-private profiles), use `noindex` — it works regardless of robots.txt. For admin protection, use authentication, not robots.txt. Modern WordPress sites need almost no custom robots.txt.
How do I declare my sitemap in robots.txt?
Add a `Sitemap:` directive with the absolute URL: `Sitemap: https://example.com/sitemap.xml`. Multiple sitemaps: one `Sitemap:` line per file. Place anywhere in the file (not tied to a specific User-agent block). All major search engines (Google, Bing, Yandex) discover sitemaps via this directive — submitting to Search Console is optional but improves visibility. For large sites, use sitemap index files referencing multiple sub-sitemaps. Use [Meta Tag Generator](/tools/meta-tag-generator/) to verify your site's other SEO infrastructure.
Share This Tool
Found this tool helpful? Share it with others who might benefit from it!
💡 Help others discover useful tools! Sharing helps us keep these tools free and accessible to everyone.