Share:
Glossary Web & HTTP

robots.txt

robots.txt is a public text file at a site's root that tells crawlers which paths they may or may not fetch.

Last reviewed:

AdSense Placeholder
Slot: header_reference_page

In more detail

Rules use User-agent, Disallow and Allow lines, plus a Sitemap line. It controls crawling, not indexing: a blocked URL can still appear in results, so use noindex to keep a page out. Never use it to hide sensitive content.

Try these tools

robots.txt Generator

Generate a valid robots.txt file with Allow/Disallow rules and a Sitemap directive from a simple form.

Robots.txt Tester

Check Disallow/Allow rules per user-agent, catch blocked pages instantly, and fix crawl errors before Google does.

See also

  • Glossary Sitemap
    An XML sitemap is a file listing a site's URLs, with optional last-modified dates, to help search engines find and crawl pages.
  • Guide How SEO Meta Tags Work
    Title, description, canonical, hreflang, robots.txt.
Back to Glossary
AdSense Placeholder
Slot: footer_leaderboard