The robots.txt file instructs search engine crawlers which pages or files they can or cannot request from your site. Syntax errors in this file can accidentally block search engines from indexing your primary content.
What This Tool Checks
- User-Agent Declarations: Validates rules targeting specific search engine bots (such as Googlebot, Bingbot) or universal wildcard rules (
User-agent: *). - Disallow and Allow Directives: Checks paths listed under
DisallowandAllowfor formatting errors and syntax conflicts. - Sitemap Declarations: Verifies that your
Sitemap:directive contains a valid, absolute URL pointing to your XML sitemap. - Syntax and Formatting Rules: Flags misplaced spaces, invalid line breaks, and unrecognized directive keywords.
What Your Results Mean
- Valid File Structure: Your
robots.txtfile uses standard syntax recognized by search engine crawlers. - Accidental Disallow Warning: Critical site directories (such as your homepage or main asset folders) are blocked by a disallow rule.
- Missing Sitemap Directive: No XML sitemap location is specified in
robots.txt, making discovery harder for crawlers.
Tool Limitations
This tool validates syntax and formatting rules according to published web standards. It cannot verify whether blocking a specific directory aligns with your business goals, nor can it force search engines to index blocked content.