robots.txt has valid syntax

robots_txt_syntax

Checks that robots.txt has valid syntax.

What does this rule check?

Robots.txt follows a specific syntax format. Invalid syntax (missing User-agent, malformed directives, wrong line endings) can cause crawlers to misinterpret your instructions.

Why is this important?

Invalid syntax may cause crawlers to ignore your directives entirely, leading to unwanted crawling of private areas or blocking of important pages.

How to fix

Fix syntax errors in your robots.txt. Ensure each group starts with a User-agent line. Use standard directives: Allow, Disallow, Sitemap.

Examples

Disallow: /admin/
<!-- Missing User-agent line! -->

user agent: Googlebot
<!-- Wrong syntax! -->

User-agent: *
Disallow: /admin/

User-agent: Googlebot
Allow: /

Agent fix snippet

Copy this snippet and paste it into your AI coding agent to fix this issue automatically.

Fix robots.txt syntax:
1. Each block must start with "User-agent:"
2. Use "Disallow:" and "Allow:" (capitalized, with colon)
3. One directive per line
4. Blank line between user-agent blocks