robots.txt is not too large

robots_txt_size

Checks that robots.txt file is not too large.

What does this rule check?

Google limits robots.txt processing to 500KB. Files larger than this limit are ignored beyond the cutoff point. Very large robots.txt files also slow down crawl scheduling.

Why is this important?

An oversized robots.txt means directives beyond the size limit are ignored, potentially allowing crawling of areas you intended to block.

How to fix

Simplify your robots.txt by using broader wildcard patterns instead of listing every individual URL. Consolidate duplicate directives.

Examples

# 600KB robots.txt with thousands of individual Disallow lines
Disallow: /page1
Disallow: /page2
...thousands more...

# Clean robots.txt with wildcard patterns
User-agent: *
Disallow: /admin/*
Disallow: /api/*
Disallow: /tmp/*

Agent fix snippet

Copy this snippet and paste it into your AI coding agent to fix this issue automatically.

Reduce robots.txt file size:
1. Use wildcard patterns: Disallow: /admin/*
2. Remove duplicate directives
3. Group similar paths under broader patterns
4. Target should be under 500KB