robots.txt file exists

robots_txt_exists

Checks if a robots.txt file exists at the domain root.

What does this rule check?

The robots.txt file instructs search engine crawlers which parts of the site they can and cannot crawl. It should be accessible at the root of the domain (e.g., https://example.com/robots.txt).

Why is this important?

Without robots.txt, crawlers have no guidance about which pages to prioritize or avoid. While they'll crawl everything by default, you miss the opportunity to manage crawl budget and block private areas.

How to fix

Create a robots.txt file at the root of your domain. Include basic directives and a reference to your sitemap.

Examples

<!-- No robots.txt file exists at /robots.txt -->
<!-- Returns 404 -->

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

Agent fix snippet

Copy this snippet and paste it into your AI coding agent to fix this issue automatically.

Create a robots.txt file at the root of your domain:

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://yourdomain.com/sitemap.xml