CI/CD SDK

The SEO Crawler SDK is a standalone CLI binary that runs SEO/GEO checks in your CI/CD pipeline. Catch SEO regressions before they reach production.

Installation

Build from source

git clone https://github.com/booltools/booltools-seo-crawler.git
cd booltools-seo-crawler
make build-sdk
# Binary: bin/seo-crawler

Download release binary

Linux / macOS

# Linux (amd64)
curl -sL https://github.com/booltools/booltools-seo-crawler/releases/latest/download/seo-crawler-linux-amd64 -o seo-crawler
chmod +x seo-crawler

# macOS (Apple Silicon)
curl -sL https://github.com/booltools/booltools-seo-crawler/releases/latest/download/seo-crawler-darwin-arm64 -o seo-crawler
chmod +x seo-crawler

# macOS (Intel)
curl -sL https://github.com/booltools/booltools-seo-crawler/releases/latest/download/seo-crawler-darwin-amd64 -o seo-crawler
chmod +x seo-crawler

Windows

# PowerShell
Invoke-WebRequest -Uri "https://github.com/booltools/booltools-seo-crawler/releases/latest/download/seo-crawler-windows-amd64.exe" -OutFile "seo-crawler.exe"

# Or with curl (available in Windows 10+)
curl -sL https://github.com/booltools/booltools-seo-crawler/releases/latest/download/seo-crawler-windows-amd64.exe -o seo-crawler.exe

On Windows, use seo-crawler.exe instead of ./seo-crawler in all commands below.

Scan Modes

Full mode (live crawl)

Crawls a running server, following links and checking all pages.

seo-crawler --mode=full --url=http://localhost:3000

Static mode (local HTML files)

Analyzes built HTML files directly — no server needed.

seo-crawler --mode=static --dir=./dist

Auto-starting a Dev Server

The SDK can start your dev/preview server before crawling and stop it after:

seo-crawler \
  --mode=full \
  --start-cmd="npm run dev" \
  --wait-for=http://localhost:3000 \
  --url=http://localhost:3000

Multiple Servers

When your app depends on both a frontend and a backend (or other services), pass multiple commands and URLs:

# Via CLI flags (comma-separated)
seo-crawler \
  --mode=full \
  --start-cmd="go run ./cmd/server,npm run dev" \
  --wait-for="http://localhost:8080/api/health,http://localhost:3000" \
  --url=http://localhost:3000

Or as an array in the YAML config:

mode: full
url: http://localhost:3000
start_cmd:
  - "go run ./cmd/server"
  - "cd web && npm run dev"
wait_for:
  - http://localhost:8080/api/health
  - http://localhost:3000
wait_timeout: 90s
fail_on: high

All commands are started in parallel. The crawler waits for every URL in wait_for before starting. All processes are stopped when the scan finishes.

Both fields accept a single string or an array in YAML. On the CLI, use comma-separated values.

Configuration File

Create .seo-crawler.yml in your project root:

mode: full
url: http://localhost:3000
start_cmd: "npm run dev"
wait_for: http://localhost:3000
wait_timeout: 60s
fail_on: high
format: text
ignore:
  - uses_https
  - hsts_header

CLI flags override config file values.

Avoiding port conflicts

If port 3000 is already in use on your machine, use --port to inject a PORT environment variable into the --start-cmd process. Make sure --url and --wait-for match the chosen port:

mode: full
url: http://localhost:3000
start_cmd: "npm run dev"
wait_for: http://localhost:3000
port: 3000

CLI Flags

URL Filtering

Use --only-urls to analyze only specific pages, or --exclude-urls to skip pages. Both support * wildcard patterns.

Only analyze specific pages

# Only analyze marketplace pages
seo-crawler --mode=full --url=http://localhost:3000 --only-urls="/marketplace/*"

# Only analyze blog and product pages
seo-crawler --mode=full --url=http://localhost:3000 --only-urls="/blog/*,/products/*"

# Analyze a specific route
seo-crawler --mode=full --url=http://localhost:3000 --only-urls="/pricing"

Exclude pages from analysis

# Skip admin and API pages
seo-crawler --mode=full --url=http://localhost:3000 --exclude-urls="/admin/*,/api/*"

# Skip legal pages
seo-crawler --mode=full --url=http://localhost:3000 --exclude-urls="/privacy,/terms"

YAML config

mode: full
url: http://localhost:3000
only_urls:
  - /marketplace/*
  - /blog/*

# Or exclude specific paths:
# exclude_urls:
#   - /admin/*
#   - /api/*

Exit Codes

GitHub Action

The SDK is available as a GitHub Action. Add it directly to your workflow:

name: SEO Audit
on: [pull_request]

jobs:
  seo:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: "22"
          cache: "npm"

      - run: npm ci

      - name: SEO Audit
        uses: booltools/booltools-seo-crawler@v0
        with:
          mode: full
          start-cmd: "npm run dev"
          wait-for: http://localhost:3000
          url: http://localhost:3000
          fail-on: high

Static mode (no server needed)

      - run: npm run build

      - name: SEO Audit
        uses: booltools/booltools-seo-crawler@v0
        with:
          mode: static
          dir: ./dist
          fail-on: high

Action Inputs

Action Outputs

Manual Setup (without the Action)

If you prefer to download the binary directly:

name: SEO Audit
on: [push, pull_request]

jobs:
  seo-audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: "22"
          cache: "npm"

      - run: npm ci

      - name: Download SEO Crawler SDK
        run: |
          curl -sL https://github.com/booltools/booltools-seo-crawler/releases/latest/download/seo-crawler-linux-amd64 -o seo-crawler
          chmod +x seo-crawler

      - name: Run SEO audit
        run: |
          ./seo-crawler \
            --mode=full \
            --start-cmd="npm run dev" \
            --wait-for=http://localhost:3000 \
            --url=http://localhost:3000 \
            --fail-on=high

Ignoring Rules for Localhost

When testing localhost, some rules will always fail (e.g. HTTPS, HSTS). Ignore them in your config:

ignore:
  - uses_https
  - hsts_header
  - mixed_content