2. Technical SEO & GEO Quality Gate

The Technical SEO & GEO Quality Gate runs pre-flight quality inspections on discovered URLs before allowing them into search engine submission queues. It prevents crawling budget wastage by blocking broken pages, misconfigured indexing tags, and canonical conflicts.

Hard Gate Decision Rules (passed = false)

If a page violates any of the following four core rules, it receives seo_status = 'FAIL', is assigned a specific block_reason, and is strictly barred from outbound search engine submission queues:

┌──────────────────────────────────────────────────────────────┐
│                    Pre-Flight SEO Gate                       │
├──────────────────────────────────────────────────────────────┤
│ 1. HTTP Status Code  ─── Must be 200 OK (3xx, 4xx, 5xx FAIL) │
│ 2. Noindex Directive ─── No noindex in Meta or X-Robots-Tag  │
│ 3. Canonical Tag     ─── Canonical URL must match Page URL   │
│ 4. Page Title        ─── Non-empty <title> tag required      │
└──────────────────────────────────────────────────────────────┘

1. HTTP Status Check

• Rule: The URL must return an exact 200 OK status.
• Redirects are Failures: IndexFlow probes URLs without following redirects (redirect::Policy::none()). A 301 or 302 response fails the gate with HTTP 301 / HTTP 302 because sitemaps must only list final canonical target URLs.

2. noindex Directives

• Rule: The response must not instruct crawlers to skip indexing.
• Inspection Points:
  • <meta name="robots" content="noindex">
  • <meta name="googlebot" content="noindex">
  • <meta http-equiv="robots" content="noindex">
  • HTTP header: X-Robots-Tag: noindex or X-Robots-Tag: none

3. Canonical URL Equivalence (canonical_matches_page)

• Rule: If a <link rel="canonical" href="..."> is declared, it must resolve to the inspected page URL.
• Normalization Rules:
  • Scheme and hostname are converted to lowercase.
  • Default ports (:80 for HTTP, :443 for HTTPS) are removed.
  • Relative (./post, ../post) and protocol-relative (//example.com/post) paths are resolved against the base URL.
  • Path segments . and .. are normalized.
  • UTF-8 percent-encoded path characters are decoded.
  • Trailing slashes are stripped (except for the root /).
  • URL fragments (#heading) are stripped.
  • Query parameters are sorted alphabetically by key and value.
  • Note: URL paths remain case-sensitive according to RFC 3986.

4. Page Title Tag Presence

• Rule: The HTML document must contain a non-empty <title>...</title> element outside <script> or comment blocks.

Soft Warnings & Optimization Signals (seo_status = 'WARN')

When a page passes all four hard gate rules, IndexFlow evaluates secondary SEO and GEO (Generative Engine Optimization) signals. If any issues are found, the URL is marked with seo_status = 'WARN' and allowed into push queues with advisory warnings:

GEO & AI Crawler Auditing

IndexFlow tracks indexing and training permissions for leading Generative Engine Optimization (GEO) bots:

Schema.org Structured Data Extraction

The engine extracts JSON-LD blocks and resolves all declared @type entities (including @graph trees and array-based type declarations):

{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Organization",
      "name": "IndexFlow",
      "url": "https://indexflowing.com"
    },
    {
      "@type": "TechArticle",
      "headline": "Search Indexing Infrastructure",
      "author": {
        "@type": "Person",
        "name": "Developer"
      }
    }
  ]
}

Extracted Types: ["Organization", "TechArticle"] (nested non-root entities like Person inside attributes are ignored).

Concurrency & Probing Engine

• Concurrency: SeoAuditWorker runs up to 30 concurrent audit tasks bounded by a tokio::sync::Semaphore.
• Payload Safety Cap: Response bodies are streamed with a hard cap of 5 MiB (MAX_HTML_BODY). Responses declaring Content-Length > 20 MiB have their body parsing skipped to prevent out-of-memory errors on large files.
• Custom Crawler UA: Automatically appends the site's custom_user_agent token to bypass Cloudflare Bot Fight Mode.

Executing Technical SEO Audits

Trigger Audit Pipeline via Web Console

Navigate to /seo and click Run SEO Audit.

Audit a Single URL via REST API

curl -X POST "https://www.indexflowing.com/docs/api/v1/urls/101/inspect?engine=seo" \
  -H "Cookie: if_token=<YOUR_JWT_TOKEN>"

Batch Audit Multiple URLs

curl -X POST "https://www.indexflowing.com/docs/api/v1/urls/batch-inspect" \
  -H "Content-Type: application/json" \
  -H "Cookie: if_token=<YOUR_JWT_TOKEN>" \
  -d '{
    "selected_ids": [101, 102, 103],
    "engine": "seo"
  }'