Crate: indexflow-seo

indexflow-seo is a standalone, high-performance, zero-panic Rust library designed for technical SEO quality gate evaluations and Generative Engine Optimization (GEO / AI search bot) HTML inspections.

It powers the pre-flight quality gate inside IndexFlow-core and can also be used independently in custom web crawlers, static site generators, test suites, and CI/CD pipelines.

Key Features

• Zero-Panic HTML Parser: Safe string slicing with character boundary clamping (safe_slice) preventing panics on multibyte Unicode (CJK, emojis).
• Non-Following HTTP Prober (SeoProbeClient): Probes URLs without following redirects (exposing 3xx status codes directly as gate issues) and applies a 5 MiB memory safety streaming cap.
• RFC 3986 Canonical Normalization: Comprehensive canonical matching supporting relative, protocol-relative, and query-sorted URL equivalence.
• GEO & AI Search Bot Auditing: Detects robots directives specifically targeting GPTBot, ClaudeBot, PerplexityBot, and Google-Extended.
• Structured Data Extraction: Walks Schema.org JSON-LD blocks, resolving @graph trees and multi-typed entities.
• Single-Pass Entity Decoding: Decodes XML predefined entities, HTML named entities, and numeric/hexadecimal character references.

Installation

Add indexflow-seo to your Cargo.toml:

[dependencies]
indexflow-seo = { version = "0.1.0", path = "crates/indexflow-seo" }

Feature Flags

To use in pure in-memory or WebAssembly (WASM) environments without network dependencies:

[dependencies]
indexflow-seo = { version = "0.1.0", default-features = false }

Data Models Reference

1. SeoAuditResult

The comprehensive output produced by both in-memory evaluation and live network probes:

pub struct SeoAuditResult {
    // Network Layer
    pub http_status: Option<i32>,
    pub response_time_ms: Option<i32>,
    pub payload_bytes: Option<i32>,

    // Core Technical SEO Tags
    pub page_title: Option<String>,
    pub meta_description: Option<String>,
    pub h1_content: Option<String>,
    pub h1_count: usize,

    // Indexing & Canonical Directives
    pub canonical_url: Option<String>,
    pub has_canonical: bool,
    pub has_noindex: bool,
    pub has_nofollow: bool,
    pub robots_directive: Option<String>,
    pub hreflang: Vec<HreflangItem>,

    // Social & Mobile Metadata
    pub opengraph: OpenGraphMeta,
    pub twitter_card: TwitterCardMeta,
    pub has_viewport: bool,
    pub html_lang: Option<String>,
    pub images_missing_alt: usize,

    // GEO / AI Search & Structured Data
    pub json_ld: Vec<JsonLdBlock>,
    pub ai_directives: AiBotDirectives,

    // Decision Verdict & Soft Warnings
    pub passed: bool,
    pub block_reason: Option<String>,
    pub warnings: Vec<String>,
}

Helper Methods on SeoAuditResult

• result.schema_types() -> Vec<String>: Returns all unique Schema.org @type strings extracted across all JSON-LD blocks in document order.
• result.hreflang_json() -> Option<String>: Serializes the hreflang vector into a JSON string for database persistence.

2. AiBotDirectives

Sniffs indexing and training restrictions targeting major AI search crawlers:

pub struct AiBotDirectives {
    pub gptbot_blocked: bool,          // OpenAI GPTBot / ChatGPT Search
    pub perplexity_blocked: bool,      // Perplexity AI Crawler
    pub claudebot_blocked: bool,       // Anthropic ClaudeBot / Claude-Web
    pub google_extended_blocked: bool, // Google-Extended (Gemini Training / AI Overviews)
}

impl AiBotDirectives {
    pub fn any_blocked(&self) -> bool;
    pub fn blocked_names(&self) -> Vec<&str>;
}

3. OpenGraphMeta & TwitterCardMeta

pub struct OpenGraphMeta {
    pub title: Option<String>,
    pub description: Option<String>,
    pub image: Option<String>,
    pub og_type: Option<String>,
    pub url: Option<String>,
    pub site_name: Option<String>,
}

pub struct TwitterCardMeta {
    pub card: Option<String>,
    pub title: Option<String>,
    pub description: Option<String>,
    pub image: Option<String>,
}

Public API & Code Examples

1. In-Memory Evaluation (evaluate_html)

Evaluates an HTML string and optional HTTP response headers without network calls:

use indexflow_seo::evaluate_html;

fn main() {
    let page_url = "https://example.com/blog/rust-guide";
    let status_code = 200;
    let elapsed_ms = 45;
    let x_robots_header = Some("noarchive");
    let html_body = r#"<!DOCTYPE html>
    <html lang="en">
    <head>
      <meta charset="UTF-8">
      <meta name="viewport" content="width=device-width, initial-scale=1">
      <title>Rust SEO Best Practices &amp; Guide</title>
      <meta name="description" content="A complete guide to technical SEO and GEO in Rust." />
      <link rel="canonical" href="https://example.com/blog/rust-guide" />
      <link rel="alternate" hreflang="zh-Hans" href="https://example.com/zh/blog/rust-guide" />
      
      <!-- OpenGraph & Twitter -->
      <meta property="og:title" content="Rust SEO Best Practices" />
      <meta property="og:image" content="https://example.com/cover.jpg" />
      <meta property="og:type" content="article" />
      <meta name="twitter:card" content="summary_large_image" />

      <!-- AI Bot Directives -->
      <meta name="gptbot" content="noindex" />
      <meta name="perplexitybot" content="index, follow" />

      <!-- Structured Data (Schema.org) -->
      <script type="application/ld+json">
      {
        "@context": "https://schema.org",
        "@graph": [
          {
            "@type": "Organization",
            "name": "IndexFlow"
          },
          {
            "@type": "TechArticle",
            "headline": "Rust SEO Best Practices"
          }
        ]
      }
      </script>
    </head>
    <body>
      <h1>Complete Rust Guide</h1>
      <img src="/arch.png" alt="Architecture" />
      <img src="/chart.png" />
    </body>
    </html>"#;

    let res = evaluate_html(page_url, status_code, elapsed_ms, x_robots_header, html_body);

    // Hard Gate Decision
    assert!(res.passed);
    assert_eq!(res.block_reason, None);

    // Core Tags
    assert_eq!(res.page_title.as_deref(), Some("Rust SEO Best Practices & Guide"));
    assert_eq!(res.h1_content.as_deref(), Some("Complete Rust Guide"));
    assert_eq!(res.h1_count, 1);
    assert!(res.has_canonical);
    assert!(res.has_viewport);
    assert_eq!(res.html_lang.as_deref(), Some("en"));
    assert_eq!(res.images_missing_alt, 1);

    // GEO / AI Search Directives
    assert!(res.ai_directives.gptbot_blocked);
    assert!(!res.ai_directives.perplexity_blocked);

    // Schema.org Extraction
    assert_eq!(res.schema_types(), vec!["Organization".to_string(), "TechArticle".to_string()]);
}

2. Live URL Network Probe (SeoProbeClient)

Probes live web endpoints. Redirects (301/302) are not followed and are surfaced as gate failures:

use indexflow_seo::SeoProbeClient;
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let prober = SeoProbeClient::new(
        "Mozilla/5.0 (compatible; IndexFlowBot/1.0; +https://www.indexflowing.com)",
        Duration::from_secs(15),
    )?;

    let result = prober.check_url("https://example.com/blog/rust-guide").await;

    if result.passed {
        println!("✅ Quality Gate Passed (Ready for engine submission)");
    } else {
        println!("❌ Quality Gate Blocked: {:?}", result.block_reason);
    }

    println!("HTTP Status   : {:?}", result.http_status);
    println!("Response Time : {:?} ms", result.response_time_ms);
    println!("Payload Size  : {:?} bytes", result.payload_bytes);
    println!("Warnings ({}) :", result.warnings.len());
    for warning in &result.warnings {
        println!("  - {}", warning);
    }

    Ok(())
}

3. Canonical URL Equivalence Normalization (canonical_matches_page)

Verifies canonical URL equivalence against RFC 3986 normalization rules:

use indexflow_seo::canonical_matches_page;

fn main() {
    let page = "https://example.com/blog/post-1/";

    // Equivalent URLs (Returns true)
    assert!(canonical_matches_page(page, "https://example.com/blog/post-1"));
    assert!(canonical_matches_page(page, "https://example.com:443/blog/post-1"));
    assert!(canonical_matches_page(page, "/blog/post-1"));
    assert!(canonical_matches_page(page, "./post-1"));
    assert!(canonical_matches_page(page, "//example.com/blog/post-1"));
    assert!(canonical_matches_page(page, "https://example.com/blog/post-1?b=2&a=1")); // Query sorted match

    // Non-equivalent URLs (Returns false)
    assert!(!canonical_matches_page(page, "https://example.com/blog/other-post"));
    assert!(!canonical_matches_page(page, "https://example.com/BLOG/post-1")); // Path is case-sensitive
}

4. Zero-Panic Entity Decoding (decode_basic_entities)

Decodes XML predefined entities, HTML named entities, and numeric references in a single pass:

use indexflow_seo::decode_basic_entities;

fn main() {
    let raw = "Tom & Jerry 'Special' — 😀 © 2026";
    let decoded = decode_basic_entities(raw);
    assert_eq!(decoded, "Tom & Jerry 'Special' — 😀 © 2026");
}