Crate: indexflow-sitemap

indexflow-sitemap is a high-performance, fault-tolerant, Google-compliant XML and plaintext sitemap parser and recursive crawler written in Rust.

Installation

Add indexflow-sitemap to your Cargo.toml:

[dependencies]
indexflow-sitemap = { version = "0.1.0", path = "crates/indexflow-sitemap" }

Feature Flags

To use in standalone minimal mode without HTTP capabilities:

[dependencies]
indexflow-sitemap = { version = "0.1.0", default-features = false, features = ["gzip"] }

Core Data Models

ParsedSitemap Enum

The primary output when parsing any raw sitemap document:

pub enum ParsedSitemap {
    /// A sitemap index pointing to child sitemaps (<sitemapindex>)
    Index { child_urls: Vec<String> },
    /// A standard URL set containing page records (<urlset>)
    UrlSet { entries: Vec<SitemapUrlEntry> },
    /// A line-oriented plaintext URL list (.txt)
    PlainText { urls: Vec<String> },
}

SitemapUrlEntry Struct

Represents a single <url> entry with Google extensions:

pub struct SitemapUrlEntry {
    pub loc: String,
    pub lastmod: Option<DateTime<Utc>>,
    pub changefreq: Option<ChangeFreq>,
    pub priority: Option<f64>,
    pub hreflangs: Vec<HreflangEntry>,
    pub images: Vec<SitemapImage>,
    pub videos: Vec<SitemapVideo>,
    pub news: Option<SitemapNews>,
}

Public API Reference

1. In-Memory Parsing (parse_sitemap)

Automatically detects whether the input string is XML or plaintext and extracts all entities:

use indexflow_sitemap::{parse_sitemap, ParsedSitemap, ChangeFreq};

fn main() {
    let xml_content = r#"<?xml version="1.0" encoding="UTF-8"?>
    <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
            xmlns:xhtml="http://www.w3.org/1999/xhtml"
            xmlns:image="http://www.google.com/schemas/sitemap-image/1.1">
      <url>
        <loc>https://example.com/blog/rust-guide</loc>
        <lastmod>2026-03-30T10:00:00Z</lastmod>
        <changefreq>weekly</changefreq>
        <priority>0.8</priority>
        <xhtml:link rel="alternate" hreflang="zh-Hans" href="https://example.com/zh/blog/rust-guide" />
        <image:image>
          <image:loc>https://example.com/cover.jpg</image:loc>
          <image:title>Rust Guide Cover</image:title>
        </image:image>
      </url>
    </urlset>"#;

    match parse_sitemap(xml_content) {
        ParsedSitemap::UrlSet { entries } => {
            for entry in entries {
                println!("URL: {}", entry.loc);
                println!("Last Modified: {:?}", entry.lastmod);
                println!("Priority: {:?}", entry.priority);
                println!("Hreflang count: {}", entry.hreflangs.len());
                println!("Images count: {}", entry.images.len());
            }
        }
        ParsedSitemap::Index { child_urls } => {
            println!("Sitemap index with {} child sitemaps", child_urls.len());
        }
        ParsedSitemap::PlainText { urls } => {
            println!("Plaintext sitemap with {} URLs", urls.len());
        }
    }
}

2. Recursive HTTP Expansion (SitemapFetcher)

Asynchronously crawls a root sitemap URL, recursively expanding all nested <sitemapindex> files while isolating branch failures and preventing cyclic loops:

use indexflow_sitemap::SitemapFetcher;
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let http_client = reqwest::Client::builder()
        .timeout(Duration::from_secs(15))
        .user_agent("Mozilla/5.0 (compatible; IndexFlowBot/1.0)")
        .build()?;

    let fetcher = SitemapFetcher::new(http_client);

    // Recursively expand up to 3 levels deep
    let (is_index, entries) = fetcher
        .expand_all("https://example.com/sitemap_index.xml", 3)
        .await?;

    println!("Root is index: {}", is_index);
    println!("Total extracted URLs: {}", entries.len());

    for (idx, entry) in entries.iter().take(5).enumerate() {
        println!("[#{}] {} (lastmod: {:?})", idx + 1, entry.loc, entry.lastmod);
    }

    Ok(())
}

3. Transparent Gzip Decompression (decode_if_gzipped)

Sniffs magic bytes (0x1F, 0x8B) or Unicode BOMs and safely inflates gzipped sitemaps:

use indexflow_sitemap::compression::{decode_if_gzipped, decode_if_gzipped_with_limit};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let raw_bytes: Vec<u8> = load_sitemap_bytes();

    // Default safety cap: 50 MiB
    let xml_string = decode_if_gzipped(&raw_bytes)?;

    // Custom safety cap: 10 MiB
    let custom_capped_string = decode_if_gzipped_with_limit(&raw_bytes, 10 * 1024 * 1024)?;

    Ok(())
}

Error Handling (SitemapError)

pub enum SitemapError {
    Xml(quick_xml::Error),
    Decompression(String),
    DecompressionBomb { limit: usize },
    PayloadTooLarge { size: u64, limit: u64 },
    Encoding(String),
    InvalidUrl(url::ParseError),
    Io(std::io::Error),
    Network(reqwest::Error),
    HttpStatus(u16),
    MaxDepthExceeded(u8),
    CircularReference(String),
}

• Decompression Bomb: Triggered if inflated size exceeds limit.
• Payload Too Large: Triggered if streaming HTTP response exceeds MAX_DOWNLOAD_BYTES (52 MiB).
• Circular Reference: Triggered when a child sitemap resolves to an already visited canonical ancestor.