1. Sitemap Engine

The Sitemap Engine is responsible for asset discovery. It crawls, decompresses, parses, and normalizes URL entries from standard XML sitemaps, sitemap index hierarchies, .xml.gz archives, and plaintext URL lists.

Technical Specifications & Protocol Limits

The Sitemap Engine enforces Google Sitemap protocol limits and defensive safeguards:

Parameter	Constant	Value	Description
Max Document Size	`MAX_UNCOMPRESSED_BYTES`	50 MiB (52,428,800 bytes)	Maximum uncompressed payload permitted for a single sitemap file.
Max Download Size	`MAX_DOWNLOAD_BYTES`	52 MiB (54,525,952 bytes)	Maximum HTTP streaming download size limit.
Max URLs Per File	`MAX_URLS_PER_SITEMAP`	50,000	Maximum `<url>` or `<sitemap>` records parsed per document.
Max Expansion Cap	`MAX_EXPAND_URLS`	1,000,000	Global URL accumulation limit during recursive index expansion.
Max Loc Length	`MAX_URL_LEN`	2,048 bytes	RFC 7230 loc length limit; longer URLs are dropped.
Default Max Depth	`max_depth`	3 levels	Maximum recursion depth for nested `<sitemapindex>` files.

Core Capabilities

1. Transparent Gzip Decompression Bomb Protection

Sitemaps frequently use .xml.gz compression. A malicious or corrupted 1 KiB gzip file can inflate into tens of gigabytes in memory. IndexFlow streams decompressions through a limited flate2 reader (Take limiter). If uncompressed bytes exceed 50 MiB, decompression aborts immediately with SitemapError::DecompressionBomb.

2. Prefix-Agnostic XML Parsing

XML sitemaps in the wild use varying namespace prefixes (e.g. <image:image>, <img:image>, or unprefixed <image>). The engine evaluates tags strictly by their local byte name (Tag::from_local), guaranteeing consistent extraction regardless of XML namespace aliases.

3. Fault-Tolerant Ampersand Sanitization

Hand-written sitemaps often contain unencoded literal & characters in query strings (e.g. https://example.com/?a=1&b=2 instead of &). IndexFlow runs a fast, single-pass sanitizer (sanitize_bare_ampersands) that rewrites unescaped ampersands while preserving valid XML entities and <![CDATA[...]]> blocks.

4. Circular Reference & Cycle Detection

When expanding <sitemapindex> hierarchies, recursive loops can occur if child sitemaps point back to an ancestor. IndexFlow calculates a canonical identity key for each URL:

Lowercases scheme and host.
Strips standard default ports (:80, :443).
Drops URL fragments (#section).
Strips trailing slashes (except root /).

If an identity key has already been visited in the current expansion branch, it is logged and skipped without failing the entire crawl.

Google Extensions Extraction

IndexFlow extracts rich Google sitemap extension metadata into structured Rust models:

1. Images (`<image:image>`)

<image:image>
  <image:loc>https://example.com/cover.jpg</image:loc>
  <image:title>Page Cover</image:title>
  <image:caption>Technical architecture diagram</image:caption>
  <image:geo_location>San Francisco, CA</image:geo_location>
  <image:license>https://example.com/license</image:license>
</image:image>

2. Videos (`<video:video>`)

<video:video>
  <video:thumbnail_loc>https://example.com/thumb.jpg</video:thumbnail_loc>
  <video:title>Rust Monolith Guide</video:title>
  <video:description>Building high-performance backends with Rust.</video:description>
  <video:duration>600</video:duration>
  <video:family_friendly>yes</video:family_friendly>
  <video:tag>rust</video:tag>
  <video:tag>seo</video:tag>
</video:video>

3. News (`<news:news>`)

<news:news>
  <news:publication>
    <news:name>Tech Chronicle</news:name>
    <news:language>en</news:language>
  </news:publication>
  <news:publication_date>2026-03-30T10:00:00Z</news:publication_date>
  <news:title>Modern SEO Infrastructure</news:title>
  <news:keywords>rust, indexing, sitemap</news:keywords>
</news:news>

4. Alternates & Hreflang (`<xhtml:link>`)

<xhtml:link rel="alternate" hreflang="zh-Hans" href="https://example.com/zh/page" />
<xhtml:link rel="alternate" hreflang="en" href="https://example.com/page" />

URL Locale & Path Prefix Extraction

During ingestion, SitemapService parses the URL path and hreflang metadata to categorize content:

Locale Extraction: Recognizes standard ISO 639-1 language prefixes (e.g. /zh/, /en-US/, /ja/) or uses declared xhtml:link tags.
Path Prefix: Categorizes the root folder (e.g. /docs, /blog, /products) to support localized filtering in the dashboard.

Executing Sitemap Sync

Trigger via Web Console

Navigate to /sitemap and click Sync Sitemap.

Trigger via REST API

curl -X POST "https://www.indexflowing.com/docs/api/v1/pipeline/sitemap/start?site_id=1" \
  -H "Cookie: if_token=<YOUR_JWT_TOKEN>"

Stop a Running Sync

curl -X POST "https://www.indexflowing.com/docs/api/v1/pipeline/sitemap/stop?site_id=1" \
  -H "Cookie: if_token=<YOUR_JWT_TOKEN>"

Database Upsert Behavior

Discovered URLs are written to the database in transactions of 500 entries. Existing URLs have their sitemap_lastmod, sitemap_synced_at, and priority updated automatically while preserving historical audit logs.