System Architecture

IndexFlow is engineered as a high-throughput, compiled Rust service. It combines an asynchronous web server, an embedded SQLite database with Write-Ahead Logging (WAL), background workers, and independent technical SEO and sitemap libraries into a single deployable binary.

1. High-Level Architectural Layers

┌─────────────────────────────────────────────────────────────────────────┐
│                           1. Presentation Layer                         │
│  - Axum 0.7 HTTP Router                                                 │
│  - Interactive Web Console (Askama HTML Templates + HTMX)               │
│  - JSON REST & API Endpoints (/api/v1/*)                                │
└────────────────────────────────────┬────────────────────────────────────┘
                                     │
                                     ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                           2. Application Layer                          │
│  - PipelineManager (Arc<RwLock<HashSet<PipelineStage>>>)                │
│  - SiteService         - UrlService           - SitemapService          │
│  - HealthService       - GscService           - BingService             │
│  - SubmissionService   - ViewBoostRegistry                              │
└────────────┬───────────────────────────────────────────────┬────────────┘
             │                                               │
             ▼                                               ▼
┌──────────────────────────┐                    ┌────────────────────────┐
│  3. Background Workers   │                    │  4. Standalone Crates  │
│  - SyncWorker            │                    │  - indexflow-sitemap   │
│  - SeoAuditWorker (x30)  │                    │  - indexflow-seo       │
│  - GscInspectWorker (x3) │                    └────────────────────────┘
│  - BingInspectWorker(x2) │                                 │
│  - BingSubmitWorker      │                    ┌────────────┴───────────┐
│  - GoogleSubmitWorker    │                    │  5. External Providers │
└────────────┬─────────────┘                    │  - Google OAuth2 / GSC │
             │                                  │  - Bing IndexNow / BWT │
             ▼                                  └────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────┐
│                      6. Infrastructure & Storage                        │
│  - SQLite 3 (WAL Mode, synchronous = NORMAL, busy_timeout = 5s)         │
│  - Repositories: UrlRepo, SiteRepo, AdminRepo, HealthCheckRepo,         │
│    SubmissionLogRepo, IndexHistoryRepo                                  │
└─────────────────────────────────────────────────────────────────────────┘

2. Pipeline Lifecycle & Concurrency Management

IndexFlow coordinates background workflows using a thread-safe PipelineManager. Instead of scattered atomic flags, the engine maintains an active set of running stages:

pub enum PipelineStage {
    Sitemap,     // Stage 1: Sitemap asset discovery
    SeoGate,     // Stage 2: Technical SEO & GEO pre-flight audit
    GscInspect,  // Stage 3a: Google Search Console inspection & pool sync
    BingInspect, // Stage 3b: Bing Webmaster Tools official inspection
    PushSubmit,  // Stage 4: Engine active push (IndexNow & Google Indexing)
}

Stage Transitions and Concurrency Budgets

[ Trigger: UI or API ]
         │
         ▼
[ PipelineManager::start(stage) ]
         │
         ├───> SyncWorker (1 Task per Site, chunks of 500 records)
         │
         ├───> SeoAuditWorker (Concurrency: 30 Semaphore, 5MB body cap)
         │
         ├───> GscInspectWorker (Concurrency: 3, 250ms interval, 429 auto-stop)
         │
         ├───> BingInspectWorker (Concurrency: 2, 500ms interval, 1.5 QPS throttle backoff)
         │
         └───> PushWorkers (Google: Rolling 24h quota; Bing: Batch JSON)
         │
         ▼
[ Queue Exhausted -> PipelineManager::stop(stage) ]

3. URL Priority Scheduling Algorithm

When URLs are discovered or synced from sitemaps, IndexFlow computes a schedule priority integer from 1 to 999 (where lower values indicate higher urgency). This directly determines the ORDER BY priority ASC, id ASC claim sequence used by all background inspection and audit workers.

The score is calculated via compute_url_priority:

$$\text{Priority} = \text{Score}{\text{sitemap}} + \text{Score}{\text{age}} + \text{Penalty}{\text{known}} + \text{Boost}{\text{update}}$$

pub fn compute_url_priority(
    sitemap_priority: Option<f64>,
    lastmod: Option<DateTime<Utc>>,
    previous_lastmod: Option<DateTime<Utc>>,
    is_new: bool,
    now: DateTime<Utc>,
) -> i32 {
    let sp = sitemap_priority.unwrap_or(0.5).clamp(0.0, 1.0);
    let from_sitemap = ((1.0 - sp) * 400.0).round() as i32;

    let from_lastmod = match lastmod {
        None => 150,
        Some(lm) => {
            let age = now.signed_duration_since(lm);
            if age <= Duration::hours(24) { 0 }
            else if age <= Duration::days(7) { 40 }
            else if age <= Duration::days(30) { 100 }
            else if age <= Duration::days(90) { 180 }
            else { 280 }
        }
    };

    let from_new = if is_new { 0 } else { 80 };

    let from_update = match (lastmod, previous_lastmod) {
        (Some(cur), Some(prev)) if cur > prev => -60,
        (Some(_), None) if !is_new => -30,
        _ => 0,
    };

    (from_sitemap + from_lastmod + from_new + from_update).clamp(1, 999)
}

4. View Boost Mechanism

When an operator views a specific page of URLs in the Web Console (/indexing or /seo), the user expects the currently visible URLs to be inspected first.

IndexFlow implements a View Boost Registry (ViewBoostRegistry):

1. Upon rendering a paginated table view, the handler registers the visible URL IDs:

   state.view_boost.record(current_site_id, items.iter().map(|u| u.id).collect());
   

2. Worker fetch queries prioritize these active IDs with a 30-second TTL:

   SELECT * FROM urls
   WHERE gsc_index_status = 'UNKNOWN'
   ORDER BY (CASE WHEN id IN ($1, $2, $3...) THEN 0 ELSE 1 END), priority ASC, id ASC
   LIMIT 50;
   

3. This provides responsive, real-time UI feedback while background workers process large queues.

5. Token Caching & Provider Authentication

Google OAuth2 Architecture

Google APIs require RSA-SHA256 signed JWT assertions to exchange for access tokens.

• GoogleAuthClient parses the Service Account JSON key.
• Signs a JWT valid for 1 hour with scopes webmasters.readonly and indexing.
• Caches the exchanged OAuth2 Bearer token in an in-memory RwLock<HashMap<String, CachedToken>>.
• Automatically refreshes the token when less than 5 minutes of validity remain, avoiding redundant network round-trips.

Bing Webmaster Tools API Cache

Bing uses API keys passed via query parameters (?apikey=...).

• BingProvider resolves the canonical site identity prefix via GetUserSites.
• The resolved property URL is cached in memory per site, reducing external API calls.