IF
IndexFlowing / Docs
🌐 切换中文文档 GitHub ↗

Configuration Reference

IndexFlow is configured through system environment variables, a local .env file, and per-site database credentials.

1. Global Environment Variables

Global variables control the network binding, database pool, worker polling schedules, and security secrets. They are read on application boot (src/config/mod.rs).

Variable Type Default Value Description
SERVER_HOST String 0.0.0.0 Network IP address to bind the Axum web server.
SERVER_PORT u16 8080 Port for the HTTP Web Console, partials, and REST API.
DATABASE_URL String sqlite:indexflow.db?mode=rwc SQLite database file connection string. WAL mode is enabled automatically.
DB_MAX_CONNECTIONS u32 5 Maximum number of concurrent connections in the SQLite pool.
WORKER_POLL_INTERVAL_SECS u64 2 Polling tick interval (in seconds) for Sitemap sync and Technical SEO audit workers.
SUBMIT_WORKER_INTERVAL_SECS u64 5 Polling tick interval (in seconds) for Google and Bing inspection and push workers.
SUBMIT_WORKER_BATCH i64 10 Maximum number of URLs fetched and processed per submission tick.
JWT_SECRET String indexflow-secret-key-change-in-production Secret string used to sign administrator authentication JWT cookies (if_token). Must be changed in production.
DRY_RUN bool true When set to true or 1, all outbound push calls to Google Indexing and Bing IndexNow are intercepted and mocked with local success logs. Set to false in production.
RUST_LOG String indexflow_core=info,tower_http=info Tracing log filter expression. Supports error, warn, info, debug, trace.

2. .env File Example

Create a .env file in the root directory where you launch indexflow-core:

# Server Network Settings
SERVER_HOST=0.0.0.0
SERVER_PORT=8080

# Storage & Connection Pool
DATABASE_URL=sqlite:/var/lib/indexflow/indexflow.db?mode=rwc
DB_MAX_CONNECTIONS=5

# Worker Schedules
WORKER_POLL_INTERVAL_SECS=2
SUBMIT_WORKER_INTERVAL_SECS=5
SUBMIT_WORKER_BATCH=10

# Security (Generate with: openssl rand -hex 32)
JWT_SECRET=c79f8b4a2e5d9c1b3f7a6e8d2c4b0a1f9e8d7c6b5a4f3e2d1c0b9a8f7e6d5c4b

# Safety Mode: Set to false to enable live API push
DRY_RUN=false

# Logging Level
RUST_LOG=indexflow_core=info,tower_http=info

3. Site-Level Credentials & Integrations

Credentials for search engine APIs and crawl settings are configured per site via the Web Console (/settings) or the REST API (POST /api/v1/settings).

1. Cloudflare WAF Bypass / Custom User-Agent (custom_user_agent)

  • Purpose: Prevents Cloudflare Bot Management or Web Application Firewalls from returning 403 Forbidden during sitemap crawling or SEO gate audits.
  • Behavior:
    • If you enter a secret string (e.g. MySeoCrawlerSecret), IndexFlow constructs: 
      Mozilla/5.0 (compatible; IndexFlowBot/1.0; +https://www.indexflowing.com); MySeoCrawlerSecret
    • If you enter a full User-Agent starting with Mozilla/, it replaces the default crawler User-Agent completely.
  • Cloudflare Rule Setup: Create a WAF Custom Rule on your domain:
    • Expression: (http.user_agent contains "MySeoCrawlerSecret")
    • Action: Bypass / Skip (select WAF Managed Rules, Bot Fight Mode, and Rate Limiting).

2. Bing IndexNow Key (bing_indexnow_key)

  • Format: 32-character hexadecimal string (e.g. a1b2c3d4e5f6789012345678abcdef01).
  • File Verification: You must host a text file containing the key at: 
    https://example.com/a1b2c3d4e5f6789012345678abcdef01.txt
  • Test Endpoint: POST /api/v1/sites/test-indexnow verifies whether this text file is publicly accessible.

3. Bing Webmaster Tools API Key (bing_webmaster_api_key)

  • Acquisition: In Bing Webmaster Tools, go to Settings -> API Access -> API Key.
  • Role: Allows IndexFlow to call GetUserSites to resolve canonical site URLs and GetUrlInfo to check indexing and crawl timestamps.
  • Test Endpoint: POST /api/v1/sites/test-bing-webmaster validates key access and returns all verified domains.

4. Google Service Account Key (google_service_account_json)

  • Format: Complete JSON credentials file downloaded from Google Cloud Platform IAM.
  • GCP Setup:
    1. Create a Service Account in GCP IAM.
    2. Enable the Google Search Console API (webmasters.readonly) and the Web Search Indexing API (indexing).
    3. Create and download an asymmetric RSA JSON key for the service account.
    4. Add the service account's client_email (e.g. [email protected]) as an Owner or Full Permission user inside Google Search Console for your domain property.
  • Test Endpoint: POST /api/v1/sites/test-google verifies OAuth2 JWT signing, connects to GSC, and resolves the property identity (e.g. sc-domain:example.com).

4. Dry-Run Mode Mechanics

IndexFlow includes an integrated circuit breaker for testing and staging:

               ┌───────────────────────┐
               │ Submit URL Request    │
               └───────────┬───────────┘
                           │
                 [ Is DRY_RUN == true? ]
                  /                   \
            YES  /                     \  NO
                ▼                       ▼
   ┌────────────────────────┐  ┌────────────────────────┐
   │ Mock Success Log (200) │  │ Sign OAuth2 / IndexNow │
   │ No external HTTP call  │  │ Dispatch to Google/Bing│
   └────────────────────────┘  └────────────────────────┘
  • When DRY_RUN=true:
    • GoogleIndexingClient and BingProvider intercept submit batches.
    • A mock success response (DRY_RUN_MOCK: 已演练并记录日志，未向 API 发起网络请求) is written to submission_logs.
    • Database status updates to SUBMITTED, verifying the pipeline without expending daily quotas.
  • When DRY_RUN=false:
    • Full JWT assertions are signed and live API requests are dispatched to indexing.googleapis.com and api.indexnow.org.
← Introduction API Reference →