IF
IndexFlowing / Docs
🌐 切换中文文档 GitHub ↗

3. Search Indexing Radar

The Search Indexing Radar provides dual-engine verification of real-world indexing statuses by directly communicating with the official APIs of Google Search Console (GSC) and Bing Webmaster Tools (BWT).

Google Search Console (GSC) Integration

Google enforces a strict limit of 2,000 URL Inspection requests per day per Google Cloud project. IndexFlow employs a dual-strategy approach to maximize quota efficiency.

                             ┌──────────────────────────────────────┐
                             │       Discovered Sitemap URLs        │
                             └──────────────────┬───────────────────┘
                                                │
                 ┌──────────────────────────────┴──────────────────────────────┐
                 │                                                             │
                 ▼                                                             ▼
   ┌───────────────────────────────┐                             ┌───────────────────────────────┐
   │ Strategy A: Search Analytics  │                             │ Strategy B: Inspection API    │
   │ - Past 30 Days Impressions    │                             │ - Remaining Unconfirmed Pages │
   │ - Up to 25,000 URLs / request │                             │ - Parses Coverage State       │
   │ - 0 Inspection Quota Used     │                             │ - 3 Concurrency / 250ms delay │
   └──────────────┬────────────────┘                             └───────────────┬───────────────┘
                  │                                                              │
                  └──────────────────────────────┬───────────────────────────────┘
                                                 │
                                                 ▼
                                ┌───────────────────────────────────┐
                                │   `gsc_index_status` in Database  │
                                │   INDEXED / CRAWLED / DISCOVERED  │
                                └───────────────────────────────────┘

Strategy A: 0-Quota Bulk Sync via Search Analytics

Before expending inspection quotas, IndexFlow queries the Google Search Console Search Analytics API (/webmasters/v3/sites/{siteUrl}/searchAnalytics/query):

  • Pulls all URLs that received search impressions over the past 30 days (dimensions: ["page"], rowLimit: 25000).
  • Performs exact SHA-256 hash matching and trailing-slash resilient matching against sitemap URLs.
  • Automatically marks matching records as gsc_index_status = 'INDEXED' with gsc_coverage_state = 'Indexed (Search Analytics Confirmed)'.
  • Saves inspection API quotas for newly published or unranked pages.

Strategy B: Granular URL Inspection API

For URLs not observed in Search Analytics, the GscInspectWorker queries https://searchconsole.googleapis.com/v1/urlInspection/index:inspect:

GSC Raw coverageState Mapped gsc_index_status Description
Submitted and indexed / Indexed INDEXED Page is confirmed in Google's index.
Crawled - currently not indexed CRAWLED_NOT_INDEXED Google crawled the page but chose not to index it.
Discovered - currently not indexed DISCOVERED_NOT_INDEXED Google knows the URL but has not crawled it yet.
URL is unknown to Google / other NOT_INDEXED URL has no crawl or index footprint in Google.
(Uninspected) UNKNOWN Page has not been evaluated yet.

GSC Inspection Worker Architecture

  • Concurrency: 3 concurrent tasks (tokio::sync::Semaphore::new(3)).
  • Pacing: 250ms sleep delay between requests.
  • Quota Circuit Breaker: If Google returns HTTP 429, the worker immediately trips the circuit breaker, halts execution, and notifies the pipeline manager to prevent project-level API suspensions.

Bing Webmaster Tools Integration

IndexFlow communicates with the official Bing Webmaster Tools JSON API (https://ssl.bing.com/webmaster/api.svc/json):

1. Dynamic Site URL Resolution (GetUserSites)

Bing requires API calls to specify the exact registered site URL format (e.g. https://example.com/ vs http://example.com/).

  • BingProvider::resolve_site_url queries GetUserSites using the configured API key.
  • Matches the clean domain and caches the resolved property URL in memory (Arc<RwLock<HashMap<String, String>>>) to eliminate redundant lookups.

2. Live URL Inspection (GetUrlInfo)

IndexFlow calls GetUrlInfo with parameters siteUrl, url, and apikey:

  • Extracts HttpStatus: Crawler response status code.
  • Extracts LastCrawledDate: Microsoft serialized date format (e.g. Date(1711785600000)), converted to UTC DateTime<Utc>.
  • Extracts IsPage: Boolean confirmation of webpage indexing.
  • Decision Rule: If LastCrawledDate contains a valid timestamp, the URL is marked as bing_index_status = 'INDEXED'. If Bing returns HTTP 404 or an empty d object, the URL is marked as NOT_INDEXED.

Rate Limit Throttling (1.5 QPS Safe Zone)

Bing enforces a maximum rate of 1.5 queries per second per API key:

  • Worker Concurrency: 2 concurrent workers with a 500ms request delay.
  • Backoff Handling: If Bing returns ThrottleHost, ErrorCode: 5, or HTTP 429, the worker pauses for 2 seconds and retains bing_index_status = 'UNKNOWN', allowing safe retries on subsequent ticks.

URL Monitoring & Watchlist Timeline

IndexFlow allows site operators to place critical URLs on a high-priority Watchlist to track indexing state transitions over time.

Watchlist Mechanics

  • URLs marked as is_watched = 1 record historical events in the index_status_history table: 
    CREATE TABLE index_status_history (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    url_id INTEGER NOT NULL REFERENCES urls(id) ON DELETE CASCADE,
    provider TEXT NOT NULL,          -- 'google' | 'bing'
    index_status TEXT NOT NULL,      -- 'INDEXED' | 'NOT_INDEXED' | etc.
    coverage_state TEXT,
    last_crawled_at TEXT,
    checked_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
);
  • Only state changes (where status differs from the last recorded check) are inserted, preventing duplicate database entries.

Visual Timeline

The timeline view (/monitoring/:id) combines all lifecycle events for a single URL:

  1. Sitemap Synced: Time discovered and declared lastmod.
  2. SEO Audit: HTTP response code, canonical verdict, and noindex checks.
  3. Submissions: Provider responses and HTTP response codes.
  4. Index Status History: Provider coverage changes and crawl date updates.

Executing Inspection Operations

Trigger Search Analytics Pool Sync (0 Quota)

curl -X POST "https://www.indexflowing.com/docs/api/v1/pipeline/gsc-inspect/sync?site_id=1" \
  -H "Cookie: if_token=<YOUR_JWT_TOKEN>"

Inspect Single URL on Google or Bing

# Query Google Search Console
curl -X POST "https://www.indexflowing.com/docs/api/v1/urls/101/inspect?engine=google" \
  -H "Cookie: if_token=<YOUR_JWT_TOKEN>"

# Query Bing Webmaster Tools
curl -X POST "https://www.indexflowing.com/docs/api/v1/urls/101/inspect?engine=bing" \
  -H "Cookie: if_token=<YOUR_JWT_TOKEN>"

Toggle Watchlist Status

curl -X POST "https://www.indexflowing.com/docs/api/v1/monitoring/101/watch" \
  -H "Content-Type: application/json" \
  -H "Cookie: if_token=<YOUR_JWT_TOKEN>" \
  -d '{"watched": true}'
← Introduction API Reference →