IF
IndexFlowing / Docs
🌐 切换中文文档 GitHub ↗

4. Engine Push & IndexNow

The Engine Push pipeline delivers validated, unindexed URLs directly to search engines using the Bing IndexNow API and the Google Indexing API.

Smart Push Eligibility Rules

IndexFlow eliminates redundant API calls and protects your search engine quotas. A URL is eligible for background submission if and only if it meets all qualification criteria:

┌──────────────────────────────────────────────────────────────┐
│                  Submission Eligibility Gate                 │
├──────────────────────────────────────────────────────────────┤
│ 1. SEO Gate Status   ─── seo_status != 'FAIL'                │
│ 2. Index Status      ─── {provider}_index_status != 'INDEXED'│
│ 3. Push History      ─── {provider}_status IN ('NONE','FAIL')│
│ 4. Engine Readiness  ─── Site credentials configured & active│
└──────────────────────────────────────────────────────────────┘

Worker Claim Queries

Bing Queue Claim (fetch_pending_bing):

SELECT * FROM urls
WHERE seo_status != 'FAIL'
  AND bing_status IN ('NONE', 'FAILED')
  AND bing_index_status != 'INDEXED'
ORDER BY priority ASC, id ASC
LIMIT $1;

Google Queue Claim (fetch_pending_google):

SELECT * FROM urls
WHERE seo_status != 'FAIL'
  AND google_status IN ('NONE', 'FAILED')
  AND gsc_index_status != 'INDEXED'
ORDER BY priority ASC, id ASC
LIMIT $1;

Bing IndexNow Protocol

IndexNow allows webmasters to notify participating search engines (including Bing, Yandex, Naver, and Seznam) about content additions and updates.

1. Verification File

Before submitting, ensure your 32-character hexadecimal key is hosted at the root of your domain:

https://example.com/a1b2c3d4e5f6789012345678abcdef01.txt

The file content must match the key string exactly.

2. Batch Request Structure

IndexFlow batches multiple URLs into a single HTTP POST request to minimize connection overhead:

  • Endpoint: POST https://api.indexnow.org/indexnow
  • Headers: Content-Type: application/json; charset=utf-8
  • Request Body:
{
  "host": "example.com",
  "key": "a1b2c3d4e5f6789012345678abcdef01",
  "urlList": [
    "https://example.com/blog/rust-indexing-guide",
    "https://example.com/docs/installation",
    "https://example.com/pricing"
  ]
}

3. Response Status Evaluation

  • HTTP 200 / 202: Key and URL batch accepted. URLs are marked as bing_status = 'SUBMITTED'.
  • HTTP 400: Invalid format or invalid key location.
  • HTTP 403: Key does not match file or domain ownership verification failed.
  • HTTP 429: Rate limited. The worker halts the batch and logs is_quota_exceeded = true.

Google Indexing API

The Google Indexing API allows site owners to notify Google when pages are added or updated.

1. Authentication & Scope

  • Protocol: OAuth2 Bearer Token generated via RSA-SHA256 JWT assertion.
  • Scope: https://www.googleapis.com/auth/indexing
  • Target Endpoint: POST https://indexing.googleapis.com/v3/urlNotifications:publish

2. Request Body

Google requires individual per-URL notifications:

{
  "url": "https://example.com/blog/rust-indexing-guide",
  "type": "URL_UPDATED"
}

3. 24-Hour Rolling Quota Circuit Breaker

Google enforces a default quota of 200 URL submissions per day. IndexFlow tracks submission volume per site using an exact 24-hour rolling window query:

SELECT COUNT(*), MIN(sl.created_at)
FROM submission_logs sl
JOIN urls u ON u.id = sl.url_id
WHERE sl.provider = 'google'
  AND sl.success = 1
  AND u.site_id = $1
  AND sl.created_at > datetime('now', '-24 hours');

[ Used Submissions in Last 24h >= Daily Quota ]
                       │
                      YES
                       ▼
[ Calculate: oldest_submission_timestamp + 24 hours ]
                       │
                       ▼
[ Set sites.google_quota_paused_until = Next Free Time ]
                       │
                       ▼
[ GoogleSubmitWorker automatically enters standby mode ]

When quota is exhausted:

  1. IndexFlow calculates next_free_at (the timestamp of the oldest submission in the 24-hour window plus 24 hours).
  2. Sets sites.google_quota_paused_until = next_free_at.
  3. The GoogleSubmitWorker skips the site until the earliest quota slot expires, preventing HTTP 429 errors.

Dry-Run Mode Mechanics

To test workflows without consuming official API quotas:

  1. Ensure DRY_RUN=true in your .env configuration.
  2. In the Web Console header, the Dry-Run Badge will be active.
  3. Triggering push operations will generate mock entries in submission_logs: 
    {
  "provider": "google",
  "response_code": 200,
  "response_body": "DRY_RUN_MOCK: 已演练并记录日志，未向 API 发起网络请求",
  "success": 1
}
  4. URL statuses will transition to SUBMITTED locally.

Executing Push Operations

Start Automated Push Pipeline via Web Console

Navigate to /submit and click Submit to All Engines.

Submit a Single URL via REST API

# Push single URL to Bing IndexNow
curl -X POST "https://www.indexflowing.com/docs/api/v1/urls/101/submit?provider=bing" \
  -H "Cookie: if_token=<YOUR_JWT_TOKEN>"

# Push single URL to Google Indexing API
curl -X POST "https://www.indexflowing.com/docs/api/v1/urls/101/submit?provider=google" \
  -H "Cookie: if_token=<YOUR_JWT_TOKEN>"

Batch Submit Selected URLs

curl -X POST "https://www.indexflowing.com/docs/api/v1/urls/batch-submit" \
  -H "Content-Type: application/json" \
  -H "Cookie: if_token=<YOUR_JWT_TOKEN>" \
  -d '{
    "selected_ids": [101, 102, 103],
    "provider": "bing"
  }'

Response:

{
  "submitted_count": 3,
  "success": true
}
← Introduction API Reference →