MCP documentation menu

datapulse_scrape_bulk

OtherScope: datapulse:scrapeRead-only

Submit up to 250 URLs for scraping per request. Asynchronous: the answer is a receipt, not scrape results.

Description

Submit up to 250 URLs for scraping per request. Asynchronous: the answer is a receipt, not scrape results.

Scrapes are public by design: each is a fetch of a public web page, kept in one cache shared by all users, so every result is readable by every user.

LOWER PRIORITY: bulk work runs behind interactive submissions, can take hours during large background runs, and pauses while the extraction model and its overflow capacity are both unavailable. A long-pending URL has not failed, and resubmitting the batch does not make it run sooner. When someone is waiting, datapulse_scrape_submit (wait=true) per URL or datapulse_domain_overview is faster; one URL from a batch is rushed with datapulse_scrape_submit(url=…, force=true, wait=true).

Each item is {url, force?}. Receipt: accepted, cached, rejected (the API’s counts over the URLs sent); sent (index, url as sent, url_hash) for every URL sent; rejected_urls (refused by the API); rejected_local (refused before the request: a malformed item, blank, not a web URL, credentials; NOT counted in rejected); duplicates (index, first_index; sent once); errors. accepted + cached + rejected + len(rejected_local) + len(duplicates) equals the number of URLs. When no item can be sent, nothing is sent and the call is an error whose text is this receipt: every item in rejected_local, the counts 0, the other lists [], and a note.

Results: sent[].url_hash is the key for every URL sent, accepted or cached. datapulse_scrape_bulk_results(url_hashes=[sent[].url_hash, …]) reads them all in one call; its counts.by_status (pending, processing, completed, failed, over every result found) shows how far the batch has got, so the same call repeated with backoff polls the batch.

Reference: datapulse_help(topic="scraping")

Parameters

ParameterTypeDescription
urls
required
object[]
Array of URLs to scrape. Maximum 250 per request.
Min items: 1Max items: 250
urls[].url
required
string
The URL to scrape.
Min length: 1Max length: 2048
urls[].forceboolean
Bypass 90-day cache.
Input schema (JSON)
{
  "additionalProperties": false,
  "properties": {
    "urls": {
      "description": "Array of URLs to scrape. Maximum 250 per request.",
      "items": {
        "additionalProperties": false,
        "properties": {
          "force": {
            "description": "Bypass 90-day cache.",
            "type": "boolean"
          },
          "url": {
            "description": "The URL to scrape.",
            "maxLength": 2048,
            "minLength": 1,
            "type": "string"
          }
        },
        "required": [
          "url"
        ],
        "type": "object"
      },
      "maxItems": 250,
      "minItems": 1,
      "type": "array"
    }
  },
  "required": [
    "urls"
  ],
  "type": "object"
}

Generated from the live server (DataPulse MCP v0.9-341-g27163c2) on October 20, 2026.