datapulse_scrape_bulk

Submit multiple URLs for scraping in a single request (up to 1000).

LOWER PRIORITY: Bulk submissions run at lower priority than single submissions. They are processed behind interactive work and can take hours during large background runs. Bulk work also pauses while the extraction model is unavailable, and resumes automatically when it recovers, so some URLs can stay pending or processing much longer than usual. That is not a failure: keep polling with backoff, and do not resubmit the batch, which will not make it run sooner. For interactive or time-sensitive work (someone waiting on the answer), call datapulse_scrape_submit per URL (wait=true) or datapulse_domain_overview instead. To rush one URL already in a bulk batch, resubmit it with datapulse_scrape_submit(url=…, force=true, wait=true); without force=true it waits for the queued job and usually times out.

This tool is asynchronous: it returns job receipts, not scrape results. Results are read with datapulse_scrape_bulk_results(url_hashes=[…]) for many at once, datapulse_scrape_list(batch_id=…) for the whole batch, or datapulse_scrape_result(url_hash=…) for one. A scrape takes 5-30 seconds once it starts, but may wait in the queue far longer first.

For DNS/RDAP lookups, use datapulse_live_dns_bulk or datapulse_live_rdap_bulk.

Returns:

• accepted/rejected/cached: Submission counts
• batch_id: UUID grouping all accepted URLs in this submission — use with datapulse_scrape_list(batch_id=…) to retrieve all results at once. NOTE: Cached URLs (already cached/in-flight) are excluded from the batch — only newly accepted URLs appear in batch results.
• url_hashes: the API’s flat list of hashes for the accepted and cached URLs, in no per-input order; use sent[].url_hash to map a hash to its input
• domain_hashes: Array of domain hashes for tracking
• job_ids: Array of job IDs for tracking
• errors: the API’s own error codes for the URLs it rejected
• rejected_urls: one entry per input the API refused: index (position in your urls array), url as sent, and error code
• rejected_local: one entry per input refused before the request because it is blank or not a URL (unsupported scheme, no host, malformed host): index, url and error (the reason); these are NOT counted in rejected — read both lists to find every failure. One bad item never fails the batch
• sent: one entry per URL submitted: index (position in your urls array), url (the form it was sent in: a URL naming http/https with scheme and host case-folded; one without a scheme as written, so it is tried over https, then http) and url_hash (the key its result is stored under; pass it to datapulse_scrape_result or datapulse_scrape_bulk_results). An entry whose index is also in rejected_urls was refused by the API and has no result

For usage guidance, call datapulse_help(topic="scraping")