Web scraping
Five tools cover web scrapes: three submit pages that a headless browser renders and an LLM extracts structured entity data from (datapulse_domain_overview, datapulse_scrape_submit, datapulse_scrape_bulk), and two read stored results:
| Tool | Purpose |
|---|---|
datapulse_domain_overview | Scrape + DNS + RDAP for a domain in one call: the starting point for domain research |
datapulse_scrape_submit | One URL: synchronous with wait=true, otherwise a job receipt |
datapulse_scrape_bulk | Up to 250 URLs per request, always asynchronous, lower priority |
datapulse_scrape_result | One stored result by url_hash, with the DOM and the full links array on request |
datapulse_scrape_bulk_results | Up to 250 stored results by url_hash, paged |
Results are stored for 90 days; force=true re-scrapes. Live DNS and RDAP lookups are documented in datapulse_help(topic="dns") and datapulse_help(topic="rdap"). Name canonicalization, the hash kinds, paging signals, found: false answers and bulk-receipt accounting are common to every tool: datapulse_help(topic="conventions").
Scrapes are public
Every scrape is a fetch of a public web page: the content anyone on the internet gets by visiting the URL. It sends no cookies or login sessions, and URLs carrying credentials (user:pass@) or an IP address in place of a hostname are refused before anything is fetched. Results are kept in one cache shared by all users of this deployment.
Because the source pages are public, nothing in the cache is secret, and keeping results per caller would protect nothing. Shared reads are the design, not a gap:
datapulse_scrape_resultanddatapulse_scrape_bulk_resultsread any result by its url hash, whoever submitted it.- A URL one user submits is served from the cache to the next user who asks for it, which is what makes repeat lookups instant.
- What a submission reveals is that someone asked about a URL. Any URL submitted here is visible to every user of the deployment.
- A submission carries only the URL and the fetch options (
force,wait,wait_timeout,link_data), and stored rows record nothing about who submitted them.
Domain overview
datapulse_domain_overview(domain="osu.edu") submits whatever is missing (the root URL’s scrape, a DNS lookup, an RDAP lookup), waits for them, and returns:
| Key | Content |
|---|---|
summary | The main facts, described below |
scrape | The root URL’s scrape row (the fields in “Result fields”), or null when no scrape row exists yet; a top-level note then says so |
other_scrapes | Further rows the API returned for the domain, with crawler_policies_present in place of the policy text; absent when there are none |
dns | The same compact summary datapulse_live_dns returns: answer records only, names without a trailing dot, dmarc; null when the API returned no DNS row |
rdap | The RDAP envelope {query, datetime, results} as the API sent it, names and codes case-folded; null when the row carries no envelope |
rdap_status, rdap_error | The RDAP row’s own status, and its error when the lookup failed |
domain, domain_hash, submitted_domain | See datapulse_help(topic="conventions") |
A first call waits up to 45 seconds for the parts it had to submit (a scrape typically takes 5–30 seconds); a part not ready in time comes back null (a missing scrape row with a note). Later calls answer from storage unless force=true. A domain is looked up as given (www. kept); a single label is refused.
summary. resolves, dnssec and registrar_name are always present. dnssec is signed, unsigned, broken (the validating resolver answered SERVFAIL), nonexistent (NXDOMAIN: no zone to judge) or unknown. unknown, with resolves: false, means the DNS lookup failed or returned nothing usable; dns.status says whether false is a finding. registrar_name is looked up from the IANA registrar list by the newest RDAP observation’s ianaid: "unknown" when RDAP failed, and "not assigned (registry without IANA IDs)" for a registry that publishes no IANA registrar IDs (Nominet .uk), with registrar_iana_id then absent. The other fields (registrar_iana_id, registration_date, expiration_date, entity_name, entity_type, nature, country) are absent when their source had no value.
The four extraction fields are passed through exactly as the extractor wrote them, including its placeholders: entity_name: "unknown", entity_type: "other" and country: "unknown" are the extractor declining to classify, not classifications. They describe what the site’s web content claims, not RDAP registrant data, which is GDPR-redacted. country is the extractor’s location_country: it is asked for an ISO 3166-1 alpha-3 code ("USA", "DEU") and nothing validates it, so it can be other wording; RDAP’s country codes have two letters and are not comparable with it.
Page content and links. include_dom=true adds the first 40 KB part of the cleaned DOM. link_data (default domains) selects the link structures, as on datapulse_scrape_submit and datapulse_scrape_result:
link_data | Returns | Typical size |
|---|---|---|
none | nothing | — |
domains (default) | link_domains | ~500 bytes |
links | the links array, first page of 300 records | up to ~37 KB |
both | both | up to ~38 KB |
An unrecognised value or a non-string is refused. A DOM or links array longer than one part carries a note naming the datapulse_scrape_result call for the next part.
For the raw DNS answer (flags, authorities, RRSIGs) use datapulse_live_dns(full=true).
Submitting a URL
datapulse_scrape_submit(url=..., wait=true) blocks until the scrape finishes (wait_timeout, default and maximum 50 seconds) and returns the result. Without wait=true it returns a job receipt {status, job_id, domain_hash, url_hash} and the result is read later with datapulse_scrape_result(url_hash=...). A scrape takes 5–30 seconds once it starts.
What is fetched is the URL as you gave it: www. is kept, and an explicit http:// or https:// is honoured. A URL that names its scheme goes up with the scheme and host case-folded and the host in A-label form (submitted_url echoes your input when that changed it). A URL that names no scheme goes up as written and is tried over https:// first, then http://. The path, query and fragment go up byte for byte.
What is stored is keyed on url_hash (its identity rules: “URL identity” in datapulse_help(topic="conventions")). Every variant of a URL therefore shares one row: within the 90-day window, whichever variant was fetched first answers for all of them, and the row’s url and redirect_chain say what was actually fetched. Paths are significant: example.com/about and example.com/contact are different scrapes. force=true fetches the variant you asked for.
Status values
status | Where | Meaning |
|---|---|---|
queued | async submit receipt | New job accepted; the result comes from datapulse_scrape_result |
in_flight | submit receipt | The URL is already being scraped; the result comes from datapulse_scrape_result |
cached | submit (sync or async), full row inline | A successful scrape stored within the last 90 days, cached: true; no polling needed |
completed | submit (wait=true) or a stored row | The scrape finished; extraction is ready (absent for PDFs and other non-HTML content) |
failed | submit inline, or a stored row | The scrape failed; error says why. A failure stored within the last 7 days keeps failed with cached: true |
pending, processing | stored row | Not finished yet; a later read returns it |
A submit that returns a row (cached, completed, failed) returns it in the shape of datapulse_scrape_result without the DOM; link_data applies to it. Only a scrape that ran for this call reports completed; last_scraped_at is the time of the stored scrape.
A wait=true call whose scrape outlasts wait_timeout is not an error: the tool resubmits the URL without waiting and without force, which starts no second scrape, and answers with the running job’s in_flight receipt (url_hash) and a note. If the scrape finished in between, the full result comes back instead.
Queue priority: interactive vs bulk scrapes
| Submitted via | Class | When it runs |
|---|---|---|
datapulse_scrape_submit, datapulse_domain_overview | Interactive | One of each worker’s scrape slots is reserved for interactive work, so it never waits behind bulk |
datapulse_scrape_bulk | Bulk | Lower priority. Runs only in the slots not reserved for interactive work. During large background runs it can take hours |
datapulse_scrape_bulk suits background work whose results can arrive later. When someone is waiting on the answer, datapulse_scrape_submit(url=..., wait=true) per URL or datapulse_domain_overview is faster: both are processed ahead of datapulse_scrape_bulk submissions. This applies to web scrapes only; datapulse_live_dns_bulk and datapulse_live_rdap_bulk share one queue with their single-lookup forms.
Bulk work can pause
Bulk scrapes pause while the extraction model is unavailable and the day’s overflow extraction capacity is used up. Queued bulk jobs wait or are retried, and resume on their own when either returns, so a batch can stall for as long as an outage lasts on top of waiting behind interactive work. Interactive submissions are not affected.
Polling a bulk batch
A bulk batch is followed by its url hashes:
datapulse_scrape_bulkreturns a receipt whosesent[].url_hashis the key of every URL sent, accepted or cached.datapulse_scrape_bulk_results(url_hashes=[sent[].url_hash, ...])reads those results, up to 250 in one call.- Its
counts.by_statusgivespending,processing,completedandfailedover every result found, on every page, so the first page alone shows progress. The batch is finished whenpendingandprocessingare both 0; eachfailedrow carrieserror.
- A long-pending result is not a failure.
pendingorprocessingmeans the URL is still queued or in progress, however long it has been. Onlystatus: "failed"is a failure. - Polling backs off: a first read after at least 5 seconds, then intervals doubling (5 s → 10 s → 20 s …) up to a few minutes. One
datapulse_scrape_bulk_resultscall covers the whole batch, in place of adatapulse_scrape_resultcall perurl_hash. - Resubmitting the batch does not unstick it:
datapulse_scrape_bulkuses the same lower-priority queue. - One urgent URL is resubmitted interactively with
datapulse_scrape_submit(url=..., force=true, wait=true).force=trueis required: without it the submit attaches to the job still queued from the batch, waits behind it, and ends with that job’sin_flightreceipt.
Bulk submission
datapulse_scrape_bulk(urls=[{"url": ..., "force"?}, ...]) takes up to 250 URLs; any other item field is refused, naming the item, before anything is sent. It returns a receipt, not results. A real receipt for five inputs, where index 1 and 4 are other spellings of index 0, index 3 is not a web URL, and both URLs sent were already stored:
{
"accepted": 0, "rejected": 0, "cached": 2,
"errors": [], "rejected_urls": [],
"rejected_local": [
{"index": 3, "url": "ftp://example.net/file", "error": "invalid urls[3] \"ftp://example.net/file\": unsupported URL scheme in 'ftp://example.net/file' (only http and https)"}
],
"duplicates": [
{"index": 1, "url": "http://WWW.EXAMPLE.COM/", "first_index": 0},
{"index": 4, "url": "example.com", "first_index": 0}
],
"sent": [
{"index": 0, "url": "https://example.com/", "url_hash": "a379a6f6..."},
{"index": 2, "url": "https://example.org/", "url_hash": "bfabc374..."}
]
}
The fields every bulk receipt shares are in “Bulk receipts” in datapulse_help(topic="conventions"); for scrapes:
sentlists every URL that was sent, in the form sent, with theurl_hashits result is stored under. It is the key for reading results back.rejected_urls(index,url,error) is what the API refused; that index has no result.rejected_localis what was refused before the request (a blank item, a non-web scheme, credentials, a scheme-relative URL), with the reason. These are not counted inrejected.accepted + cached + rejected + len(rejected_local) + len(duplicates)equals the number of URLs passed (0 + 2 + 0 + 1 + 2 = 5 above).- When no item can be sent, nothing is sent: the call is an error whose text is this receipt, with every item in
rejected_local, the counts 0, the other lists[], and anote.
Results are read with datapulse_scrape_bulk_results(url_hashes=[sent[].url_hash...]), which covers every URL sent whether accepted or cached; “Polling a bulk batch” above follows a batch to the end.
Reading results
datapulse_scrape_result
datapulse_scrape_result(url_hash=...) returns one stored row. A hash with no row answers found: false with a note.
| Parameter | Type | Required | Description |
|---|---|---|---|
url_hash | string | Yes | 64 hex characters, from a submit receipt or sent[].url_hash |
include_dom | boolean | No | Include the cleaned HTML DOM, 40 KB per part (default false) |
text_offset | integer | No | With include_dom=true: the byte offset of the DOM part, from next_text_offset |
link_data | string | No | none, domains (default), links or both |
links_offset | integer | No | With link_data links or both: the first record of the links page, from links_next_offset |
DOM parts. The cleaned DOM runs from about 1 KB (example.com) to the scraper’s 200 KB cap (www.cnn.com) and comes 40 KB per part: cleaned_dom_total is its size, text_offset where this part starts, and next_text_offset, when present, the offset of the next part, which the note names. Parts never split a UTF-8 character, and joined in order they are the stored DOM byte for byte. An offset past the end returns no cleaned_dom, with cleaned_dom_total and a note naming the DOM’s size.
Links pages. The links array comes 300 records per page (about 37 KB): links_total, links_truncated: true and links_next_offset (passed back as links_offset) appear when the page has more. datapulse_scrape_submit returns the first links page; datapulse_domain_overview returns the first DOM part (with include_dom) and the first links page.
Result fields
| Field | Description |
|---|---|
id | Job ID (UUID) |
domain, domain_hash | The URL’s domain and its canonical hash (datapulse_help(topic="conventions")) |
url | The URL that was fetched, as submitted; the row is shared by every scheme and www. variant, so this says which one produced it |
url_hash | The key the row is stored under |
status, cached | See “Status values”; cached (on rows returned by submit) is true when the row was served from storage rather than scraped for this call. datapulse_scrape_result reads a stored row and carries no cached |
error | Present when status is failed |
final_url | The page as captured after redirects and protocol fallback; a script’s rewrite after load (google.com’s ?zx=) shows here but not in redirect_chain |
redirected | true when final_url differs from the requested URL |
redirect_chain | The main-frame navigation the browser followed, one {url, status, location?, kind, error?} per document response. kind is http (a 30x to location), client (meta refresh or script), final (the rendered page) or error (navigation died there). Absent when nothing was fetched (a DNS failure) |
tls | protocol, issuer, subject, valid_from, valid_to; absent for a plain-HTTP fetch |
quic | QUIC/HTTP3 probe: supported: true with alpn, tls_version, server_addr, handshake_ms; supported: false with error and reason (timeout, tls_rejected, resolve_failed, version_negotiation, stateless_reset, transport_error, application_error, listen_failed, invalid_args, other), plus tls_alert and tls_alert_code for tls_rejected (alert 40 is what a CDN with HTTP/3 disabled returns). Absent when the probe did not run |
server_addr | IP:port of the server that answered |
created_at, completed_at, last_scraped_at | Job submitted, job finished, and the time of the scrape the row holds (earlier than completed_at when served from storage) |
render_time_ms, llm_time_ms, total_time_ms | Timings; llm_time_ms is absent when no extraction ran |
extraction | The LLM-extracted entity data (below); absent for PDFs and other non-HTML content |
crawler_policies | robots_txt, llms_txt, ai_txt, ai_json as found, each a string or absent; {} when none. Collected and returned verbatim, not applied: a Disallow does not stop the scrape. Multi-row reads carry crawler_policies_present (the file names, [] when none) instead of the text, which runs to 50 KB |
link_domains | {internal, external}: internal holds full HOSTNAMES under the page’s own registrable domain (which subdomains a site runs); external the distinct third-party REGISTRABLE DOMAINS (the portfolio signal). Different units, so counts across them are meaningless. Derived from every link; carries links_truncated: true and links_total when the page had more links than the scrape keeps. Links whose host is an IP address are in links only |
links | The page’s referenced URLs, including JS-inserted ones: {url, source, rel?, text?} with source one of a, link, script, iframe, img, media, form, jsonld, meta. The scrape keeps the first 1,500; a rendered page with no links has links: [] |
cleaned_dom | Only with include_dom=true |
submitted_url | On a submit, your URL when the form sent differs from it |
PDF and other non-HTML results have no extraction, llm_time_ms, links or link_domains, since nothing was rendered; a completed result without extraction is such a document. The MCP tools do not return a PDF’s markdown_content.
Extraction data
| Field | Type | Description |
|---|---|---|
is_error | yes/no | The page was an error page |
entity_name | string | Organization or entity name |
entity_type | enum | commercial, non-profit, government, personal, education, religious, media, other |
nature | string | What the page is |
nature_adult | yes/no | Adult content |
nature_domain | string | The domain’s purpose |
location | string | Physical location |
location_country | string | Requested as an ISO 3166-1 alpha-3 code (USA, DEU); not validated, so it can be other wording or unknown |
phone, email | string | Contact details shown on the page; none when the page shows none |
domain_parked | yes/no | The domain is parked |
domain_mismatch | yes/no | The page content does not match what the domain name claims |
domain_random | yes/no | The domain looks randomly generated |
ecommerce | yes/no | E-commerce site |
language | string | ISO 639-3 language code |
bot_detected | yes/no | A bot-detection or CAPTCHA page was served |
The yes/no fields are the strings "yes" and "no".
datapulse_scrape_bulk_results
Up to 250 hashes (one bulk submission) in one call, paged: limit default 15, max 30 (larger values are capped), about 2 KB per result. Results come in url_hashes order. Each result has the fields above except cleaned_dom, the links array and the per-connection fields (final_url, redirected, tls, quic, server_addr), which come only from datapulse_scrape_result; link_domains and crawler_policies_present are included.
{
"total": 2,
"counts": {"not_found": 1, "duplicates": 0,
"by_status": {"pending": 1, "processing": 0, "completed": 1, "failed": 0}},
"returned": 2, "offset": 0, "truncated": false,
"results": [{"url_hash": "a379a6f6...", "status": "completed", "...": "..."}, {"url_hash": "a1b2c3d4...", "status": "pending"}],
"not_found": ["deadbeef..."],
"duplicates": []
}
totalcounts results found across all pages;truncated: true,next_offsetandnotemark a page that is a SUBSET.not_foundlists hashes with no row at all; a URL stillpendingorprocessingis inresults, not here.duplicateslists entries naming a row an earlier entry already asked for (hashes match in any letter case):{index, url_hash, first_index}.not_foundandduplicatescome complete on theoffset=0page;countsgives their sizes on every page.total + counts.not_found + counts.duplicatesequals the number of hashes sent.counts.by_statuscounts every result found bystatus(pending,processing,completed,failed), across all pages and on every page; each key is present, 0 when no row has it, and the four sum tototal. A stored row is nevercached(that status labels a submit answer); a status outside the four would be counted asother, which is otherwise absent.
Comparing two sites
To judge whether a suspicious domain imitates a legitimate one, scrape both with wait=true, then read each with include_dom=true. Similar page structure, copied text (slogans, disclaimers, product descriptions), replicated forms, matching CSS class names or script patterns, and reused images or branding are evidence of impersonation.
datapulse_scrape_submit(url="huntington.com", wait=true)
datapulse_scrape_submit(url="huntingtonbankonline.com", wait=true)
datapulse_scrape_result(url_hash="...", include_dom=true) for each
Errors
A Scrape API failure is returned as an error naming what failed, the HTTP status and the API’s own reason; internal paths and raw bodies stay in the server log. A hash with no stored row is not an error: it answers found: false with a note.
Generated from the live server (DataPulse MCP v0.9-341-g27163c2) on October 20, 2026. Your AI assistant reads this page by calling datapulse_help(topic="scraping").