MCP documentation menu

Web scraping

Five tools cover web scrapes: three submit pages that a headless browser renders and an LLM extracts structured entity data from (datapulse_domain_overview, datapulse_scrape_submit, datapulse_scrape_bulk), and two read stored results:

ToolPurpose
datapulse_domain_overviewScrape + DNS + RDAP for a domain in one call: the starting point for domain research
datapulse_scrape_submitOne URL: synchronous with wait=true, otherwise a job receipt
datapulse_scrape_bulkUp to 250 URLs per request, always asynchronous, lower priority
datapulse_scrape_resultOne stored result by url_hash, with the DOM and the full links array on request
datapulse_scrape_bulk_resultsUp to 250 stored results by url_hash, paged

Results are stored for 90 days; force=true re-scrapes. Live DNS and RDAP lookups are documented in datapulse_help(topic="dns") and datapulse_help(topic="rdap"). Name canonicalization, the hash kinds, paging signals, found: false answers and bulk-receipt accounting are common to every tool: datapulse_help(topic="conventions").

Scrapes are public

Every scrape is a fetch of a public web page: the content anyone on the internet gets by visiting the URL. It sends no cookies or login sessions, and URLs carrying credentials (user:pass@) or an IP address in place of a hostname are refused before anything is fetched. Results are kept in one cache shared by all users of this deployment.

Because the source pages are public, nothing in the cache is secret, and keeping results per caller would protect nothing. Shared reads are the design, not a gap:

  • datapulse_scrape_result and datapulse_scrape_bulk_results read any result by its url hash, whoever submitted it.
  • A URL one user submits is served from the cache to the next user who asks for it, which is what makes repeat lookups instant.
  • What a submission reveals is that someone asked about a URL. Any URL submitted here is visible to every user of the deployment.
  • A submission carries only the URL and the fetch options (force, wait, wait_timeout, link_data), and stored rows record nothing about who submitted them.

Domain overview

datapulse_domain_overview(domain="osu.edu") submits whatever is missing (the root URL’s scrape, a DNS lookup, an RDAP lookup), waits for them, and returns:

KeyContent
summaryThe main facts, described below
scrapeThe root URL’s scrape row (the fields in “Result fields”), or null when no scrape row exists yet; a top-level note then says so
other_scrapesFurther rows the API returned for the domain, with crawler_policies_present in place of the policy text; absent when there are none
dnsThe same compact summary datapulse_live_dns returns: answer records only, names without a trailing dot, dmarc; null when the API returned no DNS row
rdapThe RDAP envelope {query, datetime, results} as the API sent it, names and codes case-folded; null when the row carries no envelope
rdap_status, rdap_errorThe RDAP row’s own status, and its error when the lookup failed
domain, domain_hash, submitted_domainSee datapulse_help(topic="conventions")

A first call waits up to 45 seconds for the parts it had to submit (a scrape typically takes 5–30 seconds); a part not ready in time comes back null (a missing scrape row with a note). Later calls answer from storage unless force=true. A domain is looked up as given (www. kept); a single label is refused.

summary. resolves, dnssec and registrar_name are always present. dnssec is signed, unsigned, broken (the validating resolver answered SERVFAIL), nonexistent (NXDOMAIN: no zone to judge) or unknown. unknown, with resolves: false, means the DNS lookup failed or returned nothing usable; dns.status says whether false is a finding. registrar_name is looked up from the IANA registrar list by the newest RDAP observation’s ianaid: "unknown" when RDAP failed, and "not assigned (registry without IANA IDs)" for a registry that publishes no IANA registrar IDs (Nominet .uk), with registrar_iana_id then absent. The other fields (registrar_iana_id, registration_date, expiration_date, entity_name, entity_type, nature, country) are absent when their source had no value.

The four extraction fields are passed through exactly as the extractor wrote them, including its placeholders: entity_name: "unknown", entity_type: "other" and country: "unknown" are the extractor declining to classify, not classifications. They describe what the site’s web content claims, not RDAP registrant data, which is GDPR-redacted. country is the extractor’s location_country: it is asked for an ISO 3166-1 alpha-3 code ("USA", "DEU") and nothing validates it, so it can be other wording; RDAP’s country codes have two letters and are not comparable with it.

Page content and links. include_dom=true adds the first 40 KB part of the cleaned DOM. link_data (default domains) selects the link structures, as on datapulse_scrape_submit and datapulse_scrape_result:

link_dataReturnsTypical size
nonenothing—
domains (default)link_domains~500 bytes
linksthe links array, first page of 300 recordsup to ~37 KB
bothbothup to ~38 KB

An unrecognised value or a non-string is refused. A DOM or links array longer than one part carries a note naming the datapulse_scrape_result call for the next part.

For the raw DNS answer (flags, authorities, RRSIGs) use datapulse_live_dns(full=true).

Submitting a URL

datapulse_scrape_submit(url=..., wait=true) blocks until the scrape finishes (wait_timeout, default and maximum 50 seconds) and returns the result. Without wait=true it returns a job receipt {status, job_id, domain_hash, url_hash} and the result is read later with datapulse_scrape_result(url_hash=...). A scrape takes 5–30 seconds once it starts.

What is fetched is the URL as you gave it: www. is kept, and an explicit http:// or https:// is honoured. A URL that names its scheme goes up with the scheme and host case-folded and the host in A-label form (submitted_url echoes your input when that changed it). A URL that names no scheme goes up as written and is tried over https:// first, then http://. The path, query and fragment go up byte for byte.

What is stored is keyed on url_hash (its identity rules: “URL identity” in datapulse_help(topic="conventions")). Every variant of a URL therefore shares one row: within the 90-day window, whichever variant was fetched first answers for all of them, and the row’s url and redirect_chain say what was actually fetched. Paths are significant: example.com/about and example.com/contact are different scrapes. force=true fetches the variant you asked for.

Status values

statusWhereMeaning
queuedasync submit receiptNew job accepted; the result comes from datapulse_scrape_result
in_flightsubmit receiptThe URL is already being scraped; the result comes from datapulse_scrape_result
cachedsubmit (sync or async), full row inlineA successful scrape stored within the last 90 days, cached: true; no polling needed
completedsubmit (wait=true) or a stored rowThe scrape finished; extraction is ready (absent for PDFs and other non-HTML content)
failedsubmit inline, or a stored rowThe scrape failed; error says why. A failure stored within the last 7 days keeps failed with cached: true
pending, processingstored rowNot finished yet; a later read returns it

A submit that returns a row (cached, completed, failed) returns it in the shape of datapulse_scrape_result without the DOM; link_data applies to it. Only a scrape that ran for this call reports completed; last_scraped_at is the time of the stored scrape.

A wait=true call whose scrape outlasts wait_timeout is not an error: the tool resubmits the URL without waiting and without force, which starts no second scrape, and answers with the running job’s in_flight receipt (url_hash) and a note. If the scrape finished in between, the full result comes back instead.

Queue priority: interactive vs bulk scrapes

Submitted viaClassWhen it runs
datapulse_scrape_submit, datapulse_domain_overviewInteractiveOne of each worker’s scrape slots is reserved for interactive work, so it never waits behind bulk
datapulse_scrape_bulkBulkLower priority. Runs only in the slots not reserved for interactive work. During large background runs it can take hours

datapulse_scrape_bulk suits background work whose results can arrive later. When someone is waiting on the answer, datapulse_scrape_submit(url=..., wait=true) per URL or datapulse_domain_overview is faster: both are processed ahead of datapulse_scrape_bulk submissions. This applies to web scrapes only; datapulse_live_dns_bulk and datapulse_live_rdap_bulk share one queue with their single-lookup forms.

Bulk work can pause

Bulk scrapes pause while the extraction model is unavailable and the day’s overflow extraction capacity is used up. Queued bulk jobs wait or are retried, and resume on their own when either returns, so a batch can stall for as long as an outage lasts on top of waiting behind interactive work. Interactive submissions are not affected.

Polling a bulk batch

A bulk batch is followed by its url hashes:

  1. datapulse_scrape_bulk returns a receipt whose sent[].url_hash is the key of every URL sent, accepted or cached.
  2. datapulse_scrape_bulk_results(url_hashes=[sent[].url_hash, ...]) reads those results, up to 250 in one call.
  3. Its counts.by_status gives pending, processing, completed and failed over every result found, on every page, so the first page alone shows progress. The batch is finished when pending and processing are both 0; each failed row carries error.
  • A long-pending result is not a failure. pending or processing means the URL is still queued or in progress, however long it has been. Only status: "failed" is a failure.
  • Polling backs off: a first read after at least 5 seconds, then intervals doubling (5 s → 10 s → 20 s …) up to a few minutes. One datapulse_scrape_bulk_results call covers the whole batch, in place of a datapulse_scrape_result call per url_hash.
  • Resubmitting the batch does not unstick it: datapulse_scrape_bulk uses the same lower-priority queue.
  • One urgent URL is resubmitted interactively with datapulse_scrape_submit(url=..., force=true, wait=true). force=true is required: without it the submit attaches to the job still queued from the batch, waits behind it, and ends with that job’s in_flight receipt.

Bulk submission

datapulse_scrape_bulk(urls=[{"url": ..., "force"?}, ...]) takes up to 250 URLs; any other item field is refused, naming the item, before anything is sent. It returns a receipt, not results. A real receipt for five inputs, where index 1 and 4 are other spellings of index 0, index 3 is not a web URL, and both URLs sent were already stored:

{
  "accepted": 0, "rejected": 0, "cached": 2,
  "errors": [], "rejected_urls": [],
  "rejected_local": [
    {"index": 3, "url": "ftp://example.net/file", "error": "invalid urls[3] \"ftp://example.net/file\": unsupported URL scheme in 'ftp://example.net/file' (only http and https)"}
  ],
  "duplicates": [
    {"index": 1, "url": "http://WWW.EXAMPLE.COM/", "first_index": 0},
    {"index": 4, "url": "example.com", "first_index": 0}
  ],
  "sent": [
    {"index": 0, "url": "https://example.com/", "url_hash": "a379a6f6..."},
    {"index": 2, "url": "https://example.org/", "url_hash": "bfabc374..."}
  ]
}

The fields every bulk receipt shares are in “Bulk receipts” in datapulse_help(topic="conventions"); for scrapes:

  • sent lists every URL that was sent, in the form sent, with the url_hash its result is stored under. It is the key for reading results back.
  • rejected_urls (index, url, error) is what the API refused; that index has no result.
  • rejected_local is what was refused before the request (a blank item, a non-web scheme, credentials, a scheme-relative URL), with the reason. These are not counted in rejected.
  • accepted + cached + rejected + len(rejected_local) + len(duplicates) equals the number of URLs passed (0 + 2 + 0 + 1 + 2 = 5 above).
  • When no item can be sent, nothing is sent: the call is an error whose text is this receipt, with every item in rejected_local, the counts 0, the other lists [], and a note.

Results are read with datapulse_scrape_bulk_results(url_hashes=[sent[].url_hash...]), which covers every URL sent whether accepted or cached; “Polling a bulk batch” above follows a batch to the end.

Reading results

datapulse_scrape_result

datapulse_scrape_result(url_hash=...) returns one stored row. A hash with no row answers found: false with a note.

ParameterTypeRequiredDescription
url_hashstringYes64 hex characters, from a submit receipt or sent[].url_hash
include_dombooleanNoInclude the cleaned HTML DOM, 40 KB per part (default false)
text_offsetintegerNoWith include_dom=true: the byte offset of the DOM part, from next_text_offset
link_datastringNonone, domains (default), links or both
links_offsetintegerNoWith link_data links or both: the first record of the links page, from links_next_offset

DOM parts. The cleaned DOM runs from about 1 KB (example.com) to the scraper’s 200 KB cap (www.cnn.com) and comes 40 KB per part: cleaned_dom_total is its size, text_offset where this part starts, and next_text_offset, when present, the offset of the next part, which the note names. Parts never split a UTF-8 character, and joined in order they are the stored DOM byte for byte. An offset past the end returns no cleaned_dom, with cleaned_dom_total and a note naming the DOM’s size.

Links pages. The links array comes 300 records per page (about 37 KB): links_total, links_truncated: true and links_next_offset (passed back as links_offset) appear when the page has more. datapulse_scrape_submit returns the first links page; datapulse_domain_overview returns the first DOM part (with include_dom) and the first links page.

Result fields

FieldDescription
idJob ID (UUID)
domain, domain_hashThe URL’s domain and its canonical hash (datapulse_help(topic="conventions"))
urlThe URL that was fetched, as submitted; the row is shared by every scheme and www. variant, so this says which one produced it
url_hashThe key the row is stored under
status, cachedSee “Status values”; cached (on rows returned by submit) is true when the row was served from storage rather than scraped for this call. datapulse_scrape_result reads a stored row and carries no cached
errorPresent when status is failed
final_urlThe page as captured after redirects and protocol fallback; a script’s rewrite after load (google.com’s ?zx=) shows here but not in redirect_chain
redirectedtrue when final_url differs from the requested URL
redirect_chainThe main-frame navigation the browser followed, one {url, status, location?, kind, error?} per document response. kind is http (a 30x to location), client (meta refresh or script), final (the rendered page) or error (navigation died there). Absent when nothing was fetched (a DNS failure)
tlsprotocol, issuer, subject, valid_from, valid_to; absent for a plain-HTTP fetch
quicQUIC/HTTP3 probe: supported: true with alpn, tls_version, server_addr, handshake_ms; supported: false with error and reason (timeout, tls_rejected, resolve_failed, version_negotiation, stateless_reset, transport_error, application_error, listen_failed, invalid_args, other), plus tls_alert and tls_alert_code for tls_rejected (alert 40 is what a CDN with HTTP/3 disabled returns). Absent when the probe did not run
server_addrIP:port of the server that answered
created_at, completed_at, last_scraped_atJob submitted, job finished, and the time of the scrape the row holds (earlier than completed_at when served from storage)
render_time_ms, llm_time_ms, total_time_msTimings; llm_time_ms is absent when no extraction ran
extractionThe LLM-extracted entity data (below); absent for PDFs and other non-HTML content
crawler_policiesrobots_txt, llms_txt, ai_txt, ai_json as found, each a string or absent; {} when none. Collected and returned verbatim, not applied: a Disallow does not stop the scrape. Multi-row reads carry crawler_policies_present (the file names, [] when none) instead of the text, which runs to 50 KB
link_domains{internal, external}: internal holds full HOSTNAMES under the page’s own registrable domain (which subdomains a site runs); external the distinct third-party REGISTRABLE DOMAINS (the portfolio signal). Different units, so counts across them are meaningless. Derived from every link; carries links_truncated: true and links_total when the page had more links than the scrape keeps. Links whose host is an IP address are in links only
linksThe page’s referenced URLs, including JS-inserted ones: {url, source, rel?, text?} with source one of a, link, script, iframe, img, media, form, jsonld, meta. The scrape keeps the first 1,500; a rendered page with no links has links: []
cleaned_domOnly with include_dom=true
submitted_urlOn a submit, your URL when the form sent differs from it

PDF and other non-HTML results have no extraction, llm_time_ms, links or link_domains, since nothing was rendered; a completed result without extraction is such a document. The MCP tools do not return a PDF’s markdown_content.

Extraction data

FieldTypeDescription
is_erroryes/noThe page was an error page
entity_namestringOrganization or entity name
entity_typeenumcommercial, non-profit, government, personal, education, religious, media, other
naturestringWhat the page is
nature_adultyes/noAdult content
nature_domainstringThe domain’s purpose
locationstringPhysical location
location_countrystringRequested as an ISO 3166-1 alpha-3 code (USA, DEU); not validated, so it can be other wording or unknown
phone, emailstringContact details shown on the page; none when the page shows none
domain_parkedyes/noThe domain is parked
domain_mismatchyes/noThe page content does not match what the domain name claims
domain_randomyes/noThe domain looks randomly generated
ecommerceyes/noE-commerce site
languagestringISO 639-3 language code
bot_detectedyes/noA bot-detection or CAPTCHA page was served

The yes/no fields are the strings "yes" and "no".

datapulse_scrape_bulk_results

Up to 250 hashes (one bulk submission) in one call, paged: limit default 15, max 30 (larger values are capped), about 2 KB per result. Results come in url_hashes order. Each result has the fields above except cleaned_dom, the links array and the per-connection fields (final_url, redirected, tls, quic, server_addr), which come only from datapulse_scrape_result; link_domains and crawler_policies_present are included.

{
  "total": 2,
  "counts": {"not_found": 1, "duplicates": 0,
             "by_status": {"pending": 1, "processing": 0, "completed": 1, "failed": 0}},
  "returned": 2, "offset": 0, "truncated": false,
  "results": [{"url_hash": "a379a6f6...", "status": "completed", "...": "..."}, {"url_hash": "a1b2c3d4...", "status": "pending"}],
  "not_found": ["deadbeef..."],
  "duplicates": []
}
  • total counts results found across all pages; truncated: true, next_offset and note mark a page that is a SUBSET.
  • not_found lists hashes with no row at all; a URL still pending or processing is in results, not here.
  • duplicates lists entries naming a row an earlier entry already asked for (hashes match in any letter case): {index, url_hash, first_index}.
  • not_found and duplicates come complete on the offset=0 page; counts gives their sizes on every page. total + counts.not_found + counts.duplicates equals the number of hashes sent.
  • counts.by_status counts every result found by status (pending, processing, completed, failed), across all pages and on every page; each key is present, 0 when no row has it, and the four sum to total. A stored row is never cached (that status labels a submit answer); a status outside the four would be counted as other, which is otherwise absent.

Comparing two sites

To judge whether a suspicious domain imitates a legitimate one, scrape both with wait=true, then read each with include_dom=true. Similar page structure, copied text (slogans, disclaimers, product descriptions), replicated forms, matching CSS class names or script patterns, and reused images or branding are evidence of impersonation.

datapulse_scrape_submit(url="huntington.com", wait=true)
datapulse_scrape_submit(url="huntingtonbankonline.com", wait=true)
datapulse_scrape_result(url_hash="...", include_dom=true)   for each

Errors

A Scrape API failure is returned as an error naming what failed, the HTTP status and the API’s own reason; internal paths and raw bodies stay in the server log. A hash with no stored row is not an error: it answers found: false with a note.

Generated from the live server (DataPulse MCP v0.9-341-g27163c2) on October 20, 2026. Your AI assistant reads this page by calling datapulse_help(topic="scraping").