Conventions shared by every tool

The rules here apply across the DataPulse tools: how names and URLs are put into canonical form, the hashes that key stored results, how list answers page, what an answer that finds nothing looks like, how a bulk receipt accounts for every input, what is cached, and which access each tool needs. Tool topics refer back to these sections instead of restating them.

Names and URLs

What every name goes through

Every domain / query parameter is put into one canonical form before anything is sent:

The U-label and A-label spellings of a name are the same lookup. A run of trailing dots is input noise, not an empty label: example.com.. is example.com. An empty label anywhere else (a..b.com, .example.com) is refused, and so is whitespace inside a name (exa mple.com).

Refused wherever a domain is required: a URL (https://example.com/path), a port (example.com:8080), credentials, an IP literal (except where a tool takes addresses), a label over 63 bytes or a name over 253, a label starting or ending with a hyphen. The refusal names the reason and nothing is sent.

Names that registries sold but IDNA 2008 forbids (emoji and symbol names, raw or as xn-- punycode) pass local validation in every tool except datapulse_disputedb_lookup, which is strict. The Scrape API then refuses them for scrapes, the domain overview, live DNS and domain health (invalid_url or invalid_domain; invalid_input on a bulk item, as the DNS bulk example in datapulse_help(topic="dns") shows for xn--ls8h.la). Live RDAP accepts them.

Three models: how much of the name is kept

Two hostname-kept tools add handling downstream without changing what is sent: the domain-health probe checks both the name and its www. sibling, and DisputeDB matches a leading www. away itself, so www.tesla.com and tesla.com find the same decisions (the response’s query shows the name as sent).

Single labels and underscores

submitted_domain / submitted_url

When the name looked up differs from the input by more than ASCII case (an IDN folded to punycode, a trailing dot dropped, a hostname reduced to its registrable domain), the response carries submitted_domain (submitted_url for datapulse_scrape_submit) with the original input. It is absent when nothing changed, and it keeps a homograph such as instagrạm.com from reading as a report about instagram.com.

Hashes

Stored results are keyed by SHA-256 hashes of canonical forms. Each is returned by the tool that creates the row, and reading a row back by its hash makes no new lookup.

A root URL’s url_hash equals its domain’s domain_hash: both hash example.com. The hashes are returned as data; computing them outside the server is not needed.

URL identity

The identity form behind url_hash normalizes more than the request does. These are the same stored page: http:// and https://; with and without a leading www.; host case and U-label / A-label spellings; with and without the default port (:80, :443); with and without a #fragment; a bare trailing / on the root; query parameters in any order; ./.. path segments and their resolved form; percent- encoded and literal spellings of the same character. Path case and a trailing / on a non-root path are significant, and a non-default port is part of the identity.

Paging

List answers are bounded. Each list tool’s description states its default and largest page and the size of a row.

• limit absent, 0 or negative means the default; above the maximum it is capped, not refused. A negative offset is refused. The same offset returns the same rows.
• complete: false is a different signal: the source stopped early (a back-end result cap or compute limit), so paging cannot reach the rest. The remedy is a narrower query, as the hint says. CSV tools put it as a hint line, never inside the rows.
• A nested list that is capped inside a row carries <field>_total and <field>_truncated beside it.
• Long text comes in parts: part=N, or text_offset / next_text_offset, each part naming the call for the next.

Arguments

• An integer or boolean sent as a string ("20", "true", "0.9") is read as the value it spells. A fraction where an integer is required, a value outside the 32-bit range, NaN and infinity are refused with the value sent.
• JSON null is an absent argument. When a tool chooses a mode from “exactly one of” selectors, a blank string (empty or whitespace) selects nothing either.
• An argument that belongs to the mode not chosen is refused when it selects something; one holding its documented default (offset: 0, force: false) selects nothing and is accepted.
• An unknown argument is refused with the list of accepted ones. String arguments have a maximum length, counted in characters; a value echoed in an error is cut to 64 characters.

Answers that find nothing

“No data for this input” is an answer, not an error:

• Lookups by name or hash (datapulse_dns_dphistory, datapulse_dns_rdap_history, stored-row reads by domain_hash, dns_query_hash, url_hash, and datapulse_scrape_list(batch_id=...) for an unknown batch) answer found: false with the identifying field, the back end’s own reason as detail when it gave one, and a note naming the next call.
• datapulse_disputedb_lookup with a case number that does not exist answers found: false with total: 0.
• CSV tools (datapulse_dns_dptechsim, datapulse_dns_searchlabels) answer with the header row and a hint line.
• datapulse_scrape_bulk_results lists hashes with no row in not_found.
• An empty list is [], never null.

Bulk receipts

datapulse_live_dns_bulk, datapulse_live_rdap_bulk and datapulse_scrape_bulk validate every item before the request; one bad item never costs the batch (a batch with nothing usable is an error).

Every index is the position in the array passed. Every input is accounted for:

accepted + cached + rejected + len(rejected_local) + len(duplicates) = number of items passed

Finding every failure takes both rejected_items (rejected_urls) and rejected_local. Each tool’s topic names how its results are read back.

Caching

Access and scopes

Sign-in is OAuth 2.1 with PKCE (S256). The server’s /.well-known/oauth-protected-resource document names the authorization server. Clients identify themselves with a Client ID Metadata Document; the server offers no dynamic client registration. The access token goes in the Authorization: Bearer header only (query-string and form tokens are refused), and the server reads the caller’s permissions from it.

Every request needs the datapulse:mcp permission; without it the answer is 403 insufficient_scope. Each tool also needs the scope below, if any:

A tool call runs for at most 120 seconds. Each account may have only a few slow calls (scrapes with wait=true, domain overviews, domain-health checks, bulk submissions) running at once; one more is refused as a tool error saying how many are running.