How names and URLs are normalized

Every tool that takes a domain name, hostname, IP address or URL puts it into one canonical form before anything is sent. What differs between tools is how much of the name is kept, and what each tool refuses. This topic is the single statement of those rules; the per-tool topics refer back to it.

What every name goes through

These apply to every domain / query parameter on every tool:

The U-label and A-label spellings of a name are the same lookup. A run of trailing dots is input noise, not an empty label: example.com.. is example.com, never an error. An empty label anywhere else (a..b.com, .example.com) is refused, and so is whitespace inside a name (exa mple.com).

Refused everywhere a domain is required: a URL (https://example.com/path), a port (example.com:8080), credentials, an IP literal (except where the tool takes addresses, below), a label over 63 bytes or a name over 253, a label starting or ending with a hyphen. The refusal names the reason and nothing is sent.

Most lookup tools are lenient about one thing (datapulse_disputedb_lookup is strict): names that registries actually sold but IDNA 2008 forbids (emoji and symbol names, raw or as xn-- punycode) are accepted, because they exist and must be queryable.

Three models: how much of the name is kept

“Hostname kept” does not mean “sent as typed”: case, whitespace, IDN spelling, Unicode separators and trailing dots are still canonicalized. It means the hostname level is preserved rather than reduced to the registrable domain.

Two tools in the middle family add their own handling downstream, without this server changing what you sent: the domain-health probe checks both the name and its www. sibling, and the DisputeDB API matches a leading www. away itself, so www.tesla.com and tesla.com find the same decisions (the response’s query still shows the name as sent).

URL identity (url_hash)

A scrape result is keyed by a hash of the URL’s identity form, which normalizes more than the request does. These are the same stored page: http:// and https://; with and without a leading www.; with and without the default port (:80, :443); with and without a #fragment; query parameters in any order; percent-encoded and literal spellings of the same character; paths with and without ./.. segments; U-label and A-label hosts. A non-default port (:8443) is part of the identity and is kept.

Single labels and underscores

submitted_domain / submitted_url: what you sent, when it changed

When the name that was looked up differs from what you typed by more than ASCII case — an IDN folded to punycode, a trailing dot dropped, a hostname reduced to its registrable domain — the response carries submitted_domain (or submitted_url) with your original input beside the result. It is absent when nothing changed. This matters for homographs: pasting instagrạm.com out of a phishing sample must not silently produce a report that reads as if it were about instagram.com.

Bulk receipts: where every input is accounted for

datapulse_live_dns_bulk, datapulse_live_rdap_bulk and datapulse_scrape_bulk validate every item before the request. One bad item never costs you the batch (a batch with nothing usable is an error, since there is nothing to send).

Every index is the position in the array you passed. To account for every input:

accepted + cached + rejected + len(rejected_local) + len(duplicates) = number of items you passed

(duplicates is absent, so zero, for datapulse_scrape_bulk.)

To find everything that failed, read both rejected_items (rejected_urls) and rejected_local. An empty rejected_items with rejected: 0 means the API refused nothing; it says nothing about what was refused before the request.