URL Parser

RFC 3986 was published in 2005 and describes a grammar. The WHATWG URL Standard describes an algorithm, and its stated first goal is to "align RFC 3986 and RFC 3987 with contemporary implementations and obsolete the RFCs in the process". Browsers and anything built on the JavaScript URL class implement the algorithm; many server-side libraries in other languages still implement the grammar.

The divergence is not theoretical. The WHATWG basic URL parser begins by removing every ASCII tab and newline from anywhere in the input, and stripping leading and trailing C0 controls and spaces. Feed it http://exa followed by a newline and mple.com/x and it returns http://example.com/x without complaint. A grammar-driven parser either rejects that string or keeps the newline inside the host.

That becomes exploitable when two parsers sit in the same request path. A gateway that makes a routing or access decision with one implementation, in front of an application that resolves the target with another, can be made to disagree about which host was requested. Compare what your own stack extracts against the components shown here before assuming a URL is unambiguous.

host includes the port; hostname never does. For IPv6 the brackets belong to the hostname: http://[::1]:8080/ gives a hostname of [::1] and a host of [::1]:8080. Passing hostname straight into a DNS lookup, or comparing it against a bare ::1, fails for exactly that reason.

The port is dropped whenever it is the default for the scheme, so https://example.com:443/ has a port of the empty string rather than 443. Any code deciding whether to append a port needs to fall back on the scheme instead of assuming the field is populated.

origin is scheme plus host plus non-default port and nothing else — no userinfo, no path. URLs without a tuple origin serialise to the literal string "null": both file:///tmp/a and foo://bar/baz return it. That is a four-character truthy string, not the value null, and comparing origins without accounting for it is a recurring bug. The fragment, by contrast, is never transmitted at all — RFC 3986 §3.5 separates it before dereferencing — so anything after # was resolved entirely on the client.

One asymmetry catches people reassembling URLs by hand. search and hash return the empty string when the component is absent and when it is present but empty, so a URL ending in a bare ? or # reports nothing for that field even though href still shows the delimiter. Rebuilding from the individual getters drops a trailing delimiter that some signature schemes and cache keys treat as significant; reading href preserves it.

The WHATWG standard lists exactly six special schemes and their default ports: ftp (21), file (none), http (80), https (443), ws (80) and wss (443). Everything else is non-special, and the parsing rules change underneath you.

A non-special scheme may have an opaque path with no authority component at all. Parsing mailto:a@b.test yields an empty host and hostname, with a@b.test sitting in pathname. Code that reads hostname to route or authorise receives an empty string rather than an error, which is why a host check is only meaningful once you know the scheme has a host. Other non-special schemes do carry an authority: git+ssh://git@host.test/repo.git parses with a username of git and a hostname of host.test.

The scheme grammar is wider than most people assume. RFC 3986 §3.1 permits a letter followed by any mix of letters, digits, plus, period and hyphen, so web+myapp: is structurally valid. Schemes and hosts are lowercased during parsing while path, query and fragment keep their case, so HTTPS://ExAmPle.COM/PaTh normalises to https://example.com/PaTh.

Relative references are resolved by the algorithm in RFC 3986 §5.2, and the trailing slash on the base decides the outcome. Resolving b against https://x.test/a/c produces https://x.test/a/b, because the base's last segment is replaced. Against https://x.test/a/c/ the same input produces https://x.test/a/c/b. A leading slash discards the base path entirely, and a leading // discards the host as well.

The remove_dot_segments routine in RFC 3986 §5.2.4 then collapses . and .. and clamps at the root, so https://x.test/a/b/../../../c normalises to https://x.test/c rather than climbing above the origin. This happens during parsing; the pathname you read has already been rewritten.

WHATWG goes one step further. It defines a double-dot path segment as .. or any case-insensitive percent-encoded spelling of it — .%2e, %2e., %2e%2e — so /a/%2e%2e/b normalises to /b. RFC 3986 performs no such decoding before removing dot segments. A traversal filter reading the raw string and a server that normalises first are therefore inspecting two different paths.

Parsing https://example.com@evil.test/ gives a hostname of evil.test. Everything up to the last @ in the authority is userinfo; the host is whatever follows it. RFC 3986 gives this its own section — §7.6, "Semantic Attacks" — noting that userinfo "can be used to construct a URI intended to mislead a human user by appearing to identify one (trusted) naming authority while actually identifying a different authority".

§3.2.1 separately deprecates the user:password form and says applications should reject storing it unencrypted. Credentials placed in a URL end up in browser history, Referer headers and every access log along the path. Finding them while parsing is a finding, not a formatting quirk.

The other place a host is not what it appears to be is internationalised domains. The hostname getter returns the ASCII form, so https://exämple.test/ reads back as xn--exmple-cua.test, and a Cyrillic а (U+0430) substituted for a Latin a turns a string that renders identically to apple.com into xn--pple-43d.com. Compare the punycode form, never the display form. When you need to decide whether a parsed URL is one you should accept, the URL validator applies scheme and host rules on top of the parse.