Rollout of Happy Eyeballs v3 in Firefox

Managing the cardinality explosion through connection racing

HTTP Workshop 2026  ·  Max Inden · Mozilla

A cardinality explosion

Connecting to a modern origin means choosing among:

  • IPv4 or IPv6
  • TCP + TLS or QUIC
  • DNS A, AAAA or HTTPS records
  • Alt-Svc header or HTTPS-record ALPN
  • ECH or plaintext SNI

Part 1: Resolution

Fire the DNS queries together (HTTPS, AAAA, A), then move on as soon as one of these holds:

Either

  • some positive address answers are in, and
  • the preferred family has answered (positive or negative), and
  • the HTTPS/SVCB answer is in (or negative).

Or

  • some positive address answers are in, and
  • the Resolution Delay has passed with nothing more arriving.

Source: draft-ietf-happy-happyeyeballs-v3, §4.2.

Part 2: Connection attempts

  1. Sort the endpoints: by protocol + security (ALPN, ECH), then service priority, then RFC 6724, interleaving address families.
  2. Race attempts one at a time, staggered by the Connection Attempt Delay.
  3. First success wins; the rest are cancelled.

Default preference: IPv6 > IPv4, QUIC > TCP.

Two timers

Resolution Delay

  • How long to wait for the preferred family / HTTPS RR before committing.
  • Draft default 50 ms. Firefox: 25 ms (D309547).

Connection Attempt Delay

  • Stagger between successive attempts.
  • Draft default 250 ms. Firefox: 50 ms with a ×2 multiplier (50, 150, 350, 750 ms …).
0 ms
50
150
350
750 ms

Each dot is a staggered connection attempt: a 50 ms base with a ×2 back-off, versus the draft’s single 250 ms.

A reusable library: mozilla/happy-eyeballs

  • A deterministic, sans-I/O state machine: no network, no clock. The caller drives all I/O and passes time in explicitly.
  • No dependency on Firefox; embeds in Necko’s C++ event loop.
  • Permissive license (MIT OR Apache-2.0). Input and contributions welcome.
  • Every scenario is a plain unit test, so behaviour is easy to pin down and easy to review.

github.com/mozilla/happy-eyeballs

Testing the state machine

#[test]
fn connection_attempt_delay() {
    let (mut now, mut he) = setup();
    expect_initial_dns_queries(&mut he, now);

    he.input(in_dns_https_positive_no_alpn(Id::from(0)), now);
    he.expect(out_resolution_delay(), now);
    he.input(in_dns_aaaa_positive(Id::from(1)), now);
    he.expect(out_attempt_v6_h1_h2(Id::from(3)), now);   // preferred family first
    he.input(in_dns_a_positive(Id::from(2)), now);
    he.expect(out_connection_attempt_delay(), now);

    now += CONNECTION_ATTEMPT_DELAY;                     // time is an argument
    he.expect(out_attempt_v4_h1_h2(Id::from(4)), now);   // the staggered second attempt
}

Feed inputs, assert the next action, advance time by hand.

Enabled in Firefox Nightly

  • Pref network.http.happy_eyeballs_enabled, default on in Nightly.
  • Public Glean telemetry, aligned with Mozilla’s privacy stance. Ten metrics:
  • end-to-end time (succeeded / failed)
  • winning attempt index
  • connection attempt count
  • cancelled attempt count
  • time to first attempt
  • DNS resolution time, by record type
  • h3 discovery method
  • HTTPS-RR features
  • HTTPS-RR features, by resolver

Definitions in netwerk/protocol/http/metrics.yaml. Tracking: Bugzilla 1953459.

The happy_eyeballs_* metrics

Firefox Nightly, Glean, read from GLAM. Ten metrics follow.

1. End-to-end time, succeeded

Time from first DNS query to a connected socket, successful connects. Median 74 ms, P95 603 ms. Source: GLAM: end_to_end_time_succeeded.

2. End-to-end time, failed

When a connection fails, it fails fast (P75 50 ms, an immediate refusal) or times out at the 10 s cap. Few failures sit in between. Source: GLAM: end_to_end_time_failed.

3. Winning attempt index

Which staggered attempt actually connected. 84% of connections win on the very first attempt; few need a later one. Source: GLAM: winning_attempt_index.

4. Connection attempt count

How many attempts were opened before one succeeded. 87% need just one. Racing is cheap: it rarely fires a second attempt. Source: GLAM: connection_attempt_count.

5. Cancelled attempt count

Attempts started, then cancelled once a winner appeared: wasted work. 92% cancel nothing. Source: GLAM: cancelled_attempt_count.

6. Time to first attempt

How long before the first connection attempt fires: mostly the DNS answer arriving. Median 3 ms, P75 27 ms. Source: GLAM: time_to_first_attempt.

7. DNS resolution time, by record type

Per record type, log scale. HTTPS records resolve as fast as A/AAAA, so asking for them costs no latency. Source: GLAM: dns_resolution_time.

8. How connections learn about h3

  • 29% of connections learned h3 only via Alt-Svc: a round trip they need not have paid.
  • 21% via an HTTPS record alone, 19% via both: no round trip.
  • The Alt-Svc-only slice is exactly the gap an HTTPS record would close.
Source: GLAM: h3_discovery.

9. What HTTPS records carry

Of connections that saw an HTTPS record, the share carrying each SvcParam. Nearly all advertise h3; about half carry ECH. Source: GLAM: https_rr_features.

10. HTTPS-RR features, by resolver

Same SvcParams, split by DoH vs the OS resolver. DoH: 98/96/96/32% (h3/v4/v6/ECH). OS resolver: 96/90/89/52%. ECH shows up more over the OS resolver. Source: GLAM: by_resolver.

The wasted round trip

Alt-Svc only

HTTP/3 only on the second connection.

HTTPS record

HTTP/3 on the first connection.

Save a round trip

  • Many sites send Alt-Svc: h3 but publish no HTTPS record.
  • Every first visit then wastes a round trip: connect over HTTP/2, read the header, upgrade to HTTP/3 only next time.
  • Publish an HTTPS record with alpn="h3" and the client goes straight to QUIC on connection #1.
  • Bonus: the HTTPS record is the only way to deliver ECH.

savearoundtrip.com: check your domain.

How Firefox deviates from the draft

  • Attempt delay: the draft recommends a flat 250 ms. Firefox uses a smaller 50 ms base with a ×2 back-off (50, 150, 350, 750 ms), so the second attempt fires far sooner while later attempts still back off.
  • Resolution delay: draft 50 ms, Firefox 25 ms (D309547), tuned from the telemetry above.
  • Interleaving: the draft interleaves address families; the library also interleaves protocol variants, so QUIC and the other family are both reached early. A library choice, not fed back into the draft.

The delay tuning is exactly the kind of input the working group can fold back into the recommended defaults.

Real-world edge cases

  • Racing an unreachable IPv6 address first added up to 40 ms on cold DoH connections (bug 2044951).
  • IPv6 blackholes: an h3 path that silently drops packets needed explicit handling (bug 1993300).
  • 0-RTT: a rejected 0-RTT connection was wrongly kept as usable (bug 2040900); the fallback when 0-RTT loses the race had to be defined (bug 1994321).
  • DNS negative answers: honour the SOA-derived negative TTL (RFC 2308, bug 2049173), and do not re-resolve them mid-race (bug 2049178).
  • DNS coalescing: a v4-only or v6-only lookup could not reuse a pending AF_UNSPEC lookup (bug 2044254).
  • Multi-CDN: a non-standard HTTPS-RR/CNAME consistency check rejected legitimate multi-CDN records; reverted to resolve each TargetName independently (RFC 9460 §10.4.4, PR 104).
  • Web standards: dnsLookupEnd stopped matching the Resource Timing spec (bug 2047648).

Gotchas from the Happy Eyeballs v3 meta bug and the happy-eyeballs crate, most now fixed.

Debugging Happy Eyeballs

Inspect live state

  • about:networking: the HTTP, Sockets, and DNS tabs show live connections, open sockets, and cached lookups.
  • The Alt-Svc page: which origins advertised h3, and whether it is still valid.
  • The DNS Lookup tool: resolve a host on demand and see its A / AAAA and HTTPS records.

Capture a trace: the Firefox Profiler

  1. Enable it from profiler.firefox.com (the page walks you through adding the toolbar button).
  2. Start profiling with the Networking preset.
  3. Load the site you want to debug, then stop.
  4. Open the parent-process Socket Thread marker chart: each attempt, its timing, and which one won are right there.

Example profile: share.firefox.dev/4wOr8Fc.

Lessons and open questions

  • The 250 ms attempt delay is too coarse: real deployments want a smaller base with back-off.
  • The Alt-Svc-vs-HTTPS-RR gap is measurable and costs round trips: publish HTTPS records.
  • Still open: proxy awareness (MASQUE connect-udp vs HTTP CONNECT ordering) and WebSocket / WebTransport EXTENDED CONNECT.

Discussion

Attempt-delay defaults, the Alt-Svc / HTTPS-RR discovery gap, the ECH-by-resolver split.

github.com/mozilla/happy-eyeballs · Bugzilla 1953459 · mail@max-inden.de