Modern UDP I/O for Firefox in Rust

Closing the gap to in-kernel TCP

HTTP Workshop 2026  ·  Max Inden · Mozilla

Motivation

  • ~20% of Firefox desktop HTTP requests are HTTP/3 over QUIC (UDP), already ~28% on Android Nightly (performance.mozilla.org/networking).
  • We expect that to keep growing: broadly, and specifically through Happy Eyeballs v3.
  • Firefox’s QUIC implementation lives in user space, so every datagram costs a system call into the OS kernel.
  • We replaced Firefox’s dated NSPR UDP I/O (one sendto/recvfrom per datagram) with a modern Rust stack.
  • Goal: close the gap to heavily optimized, in-kernel TCP.

Tracking: Bugzilla 1901292: [meta] Fast UDP for Firefox. Background: max-inden.de/post/fast-udp-io-in-firefox.

Userspace QUIC on UDP is expensive

  • One syscall per datagram, < 1500 bytes each.
  • Default QUIC (RFC 9000) ACKs up to every second packet.
  • Early QUIC servers spent ~3.5× the CPU cycles per byte of TLS/TCP (Langley et al., SIGCOMM 2017).

Segmentation offload: GSO / GRO

  • Linux GSO/GRO, Windows USO/URO: one syscall carries a big buffer; the kernel or NIC splits it on send (GSO) and coalesces many datagrams on receive (GRO) (LWN).
  • One syscall then moves many packets instead of one, cutting the per-datagram cost.

Send: segments per syscall (GSO)

How many segments one sendmsg carries via GSO.

  • Median 1, but the tail batches: P99 5, P99.9 10 segments per send.
  • Firefox is a client: mostly downloading, so most sends are small ACKs with nothing to batch.
  • GSO batches only when there is bulk data to send (uploads).
  • And most connections never get there: only ~6.5% ever grow their congestion window, of those only ~5-6% exit slow start.
UDP segments per send syscall, Firefox Nightly, Linux. GLAM: udp_datagram_segments_sent. Growth: congestion_window_growth, slow_start_exited_filtered.

Receive: segments per syscall (GRO)

How many datagrams one recvmmsg / GRO read returns.

  • Median 1, but the tail coalesces heavily: P95 10, P99 27, P99.9 53 per read.
  • This is where receive offload pays off: bulk download.
  • Fewer syscalls per byte on the hot receive path.
UDP segments per receive syscall, Firefox Nightly, Linux. GLAM: udp_datagram_segments_received.

Sent segment size

The GSO segment size chosen on send.

  • The median is small (~33 to 47 B): ACKs.
  • When GSO batches bulk data, segments reach our configured MTU (~1217 B), a conservative fixed floor.
  • We do not do PMTUD yet, so we never grow to the real path MTU (bug 2048618).
UDP datagram segment size sent, bytes, Firefox Nightly, Linux. GLAM: udp_datagram_segment_size_sent.

Received segment size

The size of each datagram the network delivers to us.

  • Tightly clustered at the common 1217 B and 1448 B MTUs.
  • Near-MTU datagrams arrive consistently, little variation.
UDP datagram segment size received, bytes, Firefox Nightly, Linux. GLAM: udp_datagram_segment_size_received.

Bytes per send syscall

Total bytes handed to one sendmsg.

  • Median is tiny (an ACK or control write, ~43 B).
  • The tail packs several segments into one GSO syscall (P99 ~5 KB, P99.9 ~12 KB).
UDP datagram size sent, bytes, Firefox Nightly, Linux. GLAM: udp_datagram_size_sent.

Bytes per receive syscall

Total bytes returned by one recvmmsg / GRO read.

  • Median is one datagram (~1217 B).
  • The tail coalesces a lot into one read: P95 ~13 KB, P99 ~31 KB, P99.9 ~63 KB.
UDP datagram size received, bytes, Firefox Nightly, Linux. GLAM: udp_datagram_size_received.

Multi-message syscalls

  • Linux sendmmsg / recvmmsg; macOS sendmsg_x / recvmsg_x (Apple, undocumented, quinn#1993).
  • One syscall moves a batch of independent datagrams.
  • ~11% improvement on an OS loopback transfer (quinn#1993).

quinn-udp: one abstraction, many platforms

  • QUIC UDP I/O in Rust via quinn-udp, shared with the Quinn project.
  • Offload-optimized where Firefox ships: Linux, Windows, macOS, Android.
  • The same abstraction compiles for iOS, FreeBSD, NetBSD, OpenBSD, DragonFly, Solaris, Redox, with a generic fallback path.
  • Reads auxiliary IP metadata (ECN) from the same control messages.

No-alloc receive path

  • A single, long-lived 64 KiB receive buffer shared across all QUIC connections (neqo-udp).
  • No per-datagram allocation on the hot receive path.
  • The iterator hands out borrowed slices of that one buffer; Rust’s borrow checker proves there is no aliasing or use-after-free, at compile time.
  • Large transfers spend much less CPU: in the profile, crypto is now about as costly as UDP I/O, no longer far below the per-datagram overhead.

Result

<1 → 4 Gbit/s on CPU-bound transfers.

Flamegraph rooted at HttpConnectionUDP::OnPacketReceived, Firefox 154 loading a page: neqo’s per-packet receive path. Widest leaves are sendmsg / recvmmsg; the tall middle branch is Connection::input → decrypt. Profile: share.firefox.dev/3T9RKSh. Throughput: max-inden.de/post/fast-udp-io-in-firefox.

ECN: marking and echoing

  • Modern syscalls expose IP metadata via control messages, including ECN.
  • 58% of paths are ECN-capable; but 33% bleach the bits (strip ECN in transit).
  • On a CE mark, Firefox does a reduced congestion response, no packet loss required (neqo#3233).
  • 1 in 5 congestion responses is triggered by ECN CE, not loss (GLAM).
ECN path capability, share of connections. GLAM.

OS idiosyncrasies: Windows & macOS

Windows

  • URO on Windows-on-ARM returns no segment size, so receive coalescing is unusable, disabled (bug 1916558, quinn#2041).
    Bought the same Surface laptop, same colour, just to reproduce it.
  • USO raised P95 loss to ~6.5%, rolled back (bug 1979279).
  • USO crashed a user’s network driver (bug 1978821).

macOS

  • No NIC offload, but the undocumented sendmsg_x/recvmsg_x are now enabled (bug 2028797): a dedicated background thread exercises them once against localhost sockets before we trust them (AppleFastDatapathProbe).
  • 10.15 reports packets that are neither v4 nor v6 (bug 1987606, quinn#2387).

OS idiosyncrasies: Linux & Android

Linux

  • GSO support must be probed at runtime; sandbox tweaks needed (bug 1989895).
  • GSO can pass the probe but fail at send with EIO, resend individually (bug 2049334, quinn#2399).

Android

  • x86 dispatches via socketcall; seccomp blocks it (quinn#1966).
  • API ≤ 25 returns EINVAL when setting ECN, retry without (quinn#2079).
  • Complains about single-segment GSO (quinn#2050).

Status: shipped to most users

  • The quinn-udp UDP path is on by default beyond Nightly: most of this has shipped to Release (bug 1910360).
  • ECN shipped: marking, echoing, and a reduced CE-based congestion response (neqo#3233).
  • Send-path offload is in: GSO on Linux and Android, sendmsg_x on macOS (bug 2028797).
  • Windows USO/URO remain off (packet loss, driver crashes).
  • Still to do: a long-lived send buffer (neqo#2747).

Where the stack is today

683MiB/s download
100 MB, 1 conn
656MiB/s upload
100 MB, 1 conn
~38krequests/s
10k parallel, 1-byte

~5.7 / 5.5 Gbit/s single-connection. These are artificial benchmarks: neqo’s criterion suite over loopback (127.0.0.1), one machine, no real network and no browser, so an upper bound, not field performance. Source: neqo#3768 benchmark run.

Real users see faster uploads

HTTP/3 upload throughput is up ~60-90% across percentiles over the last ~11 Firefox releases.

No single cause: many small compounding wins, including

HTTP/3 upload throughput (Mbps, uploads > 10 MB), Firefox release, by version. GLAM.

Takeaways

Batching and offload matter, even on the client

  • These are usually server-side optimizations; it is surprising how much they help a browser.
  • Offload and batching close most of the gap to kernel TCP.
  • The hard part is not the fast path: it is every OS’s edge cases.
  • Rust’s borrow checker made the shared-buffer receive path safe by construction.

neqo · quinn-udp · mail@max-inden.de