How OpenAI delivers low-latency voice AI to 900 million users

Most backend systems treat latency as a performance metric: lower is better, but 200ms vs. 500ms rarely breaks a feature. Voice AI is different because of how humans perceive conversation. Above roughly 150ms of round-trip delay, people start to notice. Above 300ms end-to-end, turn-taking breaks down: speakers interrupt each other, there are awkward silences, and the conversation feels like a satellite phone call. This is not a soft preference, it is a hard perceptual constraint rooted in human auditory processing.

The end-to-end budget for a voice AI response looks like this: the user speaks, audio is captured and encoded on the device, it travels over the network to the server, a voice activity detector identifies the end of the utterance, a speech-to-text model transcribes it, a language model generates a response (streaming tokens), a text-to-speech model synthesizes speech, audio travels back over the network, and the client decodes and plays it. Each stage adds latency. You have roughly 300ms to do all of this.

Two things make this tractable. First, you can pipeline: start transcribing while the user is still talking, start synthesizing as soon as the first sentence of the LLM output arrives. Second, you can use native audio models like GPT-4o that bypass the ASR and TTS stages entirely by processing audio end-to-end. But the transport layer, getting audio to and from the server, is the first problem to solve, and it sets a floor that no model optimization can lower.

WebRTC (Web Real-Time Communication) is a collection of standards that make real-time audio and video communication work from a browser or native app. Before WebRTC, building a video call required proprietary plugins or native apps because browsers had no standard way to handle real-time media. WebRTC solves several hard problems that a naive "just send audio over a WebSocket" approach would leave unresolved.

The first hard problem is connectivity. Most devices are behind NAT (Network Address Translation) routers and firewalls. Two devices cannot directly connect to each other without a way to discover and test candidate network paths. WebRTC uses ICE (Interactive Connectivity Establishment) for this: both sides gather candidates (local IP addresses, translated IP addresses via STUN servers, and relayed addresses via TURN servers), exchange them via a signaling channel (typically HTTP or WebSocket), and then test each candidate pair to find one that works. This is NAT traversal.

The second hard problem is encrypted transport. WebRTC uses DTLS (Datagram Transport Layer Security, the UDP equivalent of TLS) to negotiate encryption keys, and then wraps all media in SRTP (Secure Real-Time Transport Protocol). Every packet is encrypted. Wiretapping or injection attacks on the media stream are not possible once the session is established.

The third hard problem is media quality over an unreliable network. WebRTC uses UDP, not TCP, for media. TCP guarantees delivery by retransmitting lost packets, but retransmission introduces unpredictable delay, if a packet is dropped at 100ms and retransmitted at 300ms, all subsequent packets queue behind it (head-of-line blocking). For voice, it is better to drop a 20ms audio frame than to play audio 200ms late. WebRTC uses codec-level packet loss concealment to hide brief gaps, a jitter buffer to smooth out arrival time variation, and RTCP (RTP Control Protocol) for quality feedback and adaptive bitrate. Opus, the standard WebRTC audio codec, can operate at 6 to 510 kbps and handles packet loss gracefully.

The fourth hard problem is client-side audio processing. Browser implementations of WebRTC include built-in acoustic echo cancellation (so the speaker output does not feed back into the microphone), noise suppression, and automatic gain control. These run on the client before audio ever reaches the server, a meaningful reduction in the work the AI pipeline needs to do.

In a standard WebRTC server deployment, the server must expose a public UDP port for each active session. A client connects to a unique port, and that port is bound to the server process handling that session. This is fine at small scale. At OpenAI's scale, hundreds of millions of weekly users, with many concurrent voice sessions at any given time, it creates three serious operational problems.

First, Kubernetes and cloud load balancers are not designed to expose tens of thousands of UDP ports. NodePort services in Kubernetes support a range of roughly 30,000 ports (30000 to 32767 by default). For applications that require one port per session, this is a hard ceiling. You can work around it with host networking (bypassing Kubernetes's network model entirely), but then you lose the benefits of service discovery, autoscaling, and standard network policies.

Second, a large public UDP port range is a security liability. Every exposed port is an attack surface. Scanning for open ports is trivial, and UDP-based amplification attacks target open ports. A smaller and more predictable UDP footprint is easier to firewall, monitor, and harden.

Third, port reservations conflict with autoscaling. If a pod owns a port range and that pod needs to be replaced or scaled up, you need to coordinate port assignments across the cluster. This couples the stateful port assignment problem to the stateless compute scaling problem, two things that are much easier to reason about separately.

There are three fundamentally different models for how a server participates in a real-time media session, and the choice has major implications for latency, compute cost, and what the server can actually do with the audio.

A Multipoint Control Unit (MCU) decodes all incoming media streams, mixes them into a single composite stream, and re-encodes it for each participant. Every participant receives one stream regardless of how many others are in the call. This is the traditional conference call model. The tradeoff: the server must decodeWrites the answer one token at a time. and re-encode, which is compute-intensive and adds latency. MCUs were common in the early days of video conferencing.

A Selective Forwarding Unit (SFU) is the dominant model in modern multi-party systems like Zoom, Google Meet, and WebRTC-based conferencing. The SFU forwards media packets between participants without decoding them. Each participant receives streams from all others. The server is essentially a smart router, it can apply routing policy (simulcast, bandwidth adaptation) but never decodes the actual content. This is very compute-efficient.

A transceiver is a server that fully terminates a WebRTC session and participates as a genuine peer. It maintains ICE state, holds DTLS encryption context, decodes the audio, and then does whatever it wants with the raw PCM, in OpenAI's case, that means feeding it to an ASR model or a native audio model. It produces a response and re-encodes it for transmission back to the client.

OpenAI chose the transceiver model, and for voice AI this is the only model that makes sense. An SFU is designed to forward packets from human speaker A to human speaker B. The AI is not a passive participant receiving packets, it needs to actively process the decoded audio and generate a response. An SFU would forward packets to an AI backend that still has to decodeWrites the answer one token at a time. them. The transceiver just does all of this in one place. There is also a latency argument: a transceiver can begin processing audio as it arrives (before the utterance ends), enabling streaming transcription and early response generation.

Even after choosing the transceiver model, the port explosion problem remains. Each transceiver pod needs to expose a public UDP port per session. OpenAI's solution is to split packet routing from protocol termination into two separate services.

The relay is a lightweight UDP forwarding service. It exposes a small, fixed set of public UDP ports, potentially just one or a handful per relay instance. Every incoming WebRTC packet from any client arrives at the relay first. The relay's only job is to read a routing identifier from the packet and forward it to the correct transceiver pod on the internal network. Critically, the relay does not decrypt any media, it reads metadata from the unencrypted portion of the packet header and forwards the raw bytes. This makes it stateless from a media perspective and extremely fast.

The transceiver runs on the internal network with no public UDP exposure. It maintains full WebRTC session state: the ICE connectivity state, the DTLS encryption context, the SRTP keys, and the upstream connection to the AI inference backend. It is the stateful, compute-intensive component. Because it only receives traffic from the relay (not directly from the internet), it does not need to expose any ports publicly. Multiple transceiver pods can run behind standard Kubernetes services.

The architecture separates two concerns that were previously coupled: the relay manages the public UDP surface (a small, fixed, hardened attack surface), and the transceiver manages session state (stateful, but not publicly exposed). The relay scales horizontally as a stateless forwarder. The transceiver scales by session affinity, clients are routed to the transceiver that owns their session.

This split is a specific instance of the broader data plane vs. control plane separation pattern. The relay is the data plane: it moves packets fast without caring about their content. The transceiver is the control plane: it manages state, makes decisions, and connects to backend services.

The relay needs to know which transceiver to forward each packet to. The naive approach would be to maintain a lookup table: incoming packet arrives, relay queries a central store keyed on source IP + port, gets back the transceiver address, forwards. This works but adds a round-trip for every forwarding decision and introduces a dependency on an external stateful service (the lookup table) in the critical path.

OpenAI's approach is more elegant. When a client establishes a WebRTC session, it goes through SDP (Session Description Protocol) negotiation: the client and server exchange a text description of the session parameters (codecs, ICE candidates, ICE credentials). One of the values in this exchange is the ICE ufrag, a short identifier that appears in every STUN packet during the ICE process. STUN packets are unencrypted during the connectivity phase, so the relay can read the ufrag without decrypting anything.

During session setup, the signaling layer encodes the target transceiver's address directly into the ICE ufrag. When the client's first STUN packet arrives at the relay, the relay reads the ufrag, parses the embedded transceiver address, and forwards the packet, no external lookup, no state, no round-trip. The routing metadata travels with the packet itself, inside a field that is already part of the WebRTC protocol.

This is a clean application of a general principle: when you need to route traffic, prefer embedding routing metadata in the traffic itself over maintaining external routing tables. It also means the routing logic is deterministic: the same ufrag always routes to the same transceiver, regardless of the relay instance that receives the packet. This makes the relay fully stateless.

The relay + transceiver architecture solves the Kubernetes port problem, but it does not automatically make the system fast for users everywhere. If the only relay nodes are in US-East, a user in Tokyo still has ~150ms of one-way network latency before their audio reaches the relay, before any processing has even started.

OpenAI deploys relay nodes at geographically distributed ingress points around the world. For signaling (the SDP negotiation that happens before any media flows), they use Cloudflare's geo-steering: DNS returns the address of the nearest signaling endpoint based on the user's location. The user's first network hop, the one they have the least control over, goes to a nearby relay node rather than to a distant data center.

Once the session is established, media from the client flows to the nearby relay, which forwards it over OpenAI's internal backbone to the transceiver and inference backend. Internal backbone routing is faster and more reliable than public internet routing, so the net result is that the highest-latency part of the path (the last-mile hop from client to server) is minimized.

The separation of relay and transceiver is what makes this practical. The relay is stateless and cheap to deploy globally. The transceiver is stateful and computationally expensive, you do not want to deploy GPU inference capacity at every relay PoP. The geo-distributed relay nodes act as low-latency onramps onto the backbone, and the expensive processing happens centrally. This is the same topology CDNs use: edge nodes close to users for fast delivery, origin servers for the actual content.

Even with good architecture, a poorly implemented UDP relay can become a bottleneck. A single thread reading from a single UDP socket can process only so many packets per second, not because of bandwidth, but because of CPU time per packet (system calls, memory copies, scheduling).

OpenAI's relay uses two Linux socket options to maximize throughput without needing kernel-bypass frameworks like DPDK. The first is SO_REUSEPORT, which allows multiple threads to bind to the same UDP port. The Linux kernel then distributes incoming packets across all bound threads using a hash of the source IP and port. Each thread has its own socket file descriptor and processes its own stream of packets in parallel. This turns a single-threaded bottleneck into a parallelizable workload.

The second is runtime.LockOSThread (this is Go-specific). Normally, Go's scheduler can move a goroutine between OS threads. For a tight packet processing loop, this is bad for CPU cache performance: if a goroutine moves to a different OS thread, the socket state and packet buffers that were warm in the previous CPU's cache must be re-fetched. LockOSThread pins the goroutine to its OS thread, preserving cache locality across the packet processing loop.

The result is a relay that handles high packet throughput using standard kernel networking, without the complexity of DPDK or io_uring. This is worth noting because kernel-bypass frameworks are sometimes proposed for high-performance network services, but they add significant operational complexity. At voice AI packet rates (audio is not high-bandwidth), standard kernel networking with SO_REUSEPORT is sufficient.

This is a composite system design question that combines real-time media infrastructure with AI inference. Here is how to structure a strong answer.

**Step 1: Clarify requirements and constraints.** Before drawing anything, nail down: what is the latency target? (Answer: sub-300ms end-to-end for natural conversation.) What platforms must be supported? (Web and mobile.) What is the expected scale? (Say 1 million concurrent sessions.) Is this a native audio model or a pipeline (ASR + LLM + TTS)? These answers shape every subsequent decision.

**Step 2: Sketch the high-level flow.** Client speaks into a microphone. Audio is captured and encoded on the device. It travels over WebRTC (UDP, SRTP, Opus) to the nearest relay node. The relay forwards to a transceiver, which decodes the audio and feeds it to the AI pipeline. The pipeline returns synthesized speech. The transceiver encodes it and sends it back through the relay to the client.

**Step 3: Deep dive on transport.** Justify WebRTC over WebSocket: you need UDP to avoid head-of-line blocking, SRTP for encryption, and jitter buffering for smooth playback. The client gets these for free in browser WebRTC. Use Opus codec for high-quality audio at 16-32 kbps.

**Step 4: Deep dive on server architecture.** Propose the relay + transceiver split. The relay is a stateless UDP forwarder with a small fixed port footprint, deployed globally as geo-steered ingress. The transceiver handles full WebRTC session termination and connects to the AI backend. Encode the transceiver address in the ICE ufrag for first-packet routing. This gives you: a small, auditable attack surface; relay nodes that scale as stateless compute; transceivers that scale by session affinity.

**Step 5: Deep dive on the AI pipeline.** Name the three options: pipeline (Whisper + GPT-4 + TTS), streaming pipeline (overlap each stage), or native audio model (GPT-4o audio mode). For minimum latency, streaming is critical: begin transcribing as the user speaks, begin synthesizing as the first sentence of the LLM response arrives. If using a native audio model, the pipeline overhead disappears but you trade off per-stage control.

**Step 6: Latency budget.** Walk through the budget explicitly: ~20-50ms network (user to nearest relay), ~50-100ms first-token from model, ~50ms TTS first audio chunk, ~20ms network back. With pipelining, perceived response time can be under 200ms even if total pipeline time is longer. The key metric for perceived responsiveness is time to first audio byte, not total response latency.

**Step 7: Name the tradeoffs you consciously made.** Relay adds one extra network hop vs. direct transceiver exposure, but the security and operational benefits outweigh it. Centralized transceivers add ~10-20ms backbone latency vs. fully distributed, but avoid the cost of running GPU inference at every edge PoP. Choosing native audio model reduces pipeline latency but reduces per-stage observability and control.