Kubernetes Gateway API Inference Extension routes LLM traffic by pod serving state

The Kubernetes Gateway API’s Inference Extension changes how LLM requests are routed by choosing backends based on each pod’s current serving state rather than treating all candidates as interchangeable. That shift is intended to curb wasted inference capacity and reduce tail latency that arises when generic HTTP load balancers distribute generative — model traffic blindly. Platform and ML engineers stand to gain better accelerator utilization and fewer misrouted requests when stateful pod readiness is respected.

Before a request is assigned to a pod, the Inference Extension evaluates per-pod signals that indicate readiness to serve that specific model or request type. Examples called out in the guide include Key‑Value (KV) cache readiness, Low‑Rank Adaptation (LoRA) adapter availability, and current queue length. At the same time, the gateway performs protocol checks — validating hostname and HTTPRoute rules — extracts the target model from the path, headers, or a Body‑Based Router (BBR), and maps that model name to an InferencePool representing the candidate pods.

Endpoint selection is delegated to a dedicated Endpoint Picker (EPP). Rather than performing simple round‑robin selection at the gateway, the gateway pauses routing long enough to query the pool’s EPP, which applies flow‑control logic together with the collected readiness signals to select the most suitable pod. That decoupling recognizes production inference scheduling is complex and that individual pod readiness fluctuates with workload and use of stateful resources such as caches and adapters.

The guide contrasts this approach with conventional HTTP load balancers, which were designed for stateless web traffic and default to even distribution. Because inference requests can vary widely in compute cost and duration, blind load distribution can increase latency and squander cached state. Moving to the Gateway API and enabling the Inference Extension lets teams exploit richer routing criteria to improve GPU and accelerator utilization and to reduce tail latency compared with even distribution.

Observability is emphasized across three lifecycle phases: gateway validation and model extraction, EPP endpoint selection and flow control, and the underlying model — serving instance. The guide recommends concrete metrics to instrument, including per‑pod queue length, KV cache hit/miss rates, LoRA adapter readiness, EPP decision counts and latencies, and request duration on the chosen backend, so operators can validate that routing decisions align with runtime reality.

Operational consequences include more efficient cluster capacity use and potentially lower inference costs when routing leverages cached state and adapter availability. At the same time, routing correctness now depends on transient backend state, so monitoring and alerting must evolve: dashboards and SLOs should correlate gateway decisions with EPP signals and model‑server metrics to detect misrouting or resource hot spots. Finally, the guide notes migration implications — adopting the Gateway API is a prerequisite for the Inference Extension — and advises platform teams to plan for additional telemetry, flow‑control policies, and testing of BBR paths to ensure requests are consistently routed to suitably prepared pods.