Multi-node inference with Ray LLM on EKS: how prefill and decode fit together

AWS’s multi-node Ray LLM deployment guide describes how to serve one large language model across two GPU nodes on Amazon EKS, assigning prefill and decode to separate nodes behind a single OpenAI-compatible endpoint. The Ray LLM Deep Learning Container packages the inference stack, making deployment configuration the central task. For cloud architects, the useful distinction is the separation of inference phases while preserving one application-facing API. AWS’s deployment overview

The guide provides an architecture to examine, rather than evidence of a universal performance improvement. Its practical questions concern where each phase runs, how inference state moves between nodes, and which transport the deployment actually uses.

Why separate prefill and decode?

The supplied coverage distinguishes processing a prompt during prefill from generating tokens during decode. AIToday describes the benefit of separating these phases as preventing a long, compute-heavy prompt from stalling token generation for requests already in progress. That is the publication’s stated performance rationale; the evidence contains no measurements establishing the size of the improvement or the workloads under which it holds. AIToday’s account

This makes phase separation an architectural option to evaluate against a workload’s behavior. The configuration demonstrates how to assign different responsibilities to GPU workers, but it does not establish that every application needs that separation. The evidence supplies no comparative throughput, latency, or cost results that would support such a conclusion.

The single endpoint is a separate benefit of the design. Prefill and decode have distinct placements inside the deployment, while clients access an OpenAI-compatible API. The supplied evidence establishes that interface description without documenting compatibility with every OpenAI API feature.

How the container’s components fit together

AWS’s detailed guidance describes four components in the packaged stack:

The KV cache is the inference state transferred between nodes in this design. NIXL’s role is that transfer; the EFA stack supplies the communication layer described by AWS. Keeping these responsibilities distinct helps explain why packaging the components together is only part of deployment: their configuration must also connect the two inference phases. AWS’s stack and configuration guidance

AWS describes the container as exposing an OpenAI-compatible API without application code. AIToday adds that the demonstrated deployment requires no handler class, model archiver, or custom image build, and uses YAML rather than custom handler code. Those statements describe the supplied serving approach; they do not remove the need to configure node placement and cache transfer. AIToday’s deployment description

The demonstrated EKS layout

AWS’s example separates CPU system workloads from GPU inference workers:

The Ray head pod is pinned to the system node group, with num-gpus: "0". The two GPU workers carry the phase-specific inference responsibilities. These instance types belong to the documented example; the evidence does not establish them as optimal choices for other models or traffic patterns. AWS’s node layout

AWS also pins the GPU node group to one Availability Zone, stating that EFA cannot cross Availability Zones. That constraint matters when interpreting the architecture: the demonstrated arrangement does not show a prefill worker in one zone transferring cache to a decode worker in another over EFA. The evidence does not describe an alternative cross-zone transport or a multi-zone availability design.

The configuration that connects the phases

AIToday identifies build_pd_openai_app as the helper used to build the application from prefill and decode configurations. AWS identifies kv_transfer_config as the setting that enables disaggregation, or separating the two phases. Both phases use NixlConnector, with kv_role: kv_both and distinct engine_id values: prefill for one phase and decode for the other. AIToday’s application construction account, AWS’s cache-transfer configuration

These identifiers explain the configuration relationship, but the supplied evidence does not include a complete deployable manifest, container image identifier, or command sequence. They should be read as documented settings within the architecture, rather than a self-contained installation recipe.

Transport selection deserves particular attention. AIToday reports that without the LIBFABRIC backend setting, NIXL defaults to UCX and operates at TCP speed even on EFA. This warning is supported by that secondary account, without a supplied benchmark or independent verification. Its operational implication is clear: the presence of EFA in the infrastructure does not, by itself, establish that cache transfers use the intended transport. AIToday’s transport warning

What the architecture establishes

The documented approach brings the inference engine, serving layer, cache-transfer library, and communication stack into a pre-built container. It shows a concrete division of work across a CPU system node and two GPU workers, with phase-specific configurations behind one endpoint.

The remaining decision is workload-specific. The supplied evidence supports evaluating this architecture where prompt processing interferes with ongoing token generation, but it does not quantify the performance benefit, transfer overhead, or operating cost. For an architect assessing the design, the next evidence needed is measurement of the actual workload with the documented placement and transport settings. The guide establishes the deployment pattern; those measurements would establish its value for a particular application.