Standard, Technology

NVIDIA Dynamo on Amazon SageMaker HyperPod: Distributed Inference and Its Trade-offs

Symbolic diagram: NVIDIA Dynamo on Amazon SageMaker HyperPod: Distributed Inference and Its Trade-offs

Introduction to NVIDIA Dynamo and Amazon SageMaker HyperPod

Amazon SageMaker HyperPod Inference supports NVIDIA Dynamo, bringing distributed model serving together with managed infrastructure for GPU deployments. The practical aim is to help teams serve models across multiple instances, separate prompt processing from token generation when useful, and operate those deployments with coordinated scheduling and node recovery.

In its description of the integration, AWS presents Dynamo as an open source distributed inference framework supporting vLLM, SGLang, and TensorRT-LLM. HyperPod supplies Elastic Fabric Adapter (EFA) networking, GPU scheduling, and health management. These capabilities address related problems: how to distribute inference work and how to keep the resources supporting it available.

The combination matters because serving efficiency depends on more than whether model weights fit in GPU memory. Prompt processing, token generation, cache reuse, communication between instances, and placement of workers all affect a deployment. Dynamo and HyperPod provide controls across those layers, while the managed integration connects their deployment components.

AWS reports throughput improvements from separating prompt processing and generation in one benchmark, alongside a separate experiment showing little throughput loss when a worker spans two instances. Those results establish specific examples of what the integration can deliver. They do not establish a universal performance gain: the tested models, traffic patterns, hardware, and latency conditions are essential to interpreting them.

Challenges in serving large language models

Symbolic diagram illustrating: Challenges in serving large language models

The first constraint is memory. AWS gives GLM-5.3 as an example: its FP8 model weights exceed the 640 GB of GPU memory available on one ml.p5.48xlarge instance. That comparison excludes the additional memory needed for the KV cache and runtime overhead. Sizing a deployment against weight size alone would therefore leave out two requirements the source explicitly identifies.

For a model that exceeds one instance’s capacity, inference workers must be able to span instances. Dynamo supports this through tensor and pipeline parallelism. The infrastructure then has to place the worker’s constituent pods together and provide communication between them. Multi-node serving is consequently both a model execution problem and a scheduling problem.

A second challenge is competition between inference phases. Prefill workers process prompts; decode workers generate output tokens. AWS explains that long prompts can slow token generation when both activities share the same GPUs. Aggregated serving places prefill and decode in one worker, while disaggregated serving separates them into distinct pools.

Separating those pools creates room for independent scaling and tuning, but also creates requirements of its own. Both pools load the full model weights, and KV cache moves between them over EFA. The architecture therefore needs sufficient capacity for each pool and working communication for the cache transfer. Disaggregation changes resource allocation rather than eliminating the need to size and coordinate resources.

Request routing adds another consideration. Dynamo’s KV-aware router considers reusable prefix KV cache and active worker load in either serving topology. According to AWS, reusing cached prefixes can reduce repeated prompt computation and improve time to first token; considering load helps avoid overloading workers. The potential benefit depends on the traffic presented to the system, making prefix reuse a relevant part of benchmark design.

Finally, a deployment has to survive infrastructure problems. HyperPod monitors nodes, GPUs, and EFA and manages node replacement, while orchestration components recreate affected workers. That division of responsibility is part of the integration’s operational value: serving configuration and infrastructure recovery are connected, although the supplied evidence does not quantify recovery times or availability outcomes.

Integration benefits of Dynamo with HyperPod

Symbolic diagram illustrating: Integration benefits of Dynamo with HyperPod

The integration combines Dynamo’s serving choices with HyperPod’s infrastructure controls. Teams can use aggregated or disaggregated serving, run workers on one instance or across several, and apply KV-aware routing. HyperPod supports those arrangements through EFA networking, coordinated scheduling, and health management. AWS also describes managed component installation and configuration defaults intended to simplify endpoint setup.

The strongest numerical evidence comes from two AWS benchmarks, each answering a different question. Both used AIPerf, but their configurations and purposes differ substantially.

ExperimentConfiguration and workloadAWS-reported resultScope of the finding
Aggregated versus disaggregated servingQwen3-235B-A22B-FP8 with Dynamo and TensorRT-LLM on two p5en.48xlarge instances, totaling 16 H200 GPUs; requests with 4,000 input tokens and 200 output tokensDisaggregated serving reached 207 output tokens per second per GPU versus 101 for aggregated serving, or 2.05× the highest aggregated throughput among retained pointsRetained measurements met P90 time to first token below one second and median generation speed of at least 50 tokens per second per user; the reported throughput points used concurrency 80 and 32, respectively
Single-node versus two-node aggregated servingQwen3.5-397B-A17B-FP8 with Dynamo and vLLM, TP4 × PP2, and eight active H200 GPUs; an agentic workload with high prefix reuse at concurrency 32The two-node configuration delivered 126.4 output tokens per second per GPU, retaining 99.1 percent of single-node throughputThe comparison held model, parallelism, and workload configuration constant; this model fits on one instance and was used as a controlled baseline

The first experiment supports a specific case for disaggregation under latency constraints. AWS selected measurements that satisfied both a prompt-response threshold and a per-user generation threshold, then compared the highest throughput within that set. The 2.05× result therefore reflects the tested workload and qualifying operating points, including their different concurrency levels. It should not be read as a same-concurrency comparison or a guarantee for other request patterns.

The second experiment isolates the effect of crossing an instance boundary more directly. AWS compared eight GPUs on one p5en.48xlarge with four GPUs on each of two instances. Retaining 99.1 percent of throughput suggests that the node boundary imposed little throughput penalty in that configuration. However, because the model fits on one instance, this test is a baseline for multi-node execution rather than a direct benchmark of a model that requires multiple instances for memory capacity.

These measurements support the integration’s technical premise while leaving deployment economics open. The supplied evidence does not report a cost comparison or establish the best topology for every workload. Teams still need to test their own latency targets, concurrency, prompt lengths, output lengths, and prefix reuse before deciding whether the extra deployment structure is worthwhile.

Deployment architecture and management

The HyperPod Inference add-on installs Dynamo custom resource definitions and the components responsible for serving and scheduling. Their responsibilities are distinct, which helps explain what the managed integration actually handles.

The HyperPod Inference Operator manages endpoint deployment through an InferenceEndpointConfig (IEC) custom resource. Its dynamoSpec section contains the Dynamo configuration. The Dynamo operator manages resources described by a DynamoGraphDeployment (DGD), including the frontend, model workers, replica counts, and resource requirements.

Grove coordinates related groups of pods, including multi-node workers and their scaling relationships. Kubernetes AI (KAI) Scheduler places pods on GPU resources and uses gang scheduling to arrange the pods needed for a multi-node worker together. AWS Deep Learning Containers provide the Dynamo runtime. The add-on manages integrated component versions; AWS says adopting a supported Dynamo release requires updating the add-on.

Within the serving graph, the Dynamo frontend routes requests to workers. In disaggregated deployments, prefill and decode use separate pools with independent scaling and tuning. In aggregated deployments, both phases run within a worker. Neither choice dictates whether the worker occupies one instance or spans several: topology and the number of instances per worker are separate configuration decisions.

Two configuration paths

The integration offers a choice between bringing an existing DGD and defining an IEC with dynamoSpec. Both paths support single-instance and multi-node deployments, and HyperPod handles node health and replacement in either case. Their management responsibilities differ.

ResponsibilityBring your own DGDIEC with dynamoSpec
Deployment definitionWrite the complete DGD manifestWrite an IEC containing dynamoSpec
Model weightsConfigure downloading or mountingOperator stages weights from Amazon S3 or Amazon FSx
Endpoint, TLS, and registrationConfigure and manage these resourcesManaged through the HyperPod Inference Operator
AutoscalingConfigure and manage scalingKEDA through autoScalingSpec for aggregated and disaggregated deployments

The DGD path suits teams seeking direct control of the serving graph or reuse of existing manifests. AWS specifies that adopting an existing DGD requires a supported container image and storage, networking, and pod placement configured for the cluster. Bringing the graph does not remove those integration responsibilities.

The IEC path offers managed endpoint setup within HyperPod’s configuration format and an OpenAI-compatible interface. It supports vLLM, SGLang, and TensorRT-LLM, with vLLM as the default engine. The operator also supplies defaults for KV-aware routing and replica counts. Deployments configured through IEC with dynamoSpec support invocation through SageMaker endpoints.

What the deployment example establishes

AWS illustrates the IEC path using GLM-5.3, vLLM, and ml.p5.48xlarge instances. Its aggregated configuration has one model-worker replica spanning two instances, or 16 GPUs, and two frontend replicas. Setting dynamoSpec.frontend.routingMode to kv-aware enables KV-aware routing. The model-worker count is controlled by spec.replicas, separately from frontend.replicas.

The disaggregated example replaces aggregation with disaggregation and sets prefill and decode replica counts separately. One prefill replica and two decode replicas each span two instances, producing a total of six instances. This makes the resource implications concrete: separating phases introduces independently sized worker pools, each with its own multi-node allocation.

After the complete manifest is applied, the HyperPod Inference Operator makes configured weights available and generates the DGD. Grove and KAI Scheduler coordinate placement of the multi-node workers. Once the SageMaker endpoint is ready and configured for chat completions, clients can send an OpenAI-compatible request payload through the SageMaker Runtime API.

Tuning against representative traffic

AWS recommends starting from a Dynamo recipe matching the model, inference engine, and GPU type, with memory sizing that includes weights, KV cache, and runtime overhead. For supported configurations, AISimulate can help compare serving topologies, parallelism, and worker-pool sizing before GPU testing.

Communication should be checked before benchmarking, including EFA transport for cache transfers. AWS then recommends replaying representative traffic with AIPerf or using a generated agentic trace with representative lengths and prefix reuse, sweeping concurrency, and retaining measurements that meet service-level targets. Its tuning guidance prioritizes topology and worker allocation before engine settings, changing one variable at a time and validating candidates on the target HyperPod deployment.

Conclusion on the impact of the integration

Dynamo on HyperPod gives teams a connected way to configure distributed inference and manage the infrastructure beneath it. Its clearest practical contribution is the combination of serving flexibility, coordinated placement, managed endpoint options, and node recovery. The IEC path transfers several deployment responsibilities to the operator, while the DGD path preserves direct control for teams that need it.

AWS’s benchmarks show why those options merit evaluation: disaggregation improved qualifying throughput in one test, and a separate controlled comparison retained nearly all single-node throughput across two nodes. The remaining decision is workload-specific. Choosing a topology requires representative measurements and explicit latency and cost priorities; the supplied evidence supports that evaluation process more strongly than any blanket claim of superior efficiency.