Introduction to NVIDIA Run:ai GPU Fractioning
NVIDIA Run:ai GPU Fractioning gives Kubernetes workloads enforced GPU-memory limits so that multiple Pods can share a device that would otherwise be scheduled as a whole GPU. In the Amazon EKS walkthrough documented by AWS, two Pods each receive an 8 GiB memory contract on a single NVIDIA A10G GPU and run probes designed to allocate 6 GiB apiece. The practical question is whether workloads whose memory requirements fit together can use the same GPU without each requesting the entire device.
AWS’s walkthrough describes that capability through NVIDIA Run:ai, AI workload orchestration software for Kubernetes. GPU Fractioning builds on NVIDIA Multi-Process Service (MPS), adding scheduling integration, memory-contract enforcement, and per-Pod metrics. Its demonstrated value concerns workload placement and GPU-memory sharing. A memory contract does not reserve a proportional share of compute, and the walkthrough does not test compute performance or fault containment.
The baseline makes the scheduling problem concrete. With sharing disabled, the NVIDIA device plugin exposes the single GPU as one nvidia.com/gpu resource. Two Pods that each request nvidia.com/gpu: 1 cannot run concurrently on that node, even when their combined GPU-memory demand fits. One Pod runs while the other remains Pending. Fractioning changes how those workloads request and share the available GPU memory.
AWS identifies notebooks, model evaluations, and small inference services as workloads that can encounter this mismatch between available memory and whole-device scheduling. The relevant benefit is allowing suitable workloads to coexist. Whether that also improves response times, throughput, or spending requires evidence beyond the memory-focused demonstration.
Overview of GPU sharing methods

GPU sharing covers several approaches with different isolation properties. AWS discusses Multi-Instance GPU (MIG), device-plugin time-slicing, MPS, and Run:ai GPU Fractioning. Choosing among them requires separating hardware partitioning, concurrent execution, memory enforcement, and error behavior rather than treating all sharing mechanisms as equivalent.
| Approach | Behavior described in the AWS walkthrough | Qualification |
|---|---|---|
| Multi-Instance GPU (MIG) | Provides hardware partitions | Requires supported GPUs |
| Device-plugin time-slicing | Shares a GPU | Does not provide replica memory or fault isolation |
| Multi-Process Service (MPS) | Permits concurrent CUDA processes | Fault containment depends on configuration |
| Run:ai GPU Fractioning | Adds enforced per-container GPU-memory limits and Kubernetes integration | Memory limits do not reserve proportional compute |
Run:ai GPU Fractioning supports a default time-slicing compute mode and an sm-sharing mode. AWS reports that NVIDIA documents separate workload error domains for the default mode and a shared error domain for sm-sharing. Those descriptions are not security guarantees. The example leaves the default compute mode unchanged and does not experimentally establish fault containment for either mode.
Dynamic Resource Allocation (DRA) addresses another part of the system: Kubernetes APIs for requesting and allocating devices. GPU sharing and isolation still depend on the device driver’s implementation and enabled features. Although the source says KAI Scheduler and GPU Operator support DRA, this particular walkthrough uses NvFractions annotations. Its configuration should therefore be read as an NvFractions implementation, without assuming that it demonstrates DRA behavior.
The source’s recommendation is specific: consider fractioning when enforced per-Pod GPU-memory limits are needed on GPUs without MIG and fixed-size GPU node groups are acceptable. That is a workload and infrastructure fit criterion, rather than a claim that fractioning supersedes every other sharing method. Hardware eligibility also needs separate checks; support for a GPU alone does not establish compatibility with the fractioning stack.
Benefits of GPU Fractioning in Kubernetes

The clearest benefit is finer scheduling granularity. In the baseline, one request consumes the node’s only whole-GPU resource. The fractioning example instead assigns memory contracts to two Pods on the same A10G. Each contract is 8 GiB, while each probe allocates 6 GiB. These are distinct quantities: the contract defines the memory request and limit, while the probe exercises an allocation below that limit.
The validated GPU has 23,028 MiB of memory. AWS explicitly requires room for both contracts and runtime overhead. The example consequently cannot be transferred unchanged to a smaller device merely by selecting a different instance type. Smaller devices need smaller contracts and adjusted probes, with compatibility and memory headroom checked again.
Enforcement and visibility complement placement. GPU Fractioning supplies enforced per-container memory limits, while the metricsd daemon exports per-Pod GPU-memory metrics. In the demonstrated workloads, the annotations apply the memory request and limit to the named container. Calling these per-Pod contracts describes the example’s workload allocation; the technical enforcement detail remains per-container.
For teams sharing limited GPU capacity, AWS identifies interactive notebooks with idle periods, short evaluation jobs, GPU compatibility tests, and small inference services as potential fits. The underlying reasoning is conditional: sharing is useful when memory requirements permit co-location and the workloads tolerate the resulting concurrent execution. The source recommends choosing requests from measured peak GPU-memory usage plus overhead, then testing latency under concurrent load.
That measurement requirement limits what can be concluded from the demonstration. Putting two workloads on one GPU establishes a density improvement relative to its whole-device baseline. It does not establish a throughput increase, predictable latency, or a universal reduction in cost. Those outcomes depend on workload behavior that this memory-limit walkthrough does not measure. The useful operational advance is an enforceable memory allocation model, paired with scheduling and metrics that make the shared arrangement observable.
Implementation details on Amazon EKS
AWS validates a specific stack in the US East (Ohio) Region, us-east-2. An m7i.large system node hosts operator controllers and KAI Scheduler. A separate, tainted g5.xlarge worker provides one A10G GPU. The EKS control plane runs Kubernetes 1.36, with workers in private subnets of the customer VPC. The GPU node group stays at one On-Demand instance, with desired, minimum, and maximum capacity all set to one.
The component versions matter because the walkthrough requires a custom host image:
| Component | Validated configuration |
|---|---|
| Amazon EKS | 1.36 |
| GPU | NVIDIA A10G, 23,028 MiB |
| NVIDIA driver | 615.71.10 |
| NVIDIA Container Toolkit | 1.20.1-1 |
| containerd | 2.3.5 with NRI |
| GPU Operator / device plugin | 26.7.1 / 0.20.1 |
| KAI Scheduler / GPU Fractioning Operator | 0.18.5 / 0.1.7 |
AWS says it validated the walkthrough on October 22, 2026. After updating the GPU node taint, it repeated every step except the image build on November 18, using an image with the same driver, and reported matching results. These are lab validation details, not vendor support certification or a permanent compatibility matrix.
Prepare the environment and host image
The prerequisites include permissions for the AWS services used to provision the cluster and discover images, capacity and quota for the GPU worker, system node, and temporary m5.xlarge image builder, and the required command-line tools. Those tools are AWS CLI v2, eksctl, kubectl, Helm, Packer, Git, jq, and curl. The image build targets x86_64; another Region or GPU configuration needs its own compatibility checks.
The supplied workflow uses walkthrough.env for configuration and run.sh for individual stages. Generated files and kubeconfig remain in the ignored .state directory. The initial ./run.sh check confirms the active AWS identity before provisioning. Temporary access keys require AWS_SESSION_TOKEN; AWS recommends temporary credentials and keeping credentials out of Git.
After ./run.sh create-cluster, ./run.sh build-ami builds a custom standard EKS AL2023 image. At validation time, EKS-optimized AL2023 NVIDIA images supplied R580 and R595 driver branches, while GPU Fractioning Operator 0.1.7 required R615 or newer. AWS identifies R615 as short-lived and outside the GPU Operator default.
A subtle reproducibility constraint is that NVIDIA_DRIVER_VERSION selects the R615 branch rather than pinning an exact patch. Packer installs the current patch release from that branch, and the workflow records the observed version in .state/installed-driver-version alongside image metadata. The tested driver was 615.71.10. For production builds, the source recommends exact package and base-image pins and validation of a supported driver-branch upgrade.
Install the GPU stack and establish the baseline
./run.sh add-gpu-node adds the worker using the custom image. The nvidia.com/gpu taint with effect NoSchedule excludes ordinary workloads. The script disables eksctl’s device-plugin installation because GPU Operator manages that component.
./run.sh install-gpu-operator installs GPU Operator while leaving its driver and toolkit management disabled. Ownership stays with the image. A temporary privileged inspection Pod checks the host and is then removed. The validated host uses cgroup2fs and exposes the NRI socket through /var/run/nri/nri.sock and /run/nri/nri.sock; on this AL2023 node, /var/run links to /run, so both paths identify the same socket.
Before installing fractioning, ./run.sh baseline checks whole-device scheduling. The source records whole-gpu-a as Running and whole-gpu-b as Pending. The script checks that state and removes both Pods. This comparison assumes one physical GPU with sharing disabled. On a multi-GPU node, the baseline needs adaptation and the fractional workloads must be verified as using the same device.
Configure and exercise the memory contracts
./run.sh install-kai installs the pinned KAI Scheduler chart, including GPU Fractioning Operator. Its values set gpuSharingMode to NvFractions, enable per-Pod metrics, and retain GPU-stack version checks. The worker runs fractiond with Node Resource Interface integration and MPS for sharing. In AWS’s validation, FractiondReady, MpsdReady, and Ready are True, while DriverUpgradeInProgress is False with reason NoDriverUpgrade, consistent with image-owned drivers.
./run.sh share deploys the two Pods. Each selects kai-scheduler and the nvidia runtime class. The annotations nvidia.com/container.gpu-workload.gpu-memory.request and nvidia.com/container.gpu-workload.gpu-memory.limit are both set to 8Gi. Neither Pod requests a whole nvidia.com/gpu resource. The CUDA probe allocates 6 GiB within each contract.
The source describes subsequent inspection of per-Pod metrics and a test rejecting a 9 GiB allocation. However, the supplied source extract ends during the positive sharing step and does not include the later commands or rejection output. That prevents reproducing those missing procedures or assessing the exact failure behavior from the available evidence.
Cleanup is a separate action even when using ./run.sh walkthrough. The environment creates billable resources, including the EKS control plane, NAT gateway, instances, volumes, and an AMI snapshot. AWS therefore frames it as a disposable validation environment whose cleanup must be completed explicitly.
Conclusion and future implications
Run:ai GPU Fractioning on EKS offers a concrete way to replace whole-device requests with enforced GPU-memory contracts for suitable workloads. The demonstrated configuration connects Kubernetes scheduling, container-level memory limits, and per-Pod visibility on one A10G. Its significance is the ability to place workloads together when their GPU-memory requirements fit.
Adoption still depends on two separate validations: compatibility of the host and GPU stack, and acceptable behavior of the actual workloads under concurrent load. The custom AL2023 image illustrates how driver requirements can shape deployment. AWS recommends preferring an EKS-optimized NVIDIA image once its selected driver and remaining host prerequisites meet fractioning requirements, but the sample scripts also need changes to use a stock image.
The remaining questions concern workload performance, fault behavior, and repeatability as components change. This walkthrough supports memory-sharing and enforcement analysis within its documented scope. Teams evaluating broader operational value need measurements of their own peak memory usage and concurrent latency, alongside renewed compatibility checks, before treating higher workload density as evidence of a production benefit.












