Amazon SageMaker HyperPod governance requires controls at several layers: infrastructure boundaries, project access, cluster isolation, and workload allocation. Connecting a cluster to Amazon SageMaker Unified Studio gives a team a workspace from which to launch workloads. It does not establish a strong runtime security boundary. AWS’s administration guidance explicitly describes Unified Studio projects as collaboration boundaries and says clusters remain managed through SageMaker AI interfaces and APIs. AWS administration guidance.
For platform engineers, that distinction determines the architecture. Project membership, tenant permissions, and accelerator quotas each need an explicit design. The walkthrough below follows AWS’s documented controls; it is an analysis of vendor guidance rather than an independent validation of isolation or scheduling behavior.
Start with infrastructure permissions
HyperPod provides accelerated compute for model training and fine-tuning. Before deciding how teams share that compute, establish who can create or change the infrastructure and which configurations they may select.
AWS documents three IAM condition keys for granular control over the CreateCluster and UpdateCluster operations:
| Condition key | Documented control |
|---|---|
| <code>sagemaker:InstanceTypes</code> | Control access based on specified instance types |
| <code>sagemaker:VpcSubnets</code> | Restrict cluster creation or updates to specified VPC subnets |
| <code>sagemaker:VpcSecurityGroupIds</code> | Control access based on VPC security group IDs |
These controls give platform teams a way to express infrastructure restrictions at the cluster API layer. They address which cluster configurations an authorized caller can create or update; the tenant and workload controls below address a different part of the governance design. AWS also documents cluster management through CloudFormation and Terraform, making infrastructure definition another part of the administration workflow. HyperPod release notes.
Treat project collaboration and runtime isolation separately
Unified Studio can connect a HyperPod cluster to a project so members can launch workloads from their project workspace. AWS nevertheless places responsibility for runtime boundaries in the underlying cluster controls.
For Amazon EKS, AWS recommends a namespace per tenant, combined with role-based access control (RBAC), per-tenant service accounts, and EKS Pod Identity roles. Its guidance also calls for default-deny network policies and tenant-specific storage and AWS Key Management Service (AWS KMS) permissions. Read together, these recommendations cover several distinct concerns: access to Kubernetes resources, workload identity, network communication, and access to stored data. AWS administration guidance.
The practical implication is to review those controls as a set. A project-to-cluster connection explains how a team reaches compute; a namespace, identity, network, and storage design explains how its workloads are governed once there. Project membership alone does not answer both questions.
AWS’s discussion also highlights task visibility in SageMaker AI Studio, a distinction worth preserving when reviewing the user experience. It reports that AI Studio users can see all EKS cluster tasks by default, while for Slurm every AI Studio user can view, manage, and interact with available tasks. Review that documented behavior explicitly rather than assuming the project boundary establishes task privacy. AWS administration guidance.
Choose capacity controls for the orchestrator
AWS recommends different mechanisms for sharing capacity on EKS and Slurm. The comparison matters because its guidance explicitly says Slurm does not provide the same lending-and-borrowing model.
| Governance concern | Amazon EKS guidance | Slurm guidance |
|---|---|---|
| Team allocation | HyperPod task governance quotas | Slurm accounting with hierarchical accounts and associations |
| Scheduling policy | Priority classes and preemption | Quality of service (QoS), priority, fair-share, partitions, and preemption controls |
| Shared spare capacity | Lending and borrowing through task governance | No equivalent lending-and-borrowing model documented in this guidance |
These are orchestrator-specific designs for allocating a shared accelerator pool. A platform policy should therefore identify the actual scheduling mechanism behind a team’s allocation, rather than treating an EKS quota and a Slurm account as interchangeable controls. AWS administration guidance.
On EKS, the HyperPod task governance add-on installs Kueue, which AWS describes as a Kubernetes-native system for managing quotas and how jobs consume them. The add-on is thus a concrete dependency of this governance approach, with installation prerequisites and service-managed resources that administrators need to account for. Task governance setup documentation.
Establish who manages scheduling resources
The task governance setup documentation specifies Kubernetes version 1.30 or later and requires a HyperPod node to exist in the EKS cluster before add-on installation. It also directs users with an existing Kueue installation to uninstall it before installing the EKS add-on. These prerequisites make adoption an administration change that needs planning, particularly for a cluster already using Kueue. Task governance setup documentation.
After installation, HyperPod creates and modifies SageMaker AI-managed Kubernetes resources, including KueueManagerConfig, ClusterQueues, LocalQueues, WorkloadPriorityClasses, ResourceFlavors, and ValidatingAdmissionPolicies. Administrators can modify their state, but AWS warns that the service may update and overwrite those changes. Task governance setup documentation.
That warning has a direct operational implication: a successful manual edit is insufficient evidence that a scheduling policy will persist. Platform teams should document which resources the service manages and account for possible overwrites when reviewing changes. Administrative access and durable policy ownership are separate questions in this setup.
Connect quotas to usage evidence
For EKS workloads submitted through the HyperPod CLI or kubectl, AWS documents tracking of GPU and CPU hours across namespaces and teams. Usage reports distinguish allocated from borrowed resource consumption, provide team utilization for auditing for up to 180 days, and support cost attribution aligned with Task Governance policies. The usage report infrastructure must be installed to use those reports; AWS strongly recommends configuring Task Governance to enforce quotas and enable granular attribution. EKS job documentation.
This creates a useful connection between allocation policy and its operational record. Quotas describe the capacity a team is assigned, lending and borrowing describe how shared capacity can be consumed, and reporting provides evidence of that consumption. Platform teams should establish both the allocation rules and the reporting infrastructure before relying on usage-based attribution.
The documented workflow also extends beyond the project workspace. Scientists can use the HyperPod CLI and native kubectl commands to find clusters, submit training jobs as Pods, and manage workloads. AWS documents Kubeflow Training Operator use according to HyperPod-managed compute quotas. Governance planning should therefore include these submission workflows when assessing how teams consume their allocations. EKS job documentation.
Keep operations specific to the cluster type
AWS documents CloudWatch health and performance metrics for Slurm clusters, enabled by default under /aws/sagemaker/Clusters. These cover CPU, GPU, and memory utilization, plus cluster information such as node counts and failed nodes. This Slurm monitoring capability should be distinguished from the EKS usage-reporting workflow and its installation requirement. HyperPod release notes.
The resulting governance review has concrete questions: which infrastructure changes are permitted, which runtime controls isolate tenants, which scheduler policies allocate capacity, who manages the relevant resources, and which reports demonstrate consumption. AWS’s guidance supplies mechanisms for each layer. The platform team’s work is to connect them into an explicit operating model, with the EKS and Slurm differences preserved.
Related reading: How to Implement Access Control for AI Agents
