AWS says three kernel optimizations for fused Mixture-of-Experts (MoE) inference on Amazon SageMaker AI deliver up to 10% higher throughput and up to 9% faster Time to First Token on H200 GPUs. The changes target matrix execution, backend selection for FP8 quantization, and unnecessary Triton just-in-time compilation.
The figures come from AWS’s engineering account. They are vendor-reported results, with different conditions attached to the individual optimizations. The supplied evidence does not establish that their throughput gains can be added together.
For cloud architects and ML engineers, the useful detail is where each change applies. AWS identifies separate improvements to how matrix operations use Hopper hardware, how an FP8 pipeline executes, and when Triton recompiles a kernel. Each addresses a different source of inference overhead.
Swap-AB targets matrix execution on Hopper
MoE architectures use sparse activation to increase language-model capacity while keeping per-token computation low. A fused MoE system combines token routing, data shuffling, and expert-network calculations into a single batched GPU operation. AWS’s work focuses on improving execution within that system.
In its technical explanation, AWS describes Hopper’s Warp Group Matrix Multiply Accumulate (WGMMA) protocol, which operates collectively across 128 threads. The underlying wgmma.mma_async instruction fixes the matrix row dimension, M, at 64.
AWS presents Swap-AB, a matrix transposition optimization, as a way to improve WGMMA utilization. It reports a 2–4 percent end-to-end output throughput improvement across large-scale MoE models on H200 GPUs. According to AWS, the optimization was upstreamed to vLLM and is available in version 0.24.0 or later.
The distinction between instruction-level mechanics and the reported result matters: the 2–4 percent figure describes end-to-end output throughput, rather than an isolated matrix-operation benchmark. It remains a result reported by AWS for the models and hardware in its account, rather than an established gain for every MoE deployment.
Triton improves the FP8 execution path
The second optimization concerns backend selection. AWS reports that, for FP8 tensor-quantized models on Hopper in low-concurrency MoE workloads, the Triton backend outperforms the default FLASHINFER_CUTLASS path.
AWS attributes the advantage to consolidation: Triton combines FP8 tensor scaling, matrix multiplication, and expert aggregation into a single hardware pass. It reports a 10 percent end-to-end throughput improvement for FP8-quantized fused MoE models on H200 GPUs. The backend-routing improvement is available in the AWS Large Model Inference (LMI) v28 container.
This is a specific comparison with a named default execution path. The evidence supports examining Triton for that FP8 workload; it does not establish a general ranking of the two backends across other precisions, concurrency levels, or hardware. For deployment decisions, the quantization format and workload conditions are therefore as relevant as the headline percentage.
Removing recompilation improves the first-token result
The third change addresses compilation behavior. Triton triggers fresh just-in-time (JIT) compilation when a kernel receives a previously unseen value for a specialized parameter. AWS identifies variable-length parameters in _fused_inv_rope_fp8_quant_per_head as the target of its fix.
Adding do_not_specialize annotations to those parameters removed the recompilations, AWS says. It reports a 9 percent improvement in Time to First Token (TTFT) on H200 GPUs at 8-way tensor parallelism. AWS says this optimization is available in vLLM version 0.22.0 or later.
That result measures a different aspect of inference from output throughput. It also has an explicit parallelism condition. Treating the TTFT result separately preserves what the benchmark actually says, without implying that it measures a 9 percent improvement in every part of a response.
AWS says it applies these optimizations in the LMI Deep Learning Container on SageMaker AI. The availability details nevertheless differ: Swap-AB is associated with vLLM 0.24.0 or later, the recompilation fix with vLLM 0.22.0 or later, and FP8 backend routing with LMI v28.
Together, the changes illustrate three concrete places to examine when evaluating fused MoE inference: matrix utilization, the selected FP8 execution path, and specialization-driven compilation. The evidence contains two URL forms of the same AWS article, rather than independent corroboration. Its reported gains provide workload-specific reference points; they do not establish performance outside the documented conditions.
