Standard, Technology

How Salesforce uses cross-Region inference routing to address model-serving capacity gaps

Symbolic diagram: How Salesforce uses cross-Region inference routing to address model-serving capacity gaps

Introduction to cross-Region inference routing

Salesforce has integrated cross-Region inference routing in Amazon SageMaker AI into its Internal Gateway to direct model requests toward available capacity across Regions and AWS accounts. The router prefers local endpoints, shifts traffic when destinations become unreliable, and reports where each request ran. Salesforce retains responsibility for checking data residency requirements before requests enter the broader routing path.

The immediate problem is capacity fragmentation. A Region can run out of model-serving capacity while instances elsewhere sit idle. In the AWS account of Salesforce’s implementation, that mismatch makes capacity planning an availability concern: a local shortfall can prevent a request from being served even when the wider fleet has room.

Cross-Region inference routing gives applications one invocation path to eligible SageMaker AI endpoints across several Regions and accounts. It uses Salesforce’s existing deployments rather than creating a new model fleet. The operational change is that capacity awareness, health assessment, and failover move into a managed routing layer, reducing the corresponding logic Salesforce maintains in its gateway.

The evidence supports that architectural change and describes Salesforce’s configuration checks. It does not establish a measured fleet-wide improvement in availability, utilization, or cost. The supplied source ends during its testing discussion, so conclusions about quantified production outcomes would go beyond the available record. The useful question is therefore how the design addresses Salesforce’s capacity problem, and which constraints remain.

Salesforce’s operational scale and model variety

Symbolic diagram illustrating: Salesforce's operational scale and model variety

According to the AWS account, Salesforce’s Agentforce Foundation team serves models for products including Agentforce and the Salesforce Trust Layer across 30+ global Regions and hundreds of AWS accounts. Its portfolio includes models trained in house and open source models. The challenge spans multiple models, customer demand patterns, and deployment boundaries.

Salesforce serves customers from the Region their requests come from and provisions capacity around each customer’s traffic. Customers have distinct peak windows, but those windows often overlap. Provisioning enough capacity for the overlapping peak would be expensive, the source says. Customers can also onboard new business units at short notice, requiring the serving fleet to grow quickly.

Adding capacity separately in each Region creates an uneven distribution: some locations retain unused instances while others exhaust their available capacity. A regional deployment model therefore leaves Salesforce balancing two pressures. More local provisioning can accommodate peaks, while access to eligible capacity elsewhere can help absorb demand that a local endpoint cannot handle.

Model variety adds another boundary to that capacity problem. The router groups destinations under a routing target representing one logical model, such as the toxicity classifier used in the source’s example. Each model has its own target, and problems with one model’s endpoints do not cause traffic for other models to move. Capacity sharing remains tied to destinations registered for the requested model.

That distinction limits what a global routing layer can accomplish. It can select among deployed destinations for a model; it cannot turn an arbitrary idle instance elsewhere into a ready replacement. The source explicitly states that the router does not deploy models, copy artifacts, or add instances. Endpoint autoscaling continues to add instances as endpoint traffic grows. Routing and provisioning remain separate responsibilities.

Previous traffic management methods

Symbolic diagram illustrating: Previous traffic management methods

Salesforce already had a cross-Region fallback path before adopting the inference router. Its Internal Gateway sent traffic that local endpoints could not handle to other Regions. Salesforce built and operated that path using basic health checks, retries, and Region-switching logic.

The source identifies two gaps. First, the gateway lacked capacity signals from both local and remote endpoints. It could not reliably determine when traffic should move or whether the destination had room. A fallback route alone did not provide the information needed to choose an appropriate destination under pressure.

Second, the previous cross-Region path left the virtual private cloud through a NAT gateway and an internet gateway, then traveled over public AWS networking. The described integration places a VPC endpoint using AWS PrivateLink between the Internal Gateway and the routing service. This changes the gateway’s access path, although the supplied evidence does not provide a measured comparison of network overhead or cost.

Salesforce also raised cascading failures as a design concern. Shifting traffic during an incident can overload the next Region. The source says the team needed a cap on cross-Region traffic, but the available material does not document a cap setting or explain how it was enforced. That requirement should not be mistaken for a demonstrated control in the supplied configuration.

The Internal Gateway remains part of the new architecture. The router call occupies the per-request egress position previously used for a direct InvokeEndpoint call, after the residency check and before response handling. This preserves a central policy decision while moving destination health and failover decisions into SageMaker AI.

Retries and timeouts also require coordination. Salesforce’s gateway adds no retries of its own because the router already supports a retry on another destination under specified conditions. Its timeout accounts for the cross-Region round trip so a remote request has time to complete. These choices address two practical consequences of adding a routing layer: attempts can otherwise multiply during incidents, and a failover request can require more time than a local call.

Functionality and benefits of the inference router

The inference router is a Regional SageMaker AI resource in the customer’s account. An application invokes it with a router name and routing target name, while retaining its existing request body, content type, and accept type. A routing target lists eligible endpoints and inference components by their full Amazon Resource Names, or ARNs. Those identifiers allow destinations to reside in other Regions or AWS accounts.

The endpoint owner grants cross-account access through a resource-based endpoint policy. In Salesforce’s described arrangement, that policy restricts invocation to one named router. The example uses the sagemaker:SourceInferenceRouterArn condition for that restriction, and the endpoint owner can revoke access with delete-endpoint-policy. Registering an ARN in a target does not, by itself, establish permission to use it.

Local preference and gradual failover

Requests stay local while the local destination serves successfully. The router evaluates outcomes from recent requests: platform faults, model-container 5xx errors, throttles, and timeouts lower a destination’s health. As those problems persist, it moves a growing share of new requests toward healthy destinations.

According to the source, this shift typically starts within about a minute, but a destination receiving little traffic can take several minutes. Signals including scaling progress and failed instance launches affect the response once errors appear; those signals do not independently move traffic. This is an important qualification to the description of capacity-aware routing: the documented mechanism responds to request failures rather than moving traffic solely because a scaling signal changes.

After recovery, traffic returns over several minutes, more slowly than it moved away, to avoid repeated swings. When a destination fails a request without running the model, such as by throttling it, the router retries that request once on another destination. The evidence does not describe a blanket retry of every failed model request.

Client errors such as invalid input do not trigger traffic movement, because the request would fail elsewhere too. Model latency also does not trigger movement: a slow response can reflect a long prompt or output rather than an unhealthy endpoint. Consequently, the router should not be understood as a mechanism that always selects the fastest Region.

Residency policy and destination readiness

Salesforce keeps the data residency check in its gateway. The integration narrative describes residency-tagged requests as pinned to an in-Region routing target containing only the local endpoint. Other requests use the broader routing path.

There is a difference in how the source depicts the restricted path: the architecture caption says residency-flagged requests go directly to the local endpoint, while the prose describes a local-only routing target. Both descriptions keep those requests within their Region, but they do not establish the same invocation path. The supplied evidence leaves that implementation detail unresolved.

Before sending traffic, operators call DescribeInferenceRouter and inspect each destination’s RoutingStatus. The documented states distinguish readiness from visibility:

StatusDocumented meaning
RoutableSageMaker AI could read the destination and it can serve traffic.
UnroutableSageMaker AI read the destination, but it cannot currently serve, for example because the endpoint is not InService.
UnknownSageMaker AI could not read the destination with the supplied credentials.

An Unknown result can reflect missing sagemaker:DescribeEndpoint or sagemaker:DescribeInferenceComponent permission, or a cross-account endpoint policy that does not grant Describe access. Invocation permission and the ability to inspect readiness therefore need separate attention.

Salesforce’s documented walkthrough uses a router in US West (Oregon), us-west-2, and an endpoint owned by another account in US East (Ohio), us-east-2. It describes confirming local routing, then removing the local destination and confirming requests reached the remote endpoint. That checks routing across account and Region boundaries; it is not equivalent to proving how an overloaded production fleet behaves at scale.

Importance of telemetry in routing performance

Salesforce logs two response fields, InvokedRoutedResourceArn and RoutingReason, alongside the existing request ID and trace ID, and emits them as application performance monitoring tags. The destination ARN identifies the Region and account that served the request. The routing reason explains the selection.

The four documented reasons provide different operational interpretations:

Routing reasonMeaning
local-preferredA healthy destination in the router’s Region served the request.
failoverA local destination exists but is not serving reliably, so the request went elsewhere.
capacity-availableNo destination in the router’s Region could take the request; it went to one of several healthy remote destinations.
only-healthyThe only eligible healthy destination served the request.

These fields make traffic movement visible at the request level. Salesforce joins them with the router’s Amazon CloudWatch metrics when investigating shifts. That combination connects an individual routing decision to the aggregate behavior of destinations.

The router publishes OpenTelemetry metrics to CloudWatch in its own account and Region, with the router name and destination ARN attached. aws.sagemaker.inference_router.invocations counts requests per destination. Its error.type attribute separates invocation_4xx, invocation_5xx, including timeouts, and invocation_model_error for errors returned by the model.

Two duration metrics distinguish the time a destination took from the time the routing path added. aws.sagemaker.inference_router.destination.duration measures destination time; aws.sagemaker.inference_router.overhead.duration includes router overhead and the network journey to and from a remote Region. That separation matters when investigating whether a slower request reflects model execution or the additional routing path, especially because latency alone does not trigger failover.

The metric aws.sagemaker.inference_router.destination.status reports destinations as healthy, overloaded, or degraded. Operators can query the metrics with PromQL in CloudWatch Query Studio and place them alongside existing endpoint metrics on a cross-account, cross-Region dashboard.

The source also describes a dashboard from an AWS failover test using a test model: requests move from a degraded local endpoint in us-east-2 to endpoints in us-west-2, then return gradually after recovery. That illustration is distinct from Salesforce’s configuration walkthrough and should not be treated as a Salesforce production benchmark.

The strongest supported implication is operational: Salesforce has moved routing decisions into a managed layer while retaining residency checks and collecting per-request evidence of destination choice. Establishing the magnitude of any reliability, utilization, or cost improvement would require results beyond the supplied account. The telemetry described here provides a way to examine those outcomes rather than assuming that cross-Region routing alone guarantees them.