Running speech-to-text models at scale is a GPU problem before it is a software problem, and that is exactly what makes cost-optimized speech-to-text inference hard: every audio file triggers a pipeline of deep learning models, and at AssemblyAI’s production volume that means thousands of GPU instances running around the clock. AssemblyAI, whose API transcribes speech for high-volume workloads in technology, media, healthcare and financial services, faced that bill directly — and solved it without giving up the latency and availability guarantees its customers depend on. Its answer, described in a recent AWS case study, is to run inference on Amazon EC2 Spot Instances inside Amazon ECS, with a queue-based retry architecture that turns interruptions into routine events. The result: 70% lower inference compute costs than On-Demand pricing, sustained while compute demand grew nearly fourfold year over year.
This article walks through the architecture piece by piece: why the workload is so expensive, how Spot Instances and SQS retries fit together, how capacity providers and GPU diversification keep the fleet running, what actually happens in the two minutes before a Spot interruption, how the same design meets strict data-residency rules, and whether the pattern transfers to your own AI workloads.
Why speech-to-text inference is so expensive to run
Speech-to-text inference is computationally intensive because transcription is not a single model call. Each audio file triggers a pipeline of deep learning models — acoustic models, language models and post-processing stages — running on NVIDIA GPUs to meet latency requirements. A single short clip is cheap; millions of clips a day are not.
At AssemblyAI’s scale, that means thousands of GPU instances running continuously, each running a single inference model per GPU. Running that fleet entirely on On-Demand Instances would be cost-prohibitive, in the AWS authors’ own words. Spot Instances offer significant savings — AWS documentation puts the figure at up to 90% compared to On-Demand prices — but they come with a catch: Amazon EC2 can reclaim a Spot Instance with two minutes of notice when capacity is needed elsewhere, so any workload using them must be built to tolerate interruptions and retries.
The tension is therefore not just cost versus performance. It is cost versus the service level agreement. Customers in regulated industries, including healthcare, finance and legal, depend on transcriptions arriving on time, and the platform must keep its availability and latency guarantees while the underlying compute can disappear at two minutes’ notice. Any architecture that chases the Spot discount has to absorb interruptions gracefully, or the savings are not worth having.
Cost-optimized speech-to-text inference: Spot Instances plus queue-based retries
The core idea is to decouple job submission from job execution. When a transcription request arrives, it is placed in an Amazon SQS queue instead of being handed directly to a GPU. GPU-backed ECS tasks poll the queue, pull jobs, and execute the inference pipeline. Each task retrieves the audio file from Amazon S3 and writes the completed transcription to a results store, which AssemblyAI configures to deliver the output to the end customer.
Because the work is a queue of independent jobs rather than a live service pinned to specific machines, an interruption stops being a failure and becomes a retry. A job that is interrupted mid-flight simply goes back on the queue, where another healthy GPU instance picks it up. Individual jobs typically take between 15 seconds and several minutes depending on audio length.
The full architecture is an Amazon ECS cluster running in a VPC across multiple Availability Zones, with GPU-backed ECS tasks, an SQS main job queue, a dead-letter queue for jobs that exhaust their retries, and Amazon CloudWatch Metrics watching fleet health and Spot interruption events. It is built on four pillars that work together: ECS capacity providers mixing Spot and On-Demand, multi-family GPU Spot diversification, the queue-based retry architecture, and regional deployment for data residency.
ECS capacity providers and multi-family GPU diversification
The first cost lever is how capacity is bought. AssemblyAI configures multiple capacity providers in a single capacity provider strategy within each ECS cluster. An On-Demand capacity provider serves as an always-on baseline, and a Spot capacity provider spans different GPU instance families and Availability Zones to serve spikes of demand. ECS managed scaling automates the scaling of GPU instances based on the capacity provider weights and the pending inference tasks: when the task queue grows, the Spot capacity provider requests additional instances from the diversified pool, and when demand drops, instances scale in gracefully.
The second lever is diversification itself, and AssemblyAI calls it the most effective way to minimise Spot interruption rates. Rather than depending on a single GPU instance family, the infrastructure spans three EC2 GPU instance families:
- g4dn instances — powered by NVIDIA T4 GPUs, providing a cost-effective baseline for standard speech recognition models.
- g5 instances — equipped with NVIDIA A10G GPUs, delivering higher throughput.
- g6 instances — featuring NVIDIA L4 GPUs, offering improved performance-per-dollar.
Sizes in use include g4dn.xlarge, g4dn.2xlarge, g5.xlarge, g5.4xlarge and g6.2xlarge. Spreading demand across families and sizes draws from a significantly larger pool of available Spot capacity, because each family taps an independent capacity pool, so the fleet is not reliant on any single family. On top of that, the price-capacity optimized allocation strategy provisions instances from the Spot capacity pools with the highest available capacity at the lowest price, which further reduces the likelihood of interruptions.
This matches AWS’s own Spot guidance, which recommends being flexible across at least 10 instance types for each workload and using Spot placement scores — a 1-to-10 rating of the likelihood of successfully provisioning requested Spot capacity in a given Region or Availability Zone — to pick where to run.
Surviving a Spot interruption without breaking the SLA
The retry mechanism is where the design earns its keep. When a Spot Instance receives a two-minute interruption notice, the recovery flow works as follows:
- Amazon EventBridge invokes the Spot Interruption Handler, an AWS Lambda function, on the interruption notice.
- The handler returns the in-flight job to the Main Job SQS Queue, where it becomes immediately available to other workers.
- Another healthy GPU instance picks up the message — potentially on a different instance type or in a different Availability Zone — and completes the inference.
- Jobs that exhaust their retry attempts are routed to a Dead Letter SQS Queue for investigation.
Jobs that can complete within the two-minute window simply finish normally. To reduce recovery time further, AssemblyAI creates custom Amazon Machine Images containing model artifacts and container images in approximately two minutes. For end customers, the impact of a Spot interruption is a small increase in processing time — but the transcription completes successfully and is delivered as expected.
Data residency: running the same architecture in regulated regions
Cost is only half the story. AssemblyAI serves customers in regulated industries — healthcare, financial services and legal — where audio data must never leave specific geographic boundaries. The company implements strict data residency configurations in European AWS Regions, and the architecture supports that by being fully replicated per region rather than stretched across them.
Each AWS Region runs a fully independent copy of the inference architecture: its own SQS queues, ECS clusters, capacity providers and GPU fleets. The deployment spans US West (Oregon), us-west-2, and Europe (London), eu-west-2. Multiple GPU instance families (g4dn, g5, g6) are available in these European Regions with competitive Spot pricing, so the cost optimisation does not depend on running in a single cheap region — the instance diversity that makes Spot reliable is available where the data is allowed to live.
A job interrupted in London is retried in London, not moved to a region where the audio is not permitted to go.
The results: 70% cheaper inference compute
The headline number is the one AssemblyAI reports itself: by running Amazon EC2 Spot Instances with GPUs in production, the company achieved 70 percent cost savings on inference compute compared to On-Demand pricing. That is the figure for its speech-to-text pipeline, measured on its own production fleet — not a generic Spot discount claim. (AWS’s general documentation cites savings of up to 90% for Spot versus On-Demand, but that is a ceiling across all workloads, not this workload’s result.)
The savings held as the platform grew: compute demand rose nearly fourfold year over year, and the Spot-based architecture kept unit economics sustainable through that growth. The lessons AssemblyAI draws from operating at that scale are practical rather than theoretical. Diversifying across instance families and sizes is the most effective lever for minimising Spot interruption rates, because the families tap independent capacity pools. And designing fault tolerance from day one — queue-based retries and fast warm-up — turns Spot interruptions into routine events while maintaining SLA commitments, in the authors’ words.
It is worth noting what the 70% figure does not claim. It is a saving on inference compute, not on the total cost of running the platform. Total-platform savings are not quantified by the evidence available.
Can you apply this pattern to your own AI workloads?
Teams running batch or near-real-time AI workloads with similar characteristics can apply this pattern directly. The AWS authors name the characteristics explicitly: short-lived, independently retriable jobs on GPU compute. If your workload looks like that — transcription, batch image or video inference, offline embedding generation, evaluation runs — the recipe is to start by diversifying instance families and building queue-based fault tolerance, and only then to take advantage of Spot pricing.
The limits are just as explicit, and AWS’s own documentation states them bluntly. Spot Instances are recommended for stateless, fault-tolerant, flexible applications. They are not suitable for workloads that are inflexible, stateful, fault-intolerant, or tightly coupled between instance nodes. A real-time streaming workload that cannot tolerate a retry delay does not fit; a batch job that can wait a few minutes does.
There are also traps the documentation warns against. AWS strongly discourages failing over to On-Demand Instances to handle Spot interruptions, because that can inadvertently drive interruptions for your other Spot Instances. The design works because the queue absorbs the interruption, not because On-Demand is waiting as a safety net.
For teams that want to try it, the practical starting points are the same ones the case study points to: the Amazon ECS documentation for GPU workloads, Spot placement scores for checking accelerated-compute capacity in your target Regions, and the EC2 Spot best practices guide. And the open question worth carrying into any evaluation is whether the 70% figure transfers: it was measured on a speech-to-text pipeline of short, retriable jobs, and a different model type — large language model inference, for example — would need its own measurement before anyone quotes the number.
