Streaming Low-Latency Speech from AI Agents with Kokoro TTS

A voice agent that takes five seconds to start speaking feels broken. The problem is the wait before the first sound: synthesis holds until the full response is ready, even though the first sentence is already available. This article shows how to build low-latency agent speech with Kokoro TTS on Amazon SageMaker AI, streaming each sentence to synthesis as soon as the agent produces it.

Why voice agents sound slow

A customer asks an AI voice agent a question and waits. The agent already has an answer and is midway through a long response, but the customer hears nothing for five seconds — and by the time an answer arrives, the customer is frustrated, confused, and no longer paying attention.

The cause is how text-to-speech integrations usually work: a one-sentence response might generate in about 600 milliseconds, but the full response takes about five seconds, and the integration waits for all of it before synthesis starts. The wait is the problem. The fix is to start synthesis on sentence one: as the agent produces each sentence, it streams immediately to a hosted Kokoro-82M text-to-speech model, so audio starts before the agent finishes its full response.

Streaming synthesis alongside generation

Kokoro-82M is the model at the centre of this pattern: an open-weight text-to-speech model with 82 million parameters, built on StyleTTS 2 and ISTFTNet, version 1.0 released on 27 January 2025 under Apache 2.0 weights, covering eight languages with 54 preset voices, producing 24 kHz speech. At that size it runs cheaply on a single mid-tier GPU instance, keeping latency and cost low.

The processes overlap in time: the agent generates text token by token, each completed sentence leaves for the speech model immediately, and first-sentence audio can arrive before generation finishes. The win comes entirely from that overlap — nothing makes the model faster; it removes the serial wait.

This is not concurrent synthesis: the walkthrough’s container handler is a strictly sequential loop for each session — receive, synthesize, send audio, send synthesis_complete, repeat — so sentence two’s audio is not generated while sentence one’s is still transmitting. What incremental sending buys is narrower and still decisive: synthesis on sentence one begins the instant it exists, rather than waiting for the full response.

The bidirectional SageMaker endpoint

Kokoro-82M runs in a custom container on a SageMaker AI real-time endpoint on an ml.g5.xlarge GPU instance, exposing two routes: POST /invocations for one complete input, and WS /invocations-bidirectional-stream, which keeps one connection open across many messages in both directions — the pattern here uses the bidirectional route.

The container must declare the Docker label com.amazonaws.sagemaker.capabilities.bidirectional-streaming=true.

The first pitfall is the frame type SageMaker AI actually sends: the bidirectional transport delivers every client message as a binary WebSocket frame, while a common Starlette handler calls receive_text(), which reads message["text"] and raises KeyError if that key is absent — so it fails on every request.

It is commonly misdiagnosed: a generic handler forwarding str(e) turns KeyError('text') into an error frame reading 'text', which looks like a missing field — apparently — while the endpoint reports InService throughout. Accept either frame type explicitly:

from starlette.websockets import WebSocket

async def _receive_message(websocket: WebSocket) -> str:
    message = await websocket.receive()

    payload = message.get("text")

    if payload is None:
        payload = message.get("bytes", b"").decode("utf-8")

    return payload

Chunking sentences as tokens arrive

The client-side fix is a class that normalizes two input shapes — a complete string, or a live token stream — into complete sentences, ready to send as soon as each one is complete.

import re

_SENTENCE_RE = re.compile(r"(?<=[.!?])\s+")

class IncrementalSentenceChunker:
    def __init__(self) -> None:
        self._buffer = ""

    def feed(self, fragment: str) -> list[str]:
        self._buffer += fragment

        parts = _SENTENCE_RE.split(self._buffer)

        if len(parts) == 1:
            return []

        self._buffer = parts[-1]

        return [s.strip() for s in parts[:-1] if s.strip()]

    def flush(self) -> str | None:
        remainder = self._buffer.strip()
        self._buffer = ""
        return remainder or None

Call feed() once per fragment as it arrives; it returns newly completed sentences and holds back trailing text. Call flush() once at the end to release partial text remaining without trailing punctuation, so a stream ending before a period does not drop its final sentence.

Two details deserve attention. First, sentence boundaries are the demonstrated chunking heuristic: Kokoro’s voices perform best on roughly 100 to 200 tokens, are weak under 10 to 20, and rush over 400, so sentence length still needs attention — a short sentence can fall below the weak range and a long one can exceed the comfortable one. Second, session lifetime matters as much as chunking: after sending audio and a synthesis_complete frame, the container loop should wait on receive() for the next message on the same open session, closing only when the client sends an explicit close message:

{"type": "close"}

Closing eagerly after one response would force a new connection per sentence.

Getting low-latency agent speech back to the browser

Streaming text into the model solves half the problem; the other half is getting the audio out to whatever renders it, commonly a browser. Where a frontend already streams text over Server-Sent Events (SSE) — one-directional, natively supported by browsers — SSE is a text protocol, not built for raw binary payloads. A team adding voice to an existing SSE pipeline naturally reaches for the pattern it already has: base64-encode the audio and wrap it in a JSON event.

The transport measurements for the three encodings:

Base64 inflates every payload by close to the expected 33 percent. The subtler cost is frame alignment: SSE events are delimited by a blank line, the audio inside by MP3 frames, and the two do not necessarily align with the transport’s chunking. In one measured response, four logical MP3 frames arrived as six delivery chunks, two ending mid-frame — a consumer treating each chunk as one complete unit fails on ordinary, healthy traffic. Buffer incoming bytes and act only once a delimiter is found:

buffer = b""
for event in resp["Body"]:
    if "PayloadPart" not in event:
        continue

    buffer += event["PayloadPart"]["Bytes"]

    while b"\n\n" in buffer:
        frame, buffer = buffer.split(b"\n\n", 1)
        handle(frame)

The alternative for the model-facing leg skips base64 and SSE framing entirely, multiplexing raw binary audio and JSON control frames over the same bidirectional session already carrying incremental text input, with one rule — a chunk that fails to decode as a JSON object is audio, forwarded as received, with no encoding step anywhere in the path.

import json

def _decode_control_frame(chunk: bytes) -> dict | None:
    try:
        decoded = json.loads(chunk.decode("utf-8"))
    except (UnicodeDecodeError, json.JSONDecodeError):
        return None

    return decoded if isinstance(decoded, dict) else None

Raw PCM audio can legitimately begin with byte 0x7b, the ASCII { that opens a JSON object — so the rule requires the entire chunk to parse as valid JSON, not its first byte alone.

Pragmatically: with a working SSE pipeline for text, base64-encoded audio inside an SSE event is the pragmatic choice — roughly 33 percent extra payload and a buffer-and-reassemble step, but infrastructure already in operation. If the voice path is being built fresh, or latency is the priority, the unified binary and JSON approach avoids both costs and reuses the connection already carrying text.

What the measurements show

Two measurements matter. The first isolates the TTS transport’s own contribution: the same 267-character, four-sentence block sent two ways — as one message (batch), and as only its first sentence (incremental) — five trials each. Sending only the first sentence cut mean time to first audio byte from 329.3 ms to 156.7 ms — a 52.4 percent relative improvement, isolated to the TTS transport itself.

The second measures the fuller end-to-end case with a real language model producing the text, comparing bidi and batch modes against a live Bedrock model and endpoint. Bidi mode’s time to first audio measured 2.4 times faster than batch locally, and 3.5 times faster on the deployed AgentCore runtime. The gap widens for the deployed case once a real network hop is added: bidi absorbs that hop inside the overlap it already exploits between generation and synthesis.

To reproduce the comparison, run the bidi-versus-batch test the agent container exposes:

uv run agent-infra demo --endpoint-url <your-endpoint-url> --prompt "<your prompt>"

This sends the prompt through both modes and reports each mode’s first audio_chunk timestamp against agent_start. If errors appear instead, check for the KeyError: 'text' symptom — an endpoint reporting InService while every request fails signals that bug.

Cost is the other half of the decision: an ml.g5.xlarge endpoint bills $1.4080 per instance-hour in us-east-1 on demand — about $1,028 for a 730-hour month left running, compute only, before storage and data transfer — from the moment it reaches InService until deleted, regardless of traffic, plus $0.016 per GB of data processed in and out.

Putting it together and cleaning up

The fuller end-to-end case is a live, generative agent composing its response one token at a time: a Strands agent, backed by Amazon Bedrock, deployed to Bedrock AgentCore runtime, calling the Kokoro endpoint with the pattern from the transport section. Three legs connect in this design: the browser-facing SSE leg, the agent’s streaming Bedrock call, and the bidirectional Kokoro session.

The design compares two paths: one streams synthesis alongside generation, the other waits for the complete response before a single synthesis call, as a baseline. In the streaming path, text generation and synthesis overlap, so audio for an early sentence can reach the caller while the model is still generating later ones; the batch path does one thing at a time.

Going the other direction, the leg back to the caller does use base64, for a different reason: BedrockAgentCoreApp‘s entrypoint must yield plain, JSON-serializable dictionaries, with no way to hand it a raw binary frame — a consequence of the JSON-native managed runtime, not a design choice. Both modes pay that cost on this leg.

Two CLI tools, each wrapping its own AWS CDK stacks, handle deployment — endpoint first:

uv run speech-infra deploy kokoro-82m
uv run agent-infra deploy --bedrock-model-id <your-bedrock-model-id>

To avoid ongoing charges, tear down in reverse order, agent first, then endpoint:

uv run agent-infra destroy --bedrock-model-id <your-bedrock-model-id>
uv run speech-infra destroy kokoro-82m

Neither destroy empties the S3 bucket caching model weights or deletes the ECR repository — both can be shared across models. If no longer needed, empty and delete them manually — a stored checkpoint and image keep incurring small storage charges until removed.

One warning before running this beyond a proof of value: the sample container pins an older PyTorch with a known advisory (PYSEC-2026-2286, fixed in torch 2.10.0) — review and upgrade the pin first.

Waiting for an agent’s full response before starting speech synthesis is one source of avoidable latency; base64-encoded audio inside SSE adds a second — about a third more payload, plus frame-realignment work without buffering. Streaming each sentence into a SageMaker AI bidirectional endpoint as soon as the agent produces it cut time to first audio by more than half in isolation, by 2.4 times locally, and by 3.5 times on the deployed AgentCore runtime. Multiplexing binary audio and JSON control frames over that session, instead of encoding audio as text, removes the size inflation and the frame-realignment work a naive SSE relay requires. If voice is being added to an agent that already streams text over SSE, start with base64 and revisit the unified transport when latency matters.