Serving embeddings without melting your p99

An embedding model is fast on average and terrible at the tail. Get the average wrong and your bill hurts. Get the tail wrong and your users leave. This post is about the tail.

Where the time actually goes

For a typical retrieval path, latency splits into three buckets: embedding the query, searching the index, and everything else (serialization, network, auth). People spend weeks tuning the index and ignore that the embedding call is often the long pole under load.

Batch, but bound the wait

GPUs love batches. Users hate waiting for one to fill. The trick is a micro-batch window with a hard cap, so you get throughput without punishing the unlucky request that arrives when the queue is empty.

async def embed(text: str) -> list[float]:
    fut = loop.create_future()
    queue.append((text, fut))
    # flush when the batch is full OR the window closes, whichever first
    schedule_flush(max_wait_ms=8, max_batch=32)
    return await fut

Eight milliseconds is usually invisible to a human and plenty to fill a batch under real traffic. The max_batch cap keeps any single flush from blowing up memory.

Cache the boring queries

Query distributions are rarely uniform. A small cache keyed on the normalized query text absorbs the head of the distribution and takes real load off the model.

• Normalize aggressively: lowercase, collapse whitespace, strip trailing punctuation.
• Cache the embedding, not the final answer, so downstream ranking can still change.
• Set a TTL you can defend. Stale embeddings are usually fine. Stale answers are not.

Measure the tail, not the mean

Alert on p99, not average. A healthy mean with a rotten p99 is a system that works in the demo and fails in production. The tail is where the users you most want to keep are waiting.