How Do I Get Better LLM Uptime? A Reliability Playbook for AI in Production

Short answer. Better LLM uptime comes from treating the model as an unreliable dependency and building the pipeline around it accordingly. That means multi-provider routing, aggressive timeouts, hedged requests, structured fallbacks with graceful degradation, a caching layer for prompts and embeddings, circuit breakers per provider, and observability tied to specific model versions rather than a single “AI works” metric. Nine times out of ten the outage is not the model. It is the request path around the model.

This post is the playbook we use when clients ask us why their AI product went down last week, or why the 95th percentile response time is 40 seconds when the average looks fine.

Why LLM uptime is not the same as web-service uptime

A conventional web service has predictable latency, deterministic responses, and failure modes that fit neatly into HTTP status codes. LLMs have none of that.

A single call to GPT-4 or Claude can take 200 milliseconds or 90 seconds depending on prompt length, output length, provider load, and how many other requests are in flight at that moment. The provider itself can return a 200 OK with a completion that is subtly wrong, an incomplete stream, or a refusal. The API can throttle mid-conversation, degrade quality without saying so, or fall over entirely for 30 minutes without warning. And the model version you tested on last month is not the model version you are running against today.

If you treat the LLM API like a REST endpoint, you will spend the next six months explaining to your CEO why “the AI is broken” every other Tuesday. If you treat it like an unreliable dependency you designed around, you can hit three or four nines on user-visible availability with commodity providers.

The rest of this post is how.

The failure modes you actually need to plan for

Before you can improve uptime, you need to be honest about what is failing. In our production incidents across the AI-in-production work we have done for clients (including the sanctions-compliance platform we built for a global financial-crime advisory that is now live across 20+ financial institutions), the vast majority of outages fall into one of six patterns.

Provider-side rate limiting. You built for average load, then a marketing event ran, and now every request is 429ing. Your users see spinners. The model itself is fine; the problem is your account’s per-minute token bucket.

Provider-side latency spikes without status page acknowledgement. OpenAI or Anthropic slows down by 4-8x for 20-40 minutes, does not update their status page, and your p95 tanks. If you had a 15-second client-side timeout, half your users just saw a failure page.

Model version silently changed. You built against gpt-4-turbo or claude-3-opus at some earlier date. The provider quietly rotated the underlying weights or default. Your carefully-tuned prompt now returns malformed JSON 3% of the time. This is not an outage in any provider’s dashboard, but it is an outage in your product’s ability to serve responses.

Context window overrun. Your app grew, average conversations got longer, and now some requests exceed the model’s context window. The provider returns an error you never handled. This looks like a 5% outage to a subset of your users.

Downstream dependency failure. The model call succeeded, but your vector database timed out, your feature-flag service is slow, or your embeddings cache is cold. The user sees a broken app; your LLM logs show 100% success.

Cost-driven throttling. Someone at the provider spikes your account or your engineering team hits an emergency budget cap and pulls the plug on the top-tier model. Requests now go to a smaller model that cannot do the task, and quality craters even though the API returns 200s.

Each of these has a specific mitigation. They do not have a single “just add retries” fix, and treating them as one problem is a large part of why so many LLM apps have miserable uptime.

The playbook: 9 concrete moves that actually improve LLM uptime

1. Route across multiple providers from day one

The single highest-leverage improvement you can make. Wire your app to at least two model providers (typically OpenAI + Anthropic, or one of those plus a hosted open-source model on Together, Fireworks, or your own inference stack). Route new requests based on a health signal: recent latency, recent error rate, cost budget, or a manual override. If provider A is having a rough afternoon, you fail over to provider B within seconds, not the 40 minutes it takes the provider to acknowledge an incident.

The prompt engineering overhead is real but bounded. Most modern prompts port between GPT-4 class and Claude-3 class with minor formatting changes. Build the abstraction once, test it in staging, and never again explain to your CEO that “OpenAI is down” is the reason the product is broken.

2. Set aggressive per-provider timeouts and use hedged requests

The provider will happily leave you hanging for 90+ seconds on a call that should take 3. Set a client-side timeout that is a small multiple of your target p95 (for interactive UX, 8-15 seconds; for background jobs, 30-60). When you time out, either retry against the alternate provider or return a graceful fallback (see below).

For interactive UX, use hedged requests: fire the same query to two providers simultaneously after a delay of, say, 800 ms, and take whichever comes back first. This burns extra tokens but transforms your p99 latency. We use this pattern on user-facing chat interfaces where a two-second response is table stakes.

3. Build structured fallbacks with graceful degradation

When everything upstream fails, what does your product show the user? “AI is unavailable, try again” is the answer that will lose you subscribers. Real graceful degradation means: cached response for common queries, deterministic non-AI response for structured intents you can detect (a rule-based extractor for form fields, a keyword search for retrieval), a slower but more reliable model for the same task, or a clearly-marked “operating in limited mode” state that keeps the app usable.

The best AI products are indistinguishable from non-AI products during an LLM outage. That is the bar to aim at.

4. Cache prompts, responses, and embeddings

Most production LLM workloads have significant repetition. Product descriptions, category classifications, boilerplate responses, common questions. These do not need to hit the model every time. A response cache keyed on (prompt hash, model version, temperature) can absorb 30-60% of traffic on many workloads.

Embeddings caching is even easier and higher-value. Once you have embedded a document or a query, store it. Recomputing embeddings on every request is wasted spend and wasted latency. Redis, Postgres with pgvector, or a purpose-built vector store all work.

5. Circuit breakers, per provider and per model

If Anthropic’s Claude endpoint has returned errors on 20% of the last 100 requests, stop sending traffic to it for 60 seconds. Not because the retries will hurt (they might), but because the queueing will amplify your latency for every subsequent request. A circuit breaker library like Resilience4j (Java), Polly (.NET), or Sentinel (Go) drops in on top of your provider abstraction and prevents cascading failure.

Set the trip threshold conservatively at first. Circuit breakers that trip too eagerly cause their own kind of outage.

6. Version-pin models and test on the pinned version in CI

Every provider gives you a way to pin to a specific model version (gpt-4-0613, claude-3-opus-20240229, etc). Use it. Do not run production traffic against a floating alias like gpt-4 unless you have a very good reason. When you want to upgrade, do it deliberately with a canary rollout and a test suite that verifies your prompts still produce the outputs you expect.

Add representative prompts to your CI. If the model’s output changes in a way that breaks your JSON schema or your extraction logic, you want to find out before your users do.

7. Enforce and observe token budgets per user, per session, per endpoint

Runaway conversations, adversarial users, or a buggy client that retries in a hot loop can burn through your rate limit and knock your entire user base offline. Enforce a per-user token budget and a per-session context window ceiling. Truncate old messages before they push you past the model’s context limit. And track token spend per endpoint so you can find out early which feature is going to bankrupt you before your CFO does.

8. Instrument the model-specific SLOs, not just “the app is up”

Standard APM will tell you your endpoint returned a 200. It will not tell you that the LLM’s completion was empty, malformed, refused, or off-topic. Add application-level SLOs that measure the things you actually care about:

• User-visible latency percentiles at p50, p95, and p99, split by provider and by model version.
• Completion success rate: the fraction of requests where the model returned a completion that parsed correctly against the schema you expected.
• Response quality proxy: a lightweight heuristic (word count in expected range, JSON valid, no obvious refusal phrase) that catches degradation before your users complain.
• Provider error rate per model per hour.
• Fallback trigger rate: how often you had to fall over to provider B, cache, or degraded mode.

If your SLOs do not include response validity, you do not know what your LLM uptime actually is.

9. Run a chaos-engineering game day against your LLM pipeline

Once a quarter, deliberately kill your primary provider in staging and verify the fallback works, the SLOs alert correctly, and the user experience degrades gracefully. If you cannot do this because you have never built the fallback, that is the finding: build one. If you can do it and something breaks, that is a bug you found before your customers did.

The metrics you should be tracking week to week

If you want to hold your team accountable to LLM reliability, the following weekly review is a good baseline. We use a variant of this with clients where reliability is a P0 concern.

• Availability at p99 for user-facing AI endpoints, calculated as: (successful completions that met the response-validity check) / (total requests). Target: 99.5% is the floor for a serious product; 99.9% is achievable with the playbook above.
• p95 and p99 latency per model, per endpoint.
• Fallback trigger rate as a percentage of total traffic. If this is above 5% consistently, you have a primary-provider problem worth escalating.
• Cost per successful completion. An early-warning system for silent quality drift. If cost is climbing and quality is not, someone changed a prompt or the model changed underneath you.
• Time to acknowledge and time to recover for each incident. Aim for TTA under 5 minutes on P0 (you should be alerted before your users complain) and TTR under 30 minutes with fallbacks in place.

When to build this yourself vs bring in help

If your team has done SRE work at scale before, the playbook above is well within reach. You need one senior engineer with a month of focused time to stand up the provider abstraction, the caching, the circuit breakers, and the SLO instrumentation. After that, ongoing maintenance is measured in hours per month.

If your team is a mix of ML engineers and web engineers who have not previously owned production reliability for a stateful service, this work will collide with everything else they are trying to build. That is when clients bring us in. We have a 10-day AI Readiness Audit that ends with a fixed-price plan for exactly the reliability engineering above, and if the right answer is “your team can do this in a month,” we say so.

The sanctions-compliance platform is the reference: an AI-inside-production system where reliability is not optional (financial-institution customers, regulated data, audit trails), built and handed over so the client’s team owns it. The pattern transfers to any regulated or high-stakes LLM product.

We also covered this on the AI Briefing podcast, LLM Uptime Crisis: what happens when AI services like Claude go offline, with real incident examples and the mitigations that actually work.

The short version

The path to better LLM uptime is not “wait for the models to get more reliable.” Models are reliable enough. It is:

1. Two providers, one abstraction.
2. Timeouts and hedged requests.
3. Structured fallbacks that keep the product usable.
4. Cache aggressively.
5. Circuit breakers per provider.
6. Pin model versions and test on the pin.
7. Enforce token budgets.
8. Instrument SLOs that measure completion success, not HTTP 200s.
9. Chaos-test the fallbacks before your customers do it for you.

Do these nine things and three-nines LLM uptime is achievable on commodity APIs. Skip them and you are one provider incident away from a bad Monday.

If you want the version of this scoped to your specific stack and your specific AI product, that is what the AI Readiness Audit is for. Ten days, $8K, ends with a fixed-price plan and the exact SLO instrumentation you need. Or we can just talk about it first.