Logo
Login Contact Us

Amish Kushwaha - August 7, 2026

llmrouter Part 3: Middleware, Retry, Timeout, and Circuit Breaking

Part of the llmrouter Series

Production LLM applications fail in predictable ways. Providers rate-limit at peak usage. An endpoint goes cold and throws 503s for thirty seconds. A slow request holds a connection open longer than your client is willing to wait. None of this is surprising, it’s the same category of failure you’d handle for any remote dependency, except LLM providers tend to have less consistent SLAs and messier error semantics than a typical internal service.

The instinct is to write retry logic directly into the calling code:

// The instinct: retry logic bolted onto the call site
func callWithRetry(client *openai.Client, req openai.ChatRequest) (*openai.Response, error) {
    for attempt := 0; attempt < 3; attempt++ {
        resp, err := client.Chat(ctx, req)
        if err == nil {
            return resp, nil
        }
        var apiErr *openai.APIError
        if errors.As(err, &apiErr) && apiErr.StatusCode == 429 {
            time.Sleep(time.Second * time.Duration(attempt+1))
            continue
        }
        return nil, err
    }
    return nil, fmt.Errorf("max retries exceeded")
}

That handles one status code, on one provider. No circuit breaker, no distinction between “the caller gave up” and “the server had a bad second.” Add Anthropic and you write a version of this again with different error types. Add Gemini, again.

llmrouter moves this into the infrastructure layer once, as composable middleware.

MiddlewareFunc: A Function Type, Not an Interface 

type MiddlewareFunc func(Provider) Provider

Middleware is a function that takes a Provider and returns a Provider. To add behavior, you wrap the incoming provider in something that does the extra work and then delegates to it. That’s the whole contract.

The reason this works as a plain function type instead of an interface with a Handle method is composability. Chaining is nothing more than function composition, applied in Router.buildChain, which wraps a provider with each configured middleware in order.

The practical payoff is testability. Custom middleware is a pure function: given a Provider, return a Provider. You can unit test it by handing it a mock provider and inspecting what the wrapped one does, no router, no HTTP, no credentials required.

Middleware Order

Middleware declared in WithMiddleware wraps in declaration order. The first one listed is the outermost layer, it sees the request first and the response last:

llmrouter.WithMiddleware(
    middleware.Retry(3, time.Second), // outermost
    cb.Wrap,                          // middle
    middleware.Timeout(60*time.Second), // innermost
)
Request  → Retry → CircuitBreaker → Timeout → Provider
Response ← Retry ← CircuitBreaker ← Timeout ← Provider

The ordering matters. Retry sits outside the circuit breaker so each retry attempt also passes through (and counts against) the breaker, a provider throwing five consecutive 503s trips the breaker and stops further retries from even trying. Timeout sits innermost so each individual attempt gets its own fresh deadline rather than sharing one deadline across every retry. Put timeout outside retry and a single slow attempt can burn the whole budget before a second attempt ever starts.

Retry Middleware 

mw := middleware.Retry(3, time.Second)

Three total attempts, one second base delay. Backoff between attempts is exponential: baseDelay * 2^(attempt-1), capped at a default max delay of 30 seconds (override with WithMaxDelay). With Retry(3, time.Second) specifically, that’s the first attempt immediately, then a 1s wait before the second, then a 2s wait before the third, three attempts total, two delays.

mw := middleware.Retry(
    3,
    time.Second,
    middleware.WithMaxDelay(30*time.Second),
    middleware.WithRetryFunc(customShouldRetry),
)

Classification, Not Just Backoff

The hard part of retry logic is knowing when to bother. llmrouter.IsRetryable(err) is the classifier the retry middleware uses by default:

Retryable:

  • 429 Too Many Requests
  • 500 Internal Server Error
  • 502 Bad Gateway
  • 503 Service Unavailable
  • 504 Gateway Timeout

Not retried:

  • 400 Bad Request: your payload is malformed; retrying sends the same malformed payload again
  • 401 Unauthorized, 403 Forbidden: credentials or permissions, retrying doesn’t fix either
  • context.Canceled / context.DeadlineExceeded: the caller already gave up

Getting the 401 case wrong is the one that actually bites people. A rotated key that didn’t get updated everywhere fails every request with 401. Without that non-retryable classification, a three-attempt retry middleware would make three failing authenticated calls before surfacing the same error that was obvious on attempt one, burning latency and quota for nothing.

func customShouldRetry(err error) bool {
    if errors.Is(err, ErrBackendBusy) {
        return true
    }
    return llmrouter.IsRetryable(err) // delegate for everything else
}

IsRetryable is exported, so you can call it directly anywhere you’re building your own error handling outside the retry middleware too.

Timeout Middleware 

mw := middleware.Timeout(60 * time.Second)

Wraps each Complete or Stream call in context.WithTimeout. Combined with retry (timeout nested inside retry, as shown above), every individual attempt gets its own 60-second budget rather than sharing one across the whole retry sequence.

For streaming, the deadline is applied to the context used for the underlying provider call, and the timeout middleware hooks its cancel() into the StreamResult’s close handlers so resources get released whether the stream finishes normally or the caller walks away early. If the deadline fires while a stream is still open, the failure surfaces through the normal channel, stream.Next() eventually returns false and stream.Err() carries the error, rather than the loop hanging.

stream, err := router.Stream(ctx, req)
if err != nil {
    return err
}
defer stream.Close()

for stream.Next() {
    event := stream.Event()
    // handle event
}

if err := stream.Err(); err != nil {
    return err // check this, not just the initial error from Stream()
}

Circuit Breaker 

cb := middleware.NewCircuitBreaker(5, 30*time.Second)

Five consecutive failures trips it open, thirty seconds before it allows a recovery probe. Once open, requests fail immediately with ErrCircuitOpen, no network call made, sparing you from piling retries on a provider you already know is down.

The State Machine

Closed: normal operation, failures are counted.

Open: every request rejected immediately with ErrCircuitOpen until the configured timeout elapses.

HalfOpen: once the timeout passes, a limited number of probe requests (up to the same failure count that tripped it) are let through. The first one to succeed closes the circuit and resets the failure count. The first one to fail reopens it and restarts the timeout.

Observability 

state := cb.State() // middleware.CBStateClosed, CBStateOpen, or CBStateHalfOpen

Expose this on a health check or metrics endpoint. A breaker sitting in CBStateOpen for a specific provider is directly actionable, it tells you exactly which upstream is degraded without waiting for a support ticket.

Making It Actually Per-Provider

Here’s a detail that’s easy to get wrong. If you build one circuit breaker and pass its Wrap method into WithMiddleware, that single breaker’s state gets applied to every provider the router resolves to, because WithMiddleware middleware wraps whichever provider is currently being called, but they all close over the same underlying breaker instance:

// This shares ONE breaker's failure count across every provider
cb := middleware.NewCircuitBreaker(5, 30*time.Second)
router := llmrouter.New(
    llmrouter.WithProvider("anthropic", anthropic.NewFromEnv()),
    llmrouter.WithProvider("deepseek",  openai.NewFromEnv("deepseek", "DEEPSEEK_API_KEY")),
    llmrouter.WithMiddleware(cb.Wrap), // same cb for both providers
)

Five consecutive failures on Anthropic would trip the same breaker that deepseek’s requests also pass through, even though deepseek was never the problem.

To get independent failure tracking per provider, give each provider its own breaker and wrap it at registration time instead of going through global middleware:

anthropicCB := middleware.NewCircuitBreaker(5, 30*time.Second)
deepseekCB  := middleware.NewCircuitBreaker(5, 30*time.Second)

router := llmrouter.New(
    llmrouter.WithProvider("anthropic", anthropicCB.Wrap(anthropic.NewFromEnv())),
    llmrouter.WithProvider("deepseek",  deepseekCB.Wrap(openai.NewFromEnv("deepseek", "DEEPSEEK_API_KEY"))),
    llmrouter.WithFallback("anthropic", "deepseek"),
    llmrouter.WithMiddleware(
        middleware.Retry(3, time.Second),
        middleware.Timeout(60*time.Second),
    ),
)

Now a degraded Anthropic trips only anthropicCB. Requests fall through to the deepseek fallback, which has its own clean failure history and keeps serving traffic.

Writing Custom Middleware

Because it’s just a function over Provider, custom middleware is straightforward to write. Here’s one that logs the outcome of every request:

func RequestLogger(logger *slog.Logger) llmrouter.MiddlewareFunc {
    return func(p llmrouter.Provider) llmrouter.Provider {
        return &loggingProvider{inner: p, logger: logger}
    }
}

type loggingProvider struct {
    inner  llmrouter.Provider
    logger *slog.Logger
}

func (l *loggingProvider) Name() string     { return l.inner.Name() }
func (l *loggingProvider) Models() []string { return l.inner.Models() }

func (l *loggingProvider) Complete(ctx context.Context, req *llmrouter.Request) (*llmrouter.Response, error) {
    resp, err := l.inner.Complete(ctx, req)
    if err != nil {
        l.logger.Error("llm_request_failed", "provider", l.inner.Name(), "error", err)
        return nil, err
    }
    l.logger.Info("llm_request",
        "provider", resp.Provider,
        "model",    resp.Model,
    )
    return resp, nil
}

func (l *loggingProvider) Stream(ctx context.Context, req *llmrouter.Request) (*llmrouter.StreamResult, error) {
    return l.inner.Stream(ctx, req)
}

Register it like any other middleware:

logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))

router := llmrouter.New(
    llmrouter.WithProvider("openai", openai.NewFromEnv("openai", "OPENAI_API_KEY")),
    llmrouter.WithMiddleware(
        RequestLogger(logger),           // outermost: logs the final outcome
        middleware.Retry(3, time.Second),
        cb.Wrap,
        middleware.Timeout(60*time.Second),
    ),
)

Placed outside retry, it logs once per logical request rather than once per attempt, exactly what you want for an outcome log.

Testing Middleware in Isolation

Since middleware is a function over Provider, you can test it against a mock without touching a real API:

type mockProvider struct {
    errors    []error
    responses []*llmrouter.Response
    callCount int
}

func (m *mockProvider) Name() string     { return "mock" }
func (m *mockProvider) Models() []string { return []string{"mock-model"} }

func (m *mockProvider) Complete(ctx context.Context, req *llmrouter.Request) (*llmrouter.Response, error) {
    i := m.callCount
    m.callCount++
    if i < len(m.errors) && m.errors[i] != nil {
        return nil, m.errors[i]
    }
    return m.responses[i], nil
}

func (m *mockProvider) Stream(ctx context.Context, req *llmrouter.Request) (*llmrouter.StreamResult, error) {
    return nil, nil
}

func TestRetryOnRateLimit(t *testing.T) {
    mock := &mockProvider{
        errors: []error{
            &llmrouter.APIError{StatusCode: 429},
            &llmrouter.APIError{StatusCode: 429},
            nil,
        },
        responses: []*llmrouter.Response{
            nil,
            nil,
            {Model: "mock-model"},
        },
    }

    wrapped := middleware.Retry(3, 0)(mock) // zero base delay for a fast test

    _, err := wrapped.Complete(t.Context(), &llmrouter.Request{
        Model:    "mock-model",
        Messages: []llmrouter.Message{{Role: llmrouter.RoleUser, Content: "hi"}},
    })

    if err != nil {
        t.Fatalf("expected success after retries, got: %v", err)
    }
    if mock.callCount != 3 {
        t.Errorf("expected 3 calls, got %d", mock.callCount)
    }
}

No HTTP, no API key, no rate limit surprises in CI. The retry behavior is verified against a mock that returns a controlled sequence of errors.

The next part moves up the stack, streaming responses and tool calling, which cover most of what a production AI application actually needs day to day: Part 4: Streaming and Tool Calling Across Every Provider.

Share this article
LinkedIn

Related Articles