Amish Kushwaha - August 11, 2026

llmrouter Part 5: Cost Tracking, Prompt Caching, and Production Observability

Part of the llmrouter Series

• Part 1: LLM Routing Is Infrastructure, Not Application Logic

• Part 2: Providers — Wiring Up OpenAI, Anthropic, Gemini, and Five More

• Part 3: Middleware, Retry, Timeout, and Circuit Breaking

• Part 4: Streaming and Tool Calling Across Every Provider

• Part 5: Cost Tracking, Prompt Caching, and Production Observability

Most LLM applications in production have the same blind spot: nobody actually knows what they’re spending, until the invoice shows up. Tokens go in, a response comes out, and cost calculation gets pushed to “check the billing dashboard later,” because doing it properly means tracking token counts, per-model pricing, and cache state at the request layer, and that’s annoying enough that it gets skipped.

“Later” is too late for catching a bug. And a provider’s billing dashboard won’t give you per-endpoint or per-user breakdowns anyway, so even when you do check, you’re not getting the number you actually need.

llmrouter computes cost at the call site instead. Every Complete response, and every EventDone on a stream, carries Usage.Cost, calculated from the real token counts and a built-in pricing table. This post covers how that works, where it can quietly return zero, and how prompt caching changes the math.

Cost Calculation Without a Router

What this usually looks like when you’re calling a provider directly:

resp, err := openaiClient.Chat(ctx, openai.ChatRequest{Model: "gpt-5.6-sol", Messages: messages})
if err != nil {
    return err
}

inputTokens  := resp.Usage.PromptTokens
outputTokens := resp.Usage.CompletionTokens

// hardcoded, and wrong the moment OpenAI changes pricing
const gpt56SolInputPerM  = 5.00
const gpt56SolOutputPerM = 30.00

cost := float64(inputTokens)/1e6*gpt56SolInputPerM + float64(outputTokens)/1e6*gpt56SolOutputPerM
log.Printf("cost: $%.4f", cost)

Pricing constants hardcoded per model, reimplemented for every provider you add, silently wrong the day pricing changes. With llmrouter, the pricing table lives in the library and the calculation runs automatically on every response:

resp, err := router.Complete(ctx, req)
if err != nil {
    return err
}
log.Printf("cost: $%.4f | provider: %s | cache_hit_rate: %.1f%%",
    resp.Usage.Cost,
    resp.Provider,
    resp.Usage.CacheHitRate()*100,
)

Nothing to maintain in your application code.

How the Pricing Table Works

llmrouter ships DefaultPrices, a table covering eighteen models, priced per million tokens with separate input, output, and cache-read rates:

type ModelPrice struct {
    InputPerMillion     float64
    OutputPerMillion    float64
    CacheReadPerMillion float64 // 0 if the provider doesn't offer cache pricing
}

CalculateCost looks up the response’s exact model string in this table, bills uncached prompt tokens at the input rate, cached prompt tokens at the cache-read rate, and completion tokens at the output rate. No match, no error, Cost is just 0.

That lookup being an exact string match is worth sitting with for a second. The Gemini provider’s own default model list (gemini-1.5-pro, gemini-1.5-flash, gemini-2.0-flash-exp, gemini-1.0-pro, see Part 2) doesn’t actually overlap with the Gemini entries in DefaultPrices above at all. Run Gemini through llmrouter using its default models and Usage.Cost will silently sit at 0.00 all day, not because caching or pricing is broken, but because the model string the provider returns just isn’t a key in the pricing map. If cost tracking suddenly reads zero for a provider, checking for an exact string mismatch here is the first thing to try, not a caching problem. The fix is a custom price table (next section) with entries matching whatever resp.Model actually comes back as, gemini-3.1-pro-preview and friends included.

Prompt Caching

For applications with repetitive context, a long system prompt, a document, tool schemas, few-shot examples, caching is the single biggest lever on cost. The mechanics differ by provider.

Anthropic: Explicit CacheControl

Anthropic’s caching is opt-in per message. Setting CacheControl on a message tells Anthropic to write everything up to that point into a short-lived cache; a later request sharing the same prefix reads from it instead of paying full price.

systemPrompt := `You are an expert ABAP developer with deep knowledge of SAP systems.
[... a couple thousand tokens of standards, patterns, and debugging guidance ...]`

messages := []llmrouter.Message{
    {
        Role:         llmrouter.RoleSystem,
        Content:      systemPrompt,
        CacheControl: &llmrouter.CacheControl{Type: "ephemeral"},
    },
    {Role: llmrouter.RoleUser, Content: "How do I optimize this SELECT statement?"},
}

resp, err := router.Complete(ctx, &llmrouter.Request{
    Model:    "claude-sonnet-5",
    Messages: messages,
})

fmt.Printf("cache creation tokens: %d\n", resp.Usage.CacheCreationTokens)
fmt.Printf("cached tokens served:  %d\n", resp.Usage.CachedPromptTokens)
fmt.Printf("cache hit rate:        %.1f%%\n", resp.Usage.CacheHitRate()*100)
fmt.Printf("cost:                  $%.4f\n", resp.Usage.Cost)

First request, a cache write: CacheCreationTokens is non-zero, CachedPromptTokens is zero, you pay full price for the system prompt plus a write premium (1.25x base input for a 5-minute cache, 2x for a 1-hour cache). Later requests that hit the same cached prefix: CachedPromptTokens is non-zero, billed at CacheReadPerMillion, $0.20/M instead of $2.00/M for claude-sonnet-5, a 90% reduction on those tokens specifically.

Change the system prompt and you write a new cache entry; the old one just goes stale and stops matching.

OpenAI and Gemini: Automatic

OpenAI (gpt-5.6/gpt-5.4 family) and Gemini cache automatically, no annotation required. Once a request’s prefix passes roughly 1024 tokens and matches a recent request, the provider serves it from cache on its own and resp.Usage.CachedPromptTokens reflects it. You don’t control which messages get cached, so keep your stable content, system prompt, tool schemas, reference documents, at the start of the message array. A cache is more likely to match a prefix that starts at position zero.

CacheHitRate()

func (u *Usage) CacheHitRate() float64

Returns CachedPromptTokens / PromptTokens, the fraction of prompt tokens actually served from cache. 0.0 means every request is paying full price for every token, worth investigating if you have a static system prompt and expected better. A hit rate above 0.8 on Anthropic is realistic for a chatbot with a long, stable system prompt, the first turn in a session pays full price, later turns in the same session hit the cache.

Custom Price Table

If you’re on a negotiated or volume-discounted rate, replace the default table with WithPriceTable. One easy mistake here: Go maps are reference types, so customPrices := llmrouter.DefaultPrices doesn’t copy anything, it hands you a second name for the exact same map, and mutating it mutates the library’s global default for every other consumer in the process. Copy it explicitly instead:

customPrices := make(map[string]llmrouter.ModelPrice, len(llmrouter.DefaultPrices))
for k, v := range llmrouter.DefaultPrices {
    customPrices[k] = v
}
customPrices["claude-sonnet-5"] = llmrouter.ModelPrice{
    InputPerMillion:     1.60, // negotiated rate
    OutputPerMillion:    8.00,
    CacheReadPerMillion: 0.16,
}

router := llmrouter.New(
    llmrouter.WithProvider("anthropic", anthropic.NewFromEnv()),
    llmrouter.WithPriceTable(customPrices),
)

Note that WithPriceTable fully replaces the default table rather than merging into it, so make sure every model you actually route to has an entry, or its cost will read as 0 the same way the Gemini mismatch does above.

Cost Aggregation Middleware

Per-request cost is good for debugging. For production, you want it aggregated, by model, by endpoint, over time. Building on the custom middleware pattern from Part 3:

type CostAggregator struct {
    mu      sync.Mutex
    total   float64
    byModel map[string]float64
}

func (a *CostAggregator) Add(resp *llmrouter.Response) {
    if resp.Usage == nil {
        return
    }
    a.mu.Lock()
    defer a.mu.Unlock()
    a.total += resp.Usage.Cost
    a.byModel[resp.Model] += resp.Usage.Cost
}

func (a *CostAggregator) Report() map[string]any {
    a.mu.Lock()
    defer a.mu.Unlock()
    return map[string]any{"total_cost_usd": a.total, "by_model": a.byModel}
}

func CostTracking(agg *CostAggregator) llmrouter.MiddlewareFunc {
    return func(p llmrouter.Provider) llmrouter.Provider {
        return &costTrackingProvider{inner: p, agg: agg}
    }
}

type costTrackingProvider struct {
    inner llmrouter.Provider
    agg   *CostAggregator
}

func (c *costTrackingProvider) Name() string     { return c.inner.Name() }
func (c *costTrackingProvider) Models() []string { return c.inner.Models() }

func (c *costTrackingProvider) Complete(ctx context.Context, req *llmrouter.Request) (*llmrouter.Response, error) {
    resp, err := c.inner.Complete(ctx, req)
    if err == nil {
        c.agg.Add(resp)
    }
    return resp, err
}

func (c *costTrackingProvider) Stream(ctx context.Context, req *llmrouter.Request) (*llmrouter.StreamResult, error) {
    return c.inner.Stream(ctx, req)
    // Streaming cost lands on EventDone, not at the point Stream() returns.
    // Aggregating stream cost means wrapping the StreamResult to intercept
    // that event rather than hooking Complete alone.
}

Wire it in and expose it:

agg := &CostAggregator{byModel: make(map[string]float64)}

router := llmrouter.New(
    llmrouter.WithProvider("openai",    openai.NewFromEnv("openai", "OPENAI_API_KEY")),
    llmrouter.WithProvider("anthropic", anthropic.NewFromEnv()),
    llmrouter.WithMiddleware(
        CostTracking(agg),
        middleware.Retry(3, time.Second),
        middleware.NewCircuitBreaker(5, 30*time.Second).Wrap,
        middleware.Timeout(60*time.Second),
    ),
)

http.HandleFunc("/metrics/llm", func(w http.ResponseWriter, r *http.Request) {
    json.NewEncoder(w).Encode(agg.Report())
})

What to Log Per Request

slog.Info("llm_request",
    "request_id",          requestID,
    "model",                resp.Model,
    "provider",             resp.Provider,
    "input_tokens",         resp.Usage.PromptTokens,
    "output_tokens",        resp.Usage.CompletionTokens,
    "cached_tokens",        resp.Usage.CachedPromptTokens,
    "cache_creation_tokens", resp.Usage.CacheCreationTokens,
    "cache_hit_rate",       resp.Usage.CacheHitRate(),
    "cost_usd",             resp.Usage.Cost,
    "endpoint",             r.URL.Path,
    "user_id",              userID,
)

With that structured log in place, questions like “which endpoint is burning the most” or “is caching actually working on this system prompt” become a query instead of a guess, group by endpoint or user_id and sum cost_usd, filter to a provider and plot cache_hit_rate over time.

Cost Comparison: 1M Input, 200k Output Tokens

A rough document-analysis-shaped workload, computed directly from DefaultPrices:

With Anthropic caching at a realistic 85% hit rate on the input side, claude-sonnet-5’s effective input cost drops to 0.85 × $0.20 + 0.15 × $2.00 ≈ $0.47/M instead of the full $2.00/M. On this workload that brings the total to roughly $0.47 + $2.00 ≈ $2.47, landing it just above claude-haiku-4-5-20251001 rather than sitting at the full baseline price. Caching changes which row of this table you’re actually competing against.

This table isn’t an argument for always picking the cheapest row. It’s an argument for knowing the number before you pick, and routing deliberately once you do.

Where This Series Leaves You

Across these five posts: providers and how model resolution actually decides where a request goes (Part 2), retry, timeout, and circuit breaking as composable middleware (Part 3), streaming and tool calling with one interface across providers (Part 4), and cost visibility with prompt caching (this post). If you’re starting from scratch, Part 1 lays out why routing belongs at the infrastructure layer in the first place.

A few of these pieces are relatively recent additions to the library. Prompt caching support landed early, cost calculation and CacheHitRate() came somewhat later, and the opt-in RoutingPolicy layer from Part 1, for complexity- or outcome-aware model selection when static resolution isn’t enough, is the newest of the bunch. The static, deterministic core described across this series has stayed stable the whole way through, everything since has been additive.

The library is Apache 2.0, open source, written in Go, at github.com/bluefunda/llmrouter. Issues, contributions, and pushback on any of this are genuinely welcome.