Amish Kushwaha - August 6, 2026

llmrouter Part 2: Providers: Wiring Up OpenAI, Anthropic, Gemini, and Five More

Part of the llmrouter Series

• Part 1: LLM Routing Is Infrastructure, Not Application Logic

• Part 2: Providers — Wiring Up OpenAI, Anthropic, Gemini, and Five More

• Part 3: Middleware, Retry, Timeout, and Circuit Breaking

• Part 4: Streaming and Tool Calling Across Every Provider

• Part 5: Cost Tracking, Prompt Caching, and Production Observability

A router is only useful if the providers worth using are actually supported. This post walks through every provider llmrouter ships with, how each one is instantiated, what models it exposes, and how the router decides which provider handles a given request.

There’s no wrapper around a wrapper here. Each provider is a thin adapter over the official SDK for that service, OpenAI’s Go SDK, Anthropic’s Go SDK, Google’s generative AI library. You get retry, circuit breaking, and fallback without losing access to how each provider actually behaves.

The Provider Inventory

That’s eight provider identities across three packages. All of them implement the same interface:

type Provider interface {
    Name() string
    Models() []string
    Complete(ctx context.Context, req *Request) (*Response, error)
    Stream(ctx context.Context, req *Request) (*StreamResult, error)
}

That uniformity is what makes everything else in this series work.

OpenAI and the OpenAI-Compatible Presets

DeepSeek, Groq, Together AI, Ollama, and Sarvam all speak an API close enough to OpenAI’s chat completions format that llmrouter handles them through one adapter package, providers/openai, with a built-in Presets table of base URLs and default models.

import "github.com/bluefunda/llmrouter/providers/openai"

openaiProvider   := openai.NewFromEnv("openai",   "OPENAI_API_KEY")
deepseekProvider := openai.NewFromEnv("deepseek", "DEEPSEEK_API_KEY")
groqProvider     := openai.NewFromEnv("groq",     "GROQ_API_KEY")
togetherProvider := openai.NewFromEnv("together", "TOGETHER_API_KEY")
ollamaProvider   := openai.NewFromEnv("ollama",   "") // no key needed

NewFromEnv(name, envKey) is the same function for every one of these. The first argument has to match a key in the Presets map (that’s how it finds the base URL and default model), and the second is the environment variable holding the key. There are also named convenience constructors if you’d rather not think about the string, openai.NewOpenAI(key), openai.NewDeepSeek(key), openai.NewGroq(key), openai.NewTogether(key), openai.NewOllama(baseURL).

Each preset’s model list, as it actually ships today:

Ollama is the local development case. No API key required, no billed request. The preset points at http://localhost:11434/v1/. Anything you’ve pulled with ollama pull is addressable by its Ollama model name in a request, even though the preset’s static model list is empty.

Sarvam needs one extra step. Its API doesn’t accept OpenAI’s array-of-parts content format for plain text messages, so the adapter has a StringContentOnly flag that flattens single-part text messages down to a plain string. This isn’t set automatically by the preset, you have to construct the provider directly instead of using NewFromEnv:

sarvamProvider := openai.New(llmrouter.ProviderConfig{
    Name:              "sarvam",
    APIKey:            os.Getenv("SARVAM_API_KEY"),
    StringContentOnly: true,
})

Worth knowing: StringContentOnly only collapses the simple case, a message with exactly one text content part. It doesn’t validate or block anything, it’s not a safety check, it just controls serialization. If you send Sarvam a multimodal message with an image part, llmrouter will still build the structured content array and send it, it isn’t something the library actively rejects on Sarvam’s behalf. Whether Sarvam’s API accepts that is between you and Sarvam.

Anthropic

import "github.com/bluefunda/llmrouter/providers/anthropic"

p := anthropic.NewFromEnv()

Name() is hardcoded to "anthropic". The key comes from ANTHROPIC_API_KEY.

The model list is not a set of clean aliases, it’s Anthropic’s real dated snapshot IDs:

claude-opus-5
claude-sonnet-5
claude-fable-5
claude-sonnet-4-6
claude-haiku-4-5-20251001
claude-opus-4-5
claude-haiku-3-5-20241022

This matters for two reasons. First, if you leave Request.Model empty (or set it to "anthropic"), the provider falls back to whatever model you configured as its default when you constructed it, claude-sonnet-5 unless you override it. Second, for anything else, whatever string you put in Request.Model is sent to Anthropic exactly as written. WithModelMapping decides which provider handles a name, it does not rewrite the model string on the way out. Note that these aren’t uniformly dated snapshot IDs anymore, the newest flagships (claude-opus-5, claude-sonnet-5, claude-fable-5) are plain aliases, while older and interim releases (claude-haiku-4-5-20251001, claude-haiku-3-5-20241022) still carry a date suffix. If you request "claude-haiku-4-5" without that suffix, that string goes straight to Anthropic’s API as-is, not the resolved snapshot. So for Anthropic in particular, use the real, exact model ID in your requests unless you’re deliberately relying on the provider’s configured default.

Gemini

import "github.com/bluefunda/llmrouter/providers/gemini"

p, err := gemini.NewFromEnv()
if err != nil {
    log.Fatal(err)
}

Note the second return value. Constructing the Gemini provider can fail (it builds a live gRPC client as part of construction), so NewFromEnv returns (*Provider, error), not just a provider.

Default models: gemini-2.5-pro, gemini-2.5-flash, gemini-2.0-flash, gemini-2.0-flash-exp.

The Gemini provider talks gRPC rather than HTTP, invisible to your application code, but it has one operational consequence: it holds an open connection that needs to be released. The Provider type implements io.Closer, and Router.Close() will call Close() on any registered provider that implements it:

router := llmrouter.New(/* ... */)
defer router.Close()

Skip this in a long-running service that reinitializes providers and you leak a connection each time. It’s harmless in a short-lived CLI, but get in the habit anyway.

The Three-Step Model Resolution Order

When you call router.Complete(ctx, req) with a model name and no RoutingPolicy configured, resolution runs a fixed, deterministic sequence:

1. Explicit mapping: check WithModelMapping registrations for this exact string.
2. Provider name match: check if the string equals a registered provider’s name.
3. Models() scan: walk registered providers in the order they were registered, and use the first one whose Models() list contains the string.

If none of the three hit, you get ErrUnknownModel immediately, before any network call. No fuzzy matching, no scoring, nothing implicit. (You can opt into scoring-based selection with a RoutingPolicy, covered in Part 1, but it’s off by default and none of this series assumes it.)

router := llmrouter.New(
    llmrouter.WithProvider("openai",    openai.NewFromEnv("openai", "OPENAI_API_KEY")),
    llmrouter.WithProvider("anthropic", anthropic.NewFromEnv()),

    llmrouter.WithModelMapping("gpt-5.6-sol",   "openai"),
    llmrouter.WithModelMapping("claude-sonnet-5", "anthropic"),
)

In this example the explicit mappings are actually redundant with step 3, both strings already appear in their provider’s Models() list. Where WithModelMapping earns its keep is disambiguation: if two registered providers both happen to expose a model with the same name, the scan in step 3 would silently pick whichever was registered first. An explicit mapping makes the choice deterministic and visible in your config instead of depending on registration order.

Fallback Strategy

WithFallback registers providers to try, in order, if the primary fails:

llmrouter.WithFallback("anthropic", "deepseek"),

One detail worth being precise about: fallback triggers on any error the primary returns after its middleware chain gives up, not just retryable ones. If your retry middleware exhausts its attempts on a 503, that counts. If the primary comes back with a 401 because a key rotated and nobody updated the secret, that also counts, the router will still try the fallback chain. There’s no filtering on error type at the fallback layer itself; retryable-vs-not is entirely the retry middleware’s job (Part 3 covers this). Keep that in mind when choosing a fallback: a misconfigured primary can quietly route all your traffic to the fallback provider until someone notices the bill.

Some practical things worth checking before wiring a fallback pair together:

Capability alignment. If your primary request uses tool calling, make sure the fallback supports it too. Gemini’s tool-choice handling is thinner than OpenAI’s or Anthropic’s (more on that in Part 4), so it’s a weaker fallback for a tool-heavy flow.

Cost alignment. A fallback shouldn’t be dramatically pricier than the primary, or a rough hour for your primary provider turns into a rough invoice.

Multimodal alignment. Sarvam only cleanly handles single-part text messages. It’s a poor fallback for anything that might carry an image.

A Complete Router With All Eight Providers

package main

import (
    "log"
    "os"
    "time"

    "github.com/bluefunda/llmrouter"
    "github.com/bluefunda/llmrouter/middleware"
    "github.com/bluefunda/llmrouter/providers/anthropic"
    "github.com/bluefunda/llmrouter/providers/gemini"
    "github.com/bluefunda/llmrouter/providers/openai"
)

func buildRouter() (*llmrouter.Router, error) {
    geminiProvider, err := gemini.NewFromEnv()
    if err != nil {
        return nil, err
    }

    cb := middleware.NewCircuitBreaker(5, 30*time.Second)

    router := llmrouter.New(
        // First-class providers
        llmrouter.WithProvider("openai",    openai.NewFromEnv("openai", "OPENAI_API_KEY")),
        llmrouter.WithProvider("anthropic", anthropic.NewFromEnv()),
        llmrouter.WithProvider("gemini",    geminiProvider),

        // OpenAI-compatible presets
        llmrouter.WithProvider("deepseek", openai.NewFromEnv("deepseek", "DEEPSEEK_API_KEY")),
        llmrouter.WithProvider("groq",     openai.NewFromEnv("groq", "GROQ_API_KEY")),
        llmrouter.WithProvider("together", openai.NewFromEnv("together", "TOGETHER_API_KEY")),
        llmrouter.WithProvider("ollama",   openai.NewFromEnv("ollama", "")),
        llmrouter.WithProvider("sarvam", openai.New(llmrouter.ProviderConfig{
            Name:              "sarvam",
            APIKey:            os.Getenv("SARVAM_API_KEY"),
            StringContentOnly: true,
        })),

        // Fallback chain
        llmrouter.WithFallback("anthropic", "deepseek"),

        // Middleware, applied to every provider
        llmrouter.WithMiddleware(
            middleware.Retry(3, time.Second),
            cb.Wrap,
            middleware.Timeout(60*time.Second),
        ),
    )

    return router, nil
}

func main() {
    router, err := buildRouter()
    if err != nil {
        log.Fatal(err)
    }
    defer router.Close()
    // ...
}

You won’t need all eight in most applications. This is a reference config. Most services register two or three, a primary, a fallback, and maybe Ollama for local development.

Runtime Provider Management

The router is safe for concurrent use, guarded internally by a sync.RWMutex. You can register providers, remap models, and update fallbacks after startup:

// Register a provider discovered at runtime, e.g. from config
router.RegisterProvider("groq", openai.NewFromEnv("groq", "GROQ_API_KEY"))

// Inspect what's registered
for _, name := range router.Providers() {
    p, _ := router.GetProvider(name)
    log.Printf("%s: %v", name, p.Models())
}

// Remap a model without restarting
router.MapModel("balanced", "openai")

This is useful for multi-tenant services where different tenants carry different provider credentials, and for shifting traffic between providers without a redeploy.

The next part covers middleware, how retry, timeout, and circuit breaking actually compose, what each one protects against, and how to write your own: Part 3: Middleware, Retry, Timeout, and Circuit Breaking.